contrastors

Why we included this project

This is the training code behind Nomic's embed models, which makes it a useful reference if you are training your own text or image embedding models instead of calling a hosted API. It covers the full pipeline: masked language modeling pretraining, contrastive finetuning, and CLIP-style vision-language alignment, all driven by config files that run across multiple GPUs with DeepSpeed. The Flash Attention integration and GradCache support let teams train with large batch sizes without exhausting GPU memory, and the Matryoshka option trains one model that can serve embeddings at several sizes. Researchers reproducing embedding papers and engineers fine-tuning a retrieval model on their own data will both find the configs and data-loading helpers a solid starting point, even if they end up adapting the code rather than using it as-is.