#53 · Primary category： Foundation Models
ModernBERT
Bringing BERT into modernity via both architecture changes and scaling
Project last updated：03/01/26
GitHub Stars
1.7K
Forks
144
Contributors
17
License
Apache-2.0
Why we included this project
ModernBERT is the research codebase behind the bidirectional encoder from Answer.AI and LightOn, and it's where you go to reproduce or extend the pre-training pipeline instead of just loading a checkpoint. The repo builds on Composer and drives all experiments through YAML configs, with documented entry points for pre-training and GLUE evaluation, plus loaders for streaming and local datasets. It also includes working examples for training dense Sentence-Transformers models and ColBERT models with PyLate on top of ModernBERT. If you want the production-grade inference integration, that lives in the Hugging Face model collection; this repository is aimed at practitioners who want to train, evaluate, and experiment with encoder models themselves.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
CLIP
CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
MiniCPM-V
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
generative-models
Generative Models by Stability AI
unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities