large_concept_model

Why we included this project

Researchers and ML engineers wondering what might follow token-by-token language modeling will find something genuinely different here. This is the official FAIR implementation of the Large Concept Model, a sequence-to-sequence architecture that generates whole sentences at once by predicting in SONAR, a multilingual embedding space covering up to 200 languages for text. Alongside the trained models you get reproducible recipes for pretraining and fine-tuning 1.6B-parameter variants on roughly 1.3T tokens, including both an MSE regression model and a two-tower diffusion approach. Because it works on a language- and modality-agnostic representation rather than individual word pieces, it makes a handy playground for experiments on longer-horizon generation, cross-lingual transfer, and generating text from meaning instead of surface tokens.