#110 · Primary category： Foundation Models
ru-gpts
Russian GPT3 models.
Project last updated：12/12/22
GitHub Stars
2.1K
Forks
433
Contributors
5
License
Apache-2.0
Why we included this project
Anyone building with Russian-language text gets a solid head start here: a family of pretrained autoregressive transformers ranging from a compact GPT-2 large checkpoint up to the 1.3-billion-parameter ruGPT3XL. Sequence lengths run from 1024 to 2048 tokens, with a mix of dense and sparse attention blocks, so the same weights work for quick text generation experiments and for fine-tuning on domain-specific Russian corpora. Each model comes with Colab notebooks covering both generation and fine-tuning, which makes it easy to judge output quality and training behavior before you integrate anything into your own stack. If you want a Russian-language baseline without pretraining from scratch, this is a practical place to start adapting ready-made weights.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
CLIP
CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
MiniCPM-V
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
generative-models
Generative Models by Stability AI
unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities