#52 · Primary category： Foundation Models
lit-llama
Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.
Project last updated：07/01/25
GitHub Stars
6.1K
Forks
517
Contributors
35
License
Apache-2.0
Why we included this project
Most people will never need to open this repo, but if you want to see how a LLaMA-class model is actually trained and tuned, the code reads cleanly because it builds on nanoGPT and stays deliberately plain. Pretraining, LoRA and LLaMA-Adapter fine-tuning, generation, flash attention, and Int8 or GPTQ 4-bit quantization all live in readable Python you can trace end to end, under an Apache-2.0 license that sidesteps the GPL restrictions on Meta's original code. The honest caveat is that Lightning has moved on to its LitGPT successor, so this repository no longer gets updates. Treat it as solid study material and a starting point for experiments rather than a supported product.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
CLIP
CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
MiniCPM-V
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
generative-models
Generative Models by Stability AI
unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities