LLaVA

Why we included this project

LLaVA is the reference implementation of visual instruction tuning, the approach that lets a language model actually see and reason about images. It was presented as an Oral at NeurIPS 2023, and its simple architecture and public data have made it the basis for many later multimodal models. If you're building something that needs to answer questions about screenshots, photos, diagrams, or documents, this repo gives you the training recipe, data pipeline, and model checkpoints to fine-tune a vision-language assistant instead of starting from scratch. The project also ships a model zoo and a demo, so you can try the released checkpoints on your own tasks before committing to training. For teams looking for a proven, well-documented starting point for multimodal understanding, this is the codebase to learn from and build on.