LLaVA-Plus-Codebase

Why we included this project

LLaVA-Plus is the research codebase behind a large multimodal model that learns to use external vision tools rather than trying to handle every image task itself. Given an image and a user instruction, the assistant picks a tool from its skill repository, writes the prompt for that tool, and folds the result back into its answer. Teams building tool-using multimodal agents will find the training data pipeline especially useful, since it curates instruction-following examples for tool use. The repo includes install and training scripts, evaluation code, and a demo, though parts of the release are still being finalized. Note that the dataset and checkpoints are licensed for research use only, so treat this as a design reference rather than a drop-in production service.