VLM2Vec

Why we included this project

Most embedding tooling assumes your data is text. VLM2Vec is the reference implementation for the other case: adapting a vision-language model into a dense retriever that handles images, video, audio, visual documents, and agent-centric queries alongside text. It ships the training recipe plus the MMEB benchmark, now at V3 with 190 tasks, which scores how well any embedding model follows modality-specific retrieval instructions. That makes it useful both for training your own model and for evaluating third-party ones before you commit. The OmniSET diagnostic component helps separate failures caused by modality handling from failures in instruction following. It is research code first, so plan to adapt the scripts rather than drop them into production unchanged.