#95 · Primary category： Foundation Models
Otter
🦦 Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
Project last updated：03/05/24
GitHub Stars
3.4K
Forks
210
Contributors
12
License
MIT
Why we included this project
Teams building or fine-tuning vision-language models will find Otter useful as a tested codebase to work from instead of a paper you have to reimplement. It follows OpenFlamingo's Flamingo-style architecture, so one model can take interleaved images and text, and the repo includes training scripts for pretraining, instruction tuning, and RLHF. OtterHD, built on Fuyu-8B, drops the explicit vision encoder and processes high-resolution image patches directly, with a fine-tune script that uses FlashAttention-2. Researchers also get a unified loader for common multimodal datasets and a one-click harness for running GPT-4V evaluation across eight benchmarks. If your work centers on in-context learning over mixed visual and text inputs, this is a solid base to study and adapt.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
CLIP
CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image
MiniCPM-V
A Pocket-Sized MLLM for Ultra-Efficient Image and Video Understanding on Your Phone
generative-models
Generative Models by Stability AI
unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities