flamingo-pytorch

Why we included this project

Researchers and engineers who want to experiment with vision-language models will find this a handy PyTorch reimplementation of DeepMind's Flamingo architecture. Instead of a single monolithic model, it splits the design into a perceiver resampler that condenses long media sequences into a compact set of learned latents, and gated cross-attention blocks that let a language model attend to those images at chosen intervals. A worked example pairs a ViT encoder with a PaLM-style backbone and shows how training can freeze everything except the resampler and cross-attention parts. That makes it a practical starting point for studying few-shot visual question answering or extending the architecture, though it is delivered as composable building blocks rather than a ready-to-deploy application.