perception_models

Why we included this project

Meta's latest perception research ships here as one codebase: the Perception Encoder, a CLIP-style family that embeds images, video, and audio into a shared space and reports strong results on tasks like zero-shot classification, retrieval, and dense prediction, plus the Perception Language Model, a multimodal decoder that turns those embeddings into text for question answering. If your pipeline needs a current, well-benchmarked visual encoder for retrieval, captioning, or a multimodal LLM, this saves you training one yourself. The checkpoints are Apache-2.0 and integrate with Hugging Face transformers and timm, so you can probe the small-to-giant size variants or run the Colab demo before committing. Expect research-grade code though: you will be reproducing benchmark setups and adapting things yourself rather than consuming a polished application.