CoCa-pytorch

Why we included this project

Phil Wang's CoCa-pytorch is a research-grade PyTorch implementation of the CoCa architecture, a contrastive captioner that folds contrastive learning into a conventional image-to-text encoder-decoder transformer. The paper reports 91.0% top-1 accuracy on ImageNet after fine-tuning the encoder, and the code follows the PaLM-style transformer layout with parallel SwiGLU feedforwards. If you want to train your own image-text model rather than call a hosted API, the usage examples show how to slot in a pretrained vision transformer and pull out caption logits or CLIP-style embeddings. This is not a pretrained release, so expect to bring your own data and training loop; the OpenClip team has trained CoCa weights you can fine-tune instead. It suits researchers and engineers who prefer seeing exactly how the architecture.