Transformer-MM-Explainability

Why we included this project

Anyone debugging a Transformer that has to look at images, whether for object detection, visual question answering, or CLIP-style zero-shot tasks, can get real value from seeing where the model actually attends. The repo is the official ICCV 2021 implementation of the Generic Attention-model Explainability method, and it turns attention heads and layers into heatmaps over the input image. It ships runnable notebooks for DETR, LXMERT/VQA, CLIP, and ViT, so you can plug in your own image, and sometimes a free-form question, without building a demo from scratch. Since the method does not depend on a particular architecture, the code is also a decent reference for adding attention-based explanations to a model that none of the included examples cover. Keep expectations in line: this is a research codebase of scripts and notebooks, not a maintained library, so treat it as a starting point rather than production software.