Vision-language models like CLIP map images and text into a shared embedding space. This simple objective has turned them into a default building block of modern computer vision, used for zero-shot classification, cross-modal retrieval, and as the perceptual backbone of many generative systems. As they take on this central role, a practical question follows: what has a CLIP model actually learned, and where inside it does that knowledge live?
Mechanistic interpretability answers questions like these by looking past a model’s outputs to the computations that produce them, mapping internal components onto functions a human can understand. In language models the program has been fruitful, uncovering linear representations of semantic attributes, attention heads with identifiable roles, and circuits that carry out specific behaviors. Vision-language models have received far less of this scrutiny, and the methods that do exist mostly share one starting point, which, we argue, limits what they can reveal.
The code is available here.