(And How We Taught Them to Read the Fine Print)
Vision-language models like CLIP are the backbone of modern AI tools, powering everything from zero-shot image classification to text-to-image generators like Stable Diffusion.
But these models have a blind spot: their training is heavily dominated by images paired with short, simple captions. This biases them toward focusing only on early words and prominent objects, causing them to struggle when aligning complex scenes with dense, paragraph-length descriptions.
In our recent CVPR 2026 highlight paper, we introduce a straightforward fix for this "early-token bias". Instead of adding complex architectures, we simply change how the model learns from data, forcing it to process the full context rather than just skimming the surface.
Exposing biases of CLIP models
Until now, efforts to fix CLIP's inability to process long text generally involved fine-tuning the models on curated, long-caption datasets. However, our research identifies a hidden shortcut these models take: both human- and AI-generated long captions often begin with a one-sentence summary.
Because existing models are already biased toward early words, they latch onto this opening summary to achieve their training objectives, effectively ignoring the rich, detailed descriptions that follow.
We expose this limitation by testing what happens when that crucial first sentence is moved or removed. For example, simply swapping the first and second sentences in a caption (“Move”) severely degrades previous models' abilities to find the matching image. Removing the summary sentence altogether (“Remove”) causes an even more dramatic drop. However, as the figure below demonstrates, while models like Long-CLIP and TULIP see their performance severely degrade in these scenarios, our proposed DeBias-CLIP model stays consistently accurate.
Data-focused approach: remove, sample, and pad
Our model is a drop-in replacement that requires no additional trainable parameters. Rather than changing the model's internal mechanics, we simply change how it consumes text during training using three simple caption-level augmentations:
- Remove: We completely drop the opening summary sentence from the training captions, forcing the model to stop relying on the shortcut and start mapping the image to the fine-grained details hidden deeper in the text.
- Sample: We randomly select a subset of the remaining sentences. This introduces diversity and encourages the model to treat all details as potentially important, regardless of their order.
- Pad: We add blank "padding" tokens to the beginning of the text sequence. This pushes the informative words further back into the model's context window, training the model to effectively extract meaning from later positions in a paragraph.
Ultimately, eliminating these easy shortcuts forces the model to align images with dense, highly detailed text rather than just simple concepts. While previous models like Long-CLIP and SmartCLIP see a drop in attention as they read deeper into a paragraph, our model maintains a steady, consistent focus across every word.
Beyond the summary: unlocking accurate image search
By distributing the model's attention more evenly across all words, DeBias-CLIP achieves state-of-the-art performance for long-text retrieval, while simultaneously maintaining or improving short-text retrieval capabilities. Crucially, it is far more robust than its predecessors. As shown in the example below, when faced with a dense paragraph describing a scene, previous models like Long-CLIP fail to find the right match. In contrast, DeBias-CLIP successfully uses specific details buried in the text, such as the color of the cat’s paws or the chair's armrests, to retrieve the correct image.
Enhancing text-to-image generation
This newfound attention to detail translates directly to better downstream applications. When we swap our DeBias-CLIP model into a text-to-image image generation model like Stable Diffusion, the generated images preserve much finer details from the prompts, such as specific colors, objects, and nuanced interactions, that previous models simply ignore. In the generation example below, DeBias-CLIP captures details that appear late in the prompt more accurately than other models.
Ultimately, our approach empowers researchers and practitioners to enhance any application dependent on strong image-text alignment, from building more precise visual search engines to unlocking richer, more detailed text-to-image generation.
Open Source Release
Project page: https://trailab.github.io/DeBias-CLIP-website/
Arxiv paper version: https://arxiv.org/abs/2602.22419
