Look Twice: Training-Free
Evidence Highlighting
for Knowledge-based Visual Question Answering

A training-free inference-time framework that highlights relevant multimodal evidence, discards distracting context, and improves how pretrained MLLMs answer knowledge-intensive visual questions.

Marco Morini Sara Sarto Marcella Cornia Lorenzo Baraldi Rita Cucchiara

University of Modena and Reggio Emilia

arXiv Paper GitHub Code AImageLab

Abstract

From noisy evidence to precise answers

Knowledge-based Visual Question Answering requires Multimodal Large Language Models (MLLMs) to combine fine-grained visual cues with retrieved textual evidence. However, retrieved documents often contain noisy or only partially relevant information, while images may include distracting visual regions.

We introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT identifies query-relevant image regions and textual sentences without parameter updates or architectural modifications.

The selected visual crop and high-relevance sentences are emphasized through lightweight prompt-level markers. Crucially, LoT also discards sentences whose relevance falls below a lower attention threshold, while retaining intermediate-scoring sentences unchanged. The resulting focused multimodal input is then processed by the same MLLM for final answer generation. Its inference-time design also allows LoT to complement existing KB-VQA strategies based on reasoning, passage filtering, and multimodal re-ranking.

Teaser figure illustrating the Look Twice (LoT) framework

Figure 1. Look Twice (LoT) identifies query-relevant visual and textual evidence, removes low-relevance context, and reformulates the input to guide final answer generation.

+12.5

Maximum average accuracy gain

Parameter updates or fine-tuning

71.93%

Final input reduction on E-VQA (3B)

Models evaluated on 4 KB-VQA benchmarks

Method

A two-pass inference strategy

LoT operates entirely at inference time. The model first generates a single token to expose its internal attention patterns. These signals are used to localize visual evidence and assign sentence-level relevance scores to the retrieved context. The refined input is then constructed by highlighting high-relevance evidence, discarding low-relevance sentences, and retaining intermediate-scoring sentences unchanged before final answer generation.

First pass

Attention Analysis

Generate a single token to read the model's internal attention patterns. Extract object-to-visual attention to locate relevant image regions, and last-to-context attention to score retrieved sentences.

Filtering

Attention Sink Filtering

Identify and suppress spurious visual tokens acting as attention sinks — tokens that attract disproportionate attention regardless of semantic relevance — using hidden-state activation statistics.

Selection

Multimodal Evidence Selection

Aggregate filtered visual attention into a 2D map and extract a bounding box via weighted centroid and spread. In parallel, average textual attention within each retrieved sentence to obtain sentence-level relevance scores.

Second pass

Refined Answer Generation

The selected crop and high-relevance sentences are enclosed by importance markers. Sentences below the dropping threshold are removed, while intermediate-scoring sentences remain as unmarked supporting context.

Figure 2. Overview of LoT: attention-derived visual localization is combined with three-level textual refinement, which highlights relevant sentences, discards low-relevance ones, and preserves intermediate context.

Components

What makes LoT work

🔍

Self-Guided Visual Evidence

Object-conditioned attention between question tokens and visual tokens produces a query-specific relevance map, explicitly capturing how the queried object interacts with the visual input.

🚫

Attention Sink Filtering

Tokens with disproportionately high hidden-state activations in sink dimensions (identified from the base LLM) are suppressed, yielding cleaner grounding maps without modifying the model.

📦

Weighted Centroid Bounding Box

The attention map is converted into a precise spatial region by computing the attention-weighted centroid and standard deviation, outperforming min-max and morphological alternatives.

📝

Self-Guided Textual Evidence

Last-token-to-context attention is aggregated across deep decoder layers and averaged within each sentence. High-relevance sentences are highlighted, low-relevance sentences are discarded, and intermediate-scoring sentences are retained without markers.

Three-level textual refinement. Given sentence-level relevance scores, LoT highlights sentences above α, removes sentences below α_drop, and preserves the remaining sentences as unmarked supporting context.

Qualitative examples of attention sink filtering, showing raw and filtered attention maps

Figure 3. Qualitative examples of attention sink filtering. Raw maps (center) contain scattered activations; filtered maps (right) are tightly concentrated around the target object.

Results

Consistent gains across models & benchmarks

LoT is evaluated on four KB-VQA benchmarks (E-VQA, InfoSeek, OVEN, and ViQuAE) across ten off-the-shelf MLLMs ranging from 2B to 38B parameters. In the training-free setting, LoT improves every evaluated backbone, with average gains ranging from +3.1 to +12.5 accuracy points.

Model	E-VQA All	InfoSeek All	OVEN All	ViQuAE	Avg
Small Models
Qwen2.5-VL-3B	28.0	22.4	11.6	22.9	21.2
+ LoT (Ours)	30.2	30.1	29.5	34.6	31.1 +9.9
InternVL3.5-4B	26.4	29.0	7.9	36.4	24.9
+ LoT (Ours)	31.0	29.5	16.3	41.9	29.7 +4.8
Medium Models
Qwen2-VL-7B	22.9	24.4	11.1	33.0	22.9
+ LoT (Ours)	26.6	33.6	30.5	50.6	35.3 +12.5
Qwen3-VL-8B	34.8	29.7	18.1	43.7	31.6
+ LoT (Ours)	36.3	32.5	19.6	54.0	35.6 +4.0
Large Models
InternVL3.5-38B	31.6	33.1	20.2	51.5	34.1
+ LoT (Ours)	33.4	33.4	24.8	57.6	37.3 +3.2

Table 1. Performance on KB-VQA benchmarks. LoT improves every evaluated backbone without parameter updates or task-specific training.

LoT analysis under different evidence-availability and retrieval-depth settings

Figure 4. Performance on E-VQA under different evidence-availability and retrieval-depth settings. LoT remains beneficial when answer-bearing evidence is retrieved, absent, or provided through oracle context.

Compatibility

Complementary to existing KB-VQA strategies

LoT can be integrated as an inference-time refinement stage within diverse KB-VQA pipelines. We combine it with chain-of-thought prompting, zero-shot and trained passage critics, and specialized multimodal re-rankers, while leaving their underlying components unchanged. Across these settings, LoT provides further gains over already refined inputs, showing that attention-guided multimodal evidence selection remains useful alongside reasoning, filtering, and re-ranking.

Citation

BibTeX

@article{morini2026looktwice,
  title   = {{Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answering}},
  author  = {Morini, Marco and Sarto, Sara and Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita},
  journal = {arXiv preprint arXiv:2604.01280},
  year    = {2026}
}