CounterVid creates controlled counterfactual videos that differ in action or temporal order—then uses them to teach video-language models to trust visual evidence over language priors.
Dataset and three model checkpoints available now
Tobia Poppi1,2,3,
Burak Uzkent1, Amanmeet Garg1, Lucas Porto1, Garin Kessler1, Yezhou Yang1,
Marcella Cornia2, Lorenzo Baraldi2, Rita Cucchiara2, Florian Schiffers1
The problem
Strong VLMs often infer actions and event order from linguistic plausibility instead of fine-grained motion. CounterVid turns those near-miss cases into supervision.
The scene is right, but the action is wrong. Semantically matched clips force the model to attend to motion rather than static context.
The events are present, but their order is not. Controlled permutations isolate whether the model actually follows the timeline.
Abstract
Video-language models achieve strong multimodal understanding but remain prone to hallucinations, especially when reasoning about actions and temporal order. We introduce a scalable framework that synthesizes videos differing only in action dynamics or temporal structure while preserving scene context. From these controlled counterfactuals, we build CounterVid and train with MixDPO—a unified objective combining textual and visual preferences—to improve grounding and temporal sensitivity across VLM families.
Generation pipeline
One anchor frame becomes a family of plausible, same-scene action videos—semantic hard negatives built for fine-grained visual reasoning.
Coarse-to-fine frame retrieval selects a representative, caption-aligned starting state.
A multimodal LLM suggests plausible, distinct, and visually expressible actions.
Image editing creates end states; image-to-video diffusion connects them coherently.
Clips become action-recognition and temporal-ordering preference pairs at scale.
MixDPO
A grounded model should choose the right answer for a video—and assign that answer to the right video. MixDPO optimizes both directions in one objective.
Prefer a grounded answer over a plausible but visually unsupported answer under the same video.
Prefer the correct visual evidence over a same-scene counterfactual for the same question and answer.
ℒMixDPO = ℒt-pref + λ ℒv-pref
output grounding + input sensitivityCounterVid dataset
26,167 training preference pairs built from synthetic clips with controlled action variations, plus 2,910 held-out examples.
Results
Across Qwen2.5-VL and InternVL3 backbones, MixDPO improves action recognition and temporal ordering, transfers positively to standard video hallucination benchmarks, and largely preserves general video understanding.
Research story
This research was conducted during Tobia Poppi's Applied Scientist internship with the Amazon Prime Video team in Seattle.
Tobia on GitHubCitation
If this work helps your research, please cite the paper.
@inproceedings{poppi2026countervid,
title = {{CounterVid}: Counterfactual Video Generation for
Mitigating Action and Temporal Hallucinations in
Video-Language Models},
author = {Poppi, Tobia and Uzkent, Burak and Garg, Amanmeet and
Porto, Lucas and Kessler, Garin and Yang, Yezhou and
Cornia, Marcella and Baraldi, Lorenzo and Cucchiara,
Rita and Schiffers, Florian},
booktitle = {Proceedings of EMNLP},
year = {2026}
}