Polished reports, weaker support
The benchmark's results separate the quality of a report's presentation from the strength of its evidence. Average presentation scores reached 81.8 points, compared with 45.4 for content and 47.8 for citation quality. A dev.to analysis describes the results as a roughly 37% shortfall in grounded reasoning despite strong answer accuracy.
OmniDR-Bench tests a sequence that answer-focused evaluations have not required: finding relevant media, identifying the segment that supports a claim, and explaining how that evidence justifies the finding. Earlier multimodal benchmarks largely assessed object identification, video question answering, or retrieval through measures such as VQA accuracy and recall-based scores, without requiring timestamps or supporting clips.
What agents must produce
According to the OmniDR-Bench official repository, agents must deliver a self-contained, interactive HTML report whose claims can be traced to evidence. Findings must draw on the video's audiovisual material and external multimodal research, with every claim connected to both a citation and a video timestamp.
The dataset includes 50 tasks: 27 in English and 23 in Chinese. These cover four task families and 19 real-world subcategories, drawing on 42 source videos. The repository lists 3,102 verified facts and 6,414 binary rubric criteria, or roughly 128 criteria per task.
Reports are evaluated using Omni Atomic Rubric Trees, or Omni-ARTs. Each tree organizes task-specific yes/no criteria under three dimensions and 23 shared evaluation directions. A multimodal model or a deterministic program/browser probe judges each criterion. Some directions include gates; failing a gate gives that entire direction a score of zero.
Scope and remaining limits
The benchmark is introduced in "Beyond Watching and Searching: Benchmarking Agents for Video-centered Deep Research Reports" (arXiv:2610.31158), submitted to arXiv on October 22, 2026. The authors are Jiahao Meng, Qize Yang, Yuxuan Wang, Yunfei Chu, Wensi Dai, Jin Xu, and Yunhai Tong.
Its evaluation does not yet cover long-term consistency or cross-modal causal reasoning beyond the annotated criteria. The authors identify several remaining needs: coverage of more content, better causal and logical analysis, and more precise citation support.
Tools and access
The official repository makes the report-generation workflow available through the "omnidr" capability in Qwen-MM-Plugins. For evaluation, the harness uses "gpt-5.6-sol" as its default judge model, matching the paper.
The evaluation code carries an MIT License, while the dataset's texts and arrangement are licensed under CC BY-NC 4.0. Source video and audio files are not included; users must obtain those independently.
