StoryTR Benchmark Reveals LLMs Lack Narrative Theory of Mind in Video Retrieval

The current generation of video moment retrieval systems, while adept at identifying explicit actions, demonstrably fails to grasp the underlying narrative causality and implicit intentions within video content. This semantic chasm, attributed to a lack of Theory of Mind (ToM), represents a significant barrier to developing truly intelligent video understanding. The introduction of StoryTR, a novel benchmark specifically designed to test ToM reasoning in narrative short-form videos, exposes this critical deficiency, revealing that even state-of-the-art models struggle with inferring "why" events occur.

StoryTR comprises 8.1k samples from high-information-density narrative videos, providing a challenging testbed where subtle multimodal cues encode complex meaning. Performance metrics highlight the severity of the reasoning gap: Gemini-3.0-Pro, a leading model, achieved a mere 0.53 Avg IoU on the benchmark. To address this, researchers proposed an "Agentic Data Pipeline" that generates training data structured with explicit three-tier ToM chains, encompassing intent decoding, narrative reasoning, and boundary localization. A 7B "Shorts-Moment" model, trained using this ToM-guided data, demonstrated a +15.1% relative IoU improvement over baselines, underscoring that specialized narrative reasoning capabilities are more impactful than raw parameter scale for this task.

The implications extend beyond video retrieval, impacting any AI application requiring nuanced understanding of human behavior and narrative context. The success of the ToM-guided data pipeline suggests a viable pathway for developing AI that can infer mental states and narrative causality, moving beyond surface-level observations. This research indicates a strategic shift towards architecting data generation processes that explicitly encode higher-order cognitive abilities, rather than solely relying on larger models or more data. Overcoming this ToM deficit is crucial for AI's integration into complex human-centric domains, from content moderation to personalized assistance, where understanding implicit meaning is paramount.