NeurIPS 2026 Workshop on Evaluation of Interactive Agents (IAEval)

ReactionBench

A human-anchored benchmark for grounding an observed viewer reaction in the video that caused it.

Yibo Zhao^1,2* Ao Qu^2* Xuan Jiang² Jiahao Xu³ Keane Ong⁴ Hang Jiang⁵ Zhaofeng Wu² Dingyi Zhuang² Yihong Tang⁶ Kaichen Zhou² Jinhua Zhao² Paul Pu Liang²

¹ University of Illinois Urbana-Champaign ² Massachusetts Institute of Technology ³ Independent Researcher ⁴ National University of Singapore ⁵ Northeastern University ⁶ McGill University

^*Equal contribution

arXiv coming soon Code Dataset Leaderboard

🔥 News

2026-10 Code, frozen task files, and every model output are released on GitHub.
2026-09-30 ReactionBench is accepted to the NeurIPS 2026 Workshop on Evaluation of Interactive Agents (IAEval).

Overview of ReactionBench: synchronized stimulus–reaction videos, human evidence and task construction, and three tasks — intensity profiling, stimulus–reaction matching, and grounded rationale generation. — **Overview.** Built from synchronized in-the-wild stimulus–reaction videos, ReactionBench tests whether a multimodal model can connect a stimulus to an observed viewer reaction through three tasks. Intensity scores and rationale references are human-authored; matching answers come from temporal alignment.

TL;DR

Video benchmarks ask what happens in a video. Socially responsive systems also need to relate what a person watches to how that person responds: a laugh follows a visual gag; a pause matters only against the same viewer’s surrounding behavior. ReactionBench asks whether current multimodal models can rank, temporally match, and explain a viewer’s reaction against the stimulus it was watching. Across seven models, the answer is: not yet.

31,834

synchronized stimulus–reaction clips

433 h

of video across 8 domains

14,894

human intensity ratings on 3,257 moments

447

blind-written human rationales

Three tasks

Each task states its input, its supervision source, and its scoring rule. No model-generated annotation is used as gold.

Intensity Profiling

250 videos · 642 moments

Input: A full reaction video with 2–5 marked time ranges.
Output · metric: A 0–100 intensity score per range; mean per-video extended Spearman ρ₀.
Ground truth: Consensus of ≥3 behaviorally anchored 0–3 human ratings per moment.

Stimulus–Reaction Matching

112 items · 97 videos

Input: One reaction window and four same-video stimulus candidates, separated in time.
Output · metric: One A/B/C/D choice; four-way accuracy (chance 25%).
Ground truth: Temporal alignment in the synchronized recording: the stimulus on screen during the reaction.

Grounded Rationale Generation

200 items · 447 references

Input: Time-aligned stimulus and reaction frames.
Output · metric: A one-sentence cue-based explanation, judged for cue, link, and reference consistency (0–5).
Ground truth: 1–5 blind-written human rationales per item; judge validated against a human-rated anchor set.

Try it yourself

Your turnWhich stimulus was on screen during this reaction?You: 0/0 · item 1/3

Reaction window

Pick A–D. The best model reaches 50.9% on this task (chance 25%).

Building the benchmark

Source videos are public reaction recordings in which a creator watches content, usually in a picture-in-picture layout. The pipeline separates the stimulus and reactor views into time-aligned streams. A model-assisted pre-screen nominates candidate moments, and people supply every label: intensity ratings, rationale references, and the audits behind each task’s construction contract.

ReactionBench annotation pipeline from source videos to human-annotated moments. — **Annotation pipeline.** Automated processing and model-based nomination reduce annotation load. People provide the intensity ratings and rationale references. In a blind pilot, 92.0% of the selected nominator’s picks received a human score ≥ 2, against 68.0% for a facial-analysis selector.

Construction of the three frozen evaluation sets. — **Evaluation-set construction.** Each set follows an explicit contract: within-video contrast for IP, audited temporal separation between candidates for SRM, and blind-written references for GRG.

Leaderboard

Zero-shot results for seven multimodal models on the frozen evaluation sets. IP reports mean per-video ρ₀ (a constant prediction scores zero) and pooled Pearson r under joint and independent scoring. GRG uses one shared text-only judge (DeepSeek-V3.1).

Click a column to sort · bold = best, underline = best open-weight

#	Model	IP joint		IP indep.		SRM	GRG
#	Model	ρ₀	r	ρ₀	r	Acc. ▼	Comp. /5	Cue /2	Cons. /1
1	GPT-5API	.026	.293	.177	.426	.509	2.775	1.245	.465
2	Gemini 3.7 FlashAPI	.176	.308	.316	.518	.500	2.654	.906	.618
3	Qwen3.6-35B-A3Bopen	.043	.187	.210	.398	.482	2.775	.950	.275
4	Qwen3.6-27Bopen	.163	.238	.329	.434	.446	2.610	.815	.270
5	Qwen3.5-397B-A17Bopen	.065	.217	.267	.387	.411	2.905	.950	.270
6	Kimi-K2.6open	.086	.241	.233	.430	.375	2.910	1.085	.295
7	Qwen3.5-9Bopen	.029	.204	.255	.389	.259	2.825	.900	.205
	Chance	—	—	—	—	.250	—	—	—

Key findings

50.9%

Matching stays hard

The best model picks the aligned stimulus in about half of four-way SRM items (chance 25%). Rationale composites peak at 2.91 / 5.

≈0

Which reaction? Models barely notice

For GPT-5, removing the reaction lowers rationale scores, but substituting another viewer’s reaction produces no detectable drop. The model responds to the presence of a reaction, not to which reaction it is shown.

+.14–.23

Protocol moves the score

Scoring moments independently instead of jointly raises intensity-ranking ρ₀ for every one of the seven models, and every paired interval excludes zero.

7 / 7

No model leads every column

Gemini leads joint IP, GPT-5 leads SRM and cue coverage, Qwen3.6-27B leads independent IP, and Kimi-K2.6 leads the GRG composite. The 27B model beats the 397B mixture on IP and SRM.

Results at a glance across IP, SRM, and GRG for seven models, with 95% intervals. — **Results at a glance**, with 95% bootstrap intervals.

Data release and responsible use

Code (MIT): evaluation harness for all three tasks, scoring, judge, and the dataset-construction pipeline.
Annotations (CC BY-NC 4.0): frozen task files, de-identified human ratings, and human rationale references. Annotator identities are replaced with pseudonyms.
Evaluation clips (2,110 clips, about 3 GB) are on a gated Hugging Face dataset. Access is reviewed by the authors and granted for non-commercial academic research. Reaction recordings belong to their creators and stimulus content to its rights holders.
Creator opt-out and takedown. Creators can request removal of their videos by emailing yibozhao@mit.edu and qua@mit.edu (see TAKEDOWN.md).

ReactionBench is intended for evaluating research systems that connect observable events with observable reactions under a disclosed protocol. It is not intended for inferring private mental states, diagnosing emotion or health, detecting deception, ranking people by emotional expressiveness, or supporting surveillance or consequential decisions about individuals. Human-participant research completed institutional ethics review. Annotators were paid an average of $12.00–$16.67 per hour.

BibTeX

@inproceedings{zhao2026reactionbench,
  title     = {ReactionBench: A Human-Anchored Benchmark for Modeling Reactions to Video},
  author    = {Zhao, Yibo and Qu, Ao and Jiang, Xuan and Xu, Jiahao and Ong, Keane and Jiang, Hang and
               Wu, Zhaofeng and Zhuang, Dingyi and Tang, Yihong and Zhou, Kaichen and Zhao, Jinhua and
               Liang, Paul Pu},
  booktitle = {NeurIPS 2026 Workshop on Evaluation of Interactive Agents (IAEval)},
  year      = {2026}
}

Contact

Questions, collaboration, and takedown requests: Yibo Zhao (yibozhao@mit.edu) and Ao Qu (qua@mit.edu). Bug reports: GitHub Issues.