Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909641258041344 |
|---|---|
| author | Zaranis, Emmanouil Farinhas, António Santos, Saul Canaverde, Beatriz Ramos, Miguel Moura Surikuchi, Aditya K Viveiros, André Liao, Baohao Bueno-Benito, Elena Sivakumaran, Nithin Vasylenko, Pavlo Yu, Shoubin Sannigrahi, Sonal Mohammed, Wafaa Peters, Ben Villegas, Danae Sánchez Stengel-Eskin, Elias Attanasio, Giuseppe Yoon, Jaehong Frank, Stella Suglia, Alessandro Zerva, Chrysoula Elliott, Desmond Dimiccoli, Mariella Bansal, Mohit Lanz, Oswald Bernardi, Raffaella Fernández, Raquel Pezzelle, Sandro Niculae, Vlad Martins, André F. T. |
| author_facet | Zaranis, Emmanouil Farinhas, António Santos, Saul Canaverde, Beatriz Ramos, Miguel Moura Surikuchi, Aditya K Viveiros, André Liao, Baohao Bueno-Benito, Elena Sivakumaran, Nithin Vasylenko, Pavlo Yu, Shoubin Sannigrahi, Sonal Mohammed, Wafaa Peters, Ben Villegas, Danae Sánchez Stengel-Eskin, Elias Attanasio, Giuseppe Yoon, Jaehong Frank, Stella Suglia, Alessandro Zerva, Chrysoula Elliott, Desmond Dimiccoli, Mariella Bansal, Mohit Lanz, Oswald Bernardi, Raffaella Fernández, Raquel Pezzelle, Sandro Niculae, Vlad Martins, André F. T. |
| contents | Despite recent progress in vision-language models (VLMs), holistic understanding of long-form video content remains a significant challenge, partly due to limitations in current benchmarks. Many focus on peripheral, ``needle-in-a-haystack'' details, encouraging context-insensitive retrieval over deep comprehension. Others rely on large-scale, semi-automatically generated questions (often produced by language models themselves) that are easier for models to answer but fail to reflect genuine understanding. In this paper, we introduce MF$^2$, a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies (50-170 minutes long). MF$^2$ includes over 50 full-length, open-licensed movies, each paired with manually constructed sets of claim pairs -- one true (fact) and one plausible but false (fib), totalling over 850 pairs. These claims target core narrative elements such as character motivations and emotions, causal chains, and event order, and refer to memorable moments that humans can recall without rewatching the movie. Instead of multiple-choice formats, we adopt a binary claim evaluation protocol: for each pair, models must correctly identify both the true and false claims. This reduces biases like answer ordering and enables a more precise assessment of reasoning. Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance, underscoring the relative ease of the task for humans and their superior ability to retain and reason over critical narrative information -- an ability current VLMs lack. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06275 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding Zaranis, Emmanouil Farinhas, António Santos, Saul Canaverde, Beatriz Ramos, Miguel Moura Surikuchi, Aditya K Viveiros, André Liao, Baohao Bueno-Benito, Elena Sivakumaran, Nithin Vasylenko, Pavlo Yu, Shoubin Sannigrahi, Sonal Mohammed, Wafaa Peters, Ben Villegas, Danae Sánchez Stengel-Eskin, Elias Attanasio, Giuseppe Yoon, Jaehong Frank, Stella Suglia, Alessandro Zerva, Chrysoula Elliott, Desmond Dimiccoli, Mariella Bansal, Mohit Lanz, Oswald Bernardi, Raffaella Fernández, Raquel Pezzelle, Sandro Niculae, Vlad Martins, André F. T. Computer Vision and Pattern Recognition Computation and Language Machine Learning Despite recent progress in vision-language models (VLMs), holistic understanding of long-form video content remains a significant challenge, partly due to limitations in current benchmarks. Many focus on peripheral, ``needle-in-a-haystack'' details, encouraging context-insensitive retrieval over deep comprehension. Others rely on large-scale, semi-automatically generated questions (often produced by language models themselves) that are easier for models to answer but fail to reflect genuine understanding. In this paper, we introduce MF$^2$, a new benchmark for evaluating whether models can comprehend, consolidate, and recall key narrative information from full-length movies (50-170 minutes long). MF$^2$ includes over 50 full-length, open-licensed movies, each paired with manually constructed sets of claim pairs -- one true (fact) and one plausible but false (fib), totalling over 850 pairs. These claims target core narrative elements such as character motivations and emotions, causal chains, and event order, and refer to memorable moments that humans can recall without rewatching the movie. Instead of multiple-choice formats, we adopt a binary claim evaluation protocol: for each pair, models must correctly identify both the true and false claims. This reduces biases like answer ordering and enables a more precise assessment of reasoning. Our experiments demonstrate that both open-weight and closed state-of-the-art models fall well short of human performance, underscoring the relative ease of the task for humans and their superior ability to retain and reason over critical narrative information -- an ability current VLMs lack. |
| title | Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.06275 |