Imagen de Portada

GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Jassim, Serwan, Holubar, Mario, Richter, Annika, Wolff, Cornelius, Ohmer, Xenia, Bruni, Elia
Formato:	Preprint
Publicado:	2023
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Bidirectional Emergent Language in Situated Environments
por: Wolff, Cornelius, et al.
Publicado: (2024)

Pixels to Principles: Probing Intuitive Physics Understanding in Multimodal Language Models
por: Ballout, Mohamad, et al.
Publicado: (2025)

From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
por: Ohmer, Xenia, et al.
Publicado: (2024)

iVISPAR -- An Interactive Visual-Spatial Reasoning Benchmark for VLMs
por: Mayer, Julius, et al.
Publicado: (2025)

On the Relationship between Skill Neurons and Robustness in Prompt Tuning
por: Ackermann, Leon, et al.
Publicado: (2023)

DevBench: A multimodal developmental benchmark for language learning
por: Tan, Alvin Wei Ming, et al.
Publicado: (2024)

\textsc{CantoNLU}: A benchmark for Cantonese natural language understanding
por: Min, Junghyun, et al.
Publicado: (2025)

DocLLM: A layout-aware generative language model for multimodal document understanding
por: Wang, Dongsheng, et al.
Publicado: (2023)

A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models
por: Li, Kunning, et al.
Publicado: (2025)

AIDBench: A benchmark for evaluating the authorship identification capability of large language models
por: Wen, Zichen, et al.
Publicado: (2024)

MaterialFigBENCH: benchmark dataset with figures for evaluating college-level materials science problem-solving abilities of multimodal large language models
por: Yoshitake, Michiko, et al.
Publicado: (2026)

WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation
por: Matos, João, et al.
Publicado: (2024)

Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
por: Rädsch, Tim, et al.
Publicado: (2025)

BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language models
por: Lavechin, Marvin, et al.
Publicado: (2023)

Protecting multimodal large language models against misleading visualizations
por: Tonglet, Jonathan, et al.
Publicado: (2025)

Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models
por: Padlewski, Piotr, et al.
Publicado: (2024)

Anthropocentric bias in language model evaluation
por: Millière, Raphaël, et al.
Publicado: (2024)

What does it mean to understand language?
por: Casto, Colton, et al.
Publicado: (2025)

Is your multimodal large language model a good science tutor?
por: Liu, Ming, et al.
Publicado: (2025)

Anchor function: a type of benchmark functions for studying language models
por: Zhang, Zhongwang, et al.
Publicado: (2024)

LongTail-Swap: benchmarking language models' abilities on rare words
por: Algayres, Robin, et al.
Publicado: (2025)

Linguini: A benchmark for language-agnostic linguistic reasoning
por: Sánchez, Eduardo, et al.
Publicado: (2024)

FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models
por: Shamsfard, Mehrnoush, et al.
Publicado: (2025)

As easy as PIE: understanding when pruning causes language models to disagree
por: Tropeano, Pietro, et al.
Publicado: (2025)

Are LLM-generated plain language summaries truly understandable? A large-scale crowdsourced evaluation
por: Guo, Yue, et al.
Publicado: (2025)

The intersection of philosophy of language and artificial intelligence: Challenges in replicating human language understanding
por: Sooraj Kumar Maurya
Publicado: (2024)

The SMeL Test: A simple benchmark for media literacy in language models
por: Ahdritz, Gustaf, et al.
Publicado: (2025)

TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain
por: Barboule, Camille, et al.
Publicado: (2024)

Towards understanding evolution of science through language model series
por: Dong, Junjie, et al.
Publicado: (2024)

Creativity Benchmark: A benchmark for marketing creativity for large language models
por: Bhat, Ninad, et al.
Publicado: (2025)

Lost without translation -- Can transformer (language models) understand mood states?
por: Shivaprakash, Prakrithi, et al.
Publicado: (2025)

Can large language models understand uncommon meanings of common words?
por: Wu, Jinyang, et al.
Publicado: (2024)

Re-evaluating Theory of Mind evaluation in large language models
por: Hu, Jennifer, et al.
Publicado: (2025)

CMMLU: Measuring massive multitask language understanding in Chinese
por: Li, Haonan, et al.
Publicado: (2023)

NLD-LLM: A systematic framework for evaluating small language transformer models on natural language description
por: Jelodar, Hamed, et al.
Publicado: (2025)

Logical forms complement probability in understanding language model (and human) performance
por: Wang, Yixuan, et al.
Publicado: (2025)

Realtime, multimodal invasive ventilation risk monitoring using language models and BoXHED
por: Pakbin, Arash, et al.
Publicado: (2024)

A dataset and benchmark for hospital course summarization with adapted large language models
por: Aali, Asad, et al.
Publicado: (2024)

Clinical named entity recognition in the Portuguese language: a benchmark of modern BERT models and LLMs
por: de Almeida, Vinicius Anjos, et al.
Publicado: (2026)

Tgea: An error-annotated dataset and benchmark tasks for text generation from pretrained language models
por: He, Jie, et al.
Publicado: (2025)