Cover Image

I am a Strange Dataset: Metalinguistic Tests for Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Thrush, Tristan, Moore, Jared, Monares, Miguel, Potts, Christopher, Kiela, Douwe
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Nearest Neighbor Normalization Improves Multimodal Retrieval
by: Chowdhury, Neil, et al.
Published: (2024)

Lynx: An Open Source Hallucination Evaluation Model
by: Ravi, Selvan Sunitha, et al.
Published: (2024)

LMUnit: Fine-grained Evaluation with Natural Language Unit Tests
by: Saad-Falcon, Jon, et al.
Published: (2024)

Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
by: D'Oosterlinck, Karel, et al.
Published: (2024)

Document Optimization for Black-Box Retrieval via Reinforcement Learning
by: Uzan, Omri, et al.
Published: (2026)

Synthetic Data for any Differentiable Target
by: Thrush, Tristan, et al.
Published: (2026)

Are Large Language Models Consistent over Value-laden Questions?
by: Moore, Jared, et al.
Published: (2024)

Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
by: Tran, Dat, et al.
Published: (2026)

Improving Pretraining Data Using Perplexity Correlations
by: Thrush, Tristan, et al.
Published: (2024)

A Benchmark for the Detection of Metalinguistic Disagreements between LLMs and Knowledge Graphs
by: Allen, Bradley P., et al.
Published: (2025)

Generative Representational Instruction Tuning
by: Muennighoff, Niklas, et al.
Published: (2024)

Great Models Think Alike and this Undermines AI Oversight
by: Goel, Shashwat, et al.
Published: (2025)

Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appear
by: McCoy, R. Thomas
Published: (2026)

Counterfactual Simulation Training for Chain-of-Thought Faithfulness
by: Hase, Peter, et al.
Published: (2026)

Anchor Points: Benchmarking Models with Much Fewer Examples
by: Vivek, Rajan, et al.
Published: (2023)

Do Large Language Models Possess a Theory of Mind? A Comparative Evaluation Using the Strange Stories Paradigm
by: Babarczy, Anna, et al.
Published: (2026)

Mission: Impossible Language Models
by: Kallini, Julie, et al.
Published: (2024)

Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP
by: Rozner, Josh, et al.
Published: (2021)

DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
by: Singhvi, Arnav, et al.
Published: (2023)

MrT5: Dynamic Token Merging for Efficient Byte-level Language Models
by: Kallini, Julie, et al.
Published: (2024)

Leveraging Diffusion Perturbations for Measuring Fairness in Computer Vision
by: Lui, Nicholas, et al.
Published: (2023)

Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructions
by: Boguraev, Sasha, et al.
Published: (2025)

ReFT: Representation Finetuning for Language Models
by: Wu, Zhengxuan, et al.
Published: (2024)

Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors
by: Huang, Jing, et al.
Published: (2025)

Language Models are Alignable Decision-Makers: Dataset and Application to the Medical Triage Domain
by: Hu, Brian, et al.
Published: (2024)

Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction
by: Dsouza, Amanda, et al.
Published: (2024)

A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i
by: Ayonrinde, Kola, et al.
Published: (2025)

OLMoE: Open Mixture-of-Experts Language Models
by: Muennighoff, Niklas, et al.
Published: (2024)

AUTALIC: A Dataset for Anti-AUTistic Ableist Language In Context
by: Rizvi, Naba, et al.
Published: (2024)

Retrieval Augmented Spelling Correction for E-Commerce Applications
by: Guo, Xuan, et al.
Published: (2024)

Diagnosing Bottlenecks in Data Visualization Understanding by Vision-Language Models
by: Tartaglini, Alexa R., et al.
Published: (2025)

The Pitfalls of Publishing in the Age of LLMs: Strange and Surprising Adventures with a High-Impact NLP Journal
by: Verma, Rakesh M., et al.
Published: (2024)

Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
by: Moore, Kyle, et al.
Published: (2025)

KTO: Model Alignment as Prospect Theoretic Optimization
by: Ethayarajh, Kawin, et al.
Published: (2024)

Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
by: Opsahl-Ong, Krista, et al.
Published: (2024)

Datasets for Large Language Models: A Comprehensive Survey
by: Liu, Yang, et al.
Published: (2024)

Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Together
by: Soylu, Dilara, et al.
Published: (2024)

CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models
by: Wagner, Eitan, et al.
Published: (2024)

Test-Time Fairness and Robustness in Large Language Models
by: Cotta, Leonardo, et al.
Published: (2024)

Zyda: A 1.3T Dataset for Open Language Modeling
by: Tokpanov, Yury, et al.
Published: (2024)