Cover Image

Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wu, Zhaofeng, Qiu, Linlu, Ross, Alexis, Akyürek, Ekin, Chen, Boyuan, Wang, Bailin, Kim, Najoung, Andreas, Jacob, Kim, Yoon
Format:	Preprint
Published:	2023
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

In-Context Language Learning: Architectures and Algorithms
by: Akyürek, Ekin, et al.
Published: (2024)

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
by: Akyürek, Ekin, et al.
Published: (2024)

Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatability
by: Akyürek, Afra Feyza, et al.
Published: (2024)

Implicit Representations of Grammaticality in Language Models
by: Wang, Yingshan Susan, et al.
Published: (2026)

Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
by: Qiu, Linlu, et al.
Published: (2023)

On the Same Wavelength? Evaluating Pragmatic Reasoning in Language Models across Broad Concepts
by: Qiu, Linlu, et al.
Published: (2025)

Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
by: Tang, Zilu, et al.
Published: (2025)

Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models
by: Qiu, Linlu, et al.
Published: (2025)

Is analogy enough to draw novel adjective-noun inferences?
by: Ross, Hayley, et al.
Published: (2025)

Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distribution
by: Ross, Hayley, et al.
Published: (2024)

ELF: Embedded Language Flows
by: Hu, Keya, et al.
Published: (2026)

Bias Amplification in Language Model Evolution: An Iterated Learning Perspective
by: Ren, Yi, et al.
Published: (2024)

A systematic framework for generating novel experimental hypotheses from language models
by: Misra, Kanishka, et al.
Published: (2024)

Code Pretraining Improves Entity Tracking Abilities of Language Models
by: Kim, Najoung, et al.
Published: (2024)

Translation or Recitation? Calibrating Evaluation Scores for Machine Translation of Extremely Low-Resource Languages
by: Chen, Danlu, et al.
Published: (2026)

Can You Learn Semantics Through Next-Word Prediction? The Case of Entailment
by: Merrill, William, et al.
Published: (2024)

Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment
by: Wu, Zhaofeng, et al.
Published: (2024)

Learning Linear Attention in Polynomial Time
by: Yau, Morris, et al.
Published: (2024)

Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets
by: Daswani, Ashwin, et al.
Published: (2024)

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities
by: Wu, Zhaofeng, et al.
Published: (2024)

Structured Code Representations Enable Data-Efficient Adaptation of Code Language Models
by: Agarwal, Mayank, et al.
Published: (2024)

Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps
by: Chuang, Yung-Sung, et al.
Published: (2024)

Self-Adapting Language Models
by: Zweiger, Adam, et al.
Published: (2025)

Toward In-Context Teaching: Adapting Examples to Students' Misconceptions
by: Ross, Alexis, et al.
Published: (2024)

Learning to Decode Collaboratively with Multiple Language Models
by: Shen, Shannon Zejiang, et al.
Published: (2024)

DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks
by: Kim, Hyunjun, et al.
Published: (2025)

Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models
by: Puri, Isha, et al.
Published: (2026)

Findings of the BlackboxNLP 2025 Shared Task: Localizing Circuits and Causal Variables in Language Models
by: Arad, Dana, et al.
Published: (2025)

Are they lovers or friends? Evaluating LLMs' Social Reasoning in English and Korean Dialogues
by: Kim, Eunsu, et al.
Published: (2025)

Personas as a Way to Model Truthfulness in Language Models
by: Joshi, Nitish, et al.
Published: (2023)

CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities
by: Mao, Yujun, et al.
Published: (2024)

RPM: Reasoning-Level Personalization for Black-Box Large Language Models
by: Kim, Jieyong, et al.
Published: (2025)

Learning to Reason via Program Generation, Emulation, and Search
by: Weir, Nathaniel, et al.
Published: (2024)

Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks
by: Kim, Sungkyung, et al.
Published: (2024)

Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning
by: Kim, Minwu, et al.
Published: (2025)

Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math
by: Xu, Haoran, et al.
Published: (2025)

Gated Linear Attention Transformers with Hardware-Efficient Training
by: Yang, Songlin, et al.
Published: (2023)

Parallelizing Linear Transformers with the Delta Rule over Sequence Length
by: Yang, Songlin, et al.
Published: (2024)

The Dialogue That Heals: A Comprehensive Evaluation of Doctor Agents' Inquiry Capability
by: Gong, Linlu, et al.
Published: (2025)

Reasoning Capabilities of Large Language Models on Dynamic Tasks
by: Wong, Annie, et al.
Published: (2025)