Image de couverture de livre

CAT: A Metric-Driven Framework for Analyzing the Consistency-Accuracy Relation of LLMs under Controlled Input Variations

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Cavalin, Paulo, Sanctos, Cassia, Grave, Marcelo, Pinhanez, Claudio, Primerano, Yago
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks
par: Pinhanez, Claudio, et autres
Publié: (2025)

Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choices
par: Cavalin, Paulo, et autres
Publié: (2025)

Exploring Performance Variations in Finetuned Translators of Ultra-Low Resource Languages: Do Linguistic Differences Matter?
par: Gonçalves, Isabel, et autres
Publié: (2025)

Sentence-level Aggregation of Lexical Metrics Correlates Stronger with Human Judgements than Corpus-level Aggregation
par: Cavalin, Paulo, et autres
Publié: (2024)

Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluations
par: Pinhanez, Claudio, et autres
Publié: (2024)

Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences
par: Pinhanez, Claudio, et autres
Publié: (2024)

Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs
par: Reusens, Manon, et autres
Publié: (2025)

A methodological analysis of prompt perturbations and their effect on attack success rates
par: Machado, Tiago, et autres
Publié: (2025)

Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Staging
par: Chang, Chia-Hsuan, et autres
Publié: (2024)

LLMs as Agentic Cooperative Players in Multiplayer UNO
par: Matinez, Yago Romano, et autres
Publié: (2025)

Accuracy and Consistency of LLMs in the Registered Dietitian Exam: The Impact of Prompt Engineering and Knowledge Retrieval
par: Azimi, Iman, et autres
Publié: (2024)

Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy
par: Deviyani, Athiya, et autres
Publié: (2025)

Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak
par: Du, Yanrui, et autres
Publié: (2023)

Analyzing FOMC Minutes: Accuracy and Constraints of Language Models
par: Kim, Wonseong, et autres
Publié: (2023)

To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
par: Sedova, Anastasiia, et autres
Publié: (2024)

Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
par: Kocmi, Tom, et autres
Publié: (2024)

Analyzing Reasoning Consistency in Large Multimodal Models under Cross-Modal Conflicts
par: Zhu, Zhihao, et autres
Publié: (2026)

AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic
par: Robinson, Nathaniel R., et autres
Publié: (2024)

Worst-Case Input Generation for Concurrent Programs under Non-Monotone Resource Metrics
par: Pham, Long, et autres
Publié: (2023)

Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
par: Kong, Jiawei, et autres
Publié: (2025)

Meta-Evaluating Local LLMs: Rethinking Performance Metrics for Serious Games
par: Isaza-Giraldo, Andrés, et autres
Publié: (2025)

LogitScope: A Framework for Analyzing LLM Uncertainty Through Information Metrics
par: Ahmed, Farhan, et autres
Publié: (2026)

MetricalARGS: A Taxonomy for Studying Metrical Poetry with LLMs
par: Kranti, Chalamalasetti, et autres
Publié: (2025)

Self-Training Meets Consistency: Improving LLMs' Reasoning with Consistency-Driven Rationale Evaluation
par: Lee, Jaehyeok, et autres
Publié: (2024)

Attention Consistency for LLMs Explanation
par: Lan, Tian, et autres
Publié: (2025)

Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction
par: Sun, Qi, et autres
Publié: (2024)

LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning
par: Mao, Yansheng, et autres
Publié: (2025)

Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo
par: Sundararajan, Barkavi, et autres
Publié: (2024)

Too Consistent to Detect: A Study of Self-Consistent Errors in LLMs
par: Tan, Hexiang, et autres
Publié: (2025)

Comparative Experimentation of Accuracy Metrics in Automated Medical Reporting: The Case of Otitis Consultations
par: Faber, Wouter, et autres
Publié: (2023)

Moneyball with LLMs: Analyzing Tabular Summarization in Sports Narratives
par: Upadhyay, Ritam, et autres
Publié: (2025)

Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks
par: Rao, Abhinav, et autres
Publié: (2023)

Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
par: Pan, Eileen, et autres
Publié: (2025)

Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs
par: Muppidi, Ananth, et autres
Publié: (2025)

The Order Effect: Investigating Prompt Sensitivity to Input Order in LLMs
par: Guan, Bryan, et autres
Publié: (2025)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
par: Sam, Dylan, et autres
Publié: (2025)

On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs
par: Ghorbanpour, Faeze, et autres
Publié: (2025)

Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric
par: Koh, Hyukhun, et autres
Publié: (2024)

Consistency Matters: Explore LLMs Consistency From a Black-Box Perspective
par: Zhao, Fufangchen, et autres
Publié: (2024)

Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking
par: Aksu, Taha, et autres
Publié: (2024)