Cover Image

UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Adeeba, Farah, Dillon, Brian, Sajjad, Hassan, Bhatt, Rajesh
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Paradigm Gap in Urdu
by: Adeeba, Farah, et al.
Published: (2025)

RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs
by: Taktasheva, Ekaterina, et al.
Published: (2024)

TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs
by: Başar, Ezgi, et al.
Published: (2025)

Irish-BLiMP: A Linguistic Benchmark for Evaluating Human and Language Model Performance in a Low-Resource Setting
by: McGiff, Josh, et al.
Published: (2025)

QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs
by: Beauchemin, David, et al.
Published: (2025)

Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks
by: Tahir, Munief Hassan, et al.
Published: (2024)

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
by: Jumelet, Jaap, et al.
Published: (2025)

BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Models
by: Gao, Yuan, et al.
Published: (2025)

Extrinsic Evaluation of Cultural Competence in Large Language Models
by: Bhatt, Shaily, et al.
Published: (2024)

CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models
by: Oba, Miyu, et al.
Published: (2026)

Evaluating Large Language Models on Urdu Idiom Translation
by: Khan, Muhammad Farmal, et al.
Published: (2025)

Ax-to-Grind Urdu: Benchmark Dataset for Urdu Fake News Detection
by: Harris, Sheetal, et al.
Published: (2024)

Generalists vs. Specialists: Evaluating Large Language Models for Urdu
by: Arif, Samee, et al.
Published: (2024)

Holmes: A Benchmark to Assess the Linguistic Competence of Language Models
by: Waldis, Andreas, et al.
Published: (2024)

COCO-Urdu: A Large-Scale Urdu Image-Caption Dataset with Multimodal Quality Estimation
by: Hassan, Umair
Published: (2025)

SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia
by: Ayash, Lama, et al.
Published: (2025)

TALE: A Tool-Augmented Framework for Reference-Free Evaluation of Large Language Models
by: Badshah, Sher, et al.
Published: (2025)

Enhanced Urdu Intent Detection with Large Language Models and Prototype-Informed Predictive Pipelines
by: Hassan, Faiza, et al.
Published: (2025)

UrduLM: A Resource-Efficient Monolingual Urdu Language Model
by: Ali, Syed Muhammad, et al.
Published: (2026)

Benchmarking Linguistic Diversity of Large Language Models
by: Guo, Yanzhu, et al.
Published: (2024)

Benchmarking Motivational Interviewing Competence of Large Language Models
by: Jha, Aishwariya, et al.
Published: (2026)

Evaluating the Deductive Competence of Large Language Models
by: Seals, Spencer M., et al.
Published: (2023)

Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models
by: Merlo, Paola, et al.
Published: (2026)

Evaluating Clinical Competencies of Large Language Models with a General Practice Benchmark
by: Li, Zheqing, et al.
Published: (2025)

Qalb: Largest State-of-the-Art Urdu Large Language Model for 230M Speakers with Systematic Continued Pre-training
by: Hassan, Muhammad Taimoor, et al.
Published: (2026)

From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
by: Al-Monef, Renad, et al.
Published: (2025)

Pragmatic Competence Evaluation of Large Language Models for the Korean Language
by: Park, Dojun, et al.
Published: (2024)

Understanding Syntactic Generalization in Structure-inducing Language Models
by: Arps, David, et al.
Published: (2025)

UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
by: Ahmad, Sarfraz, et al.
Published: (2025)

Irony Detection in Urdu Text: A Comparative Study Using Machine Learning Models and Large Language Models
by: Ahmad, Fiaz, et al.
Published: (2025)

WER We Stand: Benchmarking Urdu ASR Models
by: Arif, Samee, et al.
Published: (2024)

The Invalsi Benchmarks: measuring Linguistic and Mathematical understanding of Large Language Models in Italian
by: Puccetti, Giovanni, et al.
Published: (2024)

A System for Name and Address Parsing with Large Language Models
by: Tarannum, Adeeba, et al.
Published: (2026)

LaMP: When Large Language Models Meet Personalization
by: Salemi, Alireza, et al.
Published: (2023)

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks
by: Yamaguchi, Atsuki, et al.
Published: (2026)

Uncovering Competency Gaps in Large Language Models and Their Benchmarks
by: Bohacek, Maty, et al.
Published: (2025)

Low-Resource Transliteration for Roman-Urdu and Urdu Using Transformer-Based Models
by: Butt, Umer, et al.
Published: (2025)

Efficient Large Language Models with Zero-Shot Adjustable Acceleration
by: Kachuee, Sajjad, et al.
Published: (2025)

A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognition
by: Ahmad, Hussain, et al.
Published: (2025)

BioUNER: A Benchmark Dataset for Clinical Urdu Named Entity Recognition
by: Ali, Wazir, et al.
Published: (2026)