Imagen de Portada

CUTE: Measuring LLMs' Understanding of Their Tokens

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Edman, Lukas, Schmid, Helmut, Fraser, Alexander
Formato:	Preprint
Publicado:	2024
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

EXECUTE: A Multilingual Benchmark for LLM Token Understanding
por: Edman, Lukas, et al.
Publicado: (2025)

Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMs
por: Edman, Lukas, et al.
Publicado: (2025)

Are BabyLMs Second Language Learners?
por: Edman, Lukas, et al.
Publicado: (2024)

Beyond Literal Token Overlap: Token Alignability for Multilinguality
por: Hämmerl, Katharina, et al.
Publicado: (2025)

Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models
por: Nie, Ercong, et al.
Publicado: (2025)

CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages
por: Zhuang, Wenhao, et al.
Publicado: (2025)

On the Sensitivity of Instruction-tuned LLMs to Harmful Sentences in Long Inputs
por: Ghorbanpour, Faeze, et al.
Publicado: (2025)

LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback
por: Lai, Wen, et al.
Publicado: (2024)

Understanding Cross-Lingual Alignment -- A Survey
por: Hämmerl, Katharina, et al.
Publicado: (2024)

Style-Specific Neurons for Steering LLMs in Text Style Transfer
por: Lai, Wen, et al.
Publicado: (2024)

Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation
por: Edman, Lukas, et al.
Publicado: (2023)

PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions
por: Niazi, Ruhallah, et al.
Publicado: (2026)

ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks
por: Ma, Bolei, et al.
Publicado: (2024)

Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning
por: Xu, Zhu, et al.
Publicado: (2024)

Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study
por: Ghorbanpour, Faeze, et al.
Publicado: (2025)

Hate Personified: Investigating the role of LLMs in content moderation
por: Masud, Sarah, et al.
Publicado: (2024)

How to Solve Few-Shot Abusive Content Detection Using the Data We Actually Have
por: Hangya, Viktor, et al.
Publicado: (2023)

Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters
por: Hiraoka, Tatsuya, et al.
Publicado: (2025)

Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs
por: Dong, Jiancheng, et al.
Publicado: (2025)

Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs
por: Wang, Dingdong, et al.
Publicado: (2025)

LLM in the Loop: Creating the ParaDeHate Dataset for Hate Speech Detoxification
por: Yuan, Shuzhou, et al.
Publicado: (2025)

Why Are We Lonely? Leveraging LLMs to Measure and Understand Loneliness in Caregivers and Non-caregivers
por: Kim, Michelle Damin, et al.
Publicado: (2026)

Measuring Scalar Constructs in Social Science with LLMs
por: Licht, Hauke, et al.
Publicado: (2025)

WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
por: Thede, Lukas, et al.
Publicado: (2025)

Accelerating Production LLMs with Combined Token/Embedding Speculators
por: Wertheimer, Davis, et al.
Publicado: (2024)

Optimizing Korean-Centric LLMs via Token Pruning
por: Kim, Hoyeol, et al.
Publicado: (2026)

Beyond Tokens: Concept-Level Training Objectives for LLMs
por: Iyer, Laya, et al.
Publicado: (2026)

CrossNews-UA: A Cross-lingual News Semantic Similarity Benchmark for Ukrainian, Polish, Russian, and English
por: Dementieva, Daryna, et al.
Publicado: (2025)

EmoBench-UA: A Benchmark Dataset for Emotion Detection in Ukrainian
por: Dementieva, Daryna, et al.
Publicado: (2025)

Language Model Re-rankers are Fooled by Lexical Similarities
por: Hagström, Lovisa, et al.
Publicado: (2025)

Toward a Theory of Tokenization in LLMs
por: Rajaraman, Nived, et al.
Publicado: (2024)

LLMs are Not Just Next Token Predictors
por: Downes, Stephen M., et al.
Publicado: (2024)

StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
por: Song, Yuhan, et al.
Publicado: (2025)

Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoders
por: Goyal, Agam, et al.
Publicado: (2025)

How Language Directions Align with Token Geometry in Multilingual LLMs
por: Kim, JaeSeong, et al.
Publicado: (2025)

Speculating LLMs' Chinese Training Data Pollution from Their Tokens
por: Zhang, Qingjie, et al.
Publicado: (2025)

How does a Language-Specific Tokenizer affect LLMs?
por: Seo, Jean, et al.
Publicado: (2025)

Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs
por: Pawar, Sachin, et al.
Publicado: (2025)

Measuring Intrinsic Dimension of Token Embeddings
por: Kataiwa, Takuya, et al.
Publicado: (2025)

Do LLMs Understand Why We Write Diaries? A Method for Purpose Extraction and Clustering
por: Goloviznina, Valeriya, et al.
Publicado: (2025)