Why Mean Pooling Works: Quantifying Second-Order Collapse in Text Embeddings

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hara, Tomomasa, Kurita, Hiroto, Imaizumi, Masaaki, Inui, Kentaro, Yokoi, Sho
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Zipfian Whitening
by: Yokoi, Sho, et al.
Published: (2024)

Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps
by: Kobayashi, Goro, et al.
Published: (2023)

Linear Representations of Hierarchical Concepts in Language Models
by: Sakata, Masaki, et al.
Published: (2026)

On Entity Identification in Language Models
by: Sakata, Masaki, et al.
Published: (2025)

Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport
by: Kishino, Ryo, et al.
Published: (2024)

Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledge
by: Zhang, Ying, et al.
Published: (2025)

Spelling-out is not Straightforward: LLMs' Capability of Tokenization from Token to Characters
by: Hiraoka, Tatsuya, et al.
Published: (2025)

Repetition Neurons: How Do Language Models Produce Repetitions?
by: Hiraoka, Tatsuya, et al.
Published: (2024)

Monotonic Representation of Numeric Properties in Language Models
by: Heinzerling, Benjamin, et al.
Published: (2024)

LLMs Can Compensate for Deficiencies in Visual Representations
by: Takishita, Sho, et al.
Published: (2025)

Cell-Based Representation of Relational Binding in Language Models
by: Dai, Qin, et al.
Published: (2026)

Rectifying Belief Space via Unlearning to Harness LLMs' Reasoning
by: Niwa, Ayana, et al.
Published: (2025)

Representational Analysis of Binding in Language Models
by: Dai, Qin, et al.
Published: (2024)

Hidden Failures in Robustness: Why Supervised Uncertainty Quantification Needs Better Evaluation
by: Stacey, Joe, et al.
Published: (2026)

Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning
by: Doan, Nhi Hoai, et al.
Published: (2025)

Mechanistic Insights into Grokking from the Embedding Layer
by: AlquBoj, H. V., et al.
Published: (2025)

J-UniMorph: Japanese Morphological Annotation through the Universal Feature Schema
by: Matsuzaki, Kosuke, et al.
Published: (2024)

Syntactic Learnability of Echo State Neural Language Models at Scale
by: Ueda, Ryo, et al.
Published: (2025)

TopK Language Models
by: Takahashi, Ryosuke, et al.
Published: (2025)

A Large Collection of Model-generated Contradictory Responses for Consistency-aware Dialogue Systems
by: Sato, Shiki, et al.
Published: (2024)

Subspace Representations for Soft Set Operations and Sentence Similarities
by: Ishibashi, Yoichi, et al.
Published: (2022)

Pooling and Semantic Shift: The Fundamental Challenges in Long Text Embedding and Retrieval
by: Gao, Hang, et al.
Published: (2026)

FinchGPT: a Transformer based language model for birdsong analysis
by: Kobayashi, Kosei, et al.
Published: (2025)

How Individual Traits and Language Styles Shape Preferences In Open-ended User-LLM Interaction: A Preliminary Study
by: Chevi, Rendi, et al.
Published: (2025)

LegalViz: Legal Text Visualization by Text To Diagram Generation
by: Onami, Eri, et al.
Published: (2025)

Quantifying Positional Biases in Text Embedding Models
by: Lee, Reagan J., et al.
Published: (2024)

Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View
by: Airlangga, Muhammad Cendekia, et al.
Published: (2025)

Reducing the Cost: Cross-Prompt Pre-Finetuning for Short Answer Scoring
by: Funayama, Hiroaki, et al.
Published: (2024)

To Drop or Not to Drop? Predicting Argument Ellipsis Judgments: A Case Study in Japanese
by: Ishizuki, Yukiko, et al.
Published: (2024)

MQM-Chat: Multidimensional Quality Metrics for Chat Translation
by: Li, Yunmeng, et al.
Published: (2024)

The Curse of Popularity: Popular Entities have Catastrophic Side Effects when Deleting Knowledge from Language Models
by: Takahashi, Ryosuke, et al.
Published: (2024)

ACORN: Aspect-wise Commonsense Reasoning Explanation Evaluation
by: Brassard, Ana, et al.
Published: (2024)

Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings
by: Pan, Tengyu, et al.
Published: (2025)

Do LLMs Need to Think in One Language? Correlation between Latent Language and Task Performance
by: Ozaki, Shintaro, et al.
Published: (2025)

Large Language Models Are Human-Like Internally
by: Kuribayashi, Tatsuki, et al.
Published: (2025)

Sticking to the Mean: Detecting Sticky Tokens in Text Embedding Models
by: Chen, Kexin, et al.
Published: (2025)

Anti Mode-Collapse in Mean-Field Transformer via Auxiliary Variables
by: Imaizumi, Masaaki, et al.
Published: (2026)

LMK > CLS: Landmark Pooling for Dense Embeddings
by: Doshi, Meet, et al.
Published: (2026)

TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
by: Shing, Makoto, et al.
Published: (2025)

Evaluating Multimodal Large Language Models on Vertically Written Japanese Text
by: Sasagawa, Keito, et al.
Published: (2025)