Cover Image

Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Junxuan, Ge, Xuyang, Shu, Wentao, Tang, Qiong, Zhou, Yunhua, He, Zhengfu, Qiu, Xipeng
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Dimensional Collapse in Transformer Attention Outputs: A Challenge for Sparse Dictionary Learning
by: Wang, Junxuan, et al.
Published: (2025)

Evolution of Concepts in Language Model Pre-Training
by: Ge, Xuyang, et al.
Published: (2025)

Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition
by: He, Zhengfu, et al.
Published: (2025)

Automatically Identifying Local and Global Circuits with Linear Computation Graphs
by: Ge, Xuyang, et al.
Published: (2024)

Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
by: He, Zhengfu, et al.
Published: (2024)

Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT
by: He, Zhengfu, et al.
Published: (2024)

A Distributional View for Visual Mechanistic Interpretability: KL-Minimal Soft-Constraint Principle
by: Zhou, Guancheng, et al.
Published: (2026)

The Open-World Lottery Ticket Hypothesis for OOD Intent Classification
by: Zhou, Yunhua, et al.
Published: (2022)

DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning
by: Wang, Xinghao, et al.
Published: (2024)

How Attention Sinks Emerge in Large Language Models: An Interpretability Perspective
by: Peng, Runyu, et al.
Published: (2026)

BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments
by: Wang, Xinghao, et al.
Published: (2024)

Data-free Weight Compress and Denoise for Large Language Models
by: Peng, Runyu, et al.
Published: (2024)

Tracing the Thought of a Grandmaster-level Chess-Playing Transformer
by: Lin, Rui, et al.
Published: (2026)

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance
by: Ye, Jiasheng, et al.
Published: (2024)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
by: Peng, Runyu, et al.
Published: (2026)

Making Large Language Models Better Reasoners with Orchestrated Streaming Experiences
by: Liu, Xiangyang, et al.
Published: (2025)

Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?
by: Zeng, Zhiyuan, et al.
Published: (2025)

Layers at Similar Depths Generate Similar Activations Across LLM Architectures
by: Wolfram, Christopher, et al.
Published: (2025)

In-Memory Learning: A Declarative Learning Framework for Large Language Models
by: Wang, Bo, et al.
Published: (2024)

Training-Free Long-Context Scaling of Large Language Models
by: An, Chenxin, et al.
Published: (2024)

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
by: Zhang, Mozhi, et al.
Published: (2024)

Evaluating the Performance of Large Language Models on GAOKAO Benchmark
by: Zhang, Xiaotian, et al.
Published: (2023)

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
by: Zong, Yi, et al.
Published: (2024)

Can AI Assistants Know What They Don't Know?
by: Cheng, Qinyuan, et al.
Published: (2024)

Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts
by: Wu, Xuyang, et al.
Published: (2024)

Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
by: Xu, Ningyu, et al.
Published: (2026)

Order-Level Attention Similarity Across Language Models: A Latent Commonality
by: Liang, Jinglin, et al.
Published: (2025)

Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Models
by: Liu, Peiju, et al.
Published: (2026)

SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
by: Zhang, Xin, et al.
Published: (2023)

Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Framework
by: Sun, Yuhong, et al.
Published: (2025)

Scaling Laws for Fact Memorization of Large Language Models
by: Lu, Xingyu, et al.
Published: (2024)

LongWanjuan: Towards Systematic Measurement for Long Text Quality
by: Lv, Kai, et al.
Published: (2024)

Calibrating the Confidence of Large Language Models by Eliciting Fidelity
by: Zhang, Mozhi, et al.
Published: (2024)

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
by: Zhang, Yechen, et al.
Published: (2026)

Toward Mechanistic Explanation of Deductive Reasoning in Language Models
by: Maltoni, Davide, et al.
Published: (2025)

Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge
by: Luo, Qi, et al.
Published: (2025)

Full Parameter Fine-tuning for Large Language Models with Limited Resources
by: Lv, Kai, et al.
Published: (2023)

Code Needs Comments: Enhancing Code LLMs with Comment Augmentation
by: Song, Demin, et al.
Published: (2024)

Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem
by: Sun, Yuhong, et al.
Published: (2024)

Can Language Models Learn to Skip Steps?
by: Liu, Tengxiao, et al.
Published: (2024)