Imagen de Portada

Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Raposo, David, Ritter, Sam, Richards, Blake, Lillicrap, Timothy, Humphreys, Peter Conway, Santoro, Adam
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
por: Bae, Sangmin, et al.
Publicado: (2025)

A path to natural language through tokenisation and transformers
por: Berman, David S., et al.
Publicado: (2026)

Tracing the Representation Geometry of Language Models from Pretraining to Post-training
por: Li, Melody Zixuan, et al.
Publicado: (2025)

MoDification: Mixture of Depths Made Easy
por: Zhang, Chen, et al.
Publicado: (2024)

Detecting out-of-distribution text using topological features of transformer-based language models
por: Pollano, Andres, et al.
Publicado: (2023)

Zero-shot data citation function classification using transformer-based large language models (LLMs)
por: Byers, Neil, et al.
Publicado: (2025)

Comparison of different Unique hard attention transformer models by the formal languages they can recognize
por: Ryvkin, Leonid
Publicado: (2025)

Physical models realizing the transformer architecture of large language models
por: Chen, Zeqian
Publicado: (2025)

How do language models learn facts? Dynamics, curricula and hallucinations
por: Zucchet, Nicolas, et al.
Publicado: (2025)

Alignment faking in large language models
por: Greenblatt, Ryan, et al.
Publicado: (2024)

Question answering system of bridge design specification based on large language model
por: Zhang, Leye, et al.
Publicado: (2024)

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
por: Knupp, Jonas, et al.
Publicado: (2026)

Training Agents Inside of Scalable World Models
por: Hafner, Danijar, et al.
Publicado: (2025)

Exploring prompts to elicit memorization in masked language model-based named entity recognition
por: Xia, Yuxi, et al.
Publicado: (2024)

Attention based Bidirectional GRU hybrid model for inappropriate content detection in Urdu language
por: Shoukat, Ezzah, et al.
Publicado: (2025)

Aligning language models with human preferences
por: Korbak, Tomasz
Publicado: (2024)

Evaluating language models as risk scores
por: Cruz, André F., et al.
Publicado: (2024)

Dynamic layer selection in decoder-only transformers
por: Glavas, Theodore, et al.
Publicado: (2024)

Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
por: Chen, Yilong, et al.
Publicado: (2026)

Auditing language models for hidden objectives
por: Marks, Samuel, et al.
Publicado: (2025)

Amortizing intractable inference in large language models
por: Hu, Edward J., et al.
Publicado: (2023)

The language of time: a language model perspective on time-series foundation models
por: Xie, Yi, et al.
Publicado: (2025)

Perturbed examples reveal invariances shared by language models
por: Rawal, Ruchit, et al.
Publicado: (2023)

A mean teacher algorithm for unlearning of language models
por: Klochkov, Yegor
Publicado: (2025)

Do language models plan ahead for future tokens?
por: Wu, Wilson, et al.
Publicado: (2024)

Visualizing token importance for black-box language models
por: Rauba, Paulius, et al.
Publicado: (2025)

A meta-analysis on the performance of machine-learning based language models for sentiment analysis
por: Rohde, Elena, et al.
Publicado: (2025)

Continuous-Depth Transformers with Learned Control Dynamics
por: Jemley, Peter
Publicado: (2026)

Prompt reinforcing for long-term planning of large language models
por: Lin, Hsien-Chin, et al.
Publicado: (2025)

Machine-generated text detection prevents language model collapse
por: Drayson, George, et al.
Publicado: (2025)

No Need to Talk: Asynchronous Mixture of Language Models
por: Filippova, Anastasiia, et al.
Publicado: (2024)

Representation in large language models
por: Yetman, Cameron
Publicado: (2025)

A comparison of pipelines for the translation of a low resource language based on transformers
por: Bonfanti, Chiara, et al.
Publicado: (2025)

Boosting classification reliability of NLP transformer models in the long run
por: Kmetty, Zoltán, et al.
Publicado: (2023)

Investigating and Alleviating Harm Amplification in LLM Interactions
por: Guo, Ruohao, et al.
Publicado: (2026)

Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding
por: Guo, Ruohao, et al.
Publicado: (2023)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
por: Huang, Wei, et al.
Publicado: (2024)

Mastering Diverse Domains through World Models
por: Hafner, Danijar, et al.
Publicado: (2023)

Zero-shot generation of synthetic neurosurgical data with large language models
por: Barr, Austin A., et al.
Publicado: (2025)

Repetitions are not all alike: distinct mechanisms sustain repetition in language models
por: Mahaut, Matéo, et al.
Publicado: (2025)