Cover Image

Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ye, Chunyu, Zhang, Yunhao, Sun, Jingyuan, Li, Chong, Zong, Chengqing, Wang, Shaonan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Computational Models to Study Language Processing in the Human Brain: A Survey
by: Wang, Shaonan, et al.
Published: (2024)

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment
by: Li, Chong, et al.
Published: (2023)

Component-Level Lesioning of Language Models Reveals Clinically Aligned Aphasia Phenotypes
by: Wang, Yifan, et al.
Published: (2026)

MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models
by: Zhang, Yunhao, et al.
Published: (2024)

Navigating Brain Language Representations: A Comparative Analysis of Neural Language Models and Psychologically Plausible Models
by: Zhang, Yunhao, et al.
Published: (2024)

MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities
by: Zhao, Xinpei, et al.
Published: (2024)

Discovering Semantic Subdimensions through Disentangled Conceptual Representations
by: Zhang, Yunhao, et al.
Published: (2025)

Computational Lesions in Multilingual Language Models Separate Shared and Language-specific Brain Alignment
by: Cui, Yang, et al.
Published: (2026)

TokAlign: Efficient Vocabulary Adaptation via Token Alignment
by: Li, Chong, et al.
Published: (2025)

Bridging Brains and Models: MoE-Based Functional Lesions for Simulating and Rehabilitating Aphasia
by: Wang, Yifan, et al.
Published: (2025)

X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions
by: Li, Chong, et al.
Published: (2024)

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
by: Li, Chong, et al.
Published: (2026)

Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
by: Liang, Yupu, et al.
Published: (2025)

From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignment
by: Ye, Jing, et al.
Published: (2025)

F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation
by: Wu, Junhong, et al.
Published: (2024)

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
by: Li, Chong, et al.
Published: (2025)

Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation
by: Wu, Junhong, et al.
Published: (2024)

Dual-branch Prompting for Multimodal Machine Translation
by: Wang, Jie, et al.
Published: (2025)

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
by: Yang, Wen, et al.
Published: (2025)

SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agent
by: Ye, Jing, et al.
Published: (2024)

Self-Modifying State Modeling for Simultaneous Machine Translation
by: Yu, Donglei, et al.
Published: (2024)

Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
by: Yang, Wen, et al.
Published: (2025)

EmoHarbor: Evaluating Personalized Emotional Support by Simulating the User's Internal World
by: Ye, Jing, et al.
Published: (2026)

DEER: Disentangled Mixture of Experts with Instance-Adaptive Routing for Generalizable Machine-Generated Text Detection
by: Ma, Guoxin, et al.
Published: (2025)

TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification
by: Zhu, Junnan, et al.
Published: (2025)

UMBRAE: Unified Multimodal Brain Decoding
by: Xia, Weihao, et al.
Published: (2024)

Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs
by: Gao, Xinyu, et al.
Published: (2026)

DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning
by: He, Kang, et al.
Published: (2025)

The Case for Evaluating Multimodal Translation Models on Text Datasets
by: Vijayan, Vipin, et al.
Published: (2024)

Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion
by: Du, Yexing, et al.
Published: (2026)

Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models
by: Moussa, Omer, et al.
Published: (2025)

Enhance the Robustness of Text-Centric Multimodal Alignments
by: Yen, Ting-Yu, et al.
Published: (2024)

A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities
by: Xiang, Lu, et al.
Published: (2025)

Adding Multimodal Capabilities to a Text-only Translation Model
by: Vijayan, Vipin, et al.
Published: (2024)

SimulPL: Aligning Human Preferences in Simultaneous Machine Translation
by: Yu, Donglei, et al.
Published: (2025)

BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing
by: Wang, Chen, et al.
Published: (2023)

Prompt-tuning for Clickbait Detection via Text Summarization
by: Deng, Haoxiang, et al.
Published: (2024)

How Syntax Specialization Emerges in Language Models
by: Duan, Xufeng, et al.
Published: (2025)

Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
by: Liang, Yupu, et al.
Published: (2025)

ChineseWebText 2.0: Large-Scale High-quality Chinese Web Text with Multi-dimensional and fine-grained information
by: Zhang, Wanyue, et al.
Published: (2024)