Cover Image

Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Papi, Sara, Gilabert, Javier Garcia, Hopton, Zachary, Zouhar, Vilém, Escolano, Carlos, Gállego, Gerard I., Iranzo-Sánchez, Jorge, Kim, Ahrii, Macháček, Dominik, Schmidtova, Patricia, Züfle, Maike
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Sound
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

How "Real" is Your Real-Time Simultaneous Speech-to-Text Translation System?
by: Papi, Sara, et al.
Published: (2024)

DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs
by: Papi, Sara, et al.
Published: (2026)

A Bayesian Optimization Approach to Machine Translation Reranking
by: Cheng, Julius, et al.
Published: (2024)

Revisiting Direct Speech-to-Text Translation with Speech LLMs: Better Scaling than CoT Prompting?
by: Pareras, Oriol, et al.
Published: (2025)

AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation
by: Papi, Sara, et al.
Published: (2023)

SimulSeamless: FBK at IWSLT 2024 Simultaneous Speech Translation
by: Papi, Sara, et al.
Published: (2024)

Early-Exit and Instant Confidence Translation Quality Estimation
by: Zouhar, Vilém, et al.
Published: (2025)

MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks
by: Papi, Sara, et al.
Published: (2025)

Listening or Reading? Evaluating Speech Awareness in Chain-of-Thought Speech-to-Text Translation
by: Romero-Díaz, Jacobo, et al.
Published: (2025)

StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection
by: Papi, Sara, et al.
Published: (2024)

Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing
by: Zouhar, Vilém
Published: (2024)

Summarizing Speech: A Comprehensive Survey
by: Retkowski, Fabian, et al.
Published: (2025)

Beyond Transcripts: A Renewed Perspective on Audio Chaptering
by: Retkowski, Fabian, et al.
Published: (2026)

COMET-poly: Machine Translation Metric Grounded in Other Candidates
by: Züfle, Maike, et al.
Published: (2025)

Cross-Attention is Half Explanation in Speech-to-Text Models
by: Papi, Sara, et al.
Published: (2025)

Towards Environmental Preference Based Speech Enhancement For Individualised Multi-Modal Hearing Aids
by: Kirton-Wingate, Jasper, et al.
Published: (2024)

Speech-to-Text Translation with Phoneme-Augmented CoT: Enhancing Cross-Lingual Transfer in Low-Resource Scenarios
by: Gállego, Gerard I., et al.
Published: (2025)

Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison
by: Lam, Tsz Kin, et al.
Published: (2025)

Speech Separation for Hearing-Impaired Children in the Classroom
by: Olalere, Feyisayo, et al.
Published: (2025)

Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs
by: Futami, Hayato, et al.
Published: (2025)

Pearmut: Human Evaluation of Translation Made Trivial
by: Zouhar, Vilém, et al.
Published: (2026)

Quality and Quantity of Machine Translation References for Automatic Metrics
by: Zouhar, Vilém, et al.
Published: (2024)

Look Once to Hear: Target Speech Hearing with Noisy Examples
by: Veluri, Bandhav, et al.
Published: (2024)

SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation
by: Deng, Keqi, et al.
Published: (2025)

Hearing More with Less: Multi-Modal Retrieval-and-Selection Augmented Conversational LLM-Based ASR
by: Mu, Bingshen, et al.
Published: (2025)

How Important is `Perfect' English for Machine Translation Prompts?
by: Schmidtová, Patrícia, et al.
Published: (2025)

Using Speech Foundational Models in Loss Functions for Hearing Aid Speech Enhancement
by: Sutherland, Robert, et al.
Published: (2024)

Can We Hear from Events? Generating Speech from Event Camera
by: Fang, Jingping, et al.
Published: (2026)

LongSpeech: A Scalable Benchmark for Transcription, Translation and Understanding in Long Speech
by: Yang, Fei, et al.
Published: (2026)

SpeechT: Findings of the First Mentorship in Speech Translation
by: Moslem, Yasmin, et al.
Published: (2025)

HI-TransPA: Hearing Impairments Translation Personal Assistant
by: Ma, Zhiming, et al.
Published: (2025)

Protecting Bystander Privacy via Selective Hearing in Audio LLMs
by: Zhan, Xiao, et al.
Published: (2025)

DeepGESI: A Non-Intrusive Objective Evaluation Model for Predicting Speech Intelligibility in Hearing-Impaired Listeners
by: Luo, Wenyu, et al.
Published: (2025)

FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian
by: Papi, Sara, et al.
Published: (2025)

Continuous Speech Tokens Makes LLMs Robust Multi-Modality Learners
by: Yuan, Ze, et al.
Published: (2024)

Optimized Self-supervised Training with BEST-RQ for Speech Recognition
by: Baumann, Ilja, et al.
Published: (2025)

NaijaS2ST: A Multi-Accent Benchmark for Speech-to-Speech Translation in Low-Resource Nigerian Languages
by: Maltais, Marie, et al.
Published: (2026)

Lend me an Ear: Speech Enhancement Using a Robotic Arm with a Microphone Array
by: Turcotte, Zachary, et al.
Published: (2026)

Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025
by: Macháček, Dominik, et al.
Published: (2025)

UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice
by: Cheng, Sitong, et al.
Published: (2025)