Cover Image

How do Multimodal Foundation Models Encode Text and Speech? An Analysis of Cross-Lingual and Cross-Modal Representations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lee, Hyunji, Liu, Danni, Sinhamahapatra, Supriti, Niehues, Jan
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Do Slides Help? Multi-modal Context for Automatic Transcription of Conference Talks
by: Sinhamahapatra, Supriti, et al.
Published: (2025)

Middle-Layer Representation Alignment for Cross-Lingual Transfer in Fine-Tuned LLMs
by: Liu, Danni, et al.
Published: (2025)

How Transferable are Attribute Controllers on Pretrained Multilingual Translation Models?
by: Liu, Danni, et al.
Published: (2023)

MUSCAT: MUltilingual, SCientific ConversATion Benchmark
by: Sinhamahapatra, Supriti, et al.
Published: (2026)

CCFQA: A Benchmark for Cross-Lingual and Cross-Modal Speech and Text Factuality Evaluation
by: Du, Yexing, et al.
Published: (2025)

Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approach
by: Li, Siqi, et al.
Published: (2024)

Conditions for Catastrophic Forgetting in Multilingual Translation
by: Liu, Danni, et al.
Published: (2025)

Language-Independent Representations Improve Zero-Shot Summarization
by: Solovyev, Vladimir, et al.
Published: (2024)

Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
by: Omnilingual SONAR Team, et al.
Published: (2026)

POTSA: A Cross-Lingual Speech Alignment Framework for Speech-to-Text Translation
by: Li, Xuanchen, et al.
Published: (2025)

Cross-Lingual Interleaving for Speech Language Models
by: Moumen, Adel, et al.
Published: (2025)

Generation-Step-Aware Framework for Cross-Modal Representation and Control in Multilingual Speech-Text Models
by: Nakai, Toshiki, et al.
Published: (2026)

Cross-Lingual Transfer Learning for Speech Translation
by: Ma, Rao, et al.
Published: (2024)

EmoAra: Emotion-Preserving English Speech Transcription and Cross-Lingual Translation with Arabic Text-to-Speech
by: Hassan, Besher, et al.
Published: (2026)

MACT: Model-Agnostic Cross-Lingual Training for Discourse Representation Structure Parsing
by: Liu, Jiangming
Published: (2024)

Universal Cross-Lingual Text Classification
by: Savant, Riya, et al.
Published: (2024)

Zero Resource Cross-Lingual Part Of Speech Tagging
by: Chopra, Sahil
Published: (2024)

XTRA: Cross-Lingual Topic Modeling with Topic and Representation Alignments
by: Nguyen, Tien Phat, et al.
Published: (2025)

mOthello: When Do Cross-Lingual Representation Alignment and Cross-Lingual Transfer Emerge in Multilingual Models?
by: Hua, Tianze, et al.
Published: (2024)

Cross-Modal Robustness Transfer (CMRT): Training Robust Speech Translation Models Using Adversarial Text
by: Issam, Abderrahmane, et al.
Published: (2026)

Cross-Lingual Conversational Speech Summarization with Large Language Models
by: Nelson, Max, et al.
Published: (2024)

MuSaG: A Multimodal German Sarcasm Dataset with Full-Modal Annotations
by: Scott, Aaron, et al.
Published: (2025)

A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation
by: Meyer, Francois, et al.
Published: (2024)

Description-based Controllable Text-to-Speech with Cross-Lingual Voice Control
by: Yamamoto, Ryuichi, et al.
Published: (2024)

Contrastive Learning for Task-Independent SpeechLLM-Pretraining
by: Züfle, Maike, et al.
Published: (2024)

Modality Matching Matters: Calibrating Language Distances for Cross-Lingual Transfer in URIEL+
by: Ng, York Hay, et al.
Published: (2025)

Neither Here Nor There: Cross-Lingual Representation Dynamics of Code-Mixed Text in Multilingual Encoders
by: Mazumder, Debajyoti, et al.
Published: (2026)

Why Better Cross-Lingual Alignment Fails for Better Cross-Lingual Transfer: Case of Encoders
by: Veitsman, Yana, et al.
Published: (2026)

CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models
by: Zhang, Yongheng, et al.
Published: (2025)

Cross-Lingual Pitfalls: Automatic Probing Cross-Lingual Weakness of Multilingual Large Language Models
by: Xu, Zixiang, et al.
Published: (2025)

Blending LLMs into Cascaded Speech Translation: KIT's Offline Speech Translation System for IWSLT 2024
by: Koneru, Sai, et al.
Published: (2024)

In-context Language Learning for Endangered Languages in Speech Recognition
by: Li, Zhaolin, et al.
Published: (2025)

Cross-lingual Matryoshka Representation Learning across Speech and Text
by: Sy, Yaya, et al.
Published: (2026)

Quantifying Cross-Lingual Transfer in Paralinguistic Speech Tasks
by: Buitrago, Pol, et al.
Published: (2026)

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception
by: Han, HyoJung, et al.
Published: (2024)

Training-Free Cross-Lingual Dysarthria Severity Assessment via Phonological Subspace Analysis in Self-Supervised Speech Representations
by: Muller, Bernard, et al.
Published: (2026)

Unsupervised Cross-Lingual Part-of-Speech Tagging with Monolingual Corpora Only
by: Zheng, Jianyu
Published: (2026)

A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding
by: Torunoğlu-Selamet, Dilara, et al.
Published: (2026)

Cross-Lingual Optimization for Language Transfer in Large Language Models
by: Lee, Jungseob, et al.
Published: (2025)

SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoning
by: Wang, Bin, et al.
Published: (2023)