Image de couverture de livre

Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yadav, Saumitra, Shrivastava, Manish
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Get away with less: Need of source side data curation to build parallel corpus for low resource Machine Translation
par: Yadav, Saumitra, et autres
Publié: (2026)

Theoretical Analysis of Byte-Pair Encoding
par: Kozma, László, et autres
Publié: (2024)

Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
par: Hu, Yifan, et autres
Publié: (2025)

A Formal Perspective on Byte-Pair Encoding
par: Zouhar, Vilém, et autres
Publié: (2023)

Language Models over Canonical Byte-Pair Encodings
par: Vieira, Tim, et autres
Publié: (2025)

From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
par: Zhang, Wanpeng, et autres
Publié: (2024)

Analyzing Byte-Pair Encoding on Monophonic and Polyphonic Symbolic Music: A Focus on Musical Phrase Segmentation
par: Le, Dinh-Viet-Toan, et autres
Publié: (2024)

Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
par: Schmidt, Craig W., et autres
Publié: (2025)

Can Constructions "SCAN" Compositionality ?
par: Katrapati, Ganesh, et autres
Publié: (2025)

Hybrid Tokenization Strategy for DNA Language Model using Byte Pair Encoding and K-MER Methods
par: Sapkota, Ganesh, et autres
Publié: (2025)

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
par: Foroutan, Negar, et autres
Publié: (2025)

Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition
par: Samin, Ahnaf Mozib
Publié: (2024)

Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
par: Lian, Haoran, et autres
Publié: (2024)

Peek2: Regex-free Byte-level Byte-Pair Encoding Pretokenizer for LLM Inference on Edge Devices
par: Zai, Liu, et autres
Publié: (2026)

Beyond Single-Reward: Multi-Pair, Multi-Perspective Preference Optimization for Machine Translation
par: Wang, Hao, et autres
Publié: (2025)

Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation
par: Huang, Langlin, et autres
Publié: (2024)

MARCUS: An Event-Centric NLP Pipeline that generates Character Arcs from Narratives
par: Bhyravajjula, Sriharsh, et autres
Publié: (2025)

Automatic Normalization of Word Variations in Code-Mixed Social Media Text
par: Singh, Rajat, et autres
Publié: (2018)

MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation
par: Huang, Langlin, et autres
Publié: (2024)

LastResort at SemEval-2024 Task 3: Exploring Multimodal Emotion Cause Pair Extraction as Sequence Labelling Task
par: Mathur, Suyash Vardhan, et autres
Publié: (2024)

Segmentation-Free Streaming Machine Translation
par: Iranzo-Sánchez, Javier, et autres
Publié: (2023)

Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation
par: Allamraju, Aparajitha, et autres
Publié: (2025)

Sentiment Analysis of Code-Mixed Languages leveraging Resource Rich Languages
par: Choudhary, Nurendra, et autres
Publié: (2018)

Zero-Shot Multi-task Hallucination Detection
par: Bhamidipati, Patanjali, et autres
Publié: (2024)

TeClass: A Human-Annotated Relevance-based Headline Classification and Generation Dataset for Telugu
par: Kanumolu, Gopichand, et autres
Publié: (2024)

Emotions are Universal: Learning Sentiment Based Representations of Resource-Poor Languages using Siamese Networks
par: Choudhary, Nurendra, et autres
Publié: (2018)

Neural Network Architecture for Credibility Assessment of Textual Claims
par: Choudhary, Nurendra, et autres
Publié: (2018)

Contrastive Learning of Emoji-based Representations for Resource-Poor Languages
par: Choudhary, Nurendra, et autres
Publié: (2018)

Team ACK at SemEval-2025 Task 2: Beyond Word-for-Word Machine Translation for English-Korean Pairs
par: Lee, Daniel, et autres
Publié: (2025)

Asymmetric Conflict and Synergy in Post-training for LLM-based Multilingual Machine Translation
par: Zheng, Tong, et autres
Publié: (2025)

Segment-Based Interactive Machine Translation for Pre-trained Models
par: Navarro, Angel, et autres
Publié: (2024)

DaVinci at SemEval-2024 Task 9: Few-shot prompting GPT-3.5 for Unconventional Reasoning
par: Mathur, Suyash Vardhan, et autres
Publié: (2024)

TabXEval: Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation
par: Pancholi, Vihang, et autres
Publié: (2025)

Low-resource Machine Translation for Code-switched Kazakh-Russian Language Pair
par: Borisov, Maksim, et autres
Publié: (2025)

Different Speech Translation Models Encode and Translate Speaker Gender Differently
par: Fucci, Dennis, et autres
Publié: (2025)

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics
par: Perrella, Stefano, et autres
Publié: (2024)

A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models
par: Xu, Haoran, et autres
Publié: (2023)

Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation
par: Zhang, Ran, et autres
Publié: (2026)

A Survey of using Large Language Models for Generating Infrastructure as Code
par: Srivatsa, Kalahasti Ganesh, et autres
Publié: (2024)

Evaluating Machine Translation Models for English-Hindi Language Pairs: A Comparative Analysis
par: Shetty, Ahan Prasannakumar
Publié: (2025)