Imagen de Portada

Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Yadav, Saumitra, Shrivastava, Manish
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Get away with less: Need of source side data curation to build parallel corpus for low resource Machine Translation
por: Yadav, Saumitra, et al.
Publicado: (2026)

Theoretical Analysis of Byte-Pair Encoding
por: Kozma, László, et al.
Publicado: (2024)

Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
por: Hu, Yifan, et al.
Publicado: (2025)

A Formal Perspective on Byte-Pair Encoding
por: Zouhar, Vilém, et al.
Publicado: (2023)

Language Models over Canonical Byte-Pair Encodings
por: Vieira, Tim, et al.
Publicado: (2025)

From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
por: Zhang, Wanpeng, et al.
Publicado: (2024)

Analyzing Byte-Pair Encoding on Monophonic and Polyphonic Symbolic Music: A Focus on Musical Phrase Segmentation
por: Le, Dinh-Viet-Toan, et al.
Publicado: (2024)

Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
por: Schmidt, Craig W., et al.
Publicado: (2025)

Can Constructions "SCAN" Compositionality ?
por: Katrapati, Ganesh, et al.
Publicado: (2025)

Hybrid Tokenization Strategy for DNA Language Model using Byte Pair Encoding and K-MER Methods
por: Sapkota, Ganesh, et al.
Publicado: (2025)

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
por: Foroutan, Negar, et al.
Publicado: (2025)

Byte Pair Encoding Is All You Need For Automatic Bengali Speech Recognition
por: Samin, Ahnaf Mozib
Publicado: (2024)

Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal
por: Lian, Haoran, et al.
Publicado: (2024)

Peek2: Regex-free Byte-level Byte-Pair Encoding Pretokenizer for LLM Inference on Edge Devices
por: Zai, Liu, et al.
Publicado: (2026)

Beyond Single-Reward: Multi-Pair, Multi-Perspective Preference Optimization for Machine Translation
por: Wang, Hao, et al.
Publicado: (2025)

Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation
por: Huang, Langlin, et al.
Publicado: (2024)

MARCUS: An Event-Centric NLP Pipeline that generates Character Arcs from Narratives
por: Bhyravajjula, Sriharsh, et al.
Publicado: (2025)

Automatic Normalization of Word Variations in Code-Mixed Social Media Text
por: Singh, Rajat, et al.
Publicado: (2018)

MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation
por: Huang, Langlin, et al.
Publicado: (2024)

LastResort at SemEval-2024 Task 3: Exploring Multimodal Emotion Cause Pair Extraction as Sequence Labelling Task
por: Mathur, Suyash Vardhan, et al.
Publicado: (2024)

Segmentation-Free Streaming Machine Translation
por: Iranzo-Sánchez, Javier, et al.
Publicado: (2023)

Breaking It Down: Domain-Aware Semantic Segmentation for Retrieval Augmented Generation
por: Allamraju, Aparajitha, et al.
Publicado: (2025)

Sentiment Analysis of Code-Mixed Languages leveraging Resource Rich Languages
por: Choudhary, Nurendra, et al.
Publicado: (2018)

Zero-Shot Multi-task Hallucination Detection
por: Bhamidipati, Patanjali, et al.
Publicado: (2024)

TeClass: A Human-Annotated Relevance-based Headline Classification and Generation Dataset for Telugu
por: Kanumolu, Gopichand, et al.
Publicado: (2024)

Emotions are Universal: Learning Sentiment Based Representations of Resource-Poor Languages using Siamese Networks
por: Choudhary, Nurendra, et al.
Publicado: (2018)

Neural Network Architecture for Credibility Assessment of Textual Claims
por: Choudhary, Nurendra, et al.
Publicado: (2018)

Contrastive Learning of Emoji-based Representations for Resource-Poor Languages
por: Choudhary, Nurendra, et al.
Publicado: (2018)

Team ACK at SemEval-2025 Task 2: Beyond Word-for-Word Machine Translation for English-Korean Pairs
por: Lee, Daniel, et al.
Publicado: (2025)

Asymmetric Conflict and Synergy in Post-training for LLM-based Multilingual Machine Translation
por: Zheng, Tong, et al.
Publicado: (2025)

Segment-Based Interactive Machine Translation for Pre-trained Models
por: Navarro, Angel, et al.
Publicado: (2024)

DaVinci at SemEval-2024 Task 9: Few-shot prompting GPT-3.5 for Unconventional Reasoning
por: Mathur, Suyash Vardhan, et al.
Publicado: (2024)

TabXEval: Why this is a Bad Table? An eXhaustive Rubric for Table Evaluation
por: Pancholi, Vihang, et al.
Publicado: (2025)

Low-resource Machine Translation for Code-switched Kazakh-Russian Language Pair
por: Borisov, Maksim, et al.
Publicado: (2025)

Different Speech Translation Models Encode and Translate Speaker Gender Differently
por: Fucci, Dennis, et al.
Publicado: (2025)

Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics
por: Perrella, Stefano, et al.
Publicado: (2024)

A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models
por: Xu, Haoran, et al.
Publicado: (2023)

Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation
por: Zhang, Ran, et al.
Publicado: (2026)

A Survey of using Large Language Models for Generating Infrastructure as Code
por: Srivatsa, Kalahasti Ganesh, et al.
Publicado: (2024)

Evaluating Machine Translation Models for English-Hindi Language Pairs: A Comparative Analysis
por: Shetty, Ahan Prasannakumar
Publicado: (2025)