Image de couverture de livre

Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Akabe, Koichi, Kanda, Shunsuke, Oda, Yusuke, Mori, Shinsuke
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Engineering faster double-array Aho-Corasick automata
par: Kanda, Shunsuke, et autres
Publié: (2022)

LegalRikai: Open Benchmark -- Benchmark for Complex Japanese Corporate Legal Tasks
par: Fujita, Shogo, et autres
Publié: (2025)

llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length
par: Sugiura, Issa, et autres
Publié: (2025)

An Experimental Evaluation of Japanese Tokenizers for Sentiment-Based Text Classification
par: Rusli, Andre, et autres
Publié: (2024)

Exploring the Effect of Segmentation and Vocabulary Size on Speech Tokenization for Speech Language Models
par: Kando, Shunsuke, et autres
Publié: (2025)

Token Masking Improves Transformer-Based Text Classification
par: Xu, Xianglong, et autres
Publié: (2025)

Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works
par: Kanda, Taisei, et autres
Publié: (2025)

WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Models
par: Sugiura, Issa, et autres
Publié: (2025)

Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar
par: Gambardella, Andrew, et autres
Publié: (2025)

Instability in Downstream Task Performance During LLM Pretraining
par: Nishida, Yuto, et autres
Publié: (2025)

Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
par: Harada, Yuto, et autres
Publié: (2025)

Llama-Mimi: Exploring the Limits of Flattened Speech Language Modeling
par: Sugiura, Issa, et autres
Publié: (2025)

Do Self-Supervised Speech Models Exhibit the Critical Period Effects in Language Acquisition?
par: Koga, Yurie, et autres
Publié: (2025)

Automatic Construction of a Large-Scale Corpus for Geoparsing Using Wikipedia Hyperlinks
par: Ohno, Keyaki, et autres
Publié: (2024)

Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction
par: Qian, Junlang, et autres
Publié: (2025)

Improving Interpretability of Lexical Semantic Change with Neurobiological Features
par: Oda, Kohei, et autres
Publié: (2026)

A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP
par: Ono, Shinnosuke, et autres
Publié: (2025)

Adaptive Targeted Dynamic Chunking for Tokenization-Free Hierarchical Model
par: Dang, Thang, et autres
Publié: (2026)

Difficult for Whom? A Study of Japanese Lexical Complexity
par: Nohejl, Adam, et autres
Publié: (2024)

EfficientXLang: Towards Improving Token Efficiency Through Cross-Lingual Reasoning
par: Ahuja, Sanchit, et autres
Publié: (2025)

Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses
par: Yu, Fangyi, et autres
Publié: (2025)

Linear-size Suffix Tries and Linear-size CDAWGs Simplified and Improved
par: Inenaga, Shunsuke
Publié: (2024)

Textless Dependency Parsing by Labeled Sequence Prediction
par: Kando, Shunsuke, et autres
Publié: (2024)

Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention
par: Fountzoulas, George
Publié: (2026)

ModernBERT is More Efficient than Conventional BERT for Chest CT Findings Classification in Japanese Radiology Reports
par: Yamagishi, Yosuke, et autres
Publié: (2025)

Refactoring Programs Using Large Language Models with Few-Shot Examples
par: Shirafuji, Atsushi, et autres
Publié: (2023)

Recipe Generation from Unsegmented Cooking Videos
par: Nishimura, Taichi, et autres
Publié: (2022)

Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
par: Xu, Yuzheng, et autres
Publié: (2026)

Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification
par: Yun, Jungmin, et autres
Publié: (2024)

MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
par: Ahia, Orevaoghene, et autres
Publié: (2024)

A BERTology View of LLM Orchestrations: Token- and Layer-Selective Probes for Efficient Single-Pass Classification
par: Meyoyan, Gonzalo Ariel, et autres
Publié: (2026)

Improving Applicability of Deep Learning based Token Classification models during Training
par: Mehra, Anket, et autres
Publié: (2025)

Semantic Token Clustering for Efficient Uncertainty Quantification in Large Language Models
par: Cao, Qi, et autres
Publié: (2026)

Extended Japanese Commonsense Morality Dataset with Masked Token and Label Enhancement
par: Ohashi, Takumi, et autres
Publié: (2024)

Empirical Study of Mutual Reinforcement Effect and Application in Few-shot Text Classification Tasks via Prompt
par: Gan, Chengguang, et autres
Publié: (2024)

The Devil Is in the Word Alignment Details: On Translation-Based Cross-Lingual Transfer for Token Classification Tasks
par: Ebing, Benedikt, et autres
Publié: (2025)

Proof systems for partial incorrectness logic (partial reverse Hoare logic)
par: Oda, Yukihiro
Publié: (2025)

Just Pass Twice: Efficient Token Classification with LLMs for Zero-Shot NER
par: Ewais, Ahmed, et autres
Publié: (2026)

Text Classification Based on Knowledge Graphs and Improved Attention Mechanism
par: Li, Siyu, et autres
Publié: (2024)

Efficient Pre-Training with Token Superposition
par: Peng, Bowen, et autres
Publié: (2026)