Cover Image

Pretraining Finnish ModernBERTs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Reunamo, Akseli, Peltonen, Laura-Maria, Moen, Hans, Pyysalo, Sampo
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models
by: Kytöniemi, Joona, et al.
Published: (2025)

Got Compute, but No Data: Lessons From Post-training a Finnish LLM
by: Zosa, Elaine, et al.
Published: (2025)

A Survey of Large Language Models for European Languages
by: Ali, Wazir, et al.
Published: (2024)

Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation
by: Myntti, Amanda, et al.
Published: (2025)

Patent Language Model Pretraining with ModernBERT
by: Yousefiramandi, Amirhossein, et al.
Published: (2025)

Chinese ModernBERT with Whole-Word Masking
by: Zhao, Zeyu, et al.
Published: (2025)

BERTs are Generative In-Context Learners
by: Samuel, David
Published: (2024)

Clinical ModernBERT: An efficient and long context encoder for biomedical text
by: Lee, Simon A., et al.
Published: (2025)

A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT
by: Estève, Louis, et al.
Published: (2026)

Fine-Tuning BERTs for Definition Extraction from Mathematical Text
by: Horowitz, Lucy, et al.
Published: (2024)

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish
by: Türker, Melikşah, et al.
Published: (2025)

Spatial ModernBERT: Spatial-Aware Transformer for Table and Key-Value Extraction in Financial Documents at Scale
by: Javis AI Team, et al.
Published: (2025)

Poro 34B and the Blessing of Multilinguality
by: Luukkonen, Risto, et al.
Published: (2024)

ModernBERT + ColBERT: Enhancing biomedical RAG through an advanced re-ranking retriever
by: Rivera, Eduardo Martínez, et al.
Published: (2025)

NorBERTo: A ModernBERT Model Trained for Portuguese with 331 Billion Tokens Corpus
by: Silva, Enzo S. N., et al.
Published: (2026)

ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance
by: Antoun, Wissam, et al.
Published: (2025)

ModernBERT is More Efficient than Conventional BERT for Chest CT Findings Classification in Japanese Radiology Reports
by: Yamagishi, Yosuke, et al.
Published: (2025)

BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP
by: Sounack, Thomas, et al.
Published: (2025)

Exploring Linguistic Properties of Monolingual BERTs with Typological Classification among Languages
by: Ruzzetti, Elena Sofia, et al.
Published: (2023)

llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length
by: Sugiura, Issa, et al.
Published: (2025)

fairBERTs: Erasing Sensitive Information Through Semantic and Fairness-aware Perturbations
by: Li, Jinfeng, et al.
Published: (2024)

Pretraining and Benchmarking Modern Encoders for Latvian
by: Znotins, Arturs
Published: (2026)

Scaling Data-Constrained Language Models
by: Muennighoff, Niklas, et al.
Published: (2023)

CBR-to-SQL: Rethinking Retrieval-based Text-to-SQL using Case-based Reasoning in the Healthcare Domain
by: Nguyen, Hung, et al.
Published: (2026)

A New Massive Multilingual Dataset for High-Performance Language Technologies
by: de Gibert, Ona, et al.
Published: (2024)

Combining Qualitative and Computational Approaches for Literary Analysis of Finnish Novels
by: Ohman, Emily, et al.
Published: (2024)

LLMs' morphological analyses of complex FST-generated Finnish words
by: Moisio, Anssi, et al.
Published: (2024)

PretrainZero: Reinforcement Active Pretraining
by: Xing, Xingrun, et al.
Published: (2025)

Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations
by: Nuutinen, Emil, et al.
Published: (2025)

Analyzing Finnish Inflectional Classes through Discriminative Lexicon and Deep Learning Models
by: Nikolaev, Alexandre, et al.
Published: (2025)

Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs
by: Laato, Joonatan, et al.
Published: (2025)

Query-Guided Self-Supervised Summarization of Nursing Notes
by: Gao, Ya, et al.
Published: (2024)

Hopes and Fears -- Emotion Distribution in the Topic Landscape of Finnish Parliamentary Speech 2000-2020
by: Ristilä, Anna, et al.
Published: (2026)

Vision-and-Language Pretraining
by: Nguyen, Thong, et al.
Published: (2022)

Threefold model for AI Readiness: A Case Study with Finnish Healthcare SMEs
by: Alnajjar, Mohammed, et al.
Published: (2025)

Tokenization Strategies for Low-Resource Agglutinative Languages in Word2Vec: Case Study on Turkish and Finnish
by: Hu, Jinfan Frank
Published: (2025)

Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations
by: Luisto, Rami, et al.
Published: (2026)

Efficient Pretraining Length Scaling
by: Wu, Bohong, et al.
Published: (2025)

Reformulation for Pretraining Data Augmentation
by: Hao, Xintong, et al.
Published: (2025)

Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models
by: Linzbach, Stephan, et al.
Published: (2024)