Cover Image

Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jiang, Changhao, Zhang, Ming, Cao, Yifei, Ye, Junjie, Fan, Xiaoran, Dou, Shihan, Xi, Zhiheng, Sun, Jiajun, Dong, Yi, Shen, Yujiong, Tong, Jingqi, Fan, Baoyu, Gui, Tao, Zhang, Qi, Huang, Xuanjing
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents
by: Jiang, Changhao, et al.
Published: (2025)

CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges
by: Li, Hui, et al.
Published: (2025)

LLMEval-Med: A Real-world Clinical Benchmark for Medical LLMs with Physician Validation
by: Zhang, Ming, et al.
Published: (2025)

Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments
by: Ye, Junjie, et al.
Published: (2025)

Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations
by: Li, Shuo, et al.
Published: (2025)

From Scores to Preferences: Redefining MOS Benchmarking for Speech Quality Reward Modeling
by: Cao, Yifei, et al.
Published: (2025)

JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
by: Wang, Yuhui, et al.
Published: (2026)

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents
by: Shen, Yujiong, et al.
Published: (2026)

LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models
by: Zhang, Ming, et al.
Published: (2025)

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
by: Dou, Shihan, et al.
Published: (2023)

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening
by: Zhang, Ming, et al.
Published: (2026)

Distill Visual Chart Reasoning Ability from LLMs to MLLMs
by: He, Wei, et al.
Published: (2024)

ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios
by: Ye, Junjie, et al.
Published: (2024)

OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment
by: Zhang, Ming, et al.
Published: (2026)

Upper Bounds for Sequence Saturation
by: Kanungo, Shihan
Published: (2025)

TransferTOD: A Generalizable Chinese Multi-Domain Task-Oriented Dialogue System with Transfer Capabilities
by: Zhang, Ming, et al.
Published: (2024)

PFDial: A Structured Dialogue Instruction Fine-tuning Method Based on UML Flowcharts
by: Zhang, Ming, et al.
Published: (2025)

ChartE$^{3}$: A Comprehensive Benchmark for End-to-End Chart Editing
by: Li, Shuo, et al.
Published: (2026)

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
by: Dou, Shihan, et al.
Published: (2024)

TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use
by: Ye, Junjie, et al.
Published: (2024)

Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
by: Wang, Junzhe, et al.
Published: (2026)

Pre-Trained Policy Discriminators are General Reward Models
by: Dou, Shihan, et al.
Published: (2025)

CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models
by: Lv, Huijie, et al.
Published: (2024)

Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric
by: Yang, Yuming, et al.
Published: (2025)

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning
by: Xi, Zhiheng, et al.
Published: (2024)

Muse: Towards Reproducible Long-Form Song Generation with Fine-Grained Style Control
by: Jiang, Changhao, et al.
Published: (2026)

VRPO: Rethinking Value Modeling for Robust RL Training under Noisy Supervision
by: Zhu, Dingwei, et al.
Published: (2025)

EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
by: Zhou, Yuhao, et al.
Published: (2025)

Steering LLMs via Scalable Interactive Oversight
by: Zhou, Enyu, et al.
Published: (2026)

Improving RL Exploration for LLM Reasoning through Retrospective Replay
by: Dou, Shihan, et al.
Published: (2025)

Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
by: Zhang, Zhihao, et al.
Published: (2025)

RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
by: Zhou, Enyu, et al.
Published: (2024)

Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals
by: Zheng, Rui, et al.
Published: (2024)

MouSi: Poly-Visual-Expert Vision-Language Models
by: Fan, Xiaoran, et al.
Published: (2024)

MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning
by: Lin, Jiahang, et al.
Published: (2026)

Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning
by: Xi, Zhiheng, et al.
Published: (2025)

DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training
by: Zhu, Dingwei, et al.
Published: (2026)

RoCoIns: Enhancing Robustness of Large Language Models through Code-Style Instructions
by: Zhang, Yuansen, et al.
Published: (2024)

Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
by: Lin, Jiahang, et al.
Published: (2026)

The Role of Entropy in Visual Grounding: Analysis and Optimization
by: Li, Shuo, et al.
Published: (2025)