Compressed code: the hidden effects of quantization and distillation on programming tokens
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912886581886976 |
|---|---|
| author | Siniaev, Viacheslav Chelombitko, Iaroslav Komissarov, Aleksey |
| author_facet | Siniaev, Viacheslav Chelombitko, Iaroslav Komissarov, Aleksey |
| contents | Large Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming language token representations, we characterize how programming languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization, distillation, model scaling, and task-specific fine-tuning - affect token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02563 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Compressed code: the hidden effects of quantization and distillation on programming tokens Siniaev, Viacheslav Chelombitko, Iaroslav Komissarov, Aleksey Software Engineering Computation and Language Machine Learning Programming Languages I.2.7; I.2.2; D.3.4 Large Language Models (LLMs) have demonstrated exceptional code generation capabilities, yet their token-level mechanisms remain underexplored, particularly in compressed models. Through systematic analysis of programming language token representations, we characterize how programming languages are encoded in LLM tokenizers by analyzing their vocabulary distribution and keyword coverage patterns. We introduce a novel cold-start probability analysis method that provides insights into model behavior without requiring explicit prompts. Additionally, we present a comprehensive evaluation of how different model optimization techniques - including quantization, distillation, model scaling, and task-specific fine-tuning - affect token-level representations and code generation quality. Our experiments, supported by comprehensive probability distribution analysis and evaluation metrics, reveal critical insights into token-level behavior and provide empirically-validated guidelines for maintaining code generation quality under various optimization constraints. These findings advance both theoretical understanding of LLM code generation and practical implementation of optimized models in production environments. |
| title | Compressed code: the hidden effects of quantization and distillation on programming tokens |
| topic | Software Engineering Computation and Language Machine Learning Programming Languages I.2.7; I.2.2; D.3.4 |
| url | https://arxiv.org/abs/2601.02563 |