Kotlin ML Pack: Technical Report
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916265231122432 |
|---|---|
| author | Titov, Sergey Evtikhiev, Mikhail Shapkin, Anton Smirnov, Oleg Boytsov, Sergei Boytsov, Sergei Karaeva, Dariia Sheptyakov, Maksim Arkhipov, Mikhail Bryksin, Timofey Bogomolov, Egor |
| author_facet | Titov, Sergey Evtikhiev, Mikhail Shapkin, Anton Smirnov, Oleg Boytsov, Sergei Boytsov, Sergei Karaeva, Dariia Sheptyakov, Maksim Arkhipov, Mikhail Bryksin, Timofey Bogomolov, Egor |
| contents | In this technical report, we present three novel datasets of Kotlin code: KStack, KStack-clean, and KExercises. We also describe the results of fine-tuning CodeLlama and DeepSeek models on this data. Additionally, we present a version of the HumanEval benchmark rewritten by human experts into Kotlin - both the solutions and the tests. Our results demonstrate that small, high-quality datasets (KStack-clean and KExercises) can significantly improve model performance on code generation tasks, achieving up to a 16-point increase in pass rate on the HumanEval benchmark. Lastly, we discuss potential future work in the field of improving language modeling for Kotlin, including the use of static analysis tools in the learning process and the introduction of more intricate and realistic benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_19250 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Kotlin ML Pack: Technical Report Titov, Sergey Evtikhiev, Mikhail Shapkin, Anton Smirnov, Oleg Boytsov, Sergei Boytsov, Sergei Karaeva, Dariia Sheptyakov, Maksim Arkhipov, Mikhail Bryksin, Timofey Bogomolov, Egor Software Engineering Artificial Intelligence Programming Languages In this technical report, we present three novel datasets of Kotlin code: KStack, KStack-clean, and KExercises. We also describe the results of fine-tuning CodeLlama and DeepSeek models on this data. Additionally, we present a version of the HumanEval benchmark rewritten by human experts into Kotlin - both the solutions and the tests. Our results demonstrate that small, high-quality datasets (KStack-clean and KExercises) can significantly improve model performance on code generation tasks, achieving up to a 16-point increase in pass rate on the HumanEval benchmark. Lastly, we discuss potential future work in the field of improving language modeling for Kotlin, including the use of static analysis tools in the learning process and the introduction of more intricate and realistic benchmarks. |
| title | Kotlin ML Pack: Technical Report |
| topic | Software Engineering Artificial Intelligence Programming Languages |
| url | https://arxiv.org/abs/2405.19250 |