umayer16/VIBEBENCH: v1.3.0 — Bug Fixes, CLI, Test Suite, Multi-Model Benchmark Results
Fuente:
Zenodo
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Recurso digital |
| Veröffentlicht: |
Zenodo
2026
|
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866902154175840256 |
|---|---|
| author | umayer16 |
| author_facet | umayer16 |
| contents | <h2>What's New in v1.1.0</h2> <h3>Bug Fixes</h3> <ul> <li>Fixed Halstead Volume formula (bit_length → log2)</li> <li>Fixed Halstead operator counting using Counter() to track N1/N2 separately</li> <li>Implemented ghost comment detection</li> <li>Fixed duplicate ImportFrom detection</li> <li>Expanded operator detection to all Python AST types</li> <li>Fixed misleading 100% docstring coverage for files with no functions</li> <li>Fixed credential detection regex to match keys containing underscores</li> </ul> <h3>New Features</h3> <ul> <li>Full CLI interface (analyze + benchmark subcommands via argparse)</li> <li>Google Gemini model support via core/gemini_generator.py</li> <li>Groq API support via core/groq_generator.py (LLaMA 3.3 70B, Mixtral)</li> <li>Comprehensive pytest test suite (17 tests, all passing)</li> <li>GitHub Actions CI pipeline (green)</li> <li>Proper README with installation, usage, and citation docs</li> <li>.gitignore and .env.example added</li> </ul> <h3>Research Results</h3> <p>First benchmark run comparing 6 LLMs against human baseline across 5 tasks (Data Structures, Cybersecurity, Algorithms, File I/O, Math/Logic):</p> <p>| Model | Success Rate | Avg Complexity | Avg Doc Coverage | |-------|-------------|----------------|-----------------| | ChatGPT | 100% | 4.76 | 80.0% | | Gemini | 100% | 3.90 | 95.0% | | Claude | 80% | 3.77 | 0.0% | | LLaMA 3.3 70B | 80% | 5.30 | 0.0% | | DeepSeek | 40% | 4.65 | 93.1% | | Grok | 20% | 5.24 | 82.0% | | HUMAN BASELINE | 100% | 3.60 | 20.0% |</p> <p>Full results: vibebench_multimodel_20260309_1926.json</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19094981 |
| institution | Zenodo |
| language | |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | umayer16/VIBEBENCH: v1.3.0 — Bug Fixes, CLI, Test Suite, Multi-Model Benchmark Results umayer16 <h2>What's New in v1.1.0</h2> <h3>Bug Fixes</h3> <ul> <li>Fixed Halstead Volume formula (bit_length → log2)</li> <li>Fixed Halstead operator counting using Counter() to track N1/N2 separately</li> <li>Implemented ghost comment detection</li> <li>Fixed duplicate ImportFrom detection</li> <li>Expanded operator detection to all Python AST types</li> <li>Fixed misleading 100% docstring coverage for files with no functions</li> <li>Fixed credential detection regex to match keys containing underscores</li> </ul> <h3>New Features</h3> <ul> <li>Full CLI interface (analyze + benchmark subcommands via argparse)</li> <li>Google Gemini model support via core/gemini_generator.py</li> <li>Groq API support via core/groq_generator.py (LLaMA 3.3 70B, Mixtral)</li> <li>Comprehensive pytest test suite (17 tests, all passing)</li> <li>GitHub Actions CI pipeline (green)</li> <li>Proper README with installation, usage, and citation docs</li> <li>.gitignore and .env.example added</li> </ul> <h3>Research Results</h3> <p>First benchmark run comparing 6 LLMs against human baseline across 5 tasks (Data Structures, Cybersecurity, Algorithms, File I/O, Math/Logic):</p> <p>| Model | Success Rate | Avg Complexity | Avg Doc Coverage | |-------|-------------|----------------|-----------------| | ChatGPT | 100% | 4.76 | 80.0% | | Gemini | 100% | 3.90 | 95.0% | | Claude | 80% | 3.77 | 0.0% | | LLaMA 3.3 70B | 80% | 5.30 | 0.0% | | DeepSeek | 40% | 4.65 | 93.1% | | Grok | 20% | 5.24 | 82.0% | | HUMAN BASELINE | 100% | 3.60 | 20.0% |</p> <p>Full results: vibebench_multimodel_20260309_1926.json</p> |
| title | umayer16/VIBEBENCH: v1.3.0 — Bug Fixes, CLI, Test Suite, Multi-Model Benchmark Results |
| url | https://doi.org/10.5281/zenodo.19094981 |