Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tang, Bangsheng, Fu, Carl Chengyan, Kou, Fei, Sizov, Grigory, Zhang, Haoci, Park, Jason, Liu, Jiawen, You, Jie, Yang, Qirui, Mehta, Sachin, Cai, Shengyong, Wang, Xiaodong, Liu, Xingyu, Li, Yunlu, Zhou, Yanjun, Wei, Wei, Zhao, Zhiwei, Qi, Zixi, Victoria, Adolfo, Ibrahim, Aya, Wasti, Bram, Kim, Changkyu, Haziza, Daniel, Sun, Fei, Delfin, Giancarlo, Guo, Emily, Ouyang, Jialin, Lee, Jaewon, Huang, Jianyu, Reizenstein, Jeremy, Fang, Lu, Zhu, Quinn, Verma, Ria, Mihailescu, Vlad, Guo, Xingwen, Cui, Yan, Hu, Ye, Lee, Yejin
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

The protection of "state overseas interests" in China's foreign policy strategy: conceptual, legal and expert dimensions of the discussion
by: Sizov, Georgiy A.
Published: (2024)

Relative derived equivalences and relative Igusa-Todorov dimensions
by: Guo, Peizheng, et al.
Published: (2025)

Obstructing two-torsion in the rational knot concordance group
by: Lee, Jaewon
Published: (2024)

Rational concordance of double twist knots
by: Lee, Jaewon
Published: (2025)

Interactions between 3Al2O3·2SiO2, CA2, CA6, and Molten Slags
by: Xingwen Wei, et al.
Published: (2024)

Letter to the Editor in Response to “Association between severity of diabetic complications and risk of cancer in middle‐aged patients with type 2 diabetes”
by: Syeda Aamna Wasti
Published: (2025)

The strong Fermat-Catalan Equation
by: Mihailescu, Preda
Published: (2025)

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
by: Bambhaniya, Abhimanyu, et al.
Published: (2026)

Hydrochemistry measured on water bottle samples during Mikhail Lomonosov cruise ML33
by: Sizov, Evgeni M, et al.
Published: (2012)

Set Block Decoding is a Language Model Inference Accelerator
by: Gat, Itai, et al.
Published: (2025)

Machine learning methods for finite population parameter estimation in survey sampling
by: Dagdoug, Mehdi, et al.
Published: (2026)

Remember Your Trace: Memory-Guided Long-Horizon Agentic Framework for Consistent and Hierarchical Repository-Level Code Documentation
by: Bae, Suyoung, et al.
Published: (2026)

Denoising-Contrastive Alignment for Continuous Sign Language Recognition
by: Guo, Leming, et al.
Published: (2023)

Conjugation, loop and closure invariants of the iterated-integrals signature
by: Diehl, Joscha, et al.
Published: (2024)

Structural Thermodynamics of Membrane Electroporation: Voltage-Driven Interdigitation Induces Electro-Topological Disparity
by: Guo, Fei
Published: (2026)

Corrosion of MgO–C Refractory with Ladle Slags
by: Xingwen Wei, et al.
Published: (2024)

Variable Selection for Linear Regression Imputation in Surveys
by: An, Ziming, et al.
Published: (2026)

IR-UWB Radar-Based Contactless Silent Speech Recognition with Attention-Enhanced Temporal Convolutional Networks
by: Lee, Sunghwa, et al.
Published: (2025)

On homology spheres of the trivial local equivalence class
by: Lee, Jaewon, et al.
Published: (2025)

Architectural Tactics to Improve the Environmental Sustainability of Microservices: A Rapid Review
by: Xiao, Xingwen
Published: (2024)

Acylsilanes for Palladium‐Catalyzed Reversed‐Polarity Synthesis of Diaryl Ketones
by: Jaehoon Lee, et al.
Published: (2025)

Universal deformation rings and derived equivalences
by: Pan, Shengyong
Published: (2025)

CLIP-KOA: Enhancing Knee Osteoarthritis Diagnosis with Multi-Modal Learning and Symmetry-Aware Loss Functions
by: Jeong, Yejin, et al.
Published: (2025)

TEEMATE: Fast and Efficient Confidential Container using Shared Enclave
by: Lee, Chulmin, et al.
Published: (2024)

A simplified fixed‐time insemination protocol using frozen–thawed stallion spermatozoa stored at 17°C for up to 24 h before insemination
by: Lee Morris, et al.
Published: (2024)

Llama-Polya: Instruction Tuning for Large Language Model based on Polya's Problem-solving
by: Lee, Unggi, et al.
Published: (2026)

Beyond and slightly aside of the Fermat Desert
by: Bartolomé, Boris, et al.
Published: (2021)

Mind the Gap: Anti-Critical Quantum Metrology
by: Mihailescu, George, et al.
Published: (2026)

Statistical inference in the presence of imputed survey data through regression trees and random forests
by: Mehdi Dagdoug, et al.
Published: (2025)

On high‐dimensional variance estimation in survey sampling
by: Esther Eustache, et al.
Published: (2025)

Stability of Nonlinear System With External Disturbance Under Multiple Cyber‐Attacks Via Delay‐Dependent Impulsive Control
by: Junfeng Guo, et al.
Published: (2025)

Adaptive display white point under various ambient conditions
by: Jiamin Guo, et al.
Published: (2024)

Multiscale Feature Importance-based Bit Allocation for End-to-End Feature Coding for Machines
by: Liu, Junle, et al.
Published: (2025)

The Llama 3 Herd of Models
by: Grattafiori, Aaron, et al.
Published: (2024)

Workplace Cyberbullying Among Healthcare Workers: A Systematic Review of the Prevalence, Antecedents and Consequences
by: Wei Zhang, et al.
Published: (2025)

TranslationCorrect: A Unified Framework for Machine Translation Post-Editing with Predictive Error Assistance
by: Wasti, Syed Mekael, et al.
Published: (2025)

A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama
by: Cursaru, Vlad-Andrei, et al.
Published: (2024)

Two New Cembranoids From the Soft Coral Sinularia Sp. With Antioxidant Activity
by: Wei‐Kang Xie, et al.
Published: (2025)

TinyLlama: An Open-Source Small Language Model
by: Zhang, Peiyuan, et al.
Published: (2024)

if (filemtime($filename) + $maxlifetime < time()) { unlink($filename); $count++; } } return $count; } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ protected function saveSession($sessId, $data): bool { $sessFile = $this->path . '/sess_' . $sessId; if ($handle = fopen($sessFile, 'w')) { $return = false; // Lock the file for exclusive access to avoid issues with multiple // processes writing session simultaneously: if (flock($handle, LOCK_EX)) { $return = fwrite($handle, $data); // Make sure that there's no trailing data by truncating the file to // the correct length: ftruncate($handle, strlen($data)); fflush($handle); flock($handle, LOCK_UN); } fclose($handle); if ($return !== false) { return true; } } // If we got this far, something went wrong with the file output... // It is tempting to throw an exception here, but this code is called // outside of the context of exception handling, so all we can do is // echo a message.

// Anecdotal testing Today and Yesterday seems to indicate destroy() // is called by the garbage collector and everything is good. // Something to keep in mind though. return 0; } /** * Write function that is called when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ public function write($sessId, $data): bool { if ($this->writesDisabled) { return true; } return $this->saveSession($sessId, $data); } /** * A function that is called internally when session data is to be saved. * * @param string $sessId The current session ID * @param string $data The session data to write * * @return bool */ abstract protected function saveSession($sessId, $data): bool; }

* * @return void */ public function writeClose() { // The assumption is that we're using PHP's ext/session. // session_write_close() will actually overwrite $_SESSION with an // empty array on completion -- which leads to a mismatch between what // is in the storage object and $_SESSION. To get around this, we // temporarily reset $_SESSION to an array, and then re-link it to // the storage object. // // Additionally, while you _can_ write to $_SESSION following a // session_write_close() operation, no changes made to it will be // flushed to the session handler. As such, we now mark the storage // object isImmutable. $storage = $this->getStorage(); if (! $storage->isImmutable()) { $_SESSION = $storage->toArray(true); session_write_close(); $storage->fromArray($_SESSION); $storage->markImmutable(); } } /** * Attempt to set the session name * * If the session has already been started, or if the name provided fails * validation, an exception will be raised. * * @param string $name * @return SessionManager * @throws Exception\InvalidArgumentException */ public function setName($name) { if ($this->sessionExists()) { throw new Exception\InvalidArgumentException( 'Cannot set session name after a session has already started'

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
language	"en"
VUFIND_SESSION	"b304aefc1b9196f658fdd9ddd0e16b7c"
ui	"standard"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1790951248.4287 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "b304aefc1b9196f658fdd9ddd0e16b7c" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1433}
SessionHelper	Laminas\Stdlib\ArrayObject {#1759}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"language=en; VUFIND_SESSION=b304aefc1b9196f658fdd9ddd0e16b7c; ui=standard"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.142"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"37390"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2508_08192/Similar """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2508_08192/Similar"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1790951248.4243
REQUEST_TIME	1790951248

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Environment & details:

GET Data empty

POST Data empty

Files empty

Key	Value
language	"en"
VUFIND_SESSION	"b304aefc1b9196f658fdd9ddd0e16b7c"
ui	"standard"

Session

Key	Value
__Laminas	array:2 [`"_REQUEST_ACCESS_TIME" => 1790951248.4287 "_VALID" => array:1 [ "Laminas\Session\Validator\Id" => "b304aefc1b9196f658fdd9ddd0e16b7c" ]`]
SessionState	Laminas\Stdlib\ArrayObject {#938}
FlashMessenger	Laminas\Stdlib\ArrayObject {#1433}
SessionHelper	Laminas\Stdlib\ArrayObject {#1759}

Server/Request Data

Key	Value
REDIRECT_VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
REDIRECT_HTTPS	"on"
REDIRECT_SSL_TLS_SNI	"info.umar.mx"
REDIRECT_STATUS	"200"
VUFIND_LOCAL_DIR	"/usr/local/vufind/local"
HTTPS	"on"
SSL_TLS_SNI	"info.umar.mx"
HTTP_ACCEPT	"/"
HTTP_USER_AGENT	"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
HTTP_COOKIE	"language=en; VUFIND_SESSION=b304aefc1b9196f658fdd9ddd0e16b7c; ui=standard"
HTTP_ACCEPT_ENCODING	"gzip, br, zstd, deflate"
HTTP_HOST	"info.umar.mx"
PATH	"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/snap/bin"
SERVER_SIGNATURE	"<address>Apache/2.4.66 (Ubuntu) Server at info.umar.mx Port 443</address>\n"
SERVER_SOFTWARE	"Apache/2.4.66 (Ubuntu)"
SERVER_NAME	"info.umar.mx"
SERVER_ADDR	"200.23.222.19"
SERVER_PORT	"443"
REMOTE_ADDR	"216.73.217.142"
DOCUMENT_ROOT	"/usr/local/vufind/public"
REQUEST_SCHEME	"https"
CONTEXT_PREFIX	""
CONTEXT_DOCUMENT_ROOT	"/usr/local/vufind/public"
SERVER_ADMIN	"admin@localhost"
SCRIPT_FILENAME	"/usr/local/vufind/public/index.php"
REMOTE_PORT	"37390"
REDIRECT_URL	""" /Record/\n arxiv_https___arxiv_org_abs_2508_08192/Similar """
GATEWAY_INTERFACE	"CGI/1.1"
SERVER_PROTOCOL	"HTTP/1.1"
REQUEST_METHOD	"GET"
QUERY_STRING	""
REQUEST_URI	"/Record/%0Aarxiv_https___arxiv_org_abs_2508_08192/Similar"
SCRIPT_NAME	"/index.php"
PHP_SELF	"/index.php"
REQUEST_TIME_FLOAT	1790951248.4243
REQUEST_TIME	1790951248

Environment Variables empty

Registered Handlers

0. Whoops\Handler\PrettyPageHandler

Similar Items

Similar Items

Environment & details:

Environment & details: