BlockLock: Reproducible LLM Inference with Fixed Batch Geometry
Paper · https://arxiv.org/abs/2611.06977
arXiv:2611.06977v1 Announce Type: new Abstract: Reproducible LLM inference requires that a request produces the same output regardless of the other requests in the batch. Existing solutions pay for this in different ways: batch-invariant kernels constrain every operator, and isolating prefill with fixed-shape decode ({our CoRun-style baseline}) gives up shared prefill compute and pads decode. We present BlockLock, which keeps shared batched prefill and pins only the numerical path of the protected request: it fixes the geometry that the request sees, with {two modes: BlockLock-Rows uses fixed-width rows and retains batched decode of all requests; BlockLock-Pack uses terminal-anchored packing with an isolated dummy segment, then extracts the protected request's KV state for target-only decode. Under a stated per-deployment contract, both modes preserve the target's output bitwise when its companions change.} Across five {dense} models (Qwen3-8B/14B, Llama-3.1-8B-Instruct, and Gemma-3-4B/12B-IT) and three single-GPU deployments (NVIDIA A6000, A100, and H100), we evaluate our approach on both tinyBenchmarks and BFCL{; we also include initial tensor-parallel experiments on two NVIDIA A6000 GPUs. Both modes preserve target sequences and Top20-logit bits in the reported tests.} At equal delivered work, {BlockLock-Rows} reduces latency by 53.5--55.0\% and improves throughput by 115.2--122.2\% relative to vLLM Batch-Invariant; compared with batch-invariant kernels, it needs no custom kernel implementations and constrains only the protected requests.
First seen by Oriel on .
AI summary
BlockLock presents a fixed-batch-geometry approach to bitwise-reproducible LLM inference that preserves shared prefill while limiting numerical-path constraints to protected requests.
- BlockLock-Rows uses fixed-width rows and retains batched decode for all requests.
- BlockLock-Pack uses terminal-anchored packing, an isolated dummy segment, and extracted KV state for target-only decode.
- Across five dense models and NVIDIA A6000, A100, and H100 deployments, both modes preserved target sequences and Top20-logit bits in reported tests.
- At equal delivered work, BlockLock-Rows reduced latency by 53.5--55.0% and improved throughput by 115.2--122.2% versus vLLM Batch-Invariant.
- It requires no custom kernel implementations and constrains only protected requests, under a stated per-deployment contract.
#llm inference #batching #reproducibility
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.06977v1 | https://arxiv.org/abs/2611.06977 Read discussion |