Overview

The hardest model-serving failures may look perfectly healthy: no crash, warning, or obvious loss of confidence, yet the generated output is wrong. AI21 encountered two such failures while training and serving Jamba, a hybrid architecture combining attention and Mamba state-space layers. The first appeared as rare, load-dependent gibberish confined to the VLM engine. Engineers accelerated reproduction by constraining GPU memory, fixing sampling at temperature zero, and comparing VLM token probabilities against a Hugging Face Transformers baseline. After CUDA inspection and memory sanitization found nothing, they propagated request IDs into the forward pass and discovered that the scheduler sometimes sent a new request through decode before prefill, causing Mamba to read stale state. A second failure produced periodic log-probability spikes between rollout and FSTP computations despite identical weights and inputs. Increasing rollouts made the failure occur sooner, while reducing GPU memory unexpectedly hid it. The underlying cache offset used an unsigned 32-bit index that silently wrapped beyond roughly four billion positions; replacing it with size_t prevented the overflow. Together, the cases show that stateful inference systems can corrupt results without failing loudly. Effective diagnosis therefore depends on differential probability analysis, deterministic stress reproduction, experiments that change the failure’s shape, and end-to-end request identity.

Sections

Technical Details

Implementation and diagnostic details behind the two failures.

The request lifecycle was described as prompt tokenization, forward-pass prefill, iterative decode, and detokenization back to text.
The reproduction reduced the engine's GPU-memory-utilization setting from 0.9 to 0.2, applied concurrent load, and used temperature-zero sampling to stabilize the failing output.
The differential checker collected VLM-generated responses and log probabilities, replayed each prompt-plus-generation through a Hugging Face Transformers prefill forward pass, applied softmax to the logits, and compared token-distribution divergence.
NVIDIA Compute Sanitizer was used to check for out-of-bounds access and other memory errors, but it did not expose the scheduling defect.
The first fix required the scheduler to classify requests with no previously computed tokens as prefill rather than decode or chunked execution.
The second fix changed the Mamba state-cache index type from unsigned 32-bit to size_t, avoiding wraparound at the observed scale.

Key Definitions

Terms needed to understand the request lifecycle and diagnostic method.

Prefill: the initial forward computation over the prompt tokens that establishes the state required for subsequent generation.
Decode: the iterative generation phase that reads previously established model state to produce additional tokens.
Log probabilities: token-level probability information used here to compare two inference implementations and identify where their output distributions diverge.
Mamba state cache: persistent per-request state read during Mamba decoding; if it is stale or indexed incorrectly, later computation can remain numerically valid while producing corrupted output.
GPU memory utilization: an engine setting controlling how much GPU memory is allocated across weights, activations, the KV cache, and related runtime state.
Failure shape: the observable timing, frequency, location, or form of a failure as runtime parameters are changed.

Risks and Pitfalls

Failure modes that can conceal corruption or mislead an investigation.

A stateful inference system may return confident gibberish without crashing or emitting a warning.
Decode-before-prefill can make a new Mamba request consume state left by previous requests.
A 32-bit cache index can wrap silently under sufficiently large accumulated offsets.
Reducing memory pressure can hide an overflow rather than prove that the bug is fixed.
Anonymous tensors at the kernel boundary make it difficult to connect corruption to the responsible request and scheduler decision.
Switching entirely to the prefill kernel can suppress the symptom and create premature confidence that the decode kernel itself is defective.

Lessons Learned

General debugging practices derived from the investigations.

Build a reusable log-probability comparison script against an independent inference implementation.
Create a short deterministic feedback loop by constraining resources, increasing workload, and removing sampling randomness.
Prefer experimental knobs that change when or where a failure occurs because they reveal causal relationships inside the engine.
Validate scheduler lifecycle invariants as carefully as kernel mathematics.
Thread logical request identity through batching and low-level execution so a corrupt output can be traced to its exact runtime path.
Inspect complex framework internals directly when high-level diagnostics cannot explain silent corruption.

Overview

Sections

Technical Details

Implementation and diagnostic details behind the two failures.

The request lifecycle was described as prompt tokenization, forward-pass prefill, iterative decode, and detokenization back to text.
The reproduction reduced the engine's GPU-memory-utilization setting from 0.9 to 0.2, applied concurrent load, and used temperature-zero sampling to stabilize the failing output.
The differential checker collected VLM-generated responses and log probabilities, replayed each prompt-plus-generation through a Hugging Face Transformers prefill forward pass, applied softmax to the logits, and compared token-distribution divergence.
NVIDIA Compute Sanitizer was used to check for out-of-bounds access and other memory errors, but it did not expose the scheduling defect.
The first fix required the scheduler to classify requests with no previously computed tokens as prefill rather than decode or chunked execution.
The second fix changed the Mamba state-cache index type from unsigned 32-bit to size_t, avoiding wraparound at the observed scale.

Key Definitions

Terms needed to understand the request lifecycle and diagnostic method.

Prefill: the initial forward computation over the prompt tokens that establishes the state required for subsequent generation.
Decode: the iterative generation phase that reads previously established model state to produce additional tokens.
Log probabilities: token-level probability information used here to compare two inference implementations and identify where their output distributions diverge.
Mamba state cache: persistent per-request state read during Mamba decoding; if it is stale or indexed incorrectly, later computation can remain numerically valid while producing corrupted output.
GPU memory utilization: an engine setting controlling how much GPU memory is allocated across weights, activations, the KV cache, and related runtime state.
Failure shape: the observable timing, frequency, location, or form of a failure as runtime parameters are changed.

Risks and Pitfalls

Failure modes that can conceal corruption or mislead an investigation.

A stateful inference system may return confident gibberish without crashing or emitting a warning.
Decode-before-prefill can make a new Mamba request consume state left by previous requests.
A 32-bit cache index can wrap silently under sufficiently large accumulated offsets.
Reducing memory pressure can hide an overflow rather than prove that the bug is fixed.
Anonymous tensors at the kernel boundary make it difficult to connect corruption to the responsible request and scheduler decision.
Switching entirely to the prefill kernel can suppress the symptom and create premature confidence that the decode kernel itself is defective.

Lessons Learned

General debugging practices derived from the investigations.

Build a reusable log-probability comparison script against an independent inference implementation.
Create a short deterministic feedback loop by constraining resources, increasing workload, and removing sampling randomness.
Prefer experimental knobs that change when or where a failure occurs because they reveal causal relationships inside the engine.
Validate scheduler lifecycle invariants as carefully as kernel mathematics.
Thread logical request identity through batching and low-level execution so a corrupt output can be traced to its exact runtime path.
Inspect complex framework internals directly when high-level diagnostics cannot explain silent corruption.

Two Bugs That Hid in Plain Sight: A vLLM Debugging Detective Story - Asaf Gardin & Yuval Belfer

Summary

Key ideas

Silent corruption is an engineering failure, not necessarily a model-quality failure

The first bug was rare, load-dependent, and engine-specific

Memory pressure created a faster deterministic reproduction

Debugging Sequence for Silent Inference Corruption

Ask about this source

Read the full digest

Overview

Sections

Technical Details

Key Definitions

Risks and Pitfalls

Lessons Learned

Overview

Sections

Technical Details

Key Definitions

Risks and Pitfalls

Lessons Learned