Overview
LLM inference becomes expensive because every user, token, and session adds recurring compute and memory costs. The workshop builds a first-principles framework around three initial pain points—memory consumption, time to first token, and throughput—then adds inter-token latency as a fourth operational metric. It explains that transformer attention dominates inference compute, while the growing key-value state creates substantial memory pressure as context length and concurrency increase. Inference itself has two distinct phases: prefill is compute-bound and determines time to first token, whereas autoregressive decode is largely memory-bandwidth-bound and governs token-generation speed. These mechanics create a practical trade-off among output quality, latency, and concurrent throughput. Model-side techniques such as quantization, grouped-query attention, latent attention, sparse attention, and FlashAttention reduce memory or computation, but their quality effects must be validated for the intended use case. Serving-side techniques—including KV caching, paged attention, continuous batching, prefix caching, and KV-cache quantization—improve hardware utilization and capacity. The recommended deployment strategy is workload-specific: vLLM is presented as the general production default, while SGLang may perform better for repetitive agentic workflows. Ultimately, teams should benchmark their own traffic, fix the most important service constraint first, and adopt only optimizations that solve a demonstrated problem.
Sections
Core Inference Concepts
Terms used to reason about LLM serving performance and capacity.
- LLM inference: Executing a trained model to produce or analyze content, including text, audio, video, medical reports, or tax documents.
- Time to first token (TTFT): The time spent processing the input and producing the first output token; it is primarily determined by the prefill phase.
- Inter-token latency: The time taken by each decode step between successive generated tokens.
- Throughput: The number of tokens or user requests that an inference system can serve per second.
- Arithmetic intensity: The number of floating-point operations performed per byte of data transferred, used to distinguish compute-bound from memory-bound workloads.
- KV cache: Stored key and value representations for previously processed tokens, reused during decoding to avoid redundant computation.
- Quantization: Reducing the numerical precision of model weights or KV state to decrease memory requirements, potentially affecting quality or throughput.
- Paged attention: A memory-management technique that maps logically contiguous KV state onto dynamically allocated physical memory blocks.
- Continuous batching: A scheduling technique that inserts new requests when capacity becomes available rather than waiting for every request in a batch to finish.
- Prefix caching: Reusing previously computed KV state when multiple requests share a common prompt prefix.
Technical Mechanics and Example Data
Specific formulas, measurements, architectures, and serving techniques discussed in the workshop.
- For a 7-billion-parameter model at 16-bit precision, the workshop calculates approximately 14.6 GB of model-weight memory.
- The stated Mistral 7B KV footprint is approximately 131 KB per token, derived from K and V vectors, 128 dimensions, 32 transformer layers, and eight KV heads.
- The example estimates about 0.5 GB of KV state for a 4K context and 2.1 GB for a 16K context per user; 80 users at 4K context would require roughly 42 GB.
- Maximum concurrent users are approximated by dividing GPU memory available after model weights and overhead by the KV-cache size required per user.
- Prefill processes many input tokens in parallel and is compute-bound; decode generates one token at a time and is constrained primarily by memory bandwidth.
- The workshop reports a Hugging Face baseline of about 51 tokens per second, a TTFT value of 54 in the benchmark's unspecified unit, and an inter-token latency of 19 in an unspecified unit on an H100.
- The reported default vLLM configuration includes paged attention, continuous batching, and KV caching; prefix caching and KV-cache quantization can be enabled as additional optimizations.
- FlashAttention divides large attention matrices into tiles processed in faster on-chip memory and maintains limited intermediate state to compute softmax online.
- The presenters corrected their multi-head latent attention estimate during the workshop: the claimed saving relative to multi-head attention was 14x, not 50x or 56x, because the earlier calculation omitted the number of layers.
- Speculative decoding uses a smaller draft model to propose several tokens that a larger model verifies, but one presenter reported poor personal results and preferred EAGLE-style feature prediction.
Key Trade-Offs and Engine Choices
Explicit comparisons among workload priorities, attention architectures, and serving engines.
- Premium chat should favor output quality and low latency, even if that reduces the number of concurrent users served per GPU; asynchronous agent workloads can prioritize quality and throughput while accepting longer latency.
- Multi-head attention retains independent heads and stronger quality, multi-query attention maximizes KV compression with a greater quality risk, and grouped-query attention offers a practical middle ground.
- The workshop found vLLM and SGLang approximately equivalent for its standard API test, but SGLang was reportedly three to four times better for its repetitive agentic branching test.
- Static prefix caching depends on exact prefix matching, whereas radix-tree caching can organize and reuse repeated prompt segments more effectively in agentic workloads.
- A higher-priced GPU can still deliver a lower cost per million tokens if its memory, bandwidth, and throughput support substantially more useful work.
Tools and References Mentioned
Repositories, platforms, engines, datasets, and research references named by the presenters.
- LLM Inference at Scale: The presenters' open-source repository containing the workshop slides, notebooks, exercises, and benchmark report.
- Modal: A hosted compute platform described as a Google Colab alternative and used for prepared GPU notebooks.
- Google Colab: A hosted notebook environment used for the capacity calculator after widget compatibility issues on Modal.
- Hugging Face: The source used to download model weights and the basis of the raw inference baseline.
- vLLM: A production-oriented LLM serving engine offering KV caching, paged attention, continuous batching, prefix caching, and KV-cache quantization.
- SGLang: An inference and programming system highlighted for radix-tree prefix caching and potentially stronger agentic-workload performance.
- TensorRT-LLM: NVIDIA's LLM inference engine, described as optimizing model operations at both software and hardware levels.
- NVIDIA Dynamo: An emerging serving system mentioned in connection with agentic session routing.
- ShareGPT dataset: The question dataset used for the presenters' standard API workload comparison.
- Attention Is All You Need: The foundational paper referenced while introducing transformer attention.