The 4 LLM Inference Patterns — and Why Deployment Feels Different
🤔 Curiosity: The Question
Over the last year, I’ve deployed agents into real workflows and hit the same surprise: LLM inference doesn’t behave like classic ML inference. The output is generated, not fixed. That means the cost and latency curve bends in ways you don’t see with “normal” models.
The LinkedIn post on “4 patterns of LLM inference” made the framing click. It’s not just prefill and decode—it’s how those stages combine into four distinct runtime profiles that should change how we deploy.
📚 Retrieve: The Knowledge
Key terms worth re‑anchoring
- Prefill: the model processes the full prompt and builds the KV cache. This is typically compute‑bound and fast per token.
- Decode: the model generates tokens one‑by‑one using the KV cache. This is memory‑bound and grows with output length.
- KV Cache: cached key/value tensors from attention layers that prevent recomputation across steps—critical for fast decoding.
Useful references that explain this split clearly:
- NVIDIA on inference optimization & KV cache: https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
- vLLM anatomy of prefill vs decode: https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html
The 4 patterns
| Pattern | Input | Output | Primary bottleneck | Typical risk |
|---|---|---|---|---|
| SISO | Short | Short | none (fast) | Under‑utilized GPU |
| LISO | Long | Short | prefill | prompt cost dominates |
| SILO | Short | Long | decode | token generation latency |
| LILO | Long | Long | both | heavy infra & memory pressure |
Pattern‑specific optimization playbook
| Pattern | Optimization focus | Serving tactic |
|---|---|---|
| SISO | High concurrency, batching | Aggressive dynamic batching, smaller models |
| LISO | Prefill throughput, KV cache reuse | Larger VRAM, prompt‑cache, parallel prefill |
| SILO | Decode speed, memory bandwidth | KV cache efficiency, speculative decoding |
| LILO | Both stages + memory pressure | Tensor/pipe parallelism, multi‑node, paging KV |
Minimal mental model
flowchart LR
A[Prompt] --> B[Prefill / KV cache]
B --> C[Decode loop]
C --> D[Output]
The longer the output, the more time is spent in decode. The longer the prompt, the more cost you pay in prefill. Deployment strategy should follow that split.
💡 Innovation: The Insight
Why this matters for AI × Games
Game pipelines are a mix of short queries (SISO) and long chain‑of‑thought tasks (SILO/LILO). If we route everything to one model, we overpay and slow the system down.
What I’d do in production
1) Route by inference pattern
- Short NPC tasks → smaller models (SISO)
- Long narrative generation → larger models (SILO/LILO)
2) Optimize for decode‑heavy workloads
- When output length dominates, KV cache + memory bandwidth becomes the priority.
3) Match infra to pattern
- LISO/LILO workloads need higher VRAM and parallelism; SISO should focus on concurrency.
New Questions This Raises
- Can we classify inference pattern at runtime and auto‑route models in <1ms?
- What’s the simplest pattern‑aware router that still saves real money?
- How should we benchmark agent productivity per dollar across these patterns?
References
- LinkedIn post: https://www.linkedin.com/posts/arazvant_the-4-patterns-of-llm-inference-deploying-activity-7432739901659299843-X0qG
- NVIDIA: https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
- vLLM blog: https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html