How do you serve LLMs in production?

In this question, we will learn how to serve LLMs in production so they meet latency, throughput, and reliability targets while staying cost effective. We will walk through runtime patterns, a concrete sizing example, observability needs, and common failure modes.

We will cover the following:

• The intuition
• How it actually works
• Serving patterns
• Observability and autoscaling
• Tradeoffs and failure modes
• Questions the interviewer might ask

Direct answer: Serve LLMs by choosing the right runtime pattern (single-replica low-latency, batched GPU throughput, or model-sharded across accelerators), then add batching, caching, autoscaling, and strong observability. Measure p50 and p99 latency, plan capacity from your throughput target, and iterate on optimization knobs like quantization, concurrency, and batch size.

The intuition (an analogy that makes it click)

Think of serving like a restaurant kitchen. If a single diner needs a dish quickly we cook it to order. If a tour bus of 40 guests arrives we batch similar orders to cook more efficiently. If the dish needs many chefs to lift a heavy vat we coordinate across stations. Serving LLMs is the same: choose per-request speed, batched throughput, or cross-machine sharding based on demand and resource limits.

How it actually works (the real mechanics, with a worked example)

Core knobs: batch size, concurrency, hardware choice, quantization, and cache hit rate. Two useful formulas are concurrency need and throughput.

Concurrency needed for stable latency is approximated as inline product of request rate and average service latency. In symbols, required concurrency roughly equals $\text{qps} \times \text{latency}$. Throughput per second depends on batch size and per-batch latency:

$\text{throughput} = \frac{\text{batch\_size} \times \text{concurrency}}{\text{latency\_per\_batch}}$

Worked example. Suppose you must serve a 13B model at target $50$ requests per second and user latency budget of $200$ milliseconds. If we use batch size $4$, then each batch returns $4$ responses. Batches per second required: $50/4=12.5$ batches per second. That implies per-batch latency must be at most $80$ milliseconds to meet the user latency budget.

If a single GPU inference call for batch $4$ takes $120$ milliseconds, we either reduce model latency (quantize, faster runtime) or increase concurrency and parallelism. Concurrency estimate using latency per request $L$ seconds and qps $R$ is:

$\text{concurrency} \approx R \times L$

For $L=0.12$ and $R=50$ we get $6$ concurrent requests in flight, which informs how many replicas or async threads we need.

Compare common options in a simple table:

Serving patterns

1. Single-replica low-latency: small models can run on a single GPU or CPU for sub-100ms latencies. No batching makes tail latency predictable.

2. Batched GPU serving: accumulate requests up to a batch size or short time window, then run a single forward pass. This improves GPU utilization and reduces cost per token, at the cost of added batching latency.

3. Sharded or pipeline-parallel: split weights across multiple accelerators for very large models. This adds network and synchronization overhead, so engineering is more complex.

4. Hybrid edge-cloud pattern: run a small local model for initial replies and escalate complex prompts to a larger cloud model.

Observability and autoscaling

We need metrics: p50, p95, p99 latency, request qps, GPU utilization, memory pressure, queue depth, and cache hit rate. Alerts should focus on p99 latency and OOM events.

Autoscaling rules often use target utilization and queue depth. A simple reactive scaling policy might aim to keep GPU utilization at $\sim70\%$ while ensuring the request queue length stays below a threshold. Horizontal autoscaling for stateless replications is simplest. For sharded models we need careful choreography to scale without downtime.

A short checklist before shipping:

• Define SLOs for p50 and p99 latency and error budget.
• Measure cold start time when loading models into GPU memory.
• Implement a warm-pool of replicas for predictable latency.
• Add request-level tracing to find where tails come from.

Tradeoffs and failure modes

Common failure modes: out-of-memory on GPU, queue amplification where batching increases tail latency, noisy neighbors sharing GPU, and stale autoscaler policies that overreact and thrash replicas.

Other tradeoffs include model quality versus cost when using quantization and reduced precision. Operational complexity rises with sharding and custom kernels. Balance those with business needs.

Questions the interviewer might ask

Some follow-up questions you might get:

How do you reduce p99 latency without increasing cost a lot? Use a warmed pool of replicas, small batch sizes for latency-sensitive endpoints, and a fast lightweight cache for common prompts. Also consider model distillation or a smaller on-path model for first-pass answers.

When would you choose model sharding over more replicas? When the model cannot fit on a single accelerator memory or when single-GPU compute is insufficient for throughput. Sharding adds complexity, so prefer it only for very large weights or extreme throughput demands.

How do you handle hallucinations and safety in production? Add response filters, rerankers, and a human-in-the-loop review pipeline for risky outputs. Log examples and build automated checks to detect unsafe patterns.

What metrics do you monitor for capacity planning? Monitor qps, p50/p95/p99 latency, GPU memory, utilization, queue depth, cache hit rate, and model load times. Use these metrics to size replicas and adjust batch sizes.

How do you do A B testing with LLMs? Route a percentage of traffic to new models via stable orchestration, collect both quality metrics and latency/cost signals, and use blind evaluation where possible.

Some things to note:

• Tail latency usually dominates user experience more than median latency.
• Caching common prompts reduces load significantly if usage has repetition.
• Warm starts and warm pools reduce variance from cold model loads.

What the interviewer is really testing

They want to see systems thinking applied to ML models: how you translate SLOs into capacity and design choices, how you trade latency for cost, and how you instrument and react to production signals. They also expect practical knowledge of batching, quantization, autoscaling, and observability to avoid common pitfalls.