How do you optimize LLM inference costs in production?

In this question, we will learn how to optimize LLM inference costs in production by combining model selection, serving patterns, and measurement. You will learn practical levers that reduce spend while keeping latency and quality within SLOs.

We will cover the following:

• The intuition
• How it actually works
• Efficient serving patterns
• Practical metrics and SLOs
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Answer: Optimize costs by choosing the right model and runtime, route and cache requests to cheaper paths, and use batching and quantization to improve throughput. Measure cost per useful response and tune for the SLOs that matter, like tail latency and accuracy.

The intuition (an analogy that makes it click)

Think of LLM inference like a kitchen with multiple cooks. A slow gourmet chef makes exceptional dishes but is costly per plate. A competent line cook is much cheaper and faster for simple orders. For busy times we batch orders to use the oven efficiently. For repeat orders we keep a prepared dish in the fridge. We want to match the customer request to the right chef, batch where it helps, and store popular items.

This maps to choosing model size, routing simple queries to smaller models, batching tokens to fill GPU throughput, and caching repeated outputs.

How it actually works (the real mechanics, with one concrete worked example)

We use a few simple formulas to reason about cost. Let $q$ be queries per second, $T$ be average tokens generated per query, and $c_t$ be cost per generated token for a given model and runtime. Then expected cost per query is

and cost per second is

Batching improves throughput by increasing effective tokens per GPU run at near constant overhead. If batching increases throughput by factor $b$ for the same GPU cost, then effective $c_t$ drops roughly by $1/b$.

Worked example. Suppose:

Cost per query:

• Large model $C_{large} = 0.0006 \times 150 = 0.09$ USD
• Small model $C_{small} = 0.00005 \times 150 = 0.0075$ USD

If 70% of queries are simple and can be answered by the small model via routing, overall average cost per query becomes

That is a 64 percent reduction versus always using the large model. Batching these requests can further reduce $c_t$ by increasing throughput on GPU time.

Efficient serving patterns

Routing and model choice

• Serve a spectrum: tiny specialized models for classification, small chat models for short replies, and large models for complex generation. Use a cheap classifier to route.
• Use confidence thresholds or a lightweight reranker to decide when to escalate to a bigger model.

Caching and memoization

• Cache exact prompts and responses and also cache partial computations such as key value cache in streaming setups when applicable.
• Time to live matters. Cache hits reduce cost to near zero per hit but increase risk of staleness.

Batching and token-level batching

• Group multiple short requests into one GPU batch to amortize kernel launch and memory overhead.
• Use dynamic batching or token-level batching where you fill the token budget across requests. This reduces $c_t$ while slightly increasing latency for queued requests.

Quantization and distillation

• INT8 or 4-bit quantization can reduce memory and increase throughput. Expect some drop in task accuracy depending on model and application.
• Distilled models give larger accuracy tradeoffs but often large cost reductions.

Efficient runtimes and hardware

• Use inference-optimized runtimes such as TensorRT, ONNX Runtime, or Triton. They improve $c_t$ and reduce latency.
• Match GPU type to batch size and latency SLO. For very low latency, a high clock GPU with more memory can be better than many smaller GPUs.

Practical metrics and SLOs

Measure the things you will optimize. Useful metrics:

• Cost per useful response: include only responses that meet quality SLO.
• Tail latency, for example $p_{99}$ latency.
• Cache hit rate and routing accuracy.

Set objectives like: average cost per request under $0.02 with$p_99$ latency under 800 ms and accuracy above 95 percent for critical tasks.

Tradeoffs and failure modes

Every optimization has costs. Quantization and distillation reduce accuracy. Batching increases average latency and can worsen tail latency if not carefully managed. Caching can return stale info. Routing adds complexity and the risk of misclassification which may silently lower quality.

Questions the interviewer might ask

Some follow-up questions you might get:

How would you measure the benefit of adding a cache layer? Measure cache hit rate, time to first byte reduction, and the delta in cost per second. Use A B testing to compare quality and latency impact.

What routing criteria would you use to send traffic to a small model? Use lightweight heuristics like prompt length, classification confidence, or a fast semantic similarity model. Prioritize precision for escalations.

How do you avoid increasing p99 latency when batching? Use adaptive batching with a max wait time, prioritizing interactive requests. Reserve a small pool of low-latency workers for tail requests.

When should you prefer quantization over distillation? Quantization often keeps the same model behavior with lower compute. Use it first when small accuracy loss is acceptable. Distillation is better when you need larger cost reductions and can accept more behavior change.

How do you handle models with different tokenization or output distributions? Canonicalize inputs and measure end-to-end quality. Build tests that validate outputs for each path in the routing policy.

Some things to note:

• Always measure end-to-end user quality, not only raw model perplexity.
• Automate rollback and monitoring when deploying optimizations that change model behavior.

What the interviewer is really testing

They want to see that you can balance cost, latency, and accuracy and pick practical engineering levers rather than only theoretical ones. They also want to know you can measure impact, handle privacy and safety implications, and design a deployment that can incrementally roll back if quality drops. Show both cost math and operational controls.