How do you optimize prompts for cost and latency?

In this question, we will learn how to optimize prompts for cost and latency in practical interviewer-style problems. We will explain straightforward tactics you can use, show a worked example with numbers, and discuss when those tactics backfire.

We will cover the following:

• The short answer
• The intuition
• How it actually works with a worked example and a comparison table
• Orchestration and engineering patterns
• Tradeoffs and failure modes
• Questions the interviewer might ask

Short answer: reduce token usage, choose the smallest capable model, and reduce round trips through caching, batching, streaming, or on-device work. Keep prompts compact and structured, move static context out of the query, and measure quality to find the smallest acceptable configuration.

The intuition (an analogy that makes it click)

Think of each API call as sending a parcel. Larger parcels cost more to ship and take longer to arrive. If you can compress the contents, use a slower but cheaper route for nonurgent parcels, or avoid sending the same parcel twice by keeping a local copy, you will save money and time. We follow the same ideas for prompts: compress, choose the right carrier, and reduce trips.

How it actually works

There are three main levers we control: prompt length in tokens, model size and latency, and orchestration that affects round trips. Tokens are the unit of billing and transmission. If $N$ is the number of tokens in a request and $p_{1000}$ is the price per 1000 tokens, the cost per request is

Latency is affected by model inference time and network round trips. Choose a model with lower latency or run fewer calls. Here is a worked example comparing three prompt strategies for a single user request.

We used $p_1000=0.02$ as an example price per 1000 tokens. You can see reducing tokens cuts cost linearly and often reduces latency because less data crosses the network and the model spends less time generating.

Practical tactics and a step-by-step pattern

1. Move static context outside the request. Store long policy text, examples, or documents in a separate store and retrieve only the minimal snippet needed. This lowers $N$ per call.
2. Compress examples. Turn several long few-shot examples into a single short templated example or use one representative example plus constraints.
3. Use the smallest model that meets accuracy targets. For many tasks, a medium-size model produces acceptable outputs with much lower latency. Test with real prompts.
4. Cache and reuse responses where possible. Immutable prompts or deterministic outputs can be cached per key.
5. Batch similar requests. Putting multiple user asks into a single API call can amortize model overhead.
6. Stream or early-stop. If you only need the start of a long completion, stream and stop once you have enough.

When you test, measure both cost per request and end-to-end latency. Track quality metrics like accuracy or human ratings alongside cost.

Orchestration patterns

• Retrieval augmented prompting: replace long context by an index lookup that returns a small, relevant snippet. This lowers tokens per call while preserving context.
• Distillation or smaller fine-tuned model: distill frequent patterns into a smaller model you host for low-latency inference. This requires upfront engineering but pays off at scale.
• Hybrid pipelines: use a cheap model to filter and a larger model only for hard cases. That reduces average cost and latency.

Tradeoffs and failure modes

Shorter prompts and smaller models reduce cost and latency but can harm correctness, creativity, or robustness. Moving context to retrieval introduces retrieval errors. Caching can serve stale information.

Questions the interviewer might ask

Some follow-up questions you might get:

How do you measure the impact of prompt changes on quality? Measure with a held-out test set and metrics that reflect user needs, such as accuracy, F1, or human rating. Run A B tests before and after changes.

When should you pick a smaller model versus changing the prompt? Pick a smaller model when it meets your accuracy and latency targets on representative examples. Otherwise, optimize prompts first to reduce token use, then evaluate model swaps with controlled tests.

How do you safely cache outputs that depend on user context? Cache only deterministic or idempotent responses keyed by a stable input fingerprint. Use short TTLs for time-sensitive data and include versioning for prompt templates.

What is retrieval augmented generation and why does it help with cost? It stores long documents externally and returns a small relevant passage, replacing many tokens of context with a short retrieval result. That cuts tokens in the prompt and keeps context precise.

How do you choose between batching and streaming? Batching reduces per-call overhead but adds queuing delay. Use batching for high throughput with tolerant latency. Use streaming when the user expects immediate partial output.

How do you handle prompts that require many few-shot examples? Move examples to a retrieval step or create a compact pattern that summarizes multiple cases. Consider fine-tuning or distilled models if examples are essential and repeated.

Some things to note:

• Always baseline cost and latency with real traffic patterns. Synthetic tests can mislead.
• Small changes in token count add up at scale. Multiply per-request savings by expected volume.

What the interviewer is really testing

They want to see that you can balance competing goals: cost, latency, and output quality. Show that you know practical levers like token reduction, model selection, and orchestration, and that you validate changes with metrics and safety checks. Demonstrate an engineer mindset: measure, iterate, and guard against silent regressions.