What is the difference between prompt chaining and an orchestrator-worker architecture, and when do you use each?

Prompt chaining is a static pipeline: the steps and their order are determined by the developer at design time, and each step executes exactly once in sequence. An orchestrator-worker architecture is dynamic: the orchestrator LLM decides which sub-tasks to create and can adjust that plan based on intermediate results. Chaining is appropriate when the task structure is predictable, the steps are few, and determinism matters because it keeps the system testable and cost-bounded. Orchestrator-worker is appropriate when the required sub-tasks are not knowable until the task is partially analyzed, or when sub-tasks span different tools and domains that benefit from specialization. In practice I reach for a chain first and promote to orchestrator only when I find myself hardcoding branching logic that genuinely depends on LLM output I cannot predict ahead of time.

How do you prevent an evaluator loop from running forever, and what do you do when the maximum iteration count is hit?

Set three independent termination conditions: a maximum iteration count (typically three to five), a maximum total token budget for the loop, and a maximum wall-clock time. The iteration count is the most important because it bounds cost directly. When the count is hit without the output passing, do not raise an exception. Return the best-scoring output seen during the loop along with a flag indicating it did not clear the threshold, and log the full iteration history. The flag lets the application layer decide whether to surface the output to the user, route to a human reviewer, or silently degrade. You want graceful degradation rather than crashes, because loops often fail due to a threshold that is miscalibrated rather than because the output is genuinely unusable.

When would you use parallelization with a voting ensemble instead of a single model call, and what are the cost tradeoffs?

Voting is useful when multiple independent attempts converge on a correct answer more reliably than a single attempt, particularly for factual question answering and classification. At temperature above zero the model samples from a distribution, so running N calls gives you N draws from that distribution. Majority voting selects the mode, which is more reliable than a single sample when the correct answer is probable but not certain. The cost is N times the per-call cost plus a synthesis or comparison step. For tasks where correctness matters more than speed (medical, legal, or financial fact lookups) this is usually worth it. For tasks where variance is already low (structured extraction from clean data, summarization of short text), voting adds cost without improving quality measurably. I benchmark both before committing to the pattern in production.

Describe the difference between reflection and an evaluator-optimizer loop. When does each one fail?

Reflection is self-critique: the same model re-reads its own output and revises it, often in a second call with a checklist in the prompt. The evaluator-optimizer loop uses a separate evaluator call that can run different logic, a different model, or even an external tool (such as code execution or schema validation) to produce the critique. Reflection is cheaper and works well for structural improvements: formatting, completeness, tone, and missing sections. It fails on factual errors because the same bias that produced a hallucination also reads it as plausible. The evaluator-optimizer loop can catch factual errors if the evaluator is grounded (comparing output against retrieved source chunks or running generated code against test cases). The evaluator loop fails when the evaluator itself is miscalibrated, or when the generator and evaluator share the same systematic bias, which happens when you use the same model family for both roles.

A colleague proposes solving every new LLM feature with an agent loop. How do you push back constructively?

The question to ask is whether the task structure is actually unknown at request time. If you can enumerate the steps before the request arrives, a prompt chain is simpler, cheaper, and easier to test than an agent loop. Agent loops introduce non-determinism: the same input can produce different execution paths, which makes regression testing hard and cost forecasting harder. They also introduce failure modes that pipelines do not have: the model might take an unnecessary detour, call the wrong tool, or get stuck cycling. The cases where agent loops genuinely earn their cost are tasks requiring true open-ended exploration, tasks where sub-task generation requires reasoning over intermediate results, and tasks where the number of required steps is unbounded and depends on environment state. Everything else should start as a chain or static pipeline. I frame this as a starting point rather than a ceiling: begin with the simplest structure, add the agent loop only when you hit a concrete problem the simpler structure cannot solve.

How would you design a production LLM system that combines routing, RAG, and tool use, with appropriate fallback handling at each stage?

I would start with the router as an independently deployable component that classifies incoming requests and sends them to one of several pipeline branches. Each branch has its own system prompt optimized for that request type. Branches that need factual grounding use RAG, with retrieval happening before the LLM call and a fallback to a templated response when retrieval returns nothing. Branches that need real-time data use tool calls with validated parameter schemas and structured error messages on failure, with a one-retry policy and a prompt-only fallback if the retry also fails. I would instrument every branch transition, routing decision, and tool call result with structured logs so I can trace any user session end to end and identify exactly where unexpected behavior originated. The fallback rate for each branch is a key operational metric: a spike tells me a request type has shifted outside the training distribution of the router, which is a signal to update the routing logic rather than tune the downstream branch.

ai design patterns

Part of the AI system design curriculum

AI Design Patterns

A catalog of recurring architectural patterns for LLM systems, with tradeoffs, failure modes, and guidance on when to combine or avoid each.

19 min read.Last reviewed: June 2026 | Content Verified

A hand-drawn diagram showing four labeled small pattern sketches: chaining, routing, orchestrator and workers, and an evaluator loop.

Software engineers reached for patterns long before they had names for them. Once you could name the structure, the conversation became productive: instead of debating implementation details you could say "this is a factory" or "we need an observer here" and everyone knew what you meant. LLM systems are accumulating the same vocabulary. A handful of structures appear again and again across production deployments of different sizes and industries, each solving a specific class of coordination problem between language models, tools, and data sources. This chapter catalogs the core eight, explains the mechanics and tradeoffs, and shows how they fit together into real architectures.

TL;DR

Prompt chaining links LLM calls sequentially where each stage's output becomes the next stage's input; it fits tasks that decompose into transformations, but latency is additive and error compounds across steps.
Routing uses a classifier to dispatch requests to specialized sub-pipelines; it separates concerns cleanly but fails silently if the router misclassifies, so log every routing decision with confidence scores.
Parallelization fans out independent sub-tasks, executes them concurrently, and aggregates results; it reduces latency for independent work but cost scales linearly with fan-out width.
The orchestrator-and-workers pattern uses a coordinator LLM to decompose tasks dynamically and dispatch sub-tasks to isolated workers; it handles open-ended complexity but makes debugging harder because failures can occur in decomposition, any worker, or synthesis.
Evaluator loops generate a draft, score it with a secondary model or validation step, and regenerate using critique until output passes or max iterations is reached; they improve quality at 2x to 5x inference cost.

Key tradeoffs: Chaining adds latency but makes each stage independently testable; parallelization cuts latency but multiplies cost; evaluator loops improve quality but require hard iteration caps to prevent unbounded expense.

The Core Patterns

Each pattern below names a recurring structure in LLM systems: the problem it solves, how it works, when it fits, and what breaks it.

Prompt Chaining

The simplest multi-step pattern: the output of one LLM call becomes the input to the next. A prompt chain is a directed pipeline where each stage transforms the content produced by the stage before it.

Chains work well when a task decomposes into a sequence of transformations where each step needs a verified, clean result before proceeding. Extracting structured data from a document, then classifying each extracted field, then generating a formatted report from the classified fields is a natural chain. Each step is independently testable: run the pipeline, inspect each stage's output, and the failing step is immediately visible.

The costs are additive latency (each link adds one LLM round-trip) and error compounding (noise introduced in step one propagates through every downstream step). Latency is bounded because the number of steps is fixed at design time. Error compounding is controlled by using structured outputs at each stage boundary: each step receives a well-typed object rather than free text, and validation failures are loud rather than silent.

The failure mode to watch is an informal handoff. Step one returns prose and step two parses it with a fragile regex. When the prose format shifts slightly, the whole chain breaks. Define explicit output schemas between steps using JSON or a validation library, and validate before passing downstream.

Routing

Routing separates dispatch from execution. A classifier examines the incoming request and selects which sub-pipeline processes it. The router adds a small overhead (one classification call) and gives you clean separation between request types that would otherwise require one monolithic prompt to handle inconsistently.

Routing fits when the system handles genuinely different request types: a product question needs RAG against a catalog, a code question needs a sandboxed interpreter, and a billing query needs a constrained handler with access to account state. Without routing, all three behaviors are crammed into one system prompt and optimized for none.

The danger is a router that fails silently. A misclassified request follows the wrong pipeline, receives a plausible-looking but wrong answer, and the failure is invisible. Build a fallback route (a conservative general handler) and log every routing decision with its confidence score. If routing confidence drops below a threshold, surface the request for human review rather than proceeding on a low-confidence classification.

Left: a four-step prompt chain where each LLM call transforms output from the previous step. Right: a router that classifies input and dispatches to one of three specialized pipelines.

Parallelization and Fan-out Fan-in

Some tasks are naturally independent. If you need to analyze five documents, run three searches, or collect opinions from several model configurations, there is no reason to serialize them. Parallelization fans out: dispatch all sub-tasks at the same time and collect results as they complete. The fan-in step then aggregates the results in a final synthesis.

The fan-in step carries most of the complexity. You need to handle partial failures (what if two of five workers return errors?), ordering (some aggregators care about rank, others do not), and synthesis quality (a model asked to reconcile five contradictory summaries can hallucinate a consensus that does not exist). Design the fan-in prompt explicitly for the aggregation task; do not assume the model will naturally handle noisy or contradictory inputs.

One strong application of parallelization is voting: run the same prompt N times with temperature above zero, then pick the most common answer or pass all N responses to a separate judge. This reduces variance without changing the underlying model and is especially useful for factual question answering where correct answers converge but hallucinations diverge.

Cost scales linearly with fan-out width. If each call costs $0.01 and you fan out to ten workers, each user request costs$ 0.10 plus the fan-in call. Budget for this before shipping.

The Orchestrator and Workers

The orchestrator pattern extends fan-out into dynamic territory. An orchestrator LLM reads the task, decides what sub-tasks are needed, dispatches workers to handle them, and synthesizes the results. Unlike a static pipeline, the orchestrator can decide mid-execution to add sub-tasks, change direction, or short-circuit when early results are sufficient.

This is the pattern behind most production multi-agent systems. The orchestrator does not need to be a full autonomous agent loop: it can be a single LLM call that produces a task plan, which a deterministic dispatcher then executes by routing each sub-task to the right worker. Workers are isolated: each worker handles exactly one sub-task and knows nothing about the others. Isolation makes debugging tractable because you can replay any individual worker call without running the full pipeline.

An orchestrator delegates three parallel sub-tasks to specialist workers, then a synthesizer merges their outputs into a single final answer.

The tradeoffs are higher coordination overhead and harder end-to-end debugging. When the final output is wrong, the failure could be in the orchestrator's decomposition, in any worker, or in the synthesis step. Instrument every handoff with structured logs and keep intermediate results observable.

The Evaluator and Optimizer Loop

Quality-critical outputs often benefit from an explicit evaluation step. Generate a draft, pass it to an evaluator (a second LLM call with a scoring prompt), receive a score and critique, and if the score falls below a threshold, regenerate using the critique as additional context. Repeat until the output passes or a maximum iteration count is reached.

This is not just cosmetic cleanup. For code generation, the evaluator can actually run the code and pass back the stack trace. For structured data extraction, the evaluator can validate the output against a schema and flag each invalid field. Precise, machine-readable critique is dramatically more useful feedback than asking the model to "improve" generically.

A worked example: an LLM generates a product description. The evaluator checks that the description includes the required fields (price range, key features, call to action) and returns a structured list of which fields are missing. The generator receives this list and regenerates. After two iterations, coverage reaches 100%. The whole loop takes three LLM calls and less than two seconds.

Generator produces a draft; evaluator scores it; if the score fails the threshold the critique loops back to the generator; a pass exits to final output.

An evaluator loop without a hard iteration cap is an unbounded cost. If the threshold is set too high or the generator keeps making the same mistake, the loop runs until your rate limit or budget stops it. Always add max_iterations and handle the case where the output never passes by returning the best result seen so far along with a status flag.

Reflection

Reflection is a lightweight self-check: the model generates a draft, then re-reads it against the original prompt or a checklist, and produces a revised version. The same model plays both roles in sequence. This is cheaper than a separate evaluator call and effective for catching format violations, missing required sections, and tone drift.

Do not expect reflection to fix factual errors. The model re-reading its own output will rarely catch hallucinated facts because the same bias that produced them also reads them as plausible. Reflection improves structure and completeness. A separate grounded evaluator is needed for factual correctness.

Reflection is a good first step before adding a full evaluator loop. If reflection alone resolves the quality issues you observe, you have saved the cost and complexity of a second model call. If it does not, you have a clear diagnosis that the problem is factual rather than structural, and you can target the right fix.

Tool Use

A model with tools can call external functions during inference and incorporate their results before completing the generation. The model reasons about when a tool is needed, emits a structured function call, receives the result, and continues generation. This enables precise computation, real-time data access, and side-effecting actions like writing to a database or sending a notification.

The most common failure is a malformed function call. When the model emits an invalid tool call signature, the runner must decide whether to retry, return an error, or skip. Validate tool call parameters before execution and return structured error messages when validation fails, not opaque exceptions. The model can recover from "field date must be ISO 8601 format"; it cannot recover from an HTTP 500.

Tool loops share the termination problem with evaluator loops. Set a maximum number of tool calls per request. A model that keeps calling the same search tool because it is not satisfied with the results will burn budget without converging.

LLM reasons, calls an external tool, receives the result, then generates with enriched context; a multi-tool loop allows additional calls before the final answer.

Retrieval Augmentation

Before calling the LLM, fetch relevant content from an external store and prepend it to the prompt. The model reasons over retrieved facts rather than parametric recall. The retrieval module documents the full pipeline mechanics in depth. In the context of design patterns, RAG is a composable building block: it fits inside a prompt chain as one step, inside a routing branch as the knowledge handler, or as the evidence-gathering phase of an evaluator loop that checks whether the generated answer is grounded in the retrieved chunks.

The key difference between RAG and the other patterns above is that RAG modifies what the model can draw on, not how the system coordinates. You can combine RAG with any of the other patterns without conflict.

Pattern Comparison

Pattern	Latency profile	Cost profile	Determinism	Best suited for
Prompt chaining	Additive per step	Low to medium	High	Sequential multi-step transformation
Routing	Plus one classification call	Low	High	Mixed request types with different handlers
Parallelization	Parallel end to end (fast)	Scales with fan width	Medium	Independent sub-tasks, voting ensembles
Orchestrator and workers	High (coordination plus workers)	High	Low	Complex cross-domain tasks
Evaluator loop	Variable (iterates)	Medium to high	Medium	Quality-critical outputs with clear pass criteria
Reflection	Roughly two times a single call	Medium	Medium	Format, completeness, and tone checks
Tool use	Low plus tool latency	Low to medium	Medium	Precise computation and real-time data
RAG	Low plus retrieval	Low to medium	High	Knowledge-grounded answers

Start with the simplest pattern that could work. A three-step prompt chain with structured output at each boundary solves more problems correctly than an orchestrated multi-agent loop solves impressively. Reach for agents when the task requires genuine branching decisions that are not knowable in advance, not because the problem feels complex enough to deserve them.

Combining Patterns

The patterns compose. A realistic customer support system might use:

A router at the entry point to classify requests as product questions, billing questions, or technical issues
RAG inside the product question branch to retrieve catalog and documentation content
A prompt chain inside the technical issue branch (reproduce the issue, diagnose the cause, propose a fix as sequential steps)
An evaluator loop on the final answer to verify the response cites a support article and avoids promising outcomes that are not guaranteed
Tool use in the billing branch to look up live account state before generating a response

These five patterns are not five agents. They are five structural decisions that a single request may traverse, each handled by deterministic application code except where an LLM call is actually needed. The routing logic and conditional execution are ordinary code. Complex behavior emerges from simple, well-bounded patterns combined by application logic, not from one all-purpose agent doing everything.

This is the worked example worth internalizing for interviews: sketch the routing topology first, identify which branches need knowledge retrieval, mark which outputs benefit from evaluation, and call out where tool access is genuinely required. By the time you invoke an agent loop, you should be able to articulate why the task graph is not knowable in advance.

A composed pattern system is also easier to evaluate than a single agent. Each component has a narrow contract, which means you can write targeted tests for routing accuracy, retrieval recall, and evaluator calibration independently. An all-in-one agent loop can only be evaluated end to end, which makes it expensive to diagnose when output quality drops.

Common Anti-Patterns

Agents When a Pipeline Would Do

The most common mistake in LLM system design is reaching for an agent loop when the task steps are actually known in advance. If you can write down the three steps before the request arrives, write a three-step prompt chain. Agent loops add non-determinism, make debugging harder, and cost more per request. Reserve them for tasks where the required steps genuinely depend on intermediate results that are not predictable at design time.

A useful test: if you can hardcode the control flow as an if-else tree without an LLM deciding which branch to take, you do not need an agent. A chain or a router will serve you better.

Unbounded Loops

Any loop in an LLM system (evaluator loop, tool use loop, agent loop) needs a hard termination condition that is not the model's judgment. The model will sometimes never converge. Set maximum iteration counts, maximum token budgets, and maximum wall-clock time limits. Treat hitting the limit as a recoverable error: return the best output seen so far, log the event, and surface it for monitoring. Do not crash.

No Fallback

Routing needs a fallback route. Tool use needs behavior for when a tool call fails. RAG needs behavior for when retrieval returns nothing. Every branching decision in an LLM pipeline has an unexpected-input case, and raising an unhandled exception is not the right answer for user-facing systems. Design the fallback explicitly, test it in staging, and monitor how often it triggers in production. A spike in fallback rate is one of the most useful early-warning signals you can have about distribution shift in incoming requests.

Prompt Spaghetti

A system prompt that handles ten different cases through ten nested conditional instructions is the LLM equivalent of legacy string-concatenation code. Nobody fully understands it, every edit has side effects, and the model starts ignoring instructions in the lower half once the prompt grows long enough. The fix is routing: pull each distinct behavior into its own focused prompt and route to it. The router can be a simple classifier or even a keyword match for high-confidence categories.

Prompt spaghetti compounds over time. A system prompt that starts at 200 tokens and grows to 3,000 through accumulated feature additions loses coherence long before it hits any context limit. The model attends more strongly to the beginning of the prompt; instructions buried deep are frequently ignored. Treat your system prompt like application code: refactor when complexity grows, not after it breaks.

Interview Angle

Keep going

Practice what you just read against real interview questions, or carry on through the curriculum.

Practice interview questions Browse the curriculum Visit studyaidesign.com

Follow along for new chapters and explainers:Instagram