How do you design an AI system that gracefully degrades when the model is unavailable?

In this question, we will learn how to design an AI system that gracefully degrades when the model is unavailable. We will focus on concrete patterns that keep users productive, maintain safety, and make outages predictable. Let's treat failure modes as first class features of the design.

We will cover the following:

• The intuition
• How it actually works
• Implementation patterns
• Monitoring and recovery
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: design layered fallbacks, clear detection, and predictable UX so the system still serves useful, safe outputs when the primary model is down. Use cached responses and deterministic heuristics first, a smaller or specialized model next, and human escalation last; protect these paths with circuit breakers and health checks.

The intuition (an analogy that makes it click)

Think of a live event that normally has a headline speaker with a microphone. If the microphone fails, you first pass a backup microphone. If that is not available, you use a prepared statement read aloud. If neither works, you display the transcript or a notice and route VIP questions to staff. Each step trades some richness for reliability.

Graceful degradation is exactly that ladder. We trade model expressiveness for availability in predictable steps so users are never left with silence.

How it actually works (the real mechanics, with one concrete worked example)

We will use a chat assistant that normally calls a large LLM. Our goals are to keep latency under 1.5s for cached replies and under 3s for a small local model, and to avoid incorrect confident answers.

Key components:

• Primary model service with autoscaling
• Cache of recent QA pairs and template replies
• Deterministic rules for simple queries
• Smaller local model as fallback
• Circuit breaker and health service
• Human-in-the-loop escalation

Failure detection relies on health and confidence. Health checks return a simple 200 or error and latency. Confidence arrives with the model response as a score $c$ where $0\le c\le 1$. We set thresholds like $c_{safe}=0.7$ for primary, and $c_{fallback}=0.6$ for smaller model.

Availability math helps sizing. If the primary model has availability $A_1$ and the fallback model has $A_2$, the combined availability for at least one working model in parallel is:

If $A_1=0.98$ and $A_2=0.95$, then $A_{parallel}=1-(0.02)(0.05)=0.999$.

Concrete example table comparing options for a single user request path:

A request flow example:

1. Check cache. If a recent exact or high-similarity hit exists with freshness rules, return cached reply.
2. If not cached, send to primary model and start a timer of 2 seconds.
3. If primary fails health or returns $c<c_{safe}$, open the circuit and try smaller model or deterministic rule.
4. If all automated paths fail, show a graceful message and route to human support when required.

Implementation patterns

Layered fallback pattern: prioritize cheap, deterministic outputs before expensive models. Implement a priority queue for fallback strategies and attach an estimated latency and confidence to each step.

Circuit breaker and backoff: the circuit breaker has three states: closed, open, half-open. Use rolling error windows and latency thresholds to open the circuit. When open, route automatically to fallback without attempting the primary.

Confidence and gating: never return model outputs that have confidence below threshold for that intent. For safety-critical domains, require $c\ge 0.9$ or human review.

Caching strategy: store (input fingerprint, response, timestamp, quality label). Define freshness rules per intent. For example, FAQs may be valid for 7 days, dynamic data only 30 seconds.

Monitoring and recovery

Monitor three families of signals: health, quality, and user experience. Health includes success rate and latency. Quality includes confidence distribution and post-hoc NPS or human review. User experience includes observed fallback rates and time to resolution.

Instrument metrics such as:

Use automatic remediation where safe. For example, if error rate exceeds a threshold, increase fallback routing and notify on-call. Progressive recovery can reconnect primary only in small traffic samples first.

Tradeoffs and failure modes

Other failure modes: inconsistent state between fallback and primary responses, cache poisoning, incorrect confidence calibration, or overfitting fallback models to edge cases. Plan tests for these.

Questions the interviewer might ask

Some follow-up questions you might get:

How do you decide which queries get cached? Cache queries that are idempotent and content-stable, like FAQs or confirmation messages. Use intent classification to mark cacheable requests.

How do you calibrate model confidence for gating? Use held-out labeled data and reliability diagrams. Adjust thresholds to balance precision and recall for the target intent.

What latency budget do you give each fallback? Set budgets based on SLAs and user tolerance. Example: 100 ms for cache, 1.5 s for a local model, 3 s total before offering human fallback.

How do you avoid inconsistent answers after recovery? Record which path produced each response and tag it. On recovery, reconcile by re-running the primary or flagging conflicting answers for human review.

When do you escalate to humans? Escalate for low confidence on safety-critical intents, repeated fallback failures, or user requests for human help. Track escalation rate as a metric.

Some things to note:

• Test degradation paths regularly with chaos experiments so behavior is predictable.
• Keep UX transparent: communicate when an answer is cached or simplified.

What the interviewer is really testing

They want to see that you can design for real-world failure, not only ideal model performance. The interviewer checks that you know detection, fallback options, and tradeoffs between quality, cost, and latency. They also want to hear about monitoring, operator workflows, and how you protect user safety and trust under failure.