How do you handle agent failures and implement error recovery?

In this question, we will learn how to handle agent failures and implement error recovery so your system stays correct and available when agents misbehave. We will focus on practical detection, recovery patterns, and how to protect state when an agent restarts or another takes over.

We will cover the following:

• The intuition (an analogy that makes it click)
• How it actually works (with a concrete worked example)
• Failure detection techniques
• State reconciliation and idempotency
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: build a short recovery loop: detect failures quickly, classify whether they are transient or permanent, attempt safe automated recovery (retries with backoff, restart, or failover) and then reconcile state with idempotent operations and checkpoints. Instrument everything and prefer simple, well-tested primitives like retries, circuit breakers, and leader election rather than ad hoc code.

The intuition (an analogy that makes it click)

Think of an agent as a courier delivering parcels. If a courier is late we first check why: traffic or injury. For traffic we wait and retry with more time. For injury we replace the courier and recover the undelivered packages from the manifest. The courier manifest, safe retry rules, and replacement plan are our checkpoints, idempotency, and failover strategy.

How it actually works (the real mechanics, with a concrete worked example)

Example: a worker agent pulls tasks from a queue, processes them, and marks a database row complete. Failures we might see include transient network errors, process crashes, and corrupt task payloads. A robust recovery flow looks like this:

1. Detection: use heartbeats and per-task timeouts so you do not wait forever.
2. Classification: if a worker times out but queue still has messages, treat this as transient until repeated failures suggest a permanent bug.
3. Recovery:
   • Retry the operation with exponential backoff for transient errors. Use a circuit breaker to stop retrying when the downstream is overloaded.
   • If the process crashed, restart it and let it resume from a checkpoint or re-claim tasks.
   • If the node is unhealthy, trigger failover to another instance.
4. Reconciliation: ensure applying the same task twice is safe by making operations idempotent or storing an operation log.

Key formulas and examples:

Exponential backoff timing per retry $k$ from base $t_0$:

If a single attempt has success probability $p$, the probability of at least one success in $n$ independent retries is the complementary probability:

Recovery strategies comparison table:

Concrete worked flow for a queue worker:

• Worker reads task id 123 and checkpoint id 5.
• It sets a processing lease in the queue for $\tau$ seconds and starts work.
• On transient DB timeout, the worker retries up to 3 times with $t_0=100\,\text{ms}$ and $t_k = 100\times 2^k$.
• If all retries fail, it extends the lease, records the failure reason in a side log, and escalates if repeated failures occur.
• If the worker crashes and another picks task 123, the new worker checks the last processed checkpoint and runs idempotent apply using task id to avoid duplication.

Failure detection techniques

Failure detection is the foundation. Common techniques:

• Heartbeats and leases, with clear timeouts relative to expected work durations.
• Per-operation deadlines so hung tasks are recoverable.
• Health probes that test external dependencies, not just process liveness.

Tune timeouts to workload. Too short and you cause unnecessary failovers, too long and you increase tail latency. Use adaptive timeouts if work durations vary widely.

State reconciliation and idempotency

Design your agent so redoing a step is safe. Patterns:

• Idempotent writes using upserts or compare-and-set keyed by task id.
• Operation logs or write-ahead logs so an agent can resume and replay safely.
• Checkpoints that record progress periodically rather than per event if throughput is high.

Example idempotent pattern for marking completion:

• Write a record for task id with status and unique attempt id. Use a single update that only sets status "done" if current status is not "done".

When true undo is required, implement compensating transactions and document the invariants the compensator must restore.

Tradeoffs and failure modes

Recovery policies introduce tradeoffs. Faster recovery improves availability but may increase duplicate work or overload downstream services. More complex reconciliation reduces duplication but increases development and testing effort.

Other failure modes to watch for:

• Split-brain during leader election causing concurrent agents to act on the same data.
• Lost progress when checkpoints are too coarse.
• Silent corruption if checksums and validation are missing.

Questions the interviewer might ask

Some follow-up questions you might get:

How do you choose a timeout or heartbeat interval? Pick intervals based on observed task durations and expected recovery time. Make heartbeat timeout safely larger than median work time and allow adaptive tuning for variable workloads.

When do you use retries versus failover? Use retries for short-lived, likely transient errors. Use failover if the node is unresponsive, or retries do not improve success probability and you suspect a permanent fault.

How do you avoid duplicate side effects when you retry? Make operations idempotent with unique task ids or record attempts in a durable store and check for prior success before performing side effects.

What metrics should we monitor to know recovery is working? Monitor error rates, retry counts, circuit breaker trips, restart frequency, task processing latency, and number of stale leases. High retry or restart rates indicate an underlying issue.

How do you handle stateful agents with in-memory caches? Persist critical state to durable storage or replicate state across nodes. Use fast cold-start logic and rehydrate caches from the authoritative store on takeover.

Some things to note:

• Always test recovery paths with fault injection under load.
• Prefer simple, well-understood primitives over brittle custom code.

What the interviewer is really testing

They want to see that you can design robust systems that survive common faults without causing cascading failures. They also test for practical knowledge: how you detect and classify failures, how you recover safely, and how you preserve correct state. Clear tradeoff reasoning, observability plans, and small, testable recovery primitives will demonstrate competence.