Facio's Dead-Letter Discipline: How AI Agent Systems Handle the Work That Will Never Succeed
Some work is destined to fail. A task references a record that was deleted. A request asks for an action that violates policy and will always violate policy. An integration points at a service that no longer exists. The task is valid; the work is impossible; the failure is permanent.
Naive agents retry these tasks forever. The agent tries the task, fails. The agent retries. Fails. Retries. Fails. The retries consume resources (compute, model API calls, database connections). The retries add noise to metrics. The retries might even cause side effects if the task is partially executable. The system churns on impossible work.
The failure pattern has a name in distributed systems: a poison message. A message that can never be processed successfully. Left in the queue, it poisons the system — every retry fails, every retry costs resources, every retry delays other work.
Facio's dead-letter discipline gives every task a maximum retry budget, a quarantine path for work that can't succeed, and a recovery workflow for work that can be fixed. Failed work is contained. The system stays healthy. Humans review what matters.
Here's how the discipline works, what patterns it includes, and why dead-letter handling is what makes AI agent systems robust against impossible work.
The Poison Message Reality
Production AI agent systems encounter poison messages in many forms:
Form 1: Permanently invalid input. A customer submits an order with a negative quantity. The validation catches it. The task is permanently invalid. No retry will make it valid.
Form 2: Deleted dependencies. A task references a customer record that was deleted. The task can't complete without the customer. The customer is permanently gone.
Form 3: Policy violations. A task asks for an action that's prohibited ("transfer $1M from this account to that account"). The policy blocks it. The block is permanent for this type of request.
Form 4: Missing services. A task calls an integration that's been deprecated. The integration is gone. The task can't complete.
Form 5: Permanent authorization failures. A task tries to access data the user isn't authorized to see. The authorization check fails. The failure is permanent.
Form 6: Schema mismatches. A task's expected schema has changed. The task's payload doesn't match. The mismatch is permanent unless the payload is updated.
Form 7: Resource exhaustion patterns. A task requires a resource that doesn't exist (specific database table, specific feature flag, specific configuration). The resource is permanently absent.
In every form, the pattern is the same: the task is valid (not malformed), but the work is impossible (cannot complete given current state). Retrying doesn't help. The work needs different handling.
The Dead-Letter Discipline
Facio's dead-letter discipline has five pillars. Each addresses a different aspect of poison message handling.
Pillar 1: Retry Budget (Bounded Attempts)
Every task has a maximum retry budget:
# Retry budget per task
retry_budget = {
"default_max_retries": 5,
"default_retry_strategy": "exponential_backoff",
"default_initial_delay_seconds": 30,
"default_max_delay_seconds": 3600,
}
# Per-task-type customization
retry_policies = {
"transient_api_call": {
"max_retries": 10,
"strategy": "exponential_backoff_with_jitter",
"max_attempt_duration_seconds": 300,
},
"user_initiated_action": {
"max_retries": 3,
"strategy": "fixed_delay",
"delay_seconds": 60,
"user_notification_after_failure": True,
},
"critical_business_workflow": {
"max_retries": 20,
"strategy": "exponential_backoff",
"manual_intervention_after_max": True,
},
"low_priority_batch": {
"max_retries": 2,
"strategy": "fixed_delay",
"delay_seconds": 300,
},
}
The retry budget is bounded. After maximum retries, the task moves to the dead-letter queue.
Pillar 2: Failure Classification (Transient vs Permanent)
Not all failures are equal. The discipline classifies failures:
# Failure classification
failure_classification = {
"transient": {
"examples": ["network_timeout", "rate_limit", "service_unavailable", "model_overloaded"],
"action": "retry_with_backoff",
"retry_count": "from_retry_budget",
},
"permanent": {
"examples": ["validation_failed", "policy_violation", "authorization_denied", "data_not_found"],
"action": "send_to_dead_letter_immediately",
"retry_count": 0,
},
"unknown": {
"examples": ["unexpected_error", "exception_without_category"],
"action": "retry_then_classify",
"retry_count": "from_retry_budget",
"after_max_retries": "move_to_dead_letter",
},
}
# Classifier (simple version)
def classify_failure(error):
if error.category in failure_classification["transient"]["examples"]:
return "transient"
if error.category in failure_classification["permanent"]["examples"]:
return "permanent"
return "unknown"
The classification determines retry behavior. Permanent failures skip retries and go straight to dead-letter.
Pillar 3: Dead-Letter Queue (Quarantine for Failed Work)
Failed tasks are quarantined in a dead-letter queue:
# Dead-letter queue structure
dead_letter_queue = {
"tasks": [
{
"task_id": "task-abc123",
"original_task": {...}, # Full original task payload
"failure_history": [
{"attempt": 1, "error": "...", "category": "transient"},
{"attempt": 2, "error": "...", "category": "transient"},
{"attempt": 3, "error": "...", "category": "transient"},
{"attempt": 4, "error": "...", "category": "transient"},
{"attempt": 5, "error": "...", "category": "permanent"}, # Final attempt
],
"final_failure": {
"error": "policy_violation: refund amount exceeds customer lifetime value",
"category": "permanent",
"timestamp": "2026-07-21T10:23:45Z",
},
"metadata": {
"task_type": "refund_request",
"customer_id": "cust-12345",
"submitted_at": "2026-07-21T09:00:00Z",
"dead_lettered_at": "2026-07-21T10:23:45Z",
"failure_reason_summary": "Refund exceeds allowed maximum",
},
"recovery_options": ["manual_review", "data_correction", "policy_update"],
},
],
}
The dead-letter queue holds failed work with full context. The team can analyze, fix, and replay the work.
Pillar 4: Recovery Workflows (For Salvageable Work)
Some dead-lettered work can be fixed and replayed:
# Recovery workflow for refund request
recovery_workflow_refund = {
"trigger": "task_in_dead_letter_with_failure_reason: refund_exceeds_maximum",
"options": [
{
"action": "manual_review",
"description": "Manager reviews and approves exceptional refund",
"actor": "customer_service_manager",
"outcome": "approve_and_replay | reject_and_close",
},
{
"action": "split_refund",
"description": "Split into multiple smaller refunds within policy limits",
"actor": "system_or_human",
"outcome": "create_subtasks_and_replay",
},
{
"action": "data_correction",
"description": "Customer record was wrong, correct and replay",
"actor": "data_team",
"outcome": "correct_data_and_replay",
},
{
"action": "close",
"description": "Task cannot be salvaged, close with notification",
"actor": "system",
"outcome": "notify_customer_and_close",
},
],
}
The recovery workflow gives the team structured options for fixing work. The work is salvageable; the team has a path forward.
Pillar 5: Visibility and Alerting (Surfaces Problematic Work)
The dead-letter queue is observable:
# Dead-letter dashboard
dashboard = {
"total_dead_lettered": 234,
"by_failure_reason": {
"policy_violation": 89,
"data_not_found": 67,
"validation_failed": 45,
"authorization_denied": 23,
"service_unavailable": 10,
},
"by_age": {
"last_hour": 12,
"last_24_hours": 87,
"last_7_days": 234,
},
"recovery_actions_taken": 156,
"closed_without_recovery": 78,
"alert_thresholds": {
"dead_letter_rate_per_hour": {"warn": 5, "critical": 20},
"oldest_dead_lettered_task_hours": {"warn": 48, "critical": 168},
},
"alerts_active": [
{"level": "warn", "message": "Dead-letter rate elevated: 8/hour (threshold 5/hour)"},
],
}
The dashboard surfaces dead-letter patterns. The team knows what's failing, why, and how old. The alerts catch sudden spikes.
The Dead-Letter Patterns
Several patterns emerge from disciplined dead-letter handling.
Pattern 1: Progressive Retry Strategies
Retries use progressive strategies based on the failure type:
# Progressive retry strategies
retry_strategies = {
"exponential_backoff": {
"delay_sequence": [30, 60, 120, 240, 480, 960], # Doubling each time
"max_delay_seconds": 3600,
"use_when": "transient_failures_with_uncertain_recovery",
},
"exponential_backoff_with_jitter": {
"delay_sequence": "exponential_with_random_jitter",
"jitter_pct": 20,
"use_when": "many_concurrent_retries_to_avoid_thundering_herd",
},
"fixed_delay": {
"delay_sequence": [60] * 10, # Always 60s
"use_when": "predictable_recovery_time",
},
"circuit_breaker_integration": {
"use_when": "downstream_service_unreliable",
"behavior": "stop_retrying_when_circuit_open",
},
}
# Retry execution with progressive strategy
def execute_with_retry(task, strategy):
for attempt in range(strategy.max_retries):
delay = calculate_delay(strategy, attempt)
sleep(delay)
try:
result = execute_task(task)
return result
except TransientError as e:
log_attempt(task, attempt, e, "transient")
continue
except PermanentError as e:
log_attempt(task, attempt, e, "permanent")
return send_to_dead_letter(task, e)
return send_to_dead_letter(task, "max_retries_exceeded")
The progressive strategies handle different recovery patterns. Some failures recover quickly; others need longer waits.
Pattern 2: Automatic Retry Classification
Some failures can be automatically classified and routed:
# Automatic retry classification
auto_classification = {
"rate_limit_response": {
"detection": "http_status_429_or_provider_specific",
"classification": "transient",
"action": "respect_retry_after_header",
},
"validation_error": {
"detection": "schema_validation_failed",
"classification": "permanent",
"action": "send_to_dead_letter_with_validation_details",
},
"service_unavailable": {
"detection": "http_status_503",
"classification": "transient",
"action": "retry_with_backoff",
},
"data_not_found": {
"detection": "record_not_found_exception",
"classification": "permanent",
"action": "send_to_dead_letter_for_data_recovery",
},
}
The auto-classification routes failures immediately. No manual triage needed for known patterns.
Pattern 3: Bulkhead for Dead-Letter Processing
Dead-letter processing shouldn't compete with active work:
# Bulkhead for dead-letter processing
dead_letter_bulkhead = {
"separate_resource_pool": True,
"max_concurrent_processing": 5,
"rate_limit_per_minute": 30,
"scheduled_processing_window": "off_peak_hours", # 2am-6am
"human_review_required": True,
"manual_replay_only": True, # Don't auto-replay
}
The bulkhead prevents dead-letter processing from affecting active work. The replay is manual; humans approve before work is re-attempted.
Pattern 4: Failure Pattern Analysis
The discipline analyzes failure patterns to prevent future poison messages:
# Failure pattern analysis
pattern_analysis = {
"common_failure_reasons": {
"policy_violation": "indicates_policy_too_strict_or_customer_misunderstanding",
"data_not_found": "indicates_data_quality_issue_or_race_condition",
"validation_failed": "indicates_input_validation_gap",
"authorization_denied": "indicates_permission_structure_issue",
},
"trend_analysis": {
"weekly_dead_letter_volume": "tracking_to_identify_spikes",
"failure_reason_trends": "tracking_to_see_if_specific_reasons_grow",
"recovery_success_rate": "tracking_to_see_if_recovery_works",
},
"preventive_actions": {
"policy_violation": ["review_policy", "improve_customer_communication"],
"data_not_found": ["add_data_validation", "fix_race_condition"],
"validation_failed": ["improve_input_validation", "pre-validate_at_intake"],
},
}
The analysis turns dead-letters into improvements. Each failure pattern points to a preventive action.
Pattern 5: Customer Communication for Failed Work
When tasks fail permanently, customers are informed:
# Customer communication for failed tasks
customer_communication = {
"refund_failed": {
"subject": "Refund Request Could Not Be Processed",
"body": "Your refund request could not be processed because it exceeds our policy limits. A customer service representative will contact you within 24 hours to discuss alternatives.",
"tone": "empathetic_and_professional",
"next_action": "human_followup_within_24_hours",
},
"task_failed_validation": {
"subject": "Submission Could Not Be Processed",
"body": "The submission could not be processed because required information is missing. Please review and resubmit with the requested details.",
"tone": "helpful_and_specific",
"next_action": "user_can_resubmit_with_corrections",
},
}
The communication is structured. Customers aren't left wondering. The next action is clear.
The Dead-Letter Discipline Doesn't Do
Honest limitations:
- It doesn't fix the underlying problem. Dead-lettering contains failure; it doesn't prevent it. The team still needs to address root causes.
- It doesn't guarantee recovery. Some dead-lettered tasks can't be recovered. The team has to close them.
- It requires human review. Automatic replay is dangerous. The discipline requires humans to approve replays.
- It can grow. Dead-letter queues can accumulate. The team needs retention policies.
- It adds latency. Recovery takes time (review, correction, replay). The latency is the cost of safety.
The Dead-Letter as Operational Practice
Dead-letter handling is operational practice:
Active monitoring. The team monitors dead-letter rates. Spikes are investigated.
Pattern analysis. Failures are analyzed. Patterns are identified. Root causes are addressed.
Recovery procedures. Recovery workflows are documented. The team knows how to handle each failure type.
Customer communication. Communication templates exist. Customers get clear messages.
The practice is what makes the discipline sustainable. Without it, dead-letter queues grow unmonitored. With it, the team contains and addresses failures.
The Compound Effect of Dead-Letter Discipline
Dead-letter discipline compounds:
- System health. Impossible work doesn't consume resources. The system stays healthy.
- Clear failure signals. Dead-letter queues show what's failing. The team sees problems.
- Recovery paths. Salvageable work has a path forward. The team recovers value.
- Customer experience. Failed tasks have clear communication. Customers aren't confused.
- Continuous improvement. Failure patterns drive prevention. The system gets better.
The undisciplined approach has the opposite trajectory. Resource exhaustion, hidden failures, lost work, customer confusion, no improvement.
Bottom Line
Some work can't succeed. Without discipline, the system churns on impossible work forever. With dead-letter discipline, impossible work is contained and handled.
Facio's dead-letter discipline provides retry budgets, failure classification, dead-letter queues, recovery workflows, and visibility. The discipline makes AI agent systems robust against poison messages.
The system without dead-letter handling is a churn machine for impossible work. The system with it is a partner for handling failures. The team prefers the partner. The customers prefer the partner.
Because AI agents in production encounter impossible work. The question is whether the impossible work is contained or destroys the system. The dead-letter discipline is what makes the answer contained.
See the dead-letter documentation for retry policies, failure classification, and recovery workflows.