Facio's Idempotency Discipline: How AI Agents Avoid Double-Charging Customers When Retries Happen

AI agents fail and retry. The model returns a 500 error, the agent retries the tool call. The network drops mid-flight, the agent retries. The tool returns a timeout but the operation actually succeeded, the agent retries. The model decides to "make sure" by calling the tool again, the agent retries.

Without idempotency, retries cause duplicate side effects. The agent that retries a Stripe refund charges the customer twice. The agent that retries a Slack message sends the message three times. The agent that retries a database insert creates duplicate records. The agent that retries a deployment deploys twice. The user sees the duplicate; the team gets the support ticket; the cleanup is manual.

Facio's idempotency discipline gives every side-effecting operation a unique key, makes operations safe to retry, and detects duplicate executions before they cause damage. The agent can retry confidently; the customer doesn't get double-charged; the system stays consistent.

Here's how the discipline works, what it enforces, and why idempotency is what makes AI agent operations trustworthy in production.

The Retry Reality

Production AI agents retry operations for many reasons:

Reason 1: Transient failures. The model API returns 500, 502, 503, or 504. These are temporary; the retry succeeds. Without idempotency, the operation that "failed" might have actually succeeded; the retry creates a duplicate.

Reason 2: Network timeouts. The agent makes a request, the network drops mid-flight. The agent doesn't know if the request arrived. Without idempotency, the retry might process a request that already executed.

Reason 3: Tool timeouts. The tool runs longer than expected. The agent times out. But the tool might still complete. The retry duplicates the work.

Reason 4: Model uncertainty. The model isn't sure if a tool call succeeded. It "checks" by calling the tool again. Without idempotency, the check creates a duplicate.

Reason 5: Explicit retry policies. The agent is configured to retry on failure. The retry happens. The duplicate happens.

Reason 6: Crash recovery. The agent crashes mid-workflow. It restarts. It replays its previous steps to recover state. The replay re-executes operations that already executed.

In every case, the underlying issue is the same: the agent doesn't know if the operation actually executed. Retrying without idempotency is gambling — sometimes the operation actually failed (retry is correct), sometimes it actually succeeded (retry is a duplicate).

The discipline is to make every operation's execution status deterministic. The agent always knows if the operation ran.

The Idempotency Discipline

Facio's idempotency discipline has four pillars. Each addresses a different aspect of safe retry.

Pillar 1: Idempotency Keys (Unique Per Logical Operation)

Every side-effecting operation gets a unique key. The key identifies the logical operation, not the specific call:

# Operation with idempotency key
operation = {
    "idempotency_key": "refund-order-12345-2026-07-17",
    "operation_type": "stripe.create_refund",
    "parameters": {
        "charge_id": "ch_abc123",
        "amount_usd": 50.00,
        "reason": "customer_requested",
    }
}

# Generate idempotency key (deterministic from logical operation)
def generate_key(operation_type, parameters):
    key_input = json.dumps({"type": operation_type, "params": parameters}, sort_keys=True)
    return hashlib.sha256(key_input.encode()).hexdigest()

# Same logical operation → same idempotency key
# Different logical operation → different key

The idempotency key identifies the operation's intent. The same intent (refund for order 12345) gets the same key, regardless of how many times the agent tries.

Pillar 2: Idempotency Store (Track Execution Status)

The runtime maintains an idempotency store that tracks execution status:

# Idempotency store
idempotency_store = {
    "refund-order-12345-2026-07-17": {
        "status": "completed",            # pending, in_progress, completed, failed
        "first_seen_at": "2026-07-17T10:15:23Z",
        "completed_at": "2026-07-17T10:15:24.234Z",
        "result": {
            "refund_id": "re_xyz789",
            "status": "succeeded",
            "amount_usd": 50.00,
        },
        "execution_count": 2,              # Called twice, executed once
    },
    "send-email-cust-456-2026-07-17": {
        "status": "completed",
        "first_seen_at": "2026-07-17T10:18:45Z",
        "completed_at": "2026-07-17T10:18:46.123Z",
        "result": {
            "message_id": "msg_abc",
            "status": "delivered",
        },
        "execution_count": 1,
    },
}

The store tracks every operation's status. The agent checks the store before executing. If the operation already completed, the agent gets the cached result instead of re-executing.

Pillar 3: Operation Wrapping (Idempotent Execution)

Every side-effecting operation is wrapped to be idempotent:

# Operation wrapper
def execute_idempotent(operation):
    key = operation["idempotency_key"]

    # Check if already executed
    existing = idempotency_store.get(key)
    if existing and existing["status"] == "completed":
        # Already executed, return cached result
        return existing["result"]

    if existing and existing["status"] == "in_progress":
        # Another execution in progress, wait or fail
        # Option 1: Wait for completion
        # Option 2: Return "in progress, retry later"
        return "in_progress_retry_later"

    # Mark as in_progress and execute
    idempotency_store.set(key, {"status": "in_progress", "started_at": now()})

    try:
        # Execute the actual operation
        result = execute_operation(operation)
        idempotency_store.set(key, {
            "status": "completed",
            "completed_at": now(),
            "result": result,
        })
        return result
    except Exception as e:
        idempotency_store.set(key, {
            "status": "failed",
            "failed_at": now(),
            "error": str(e),
        })
        raise

The wrapper ensures only one execution per idempotency key. Concurrent calls see consistent state.

Pillar 4: Result Caching (Fast Retry)

The result of completed operations is cached. Retries get the cached result immediately:

# Result cache (faster than checking store)
result_cache = {
    "refund-order-12345-2026-07-17": {
        "result": {"refund_id": "re_xyz789", "status": "succeeded"},
        "cached_at": "2026-07-17T10:15:24Z",
        "ttl_seconds": 86400,             # 24 hours
    },
}

def get_cached_result(key):
    cached = result_cache.get(key)
    if cached and cached["cached_at"] + cached["ttl_seconds"] > now():
        return cached["result"]
    return None

The result cache makes retries fast. The agent doesn't wait for the store check; it gets the cached result.

The Idempotency Patterns

Several patterns emerge from disciplined idempotency.

Pattern 1: Deterministic Idempotency Keys

The idempotency key is derived from the operation's logical content:

# Deterministic key generation
def make_refund_key(order_id, refund_reason):
    return f"refund-{order_id}-{refund_reason}"

# Same order + same reason → same key
# Different order or different reason → different key

# Example:
key_1 = make_refund_key("order-12345", "customer_requested")
# "refund-order-12345-customer_requested"

# If the agent retries the refund for the same order and reason, the same key is used
# The second call sees "already executed" and returns the cached result

The deterministic key makes retries safe by construction. The agent doesn't need to remember "did I already do this?"; the key is the answer.

Pattern 2: Client-Provided Idempotency Keys

For operations where the agent provides the key explicitly:

# Agent generates idempotency key
operation = {
    "idempotency_key": "agent-session-abc123-step-7-refund",
    "operation_type": "stripe.create_refund",
    "parameters": {...},
}

# Same key for retries of the same logical operation
# Different keys for different operations

The agent-provided key gives the agent control. The agent decides what constitutes "the same operation" and generates keys accordingly.

Pattern 3: Server-Enforced Idempotency

Many APIs support server-side idempotency. The agent provides the key; the server enforces:

# Server-enforced idempotency (Stripe example)
response = stripe.Refund.create(
    amount=5000,
    charge="ch_abc123",
    idempotency_key="refund-order-12345-2026-07-17"
)

# Stripe internally tracks the key
# If a request with the same key arrives, Stripe returns the original result
# No duplicate charge happens, even if the network causes retries

The server-enforced pattern uses the API's built-in idempotency. The agent just provides the key; the server handles the rest.

Pattern 4: Idempotency for Non-Deterministic Operations

Some operations are inherently non-idempotent (send email, deploy code). The discipline uses compensating actions:

# Non-idempotent operation wrapper
def idempotent_send_email(to, subject, body, idempotency_key):
    # Check if already sent
    if idempotency_store.exists(idempotency_key):
        return idempotency_store.get_result(idempotency_key)

    # Send the email
    result = email_api.send(to, subject, body)

    # Record the send
    idempotency_store.set(idempotency_key, {
        "status": "completed",
        "result": result,
        "side_effect": {"type": "email_sent", "message_id": result.id},
    })

    return result

# If the agent retries, the same email isn't sent twice
# The cached result is returned

The non-idempotent operation is wrapped to become idempotent. The wrapper checks the store; if already executed, returns the cached result.

Pattern 5: Time-Bounded Idempotency

Idempotency windows prevent stale results:

# Time-bounded idempotency
idempotency_window = {
    "stripe.create_refund": 86400 * 30,      # 30 days
    "slack.send_message": 3600,              # 1 hour
    "email.send": 3600,                      # 1 hour
    "database.insert": 86400 * 7,            # 7 days
    "deploy.production": 86400,              # 24 hours
}

# After the window expires, the idempotency key is no longer valid
# A new key is generated for the new operation
# Prevents stale results from blocking legitimate new operations

The time-bounded pattern handles the case where "the same operation" really is a new operation after a long time.

The Idempotency Observability

Idempotency events are observable:

# Idempotency dashboard
dashboard = {
    "operations_tracked": 1247,
    "duplicates_prevented": 89,
    "duplicate_rate_pct": 7.1,
    "by_operation_type": {
        "stripe.create_refund": {"tracked": 234, "duplicates_prevented": 18},
        "slack.send_message": {"tracked": 567, "duplicates_prevented": 45},
        "email.send": {"tracked": 345, "duplicates_prevented": 23},
        "database.insert": {"tracked": 101, "duplicates_prevented": 3},
    },
    "duplicate_causes": {
        "model_retry": 42,
        "network_timeout": 28,
        "explicit_retry_policy": 12,
        "crash_recovery_replay": 7,
    }
}

The observability surfaces duplicate patterns. The team sees which operations get retried most, why retries happen, and how many duplicates were prevented.

The Idempotency Discipline Doesn't Do

Honest limitations:

• It doesn't fix the underlying failure. Idempotency makes retries safe; it doesn't make the original operation succeed. The team still needs to fix flaky tools.
• It doesn't work without key generation discipline. If the agent generates random keys for each retry, idempotency is broken. The team needs deterministic or well-managed key generation.
• It doesn't handle all operations. Pure read operations don't need idempotency (no side effects). The discipline applies to write operations.
• It adds storage overhead. The idempotency store grows. The team needs retention policies.
• It doesn't prevent all duplicates. A bug in key generation, a race condition in the store, or a malicious retry can bypass idempotency. The discipline is robust but not perfect.

The Idempotency as Operational Practice

Idempotency is operational practice:

Key strategy. The team designs key generation strategies. They document the rules. They review new operations for key strategy.

Store retention. The team manages the idempotency store. They set retention policies. They archive old entries.

Duplicate analysis. The team analyzes duplicate patterns. They identify flaky tools. They fix the underlying issues.

Testing. The team tests for idempotency. They verify that retries don't cause duplicates. They test edge cases (concurrent retries, crash mid-execution).

The practice is what makes the discipline sustainable. Without it, the team bypasses idempotency under pressure. With it, the team holds the line.

The Compound Effect of Idempotency

Idempotency compounds:

• Customer trust. Customers don't get double-charged. The trust is preserved.
• Cleaner data. No duplicate records. The database stays clean.
• Lower support costs. No "you charged me twice" tickets. The support load drops.
• Confident retries. The team can retry aggressively without fear. The operations are safe.
• Faster recovery. Crashes don't cause duplicate side effects. The recovery is clean.

The undisciplined approach has the opposite trajectory. Double charges, duplicate data, high support costs, conservative retries, slow recovery.

Bottom Line

AI agents retry. Without idempotency, retries cause duplicates. With idempotency, retries are safe.

Facio's idempotency discipline provides unique idempotency keys, an idempotency store, operation wrapping, and result caching. The discipline makes every side-effecting operation safe to retry. The agent retries confidently; the system stays consistent.

The agent without idempotency is a duplicate-creating machine. The agent with it is a safe-to-retry partner. The team trusts the safe-to-retry one. The customers don't get duplicates.

Because AI agents in production fail and retry. The question is whether the retries are safe or duplicating. The idempotency discipline is what makes the answer safe.

---

See the idempotency documentation for key generation, store configuration, and operation wrapping patterns.