Facio's Interpretability Discipline: How AI Agents Make Decisions That Customers, Auditors, and Regulators Can Actually Understand

AI agents make decisions that affect real people. A loan application is approved or denied. An insurance claim is paid or rejected. A customer is offered a discount or not. Content is flagged or allowed. A medical recommendation is followed or questioned. A hiring decision is made or deferred. The people affected by these decisions deserve to know why.

Auditors demand to know why. "Why was this claim rejected?" "Why was this loan denied?" "Why was this transaction flagged?" The audit answer can't be "the model said so." That's not an explanation; that's a non-answer.

Regulators require explanations. The EU AI Act, GDPR's right to explanation, the US Equal Credit Opportunity Act, and sector-specific regulations (insurance, healthcare, employment) all require decision-makers to explain automated decisions. The agent must produce explanations.

Naive agents produce outputs without explanations. The model returns a decision; the agent delivers it; nobody can tell why the agent decided what it decided. The team is left hand-waving; the auditor is left unsatisfied; the regulator is left concerned. The agent becomes a liability.

Facio's interpretability discipline gives every decision a structured explanation. What data the agent saw. What reasoning it followed. What alternatives it considered. Why it chose this path. The decision is explainable to the customer, auditable for the regulator, and traceable for the engineer.

Here's how the discipline works, what explanations it produces, and why interpretability is what makes AI agents deployable in high-stakes decision contexts.

The Interpretability Reality

AI agents make decisions in many high-stakes contexts:

Context 1: Credit decisions. A loan application is approved or denied. The applicant asks why. The lender must explain (ECOA, FCRA). The explanation must be specific: not "low creditworthiness" but "your debt-to-income ratio exceeds policy threshold."

Context 2: Insurance claims. A claim is paid or rejected. The policyholder asks why. The insurer must explain. The explanation must reference specific policy clauses and evidence.

Context 3: Content moderation. A piece of content is removed or kept. The user asks why. The platform must explain. The explanation must reference specific community standards and the violation.

Context 4: Medical recommendations. A treatment is recommended. The patient asks why. The doctor (and agent) must explain. The explanation must reference clinical guidelines and patient-specific factors.

Context 5: Hiring decisions. A candidate is rejected. The applicant asks why. The employer must explain. The explanation must reference job-relevant criteria.

Context 6: Fraud detection. A transaction is blocked. The customer asks why. The bank must explain. The explanation must reference risk signals (without revealing proprietary detection methods).

Context 7: Tax decisions. A return is filed. The taxpayer asks why a deduction was allowed or denied. The preparer must explain. The explanation must reference tax code and specific entries.

In every context, the explanation is a requirement. Not a nicety. Not optional. The agent that can't explain its decisions can't operate in these contexts.

The naive approach is to give explanations post-hoc. The team writes explanation templates; the agent fills them in; the explanations look plausible but aren't grounded in the agent's actual reasoning. The auditor pushes back: "did the agent actually consider these factors?" The team has no answer. The explanation is theater.

The discipline is to capture the reasoning as it happens, then generate explanations grounded in the actual reasoning.

The Interpretability Discipline

Facio's interpretability discipline has four pillars. Each addresses a different aspect of explanation quality.

Pillar 1: Reasoning Capture (Per Decision)

Every reasoning step is captured as the agent reasons:

# Reasoning capture (per decision)
reasoning_trace = [
    {
        "step": 1,
        "type": "input_review",
        "content": "Reviewed customer application: income $85,000, debt $45,000, requested loan $25,000",
        "factors_extracted": [
            {"name": "annual_income", "value": 85000},
            {"name": "existing_debt", "value": 45000},
            {"name": "requested_loan", "value": 25000},
        ],
    },
    {
        "step": 2,
        "type": "policy_check",
        "content": "Checked against credit policy: debt-to-income ratio must be <= 0.5",
        "calculation": "debt_to_income = (45000 + 25000) / 85000 = 0.82",
        "policy_threshold": 0.5,
        "result": "exceeds_threshold",
    },
    {
        "step": 3,
        "type": "alternative_consideration",
        "content": "Considered alternative: lower loan amount would meet policy",
        "alternative": {"loan_amount": 10000, "debt_to_income": 0.65, "meets_policy": False},
        "alternative": {"loan_amount": 5000, "debt_to_income": 0.59, "meets_policy": False},
        "alternative": {"loan_amount": 0, "debt_to_income": 0.53, "meets_policy": False},
    },
    {
        "step": 4,
        "type": "decision",
        "content": "Decision: DENY loan application. Reason: debt-to-income ratio of 0.82 exceeds policy maximum of 0.50",
        "decision": "denied",
        "reasoning": "exceeds_policy_threshold",
    },
]

The reasoning trace captures every step. The explanation is grounded in the actual reasoning, not in templates.

Pillar 2: Factor Attribution (What Drove the Decision)

Each decision is broken down by contributing factors:

# Factor attribution
factor_attribution = {
    "decision": "loan_denied",
    "factors": [
        {
            "factor": "debt_to_income_ratio",
            "value": 0.82,
            "policy_threshold": 0.50,
            "weight": "primary",         # This was the deciding factor
            "explanation": "Your total monthly debt of $4,167 against monthly income of $7,083 gives a debt-to-income ratio of 0.82, exceeding our policy maximum of 0.50.",
        },
        {
            "factor": "annual_income",
            "value": 85000,
            "policy_threshold": None,
            "weight": "supporting",       # This was relevant but not deciding
            "explanation": "Your annual income of $85,000 is within acceptable range.",
        },
        {
            "factor": "credit_score",
            "value": 720,
            "policy_threshold": 650,
            "weight": "supporting",
            "explanation": "Your credit score of 720 exceeds our minimum of 650.",
        },
    ],
    "primary_blocker": "debt_to_income_ratio",
}

The factor attribution shows what drove the decision. The customer knows what to fix; the auditor knows what to check.

Pillar 3: Audience-Appropriate Explanations

Explanations are tailored to the audience:

# Audience-appropriate explanations
explanation_templates = {
    "customer": {
        "tone": "empathetic_and_clear",
        "language": "plain_language_no_jargon",
        "length": "concise",
        "example": "We're unable to approve your loan application because your current debt-to-income ratio of 82% exceeds our policy maximum of 50%. To qualify, your total monthly debt payments would need to be below $3,541. Consider paying down existing debt or applying for a smaller amount.",
    },
    "auditor": {
        "tone": "precise_and_cited",
        "language": "policy_references_and_calculations",
        "length": "detailed",
        "example": "Application #LN-2026-12345 was denied per Credit Policy 4.2.1. Applicant's debt-to-income ratio calculated at 0.82 (existing debt $45,000 + requested loan $25,000 = $70,000 / annual income $85,000), exceeding the policy threshold of 0.50. Three alternative loan amounts were evaluated ($10k, $5k, $0 requested) and none brought the ratio within policy. Decision is consistent with Policy 4.2.1; no exceptions apply.",
    },
    "regulator": {
        "tone": "formal_and_compliant",
        "language": "regulatory_framework_references",
        "length": "comprehensive",
        "example": "Decision: Loan application denied. Regulatory basis: Equal Credit Opportunity Act (ECOA), Regulation B. Internal policy basis: Credit Policy 4.2.1. Decision criteria: debt-to-income ratio. Specific data points used: income, existing debt, requested loan. Specific algorithm: linear calculation per policy. Right to appeal: applicant may request reconsideration within 30 days per ECOA.",
    },
    "engineer": {
        "tone": "technical_and_debuggable",
        "language": "execution_trace_with_lineage",
        "length": "full_trace",
        "example": "Session: agent-credit-007. Reasoning trace: 4 steps captured. Tools called: postgres.lookup_customer (3 calls), policy.check_eligibility (1 call). Decisions: deny (confidence 0.97). Model: claude-opus-4. Trace ID: trace-abc-123-456.",
    },
}

Different audiences get different explanations. The customer gets clarity; the auditor gets precision; the regulator gets compliance; the engineer gets debug-ability.

Pillar 4: Counterfactual Reasoning (What Would Change the Outcome)

For high-stakes decisions, the explanation includes counterfactuals:

# Counterfactual reasoning
counterfactuals = {
    "decision": "loan_denied",
    "current_state": {"annual_income": 85000, "existing_debt": 45000, "requested_loan": 25000},
    "changes_that_would_approve": [
        {"change": "Reduce requested loan to $5,000", "new_state": {"requested_loan": 5000}, "outcome": "still_denied", "reason": "DTI still 0.59 > 0.50"},
        {"change": "Reduce existing debt to $30,000", "new_state": {"existing_debt": 30000}, "outcome": "still_denied", "reason": "DTI still 0.65 > 0.50"},
        {"change": "Reduce existing debt to $10,000", "new_state": {"existing_debt": 10000}, "outcome": "approved", "reason": "DTI 0.41 < 0.50"},
        {"change": "Increase income to $120,000", "new_state": {"annual_income": 120000}, "outcome": "approved", "reason": "DTI 0.58, but exceeds policy anyway"},
        {"change": "Reduce existing debt AND increase income", "new_state": {"existing_debt": 30000, "annual_income": 120000}, "outcome": "approved", "reason": "DTI 0.46 < 0.50"},
    ],
}

The counterfactuals show what would change the outcome. The customer knows what to do; the regulator knows the decision isn't arbitrary.

The Interpretability Patterns

Several patterns emerge from disciplined interpretability.

Pattern 1: Layered Explanations

Different audiences want different levels of detail. Layered explanations accommodate:

# Layered explanations
layered_explanation = {
    "layer_1_summary": "Loan denied due to high debt-to-income ratio.",
    "layer_2_detail": "Your debt-to-income ratio of 82% exceeds our policy maximum of 50%. To qualify, the ratio must be at or below 50%.",
    "layer_3_calculation": "Monthly debt: $4,167 (existing $3,750 + new $417). Monthly income: $7,083. Ratio: $4,167 / $7,083 = 0.59. Wait, this uses monthly figures. Annual calculation: ($45,000 + $25,000) / $85,000 = 0.82.",
    "layer_4_policy_reference": "Per Credit Policy 4.2.1, maximum allowable debt-to-income ratio is 0.50 for unsecured personal loans. Policy basis: Industry standard for consumer credit risk.",
    "layer_5_alternatives": "If requested loan reduced to $0 (no new debt), ratio = $45,000 / $85,000 = 0.53, still above policy. If existing debt paid down to $10,000, ratio = ($10,000 + $25,000) / $85,000 = 0.41, below policy.",
}

# User selects layer
user_selection = ask_user("How much detail do you want?", options=["Just the summary", "Some detail", "Full calculation", "Policy references", "What would change the decision"])

The layered approach accommodates different user needs. The summary is enough for most; the detail is there for those who want it.

Pattern 2: Confidence Calibration

Decisions include confidence levels; explanations reflect uncertainty:

# Confidence calibration
confidence_calibrated_decision = {
    "decision": "claim_approved",
    "confidence": 0.87,
    "confidence_explanation": "High confidence based on: clear policy coverage, sufficient documentation, no contradicting evidence. Some uncertainty due to: one witness statement is ambiguous, but the preponderance of evidence supports approval.",
    "factors": [
        {"factor": "policy_coverage", "confidence": 0.99, "weight": 0.6},
        {"factor": "documentation_completeness", "confidence": 0.95, "weight": 0.2},
        {"factor": "witness_consistency", "confidence": 0.65, "weight": 0.2},
    ],
    "decision_with_uncertainty": "Approved (87% confidence). If new evidence emerges that contradicts, decision may be revisited.",
}

The calibrated confidence lets stakeholders know when the decision is firm and when it's tentative. The explanation is honest about uncertainty.

Pattern 3: Reasoning Replay (Step-by-Step Walkthrough)

For complex decisions, users can request a step-by-step walkthrough:

# Reasoning replay
reasoning_replay = {
    "decision_id": "dec-abc-123",
    "interactive_replay": True,
    "replay_steps": [
        {"step": 1, "title": "I reviewed your application", "details": "...", "user_can_ask_questions": True},
        {"step": 2, "title": "I checked your debt-to-income ratio", "details": "...", "user_can_ask_questions": True},
        {"step": 3, "title": "I considered alternatives", "details": "...", "user_can_ask_questions": True},
        {"step": 4, "title": "I made the decision", "details": "...", "user_can_ask_questions": True},
    ],
    "user_questions_supported": True,
    "followup_options": [
        "Ask for more detail on any step",
        "Request counterfactual analysis",
        "Request human reviewer",
        "File appeal",
    ],
}

The interactive replay lets users understand the decision in their own time. The followup options preserve agency.

Pattern 4: Bias Detection in Explanations

The discipline checks explanations for bias indicators:

# Bias detection
bias_indicators = {
    "demographic_proxy": {
        "description": "Factors that proxy for protected characteristics",
        "examples": ["zip_code_proxy_for_race", "name_proxy_for_gender"],
        "action": "flag_for_review",
    },
    "disparate_impact": {
        "description": "Decision rates differ significantly across demographic groups",
        "detection": "statistical_test_on_decision_outcomes_by_group",
        "action": "alert_if_disparity_exceeds_threshold",
    },
    "circular_reasoning": {
        "description": "Decision uses the outcome to justify itself",
        "examples": ["denied because high risk" using risk model that predicts denials],
        "action": "flag_for_model_review",
    },
    "missing_factors": {
        "description": "Relevant factors not considered",
        "detection": "compare_explained_factors_to_expected_factors",
        "action": "suggest_additional_review",
    },
}

The bias detection catches explanations that look reasonable but hide problematic reasoning. The agent's decisions are audited for fairness.

Pattern 5: Explanation Quality Monitoring

Explanations are evaluated for quality:

# Explanation quality metrics
quality_metrics = {
    "completeness": {
        "description": "All decision factors are explained",
        "measurement": "explained_factors / actual_factors",
        "target": 0.95,
        "current": 0.92,
    },
    "groundedness": {
        "description": "Explanation matches reasoning trace",
        "measurement": "semantic_similarity(explanation, reasoning_trace)",
        "target": 0.90,
        "current": 0.87,
    },
    "specificity": {
        "description": "Explanation includes specific numbers and references",
        "measurement": "explanations_with_specific_values / total_explanations",
        "target": 0.90,
        "current": 0.94,
    },
    "user_satisfaction": {
        "description": "Users rate explanation as helpful",
        "measurement": "user_feedback_score",
        "target": 0.85,
        "current": 0.78,
    },
}

The quality monitoring surfaces explanation problems. The team improves explanations based on data.

The Interpretability Discipline Doesn't Do

Honest limitations:

• It doesn't make the model interpretable. The discipline explains what the agent did; it doesn't make the model's internal workings transparent. The model is still a black box; the agent's reasoning around the model is captured.
• It doesn't eliminate uncertainty. Some decisions are uncertain. The explanations reflect this honestly. The explanations aren't false certainty.
• It doesn't justify poor decisions. A bad decision with a good explanation is still a bad decision. The discipline ensures explanations are accurate, not that decisions are good.
• It can be complex to set up. Capturing reasoning across all decision types requires engineering. The setup takes effort.
• It doesn't replace human review. Some decisions should still be reviewed by humans. The explanations support review; they don't replace it.

The Interpretability as Operational Practice

Interpretability is operational practice:

Reasoning quality. The team monitors reasoning quality. They look for unclear logic, missing factors, biased reasoning.

Explanation review. Explanations are reviewed for accuracy. The team catches explanations that don't match the reasoning.

User feedback. Users provide feedback on explanations. The team improves based on what users find helpful.

Continuous improvement. The team iterates on reasoning capture and explanation generation. The discipline gets better over time.

The practice is what makes the discipline sustainable. Without it, explanations degrade. With it, explanations stay accurate and useful.

The Compound Effect of Interpretability Discipline

Interpretability discipline compounds:

• Customer trust. Customers trust decisions they understand. The trust is preserved.
• Regulatory compliance. Regulators accept explanations that meet standards. The compliance is achieved.
• Audit readiness. Audits are smoother with good explanations. The audits pass.
• Decision quality. Forced to reason explicitly, agents make better decisions. The decisions improve.
• Continuous learning. Explanations surface patterns. The team learns and improves.

The undisciplined approach has the opposite trajectory. Customer distrust, regulatory penalties, failed audits, opaque decisions, no learning.

Bottom Line

AI agents make decisions that affect people. Without interpretability, the decisions are opaque. With interpretability, the decisions are explainable.

Facio's interpretability discipline provides reasoning capture, factor attribution, audience-appropriate explanations, and counterfactual reasoning. The discipline makes AI agents deployable in high-stakes contexts.

The agent without interpretability is a black box. The agent with it is a transparent decision-maker. The customer prefers the transparent one. The regulator accepts the transparent one.

Because AI agents in production affect real people. The question is whether the people know why. The interpretability discipline is what makes the answer clear.

---

See the interpretability documentation for reasoning capture configuration, factor attribution setup, and explanation templates.