Right-Sized HITL: The Action Classification Engine That Decides What Actually Needs a Human
The HITL industry has a problem. The marketing says "human oversight for AI agents." The implementation says "approve every action." The result: the human is the bottleneck, the agent is the rubber stamp, and nobody is actually catching the things that matter.
The right architecture isn't "every action needs a human." The right architecture is right-sized HITL — proportional oversight that matches the action's actual risk. Most actions should be autonomous. Some should be sampled for quality measurement. A few need synchronous human review. The challenge is mapping the right oversight to the right action — at scale, with consistency, with the right review intensity.
This post is about the action classification engine: the policy layer that evaluates every action the agent proposes, classifies it by risk profile, and routes it to the right oversight pattern. Not all actions are equal. Not all oversight is equal. The classification engine is what makes HITL work at scale.
The Three Oversight Patterns
Before the classification engine, the three patterns it can route to:
Pattern 1: Autonomous (No Human Required)
The action is low-stakes, well-understood, and either fully reversible or has negligible blast radius. The agent executes. The audit trail records. No human is in the loop.
Examples: Reading data. Formatting a response. Querying an internal API. Generating a draft document that is later reviewed. Updating an internal record. Sending a status notification to an internal channel.
Pattern 2: Sampled (5–20% Human Review)
The action is moderate-stakes, somewhat reversible, with potentially visible impact but contained blast radius. The agent executes, but a percentage of actions are routed to a human reviewer for quality measurement. The human's decision is data — used to refine thresholds and detect patterns — not a gate that blocks execution.
Examples: Customer service responses sent to known-good customers. Refunds under a threshold. Internal database updates. Status communications. Standardized reports.
Pattern 3: Synchronous Review (Block Until Approved)
The action is high-stakes, hard to reverse, or has broad blast radius. The agent pauses. The human reviews in real-time. The agent proceeds only on approval.
Examples: Outbound communications to high-value customers. Production infrastructure changes. Permission grants. Data deletion. Financial transactions above threshold. Any action with regulatory implications.
The patterns aren't a sequence. They're three different routes from the same classification engine, applied per-action.
The Classification Engine: How It Decides
The classification engine evaluates every action against a structured set of dimensions. The output is a classification — autonomous, sampled, or synchronous review — plus the routing metadata (which reviewer pool, which timeout, which audit treatment).
Dimension 1: Reversibility
The single most important dimension. Can the action be undone, and at what cost?
- Fully reversible in seconds (toggle a flag, reformat a string): almost always autonomous
- Reversible in minutes, low cost (roll back a database transaction, cancel a scheduled job): usually autonomous or sampled
- Reversible in hours, moderate cost (revert a deployment, refund a payment): sampled or synchronous depending on cost magnitude
- Difficult to reverse, high cost (delete a record, change account ownership, send an email to thousands): synchronous
The reversal cost is the dominant factor. A $10 refund to a customer is reversible in 60 seconds (issue a chargeback) and costs the company ~$15 in fees and goodwill. A $10,000 refund is reversible but the cost is high. The classification engine treats them differently.
Dimension 2: Blast Radius
Who or what is affected by the action?
- Self only (the agent reads or updates its own state): always autonomous
- Single user, internal (one employee, one record): usually autonomous or sampled
- Single user, external (one customer, one record): sampled or synchronous depending on action type
- Multiple users, contained (a team, a customer segment): usually synchronous
- Multiple users, public (a customer base, the public): almost always synchronous
Blast radius compounds with reversibility. A reversible action with small blast radius is low risk. A reversible action with large blast radius is moderate risk. An irreversible action with large blast radius is high risk.
Dimension 3: Financial Impact
What is the dollar value of the action's direct effect?
- Under $25: almost always autonomous or sampled
- $25–$500: sampled (with 100% review for flagged cases)
- $500–$5,000: sampled or synchronous depending on other dimensions
- $5,000+: usually synchronous
The financial threshold isn't universal. A $500 refund to a Fortune 500 customer is different from a $500 refund to a small business. The classification engine can encode customer-specific thresholds.
Dimension 4: Regulatory Implication
Does the action touch a regulated surface?
- No regulatory implication: standard classification
- GDPR / DSGVO / data subject rights: synchronous for irreversible actions
- Financial services regulation: synchronous above the regulator's notification threshold
- Healthcare / HIPAA: synchronous for any action touching PHI
- Industry-specific (FDA, FAA, etc.): synchronous for the regulated action types
The regulatory dimension is the most strict. When a regulator is involved, the action's classification defaults to synchronous regardless of other dimensions.
Dimension 5: Confidence and Risk Signals
The classification engine also considers signals the agent provides:
- The agent's confidence in its decision (low confidence → upgrade to sampled or synchronous)
- The model's risk indicators (anomalous inputs, policy edge cases)
- The pattern of recent similar actions (a sequence of borderline actions → upgrade one notch)
The confidence dimension is a modifier, not a primary dimension. A high-confidence irreversible action is still high-stakes. A low-confidence autonomous action is still autonomous if the action itself is low-stakes.
The Manifest That Encodes the Classification
The classification engine's logic lives in a manifest — the same version-controlled policy file that defines HITL rules for the agent. The manifest specifies, per action type, the dimensions and the resulting classification:
actions:
format_response:
classification: autonomous
rationale: low blast radius, fully reversible
update_internal_record:
classification: sampled
sampling_rate: 0.05
rationale: moderate blast radius, reversible
send_customer_email:
classification: dynamic
dimensions:
blast_radius:
if: customer.tier == "enterprise" then: synchronous
else: sampled
financial_impact:
if: customer.account_value > 50000 then: synchronous
else: sampled
default: sampled
sampling_rate: 0.10
delete_customer_data:
classification: synchronous
rationale: irreversible, regulatory implication
process_refund:
classification: dynamic
dimensions:
amount:
if: amount < 25 then: autonomous
if: amount < 500 then: sampled
if: amount < 5000 then: synchronous
else: synchronous
customer:
if: customer.tenure_days < 30 and amount > 100 then: synchronous
The classification: dynamic lines encode the dimensional logic. The classification engine evaluates the dimensions at action time and produces the right pattern. The same action can route differently for different customers, different amounts, different times of day — based on the dimensions in the manifest.
Why Most HITL Systems Skip This
The classification engine approach is uncommon because most teams don't have it. They have a binary check: "is this action above the threshold?" If yes, human review. If no, autonomous. The binary check is the wrong abstraction.
The binary check fails because:
It treats all high-threshold actions the same. A $501 refund and a $50,000 refund both trigger review. A $501 refund to a known good customer and a $501 refund to a brand-new account both trigger review. The review intensity should be different.
It treats all low-threshold actions the same. A $1 refund and a $499 refund both pass through. But the $499 refund is much closer to the threshold and more likely to be on the wrong side of it. The classification engine would route the $499 to sampled; the binary check would route it to autonomous.
It ignores non-financial dimensions. A non-financial action with high blast radius (sending a public statement, deleting a record) might not cross the financial threshold but should still get synchronous review. The binary check misses it.
It doesn't adapt to the action's context. The same action, in different contexts, has different risk. Sending a $50 email to a customer who's been with the company for 5 years is different from sending a $50 email to a customer who created their account yesterday. The classification engine captures the context; the binary check doesn't.
The Architecture: Action Classification in Practice
The classification engine sits between the agent and the action execution. Every action the agent proposes is submitted to the classification engine first. The engine evaluates the action against the manifest, produces a classification, and routes to the right pattern.
Agent proposes action
↓
Classification Engine
- Reversibility check
- Blast radius check
- Financial impact check
- Regulatory check
- Confidence check
↓
Classification
↓
┌─────┼─────┐
↓ ↓ ↓
Auto Sample Sync Review
↓ ↓ ↓
Execute Audit Review
Trail
The classification engine runs the same evaluation regardless of which agent proposed the action. The same process_refund action has the same classification rules across all agents that propose it. The same delete_customer_data action has the same synchronous review requirements across all agents.
The manifest is the single source of truth. The agent code contains no classification logic. The policy engine enforces the classification. The audit trail records both the classification decision and the resulting routing.
The Metrics That Prove Right-Sizing Works
The classification engine produces the data needed to prove right-sized HITL is working:
Pattern Distribution
What percentage of actions are autonomous vs. sampled vs. synchronous? A healthy distribution has 70–85% autonomous, 10–25% sampled, 5–10% synchronous. A distribution where 80% of actions go to synchronous review is over-applied. A distribution where 99% are autonomous is under-applied.
Sampled Review Findings
What percentage of sampled reviews find an error? If the rate is 0% over 6 months, the sampling rate is too high or the action type should graduate to autonomous. If the rate is 5%+, the action type may need to upgrade to synchronous, or the agent needs improvement.
Synchronous Review Outcomes
What percentage of synchronous reviews are approved without modification? If 95%+ are approved as-is, the synchronous threshold is too low — many of these actions could be sampled. If 50%+ are modified or rejected, the synchronous threshold is appropriate or too lenient.
Override Rate by Pattern
How often do reviewers override the action, by pattern? The override rate for synchronous reviews should be 5–15% (suggests the threshold is calibrated). For sampled reviews, 1–5% (suggests the agent is generally correct but occasionally wrong). For autonomous actions, 0% in the audit trail (the human wasn't involved).
The metrics tell you whether the classification is right. If the metrics show misalignment, the manifest is adjusted. The classification is continuously tuned, not set once and forgotten.
Where Facio Fits
Facio's policy engine is the classification engine. Every action the agent proposes is evaluated against the manifest, classified by the dimensions, and routed to the right pattern. The same manifest supports autonomous, sampled, and synchronous patterns — and the dynamic classification logic that produces different patterns for different contexts.
Placet.io's review interface handles both synchronous and sampled patterns. The synchronous review is the structured approval flow. The sampled review is the same flow, but applied to a percentage of actions with the decision recorded as data. The reviewer doesn't know whether the action is sampled or synchronous — the experience is the same. The audit trail records the pattern.
The combined architecture is right-sized by default. Teams don't have to choose between "everything autonomous" and "everything reviewed." The classification engine produces the right pattern per action, with the right reviewer pool, with the right audit treatment.
Key Takeaways
- Right-sized HITL is the goal, not maximum HITL — most actions should be autonomous, some sampled, a few synchronous
- The classification engine evaluates every action on five dimensions: reversibility, blast radius, financial impact, regulatory implication, confidence
- The manifest encodes the classification logic — per action, per dimension, per routing pattern
- Most HITL systems use a binary threshold — which treats a $501 refund the same as a $50,000 refund, and a $1 refund the same as a $499 refund
- The pattern distribution should be 70-85% autonomous, 10-25% sampled, 5-10% synchronous — a different distribution suggests over- or under-application
- The metrics prove right-sizing works — pattern distribution, sampled review findings, synchronous review outcomes, override rate by pattern
- Facio's policy engine is the classification engine — same manifest, same evaluation, same routing, right-sized for every action
Sources: The right-sized HITL framework draws on risk-based compliance patterns (Basel III operational risk, FDA medical device classification), proportional review principles from financial audit and quality assurance, and production patterns from HITL deployments at scale. The dimensional classification model reflects established risk assessment methodology adapted for AI agent action surfaces.