Facio's Evaluation Harness: How AI Agent Quality Stays Measurable Instead of Vibes
"The agent feels slower today." "The agent seems to be giving worse answers." "The agent used to handle this edge case." These are real signals, but they're not actionable. Without measurement, the team can't quantify the regression, can't identify the cause, can't validate the fix, can't prevent recurrence.
Software engineering solved this decades ago. Code has tests. Tests run on every change. When a test fails, the team knows exactly what broke. The team has confidence to ship because tests catch regressions before users do.
AI agents need the same discipline. The agent has golden test sets — known inputs with known-good outputs. The agent's current behavior is compared against the golden sets. When the comparison fails, the team knows the regression. When the comparison passes, the team has confidence to ship.
Facio's evaluation harness turns agent quality from vibes into metrics. The team has golden test sets, automated regression detection, prompt-version comparison, model-version comparison, and quality dashboards. The harness runs continuously, catches regressions early, and gives the team the data to improve.
Here's how the harness works, what it measures, and why measurement is what makes AI agent quality a discipline instead of a hope.
The AI Quality Vibes Problem
AI agent quality is hard to measure because the agent's output is non-deterministic. The same input can produce different outputs across runs. The agent's output is subjective (was the response "good"?). The agent's behavior depends on many factors (prompt, model, context, temperature, tools, randomness).
The naive approach is to spot-check the agent. A team member runs the agent through a few examples, looks at the outputs, decides if it's "good enough." This works for a prototype. It doesn't work for production.
The spot-check has four structural problems:
Problem 1: Sampling is unreliable. A 5% regression is invisible in a 20-example spot-check. The team doesn't see the regression until users complain.
Problem 2: The reviewer is biased. Different team members have different standards. What one person considers "good," another considers "mediocre." The measurement is inconsistent.
Problem 3: The reviewer is busy. Manual review doesn't scale. When the team is shipping features, manual review falls behind. The agent's quality drifts without anyone noticing.
Problem 4: The review is retroactive. Manual review happens after changes ship. The team catches regressions post-deployment. Users experience the regression before the team does.
The result is agent quality that drifts unpredictably, with regressions caught late, fixes untested, and improvements unmeasured. The team's confidence in the agent is vibes, not data.
The Evaluation Harness Discipline
Facio's evaluation harness discipline has five pillars. Each addresses a different aspect of quality measurement.
Pillar 1: Golden Test Sets
The team curates golden test sets — known inputs with known-good expected outputs. The harness runs the agent against the golden sets and compares actual outputs to expected:
# Golden test set definition
golden_set = {
"name": "customer-support-v3",
"version": "2026-07-10",
"test_cases": [
{
"id": "tc-001",
"input": {
"user_message": "Where is my order #12345?",
"context": {"customer_id": "cust-789", "language": "en"}
},
"expected": {
"contains_order_status": True,
"uses_correct_customer_data": True,
"no_hallucinated_tracking_number": True,
"response_length_words": {"min": 30, "max": 150},
"tone": "professional_friendly",
},
"tags": ["order_status", "english"],
"priority": "high"
},
{
"id": "tc-002",
"input": {
"user_message": "Ich möchte meine Bestellung stornieren",
"context": {"customer_id": "cust-790", "language": "de"}
},
"expected": {
"correctly_handles_cancellation": True,
"asks_required_clarification": True,
"response_in_german": True,
},
"tags": ["cancellation", "german"],
"priority": "high"
},
# ... hundreds more
]
}
The golden set is curated. The team selects test cases that represent important scenarios, edge cases, and common patterns. The golden set evolves as the agent's capabilities evolve.
Pillar 2: Automated Evaluation Metrics
The harness evaluates the agent's outputs against the expected using multiple metrics:
# Evaluation metrics
metrics = {
"exact_match": {
"description": "Output exactly matches expected",
"applies_to": ["structured_outputs", "short_answers"]
},
"contains_keywords": {
"description": "Output contains expected keywords",
"applies_to": ["narrative_responses"]
},
"semantic_similarity": {
"description": "Output is semantically similar to expected",
"method": "embedding_cosine_similarity",
"threshold": 0.85
},
"schema_compliance": {
"description": "Output matches the expected schema",
"applies_to": ["structured_outputs"]
},
"constraint_satisfaction": {
"description": "Output satisfies declared constraints",
"applies_to": ["all"]
},
"judge_model_evaluation": {
"description": "A judge model evaluates output quality",
"applies_to": ["subjective_quality"],
"judge_prompt": "Rate the response on helpfulness, accuracy, and tone. 1-5 scale."
}
}
The harness applies the appropriate metrics to each test case. The metrics are deterministic (when possible) or use judge models with calibrated rubrics (when subjective).
Pillar 3: Regression Detection
The harness runs continuously and detects regressions automatically:
# Regression detection
regression_check = {
"baseline": "main_branch_quality_score",
"current": "candidate_branch_quality_score",
"regression_threshold_pct": 2.0,
"alert_on_regression": True,
}
# Workflow:
# 1. Run golden set against baseline (main branch)
# 2. Run golden set against candidate (PR branch)
# 3. Compare scores
# 4. If candidate is more than 2% worse: flag as regression
# 5. If candidate is more than 2% better: flag as improvement
# 6. Generate report with per-test-case deltas
The regression detection catches quality changes before they ship. The team sees the regression in their PR review, not in user complaints.
Pillar 4: Version Comparison
The harness compares quality across versions of the agent:
# Version comparison
versions_to_compare = [
{"name": "v3.2", "prompt_version": "v3.2", "model": "claude-opus-4", "commit": "abc123"},
{"name": "v3.3", "prompt_version": "v3.3", "model": "claude-opus-4", "commit": "def456"},
{"name": "v3.3-mini", "prompt_version": "v3.3", "model": "gpt-4o-mini", "commit": "def456"},
]
# Results
comparison = {
"v3.2": {"overall_score": 0.87, "test_cases_passed": 92, "test_cases_failed": 8},
"v3.3": {"overall_score": 0.91, "test_cases_passed": 96, "test_cases_failed": 4},
"v3.3-mini": {"overall_score": 0.85, "test_cases_passed": 89, "test_cases_failed": 11},
}
# Insight: v3.3 improves quality by 4 points. v3.3-mini reduces cost by 80% but loses 2 points.
# Decision: ship v3.3 for premium tier, ship v3.3-mini for cost-sensitive tier
The version comparison enables informed decisions. The team sees the trade-offs between prompt versions, model versions, configuration changes. The decisions are data-driven.
Pillar 5: Quality Dashboards
The harness publishes quality metrics to dashboards:
# Quality dashboard
dashboard = {
"overall_score_trend": [
{"date": "2026-07-01", "score": 0.87},
{"date": "2026-07-02", "score": 0.87},
{"date": "2026-07-03", "score": 0.86},
{"date": "2026-07-04", "score": 0.88},
# ... daily scores
],
"by_category": {
"order_status": 0.94,
"cancellation": 0.91,
"refund": 0.83,
"technical_support": 0.79,
},
"by_language": {
"english": 0.91,
"german": 0.87,
"french": 0.84,
},
"by_complexity": {
"simple": 0.96,
"moderate": 0.88,
"complex": 0.74,
},
"regressions_last_30_days": 3,
"improvements_last_30_days": 7,
"test_case_coverage": 0.85,
}
The dashboard shows quality trends, weak areas, improvement opportunities. The team uses the dashboard to prioritize work.
The Evaluation Harness Patterns
Several patterns emerge from disciplined evaluation harness use.
Pattern 1: Continuous Evaluation
The harness runs on every change:
# CI/CD integration
on_pull_request:
- run_golden_set(scope="changed_test_cases")
- compare_to_main()
- comment_on_pr_with_results()
on_merge_to_main:
- run_full_golden_set()
- update_quality_dashboard()
- alert_if_regression_detected()
on_schedule(daily):
- run_full_golden_set()
- update_quality_dashboard()
- detect_quality_drift()
The continuous evaluation catches regressions at the moment they're introduced. The fix is easier when the regression is fresh.
Pattern 2: Targeted Test Set Selection
The harness runs the full golden set on schedule but targeted test cases on PRs:
# Targeted test selection for PR
pr_changes = ["refund handling logic"]
relevant_test_cases = filter_by_tag(test_cases, tags=["refund"])
run_golden_set(scope=relevant_test_cases)
# Faster feedback for the PR; full evaluation runs separately
The targeted selection balances speed and coverage. PRs get fast feedback; main branch gets full evaluation.
Pattern 3: Test Case Evolution
The golden set evolves as the agent's capabilities and use cases evolve:
# Test case evolution workflow
1. Identify gap: a real production issue not covered by golden set
2. Add test case: capture the input and expected behavior
3. Validate: verify the test case catches the issue (run against broken version)
4. Promote: add to golden set with appropriate tags and priority
5. Monitor: track test case over time
# Test case lifecycle: new → validated → stable → candidate_for_removal → removed
The evolution ensures the golden set stays relevant. New failure modes get captured; obsolete test cases get removed.
Pattern 4: Judge Model Calibration
For subjective evaluations, judge models need calibration:
# Judge model calibration
calibration_set = [
{"output": "...", "human_rating": 4, "judge_rating": 4}, # Match
{"output": "...", "human_rating": 5, "judge_rating": 3}, # Judge undershot
{"output": "...", "human_rating": 2, "judge_rating": 4}, # Judge overshot
]
# Calibration process:
# 1. Have humans rate 50-100 outputs
# 2. Have judge model rate the same outputs
# 3. Compute correlation between human and judge ratings
# 4. Adjust judge prompt until correlation is high (target: >0.85)
# 5. Periodically re-calibrate
The calibration ensures the judge's ratings align with human judgment. The team trusts the harness's subjective evaluations.
The Evaluation Harness as a Feedback Loop
The harness enables a continuous improvement loop:
1. Ship version with quality score 0.87
2. Identify weak area: technical support (score 0.79)
3. Improve prompt or model for technical support
4. Test improvement against harness
5. New score: 0.84 for technical support, 0.89 overall
6. Ship improved version
7. Continue identifying weak areas
The loop is data-driven. The team knows what's weak, fixes it, validates the fix, ships it. The improvements compound.
Without the harness, the loop is vibes-driven. The team thinks something is wrong, makes changes, hopes it helped. The improvements are luck.
The Evaluation Harness Doesn't Do
Honest limitations:
- It doesn't measure what it doesn't test. The golden set is a sample. Real-world inputs may differ from the test cases. The harness measures coverage of the golden set; it doesn't measure coverage of reality.
- It doesn't replace human judgment. Even with judge models calibrated to humans, some outputs need human review. The harness automates what it can; humans handle the rest.
- It can't measure emergent properties. Some agent behaviors emerge from complex interactions the harness doesn't model. The team has to look for these manually.
- It can be gamed. Optimizing for the metrics can degrade other qualities. The team has to balance the metrics they optimize against the qualities the metrics don't capture.
- It requires maintenance. Golden sets need updating. Judge models need recalibration. Metrics need adjustment. The harness is a living system.
The Evaluation Harness as a Cultural Anchor
The harness is more than tooling; it's a cultural anchor:
Quality is measurable. The team commits to measuring quality, not hoping for it. Discussions about quality are backed by data.
Regressions are unacceptable. When the harness flags a regression, the team fixes it before shipping. Quality drops aren't tolerated.
Improvements are validated. When the team makes changes, they validate with the harness. Improvements are proven, not claimed.
Trade-offs are explicit. When the team trades quality for cost (or vice versa), the trade-off is visible. The decisions are informed.
The culture is what makes the discipline sustainable. The team holds itself to the harness's standards. Quality becomes a discipline.
Bottom Line
AI agent quality is hard to measure because outputs are non-deterministic and often subjective. Without measurement, quality drifts and regressions ship.
Facio's evaluation harness turns quality into metrics: golden test sets with hundreds of cases, automated evaluation with six metric types, regression detection on every PR, version comparison across configurations, and quality dashboards. The harness runs continuously, catches regressions early, and enables data-driven improvement.
The agent without a harness is judged by vibes. The agent with a harness is judged by metrics. The team trusts the metrics; the vibes are just anecdotes.
Because AI agents in production need to be reliable. Reliability requires measurement. The harness is what makes the measurement possible.
See the evaluation harness documentation for golden set curation, metric configuration, and dashboard setup.