AI agents evolve. New capabilities get added. New tools get integrated. New behaviors get tested. The naive approach ships the new capability to everyone at once. The new capability has a bug. The bug affects every customer. The damage is large.
Facio's feature flag discipline gives AI agents structured mechanisms to roll out new capabilities safely. The flag starts off for everyone. The flag enables for internal testing. The flag enables for a percentage of traffic. The flag enables for specific customer segments. The flag enables for everyone. The flag is rolled back if needed.
Here's how the discipline works, what feature flagging it includes, and why feature flag discipline is what makes AI agents evolve in production without breaking trust.
The Feature Flag Reality
Production AI agents face a feature rollout problem:
Problem 1: All-or-nothing rollout. A new capability is enabled for everyone. The capability has a bug. Every customer is affected. The blast radius is huge.
Problem 2: No rollback mechanism. A capability is deployed and the rollback is complex. Rolling back takes hours. The damage continues.
Problem 3: No internal testing in production. A capability is tested in staging. Staging is different from production. The capability fails in production. The team is surprised.
Problem 4: No gradual rollout. A capability is rolled out 100%. The team can't tell if a regression is from the new capability or other changes.
Problem 5: No targeted rollout. A capability works for some customer segments but not others. The team can't enable it just for the working segments.
Problem 6: No kill switch. A capability is misbehaving. The team has no immediate way to disable it. The incident continues.
The naive approach — deploy and hope — fails every test. The team ends up with a system where every change is a high-stakes gamble.
The Feature Flag Discipline
Facio's feature flag discipline has five pillars. Each addresses a different aspect of safe rollout.
Pillar 1: Flag Definition and Configuration (Declare the Flag)
Flags are defined with explicit configuration:
# Flag definition and configuration
flag_definition = {
"flag_id": "flag-new-refund-flow",
"flag_name": "New Refund Flow",
"description": "Enable new multi-step refund flow with verification",
"flag_type": "boolean",
"default_value": False,
"owner": "team-payments",
"created_at": "2026-08-01T10:00:00Z",
"expiration_date": "2026-09-01T10:00:00Z",
"tags": ["payments", "experimental", "high_stakes"],
"metadata": {
"jira_ticket": "PAY-1234",
"design_doc": "https://docs.internal/new-refund-flow",
"risk_level": "high",
},
"rollout_plan": {
"phase_1_internal_testing": {"start": "2026-08-03", "end": "2026-08-05", "audience": "internal_employees"},
"phase_2_canary": {"start": "2026-08-06", "end": "2026-08-08", "audience": "canary_customers"},
"phase_3_percentage": {"start": "2026-08-09", "end": "2026-08-15", "audience": "5pct_then_25pct_then_50pct"},
"phase_4_full_rollout": {"start": "2026-08-16", "audience": "all_customers"},
},
}
def define_flag(flag_config):
validate_flag_config(flag_config)
flag_store.create(flag_config)
notify_flag_owner(flag_config["owner"], "Flag created")
return flag_config
The flag definition is explicit. The rollout plan is documented.
Pillar 2: Flag Evaluation (Decide Per Request)
Flags are evaluated per request based on context:
# Flag evaluation
flag_evaluation = {
"evaluation_context": {
"tenant_id": "tenant-acme-corp",
"user_id": "user-jane-doe",
"session_id": "session-abc-123",
"request_id": "req-xyz-789",
"user_attributes": {"tier": "enterprise", "region": "EU", "signup_date": "2025-01-15"},
"request_attributes": {"endpoint": "/api/refund", "method": "POST"},
"time": "2026-08-03T10:23:45Z",
},
"evaluation_strategies": {
"boolean": {
"description": "Simple true/false",
"use_case": "basic_feature_toggle",
"example": "flag_value = True",
},
"percentage_rollout": {
"description": "Percentage of traffic gets enabled",
"use_case": "gradual_rollout",
"implementation": "hash(user_id) % 100 < rollout_percentage",
"example": "5% of users enabled",
},
"tenant_allowlist": {
"description": "Specific tenants enabled",
"use_case": "targeted_rollout_to_business_critical_customers",
"example": "tenant_id in [tenant-acme-corp, tenant-globex-inc]",
},
"user_attribute_match": {
"description": "Enable based on user attributes",
"use_case": "tier_based_or_region_based_rollout",
"example": "user.tier == 'enterprise' AND user.region == 'EU'",
},
"time_based": {
"description": "Enable during specific time windows",
"use_case": "scheduled_rollouts",
"example": "enable between 2026-08-09 and 2026-08-16",
},
"kill_switch": {
"description": "Always-enabled flag for emergency disable",
"use_case": "rapid_response_to_incidents",
"example": "flag_value = True (kill_switch active)",
},
},
"evaluation_priority": [
"kill_switch_if_active",
"explicit_tenant_allowlist",
"percentage_rollout",
"user_attribute_match",
"default_value",
],
"evaluation_logging": {
"every_evaluation": "logged_with_context_AND_result",
"evaluation_result": "True_OR_False",
"evaluation_reason": "why_this_result",
"audit_trail": "all_evaluations_for_compliance",
},
}
def evaluate_flag(flag_id, context):
flag = flag_store.get(flag_id)
if flag.is_kill_switch_active():
return FlagResult(False, "kill_switch_active")
if flag.has_explicit_allowlist() and context.tenant_id in flag.allowlist:
return FlagResult(True, "explicit_allowlist")
if flag.has_percentage_rollout() and hash_to_bucket(context.user_id) < flag.percentage:
return FlagResult(True, "percentage_rollout")
if flag.matches_user_attributes(context):
return FlagResult(True, "user_attribute_match")
return FlagResult(flag.default_value, "default")
The flag evaluation is per-request. The context determines the result. Every evaluation is logged.
Pillar 3: Rollout Strategies (How to Enable)
Multiple rollout strategies enable gradual deployment:
# Rollout strategies
rollout_strategies = {
"internal_testing": {
"description": "Enable for internal team first",
"audience": "employees_only",
"duration_days": 2,
"use_case": "early_feeding_and_bug_finding",
"monitoring": "every_call_logged_AND_reviewed",
"rollback_trigger": "any_critical_bug",
},
"canary_customers": {
"description": "Enable for opt-in early customers",
"audience": "canary_customers_list",
"duration_days": 3,
"use_case": "real_customer_testing_before_wider_rollout",
"monitoring": "track_metrics_compare_to_control",
"rollback_trigger": "metric_regression_OR_critical_bug",
},
"percentage_rollout": {
"description": "Gradual percentage increase",
"stages": ["5%", "25%", "50%", "100%"],
"duration_days_per_stage": 3,
"use_case": "broad_rollout_with_safety",
"monitoring": "compare_metrics_to_baseline",
"rollback_trigger": "regression_detected",
},
"tenant_based_rollout": {
"description": "Enable for specific tenant tiers or regions",
"audience": "tier_enterprise_FIRST_THEN_standard",
"use_case": "enterprise_first_then_others",
"monitoring": "per_tenant_metrics",
"rollback_trigger": "enterprise_critical_issue",
},
"dark_launch": {
"description": "Run new code without exposing to customers",
"audience": "code_runs_but_results_not_used",
"use_case": "test_new_code_path_without_user_impact",
"monitoring": "shadow_metrics_only",
"rollback_trigger": "not_applicable_dark_launch",
},
"regional_rollout": {
"description": "Enable for specific regions first",
"audience": "region_EU_FIRST_THEN_US",
"use_case": "regulatory_or_capacity_reasons",
"monitoring": "per_region_metrics",
"rollback_trigger": "regional_issue",
},
}
def execute_rollout(flag_id, strategy):
if strategy == "internal_testing":
enable_for_internal(flag_id)
elif strategy == "canary_customers":
enable_for_canary(flag_id)
elif strategy == "percentage_rollout":
start_percentage_rollout(flag_id, stages=["5", "25", "50", "100"])
elif strategy == "tenant_based_rollout":
enable_for_tenant_tier(flag_id, "enterprise")
elif strategy == "dark_launch":
enable_in_shadow_mode(flag_id)
elif strategy == "regional_rollout":
enable_for_region(flag_id, "EU")
The rollout strategies enable gradual deployment. The team controls the pace.
Pillar 4: Rollback and Kill Switch (Disable Quickly)
Flags can be disabled quickly:
# Rollback and kill switch
rollback_kill_switch = {
"rollback_triggers": {
"metric_regression": {
"description": "Key metrics regress significantly",
"examples": ["error_rate_up_50%", "latency_p95_up_2x", "customer_satisfaction_down"],
"automatic": "rollback_when_threshold_exceeded",
},
"critical_bug": {
"description": "Critical bug discovered",
"examples": ["data_corruption", "security_vulnerability", "customer_facing_error"],
"automatic": "manual_rollback_via_incident_response",
},
"customer_complaint": {
"description": "Customers complain about new capability",
"examples": ["multiple_complaints_in_short_window", "NPS_drop_for_flagged_users"],
"automatic": "rollback_when_threshold_exceeded",
},
"scheduled_rollback": {
"description": "Rollback based on schedule",
"examples": ["rollout_period_ended", "phase_complete_but_next_phase_decision_pending"],
"automatic": "scheduled_rollback_at_specific_time",
},
},
"rollback_process": {
"automatic_rollback": {
"description": "System rolls back based on triggers",
"trigger_condition": "metric_threshold_exceeded",
"rollback_action": "disable_flag_AND_alert_team",
"rollback_speed": "within_5_minutes",
},
"manual_rollback": {
"description": "Operator rolls back manually",
"trigger_condition": "operator_decision",
"rollback_action": "disable_flag_AND_alert_team",
"rollback_speed": "within_1_minute",
},
"kill_switch_rollback": {
"description": "Emergency kill switch",
"trigger_condition": "critical_incident",
"rollback_action": "force_disable_flag_immediately",
"rollback_speed": "within_5_seconds",
},
},
"rollback_safety": {
"validate_rollback": "verify_rollback_completed",
"verify_old_behavior": "verify_old_behavior_active",
"no_data_loss": "rollback_does_not_corrupt_state",
"customer_experience": "customer_does_not_see_broken_state",
},
}
def rollback_flag(flag_id, reason, rollback_type="manual"):
if rollback_type == "kill_switch":
flag_store.set_kill_switch(flag_id, True)
elif rollback_type == "manual":
flag_store.set_value(flag_id, False)
elif rollback_type == "automatic":
flag_store.set_value(flag_id, False)
log_rollback(flag_id, reason, rollback_type)
notify_team(flag_id, reason, rollback_type)
verify_rollback_completed(flag_id)
return {"status": "rolled_back", "flag_id": flag_id}
The rollback and kill switch provide safety nets. The team can disable any capability in seconds.
Pillar 5: Flag Lifecycle and Cleanup (Don't Accumulate)
Flags have a lifecycle and are cleaned up after rollout:
# Flag lifecycle and cleanup
flag_lifecycle = {
"lifecycle_stages": {
"definition": {
"description": "Flag is defined with configuration",
"duration_days": 1,
"deliverables": ["config_set", "rollout_plan", "owner_assigned"],
},
"rollout": {
"description": "Flag is being rolled out",
"duration_days": 14,
"deliverables": ["internal_testing", "canary", "percentage_rollout", "full_rollout"],
"tracking": "rollout_progress_AND_metrics",
},
"stable": {
"description": "Flag is fully rolled out and stable",
"duration_days": 30,
"deliverables": ["monitoring_metrics", "performance_baseline", "ownership_documented"],
"transition_trigger": "team_confirms_stable_for_30_days",
},
"cleanup": {
"description": "Flag is removed and code is simplified",
"duration_days": 7,
"deliverables": ["remove_flag_evaluation", "remove_dead_code", "update_documentation"],
"tracking": "flag_removal_pr_referenced",
},
},
"cleanup_policies": {
"automatic_cleanup_reminder": "remind_team_after_30_days_of_stable",
"expiration_date_enforcement": "force_cleanup_after_expiration",
"ownership_enforcement": "flag_without_owner_marked_for_cleanup",
"audit_trail": "every_lifecycle_transition_recorded",
},
"cleanup_actions": [
"remove_flag_evaluation_from_code",
"remove_old_behavior_code_path",
"update_documentation",
"notify_team_of_cleanup_completion",
],
}
def cleanup_flag(flag_id):
flag = flag_store.get(flag_id)
if not is_safe_to_cleanup(flag):
return {"status": "not_safe", "reason": get_unsafe_reason(flag)}
remove_flag_evaluation_from_code(flag_id)
remove_old_behavior_code(flag_id)
update_documentation(flag_id)
log_cleanup(flag_id)
return {"status": "cleaned_up", "flag_id": flag_id}
The flag lifecycle ensures flags don't accumulate. The cleanup prevents technical debt.
The Feature Flag Patterns
Several patterns emerge from disciplined feature flagging.
Pattern 1: Flag-Tied Code Paths
Code is structured around flag decisions:
# Flag-tied code paths
flag_code_patterns = {
"pattern_types": {
"if_else_flag": {
"description": "Simple if/else around flag",
"code_example": "if flag_enabled('new_refund_flow'): return new_refund_flow(...) else: return old_refund_flow(...)",
"use_case": "simple_feature_toggle",
"drawback": "code_duplication",
},
"strategy_pattern": {
"description": "Strategy pattern with flag-selected strategy",
"code_example": "strategy = get_strategy('refund_strategy', flag_context); return strategy.execute(...)",
"use_case": "complex_feature_variants",
"drawback": "strategy_complexity",
},
"configuration_injection": {
"description": "Configuration injected based on flag",
"code_example": "config = get_config(flag_context); return process_with_config(config)",
"use_case": "configuration_driven_features",
"drawback": "config_complexity",
},
"decorator_pattern": {
"description": "Decorator wraps function with flag check",
"code_example": "@require_flag('new_refund_flow') def refund(): ... ",
"use_case": "clean_flag_checking",
"drawback": "decorator_overhead",
},
},
"flag_checking_best_practices": {
"check_once": "check_flag_at_request_start_not_in_loops",
"cache_evaluation": "cache_flag_result_for_request_lifetime",
"consistent_within_request": "same_flag_result_throughout_request",
"log_evaluations": "log_evaluations_for_audit",
},
}
The code patterns ensure flags are integrated cleanly. The patterns prevent spaghetti code.
Pattern 2: Flag-Aware Observability
Observability is flag-aware:
# Flag-aware observability
flag_observability = {
"metrics": {
"flag_evaluation_count": "tracks_how_often_flag_is_evaluated",
"flag_evaluation_distribution": "tracks_true_vs_false_distribution",
"flag_impact_on_metrics": "compares_metrics_with_flag_true_vs_false",
"flag_rollout_progress": "tracks_rollout_stage_completion",
"flag_age": "tracks_how_long_flag_has_existed",
"flag_owner_distribution": "tracks_flags_per_owner",
},
"alerting": {
"metric_regression": "alert_when_metrics_diverge_for_flagged_traffic",
"evaluation_anomaly": "alert_when_flag_evaluations_show_unexpected_pattern",
"cleanup_overdue": "alert_when_flag_cleanup_overdue",
"ownership_gaps": "alert_when_flag_has_no_owner",
},
"dashboards": {
"flag_overview": "shows_all_flags_AND_their_states",
"rollout_progress": "shows_per_flag_rollout_progress",
"impact_analysis": "shows_metrics_comparison_for_each_flag",
},
}
def emit_flag_metrics(flag_id, evaluation_result, context):
metric_registry.counter("flag.evaluation", tags={"flag_id": flag_id, "result": evaluation_result.value})
if should_track_impact(flag_id):
emit_impact_metrics(flag_id, evaluation_result, context)
The flag-aware observability surfaces flag impact. The team understands what flags do.
Pattern 3: Flag-Based Testing
Tests are flag-aware:
# Flag-based testing
flag_testing = {
"test_types": {
"flag_evaluation_tests": {
"description": "Verify flag evaluation logic",
"examples": [
"test_percentage_rollout_returns_true_for_included_users",
"test_tenant_allowlist_returns_true_for_allowlisted_tenants",
"test_kill_switch_overrides_all_other_logic",
"test_default_value_returned_when_no_rule_matches",
],
},
"behavior_tests_for_each_flag_state": {
"description": "Verify behavior with flag true and false",
"examples": [
"test_refund_works_with_new_flow_enabled",
"test_refund_works_with_old_flow_enabled",
"test_refund_handles_both_flows_correctly",
],
},
"rollout_progression_tests": {
"description": "Verify rollout progression logic",
"examples": [
"test_rollout_starts_at_5_percent",
"test_rollout_progresses_to_25_percent_after_threshold",
"test_rollout_pauses_when_metrics_regress",
],
},
"rollback_tests": {
"description": "Verify rollback mechanism",
"examples": [
"test_automatic_rollback_triggers_on_metric_threshold",
"test_manual_rollback_disables_flag_immediately",
"test_kill_switch_overrides_all_flags",
],
},
},
"test_automation": {
"test_in_ci": "all_flag_tests_run_in_continuous_integration",
"test_with_real_flag_config": "use_test_flag_config_for_realistic_testing",
"test_with_disabled_flag": "verify_old_behavior_still_works",
},
}
def test_flag_evaluation():
context = make_test_context(user_id="user-test-001", tenant_id="tenant-test")
flag = make_test_flag(rollout_percentage=50)
result_1 = evaluate_flag(flag, context)
assert result_1 in [True, False]
The flag-based testing verifies flag logic. The tests catch rollout issues.
Pattern 4: Flag-Based Incident Response
Incidents use flag rollback as primary response:
# Flag-based incident response
flag_incident_response = {
"incident_response_workflow": {
"incident_detected": {
"description": "Production issue detected",
"first_response": "check_recent_flag_changes",
"check_action": "review_flags_changed_in_last_24_hours",
"decision": "rollback_recent_flag_OR_investigate_other_causes",
},
"rollback_decision": {
"description": "Decide to rollback flag",
"decision_criteria": ["incident_correlates_with_flag_change", "rollback_safe_to_perform", "rollback_likely_resolves_incident"],
"rollback_speed": "target_under_5_minutes",
},
"rollback_execution": {
"description": "Execute rollback",
"actions": ["disable_flag", "verify_old_behavior_restored", "verify_metrics_improving"],
},
"post_rollback": {
"description": "Post-rollback actions",
"actions": ["monitor_metrics", "communicate_to_team", "investigate_root_cause", "plan_fix"],
},
},
"rollback_runbook": {
"step_1": "identify_flag_to_rollback",
"step_2": "verify_rollback_will_not_cause_data_loss",
"step_3": "execute_rollback_via_flag_system",
"step_4": "verify_old_behavior_active",
"step_5": "monitor_metrics_for_improvement",
"step_6": "communicate_to_stakeholders",
"step_7": "document_post_mortem",
},
}
def incident_response_rollback(flag_id, incident_context):
log_rollback_decision(flag_id, incident_context)
rollback_flag(flag_id, "incident_response", "manual")
verify_rollback(flag_id)
notify_team(flag_id, incident_context)
monitor_metrics(flag_id, duration_minutes=30)
return {"status": "rolled_back", "flag_id": flag_id}
The flag-based incident response uses rollback as the primary lever. The incident response is fast.
Pattern 5: Flag Documentation and Ownership
Flags are documented with owners:
# Flag documentation and ownership
flag_documentation = {
"required_documentation": [
"flag_id",
"flag_name",
"description",
"owner_team",
"owner_contact",
"created_date",
"expiration_date",
"rollout_plan",
"rollback_plan",
"risk_assessment",
"metrics_to_monitor",
],
"ownership_principles": {
"every_flag_has_owner": "no_anonymous_flags",
"owner_responsible_for_lifecycle": "owner_manages_full_lifecycle",
"owner_responsible_for_rollback": "owner_can_rollback_quickly",
"owner_responsible_for_cleanup": "owner_cleans_up_after_completion",
},
"documentation_storage": {
"centralized_flag_registry": "all_flags_listed_in_one_place",
"searchable_metadata": "find_flags_by_owner_or_purpose",
"automated_documentation_check": "CI_fails_if_flag_lacks_documentation",
"audit_log_of_changes": "every_documentation_change_logged",
},
}
def require_documentation(flag_config):
required_fields = ["flag_id", "description", "owner", "rollout_plan", "rollback_plan"]
missing = [f for f in required_fields if not flag_config.get(f)]
if missing:
raise IncompleteFlagDocumentationError(missing)
return flag_config
The flag documentation ensures flags are manageable. The ownership provides accountability.
The Feature Flag Discipline Doesn't Do
Honest limitations:
- It doesn't prevent all rollout issues. Flags are tools, not solutions. The discipline requires judgment.
- It adds complexity. Flag-tied code is more complex. The discipline requires maintenance.
- It can mask technical debt. Old flags accumulate if not cleaned up. The discipline requires cleanup.
- It can split testing surface. Multiple flag states multiply test combinations. The discipline requires testing.
- It depends on flag system reliability. If flag system is down, behavior is unpredictable. The discipline requires redundancy.
The Feature Flag Discipline as Operational Practice
Feature flag discipline is operational practice:
Flag review. The team reviews flags weekly. They identify stale flags.
Rollout monitoring. The team monitors rollouts. They pause on regressions.
Cleanup scheduling. The team schedules cleanups. They prevent accumulation.
Owner accountability. The team ensures every flag has an owner. The owners are accountable.
The practice is what makes the discipline sustainable. Without it, flags accumulate. With it, flags are managed.
The Compound Effect of Feature Flag Discipline
Feature flag discipline compounds:
- Lower risk rollout. New capabilities are tested safely. The risk is contained.
- Faster incident response. Rollback is immediate. The incidents are smaller.
- Better customer trust. Customers aren't affected by bugs. The trust is maintained.
- Higher team confidence. The team ships more often. The cadence is faster.
- Lower technical debt. Flags are cleaned up. The debt is reduced.
The undisciplined approach has the opposite trajectory. High-risk rollout, slow incident response, broken trust, low team confidence, technical debt.
Bottom Line
AI agents evolve. New capabilities ship. Without feature flag discipline, the changes risk everything. With feature flag discipline, the changes are safe.
Facio's feature flag discipline provides flag definition and configuration, per-request evaluation, rollout strategies, rollback and kill switch, and flag lifecycle management. The discipline makes AI agent evolution safe.
The team without feature flag discipline is afraid to ship. The team with it ships often. The team trusts the safe one. The customers trust the safe one.
Because AI agents in production must evolve. The question is whether the evolution is safe or risky. The feature flag discipline is what makes the answer safe.
See the feature flag documentation for flag configuration, rollout strategies, and rollback procedures.