Facio's Versioned Prompts: How AI Agent Behavior Stays Reproducible Through Model and Code Changes

An AI agent's behavior comes from three sources working together: the prompt that defines the agent's role and instructions, the model that interprets the prompt and generates responses, and the code that orchestrates the agent's tool calls, reasoning, and output formatting. Change any of these three and the agent's behavior changes. Sometimes the change is intended. Often it's not.

The team upgrades the model from GPT-4 to Claude Opus — the agent's responses shift subtly. A previously crisp rejection becomes verbose. A previously concise summary now includes opinions. The agent's "voice" changed. The team didn't notice until customers complained.

The team tweaks the prompt to fix a bug — "the agent was hallucinating order numbers" — and the fix breaks three other things. The agent now refuses valid requests. The agent now skips a tool call it should make. The agent now produces output that fails downstream validation. The team didn't notice until they looked at metrics.

The team refactors the agent code to clean up technical debt — and the agent stops working entirely. The same prompt and model produce different behavior because the code that feeds them changed. The team didn't notice until the golden test suite started failing.

Reproducibility collapses. The team can't reliably compare yesterday's agent to today's agent. They can't pinpoint which change broke the behavior. They can't roll back to a known-good state. The agent's quality drifts unpredictably.

Facio's versioned prompts discipline makes agent behavior reproducible. Every prompt version is pinned to a specific commit. Every change is tracked in version control. Every comparison between configurations is traceable to specific changes. The team's debugging starts from a known state.

Here's how the discipline works, what it pins, and why reproducibility is what makes AI agent behavior manageable over time.

The Reproducibility Problem

Production AI agent behavior is fragile because the inputs that determine behavior are many, distributed, and easy to lose track of:

Prompts. The system prompt, the user prompt templates, the tool definitions — all are inputs to the model. A typo in any of them changes behavior.

Model versions. The same prompt produces different behavior across model versions. GPT-4 to GPT-4 Turbo. Claude Opus to Claude Sonnet. The team upgrades; the behavior shifts.

Tool definitions. The agent's tool list, the tool descriptions, the tool parameter schemas — all are part of the model's context. Adding or removing a tool changes which tools the agent chooses.

Code. The orchestration code that builds prompts, calls the model, parses responses, invokes tools, and handles errors. A refactor changes how the model is queried.

Configuration. Temperature, top-p, max tokens, response format, model parameters. A configuration change affects every output.

External dependencies. Tool responses, retrieved documents, user inputs. The model's behavior depends on what it sees.

The reproducibility problem is that any of these can change without the team noticing. The team tracks their code. They might track their prompts. They rarely track the model version, the tool descriptions, the configuration, all together in a way that lets them compare exact configurations.

The team says "the agent got worse last week." They can't answer: worse compared to what? Which inputs changed? Was it the prompt, the model, the code, the configuration, or some external dependency? The team guesses.

The discipline is to make every relevant input traceable, versioned, and comparable.

The Versioned Prompts Discipline

Facio's versioned prompts discipline has four pillars. Each addresses a different aspect of reproducibility.

Pillar 1: Prompt Version Control

Every prompt is stored in version control, alongside the agent code:

# Repository structure
agent_repository/
├── agent.py                    # Agent orchestration code
├── prompts/
│   ├── system_prompt.md        # System prompt
│   ├── user_prompt_template.md # User prompt template
│   └── tool_definitions.json   # Tool schemas and descriptions
├── config/
│   ├── model.yaml              # Model configuration
│   └── tools.yaml              # Tool configuration
├── tests/
│   ├── golden_set.yaml         # Test cases
│   └── expected_outputs/       # Reference outputs
└── CHANGELOG.md                # Human-readable change log

Every prompt change is a commit. Every commit has a message. Every change has a reviewer (in production workflows). The team uses git to track prompt evolution.

The prompt version control means the team can:

• See what the prompt was at any point in time
• Compare two prompt versions side by side
• Roll back to a known-good prompt
• Bisect to find which prompt change caused a regression

Pillar 2: Configuration Pinning

Every agent run captures the complete configuration:

# Run configuration (captured per session)
run_config = {
    "prompt_version": "abc123def456",        # Git commit SHA
    "model": "claude-opus-4-2026-06-15",     # Exact model version, not just name
    "model_parameters": {
        "temperature": 0.7,
        "max_tokens": 2048,
        "top_p": 0.9,
    },
    "tool_definitions_version": "def789",
    "agent_code_version": "v3.2.4",
    "tools_available": ["postgres", "slack", "web_search"],
    "external_dependencies": {
        "model_api_version": "2026-07-01",
        "search_index_version": "v2.1.3",
    }
}

The run configuration is captured with every session. The team can reproduce any past session by replaying the exact configuration.

Pillar 3: Reproducibility Manifests

For each deployment, a reproducibility manifest captures the full state:

# Reproducibility manifest (deployment_v3.2.4.json)
{
    "deployment_id": "deploy-2026-07-13-101000",
    "deployment_timestamp": "2026-07-13T10:10:00Z",
    "components": {
        "agent_code": {
            "version": "v3.2.4",
            "git_commit": "abc123",
            "git_branch": "main"
        },
        "prompts": {
            "system_prompt_version": "abc123",
            "user_template_version": "abc123",
            "tool_definitions_version": "abc123"
        },
        "model": {
            "provider": "anthropic",
            "model_id": "claude-opus-4-2026-06-15",
            "model_card_version": "1.2"
        },
        "tools": {
            "postgres_mcp": "v1.4.2",
            "slack_mcp": "v2.0.1",
            "web_search_mcp": "v1.8.0"
        },
        "runtime": {
            "facio_version": "2.1.0",
            "config_version": "deployment-config-v47"
        }
    }
}

The manifest is the deployment's identity. The team can roll back to a deployment by rolling back to its manifest. The team can compare two deployments' manifests to understand what changed.

Pillar 4: Behavioral Testing Against Pinned Configurations

The golden test set is run against pinned configurations:

# Test execution
test_run = {
    "test_set": "customer-support-v3",
    "test_set_version": "2026-07-10",
    "configuration_pinned": "deployment_v3.2.4",  # Specific deployment
    "results": {
        "test_cases_passed": 94,
        "test_cases_failed": 6,
        "overall_score": 0.89,
    },
    "comparison_to_baseline": {
        "baseline_config": "deployment_v3.2.3",
        "score_delta": -0.03,
        "regressions": ["tc-023", "tc-041", "tc-089"],
    }
}

The team tests against pinned configurations. They compare results across configurations. They identify regressions with specific test cases and specific configuration changes.

The Versioning Patterns

Several patterns emerge from disciplined prompt versioning.

Pattern 1: Prompts as Code

Prompts are treated as code. They go through the same review process:

# Pull request workflow for prompt change
1. Developer modifies prompt
2. Developer submits PR with prompt diff
3. CI runs:
   - Lint checks on prompt (no placeholders, valid syntax)
   - Golden test set against new prompt
   - Comparison to baseline (regression check)
4. Reviewer reviews:
   - Code review on prompt content
   - Test results review
5. Approval and merge

The prompts-as-code pattern brings software engineering discipline to prompt management.

Pattern 2: Semantic Versioning for Prompts

Prompt versions follow semantic versioning:

# Semantic versioning for prompts
prompt_version = "MAJOR.MINOR.PATCH"

# MAJOR: Breaking changes (significantly changes agent behavior)
# MINOR: Backward-compatible additions (new tool, new context)
# PATCH: Backward-compatible fixes (typo, formatting)

The semantic versioning tells the team at a glance what kind of change a version represents.

Pattern 3: Branch-Based Experimentation

Prompt experiments use branches:

# Branch-based experimentation
branches = {
    "main": "Production prompts (pinned to specific versions)",
    "experiment/refined-tone": "Experimental prompts with refined tone",
    "experiment/structured-output": "Experimental prompts for structured output",
}

# Workflow:
# 1. Create branch from main
# 2. Modify prompts in branch
# 3. Run golden test set against branch
# 4. Compare to main
# 5. If improvement: merge to main with semantic version bump
# 6. If regression: archive branch

The branch-based experimentation keeps production stable while allowing experimentation.

Pattern 4: Configuration Diffing

Configuration changes are diffed before deployment:

# Configuration diff (between current and proposed)
diff = {
    "prompts": {
        "system_prompt": {
            "added_lines": 3,
            "removed_lines": 1,
            "modified_lines": 5,
        },
        "tool_definitions": {
            "added_tools": ["sendgrid"],
            "removed_tools": [],
            "modified_tools": [{"name": "slack", "changes": ["description updated"]}],
        },
        "model": {
            "from": "claude-opus-4-2026-06-15",
            "to": "claude-opus-4-2026-07-01",
            "expected_behavior_impact": "minor phrasing changes",
        }
    }
}

The diff tells the team what changed. The team can predict the impact. The deployment is informed.

Pattern 5: Model Upgrade Playbook

Model upgrades follow a structured playbook:

# Model upgrade playbook
1. Pin current model in manifest
2. Deploy new model in staging
3. Run full golden test set against staging
4. Compare results to production manifest
5. Identify regressions
6. Refine prompts to handle regressions
7. Re-run tests until quality is on par or better
8. Canary deploy to production (5% traffic)
9. Monitor quality metrics
10. Full rollout when canary passes

The playbook makes model upgrades predictable. The team knows what to do at each step.

The Versioning Observability

Versioning data is observable:

# Versioning dashboard
dashboard = {
    "current_production": {
        "deployment_id": "deploy-2026-07-13-101000",
        "manifest": "deployment_v3.2.4",
        "uptime_hours": 8,
        "quality_score": 0.89,
    },
    "recent_changes": [
        {"commit": "abc124", "type": "patch", "description": "Fixed typo in system prompt"},
        {"commit": "abc125", "type": "minor", "description": "Added new tool: sendgrid"},
        {"commit": "abc126", "type": "major", "description": "Restructured reasoning flow"},
    ],
    "experiment_branches": [
        {"name": "experiment/refined-tone", "status": "testing", "score": 0.91},
        {"name": "experiment/structured-output", "status": "archived", "score": 0.78},
    ],
    "model_upgrade_pipeline": {
        "current_model": "claude-opus-4-2026-06-15",
        "candidate_model": "claude-opus-4-2026-07-01",
        "staging_score_delta": 0.02,
        "production_canary_status": "pending"
    }
}

The observability surfaces the configuration state. The team knows what's in production, what's being tested, what's planned.

The Versioned Prompts Discipline Doesn't Do

Honest limitations:

• It doesn't make non-deterministic models deterministic. Even with the exact same prompt, model, and code, the model can produce different outputs across runs (due to temperature, sampling, model non-determinism). The discipline reduces variance, it doesn't eliminate it.
• It doesn't capture everything that affects behavior. External services, retrieved documents, user inputs — these are out of the discipline's control. The team can capture what was retrieved, but not what would have been retrieved in different circumstances.
• It doesn't prevent regressions. The discipline catches regressions by testing against pinned configurations. It doesn't prevent regressions from being introduced.
• It requires maintenance. The team has to keep the manifests updated, the branches clean, the golden sets current. Without maintenance, the discipline becomes stale.
• It can slow down experimentation. The discipline's rigor adds overhead. The team balances rigor against speed.

The Versioning as Cultural Practice

Versioned prompts are a cultural practice:

Discipline over speed. The team prefers the slower, disciplined change over the faster, risky change. The discipline catches regressions.

Evidence over opinion. Prompt changes are validated by tests, not by opinion. The team uses data to make decisions.

Traceability over magic. The team can explain any behavior by pointing to the configuration that produced it. There's no magic.

Rollback over firefighting. When something breaks, the team rolls back to a known-good state. They don't firefight in production.

The culture is what makes the discipline sustainable. Without it, the team bypasses the discipline under pressure. With it, the team holds the line.

The Compound Effect of Versioned Prompts

Versioned prompts compound:

• Reproducible debugging. When something goes wrong, the team can reproduce the exact configuration. They can debug.
• Confident deployments. The team deploys because they know what they're deploying. They can compare to baseline.
• Fast rollback. When a deployment breaks, the team rolls back to the previous manifest. The fix is minutes, not hours.
• Knowledge accumulation. The team learns which prompt changes work, which don't. The knowledge compounds.
• Cross-team collaboration. Different team members can understand each other's changes. The version control is the shared understanding.

The undisciplined approach has the opposite trajectory. Unpredictable behavior, slow debugging, scary deployments, lost knowledge. The team operates in firefighting mode.

Bottom Line

An AI agent's behavior comes from prompts, models, code, and configuration working together. Without discipline, the behavior shifts unpredictably. With versioned prompts, the behavior is reproducible.

Facio's versioned prompts discipline pins every prompt in version control, captures every run's configuration, maintains reproducibility manifests, and tests against pinned configurations. The discipline makes agent behavior manageable over time.

The agent without versioning is a moving target. The agent with versioning is a pinned, traceable, reproducible system. The team debugs the pinned one. The moving target is guesswork.

Because AI agents in production change. The question is whether the changes are controlled or chaotic. The versioned prompts discipline is what makes the answer controlled.

---

See the versioned prompts documentation for prompt storage, manifest configuration, and reproducibility patterns.