Kitaru by ZenML (Part 3): Deterministic Trace Replay, Insights, and Enterprise CI/CD

This is the final instalment of our 3-part technical deep dive into Kitaru by ZenML. Read Part 1: Architecture & Durable Execution and Part 2: Wait Gates & Framework Adapters for the runtime foundations.

Shipping an AI agent to production is only half the battle. Once real users interact with it, failures emerge in long-tail combinations of ambiguous prompts, unexpected tool responses, and multi-turn state drift.

Traditional LLM observability tools show you one trace at a time, leaving engineers to manually click through hundreds of JSON logs. Meanwhile, standard unit-test evals only catch failures you already anticipated.

Kitaru bridges the gap between production observability and regression testing through a four-stage Replay & Evaluation Loop.

1. Stage 1: Bring Your Production Traces

You do not need to rip out your existing observability stack to use Kitaru’s evaluation engine. Kitaru supports two ingestion paths:

1. Native Runtime Recording: Any agent wrapped in Kitaru’s @flow and @checkpoint primitives (or using Kitaru’s Python/TypeScript framework adapters) automatically records full session traces and tool payloads.
2. Import External Traces: Pull historical production traces directly from Langfuse, LangSmith, Braintrust, Logfire, Arize Phoenix, or MLflow into Kitaru’s unified local or remote runtime.

2. Stage 2: Cross-Session Insights (Looking Across Thousands of Runs)

Instead of inspecting individual sessions one by one, Kitaru’s built-in analysers scan across thousands of recorded sessions simultaneously to surface systemic failure modes:

• Recurring Tool Failures: Identifying which tool schemas consistently trigger malformed JSON arguments from the model.
• Infinite Retry Loops: Flagging trajectories where an agent calls the same search or lookup tool repeatedly without advancing state.
• Divergent Outcomes: Separating successful sessions from failed ones to highlight which prompt branches or context lengths correlate with regressions.

3. Stage 3: Turn Insights into Evaluators via MCP

Once Kitaru surfaces a cluster of failing sessions, engineers still face a bottleneck: writing assertions and LLM-as-a-Judge rubrics by hand.

Kitaru ships with a Model Context Protocol (MCP) server and agent skills designed to plug directly into coding assistants such as Claude Code and Cursor. Using the MCP integration, your coding assistant can:

1. Query Kitaru for the exact failing session cluster tied to an insight.
2. Inspect the tool calls, prompt state, and final outputs.
3. Interview you on the expected domain behaviour and codify your answers into repeatable evaluators (checks) that run continuously against new sessions.

4. Stage 4: Deterministic Replay (Testing Fixes Without Side Effects)

Suppose your team updates a system prompt, fixes a Python tool parser, or migrates from an older model to Gemini 4 or Claude Opus 5.5.

If you re-run 500 historical user requests against live APIs, you face two huge problems:

• Live Side Effects: You cannot safely re-execute mutating tools (charging cards, sending customer emails, or modifying production databases).
• Non-Determinism: Live external APIs return different data today than they did last Tuesday when the bug occurred.

Kitaru solves this with Deterministic Session Replay:

Historical Production Trace
  ├── Turn 1: User Prompt ──────────────────────► Re-run against NEW Agent Code / Model
  ├── Tool Call: fetch_order(id="ORD-991") ─────► Answered from ORIGINAL Recording (No live API hit)
  ├── Turn 2: Reasoning + Policy Check ─────────► Re-run against NEW Agent Code / Model
  └── Final Output ─────────────────────────────► Scored by Kitaru Evaluators (Checks)

Because tool calls are answered directly from the original recorded checkpoint artifacts, you can validate a prompt patch or model upgrade across hundreds of real production edge cases in seconds, with zero live side effects and zero flaky external dependencies.

5. Measuring Prompt Caching Cost Savings via Replay

Deterministic replay is not only a quality gate; it is also an exact AI FinOps benchmarking tool.

When teams restructure prompts to take advantage of provider prompt caching (for example, pinning static system instructions, tool schemas, and domain policies at the top of the prompt prefix in Gemini 4 or Claude Opus 5.5), they often guess at theoretical savings. With Kitaru replay, you can measure real savings on actual multi-turn production traffic before shipping:

1. Baseline Replay Pass: Replay 100 recorded multi-turn sessions using the existing uncached prompt layout (where dynamic user timestamps or session IDs sit at the top of the prompt, invalidating prefix caches).
2. Optimised Replay Pass: Replay the exact same 100 sessions (with identical recorded tool outputs) using a cache-friendly prompt layout where static instructions and few-shot examples are pinned first.
3. Compare Cost and Quality Together: Aggregate cached_input_tokens, uncached_input_tokens, output_tokens, and evaluator pass rates across both runs.

Step-by-Step Python Cost Comparison Harness

Below is a practical Python comparison utility you can run over the telemetry emitted by two Kitaru replay runs:

from dataclasses import dataclass
from typing import List


@dataclass
class ReplayTokenMetrics:
    session_id: str
    uncached_input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    eval_passed: bool


def calculate_replay_cost_gbp(
    runs: List[ReplayTokenMetrics],
    uncached_per_1m_gbp: float = 2.40,
    cached_per_1m_gbp: float = 0.24,
    output_per_1m_gbp: float = 9.60,
) -> dict:
    """
    Computes total cost, cache hit rate, and evaluator pass rate across a replay suite.
    """
    total_uncached = sum(r.uncached_input_tokens for r in runs)
    total_cached = sum(r.cached_input_tokens for r in runs)
    total_output = sum(r.output_tokens for r in runs)
    total_input = total_uncached + total_cached

    cost_gbp = (
        (total_uncached / 1_000_000) * uncached_per_1m_gbp
        + (total_cached / 1_000_000) * cached_per_1m_gbp
        + (total_output / 1_000_000) * output_per_1m_gbp
    )
    cache_hit_rate = (total_cached / total_input) * 100 if total_input else 0.0
    pass_rate = (sum(1 for r in runs if r.eval_passed) / len(runs)) * 100 if runs else 0.0

    return {
        "sessions": len(runs),
        "cache_hit_rate_pct": round(cache_hit_rate, 2),
        "eval_pass_rate_pct": round(pass_rate, 2),
        "total_cost_gbp": round(cost_gbp, 4),
    }


def compare_prompt_caching_impact(
    baseline: List[ReplayTokenMetrics],
    optimised: List[ReplayTokenMetrics],
) -> dict:
    base_stats = calculate_replay_cost_gbp(baseline)
    opt_stats = calculate_replay_cost_gbp(optimised)

    savings_gbp = base_stats["total_cost_gbp"] - opt_stats["total_cost_gbp"]
    savings_pct = (
        (savings_gbp / base_stats["total_cost_gbp"]) * 100
        if base_stats["total_cost_gbp"] > 0
        else 0.0
    )

    return {
        "baseline": base_stats,
        "optimised": opt_stats,
        "net_savings_gbp": round(savings_gbp, 4),
        "cost_reduction_pct": round(savings_pct, 2),
        "quality_preserved": opt_stats["eval_pass_rate_pct"] >= base_stats["eval_pass_rate_pct"],
    }

By pairing cost metrics with evaluator pass rates on the exact same historical tool trajectories, engineering and FinOps teams can prove that prompt caching (or routing simpler turns to a faster model) cuts inference spend by 50% to 75% without introducing quality regressions.

6. Enterprise Production Deployment Checklist

When promoting Kitaru from a developer workstation (kitaru login --local) to a shared enterprise platform, enforce these five operational controls:

1. Managed State & Artifact Backends: Back your shared Kitaru server with managed PostgreSQL for execution metadata and Google Cloud Storage (GCS) or Amazon S3 for versioned checkpoint artifacts, authenticated via Workload Identity / IAM roles rather than static keys.
2. PII Masking Before Checkpoint Persistence: Because Kitaru persists checkpoint inputs and outputs for future replay, strip or tokenise sensitive customer PII (emails, payment tokens, national IDs) inside a guardrail layer before returning payloads from @checkpoint functions.
3. Downstream Idempotency Keys: Even with @checkpoint caching, pair side-effecting tool calls with deterministic idempotency keys (flow_execution_id + checkpoint_name) so a network drop mid-commit never double-executes an external mutation.
4. Separate Read-Only vs. Mutating Tool Replay Policies: In CI/CD replay suites, verify that mutating tools remain strictly mocked from recorded artifacts while evaluator checks validate both structural schemas and semantic correctness.
5. Gate Pull Requests on Replay Suites: Add a CI/CD step that replays a curated golden set of past production failure clusters whenever a PR modifies system prompts, tool schemas, or model routing configurations.

Summary: The Kitaru Architecture

Across this 3-part series, we have covered how Kitaru by ZenML closes the gap between fragile agent prototypes and production-grade enterprise systems:

• Part 1: Plain-Python durable execution with @flow and @checkpoint for automatic crash recovery.
• Part 2: Zero-compute human-in-the-loop pauses with kitaru.wait() and framework-agnostic adapters.
• Part 3: Trace ingestion, cross-session insights, MCP-assisted evaluator creation, and side-effect-free replay in CI/CD.

Why Alps Agility?

At Alps Agility, we combine deep AI engineering expertise with production MLOps and cloud architecture to help enterprises build durable, observable, and cost-effective agentic systems.

Contact us today to architect your production AI agent runtime and evaluation pipeline.

Struggling to move AI from prototype to production? We help enterprises build robust, scalable AI architectures. Book a Generative AI Readiness Assessment.