Chad Swenson
Principal Product Manager, Statsig + Amplitude

Introducing Agent Experimentation: From better evals to better products

Tue Nov 17 2026

You’ve built v1 of your agent, but how do you know it’s actually working? Your eval score went up last week. Your customer engagement metrics went down. You’re left scratching your head.

The only way to know whether a change to an AI feature truly works is to validate it with real users and real outcomes. That is an experimentation problem. And that’s exactly the kind of problem we like to solve.

New from Statsig, Agent Experimentation lets you move past a simple eval score to get a rigorous, holistic picture of how agentic features perform and improve. By accurately testing and measuring the impact of your changes, you’ll be able to build agents that work better for your users and for your business metrics.

What is Agent Experimentation?

From building our own agents, we understand very well that a good eval score is only half the battle.

That’s because AI succeeds and fails differently from traditional software. Every change to a prompt, model, or tool can shift quality in ways that are hard to predict. Evals help you catch regressions, but an LLM judge is still a model and hypothetical. A change can pass every offline test and fall flat with your customers.

To deal with these often-frustrating complexities, we built Agent Experimentation to connect the whole process of building and testing agentic features:

  • AI configs to standardize changes to your agents, just like you’d do for A/B variants

  • Offline evals let you grade agents on your own with LLM judges or golden datasets

  • Online evals for shadow-testing on live traffic

  • Experiments to prove how agents actually affect your users and metrics

You get the same path from an idea to a measured result that you’d take for traditional software, but with all the non-deterministic bases covered.

AI configs: The control plane for your agent

AI configs product screen

Just like with anything, if you’re trying to improve an agent, you need to be clear on what you’re changing so you can understand the results of those changes.

In Agent Experimentation, you do this with AI configs. This agent control plane lets you manage versions of your prompts, models, and settings. For example, maybe you want to explore how your agent runs on GPT-6.1 Sol versus GPT-6.1 Luna. Just make a new config and change the model in the dropdown.

AI configs are the starting point for any eval or experiment you run on your agent, but even without the rest of the Agent Experimentation suite, you can use AI configs to set up an LLM in your app.

Offline evals: Choose what to ship

Offline evals product screen

Once you have your AI configs going, offline evals let you run those versions against a shared dataset or use evaluators to compare their behavior before releasing changes.

Teams with existing workflows can also run evaluations programmatically with datasets and evaluators managed in code.

Your team can iterate on the agent and the way you judge it together. A designer can help define tone and interaction behavior. A PM can clarify what successful task completion means. An engineer can inspect traces to debug failed tool calls, understand what went wrong in individual interactions, and improve the tools and context the agent needs. Together, they can refine prompts, datasets, and evaluator instructions.

Online evals: Learn from real customer interactions

Online evals product screen

Offline evaluations give you a consistent place to test. Online evals show you what those tests missed in production.

Use traces and online evaluations to inspect live interactions and check whether the behavior you saw offline holds up with customers. When an interaction reveals a gap, add it directly from the trace to your dataset inside Statsig.

A question your agent couldn't answer today becomes a test for its next version. Your dataset grows from real customer needs, and your team can check that a fix works before releasing it.

Experimentation: Measure real AI value

Create Experiment modal for agent experimentation

A higher eval score is a good start, but to tell whether a change actually improves the outcomes and metrics you care about, you need an experiment.

The core feature that gives Agent Experimentation its name, this lets you compare agent versions in an A/B test and measure evaluator scores alongside product metrics such as task completion, engagement, conversion, or retention.

Say you’ve been using evals to gauge how different models improve an agent’s answers to user questions. Do those better answers actually help customers complete their work? Experimenting can show you. If evaluator scores improve but customer outcomes don't, that's a reason to revisit your dataset, your evaluators, or your assumptions about quality.

Connecting evaluations with experiments helps you improve both your agent and the measurement system you use to guide it.

Start improving your agent scientifically

Every release should help you understand what to improve next. Agent Experimentation connects your team's work across evaluations, production, and experiments so what you learn carries into the next version of what you build.

Agent Experimentation is now available to everyone in Statsig. Start with one AI config, a small dataset, and a few evaluators. Explore Agent Experimentation and give it a try.

We'd love to hear what works, what's missing, and what would make this more useful for your team. We'll be improving Agent Experimentation every week based on your feedback.



Please select at least one blog to continue.

Recent Posts

We use cookies to ensure you get the best experience on our website.
Privacy Policy