How Statsig MCP v3 expanded capabilities while shrinking context

Statsig’s MCP has reached a smaller interface with broader capabilities and finer access controls. Here’s how we got there, and why that’s important.

As agentic workflows became more common, our customers began asking to do more through Statsig MCP. We addressed this by adding breadth: we started with around 30 tools, each mapped to an API operation. Within a few months, we had more than 90.

Improving breadth helped. Each request exposed a gap in a customer’s workflow. We wrapped another API operation, added a tool, and closed it.

Then the catalog itself became the problem.

With each new tool, results took longer and used more tokens. An agent answering a simple question about an experiment could arrive with schemas for dozens of operations it would never use. We’d solved for breadth, but at the cost of efficiency.

Breadth is important, but if a tool is inefficient, nobody will use it. So we set out to redesign the Statsig MCP from the ground up. By prioritizing what deserved a place in the agent’s context, MCP v3 took us from 93 legacy tools to 18 default tools, with broader API coverage available through discovery.

Limitations of one tool per API

One tool per API was a reasonable starting point. Our APIs already covered gates, experiments, configurations, and results. Each wrapper gave the agent a description, an input schema, and a way to call the operation.

A client can call one API endpoint without loading the definitions of all the others. In clients that load the full MCP catalog, every tool definition occupies context before the agent decides whether it needs it.

Those schemas compete in the agent’s context with code, conversation history, and results. That also increases input token usage. Caching and deferred loading change the cost across clients, but we still needed room to expand coverage without making every connection heavier.

Before removing tools, measure the work

An endpoint test starts with the correct operation and valid arguments, while an agent starts with a request. Our tests needed to cover the decisions in between, from finding a capability to interpreting its response.

We built an evaluation harness that ran tasks through MCP and recorded calls, arguments, results, and timing. Using consistent prompts and execution conditions, we checked four things:

• Selection: Did the agent find the right capability?

• Execution: Did the arguments match the user’s request?

• Answer quality: Did the final answer faithfully represent the returned data?

• Efficiency: How many model turns, tokens, and seconds did the task take?

A tool call can succeed while the answer is wrong. For entity-list questions, we compared the answer against the returned entities to catch omissions or inventions. Wherever structured ground truth was available, we used deterministic checks.

We compared loading the full catalog, a curated core, or just discovery for every task. The full catalog was our baseline: bulky. Discovery reduced the initial schema overhead but added a round trip to tasks that previously went straight to execution. The curated core seemed to strike the right balance.

The question became which operations were worth keeping in context.

Spend context where customers spend time

Existing traffic showed customers repeatedly reading gates, inspecting experiments and results, checking logs, and listing resources. These operations were worth keeping directly available; less frequent calls could pay the discovery cost.

That led to a hybrid interface: 18 default tools, with discovery for the long tail. The V3 migration guide describes the transition.

Keeping the common path short

Tools such as gate_read, experiment_read, and log_read provide direct entry points. Related reads share an action selector, avoiding a separate top-level tool for every variation.

For example, to inspect an experiment, the agent calls experiment_read with the relevant action and arguments. There is no discovery round-trip.

Loading the long tail on demand

For capabilities outside the default direct tools, like listing a project’s segments, the agent follows three steps:

1. Find the operation. Ask discover_tools to resolve the request to list segments.

2. Read the contract. A confident match returns the input schema, an operation handle, and the execution tool. An ambiguous match returns alternatives the agent can select from before retrieving the contract.

3. Execute the read. Pass the handle and matching arguments to api_read, then use the returned segments to answer.

Separate execution tools keep consequences visible to clients and users. Reading results, changing configuration, and deleting a resource still need distinct treatment behind a generic interface.

The tool reference documents both the direct and discovery paths.

Controlling permissions

Customers also wanted control over what agents could do: disable MCP entirely, allow inspection without changes, or set different limits by role. This added an extra layer of nuance to how discovery worked, because different users would need different ability access.

To address permission governance, we added self-serve controls in Project Settings under Roles. Administrators can disable MCP for a project or cap its access. Role settings further restrict personal OAuth access to none, read only, or read and write.

These controls further restrict existing RBAC. Read-and-write MCP access does not grant permission to update every gate or delete every experiment. Entity permissions and review requirements still apply.

To make this work with our hybrid approach to loading tools, we designed discovery to filter by effective MCP access. We also made execution check access again before dispatch, so that, for example, routing an operation through api_write won’t give an agent more authority than it had as a dedicated tool.

Investigating behavior, not just error rates

As we developed MCP v3’s hybrid design, we used Amplitude Agent Analytics to analyze and evaluate agent behavior instead of just logging errors and trying to fix them.

For example, an early analysis of MCP usage showed a high error rate for gate updates. Taken on its own, that looked like a reliability problem.

When we dug into the calls, it turned out that users lacked permission to perform those operations. The server was correctly refusing the writes; the agent was just logging the errors incorrectly.

Instead of an implementation failure needing a fix, the permission denial warranted examining what the agent attempted and how clearly the refusal was explained.

Analyzing tool frequency helped us revisit the direct catalog; repeated calls, errors, and difficulty finding operations showed us where to investigate. Patterns from real usage became reproducible evals, letting us test a change against the behavior that prompted it.

Why a hybrid approach worked for Statsig

Statsig’s API catalog and concentration of usage around common workflows make this hybrid practical. The right balance between direct tools and discovery depends on the workflows a system serves.

For us, the redesign makes each addition a deliberate choice: keep it direct when usage justifies the context cost, make it discoverable otherwise, and use evals to check that agents can still complete the work. We can expand what MCP does without putting every new operation into every conversation.