Product Updates

📤 Templates

Have a standard rollout you want to leverage across the team? Want to standardize best practices for experiment design across the company? Templates enable you to codify a blueprint for config creation that fellow team members can use to bootstrap their own feature gates and experiments.

Key features of Templates

• Create a new template from scratch from within Project Settings or easily convert an existing experiment or gate into a template from the config itself

• Manage your templates all in one place within Project Settings, restricting which roles on your team have the ability to create and modify templates via Statsig's role-based access controls

• Restrict which templates a given team can select from via "Allowed Templates" settings within team settings

Read more about Templates via our documentation here.

Templates home within Project Settings

🍯Persistent Assignment on Experiments

This is a new SDK feature that makes user bucketing decisions on experiments sticky even in situations they wouldn't have been previously. Users exposed to an experiment are bucketed into Control or Test deterministically (see how); however allocation or targeting changes can cause a user to to be excluded from an experiment after they were exposed to it. Persistent Assignment ensures that users stay bucketed in the experiment even when allocation or targeting changes.

Some scenarios this unlocks

1. You can roll out an experiment to 100% of users for a week, and then drop allocation to 0%. Users exposed to the experiment in that first week will continue to experience the experiment; other users will not.

2. When you target an experiment at set of users (e.g. low engaged users), if the user state changes they usually fall out of the experiment. With Persistent Assignment they will continue to see the experiment (e.g. even if they move from low engaged -> high engaged).

Learn more about Persistent Assignment on Client SDKs, Server SDKs

🚨Alerts++

We’re excited to launch a big update to our rollout alerts product today. Here’s a quick overview of what’s changing:

• Alerts will now have any experiment-level configured stats methodologies applied- If you’ve enabled CUPED or Sequential Testing for your experiment, this will now be incorporated into your alert firing logic post-the first 24 hours of a new gate/ experiment going live (i.e. as soon as daily Pulse results are available).

• Alerts now have confidence intervals attached- Instead of just surfacing metric value relative to a threshold, we will surface metric value with confidence intervals attached, to provide you more context on how seriously you should be concerned about an alert firing.

• Alerts only fire if they are statistically significant- This should drastically reduce the noisiness of alerts on Statsig and ensure you’re only getting pinged with high-signal regressions.

As a reminder, alerts can be set up and configured via the Metrics tab, at the per-metric level. Hop on in, set up alerts for your key regression metrics, and let us know if you have any feedback!

Alert configuration UX

Support for Percentile Metrics

We're excited to announce Percentile metrics on Statsig Warehouse Native! Percentiles are often used to optimize app performance, understand feature adoption or even manage resource utilization when experimenting on backend infra and AI models.

Percentiles are particularly useful when applied to metrics that exhibit large variances. They also help understand the distribution of a metric, and can be critical to understand outliers or unusual metric behaviors. Customers can now visualize understand impact (or even alert on) p90, p95, p99, p99.9 or any other percentile.

Reach out in Slack if you want to opt into this! If you're interested in the underlying math, we'll be writing about it but it's loosely patterned on the thinking here - Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas.

🥽 Statsig now supports visionOS

The Statsig iOS SDK just added support for visionOS. You can now use Statsig in your apps for the Apple Vision Pro (iOS SDK versions v1.39.1 or higher).

🔍 Diagnostics 2.0

Today, we’re thrilled to introduce two upgrades to our Diagnostics tab, which enable easier debugging of gates & experiments-

Upgraded Logstream- We’ve added the ability to access longer-term log history, as well as filter by things like rule, reason, experiment group, user properties, and other metadata to enable easily pinpointing the most important logs for your debugging.

New Statsig logstream, with filtering capabilities

SRM Debug Helper

Imbalanced exposures are the last thing you want to see when launching a new experiment, and often seeing this failing health check kicks off a deep-dive into isolating where the imbalance is coming from. We’ve now exposed more detail into the SRM we’re observing, including how the p-value is trending over time, as well as some auto-generated cuts of p-value (e.g. by browser_version, os, region, etc.) to help you isolate where the imbalance may be disproportionately coming from.

P-value broken down by group

🧮 New Metric Types on Statsig Warehouse Native

We added two new metric types and more configurability on CUPED on metrics.

Count Distinct Metrics : We added a new metric aggregation for COUNT DISTINCT that counts the unique occurrences of each value. Learn more

Latest Value Metrics : If you only care about a count of the current state of a users (e.g. Is the user a subscriber today), use this. Configure the Time Window to be Latest Value on a User Count Metric for this. Learn more

Configurable CUPED Windows : CUPED is an advanced statistical technique that speeds up experimentation. It reduces the amount of time or users required, by reducing metric variance by looking at pre-experimental metric history for users. You can now configure the CUPED lookback window (pre-experimental period) per metric to match your app's usage pattern for it to be useful (e.g. if users are typically only monthly active users, you can configure the CUPED look back period to be a month). Learn more

🤼 Teams

Today we’re excited to announce the new Teams feature. As Statsig adoption scales across an organization, the Teams feature enables a settings/ permissions layer on top of Projects, empowering you to define and enforce best practices at the per-team level.

With Teams, you can:

• Define a team-specific standardized set of metrics that will be tracked as part of every Gate/Experiment launch.

• Configure various team settings, including allowed reviewers, default target applications, and who within the company is allowed to create/ edit configs owned by the team.

• Filter lists of configs by Team, and set your Home Feed to only include updates relevant to team(s) you’re a part of.

Teams is an Enterprise-only feature at this time. Read more about Teams in our docs here.

Teams Configuration under Project Settings

Analyze Product Metrics by Feature Gate Rule in Metrics Explorer

In January, we announced the ability to perform segment analysis based on Experiment Groups. Today, we're expanding that functionality to include Feature Gates as well. Try out this feature today by selecting a metric of interest, choosing a group by, and selecting "Experiments and Gates."

Group-by Feature Gate: Segmentation analysis is one of the most powerful tools product teams have when making targeted improvements to a product. Now, with the ability to group by Feature Gate, you can get a general sense of how a metric is performing for different Feature Gate rules, view the long-term effect of a feature, or monitor and debug the product performance of a feature before rolling it out broadly.

View a Sample of Events that Contribute to a Metric for a Given Feature Gate/Experiment in Metrics Explorer: When performing an analysis on an Experiment or Feature Gate, you can now switch from a Line chart to the "Samples" view, where you can see a sample of raw events. When grouped by an Experiment or Feature Gate, you can see a sample of events that affect your given metric, separated by the Feature Gate rule /Experiment Group the user was in. This is a great way of checking your experiment or feature roll out setup, or to gain a better sense of why specific groups are behaving in the way they are.

🚫 Read Only Metric Definitions in Console

Sync metrics from your Semantic Layer to Statsig as read-only. Users can view but not edit these metric definitions, ensuring version control and change management. This works well in tandem with Verified Metrics. (Learn more)

This feature is available on both Statsig versions - Cloud and Warehouse Native.