Product Updates

🚨Alerts++

We’re excited to launch a big update to our rollout alerts product today. Here’s a quick overview of what’s changing:

• Alerts will now have any experiment-level configured stats methodologies applied- If you’ve enabled CUPED or Sequential Testing for your experiment, this will now be incorporated into your alert firing logic post-the first 24 hours of a new gate/ experiment going live (i.e. as soon as daily Pulse results are available).

• Alerts now have confidence intervals attached- Instead of just surfacing metric value relative to a threshold, we will surface metric value with confidence intervals attached, to provide you more context on how seriously you should be concerned about an alert firing.

• Alerts only fire if they are statistically significant- This should drastically reduce the noisiness of alerts on Statsig and ensure you’re only getting pinged with high-signal regressions.

As a reminder, alerts can be set up and configured via the Metrics tab, at the per-metric level. Hop on in, set up alerts for your key regression metrics, and let us know if you have any feedback!

Alert configuration UX

Support for Percentile Metrics

We're excited to announce Percentile metrics on Statsig Warehouse Native! Percentiles are often used to optimize app performance, understand feature adoption or even manage resource utilization when experimenting on backend infra and AI models.

Percentiles are particularly useful when applied to metrics that exhibit large variances. They also help understand the distribution of a metric, and can be critical to understand outliers or unusual metric behaviors. Customers can now visualize understand impact (or even alert on) p90, p95, p99, p99.9 or any other percentile.

Reach out in Slack if you want to opt into this! If you're interested in the underlying math, we'll be writing about it but it's loosely patterned on the thinking here - Applying the Delta Method in Metric Analytics: A Practical Guide with Novel Ideas.

🔍 Diagnostics 2.0

Today, we’re thrilled to introduce two upgrades to our Diagnostics tab, which enable easier debugging of gates & experiments-

Upgraded Logstream- We’ve added the ability to access longer-term log history, as well as filter by things like rule, reason, experiment group, user properties, and other metadata to enable easily pinpointing the most important logs for your debugging.

New Statsig logstream, with filtering capabilities

SRM Debug Helper

Imbalanced exposures are the last thing you want to see when launching a new experiment, and often seeing this failing health check kicks off a deep-dive into isolating where the imbalance is coming from. We’ve now exposed more detail into the SRM we’re observing, including how the p-value is trending over time, as well as some auto-generated cuts of p-value (e.g. by browser_version, os, region, etc.) to help you isolate where the imbalance may be disproportionately coming from.

P-value broken down by group

🥽 Statsig now supports visionOS

The Statsig iOS SDK just added support for visionOS. You can now use Statsig in your apps for the Apple Vision Pro (iOS SDK versions v1.39.1 or higher).

🧮 New Metric Types on Statsig Warehouse Native

We added two new metric types and more configurability on CUPED on metrics.

Count Distinct Metrics : We added a new metric aggregation for COUNT DISTINCT that counts the unique occurrences of each value. Learn more

Latest Value Metrics : If you only care about a count of the current state of a users (e.g. Is the user a subscriber today), use this. Configure the Time Window to be Latest Value on a User Count Metric for this. Learn more

Configurable CUPED Windows : CUPED is an advanced statistical technique that speeds up experimentation. It reduces the amount of time or users required, by reducing metric variance by looking at pre-experimental metric history for users. You can now configure the CUPED lookback window (pre-experimental period) per metric to match your app's usage pattern for it to be useful (e.g. if users are typically only monthly active users, you can configure the CUPED look back period to be a month). Learn more

🤼 Teams

Today we’re excited to announce the new Teams feature. As Statsig adoption scales across an organization, the Teams feature enables a settings/ permissions layer on top of Projects, empowering you to define and enforce best practices at the per-team level.

With Teams, you can:

• Define a team-specific standardized set of metrics that will be tracked as part of every Gate/Experiment launch.

• Configure various team settings, including allowed reviewers, default target applications, and who within the company is allowed to create/ edit configs owned by the team.

• Filter lists of configs by Team, and set your Home Feed to only include updates relevant to team(s) you’re a part of.

Teams is an Enterprise-only feature at this time. Read more about Teams in our docs here.

Teams Configuration under Project Settings

Analyze Product Metrics by Feature Gate Rule in Metrics Explorer

In January, we announced the ability to perform segment analysis based on Experiment Groups. Today, we're expanding that functionality to include Feature Gates as well. Try out this feature today by selecting a metric of interest, choosing a group by, and selecting "Experiments and Gates."

Group-by Feature Gate: Segmentation analysis is one of the most powerful tools product teams have when making targeted improvements to a product. Now, with the ability to group by Feature Gate, you can get a general sense of how a metric is performing for different Feature Gate rules, view the long-term effect of a feature, or monitor and debug the product performance of a feature before rolling it out broadly.

View a Sample of Events that Contribute to a Metric for a Given Feature Gate/Experiment in Metrics Explorer: When performing an analysis on an Experiment or Feature Gate, you can now switch from a Line chart to the "Samples" view, where you can see a sample of raw events. When grouped by an Experiment or Feature Gate, you can see a sample of events that affect your given metric, separated by the Feature Gate rule /Experiment Group the user was in. This is a great way of checking your experiment or feature roll out setup, or to gain a better sense of why specific groups are behaving in the way they are.

🚫 Read Only Metric Definitions in Console

Sync metrics from your Semantic Layer to Statsig as read-only. Users can view but not edit these metric definitions, ensuring version control and change management. This works well in tandem with Verified Metrics. (Learn more)

This feature is available on both Statsig versions - Cloud and Warehouse Native.

Slicing Experiment Results by User Dimensions from your Warehouse

Entity Properties are a Statsig Warehouse Native feature that let you slice experiment results by User Dimensions that come from your warehouse (e.g. User's Country, Subscription Status). This data can be time sensitive (for when experiments change this). Learn More.

User Journeys Beta

We're thrilled to announce the beta release of User Journey charts in Metrics Explorer! These charts are designed to help you visualize and understand the most common paths users take through your product, starting from a specific point.

While it's common to envision a "golden path" through your product, users often take various routes. User Journeys provide insights into the actual paths taken, allowing you to see how users navigate through your product and identify areas where they drop off and may need improvement.

We've rolled out User Journeys in beta to most customers. We're eager to hear your feedback and refine this feature to make it an essential tool for optimizing user experience and streamlining product navigation. Explore User Journeys today and share your thoughts with us!