Your team is busy shipping features, and they are keeping a close eye on product metrics. While it’s easy (with Statsig) to understand the impact of individual feature launches, one question that often comes up is: What’s the collective impact of multiple features?
At Facebook, every team calculated the cumulative impact of all features shipped over the last 6 months. And to do that accurately, we use “Holdouts” that we create at the beginning of each half. A holdout is usually small (1–5%) in size, and as the name implies, holds that set of people out of any new features launched during that half.
This provides us with a baseline to measure the cumulative impact of multiple launches over 6 months. At the end of the half, we release that holdout and create a new one for the next half. Holdouts are powerful and have many other uses including measuring long-term effects, quantifying subtle ecosystem changes, and debugging metric movements.
Today, we’re making all this available via Holdouts on Statsig. Setting up a holdout is a cinch — you pick the size and the features you want held from users. You can also make holdouts be ‘global’ which means all new features will automatically respect the holdout. And occasionally (hopefully, sparingly) you might want to run a “back-test” and you can do that by turning on holdouts to an existing set of features.
Once set up, our Pulse engine will automatically compute the impact of all those features compared against the baseline. No additional configuration or code necessary.
Kong is our Typescript-based write-once-run on every SDK framework. “Write once, run anywhere” is always a dream for programmers, and now we have just that!
LaunchDarkly was mandatory for every new feature in Motion’s backend, web app, and Chrome extension. "It was obvious this was a huge mistake."
Last Tuesday, Statsig brought a cadre of data science and experimentation fans together at a loft space in San Francisco for the first-ever Data Science Meetup.
Well-designed experimentation is the first step in creating a rollout structure that consistently delivers optimal results—whatever they may be.
Using data and experimentation, the Obama 2012 campaign generated over one billion dollars in donations, nearly $700,000,000 of which were online.
It’s only my first week yet, but each day I am more and more impressed by the team’s velocity, excitement, and transparency, and feeling more sure that I’ve made the right decision for /me/.
Explore Statsig’s smart feature gates with built-in A/B tests, or create an account instantly and start optimizing your web and mobile applications. You can also schedule a live demo or chat with us to design a custom package for your business.