Products
Solutions
Resources
“The sample size calculator still says we need to run the experiment for 3 months, but we can’t wait that long. What do we do?”
We’ve gotten this question from many customers, so I wanted to write up an answer with the nuance it deserves. That nuance has now led to a whole blog series!
TL;DR: There are ways to change your experiment to reduce runtime, but they all introduce tradeoffs and assumptions that can affect your results. Sometimes, those tradeoffs mean the answer to this question is, “Don’t run the test.” Other times, they provide new and interesting ways to collect and analyze your data.
In Part 2 of this 4-part series, I’ll cover how to reduce runtime by adjusting your metrics. These solutions are by no means exhaustive, but they’re a good starting point.
The Running Faster Tests series
Part 1: How sample size calculators work and how to adjust their inputs
Part 2: You’re here!
Part 3 (to come): Different ways to target users for shorter runtime
Part 4 (to come): Four other types of experiments and how switching may speed things up
What does it mean to change your metric?
Your metric is what you measure in your experiment. Click-through rate, dollars spent, days to value, that sort of thing. Every metric is an attempt to represent the resulting effect of your experiment in number form.
But there’s some fuzziness to that representation. Say you’re experimenting on user spend in your checkout flow; you could represent that in raw dollars. You could also represent it as a change in spending. Or in quantities of 10 instead of down to pennies. Maybe you even represent it as a binary “dollars spent: yes or no?” Although those are all different, each still represents an aspect of the monetary transaction you’re trying to measure.
So if you want to run a faster experiment, you may be able to just measure a different aspect of what you’re testing. The three main ways this strategy plays out are:
Transform your metric to cut down on standard deviation. If you decrease the standard deviation, there’s less noise, so you need fewer samples.
Pick another metric that is more likely to be impacted by the changes you are testing (more frequent impact means faster results), or is highly correlated to the original but has a smaller standard deviation.
Use CUPED on your metric, or pick a metric on which you can. Kind of a mix of mathematical transformation and aspect shifting, CUPED uses user information from before an experiment to reduce variance and increase confidence in experimental metrics.
Just make sure you aren’t changing the metric after the fact. That’s an example of p-hacking, the process of chopping, cherry-picking, or disregarding data to try to get the result you want. When you p-hack, you can’t back up your results as statistically sound. Instead, make any changes to your metric before the test, and make sure your test is designed to work well with the metric you pick.
Transformations
Removing outliers
Standard deviation is not robust to outliers, so if we remove them, we can decrease it. For example, a simple way to define an outlier could be to say, “The top 1% of data are all outliers.” Then you’d just cut those from your metric.
You need to be careful with removing outliers, though, because there is no one correct definition of an outlier. Any pruning you do reshapes what it is you’re measuring. (If you keep removing outliers until you see the pattern you’re looking for, you’re just p-hacking.)
Another thing to be careful about is causing a sample ratio mismatch. The treatment may cause there to be more outliers than in the control, and if you remove those outliers, you’ll be left with an assignment ratio that doesn’t match your initial experiment design. A 2019 paper from researchers at Microsoft, Booking.com, and Outreach.io reported an incident where some users interacted with a treatment so much that they were flagged as bots and removed from the test, causing an SRM that made the treatment’s impact actually appear negative.
If you choose to prune outliers, you may want to include different thresholds or definitions in your analysis to show that your conclusions are not dependent on your definition of an outlier.
Winsorization
Instead of fully removing outliers, you can winsorize the data. That means you cap the outliers, saying something like y = min(x, 100). Any outliers that go above 100 now equal 100. A similar option is to cap the metric at the daily level, so saying, for example, that someone can only do the event 5 times in a day.
This method has the advantage that you aren’t throwing away data, so you’re less likely to skew your results than you are by cutting outliers entirely. You still need to be careful about the same issues, though.
One of the easiest methods to figure out where to cap is to plot a histogram and see if it looks bimodal, wherever the mode in the tail is very small. Then, pick the cap so that the second mode gets capped.
Converting to binary
What if instead of measuring dollars spent, you measure “dollars spent > 0”? This is essentially a special case of winsorization, saying that your event is capped at 1.
This is helpful because binary variables generally have a smaller standard deviation. A continuous random variable can have a standard deviation of any non-negative number. A binary random variable has a standard deviation of √(p(1-p)), where p is the probability of success.
This also has the additional advantage that the sample mean will converge to a normal distribution faster than if it were not transformed.
One drawback of the binary transformation is that if your experiment increases dollars spent > 0, that doesn’t necessarily mean you have increased dollars spent. Maybe more users are spending, but the overall level of spend is down. Also, you have collapsed a lot of information, since you are treating $3 and $100 as the same thing.
You can mitigate some of this downside by using the untransformed metric as a secondary metric, or monitoring other related secondary metrics like revenue per paying user.
Reducing skew
Sometimes, distributions have a large standard deviation because they’re skewed. If you can reduce the skew, then you can reduce the standard deviation.
For example, if you’re experimenting on how loud something sounds, measuring raw sound intensity will give skewed data because people’s perception of sound doesn’t follow intensity linearly. To get a more normalized relationship, you’d use a log transformation to turn the intensity into decibels.
Log transformations generally help when you have multiplicative effects instead of additive effects. Some other common transformations to reduce skew are taking a square root and the Box-Cox transformation.
Be careful with the transformation you pick, though, as they may have mathematical quirks. For example, the three skew transformations in the previous paragraph require your data to always be positive. If you have 0s or negative numbers in your data, you can add the minimum plus some small number to every data point. (Another alternative is to use the Yeo-Johnson transformation, which lets you evaluate negative numbers.)
Another issue is how to interpret your transformed metric. Maybe square rooting dollars spent makes the sample mean look more normally distributed, but what does it mean to increase the square root of dollars spent? Remember Jensen’s inequality, that E[f(X)] != f(E[X]). You cannot compute the treatment mean of the square root of dollars spent, and then square it to get the treatment mean of dollars spent. Your transformed metric is your new metric.
Pick a different metric
This method is a little more straightforward than lowering standard deviation with mathematical transformations: just find another metric. Look for a new metric that is highly positively correlated with the original metric but has a smaller standard deviation, or that is more likely to be impacted by the changes you are testing.
For example, say your ecommerce site is experimenting on its recommendation algorithm. Dollars spent may be the KPI you want to increase, but it may not be the best metric for your experiment. A user would have to first see the new recommendation, click on it, add it to cart, and then go through the whole checkout flow in order to register dollars spent. That’s a lot of touchpoints for them to bounce.
Instead, a better metric might be to look at the number of clicks on the recommendations, and extrapolate that more engagement means a better customer experience, which will eventually lead to more money. (As a secondary metric, you could also measure total dollars spent.) Even though making money is still the main goal, not every successful experiment will directly make more money or will have enough power to detect the change in money made.
Another example is testing different titles for blog articles to improve engagement. If you measure engagement as the number of comments on the article, it may not be the best metric since the title is so removed from commenting, and readers may not comment regardless of the article’s content. Clicks or time spent reading the article may be better metrics for the title’s impact.
Time-windowed metrics
Metrics that involve time deserve a special callout because they can lengthen your experiment runtime. That’s because you aren’t just tracking exposure, but also the entire tail after the exposure.
Say you have a metric like, “How much money was spent in the first seven days after first exposure,” or “Number of days with an app open in the first two weeks after the first exposure.” Your sample size calculator will tell you how many exposures you need for stat-sig, but you’ll need to add at least one unit of your time window to the experiment to get your first day of data. In the first example, you’d need to add a bonus week; in the second, two weeks.
CUPED
CUPED (Controlled Experiments Using Pre-Experiment Data) is a variance reduction technique first published in 2013 to improve the sensitivity of A/B tests. Essentially, CUPED takes pre-experiment data that is correlated with the experiment data and uses that information to reduce variance.
CUPED uses information about each user's behavior based on a specified time window prior to the experiment start date to normalize your experiment data and reduce the noise in the outcome metric. It’s kind of like looking at the delta of previous behavior to test behavior instead of just looking at test behavior.
How CUPED works is a complicated topic that could easily be the subject of its own post (and has been here, here, and here). If you want to try using it, though, it’s built right into Statsig.
The one area that CUPED struggles with is if you don’t have historic data, like when you’re experimenting on new users. Statsig developed our CURE methodology as a solution to this problem for warehouse native customers.
What else to try?
These strategies for adjusting your metric are a little more involved than the ones we discussed in the first post, but they’ll still be effective at reducing experiment duration.
If you’re new to metric transformations, Statsig has plenty of features to help get you started, like winsorization, logarithmic functions, and more.
If pulling these levers doesn’t get the time reduction you need, or the tradeoffs don’t work for you, there are other strategies you can look into that deal more with experiment design.
Post 1 covers how to adjust the inputs that go into your sample size calculator for faster experiments. In the next installments, I’ll go over ways to decrease runtime by changing how you target users or changing the type of experiment.