Products
Solutions
Resources
You may not put time into your hypotheses. But you should. Here's why.
When I finished my first ebook as a baby content marketer, I was excited to write the perfect subject line for its announcement email. I played with different A/B test puns, tweaked word choice for character count, and found the perfect emoji as the cherry on top. But when I handed it off to my campaigns team, they asked, “Do you want to test another one?”
I didn’t get what they meant. Were we collecting data on different phrases to see which resonated more? What would the hypothesis be? Was my pun just not very good?
“No hypothesis, your copy seems good. But we can A/B test subject lines as the send goes out and switch to whichever performs best.”
While that did take some of the pressure off wordsmithing the perfect subject line, it seemed odd to me to call that capability a test. Sure, in the colloquial sense, we were “testing” different options. And we had variants. We had a performance metric. What was clanging?
The problem was, without a hypothesis, it wasn’t actually an experiment.
Here’s why that is, and why you're missing out on the fruits of your experiments if you don't dedicate time to a hypothesis up front.
What a hypothesis is supposed to do
Your experiment needs a hypothesis, and not just because the word “hypothesis” is drilled into you from day one of elementary science class. As a former science teacher, I think science class is actually a good way to explore this idea, so please bear with my classroom imagery.
You first learn that a hypothesis is an answer to, “What do you think will happen?” It’s predictive, it’s bold, and it’s a good starting point to develop the causal thinking you need for scientific discovery. Does a toy car roll down a steeper ramp in less time? Hypothesis: I think it will. Let’s see if we’re right!
Next, you’re taught a more structured approach: “If I make X change to A variable, then Y change will happen to B variable.” That’s a little drier, but that’s because it’s clearly laying out the cause and effect you’re investigating. If I make this ramp steeper, then the car will roll down it in less time.
With a clear cause-and-effect relationship in your hypothesis, you can then clearly show how your experiment results support or refute it. That lets you develop a theory of the relationship between the variables: what the effect is, why it happens, and how you can use it to predict future outcomes. That’s the essence of the scientific method!
Why you may not care about hypotheses
Using science class as a frame of reference, it also makes sense why you may not think a by-the-book hypothesis is necessary.
As you get older, science class often shifts focus from scientific thinking to just collecting and analyzing data correctly. You don’t roll a toy car down a ramp to discover that g = 9.8 m/s²; you do it to get graded on your ability to demonstrate that assumed truth.
Because you’re “experimenting” on known relationships with known results, the form matters because of your grade, but the function doesn’t. Hypotheses turn from exciting predictions into busy work. (I’m allowed to say this because I’m a former science teacher.)
So it’s understandable why you may think that a hypothesis for your sprint’s latest A/B test is also just busy work. It seems like an exercise in formality that gets in the way of results. Even in Statsig, you could just type “hypothesis tbd” into your scorecard, and you’d still be able to run your test. Variant A, Variant B, causality, boring!
What you’re actually doing if you don’t have a hypothesis
The simple fact is, without a hypothesis, you can’t support whatever you think you’re learning from your test. Testing without learning is at best a lost opportunity, and at worst an active mistake.
If your experiment doesn’t have a hypothesis, what you’re really doing may be:
An exploration. You’re learning by observing. You don’t have a hypothesis because you aren’t trying to support or refute a causal relationship; you’re just seeing what will happen. This is totally valid. Exploring is fun! But to solidify any conclusions you make from your observations, you should experiment later.
Hedging your bets. You’re using the mechanics of a testing tool to ship multiple variants because you aren’t confident in what you’re shipping. You “test” to see what might succeed. Without a hypothesis, though, you won’t develop a theory about why one variant succeeds, and you won’t know how to repeat it. Your confidence won’t grow either.
Pokémon battles. You’re pitting two variants against each other, but you don’t really care why one wins. You just want to see something win. This is an action-oriented form of hedging, with the same pitfall that you can’t use the results to develop a predictive theory of success. Save the fighting for video games.
Post hoc theorizing. You have variants and a performance metric, so what do you need a hypothesis for? If there’s an effect, you can reason about a cause later, right? … That sound is every data scientist reading this falling out of their chair. You absolutely cannot form valid conclusions after the fact. In trying to come up with an explanation that fits your data, you may be using circular reasoning, or fishing for results, or committing any number of other logical fallacies. You’ll also have no way to prove your post hoc theory one way or the other.
How to make a good hypothesis
So you’re convinced to make rigorous hypotheses going forward, but you don’t know where to start? Don’t worry if you’re still scarred by high school physics. Here are some quick ways to start hypothesizing.
As you draft your hypothesis, a key thing to remember is that it needs to be falsifiable. That means it’s a statement that could be contradicted. If there isn’t a way to prove your hypothesis is wrong, you logically can’t prove it’s right either.
Beginner’s template
If I change [thing I want to change] by [way I want to change it], then [performance metric] will [go up/down].
Example: If I change the login flow by letting people sign in with Google IDs, then my returning user rate will go up.
Practitioner’s template
We believe that [change], for [target users], will [impact direction] the [success metric] from a baseline of X% to at least Y%, based on [data / rational insight]. We will measure this effect over [timeframe], with an MDE of Z%, and monitor [guardrail metric(s)] to ensure no harm or downside risk.
Example: We believe that adding Google IDs to the login flow, for 10% of average daily users, will increase the returning user rate from a baseline of 25% to at least 30%, based on the reduction in friction. We will measure this effect over two weeks, with an MDE of 5%, and monitor login error rate to ensure no harm or downside risk.
Statsig Hypothesis Advisor
The Statsig Hypothesis Advisor gives instant feedback on experiment hypotheses and will automatically flag anything you’re missing. Admins can also configure custom requirements, like strongly recommending a validation plan be mentioned.
This is disabled by default, but can be turned on from your Statsig AI project settings.
Here’s to hypothesizing
Some of this post has drifted into the theoretical, so I want to make sure to bring it back to grounded, practical matters. We’re not in science class anymore. We’re not getting graded. The only reason to do any of this experiment stuff is what we ourselves can get out of it.
If you sacrifice a solid hypothesis for the sake of expediency, you may be able to see that one variant outperforms another. But you won’t be able to defend why you think that happened, or predict how that performance could happen again, or connect how that result might be related to other important problems you need to solve.
Instead, take a stab at it! A hypothesis is a first idea. Say what you think might happen. See if it does. And repeat. You’ll learn something useful from the process, even if it’s that your hypothesis is wrong. Which odds are, it will be. But since we aren’t keeping score, odds are you’ll also start having fun with it. Maybe not as much fun as a baking soda volcano, but close.
PS: Yes, Isaac Newton did famously say, “Hypotheses non fingo,” or “I frame no hypotheses.” I’d argue he was talking about hypotheses as unverifiable causes, though, not the modern idea of a hypothesis. @ me on LinkedIn or Twitter if you want to talk more philosophy of science.