Natural experiments: Learning from changes

Ever tried to test whether a new policy actually works, only to realize you can't exactly randomize half your users into a parallel universe? Yeah, that's where natural experiments come in handy.

These clever research approaches let you evaluate real-world interventions by finding situations where nature (or policy makers) essentially did the randomization for you. Think of it as experimentation without the guilt of messing with people's lives.

Understanding natural experiments

Natural experiments are basically the research equivalent of making lemonade when life gives you lemons. Instead of setting up a controlled experiment where you randomly assign people to different groups, you look for situations where something in the real world created that variation for you.

Here's the deal: sometimes you just can't run a traditional A/B test. Maybe it's unethical (you can't randomly deny healthcare to people), impractical (you can't control who moves to which city), or stupidly expensive (you can't build two identical highways to test traffic patterns). Natural experiments step in when these constraints make controlled experiments impossible.

The classic example? Look at what happened when different states rolled out smoking bans at different times. Researchers didn't assign states to "ban" or "no ban" groups - the states did it themselves. But by comparing health outcomes before and after, and across state lines, you can actually tease out the causal effects of the policy.

The beauty is that you're studying real people in real situations, not lab rats or paid study participants who know they're being watched. This gives your findings what researchers call "external validity" - fancy talk for "this actually applies to the real world."

But let's be honest: natural experiments aren't perfect. You're basically working with whatever variation the universe handed you, which means you need to be extra careful about confounding factors. Just because two things happened at the same time doesn't mean one caused the other.

The power of natural experiments in evaluating interventions

Natural experiments really shine when you're trying to evaluate complex interventions in messy, real-world settings. Take healthcare, where policies often roll out in waves or affect different populations differently.

Walkey, Bor, and Cordella pulled off something clever with hospital readmission programs. They noticed that hospitals used arbitrary risk-score cutoffs to decide who got extra care coordination. Patients scoring 0.69? No intervention. Patients scoring 0.70? Full treatment. That tiny difference in scores created an almost-random assignment, letting them study the intervention's true effects.

This approach works particularly well in dynamic environments where things are constantly changing. During the COVID-19 pandemic, different regions implemented lockdowns at different times and intensities. Researchers could compare outcomes across these natural variations to understand which policies actually helped.

The real power comes from combining these natural variations with modern data infrastructure. When you have detailed health records, economic data, or user behavior logs, you can track outcomes precisely across naturally-occurring treatment and control groups. Companies like Statsig help teams identify and analyze these natural experiments in their own data, turning unplanned variations into learning opportunities.

What makes this especially valuable is that you're seeing how interventions perform in the wild, complete with all the complications and edge cases that controlled experiments might miss.

Applications of natural experiments across various fields

Natural experiments pop up everywhere once you start looking for them. Each field has its own flavor of "accidental randomization" that researchers have learned to exploit.

Economics loves natural experiments because you can't exactly crash an economy just to see what happens. Some greatest hits include:

• When East and West Germany reunified, creating a massive natural experiment in market integration

• Lottery winners providing random income shocks to study spending behavior

• Hurricane paths creating random destruction patterns to study economic recovery

Public health researchers use them to evaluate policies that affect entire populations. Obesity prevention programs often roll out city by city, creating natural comparison groups. When Philadelphia implemented a soda tax but nearby counties didn't, boom - natural experiment.

Biology gets in on the action too. While you might think of biology as all controlled lab experiments, natural variations help us understand evolution in real-time. The rapid evolution of viruses during outbreaks creates natural experiments in adaptation. Different viral strains competing in different populations? That's data gold.

Even tech companies use natural experiments. When a feature launches in one country before another, or when server outages affect random subsets of users, these "accidents" become opportunities to measure real impact. Statsig's experimentation platform helps teams spot these natural variations in their metrics and turn them into insights.

The key insight across all these fields? Real-world messiness isn't always a bug - sometimes it's a feature that helps us understand cause and effect in ways controlled experiments never could.

Challenges and considerations in conducting natural experiments

Let's talk about why natural experiments can be a pain to work with. The biggest headache? You don't control the randomization, so you're constantly playing detective to figure out if your "natural" groups are actually comparable.

The checklist of things that can go wrong:

• Selection bias (maybe healthier people moved to states with better policies)

• Timing issues (did the economy tank right when your policy launched?)

• Spillover effects (people crossing state lines for cheaper cigarettes)

• Multiple changes at once (new tax law AND new health policy? Good luck untangling that)

Researchers emphasize that you need to be obsessive about checking your assumptions. Are the treatment and control groups really exchangeable? Did people manipulate their way into the better group once they knew about the policy?

Generalizability is another beast entirely. Just because something worked when Oregon expanded Medicaid doesn't mean it'll work the same way in Texas. Different populations, different healthcare systems, different results. Context matters way more in natural experiments than in nice, clean randomized trials.

The data requirements can be brutal too. You need enough historical data to establish baselines, detailed enough measurements to spot confounders, and large enough sample sizes to detect effects amid all the noise. This is where modern data infrastructure becomes crucial - without good data systems, natural experiments are just educated guessing.

But here's the thing: despite all these challenges, natural experiments often give us the only ethical way to study important questions. You work with what you've got, control for what you can, and stay humble about what you don't know.

Closing thoughts

Natural experiments occupy this sweet spot between "perfect but impossible" randomized trials and "easy but useless" observational studies. They let us learn from the chaos of the real world while still making credible causal claims.

The key is recognizing opportunities when they arise. Whether it's a policy rolling out gradually, a system glitch affecting random users, or any other source of unplanned variation - these "accidents" are actually gifts to the analytically minded. Tools like Statsig can help you spot these natural experiments in your own data and extract meaningful insights from them.

Sure, they're messier than controlled experiments. You'll spend more time defending your assumptions and checking for confounders. But when you can't ethically or practically randomize, natural experiments give you a fighting chance at understanding what actually works.

Want to dive deeper? Check out:

• Capitalizing on Natural Experiments for a framework on identifying opportunities

• BMJ's guide on evaluating health interventions for practical examples

• Your own data - seriously, you probably have natural experiments happening right now

Hope you find this useful!