Odds ratios: Comparing binary outcomes

Ever stared at an odds ratio and wondered what the hell it actually means? You're not alone - these statistical measures trip up even experienced analysts, especially when someone asks you to explain why an odds ratio of 2.5 doesn't mean the risk is 2.5 times higher.

Here's the thing: odds ratios are everywhere in data analysis, particularly when you're working with binary outcomes like conversion rates, churn prediction, or A/B test results. But they're also one of the most misunderstood metrics out there. This guide cuts through the confusion to help you interpret, calculate, and communicate odds ratios effectively - and know when to use something else entirely.

The basics of odds ratios in binary outcomes

Let's start with what an odds ratio actually tells you. An odds ratio compares the odds of something happening in one group versus another group. If you're running an experiment on whether a new checkout flow reduces cart abandonment, the odds ratio tells you how the odds of abandonment compare between your control and treatment groups.

The calculation itself is straightforward: divide the odds in one group by the odds in the other. But here's where people get confused - odds aren't the same as probability. If 30 out of 100 users abandon their cart, the probability is 30%. The odds? That's 30 to 70, or about 0.43. This distinction matters because it affects how you interpret your results.

An odds ratio of 1 means there's no difference between groups. Values above 1 indicate higher odds in the first group, while values below 1 show lower odds. But - and this is crucial - the magnitude can be deceiving when your outcome is common. The PMC analysis on odds ratios shows this beautifully: when events happen frequently, odds ratios exaggerate the actual effect compared to what most people intuitively expect.

The real beauty of odds ratios? They're symmetric. Flip your comparison groups, and you just get the inverse. Compare men to women and get an odds ratio of 2? Compare women to men and you'll get 0.5. This property makes them particularly useful in certain statistical models, which brings us to logistic regression.

One last critical point: odds ratios don't prove causation. Just because users who see Feature A have 3x higher odds of converting doesn't mean Feature A caused the conversion. There could be confounding factors - maybe power users are more likely to see Feature A and also more likely to convert anyway. Adjusting for these confounders through stratification or regression analysis helps get closer to the true relationship.

Interpreting odds ratios in logistic regression

Logistic regression spits out coefficients, but nobody wants to interpret log-odds. That's why we exponentiate those coefficients to get odds ratios - suddenly, your model results make sense to humans. But there's a catch that trips up analysts all the time: odds ratios aren't risk ratios, even though people often interpret them that way.

Let's say you're analyzing user behavior data. For a binary predictor like "premium subscriber" (yes/no), an odds ratio of 0.704 means premium subscribers have about 30% lower odds of churning compared to free users. Notice I said "odds," not "risk" - this distinction matters more than you might think.

Continuous variables work differently. Take user engagement score as an example. An odds ratio of 0.051 for this variable means that each one-point increase in engagement score multiplies the odds of churn by 0.051. Put another way: higher engagement scores dramatically reduce churn odds - by about 95% per point increase. That's a massive effect, and it illustrates why checking your interpretations matters.

The real challenge comes when explaining these results to non-technical stakeholders. You run a test at Statsig and find that users exposed to a new feature have an odds ratio of 1.47 for making a purchase. How do you communicate this? Skip the statistics jargon. Just say: "Users who saw the new feature were about 47% more likely to purchase." Yes, this isn't technically precise (you're conflating odds and probability), but for rare outcomes, it's close enough and much clearer.

Here's a pro tip from discussions among data scientists: always report confidence intervals alongside your odds ratios. An odds ratio of 1.47 sounds impressive until you realize the confidence interval runs from 0.8 to 2.7 - suddenly, that "effect" might just be noise.

Odds ratios versus risk ratios: Understanding the differences

This is where things get interesting - and where many analyses go wrong. Odds ratios and risk ratios tell different stories about your data, and confusing them can lead to wildly incorrect conclusions.

Risk ratios (also called relative risks) compare probabilities directly. If 20% of control users convert and 30% of treatment users convert, the risk ratio is 1.5 - treatment users are 1.5 times as likely to convert. Simple, intuitive, and what most people actually want to know. Odds ratios for the same scenario? That would be 1.71, making the effect seem larger than it really is.

The gap between odds ratios and risk ratios depends entirely on how common your outcome is:

• Rare outcomes (under 10%): Odds ratios and risk ratios are practically identical

• Common outcomes (over 30%): Odds ratios can be 2-3x larger than risk ratios

• Very common outcomes (over 50%): The difference becomes extreme

This isn't just academic nitpicking. The analysis community has documented case after case where odds ratios led to overblown headlines about treatment effects. A weight loss study might report "3x higher odds of success" when the actual risk increase is only 50%.

So why use odds ratios at all? Two reasons dominate. First, case-control studies (where you start with outcomes and work backward) can only calculate odds ratios. Second, logistic regression naturally produces odds ratios, and it remains the go-to method for binary outcomes because it's robust and well-understood. As researchers have pointed out repeatedly, the key is being clear about what you're reporting and helping your audience understand the distinction.

Alternatives to odds ratios for binary data

Sometimes odds ratios just aren't the right tool. If you're analyzing conversion rates at Statsig and your baseline is 40%, reporting odds ratios will confuse more than clarify. Fortunately, you have options.

Log-binomial regression directly estimates risk ratios instead of odds ratios. It's exactly what it sounds like - regression for binary outcomes that gives you relative risks. The catch? These models can be finicky. Add a few interaction terms or random effects, and you might face convergence issues that make you want to pull your hair out.

When log-binomial models fail, modified Poisson regression offers a workaround. Yes, using Poisson regression for binary data feels wrong - it's designed for count data. But with robust standard errors, it produces valid risk ratios for binary outcomes. The statistics community has thoroughly vetted this approach, though it still feels like a hack.

Here's my practical advice for choosing your approach:

• Outcome is rare (under 10%): Stick with logistic regression and odds ratios

• Outcome is common (10-50%): Try log-binomial regression first, fall back to modified Poisson if needed

• Outcome is very common (over 50%): Definitely avoid odds ratios - they'll be misleading

The most important thing? Be transparent about what you're reporting. Nothing frustrates me more than reading "users were 3x more likely" without clarification of whether that's an odds ratio, risk ratio, or something else entirely. Your audience deserves clarity, especially when the distinction could change how they interpret your findings.

Closing thoughts

Odds ratios might seem like a statistical nuisance, but understanding them properly will make you a better analyst. They're not going away - logistic regression is too useful, and case-control studies require them. The key is knowing when odds ratios work well (rare outcomes), when they're misleading (common outcomes), and how to communicate them clearly.

Next time you're analyzing binary outcomes, take a moment to consider whether odds ratios are really what you want. If your outcome happens more than 10-20% of the time, seriously consider alternatives like risk ratios. Your stakeholders will thank you for the clarity.

Want to dive deeper? Check out UCLA's statistics resources for worked examples, or experiment with different approaches in R or Python. The best way to build intuition for these concepts is to work through real data and see how odds ratios and risk ratios diverge as outcome prevalence changes.

Hope you find this useful!