Platform
Resources
Runna runs 100+ tests within a year to drive user growth
100+ experiments
Drove key growth metrics
The early days of user growth
Since launching in 2021, Runna quickly became a favorite among runners worldwide, and the need for experimentation naturally arose as they saw exponential growth in new users.
“We were building based on user feedback but wanted something that was data-backed to scale the business,” explained Meehir Patel, Senior Software Engineer on their growth team.
“As a consumer business, building stickiness and loyalty is critical.”
Taking the first steps toward data-driven product development
Runna knew they first needed to implement a feature flagging solution for launch control, with metrics attached to every release.
Fun fact: Before using Statsig’s feature flags, they used secret gestures to unlock and test hidden features!
Meehir recalled previous roles where he had used legacy tools: “I’ve used legacy A/B testing tools in the past, but I would spend so much time checking details like data skew and making sure exposures were correct.”
He shared why Runna chose to buy a platform rather than build one in-house: “We want to focus on what we’re good at and do it really well. Our teams focus on which experiments to launch and diving deeper into insights.”
“With Statsig, we can launch experiments quickly and focus on the learnings without worrying about the accuracy of results.”
Meehir Patel
Kelly Bennetts, a full-stack developer on their growth team involved with implementing Statsig, shared that it was easy to get started and build a source of truth for experiments.
Kelly Bennetts
From 0 to 1 to 100+ tests within a year
Runna quickly onboarded their engineers across all product teams and ramped up their testing velocity, using Statsig across their website and mobile app.
They’ve run a range of experiments across the customer journey: not just focused on improving the UI, but also improving the backend (running) algorithms and driving revenue growth.
Meehir is especially proud of how the experimentation culture quickly spread across teams. “Experimentation has become self-serve now, and teams ship features behind feature gates by default. We want to ensure our customers always get the best experience.”
Kelly looked back on their progress within a year. “We started from zero and are now tracking 30 million events monthly. In Q3 of 2024, just the growth team tested over 60 experiment variants, and our velocity continues to increase.”
Running experiments to drive retention and growth
Runna’s growth team primarily focuses on building a product that runners love, while driving engagement and retention.
They run experiments aimed at improving the following core KPIs:
- Premium user conversion
- Onboarding metrics like ‘Week three completed’
- Retention
The onboarding experience is a key driver of subsequent usage, conversion to premium plans, and long-term retention. The growth team focuses their time on developing ideas to test and uses Statsig to validate which features positively impact metrics.
For example, they launched an experimental modal to nudge customers to sync their training plan to their calendar. 3x more users enabled the feature, making them more sticky.
“We saw a huge uplift in retention metrics,” Meehir noted.
Meehir Patel
Embracing the unexpected results
Meehir and Kelly recalled several experiments that revealed surprising results, unlocking new insights about their users that couldn’t have been discovered otherwise.
One example was when they wanted to ask users to input their age during sign-up.
“We initially thought that adding an extra step during onboarding would create friction and lower the activation rate. However, a better-designed version of the age input actually led to a significant increase in the metric. We learned that our users valued personalization, and we needed to make our app experience reflect that. That’s the kind of insight we want from our experiments.”
Leveraging advanced experimentation techniques
As Runna increased experimentation velocity, they implemented various testing techniques to tackle a wide range of use cases:
-
Continuous optimization with multi-armed bandits: The growth team tested different copy variations on the sign-up page, using Autotune (Statsig’s multi-armed bandit solution) to dynamically allocate traffic – explore various text options and exploit winning variants.
-
Measuring success with holdouts: Every team has its own holdout group so they can measure the success of the features they shipped over time.
“We track the uplifts from experiments via holdouts and automating this has been a game changer. If we had to do this manually, we’d have to figure out so many things. With Statsig, we can focus on writing code and the experimentation comes as a layer.”
Meehir Patel
Building the best running app
A core aspect of Runna’s app is its backend running training ecosystem which has various modeling components, many of which are powered by machine learning. Statsig helps them roll out new versions of the engine with confidence.
By using dynamic configs, the team can continuously optimize the engine – quickly adjusting various inputs and testing different versions. For example, updates to their pace state modeling, a core component across all training plans, are being gradually rolled out while measuring their impact.
Harry Pratten, Engineering Lead on the Train Team, shared that they tested and launched several key features in the app using Statsig after measuring impact on user engagement:
Three notable features include:
- Post-workout insights
- Holiday mode
- Get fit plans
Kelly Bennetts
Runna moves faster with Statsig
Kelly commented, “Statsig has become a core part of our development process and just runs in the background. We use it daily and take it for granted.”
Meehir shared his favorite features: “It’s been really powerful to easily run multivariate tests, where we can test five iterations of one screen in a single function and see the experiment results in one window.”
“Statsig releases so many new features that we’re discovering all the time. We recently tried Sidecar (no-code website testing) and loved it.”
Runna continues to grow its user base and empower runners around the globe with experimentation playing a key role in their product development.
About Runna
Runna is a personalized running coach with tailored training plans to help you achieve your goals, from running a faster 5k to finishing your first marathon!