Scaling experimentation while managing costs

Swimply runs experiments across front-end changes like landing pages and checkout flows, as well as back-end changes on search and ranking algorithms.

Swimply looked at legacy platforms like VWO and Optimizely, but these tools weren’t meeting Swimply’s data science needs. As a growing startup, Swimply needed:

• a price-competitive solution
• a straightforward experimentation tool
• the ability to conduct quick and easy tests with minimal setup

The Solution: Full-stack feature flagging and A/B testing on Statsig

Swimply adopted Statsig for affordable and reliable full-stack experimentation. One of the key benefits of using Statsig was its seamless integration with Segment. This integration enabled Statsig to pull in Swimply’s Segment events, allowing them to run experiment analyses with all existing events from Segment without requiring any additional logging.

Once Statsig receives events from Segment, they become visible and aggregated in the Metrics tab in the Statsig console. These events are then automatically included in Swimply’s Pulse results for A/B tests using Statsig’s feature flags, as well as in all their Experiment results. This streamlined data management process allowed Swimply to focus on optimizing their platform and enhancing user experience.

Michael Sheldon, Head of Data at Swimply, emphasized the importance of trust and efficiency in their choice of Statsig: “It came down to being able to trust in Statsig, and having time savings along with that. There’s not a lot of things I have to configure or set up beforehand”.

In addition, Swimply supplements its Segment data with offline calculations, such as support tickets, ingested from their Snowflake warehouse. This comprehensive data management solution ensures that Swimply has access to all relevant data for their experiments and decision-making processes.

“It came down to being able to trust in Statsig, and having time savings along with that. There’s not a lot of things I have to configure or set up beforehand.”

Michael Sheldon

Head of Data

20% increase in bookings through optimized search results

By leveraging Statsig’s feature flags, Swimply was able to split traffic in a bandit framework between shuffled search results and an algorithm that predicts which listings users are most likely to engage with. This approach not only allowed Swimply to establish a baseline for their predictive model’s performance but also, as Michael Sheldon highlights, “more importantly, it allows us to get unbiased training data for that model.”

Using shuffled search results for a portion of users as training data helps maintain the accuracy of Swimply’s model. This unbiased data reflects the overall user population and allows for continuous improvements in the recommendation algorithm without being influenced by the algorithm’s own predictions.

With Statsig’s feature flags, Swimply can effortlessly change traffic allocations. As their data team gains more confidence in a variant, they can quickly modify allocations to that model without needing any coding from the engineering team. As Michael puts it, “This granular control of data has been super helpful to us.”

This process has led to a substantial 20% increase in bookings for Swimply, demonstrating the power of combining data-driven insights with a user-friendly experimentation platform like Statsig.

Direct access to Statsig’s expert team

One aspect that sets Statsig apart is its exceptional support system, which is backed by the company’s own engineering, data science, and product teams. Instead of relying on generic customer service representatives, Swimply enjoyed direct access to the very experts who built and maintained the Statsig platform.

This level of support ensured Swimply received accurate, timely, and comprehensive assistance with any questions or concerns.

Swimply benefited from both Statsig’s public-facing community Slack and the dedicated Slack channels set up for enterprise customers. These channels provided a convenient way for Swimply to get quick answers from the Statsig team, speeding up the integration process and helping them make the most of the platform’s features.

Michael Sheldon describes his relationship with Statsig’s support system: “Having a dedicated Slack channel and support was really helpful for ramping up quickly.” The seamless communication with Statsig’s expert team allowed Swimply to focus on running experiments and optimizing their platform, rather than dealing with technical roadblocks.

In addition to the prompt support, Statsig’s hands-on approach meant that Swimply didn’t require a dedicated team for integrating the platform. As Sheldon mentioned, “We don’t have a dedicated team for integrating Statsig—we just have some hours of engineering time” This demonstrates the efficiency and effectiveness of Statsig’s support, making it an invaluable asset for Swimply and other customers seeking to leverage the platform’s capabilities.

“Having a dedicated Slack channel and support was really helpful for ramping up quickly.”

Michael Sheldon

Head of Data