Building features centered around design and UX

Notion’s product culture has been instrumental in distinguishing itself in the competitive SaaS market for collaboration tools. Wendy Jiao, a Software Engineer on the data platform team responsible for integrating the SDK and ensuring seamless metric recording on the backend, succinctly captures what sets Notion apart. “Notion is not just a tool; it’s a dynamic collaboration platform that empowers users to tailor workflows and unlock their productivity.” Her role empowers data scientists and involves close collaboration with various vendors, including Statsig.

But what catalyzed the creation of something truly distinctive? Mengying Li, a Data Science Manager who collaborates with various teams to drive data-driven business decisions, says, “Our culture revolves around design and prioritizes user experience. We embrace innovative technologies that amplify our platform’s capabilities.”

The role of experimentation in Notion’s product development

At Notion, the emphasis is on rapid feature deployment without compromising long-term value. “We maintain a very agile development cycle, but our standards for feature releases remain very high,” says Mengying. "In this dynamic environment, engineers enjoy the freedom to experiment, but the spotlight is on learning rather than quick wins.”

Data-driven decision-making is the cornerstone of their product development strategy, and early on, Notion acknowledged the importance of effective feature flagging and experimentation.

Before Statsig: The challenges of an in-house experimentation platform

However, they faced a few challenges in running exclusive experiments. “Before we got Statsig, we found it time-consuming to add feature flags to every feature and couldn’t run experiments without overlaps”, said Mengying. As one of the first data scientists on the growth team, she took on the task of exploring third-party vendors to solve the challenges of their existing in-house experimentation tools. “We couldn’t run many experiments with our in-house tool, which slowed us down. It didn’t make sense for us to spend time building a platform by ourselves,” said Mengying.

Key challenges

• Engineering resource constraints to scale their in-house tools
• Limited data science resources to analyze a large volume of experiments
• Inability to run multiple experiments in parallel due to inefficient traffic allocation

Mengying also highlighted specific pain points from a data science perspective:

• It was taking too much time to locate the correct feature flag, which delayed analysis
• There was a lot of manual work such as performing statistical analyses, p-value calculations by themselves

Mengying adds, “It would take 2 months from set up to launch to analyze an experiment and it was very hard to debug.”

Why Notion embraced Statsig

In Notion’s quest for an ideal third-party experimentation tool, Mengying outlined two fundamental guiding principles: “First, we needed a tool capable of effective feature flagging, and second, it had to empower both data scientists and non-technical users to self-serve experiment insights.”

After conducting an exhaustive evaluation of available market options, Notion selected Statsig for several compelling reasons:

• Pricing flexibility: Statsig’s pricing model (which includes no fee for seats) ensured that a wider spectrum of Notion’s team, including non-technical users, can access and benefit from experiment data.

• Seamless integration: “The integration process was seamless, tailored precisely to Notion’s unique requirements,” Wendy explained. “Statsig demonstrated dedication to building features that aligned with Notion’s needs.”

• Holistic metrics analysis: Statsig’s “scorecard view” for experiments provided an invaluable holistic perspective on metrics. As Mengying elucidated, “Notion’s data analysis spans a range of product metrics. With Statsig, we gained the ability to review all metrics within a single scorecard view, a stark contrast to other tools that required us to navigate through individual metrics one at a time.”

Above all, what set Statsig apart for Mengying and Wendy was the exceptional experience of collaborating with the Statsig team. Mengying elaborated, “We had unique requirements concerning SDK integration and metric ingestion that the team at Statsig understood well. They proactively developed features based on our specific requests.”

“Statsig’s people and vision aligned with our needs,” Wendy added. “The team exhibited remarkable dedication, humility, and responsiveness, which instilled in us the confidence to forge this relationship.”

Statsig became an important tool in Notion’s data ecosystem, which already included Snowflake and Amplitude, thus augmenting Notion’s data capabilities with powerful A/B testing and experimentation at scale.

Achieving 30X experimentation velocity

"We transitioned from conducting a single-digit number of experiments per quarter using our in-house tool to orchestrating hundreds of experiments, surpassing 300, with the help of Statsig."

Mengying Li

Data Science Manager, Notion

Statsig emerged as a game-changer for Notion, enabling the efficient execution of experiments across the board. Notion began leveraging Statsig for a broad spectrum of experiments, encompassing everything from refining onboarding processes to enhancing activation and boosting engagement.

Notion’s experimentation journey witnessed a remarkable transformation, surging from single-digit experiments to over 300, incorporating hundreds of feature flags and dynamic configurations. Mengying underscored the remarkable transformation in their experimentation velocity, stating, “We transitioned from conducting a single-digit number of experiments per quarter using our in-house tool to orchestrating hundreds of experiments, surpassing 300, with the help of Statsig.”

This exponential expansion in experimentation hasn’t necessitated a proportional increase in engineering support. Remarkably, just one engineer effectively manages the platform—a testament to its user-friendly nature. Wendy quantified the impact on productivity, stating, “The gains are substantial: a single engineer devotes around 5% of their time covering Statsig SDK integration, as well as metrics ingestion pipelines. Otherwise, we might have required a full team of four engineers.”

Driving core growth metrics with Statsig

"We've successfully launched over 600 features by deploying them behind Statsig feature flags, enabling us to ship at an impressive pace with confidence."

Wendy Jiao

Software Engineer, Notion

Notion effectively drives four key growth metrics throughout the customer journey via experimentation on the Statsig platform:

• Sign-up rate (new users)
• Activation rate
• Engagement (active use)
• Retention

Here’s a selection of notable use cases:

1. Using holdouts to measure aggregate impact: Leveraging a holdout group within Statsig, Notion’s growth team executed a series of experiments aimed at enhancing the activation rate by 10-15%, subsequently bolstering engagement and retention rates. Mengying highlighted the remarkable level of control they wielded in designing and executing experiments. “Statsig allows us to use different units for experiments, such as user IDs and device IDs, a feature not widely supported by many platforms.”

2. Uncovering unexpected yet impactful learnings from experimentation: An interesting experiment entailed randomizing the order of workspaces that were available for a new user to join during the onboarding stage, providing crucial insights that wouldn’t have been discovered otherwise. Such experiments provided valuable insights for Notion, leading to significant enhancements to the product such as the mobile interface and the sign-up flow.

3. Protecting against metric regression: Wendy, emphasizing the platform’s impact on protecting against regression of core metrics, stated, “We’ve successfully launched over 600 features by deploying them behind Statsig feature flags, enabling us to ship at an impressive pace with confidence.” She also lauded the level of control it granted engineers, highlighting the effectiveness of dynamic configs in optimizing code at runtime.

4. Improving new user onboarding experience: Notion harnessed Statsig’s capabilities to fine-tune its onboarding process. They iterated on the sequence of questions presented to users, carefully selecting the order in which they were asked during onboarding to maximize the information gathered and personalize the user experience. This strategic approach significantly contributed to improving the success of the onboarding and activation processes.

A thriving experimentation culture and the path forward

“We are an experimentation-driven company, and our leadership embraces this vision. We routinely evaluate the holdout and report our progress to the finance and leadership teams on a monthly basis,” asserts Mengying Li.

While the Growth team was the first team to embrace the experimentation culture, thanks to the convenience of self-serving experiment insights provided by Statsig, more teams are starting to use experiments in their product decision-making process. Notably, as Statsig interfaces seamlessly with customer.io, marketing teams have also actively engaged in running experiments, particularly in email campaigns.

The organization is progressively becoming more self-sufficient as the culture of experimentation permeates various teams. Wendy shares her vision for the future, stating, “Our aim is to continue advancing our self-serve capabilities in experiments, accelerating the pace of experimentation across all teams.”

Notion’s overarching vision is clear: to move swiftly, learn continuously, and foster innovation. They’ve strategically embraced tools like Statsig to expedite experimentation across all their teams, ensuring that everyone can autonomously contribute to Notion’s success.