This article is a guest contribution by authors Mengying Li and Joe Mercer.

As professional data scientists, one of the questions we get asked most frequently by founders is: “How do I know if my product/business is moving in the right direction?”

Founders are typically well-versed in the stages of product development. They know what direction their product/business should be moving in. And—if they don’t—there are many wonderful resources online that can provide clear and concise descriptions of these stages.

Where we observe the most anxiety from founders is in determining whether their product is on the right trajectory to make it through these stages: “Is my product/business on the path to success, or should I be investing my resources elsewhere?”

Or, as The Clash so succinctly put it: “Should I stay, or should I go?”

In this note, we break the product development roadmap into distinct stages, and dive deep into each stage to help you decide whether your product/business is ready for the next stage.

Should I stay, or should I go?

When developing new products, you face an inherent tradeoff between wanting to move both quickly (i.e., with minimal time, resources, effort, etc.,) and confidently (i.e., while maximizing the chance that the product will succeed in the future).

If you move with excess speed and insufficient confidence, you risk wasting resources on an idea that ultimately busts. If you move with insufficient speed and excess confidence, you risk wasting resources on an idea that would have succeeded without them.

To help balance the need for both speed and confidence, we recommend subdividing the product development process into stages, and then gating each stage by...

1. A runway: A maximum threshold of time/resources that you’re willing to spend in a stage before calling it quits

2. A set of exit criteria: A minimum threshold of success that you need to meet before moving on to the next stage

In each stage of the product lifecycle, the goal is to meet the exit criteria before hitting the end of the runway.

In the early-stage startup world, your runway may be defined by literally how long you can keep the lights on.

However, in larger companies (and even in the early-stage startup world), you should limit your runway to a fraction of your total available resources. This enables you to split your resources among many product/businesses (whether these are pursued iteratively or in parallel is up to you).

As the grannies of the world would say: “Don’t put all your eggs in one basket!”

Overview

Below is a summary of different stages with some example metrics. The exact metric definition and benchmark for each stage should depend on the nature of the product. We will discuss how to on the appropriate benchmark and metric in a follow-up post.

Please note that the “typical duration” mentioned below is based on our observation and estimation. However, it can vary significantly depending on the nature and scope of each product and its connection to the overall product ecosystem of the company.

Stage 1: Concept development

The objective of the concept development stage is to identify a sufficiently sized product-market opportunity.

In this stage, you typically...

• Conduct research to identify a market, and understand customer pain points in that market. This research can include both qualitative methods (competitor analysis, UXR sessions) and quantitative methods (surveys, data analysis).

• Brainstorm solutions to customer pain points. Design product mocks, and run lab studies to gather feedback. Iterate. Through the design process, try to identify the key components of the end-to-end product experience (which we call the minimum viable product, or MVP).

• Design an alpha test plan. The alpha test plan should identify the key assumptions underlying the future success of the product, and show how the MVP will provide conclusive feedback on the validity of those assumptions.

  • For example, a key assumption of Meta’s paid online events was that people would be willing to purchase tickets to online events. In the alpha test plan, we drew a line in the sand of how many people needed to purchase a POE ticket to validate this assumption.

• Estimate the cost of building and testing the MVP. How many Engineers? How much build time? What ideas are you not pursuing because you are pursuing this idea?

• Estimate the opportunity size (e.g., $X over Y time frame). Why does your product have a right-to-win in this market? What are the risks? Some common strengths include scale, technical excellence, and existing latent behaviors. Some common risks include integrity, fraud, competition, and regulation.

At the end of the concept development stage: review your research, product design, and test plan with key stakeholders, and make a go/no-go decision on moving to the MVP stage.

Runway:

• Duration: typically 1-3 months

Exit criteria:

• Alignment from stakeholders on the MVP designs and alpha test plan

• Gut check: Are you excited about building and testing this idea?

Stage 2: Minimum viable product (MVP)

The objective of the minimum viable product (MVP stage) is to build and ship an end-to-end product, and run an alpha test to show that it works for a set of hand-picked customers.

In the MVP stage, you typically...

• Build the MVP. Ensure that you have comprehensive logging to validate the key assumptions from the alpha test plan, and understand any gaps that materialize. For example, if people aren’t clicking on the product link, then you should have logging in place to understand why.

• Identify and recruit customers for the alpha test. Target the customer persona that has the biggest pain points your product is designed to address. If you can, find customers from reputable sources you can highlight as a customer success story. At this stage, it’s likely you’ll be giving your product away at a discount (potentially for free).

• Gather continuous feedback from your customers. This should be part of the “deal” that you make with customers in return for selling your product at a reduced rate. You should be analyzing behavior logging, but also collecting feedback from qualitative sources like customer reviews and interviews.

• Define what it means for an alpha tester to be “successful.” Success metrics at this stage typically rely on qualitative feedback from the customers themselves. For example, you might ask: “Would you recommend this product to a friend?” If your product is something customers would purchase, this is a good time to gauge the price point.

• Run the alpha test, and analyze the results with the alpha test plan.

  Did the test validate our key assumptions? What did we learn? Iterate as necessary.

At the end of the MVP stage: review your alpha test results with the “success” criteria, and make a go/no-go decision on moving to the PMF stage.

Runway:

• Duration: typically 3-6 months

• Test size: typically 10-50 customers

Exit criteria:

• Proportion of successful alpha testers: e.g., at least 50% of alpha testers are “successful” (wrt to “success” as defined in the alpha test plan)

• Gut check: Do alpha customers seem excited to continue using the product?

Stage 3: Product-market fit (PMF)

The objective of the product-market fit (PMF) stage is to validate that the product generates incremental value for a large enough and cohesive enough market.

We recommend approaching this objective iteratively, further subdividing the PMF stage into three or more milestones (beta, scaled beta, etc). Each milestone should unlock a bit more capacity for growth, while simultaneously raising the expectations for success. In this way, you can manage a graceful rollout through iterations of market testing.

In the PMF stage, you typically...

• Continue to refine the product based on feedback from testing. Note that, in the PMF stage, sample sizes are typically small, and variance high, so product design should be a mixture of art and science.

• Continue to recruit customers to test the product. Choose customers from the target market based on the likelihood that they’ll both adopt and succeed with the product, which should be a natural extension of successful customer personas.

  • We recommend releasing your product to customers in blocks, called cohorts. Each cohort defines a group of customers that were invited to use the product at the same time (or, occasionally, defines a group of customers that onboarded to the same version of the product).

  • For each cohort, measure onboarding, activation, and retention. The onboarding rate is the proportion of customers that follow through with all onboarding steps. The activation rate is the proportion of onboarded customers that go on to use the product. The retention rate is the proportion of customers that use the product in subsequent time periods (usually 28d). Together, these metrics should determine the pace of your market testing. When these metrics rise, push towards the next stage. When they fall, slow down and re-evaluate your strategy.

• Build a reporting flow for complaints and feedback. Define a set of “red flag” events that are indicative of notably negative customer experiences. Common red flag events include: requesting a refund, experiencing infra problems (e.g., app lagging/crashing), or sending negative feedback. You should pay close attention to red flag events, and pause rollouts if you see their frequency starting to spike.

At the end of the PMF stage: review your beta test results with the “success” criteria, and make a go/no-go decision on moving to the PMF stage.

Runway:

• Duration: typically 3-12 months

• Limited number of customers

Exit criteria:

• Customer adoption: e.g., at least 500 L28 active users

• Customer success (”stickiness”): e.g., at least 20% L28 user retention

• Customer experience guardrail: e.g., less than 2% of users with a “red flag” experience

• Gut check: Have you found PMF? Do you know where your next N customers will come from?

Stage 4: Optimization

The objective of the optimization stage is to prepare your product for quick growth. This is also the stage where—if your product is going to be part of a larger product offering or ecosystem—you should verify that it provides incremental value to the overall experience. For this reason, Julie refers to the optimization stage as the reconciliation state. She writes…

“Remember that people don’t see your company as a collection of products and orgs, they see it as one holistic experience. When you don’t reconcile well, your product starts to feel more complicated and confusing.”

In the optimization/reconciliation stage, you typically...

• Continue refining the product. In this stage, product improvements typically start to shift from visible elements (e.g., product design) to behind-the-scenes elements (e.g., model tuning, infra, performance, reliability).

• If your product is part of a larger product offering, then show that it has a positive and incremental impact on the overall ecosystem. If possible, you should show this by running an experiment (called a “holdout”) where the test group sees a version of the product ecosystem with your product, and the control group sees a version of the product ecosystem without your product. If this is not possible (perhaps it is not possible to limit access to your product, perhaps you do not have adequate power, etc), then a mixture of quantitative and qualitative methods can suffice.

• Identify risks associated with scaling the product, and guardrails in place to manage these risks. Common guardrail metrics include sentiment, retention, performance, and any “red flag” event from the PMF stage.

• Scope out a growth trajectory for the product. How many customers? What market size and share? What assumptions does this growth trajectory make? How much infra? How much customer support?

• Develop promotional material for GTM (go-to-market). Identify customer success stories whose stories could inspire further adoption.

The goal for teams in the optimization stage is typically twofold: (1) showing capacity for quick growth, and (2) showing incremental contribution to the product ecosystem.

Runway:

• Duration: typically 3-12 months

• Limited number of customers

Exit criteria:

• Optimized product experience: e.g., a 20% increase in meaningful time spent.

• Infra readiness: e.g., capacity for at least 10x growth

• Reconciliation: e.g., proven incremental value to the ecosystem (as measured in a product holdout, or otherwise)

• Gut check: Do you feel the product/infra is ready to expand to greatly more customers? Do you feel the product adds incremental value to the ecosystem?

Subscribe to Mengying on Substack

Mengying talks about data tooling, data careers, and making data-informed decisions.

Read now

Stage 5: Growth

The objective of the growth stage is to grow the product as rapidly as possible while continuing to monitor the health of the product ecosystem.

In the growth stage, you typically...

• Launch the product! Run a robust GTM tailored to each major customer segment. Employ growth levers to drive trial of the product across customer touch points: outbound reach etc.

• Identify and optimize the product’s growth flywheel and virality path. In the PMF stage, your growth focused on enabling the core market. In the Growth stage, however, your growth will come from scaling beyond the core market.

• Consider options for vertical or horizontal expansion. What other upstream or downstream use cases does this market have? What other markets could this product expand into? Could the product be generalized into a platform?

• Develop growth partnerships. Hold these partners accountable with shared growth goals.

The goal for teams in the growth stage is typically YoY Revenue growth. Teams should continue in this stage for as long as they can. However, once they feel they can no longer provide incremental growth for the product, then they should move on to the maturity stage.

Runway:

• Duration: 12 months - Unlimited

• Budget for growth levers as efficient

Exit criteria:

• Diminishing ROI: e.g., when revenue growth falls below 20% YoY, or when other investment opportunities become relatively more compelling

• Gut check: Does the team feel they can no longer drive incremental growth? Will the product sustain a healthy growth trajectory on its own?

Stage 6: Maturity

The objective of the maturity stage is to sustain the product’s organic growth trajectory while reducing the effort needed to maintain the product. By the end of the maturity stage, the product should be able to operate almost autonomously (i.e., with minimal oversight needed from the product team).

In the maturity stage, you typically...

• Continue monitoring YoY revenue growth and ecosystem health metrics.

• Improve the maintainability of the product: clean up the code base, reduce SEV rates, polish product-help pages, etc.

• Reduce investment in the product. As the effort needed to maintain the product decreases, then people should start spinning off to work on new projects.

The goal for teams in the maturity stage typically continues to be revenue growth, although the emphasis shifts from growth acceleration to growth maintenance. Products may exist in this stage for an indefinitely long time period, however, once the product becomes obsolete, then the team should move on to the deprecation stage.

Runway:

• Duration: Unlimited

• Limited resource investment

Exit criteria:

• Negligible contribution to the business or product ecosystem: e.g., less than 0.1% of overall Revenue, less than 0.1% of product usage

• Gut check: Would people on the team rather be working on other things?

Stage 7: Deprecation (Optional)

The objective of the deprecation stage is to deprecate the product and provide a graceful transition for users to new products. If you fail to deprecate obsolete products, then your product experience will become overly complex, and hinder your ability to innovate further.

In the deprecation stage, you typically...

• Decide to deprecate the product. Run a cost-benefit analysis. Generally, the costs are short-term revenue loss, while the benefits are product simplification and opportunity for new innovation.

• Develop a transition plan for customers. Provide ample time for users to react and adjust.

• Deprecate the product.

Runway:

• Duration: typically 3-6 months

Exit criteria:

• No more product usage; no further product maintenance required

• Gut check: Does the team feel like this product is no longer disrupting further innovation?

Follow Joe Mercer on Medium

Joe Mercer is a data scientist who covers topics varying from data science and tooling to cryptocurrencies.

Read now

In closing

In the life cycle of every product, there's a passage through different stages of development, optimization, and sometimes, retirement. From the initial flutters of an idea during concept development to the final bows in the deprecation stage, products encounter various challenges and opportunities, all of which demand strategic decisions and precise execution.

The comprehensive exploration of the seven stages above is a reflection of this journey—a roadmap to guide through the meticulous planning, building, refining, growing, and sometimes, saying goodbye to the products that once were pivotal.

Navigating through each stage with agility, empathy for users, and a staunch commitment to delivering value ensures that the lifecycle of a product, from its nascent stages through to its sunset, is a journey that enhances organizational learning, user experiences, and the progressive evolution of the business itself.

So hopefully, the next time The Clash is playing in your mind, this guide will serve as a reference.

Follow Statsig on Linkedin

Follow us on Linkedin to be the first to see company updates, experimentation content, and more!

Take me to Linkedin