The way people consume content has fundamentally changed. Rei do Pitaco understands this.

The age of simply being glued to the television is gone. Our phones are communication portals to real-time updates, opinions, commentary, and more—and that’s exactly how they’re being used.

Picture a sports fan, watching a game in real time, while simultaneously using their phone for supplementary data and insight about the game, the team, the players, and more. This is the modern sports fan—and the quintessential Rei do Pitaco user.

Rafael Blay, Rei do Pitaco’s Data Scientist hits the nail on the head: “People want to watch a sports game and be a part of it. Some data streams relevant to the game can make it better.”

Rei do Pitaco is Latin America’s premier fantasy sports mobile app. By January 2023, Rei do Pitaco already boasted more than 14 million downloads since its 2019 founding. As Rafael explains, “Technology is central to Rei do Pitaco, and the data team is one of the largest teams within the organization.” Its technology focus, combined with its deep understanding of its customer base, might be largely responsible for its rapid growth.

“Our ideal user is a general sports fan in Brazil. We used to just be soccer, but we’re expanding into more sports,” he continues. “Rei do Pitaco is for people who enjoy watching sports, following teams, and watching games with a second screen in their hand.”

“I needed a champion, so I became one”

Rafael has been with Rei do Pitaco for over a year and a half, and was the first data person hired along with another data analyst. Initially, he saw himself wearing many hats and championing experimentation efforts across the company: “A lot of my day is talking to engineers, product people, etc., and helping them describe experiments, conduct them, and analyze them.”

He goes on, “I spend a lot of time meeting with stakeholders and making sure experiments are set up properly. I also help people understand the results: If the metrics are dropping, if they’re increasing, if features are safe to roll out to all users, etc.”

Essentially, Rafael and other Data Scientists are Rei do Pitaco’s internal advocates for data science, data analytics, and facilitators of good experimentation culture.

Hitting a stride

Before Rei do Pitaco adopted Statsig, experiments were done at a mobile build level. “We would have a control build and a test build, and would make the test build available to 50% of user,” Rafael shares. This testing methodology came with its fair share of issues, including lack of insight into their metrics.

“The problem was that there were a ton of differences between one app and another, making it difficult to determine which features were causing changes in metrics.” Additionally, the existing setup was not good at showing confidence intervals or statistical significance.

Immediately, Rei do Pitaco saw value in using gates. “The whole concept of gates, and using gates to determine if a user sees something or not. The idea of using gates for defining eligibility for experiment is very powerful,” says Rafael. “We liked Statsig’s experimentation and feature flagging, but we also just wanted a simple way for employees to test out a feature.”

“The idea of using gates for defining eligibility for experiment is very powerful.”

Rafael Blay

Data Scientist, Rei do Pitaco

All in all, deploying Statsig across their entire technical environment took just over a month. “In the first few months of having Statsig deployed, we weren’t very selective about which metrics to send to Statsig.” All in all, it took about three months for Rei do Pitaco to establish their own operational guidelines around using Statsig.

Experimentation surprises: The signup experiment

Rafael describes how Statsig recently helped them get to the bottom of a data mystery: “We had an experiment running inside our signup funnel to see if we could do anything to increase signup conversion rate and simplify the process.”

It sounded normal, but Rafael quickly saw a discrepancy. He explains, “We saw that our conversion rate declined during this experiment, but the rate of topline users actually grew.” The results were momentarily baffling.

Rafael elaborates: “After looking through some metrics, we were able to determine that we were tracking an event exposure every time the user reached the login or signup funnel. We realized that every time a user logged out and then logged back in they were considered a new user, which ultimately hurt the overall conversion rate.” Fortunately, he was able to put a stop to this reporting mix-up right away.

Rafael comments that Statsig made it easy to get to the source of this error.

A team you can count on

Rafael has had nothing but good experiences with the Statsig support team, which he stays in contact with in the Statsig Slack community. “Statsig support has been one of the key parts of our relationship!”

He uses the Slack community to get quick answers to questions about everything from product features and best practices to definitions and advice. “I’m very satisfied with both the speed and quality of the responses in the Slack channel,” he adds.

“Sometimes my questions aren’t even about the tool, but like ‘how to explain to a non-technical person the intuition behind a p-value or confidence interval,’ and Statsig answers those questions as well!”

Rafael Blay

Data Scientist, Rei do Pitaco

Present-day experimentation at Rei do Pitaco

According to Rafael, Rei do Pitaco is hitting its experimentation stride. “We’re migrating towards everyone doing experiments. The main experiments here are the data and tech teams. They used to launch features without testing, but now every change goes through an experimentation phase.”

“Statsig saves a lot of time in two processes,” Rafael elaborates: “One is configuring experiments. It’s very easy to just add feature flags to the code and implement it. The other is the analysis. Statsig makes it way faster to analyze the data, as a data scientist.”

Statsig also saves Rei do Pitaco significant time on the everyday process of deploying code. “For every feature we launch, Statsig saves us about 3-5 days of extra work—and we’ve already launched many ‘very good’ experiments.”

Rafael also outlines the technical wins behind Rei do Pitaco’s organization-wide utilization of Statsig. “Statsig saves time and process that an engineer expends to implement different versions, and also saves time on the data side gathering the data and adding it.”

“Statsig support has been one of the key parts of our relationship! For every feature we launch, Statsig saves us about 3-5 days of extra work”

Rafael Blay

Data Scientist, Rei do Pitaco

The future

In the grand scheme of things, despite many major wins already under its belt, Rei do Pitaco’s experimentation is just warming up. “In 2023 especially, all our teams have started using and trusting Statsig. Now everyone is on board and is moving way faster,” Rafael reports.

Rafael goes into greater detail describing how Statsig helps support Rei do Pitaco with its future development plans. “We just finished drafting our OKRs [as of the time of writing]. There are a bunch of metrics that we want to track and improve. Essentially what we want to do is leverage Statsig so that every experiment we run is linked to an OKR.”

This approach, Rafael hopes, will further optimize Rei do Pitaco’s experimentation and ensure it has maximum business impact. He gives an example, “If our OKRs state to increase revenue around a particular feature, we might allocate X number of experiments toward that feature, for instance.”

What Rafael hopes is that this new business impact-focused experimentation culture will lead to some big wins. “There are infinite experiments we could potentially run,” he remarks. “But this way, we’re experimenting on the changes that could give us big percentages. We think Statsig is going to be very helpful in allowing us to focus on high-impact experiments.”

All in all, Rei do Pitaco has mastered the art of conducting high-quality experiments and using the results to drive business decisions. For the future, refinement and prioritization will take center stage, keeping the same experimentation culture but increasing the velocity.

On this note, Rafael imparts us with a bit of optimism: “The results of the signup experiment are already looking promising.”