Synthetic users: Testing with artificial data

Picture this: you need user feedback on your new feature, but recruiting participants takes weeks and your budget is already stretched thin. You're stuck between shipping blind or waiting another month for insights.

This is where synthetic users come in - AI-generated research participants that can give you rapid feedback without the traditional headaches. But before you fire your research team and go all-in on robot users, let's talk about what these tools can actually do (and where they fall flat).

Understanding synthetic users in research

Synthetic users are basically AI participants that pretend to be real people in your research studies. Think ChatGPT, but trained specifically to act like your target users during interviews, surveys, and usability tests.

These AI participants have gotten surprisingly good at mimicking human responses. The team behind Synthetic Users (yes, that's actually the company name) has built tools that generate participants so human-like that some researchers can't tell the difference. You can run an entire user interview in minutes instead of weeks, which sounds almost too good to be true - because sometimes it is.

The big promise here is cutting through the logistics nightmare of traditional user research. No more:

• Scheduling conflicts with participants

• No-shows ruining your timeline

• Budget overruns from recruitment fees

• Geographic limitations on who you can interview

But here's where things get interesting (and a bit controversial). The Reddit ProductManagement community has been debating whether these synthetic insights are actually worth anything. One user put it perfectly: "They're great for catching obvious stuff, but can they really tell you why your checkout flow makes people rage-quit?"

The machine learning folks on Reddit have been experimenting with synthetic data in their projects too, with mixed results. Some swear by it for training models, while others say it's like trying to learn cooking from someone who's never tasted food.

Advantages of synthetic users over traditional methods

Let's be real - traditional user research can be a pain. You spend weeks recruiting participants, half of them flake out, and by the time you have insights, your product team has already moved on to the next sprint.

Synthetic users flip this whole process on its head. You can generate hundreds of diverse participants in the time it takes to schedule one real interview. Need to test with enterprise IT managers in their 40s who use specific software? Done. Want to see how teenagers in urban areas react to your app? No problem. The customization options are pretty wild.

Here's what makes synthetic users particularly useful:

• Speed: Get initial feedback in hours, not weeks

• Scale: Test with 1,000 users as easily as 10

• Consistency: No more variations from tired participants or bad interview days

• Edge cases: Test scenarios that would be impossible or unethical with real users

The UX testing community has found some clever applications. Teams are using synthetic users for early-stage prototypes where they just need quick gut checks. As noted in Statsig's guide on experimenting with generative AI apps, these tools excel at catching obvious usability issues before you waste real users' time.

But here's the thing - and this is crucial - synthetic users work best as a first pass, not your only pass. They're like spell-check for user research: great at catching obvious problems, but you still need a human editor to make sure your message actually lands.

Limitations and considerations of synthetic users

Now for the reality check. Synthetic users have a positivity problem - they're often way too nice.

The ProductManagement subreddit discovered this the hard way. Users reported that their synthetic participants gave glowing feedback on features that real users absolutely hated. It's like asking your mom if your startup idea is good - you're probably not getting the brutal honesty you need.

Here are the main issues you'll run into:

1. Surface-level insights: They catch obvious problems but miss subtle emotional reactions

2. Bias toward positivity: AI tends to be more agreeable than real humans

3. Lack of context: They can't draw from real-life experiences they haven't had

4. Missing edge cases: Real users do weird, unexpected things that AI doesn't predict

The ServiceDesign community suggests treating synthetic user feedback as hypotheses to test, not facts to ship. One designer shared how they used synthetic users to generate 50 potential pain points, then validated the top 10 with real users. Smart approach.

You also need to be transparent about using synthetic data. Nobody wants to find out their "user research" was actually just talking to robots. The Marketresearch subreddit had a whole thread about the ethics here - the consensus was clear labeling and using synthetic users to supplement, not replace, real research.

Bottom line: Synthetic users are best for early-stage exploration and hypothesis generation. Once you're making real product decisions, you need real human feedback to validate what the robots told you.

Best practices for integrating synthetic users in research

So how do you actually use synthetic users without shooting yourself in the foot? Start by being strategic about when to deploy them.

Here's a practical framework that's been working for teams:

Use synthetic users for:

• Initial concept validation

• Generating interview questions

• Testing obvious usability issues

• Exploring edge cases safely

• Training new researchers

Stick with real users for:

• Final design decisions

• Understanding emotional responses

• Validating business-critical features

• Building genuine empathy

The key is building feedback loops between synthetic and real data. Run synthetic tests first to identify potential issues, then dig deeper with real users on the problems that surface. The MachineLearning community found that this hybrid approach actually improves both types of research - synthetic users help you ask better questions, and real users help train better synthetic models.

Transparency is non-negotiable here. When presenting findings, be upfront about your methods:

• "Our synthetic user tests suggested X, which we validated with 20 real users"

• "AI participants identified these pain points, confirmed by customer interviews"

• "Initial synthetic testing revealed three issues; real users found two additional problems"

One practical tip from the UserExperience subreddit: create separate research pipelines for synthetic and real users. This prevents accidental mixing of data sources and makes it easier to track which insights came from where. Some teams even use different slide templates for synthetic vs. real user findings.

Remember, synthetic users are tools, not replacements. They're like having a really smart intern who can work 24/7 but has never actually used your product in real life. Valuable for certain tasks, but you wouldn't bet your company on their insights alone.

Closing thoughts

Synthetic users aren't going to replace your research team anytime soon, but they're too useful to ignore. They shine when you need quick directional feedback or want to explore ideas before investing in full research.

The teams getting real value from synthetic users treat them as one tool in their research toolkit - not a magic solution. Start small, validate everything with real users, and be transparent about your methods. Your stakeholders (and users) will thank you.

Want to dive deeper? Check out:

• Statsig's synthetic testing guide for catching issues early

• The ongoing Reddit discussions in r/userexperience and r/ProductManagement

• Your own experiments with free synthetic user tools (seriously, try it yourself)

Hope you find this useful! The future of research is probably some mix of human and AI participants - might as well figure out how to use both effectively while we're still in the early days.