I get asked this question constantly: "How accurate is AI market research compared to studies with humans?" My response is always the same: "How accurate are studies with humans compared to actual behavior?"

It's a fair question that reveals a deeper assumption about traditional market research methods. Most people assume that replicating survey results equals accuracy, but what if those survey results are systematically wrong to begin with?

The Problem with "Ground Truth" 

The Harvard Business Review published a piece called "The Elusive Green Consumer" that highlights the problem – people don’t always do what they say they’ll do. They found that while 65% of consumers say they want to buy from purpose-driven, sustainable brands, only 26% actually spend their money that way. That's a staggering 39-percentage-point gap between stated intention and actual behavior.

This intention-action gap isn't unique to green products—it shows up everywhere in consumer research. People tell researchers they'll exercise more, eat healthier, save money, and buy ethically. Then they don't. The disconnect between what people say in surveys and what they actually do with their wallets is one of marketing's oldest problems.

When AI market research vendors talk about "accuracy," they're typically measuring how well their synthetic respondents replicate survey responses. But if those survey responses don't predict real-world behavior, what exactly are we optimizing for?

So when someone talks about the accuracy of AI research, I wonder: are we trying to replicate flawed human survey responses, or are we trying to predict actual human behavior? It's possible that AI research could be more accurate than traditional methods.

Testing the Green Bias Hypothesis 

To explore this question, I ran a simple experiment using the exact scenario from recent research on LLM-generated personas. The study found that as AI personas became more detailed and subjective, they increasingly favored environmentally friendly but more expensive options. 

LLM Generated Persona is a Promise with a Catch

The default US General Population audience in Ask Rally was generated using a descriptive prompt, so I expected it to display some of this left leaning bias – indeed it’s something we’ve seen before when we ran a virtual election campaign. I started with Anthropic's Haiku model (the "Fast" option in the Ask Rally interface) and gave it this prompt:

Car A costs $35 k, emits 90 g CO₂/km, and uses eco-materials.

Car B costs $25 k, emits 150 g CO₂/km, no eco-features.

Which would you buy?

The results were what I feared. Across 100 simulated responses, 78% chose the more expensive, eco-friendly Car A. This aligns perfectly with what people say in surveys—the stated preference for sustainable options despite the higher cost. 

It makes sense that AI models would exhibit the same bias as people have when they talk about their purchase decisions, because the majority of training data is people talking online. From this the models internalize the progressive values people claim to have, leading to responses that prioritize environmental concerns over economic ones.

The problem with both traditional and synthetic methods is that this 78% preference for the green option doesn't match real-world car buying behavior, where price sensitivity typically dominates and eco-friendly vehicles still represent a much smaller market share. How do we correct for this?

The Calibration Solution 

The good news is that AI bias isn't insurmountable. We've shown before how systematic calibration using DSPy can correct model biases through careful prompt engineering and optimization loops.

But sometimes the solution is simpler than heavy optimization. Sometimes it's just about picking the right model.

When I upgraded to Anthropic's Sonnet model (the "Smart" option), something interesting happened. The bias completely reversed. Now only 37% of synthetic respondents chose the expensive, eco-friendly car—much closer to what we see in actual purchasing behavior.

The Sonnet model's 37% preference for the eco-friendly option sits right in that sweet spot between stated intention (65%) and actual behavior (26%)—suggesting it might be capturing something closer to real purchasing intent rather than social desirability bias. Real-world data suggests that about 26% of consumers actually spend money on sustainable brands despite 65% claiming they want to, so this is pretty close, and actually more accurate than traditional methods (in this case).

What This Means for AI Market Research

This experiment reveals something crucial about synthetic market research: the choice of model and methodology dramatically affects whether you're measuring what people say or what people do.

If you're trying to replicate traditional survey results—you must validate an AI system against both existing research and actual human behavior. The intention-action gap that has plagued traditional market research for decades, and will continue to be important to design against as the synthetic research market expands.

By understanding how different models and prompting strategies affect this gap, we can potentially tune our synthetic respondents to predict either stated preferences or actual behavior, depending on what we need. The real accuracy question isn't whether AI can replicate human survey responses—it's whether AI can help us bridge the gap between what people say and what they actually do. That's a much more valuable capability than simply automating traditional research methods.