The People and Markets AI Doesn’t Know

Existing customer data tells us a great deal about known populations and current markets. Genuine innovation, however, often requires evidence about unknown people, practices, and opportunities that have barely entered the dataset yet.

In my previous article (Mai, 2026, August 18), I argued that AI will increasingly automate those parts of commercial research that are standardized, scalable, and concerned with measuring, retrieving, classifying, and processing what is already known or explicitly stated. The more defensible role for human insight researchers lies elsewhere: in generating genuinely new empirical evidence, entering unfamiliar contexts, noticing unexpected behaviors, making abductive connections, and judging why emerging insights matter strategically.

If companies increasingly rely on the same AI models and publicly available datasets, differentiation will depend on going where the existing data – and therefore the models trained on it – cannot go. As I argue in this article, this is where unexplored populations, fringe adopters, and extreme lead users become particularly interesting.

The average customer is often the wrong place to look for innovation

Traditional market research or even standard UX research often focus on identifying existing customers’ “unmet needs”, attitudes toward certain brands, interactions with existing products, or pain and passion points around them. Therefore, most customer data only gives us good ideas about the status quo – something that is considered “outdated” within months these days by people obsessed with micro trends.

From an innovation perspective, however, that’s table stakes and one of the reasons why traditional research mostly leads to evaluating the obvious and, at best, iterative improvement of existing core products – the lowest innovation ambition (Nagji & Tuff, 2012).
More importantly, current customers and users of existing products and services tend to be well-explored groups for many companies. Research on them is often saturated and updated at an irregular pace, adding little epistemological value. That’s not what the objective of human-centric innovation research is about.

Good innovation research discovers genuine transformation potential

Innovation research is more exploratory and future intervention-focused, enabling us to hunt for genuine transformation potential. Through primary research, we want to gain an empirically grounded, holistic understanding of how real groups of people think, act, and interact within their existing, lived contexts. Using this deep understanding as a vantage point, we aim to explore and hypothesize how potential shifts in culture and humanity at large could occur, and how potential innovations could impact the parameters of context at the point of intervention (Hartley, 2022). It is an experimental approach originally coined by Joachim Halse (2013) as “ethnographies of the possible” (p. 180).

Using the insights that are generated from these future-oriented activities as a springboard allows us to identify viable opportunity spaces for innovation. These areas hold potential for new products, services, and even business logics – all of which could introduce an entirely new market or transform existing markets in major ways.

Who are the unexplored populations at the edges?

With the key objective to push the envelope a bit further, not every unexplored population is equally fertile from an innovation research perspective. As a general rule of thumb, people who embrace extremes in certain domains of life are better suited for exploration than even the most unknown Average Joe. This includes:

• Exceptionally creative people
• Controversial experts
• Fringe adopters
• Homebrewing tinkerers
• Extreme lead users
• Members of the cultural avantgarde

By investigating people who are living on the cutting edge of a particular system – be it culture and society, technology, ecology, economics, or politics – we can already gain a glimpse of future potential. By observing their lives and conversing with them from their perspective, we gain a valuable window into the following things:

• Emerging behaviors and embodied practices
• Shifts in values and perspectives that stray from mass culture
• Changes in relational dynamics and human interaction
• Far-out visions that happen to inspire people to work toward a future state
• Improvised solutions to problems that major companies have neither identified nor commercialized yet
• Informal markets that operate according to different logics

New ideas and innovation rarely emerge out of thin air. All of these aspects listed above provide evidence as well as inspiration for where the journey could be heading. Empirical evidence is important because it enables us to avoid blue sky thinking and approach innovation in a systematic, methodological way.

The actual task of exploring the unexplored is not difficult for an ethnographically trained insights professional. But the first step is gaining physical and social access to real people who are willing to be studied.

Finding the unexplored is difficult precisely because they are unexplored

Exceptional or extreme people who are on the cutting edge rarely sign up for market research studies. They don’t leave their contacts in respondent databases – the default recruitment pathway for much of traditional commercial research. They’re also not very likely to fill out a lengthy screener that assesses whether their demographic data fit the client’s ideal sample criteria before qualifying for participation. So the process of opening up the field looks decidedly different, more laborious, and time-consuming.

To find them, you need an array of techniques. I usually start with desk research-based horizon scanning – a method from foresight that detects weak signals of change and helps identify emerging actors (cf. Cuhls, 2019). Then you need soft recruiting via email, phone, or social media; physical access to particular communities; expert referrals and network-based snowball recruiting once in the field – essentially a combination of techniques one typically uses when sampling in ethnography (cf. Brewer, 2000).

You also need to figure out incentives that actually encourage participation. Someone who is not used to being researched – who doesn’t know anything about typical incentive rates in commercial research, or who struggles with the idea of supporting corporations – requires an alternative incentive that actually works for them.

Entering their worlds requires direct face-to-face interaction in context

Finding unusual people is only the first step. Entering their worlds requires physical presence and direct social interaction that is unencumbered and unmediated by computer screens. As outlined in an older article I wrote before the hype around AI (Mai, 2021, July 27), “virtual ethnography” alone won’t cut it. Remote one-on-one interviews (or, even worse, AI-moderated interviews) can certainly tell us what people behind a computer screen are willing and able to articulate, but exploratory ethnography needs to go further to elicit the full innovation potential.

Being physically present allows researchers to “nose around”, pick up seemingly irrelevant cues, stumble upon unexpected events, observe interactions as they naturally unfold, and understand how behaviors are embedded in a wider material and sociocultural context. A family member or friend entering the room, an improvised tool sitting on a workbench, the way someone navigates their neighborhood, or a practice they consider too mundane to even mention may turn out to be more revealing than anything we could have anticipated in an interview guide.

Fieldwork also creates an embodied, multimodal form of understanding. We see, hear, touch, move, participate, and experience aspects of people’s lives that cannot be fully reconstructed from their verbal accounts alone. Data is then crafted via fieldnotes, photos, video, audio recordings, sketched maps, and even collections of artifacts that are meant to capture the full sensorial and intellectual range of what is experienced in the field.
This dual openness to coincidence and experiential modality is precisely what makes ethnographic exploration useful when we do not yet know exactly what we are looking for.

Understanding the people you encounter beyond the consumer or user lens

The next step is gaining a contextual understanding of the people we are exploring. This requires treating them not primarily as consumers but as complex human beings who embrace multiple social roles and identities. The roles of consumer, user, or patient – all common in commercial research – should not be completely discarded, but we must take into account that most social roles are situation-specific. Consumption or tool usage may take on a different manifestation or may not even be a relevant consideration when people act as mothers, spouses, employees, bosses, friends, or children. The consumer or user lens may even obscure the phenomena to which we should be paying attention.

Instead, viewing people as complex, social human beings allows us to disentangle their lives along the lines of relationships, social networks (not just the digital ones), responsibilities, and obligations. All of those are tied to distinct ideas, interactions and practices that may matter.

AI hits a particular accessibility and sensemaking limit here

When it comes to interpreting people, AI has genuine limits of accessibility and deep cultural understanding. AI can only reason over representations of a world that have entered its accessible digital information environment. It cannot independently generate empirical evidence about an insufficiently documented population whose behaviors have not been documented, analyzed, or interpreted.

For sure, AI can help us scan literature and track discourses, identify weak signals of change, generate hypotheses, organize fieldwork material, do basic pattern analysis of transcripts and field notes, and identify connections across large datasets. But somebody still has to open the field, gain trust and establish rapport with real people, encounter the phenomena, decode relevant meanings, and turn previously unrecorded human activity into empirical material.

From unusual people to commercially useful insights

Exploration of fringe populations doesn’t generate knowledge for the sake of curiosity. Innovation research follows a design thinking-oriented innovation process that is meant to have a positive impact on business growth. Observations (i.e. patterns in the data) ultimately need to be clustered into strategically relevant insights, which are then translated into future change hypotheses and opportunity spaces that can inform the ideation of new products, services, markets, or business models.

In the commercial research sphere, the term “insight” is often abused for virtually any entity of knowledge, regardless of how small or complex it is. To clarify what I mean, it makes sense to briefly talk about what actually constitutes a good and useful insight.

An insight is an analytical statement about a pattern that holds across the research sample and why this is the case. It is an analytical construct that serves the purpose of reducing complexity by bringing individual observations under a coherent explanation without flattening meaningful variation. Where the pattern differs systematically between markets, types of people, or relevant groups, these differences become part of the insight rather than exceptions to it.

An insight is not produced simply by interviewing or surveying people and summarizing what they said. Instead, we put the statements and observations into context, explaining the ‘why’ behind the ‘what’ through interpretation by harnessing our knowledge about people’s behavior. Where different patterns emerge, we not only point to them but also seek to explain how and why these differences may have emerged.

To be actionable, an insight must always conclude with a business-relevant ‘So what’ for the client organization, outlining implications that can reside on a tactical level (e.g. product / service / feature) or a strategic level (e.g. vision).

Basic structure of a good insight

It should be clear by now that a good insight is more than a pattern or aggregation of data points. Overall, a strong insight should aim to incorporate four components to become a useful storytelling device and ideation springboard downstream in the innovation process:

• Description of the phenomenon – the ‘what’ – summarizing what we saw or heard
• Explanation of the human mechanics behind the phenomenon – the ‘why’ – providing an analysis of the behavior
• Supporting observations & evidence from the dataset – snippets that form a pattern, demonstrating the ‘what’ – with the addition of representative quotes, video clips, photos
• Implications with some predictive capacity – the ‘so what’ – giving direction on what it might mean going forward for the client team, without going into solution mode yet.

As mentioned above, insights are the evidence and idea-generating foundation for opportunity space identification, systematic product ideation, and concept brief development.

Conclusion

Does this mean that we are merely playing a game of catch-up until AI has mapped every raw data point and useful insight ever created? If we keep researching unexplored populations, documenting their practices, decoding their meanings and future trajectories, and feeding those insights back into our organizational knowledge management systems, won’t they eventually cease to be unexplored? And won’t AI systems then be able to process this knowledge, as they are already doing?

This assumption would only hold true if humans, their cultures, technologies, techniques, and markets were static. But they are decidedly dynamic. New technologies elicit new practices and relationships. Social, economic, political, and environmental conditions shift. Subcultures emerge, become absorbed by the mainstream, mutate or disappear. Lead users appropriate products in ways their makers never anticipated. Humans in need develop informal solutions to emerging problems before a corporation understands how to productize and commercialize them. Fringe groups experiment with ways of living that the mainstream only discovers years later. By the time one aspect of humanity has been documented, another has already begun to change.

Exploratory research shifts commercial focus from the present to the future. Competitor differentiation then comes from learning about these shifts, embracing a longer time horizon, anticipating major transformations, and speculating what this could mean strategically.

That’s why I don’t see exploratory human research as a nostalgic defense of ethnographic fieldwork against AI. We should automate the parts of research that can sensibly be automated. AI will become increasingly useful for managing and navigating what we already know. It allows us to move across enormous amounts of existing material. But somebody still has to generate genuinely new evidence in the first place, and it had better be someone who knows what they are doing.

For genuine innovation, that means strategically going beyond the established populations and markets that have already been researched to exhaustion. It means finding people at the edges, entering their worlds, observing emerging practices in context, and working out why any of this might matter for business. As AI makes existing knowledge increasingly cheap and accessible at the push of a button, the ability to discover what is not yet sufficiently known may become more valuable, not less.
 

References

Brewer, J. D. (2000). Ethnography. Buckingham: Sage.

Cuhls, K. E. (2019). Horizon scanning in foresight: Why horizon scanning is only part of the game. Futures & Foresight Science, 2(1): e23.

Halse, J. (2013). Ethnographies of the possible. In W. Gunn, T. Otto, & R. C. Smith (Eds.), Design anthropology: Theory and practice (pp. 180–198). London: Berg Publisher.

Mai, D. (2021, July 27). Missing fieldwork: Why “virtual ethnography” is not human-centric. Dr. Daniel Mai. Retrieved from https://drdanielmai.com/missing-fieldwork/

Mai, D. (2026, August 18). What is the Future of Applied Human Insight Research? Dr. Daniel Mai. Retrieved from https://drdanielmai.com/future-human-insight-research/

Hartley, P. (2022). Radical human centricity: Fulfilling the promises of innovation research. London/New York City: Anthem Press.

Nagji, B. & Tuff, G. (2012). Innovation Ambition Matrix. Harvard Business Review. May, 5-11.