What is the Future of Applied Human Insight Research?

Commercial human research has never had more methods, tools, and technological capability available. Yet parts of the research industry and innovation consultancy market appear to be in crisis as a result of an ongoing AI transformation. On the one end, AI moderators, conversational surveys, and automated data analytics are investigating real people. On the other end, synthetic users and digital twins are suggesting that, perhaps, real people need not be researched at all. Both ends are connected by the same economic promise: conduct more research, faster, cheaper, with fewer researchers.

If primary research, analysis, synthesis, and even respondents themselves can supposedly be automated, what exactly is left for the human insight researcher? This article examines why we arrived here, which parts of the industry genuinely are becoming automatable, and where human-led research still creates distinctive commercial value – especially in innovation.

How did we get here?

Many long-standing innovation researchers, business anthropologists, UX researchers, and applied social scientists in my network lament this development and what it means for their consulting practices (see Moran et al., 2026, January 16; McQuater, 2026, April 2; Gerard, 2026, June 9). Several signs of change have been on the wall for some time; it just took some of us – including myself – a bit longer to make sense of them retrospectively.
As with most developments, there’s a history behind them. Here’s my explanation, grounded in my own experience working in business anthropology and innovation consulting with various agencies for the past 15 years:

For most commercial companies, it is a no-brainer that understanding the dynamics of markets and customers is key to being successful. The ways in which evidence for decision-making is created have always evolved. Since its humble beginnings around a century ago, market research has promised companies to uncover their customers’ perspectives, preferences, relevance systems, pain points, needs and desires (cf. Yallop et al, 2022).
With the emergence of design thinking in the 1990s and early 2000s, traditional psychology-driven market research wasn’t enough anymore. With its double diamond framework that provided an intuitive structure (discover, define, develop, and deliver; see Design Council, 2026; cf. Bánáthy, 1996), a systematic, phase-based process for innovation and design became popular.

On the front end of that process, human-centric innovation research introduced a host of different methods and approaches from various disciplines in the humanities, social sciences, industrial design and service design. It promised a more direct link between human research, business strategy, and product development.

The rise of innovation research

For many Fortune 500 companies, front-end innovation research gradually became integral to the development of new markets, business models, and, eventually, new products and services. Elaborate ethnographic studies on site, broad cultural inquiries into emerging markets, and highly participative formats in creative settings asked big questions of strategic relevance.

This methodology managed to grasp and decode people not just as consumers or users, but as complex human beings who embrace multiple social roles and who are situated in culture. More than a “human factor”, culture provided the systemic lens that must be understood when aiming to cater to people in a way that is meant to be sustainable for business because it fits into their lives. The deep qualitative insights that came out of this research were a big step up from the rather tactical and often a-theoretical market research of yesteryear.

This insights approach promised not only to explain why people behave the way they do in context. Combined with foresight (e.g. Schwartz, 1996; Hines & Bishop, 2006), it allowed us to speculate how humans and markets might shift in the future due to a range of sociocultural drivers and forces, and what these scenarios could imply for business.

More importantly, this approach also enabled designers, engineers, and marketers to ideate against solid, evidence-based opportunity spaces that emerged from these robust insights and anticipatory hypotheses toward the future. Further downstream in the innovation process, the tools and methods of innovation research provided the foundations for agencies and companies to co-create concepts with customers and test prototypes with users in real-life settings.

Human-centricity was the successful call of the day

All of this was always time-consuming, expert-driven, and expensive. For about two decades, the innovation consulting industry flourished. However, none of this success was a self-evident matter of course. Agencies, such as IDEO, FROG, ReD Associates, Idea Couture, and others, made sure to belabor the point in executive boardrooms and encourage a new level of human-centricity (e.g. Mootee, 2013; Madsbjerg & Rasmussen, 2014).

Markets like the US, UK, and Scandinavian countries were significantly quicker to adopt than some central European countries such as Germany. Nevertheless, many agencies successfully helped companies all over the world to install their own innovation teams, codify their processes, and train people. In consequence, companies widely adopted the promoted language and (some of) the thinking behind it, fashioning themselves in terms of customer centricity and empathy.

From big strategic questions around culture to software, users and their digital lives

The mid- to late 2010s were marked by a major shift toward agency consolidation. Big management consultancies and IT systems integrators, such as McKinsey, Deloitte, Accenture, Capgemini, Cognizant and Wipro, acquired dozens of boutique innovation consultancies at a rapid pace, trying to integrate them into their service offerings (Palmer, 2019, November 5). With a stronger emphasis on digital business, agency services became widely centered around software. The product-focused role of the user became dominant, contributing to the rise of narrow UX research further downstream in the innovation process.

The global COVID-19 pandemic shifted people research further to the virtual sphere – first for health reasons, when in-person fieldwork was not safe, then for cost-cutting reasons, when remote research was deemed good enough by many clients. The amount of human-centricity that was lost during this shift is not to be neglected, as I pointed out in an earlier article (Mai, 2021, July 27).

At the same time, netnographies into online communities flourished (Costello et al., 2017). Standing online access panels and online research communities were instituted that offered quick feedback loops without laborious recruitment planning and fielding (Ljepava, 2016). With the inflationary rise of social media, a never-ending stream of user-generated data offered a remote glimpse into people’s lives – albeit mediated, highly stylized, and extremely performative.

The boundaries between the digital world and the physical world did not disappear; they were rendered irrelevant by many, despite the obvious methodological limitations.

Making sense of a growing insights repository and processing customer data in-house

Throughout this digital shift, internal knowledge management became more important both for clients and agencies. Over the decades, companies had spent millions of Euros on disparate customer studies whose scopes, formats, output quality, and applicability were as varied as the array of agencies and internal business functions that had created them. On top, many companies had gathered terabytes of raw customer data through their own connected products, services, and support functions that needn’t lie dormant.

Now it was time to make sense of what an organization knows by creating a functional organizational memory that could be used to generate and retrieve insights in a longitudinal, cross-project manner. Centralized digital access to these assets made it decidedly different from the corporate archives and largely analog mnemonic techniques of yesteryear (cf. Mai, 2015). Companies now also wanted to process existing customer data into information and then elevate it into usable knowledge – while relying on the interpretations and recommendations provided by agencies as little as necessary. Doing the initial legwork themselves, the goal was to produce a continuous stream of insights in-house.

Business intelligence tools such as Microsoft Power BI promised to accelerate this on an enterprise scale, harnessing all kinds of quantitative customer, market and competitor data that swirled around an organization. On a smaller scale, countless customer intelligence tools such as Dovetail or Condens positioned themselves as centralized platforms for processing qualitative customer data from research and support tickets.

Machine readability of research outputs became key because they were the fuel of these systems. So did direct access to, and processing rights for, the raw research data generated by agencies during the studies that were still commissioned.

Automating research for the sake of speed, lower costs and efficiency

The 2022 mass-market introduction of ChatGPT (and, later, other LLM chatbots) brought the next wave of changes in commercial research. When providers released APIs for their LLMs, developers could integrate these models into other apps and services. Soon, a plethora of AI-driven services emerged that promised not just to automate desk research using public data, but even primary research with real people, including pattern analysis, sensemaking, the generation of insights and business implications in no time.

For instance, the launch of ResearchGoat by Synthetic Acumen in 2023 first introduced the commercial service of agentic AI-designed and AI-moderated in-depth qualitative user research. Needless to say, countless competitors have joined the market ever since. Overall, I doubt the usefulness of AI-moderated research for semi-structured primary data generation. However, I see value in a conversational approach to structured, open-ended surveys – because, let’s be honest, nobody likes filling out surveys, regardless of how well they are designed.

On the other end of the research spectrum, the idea of “synthetic users” emerged around 2024 – essentially AI-generated profiles that simulate a target audience’s logics, perspectives, preferences, and needs (Rosala & Moran, 2024, June 21). Their core promise was that one needn’t even investigate real people anymore – at least for certain types of inquiries. While some critics call this “stochastic [research] theatre” (Papas, 2025, June 6), others argue that synthetic user research may replace basic market research (Korst et al, 2025). And – if designed properly and methodologically sound – it could simply serve as a more interactive alternative to traditional customer typologies, archetypes, and personas – all different tools that have always had their place in the innovation research process.

While some AI-driven automation tools certainly pose the biggest shift in applied social-scientific research that we have encountered in decades, let’s not pretend that ongoing technological advancements have never played a major role in the ways we conduct our work.

Social-scientific methods go hand in hand with digitalization and automation

Social-scientific research has always incorporated new high-tech tools and techniques that opened up new possibilities for fielding, data generation, analysis, and interpretation. For example, the introduction of qualitative data analysis software, such as MAXQDA or ATLAS.ti in the early 1990s, was a game changer that – if applied correctly – required researchers to shift most of their workflows toward the digital sphere.

Compact digital cameras and digital voice recorders made data management infinitely easier. So did early speech recognition and transcription software in the early 2000s, such as Dragon Naturally Speaking – long before the emergence of neural-network-based STT systems we take for granted in our smartphones today. I still remember manually transcribing all the ethnographic interviews I conducted during my PhD fieldwork, using a footswitch and F4 on my laptop. While still the surest way to really immerse yourself in your data, I don’t want to go back to those laborious and time-consuming days.

The automation potential is high for certain types of research

We can safely assume that certain types of research inquiries, methods, and individual tasks will no longer be covered completely by human researchers. The automation potential is largest when research is standardized, conducted at scale, and concerned with eliciting and processing explicitly stated responses. Here, the entire research pipeline has considerable automation potential. This includes but is not limited to:

• Desk research and secondary research, including the gathering and synthesis of market and competitor intelligence
• Social media listening
• Questionnaire design, survey programming, and conversational surveys
• Structured online community moderation
• Basic statistical analysis of quantitative data, including segmentation
• A/B testing
• Ad testing
• Certain types of UX testing

Qualitative exploratory research remains significantly harder to automate, because of its non-standardized, semi-structured nature. Parts of the research process are already in full automation swing, however, including:

• Interview transcription, speaker identification, translation, cleaning, formatting
• Descriptive summaries of transcripts and digitized fieldnotes
• Data ingestion and organization, including file tagging, indexing and metadata generation
• Semantic search and quote retrieval
• Code frequency and co-occurrence calculation
• Literature and desk research retrieval, including literature reviews
• First-pass qualitative coding following an existing codebook
• Basic pattern recognition across large unstructured data sets
• QDA research logs for internal documentation

While I have made a point in a previous article that elaborate pattern identification is still heavily flawed with popular LLMs (Mai, 2025, December 18), we can expect this to improve as social scientists incorporate QDA functions and established analytical methods into these AI systems (see e.g. Qinsights, 2026). My own experiments have proven successful with the AI-assisted anthropological interpretation and explanation of human-generated observations (Mai, 2025, December 18).

Automating existing knowledge vs. generating genuinely new evidence

As stated before, research has always incorporated technologies that removed laborious parts of the process. Refusing automation for nostalgic reasons (or professional gatekeeping reasons) is neither methodologically nor commercially convincing. We need to distinguish between research activities where standardization and automation creates efficiency, and where standardization and automation hinder or even destroy what makes inquiry valuable in the first place.

Highly commodified human research (such as traditional market research) that is concerned with measuring, retrieving, classifying, and processing what is already known or explicitly stated will increasingly be automated. Unfortunately, many companies do not genuinely care about the methodological quality of human research (Papas, 2025, June 6). Human researchers therefore cannot justify their role simply by being better moderators, data coders, analysts, or summary producers. This is a strained market that had already seen a race to the bottom in pre-COVID times, so much of that work is becoming even cheaper.
From my perspective, human insight researchers will continue to play one defensible role that comes equipped with a range of special qualities:

• Exploring phenomena, groups of people, and asking questions for which no ready-made dataset exists
• Entering and decoding unfamiliar contexts
• Noticing and observing unexpected behaviors and their emerging shifts
• Experiencing, interpreting, and explaining what can only be fully grasped in embodied, multimodal ways
• Challenging established categories and explanations
• Making new abductive connections between pre-existing research and emerging empirical data
• Judging why these insights matter strategically
• Showing personal accountability for one’s validity of recommendations.

While the so-called (and debatable) “democratization of research” via AI makes existing knowledge increasingly accessible to everyone, genuinely new knowledge may become more valuable than less. If all companies are working with more or less the same AI models and publicly available datasets, what will help generate the differentiating ideas that lead to innovation? My straightforward answer is that we need to go where the existing data – and therefore the models trained on it – cannot go.

In my next article, I will pursue this pathway further, digging deep into the value of exploring unexplored populations, fringe adopters, and extreme lead users that hold genuine innovation potential.

References

Bánáthy, B. H. (1996). Designing social systems in a changing world. New York City, NY: Springer Science + Business Media.

Costello, L., McDermott, M.-L., & Wallace, R. (2017). Netnography: Range of practices, misperceptions, and missed opportunities. International Journal of Qualitative Methods, 16(1), 1-12.

Design Council (2026). The double diamond: A universally accepted depiction of the design process. Design Council. Retrieved from https://www.designcouncil.org.uk/resources/the-double-diamond/

Gerard, M. (2026, June 9). Problems & solutions 3: Design thinking and its decline. Human Futures. Retrieved from https://humanfutures.com/fieldnotes/problems-solutions-2-design-thinking-and-its-decline

Hines, A., & Bishop, P. (2006). Thinking about the future: Guidelines for strategic foresight. Houston, TX: Hinesight.

Korst, J., Puntoni, S., & Toubia, O. (2025). AI and machine learning: How gen AI is transforming market research. Harvard Business Review, 103(3), 84-93.

Ljepava, N. (2016). Participation in online marketing research panels: Findings from focus group discussions. Conference: XVI International Symposium SYMORG. Retrieved from https://www.researchgate.net/publication/306410020_Participation_in_online_marketing_research_panels_Findings_from_focus_group_discussions

Mai, D. (2015). Organizational Cultures of Remembrance: Exploring the Relationships between Memory, Identity, and Image in an Automobile Company. Berlin/New York: De Gruyter.

Mai, D. (2021, July 27). Missing fieldwork: Why “virtual ethnography” is not human-centric. Human Futures Studio. Retrieved from https://drdanielmai.com/missing-fieldwork/

Mai, D. (2025, December 18). Turning AI into a research partner: Why training your gen-AI on methodology leads to better results. Human Futures. Retrieved from https://drdanielmai.com/ai-research-partner/

McQuater, K. (2026, April 2). UK qualitative researchers write to MRS over AI ‘threat’. Research Live. Retrieved from https://www.research-live.com/article/news/uk-qualitative-researchers-write-to-mrs-over-ai-threat/id/5148253

Mootee, I. (2013). Design thinking for strategic innovation: What they can’t teach you at business or design school. Hoboken, NJ: John Wiley & Sons.

Moran, K., Budiu, R., & Gibbons, S. (2026, January 16). State of UX 2026: Design deeper to differentiate. NN/Group. Retrieved from https://www.nngroup.com/articles/state-of-ux-2026/

Madsbjerg, C. & Rasmussen, M. (2014). The moment of clarity: Using the human sciences to solve your hardest business problems. Boston, MS: Harvard Business Review Press.

Palmer, M. (2019, November 5). Rush to acquire innovation agencies: Big consultancies like Accenture, BCG and McKinsey have bought up a string of small innovation agencies in an attempt to remain relevant. Sifted. Retrieved from https://sifted.eu/articles/innovation-agency-acquisitions

Papas, C. (2025, June 6). Synthetic personas, stochastic Theater: Why LLM outputs aren’t qualitative insight. The Voice of the User. Retrieved from https://www.thevoiceofuser.com/synthetic-personas-stochastic-theater-why-llm-outputs-arent-qualitative-insight

Qinsights (2026). AI qualitative data analysis software. Qinsights. Retrieved from https://qinsights.ai/

Research Goat (2026). Generative AI for in-depth qualitative research. Synthetic Acumen. Retrieved from https://researchgoat.com/

Rosala, M. & Moran, K. (2024, June 21). Synthetic users. If, when, and how to use AI-generated “research”. NN/Group. Retrieved from https://www.nngroup.com/articles/synthetic-users/

Schwartz (1996). The art of the long view: Planning for the future in an uncertain world. New York City, NY: Crown Business.

Yallop, A. C., Baker, J. J., Wardle, J. (2022). Market research and insight: Past, present and future. International Journal of Market Research, 64(2), 163-168.