DAY 1
Opening Remarks
Keith Smith – Managing Director, MSI
Watch the Presentation
From Prompt to Product: Using AI to Guide Aesthetic Design Decisions
Jeffrey P. Dotson – Associate Professor, The Ohio State University, Fisher College of Business
Generative AI can produce an effectively unlimited range of product images, but that abundance does not automatically make the images useful for marketing research. Jeffrey Dotson of Ohio State University presented a bottom-up method for turning aesthetic language into controlled product-design variables that researchers can test in discrete choice studies. The work, coauthored with Elea Feit and Ethan Budge, addresses a longstanding problem: visual design strongly influences consideration and choice, yet images are high-dimensional, preferences vary across consumers and conventional studies are limited to existing market designs or the output of a small number of designers.
The process begins by identifying lower-dimensional descriptors that a text-to-image model can reliably and independently express. The team evaluates each word on three criteria: applicability, prototypicality and orthogonality. Applicability asks whether the descriptor meaningfully changes the product image, prototypicality asks whether the quality already appears in the category default, and orthogonality asks whether the descriptor can vary independently from other descriptors. A useful, manipulable descriptor must be applicable, non-prototypical, and sufficiently distinct from the other terms in the design.
Jeffrey’s team uses text and image embeddings to calculate these properties before generating large numbers of images, reducing time and computational expense. In the dining-chair illustration, the researchers began with 20 aesthetic descriptors and a base prompt with no descriptor. Embedding distances captured applicability and prototypicality, while similarities among descriptors revealed six clusters. Terms such as elegant and refined represented closely related ideas, whereas whimsical and ornate occupied more distinct regions. After screening the language, the team generated 15 chair images for each descriptor using Grok. Multiple images represented each construct so the experiment would measure the underlying aesthetic rather than react to one unusually strong or weak rendering.
The researchers then fielded 20 pairwise image-and-price choices with 298 Prolific respondents across two waves. Participants completed the survey, including screening and demographic questions, in roughly two to three minutes. Jeffrey argued that the speed reflected how quickly people process visual alternatives rather than poor engagement. Average results favored the category default: only formal and refined improved preference relative to the base image, while most descriptors reduced it. The individual-level estimates told a more useful marketing story. Some styles with weak average performance appealed strongly to meaningful subsets of respondents. For example, 26% preferred opulent chairs to the base design, and 36% preferred bulky chairs. This heterogeneity can reveal niche opportunities that average effects conceal.
For marketers, the method offers a structured way to screen thousands of early-stage design concepts before investing in prototypes. It can also identify styles for distinct consumer segments and estimate the value of visual differentiation from current market offerings. The current study treats descriptors as discrete states, while the next phase will test continuous variation, such as making a product more or less sleek, as well as expanding the initial vocabulary from 20 descriptors to roughly 2,700 words.
Key takeaways:
- Visual design belongs in demand research. Images influence consideration and choice, so concept tests should evaluate aesthetics alongside functional attributes and price.
- Generative AI expands the design space. Researchers can explore concepts beyond products already in the market or ideas produced by one designer.
- Aesthetic language must be screened before use. Descriptors should meaningfully change the image, add something beyond the category default, and remain distinct from other terms.
- Embeddings make early screening practical. Text and image embeddings let researchers evaluate large vocabularies before paying the time and computational cost of image generation.
- Multiple images should represent each construct. Variation within a descriptor helps separate consumer response to the aesthetic idea from response to one particular rendering.
- The method works with established conjoint tools. The chair study combined generated images with price in pairwise choices and analyzed the results using standard hierarchical Bayes methods.
- Average preference can hide valuable segments. Most styles underperformed the base chair on average, yet bulky and opulent designs attracted 36% and 26% of respondents, respectively.
- The business applications extend beyond concept screening. Researchers can guide differentiated product lines, design for distinct tastes, and study switching or cannibalization against existing products.
The AI Value Gap: What Marketing Leaders Do Differently
Imran Ahmed – Head of Measurement, Tech & Lifestyle, Google
Brian Dennehy – Expert Partner, Bain and Company
Hayden Ostendorf – Practice Sr. Manager, Bain & Company
Imran Ahmed from Google and Brian Dennehy and Hayden Ostendorf from Bain & Company examined why widespread AI adoption has produced faster marketing work but limited measurable business impact. The presenters began with the consumer changes driving the need for transformation. Marketers face a flood of content, fragmented attention and product discovery that increasingly happens through zero-click results and large language models. Consumers also expect brands to understand individual preferences. These shifts affect the full marketing value chain, from customer understanding and content creation to activation, measurement and the design of marketing roles.
Bain’s leaders-and-laggards research found a significant gap between expectations and results. Most marketers expect AI to generate substantial revenue or cost benefits, and many report faster execution, but only a small minority see meaningful business outcomes. Marketing leaders are more than twice as likely as laggards to realize revenue and cost benefits. Their advantage comes from three practices: they connect AI investment to a defined business need, they concentrate resources on a few important workflows, and they redesign work so employees can adopt the new processes. Broad access to general-purpose tools may stimulate experimentation, but it rarely produces value without priorities, funding and organizational change.
The Google portion of the presentation showed how data quality can prevent even strong AI systems from achieving their intended result. Automated bidding learns from the goals and conversion signals that advertisers provide. Signal loss caused by cross-device behavior, privacy regulation and platform restrictions can make a successful interaction appear unsuccessful. The model then learns from incomplete feedback, which can create a self-reinforcing bias over time. The cost includes both immediate lost revenue and weaker future intelligence because subsequent decisions rely on distorted training data.
Advertisers can improve performance by recovering privacy-safe signals and sending information that reflects real business value. Google’s Enhanced Conversions for Leads uses hashed first-party data to connect offline outcomes with the ads that generated the lead. The presentation reported that advertisers upgrading from standard offline conversion imports measured 10% more conversions overall, 11% more on YouTube, and 22% more on search. Better measurement alone is not sufficient, however. A binary conversion signal treats every customer as equally valuable, while value-based bidding can distinguish a $100 customer from a $500 customer and eventually incorporate predicted lifetime value. Google cited an average 14% increase in conversion value at a similar return on ad spend when advertisers moved from target CPA to target ROAS bidding.
The speakers closed with three practical actions. Marketers should audit where measurement signals break down and estimate the value being missed. They should align bidding instructions with the business outcome they actually want, including customer quality and value where possible. Finally, they should select one high-impact workflow and redesign it end-to-end with automation built into the future process. The central message was that AI value depends on business strategy, reliable data and operating-model change, rather than tool deployment alone.
Key takeaways:
- AI adoption and AI value remain far apart. Marketers often gain speed before they achieve measurable revenue, market-share, or cost benefits.
- Leaders begin with a business outcome. A centralized roadmap and senior-level priorities keep AI investment tied to specific sources of value.
- Fewer, larger bets outperform scattered experimentation. Leaders concentrate funding and talent on a small number of workflows that can scale.
- Workflow redesign is a major differentiator. Leaders are 3.7 times more likely to redesign processes instead of adding AI tools to existing work.
- Signal loss weakens both current and future performance. Incomplete conversion feedback reduces immediate results and trains bidding systems on misleading outcomes.
- Privacy-safe first-party data can restore useful measurement. Enhanced Conversions for Leads connects hashed customer information with offline conversions and improves the signals available to automation.
- Bidding systems need business-specific value signals. Advertisers should progress from clicks and binary conversions toward conversion value and predicted lifetime value.
- Three actions can start the transition. Run a data audit, align bidding with business goals and redesign one high-impact workflow around AI-enabled execution.
Discovering Textual Drivers of Marketing Outcomes from LLM Internals: A Sparse Autoencoder and Multi-Agent Pipeline
Shane Wang – Professor of Marketing, Pamplin College of Business, Virginia Tech University
Shane Wang of Virginia Tech presented a framework for discovering textual features that influence marketing outcomes by examining the internal representations of a large language model. Traditional text analysis often concentrates on what people say by measuring words, topics, sentiment, or semantic similarity. Shane’s work focuses on how people communicate, including formatting, organization, punctuation, emphasis and discourse structure. These signals are diverse and context dependent, and common preprocessing can erase them. For example, converting “LOVING this place!!!” to “love this place” preserves basic meaning but removes expressive information that may affect readers.
The framework uses mechanistic interpretability to open the LLM rather than relying only on its final output. The study processed 432,248 Yelp reviews of 4,600 Mexican restaurants in 10 metropolitan areas through a frozen GPT-2 small model. The researchers captured residual-stream activations before the eighth transformer layer, producing a 768-number representation for every token across more than 51 million tokens. A sparse autoencoder then separated these entangled representations into a much larger vocabulary of features while limiting the number active for any word. This process produced nameable patterns that conventional dictionaries or embeddings might miss.
The team connected the discovered features to review helpfulness. For each review, it recorded the strongest activation of every feature and used XGBoost to predict helpful votes while controlling for restaurant baselines, review order, star rating, length, readability, lexical variety, and valence. Among 18,305 features that appeared in at least 1% of reviews, the model ranked those that most affected the prediction. Human review and a multi-agent interpretation process then examined high-activation examples, proposed plain-language explanations, challenged the proposed interpretations and audited the final reports.
Two prominent features involved how reviewers organized information. One used formatted lists, while the other used structural discourse markers such as “Update,” “Overall” or “Edit,” at the beginning of a paragraph. The researchers tested these features in two between-subject experiments with 566 and 564 participants. Both increased stated helpfulness. Formatted lists raised the mean from 5.15 to 5.45 on a seven-point scale, while structural discourse markers raised it from 5.09 to 5.41. The effects remained significant after controls and did not vary significantly across the four stimulus pairs.
For practitioners, the pipeline requires an existing text corpus and a measurable outcome, but it does not require a predefined dictionary or hypothesis about which writing features matter. It can return a ranked set of writing patterns, examples from the organization’s own data and descriptions that researchers can turn into experiments. A review platform could prompt writers to organize their comments, use structural cues in review ranking, and place less emphasis on length. The same method could analyze which phrases in advertisements drive clicks, which product descriptions convert, which service-agent responses predict resolution, and which complaint patterns signal escalation.
Key takeaways:
- How text is written can matter beyond what it means. Formatting, organization, and expressive cues can affect consumer response even when semantic content stays constant.
- Standard pre-processing can remove useful signals. Normalizing capitalization, punctuation, and structure may discard features that help explain marketing outcomes.
- LLM internals offer a new source of text features. Residual representations preserve lexical, semantic, pragmatic, and paralinguistic information that final model outputs do not expose directly.
- Sparse autoencoders make hidden representations interpretable. They separate entangled neural activations into a large set of more distinct patterns that researchers can rank and examine.
- Outcome data determines which features matter. The pipeline links feature activation to helpful votes, clicks, conversion, satisfaction, or another observed business result.
- Human interpretation and validation remain essential. Researchers inspect examples, challenge feature labels, and use controlled experiments to establish whether discovered patterns cause the outcome.
- Simple structural cues improved review helpfulness. Formatted lists and discourse markers produced statistically significant gains across two experiments and four stimulus pairs.
- The approach produces testable operating guidance. Organizations can turn discovered patterns into writing prompts, ranking signals and A/B tests using text they already collect.
Panel Discussion: AI and the Future of Marketing Research
Speakers:
Raymond Burke – Professor of Marketing, Kelley School of Business, Indiana University
Leabe Commisso – Senior Vice President of Strategic Growth, Ipsos
Ankit Dhawan – Founder & CEO, BluePill AI
Stefano Puntoni – Professor of Marketing, The Wharton School
Moderator: Scott McDonald, Ph.D. – President & CEO, ARF
This dynamic and insightful panel, consisting of Raymond Burke of Indiana University, Lieb Cmiso of Ipsos, Ankit Dhawan of BluePill AI, and Stefano Puntoni of The Wharton School, and moderated by the ARF’s Scott McDonald, discussed the rapid adoption of AI in marketing research and the gap between widespread enthusiasm and disciplined implementation. The speakers agreed that organizations remain optimistic: employees use AI frequently, expertise and investment continue to grow, and many managers report positive returns. Yet, adoption often lacks coordination. Giving every employee access to a general-purpose assistant does not establish a research strategy, a shared standard of quality, or an operating model that can scale successful practices across the organization.
Current applications span research and marketing work. AI supports search and synthesis, qualitative coding, video analysis, content generation, self-service analytics, AI-moderated interviews, and synthetic customers or digital twins. The panel expected many of these tools to become mainstream, especially when vendors provide simple interfaces that do not require client teams to integrate models and data themselves. At the same time, the speakers stressed that synthetic respondents require human or behavioral foundations. Synthetic data can extend scale and accelerate testing, but it cannot eliminate the need for people, context or reliable observations of how consumers behave.
Validation emerged as the central challenge. Human survey data provide one benchmark, but stated responses can differ from actual behavior. Large language models may predict what people say while missing physical and situational influences such as product availability, shopping companions, or features of the retail environment. The panel described validation as a continuing process: comparing synthetic outputs with human research, testing predictions against observed outcomes, and recalibrating models as new evidence arrives. Because populations and opinions change, synthetic panels and digital twins also need regular updates rather than one-time training.
The discussion also highlighted the absence of consistent industry terminology. “Synthetic data” can describe aggregate modeled records, LLM-generated answers or digital twins grounded in detailed individual behavior. These approaches carry different capabilities and risks, so researchers must explain how the data were created, what human evidence anchors the model and which decisions the output can support. The desired level of accuracy also matters. Large organizations making consequential decisions may require extensive legal, technical and methodological review, while smaller firms may accept directional evidence in exchange for speed and lower cost.
Looking ahead, the panel urged marketers to pursue growth and innovation rather than treating AI only as an efficiency tool. Marketing leaders should help shape AI investments around customer acquisition, better products, and richer scenario planning, rather than leaving programs solely to operations, IT, and finance. Build-versus-buy decisions should reflect organizational capability: internal development can be risky without deep model expertise, while outside products require careful evaluation because weak systems can appear convincing. The speakers anticipated more specialized behavioral or world models designed to simulate people, along with blended research designs that combine human and synthetic evidence.
Key takeaways:
- AI optimism remains high, but adoption is often uncoordinated. Frequent use and rising investment do not guarantee shared standards, scalable workflows or measurable business value.
- General access to AI tools does not constitute a strategy. Organizations need clear research objectives, education, and governance rather than expecting employees to discover best practices independently.
- Synthetic evidence still depends on human foundations. Digital twins and modeled data require real people, observed behavior, or other credible evidence for calibration.
- Validation should include behavior as well as stated responses. Survey comparisons help, but researchers should also test predictions against purchases, movement, context, and other observed outcomes.
- Models require continuing calibration. Consumer populations, opinions, and environments change, so synthetic panels and twins need refreshed data and feedback from actual results.
- Clear terminology is necessary for responsible evaluation. Researchers should distinguish aggregate synthetic data, LLM-generated responses, and individual digital twins, and document how each was built.
- Accuracy requirements should match the decision. High-stakes decisions demand stronger validation and review, while early exploration may only require directional evidence.
- Marketing should direct AI toward growth and innovation. The largest returns may come from better customers, products, and scenarios, not only from faster or cheaper research.
Collaborative Intelligence: Reconstructing the Invisible Consumer From Fragmented Data
Alice Li – Associate Professor, Fisher School of Business, The Ohio State University
Alice Li of Ohio State University addressed a familiar problem in customer analytics: firms collect abundant information yet struggle to form a coherent view of individual consumers. Survey answers, transactions, product use, and open text often sit in separate systems, while some customers have rich records and others leave only a few traces. Customers can also become invisible when their satisfaction and purchasing resemble the majority even though different needs or preferences drive those outcomes. The proposed collaborative intelligence system organizes the evidence that remains in a usable customer representation for managerial decisions.
The approach deliberately reconstructs rather than invents. It does not create synthetic people or impute every missing field. Humans define the business objective, key outcomes, and meaningful boundaries among customer attributes. AI then connects signals across those groups at scale and produces three linked outputs: a readable narrative profile, a satisfaction prediction, and a prediction of subsequent purchase behavior. Missing facts remain explicitly unknown, so the representation should not be mistaken for a recovered record.
The researchers tested the system with 13,771 creative software users in the United States and Japan. The data included 26 survey questions organized into 16 attribute groups, brand satisfaction, and later subscription records. In a fixed-model stress test with no retraining or imputation, prediction quality declined gradually as information disappeared. With 90% of the behavioral and attitudinal groups withheld at random, satisfaction discrimination fell from 0.730 to 0.601, and purchase discrimination fell from 0.636 to 0.577. Both remained above chance, although the losses were meaningful and the experiment does not imply that every real-world pattern will behave the same way.
The study also showed that not every missing field has equal value. Removing decision-preference information reduced satisfaction Area Under Cover by 0.031, compared with 0.024 for software engagement and 0.008 for creative identity. This gives managers a practical way to prioritize future data collection: test which missing modules weaken the decision-relevant prediction most, then weigh that value against the cost and feasibility of collecting better information. The narrative profiles also aligned with human-written descriptions at approximately the same level that human annotators aligned with one another, though the researchers noted that GPT-generated reference profiles supplied the training targets.
A richer representation revealed a group that an outcome-only dashboard missed. The model identified 1,226 consumers, or 10.5% of the discovery sample, whose satisfaction and purchase frequency resembled the mainstream segment, but whose underlying attribute patterns differed. Alice reported that this group bought more expensive products and spent about 15% more on average. The model placed greater emphasis on cost sensitivity when predicting their purchases, illustrating how customers can reach similar outcomes through different pathways. These patterns are diagnostic rather than causal, but they can direct managers toward better segmentation, follow-up research and more relevant tests.
Key takeaways:
- More data does not automatically create a coherent customer view. Fragmented systems and uneven records can leave important consumers difficult to interpret.
- The method reconstructs observed evidence without filling in unknown facts. Its narrative is a usable representation, not a synthetic person or completed customer record.
- Human judgment establishes the analytical boundaries. Managers define the objective, outcomes, and attribute groups, while AI connects the information consistently at scale.
- Partial records can retain decision-relevant signal. Even after 90% of the tested attribute groups were withheld at random, satisfaction and purchase predictions remained above chance.
- The pattern of missingness matters. Managers should stress test the gaps they expect to encounter instead of relying only on average missing data performance.
- Data collection should focus on the modules that matter for the chosen outcome. The measured loss in prediction quality can help balance information value against collection cost.
- Similar outcomes can conceal different customer pathways. The richer representation exposed a 10.5% segment that conventional purchase and satisfaction views did not separate.
- Model explanations guide investigation rather than establish causality. Differences in model emphasis can generate hypotheses for segmentation and testing, but they do not prove why customers behave as they do.
Synthetic Conjoint Methods for B2B Business Decisions
Rogier Verhulst – CEO, Kwantumlabs.ai
Marco Vriens – Founder Kwantumlabs.ai
Rogier Verhulst and Marco Vriens of KwantumLabs.ai tested whether synthetic buyers can reproduce the results of human B2B buyers in this conjoint study. B2B provides a demanding setting because purchases involve multiple decision makers, long buying cycles, switching costs, and scarce respondents. These same constraints make synthetic research attractive, but they also make generic large language models especially vulnerable to unrealistic assumptions. The presenters argued that a strong top-line match can conceal serious errors in how the model represents buyer preferences.
The benchmark study included 160 real buyers of qualitative research software. The conjoint study covered nine attributes and 27 levels, with two product packages and a walk-away option in each task. The synthetic study used 26 personas drawn from 116 recorded buyer conversations, including sales calls and depth interviews. The researchers evaluated five ways of constructing the AI panel, ranging from a generic buyer to grounded personas with decision rules, category context, and additional detail. They compared synthetic and human results using the walk-away gap, attribute-importance ranking, and whether preference moved in the same direction across feature levels.
The bare model performed poorly on the most basic market outcome. It walked away from only 2.3% of choices, compared with 46.6% among real buyers, producing a 44 percentage-point gap. Grounding the panel in real buyer conversations reduced that gap to 19 points. Adding buyer-specific decline rules and brief instructions reduced it to 3.2 points, within the 3.3-point difference observed between two samples of human buyers. A shorter persona combined with a category overview came even closer on the walk-away rate, missing the human benchmark by 1.2 points.
No approach performed best on every criterion. The persona with decline rules produced the strongest attribute-importance correlation with humans at 0.80, while the shorter persona with category context achieved only 0.68 despite its close adoption estimate. Agreement on feature-level direction remained weak, ranging from 44% to 59% across the first four approaches against a 71% human benchmark. Adding more details also failed to improve the panel: the fifth approach used 44 fields and stricter decline rules but overshot the human walk-away rate by 16 points. The results suggest that longer prompts and richer personas can distort behavior when they overemphasize selected constraints.
The aggregate results also hid important errors. One approach matched the overall walk-away rate within a point because opposite mistakes among small and large buyers canceled each other. At the individual level, an AI stand-in correlated with its matched buyer at only 0.21.
The presenters therefore recommended using synthetic conjoint to screen concepts, narrow attributes, and form hypotheses before human fieldwork, rather than as a standalone basis for pricing. Teams should validate every method against a full scorecard, inspect the segments where decisions will be made, and retain a matched human sample to correct residual bias. The evidence comes from one product, category and model, so replication remains necessary.
Key takeaways:
- Generic AI buyers substantially overstate willingness to choose a B2B offer. The bare model rarely selected the walk-away option that real buyers chose nearly half the time.
- Real conversations provide a stronger foundation than generic prompting. Personas extracted from buyer interviews reduced the walk-away gap from 44 percentage points to 19.
- Decision rules can improve selected market-level outcomes. Buyer-specific decline criteria brought the synthetic walk-away rate within the sampling variation observed between human samples.
- The best method depends on the business question. One approach estimated adoption most closely, while another produced the strongest ranking of feature importance.
- More persona detail can reduce accuracy. Adding fields and stacking stricter rules shifted the panel from too willing to buy toward excessive rejection.
- A favorable average can conceal offsetting segment errors. Synthetic results should be checked among the specific buyer groups where pricing and packaging decisions will be applied.
- Individual digital twins remain unreliable for this use case. Market-level agreement did not translate into accurate predictions for each matched buyer.
- Synthetic conjoint works best as a front-end research aid with a human anchor. It can screen ideas and focus fieldwork, but pricing decisions still require human evidence and calibration.
When AI Incentives Help and Hurt
Jia Li – Associate Professor, Wake Forest University
Jia Li of Wake Forest University examined what happens when artificial intelligence is employed to make management decisions. AI can evaluate every customer service interaction, provide feedback quickly, and assign incentives before work begins. Those capabilities give organizations new ways to manage at scale, but they also change how employees respond. Drawing on two field studies in a global company’s online customer service operation, Jia showed that an accurate prediction does not guarantee an effective evaluation or reward system.
The first study changed how customer service agents were evaluated and paid. The existing system relied primarily on customer ratings, while the treatment group gradually shifted to an AI score that eventually carried 100% of the weight. During an earlier “sleeper” period, the AI scored interactions without agents knowing, creating a baseline for how its assessment related to customer ratings before employees could adapt their behavior. A control group continued under the existing evaluation system, allowing the researchers to estimate the effect of making AI the sole evaluator.
Once agents knew the AI determined their evaluations and incentive pay, they improved the behaviors the scorecard emphasized. Difference-in-differences estimates showed higher scores for service attitude, emotion management, communication and efficiency. Measures of problem identification and solution quality declined, however. The overall AI score rose by 0.066 while customer ratings fell by 0.112 relative to the sleeper baseline. The AI measure initially aligned with customers, but the alignment weakened when employees began optimizing their work for the metric.
The second study tested whether AI should decide when employees receive a bonus opportunity. A model predicted which incoming cases would be difficult, and 635 agents were randomly assigned over six weeks to treatment or control conditions covering 52,542 eligible chats. Treatment agents saw a bonus opportunity before responding to what were predicted to be difficult cases, while control agents handled comparable cases without the opportunity. On average, the incentive produced little change in behavior or customer outcomes. Beneath the average, however, the same bonus generated sharply different responses.
The current analysis identified three recurring patterns. In some situations, the bonus encouraged more productive diagnosis and handling, improving ratings and reducing repeat contacts. In others, agents devoted more effort to repair, which reduced repeat contacts without necessarily improving ratings. A third pattern involved rushed or thinner responses, lower ratings and more repeat contacts. These patterns describe context-dependent responses rather than permanent employee types. Li concluded that organizations must move from “need-aware” AI, which predicts where help may be required, to “response-aware” AI that estimates how a particular management action will affect both employees and customers. The management system requires validation beyond the predictive model itself.
Key takeaways:
- AI management changes employee behavior, not only measurement. Once workers understand how the system evaluates them, they adapt to the metric and can alter the relationship between the score and the underlying goal.
- A measure can predict well before it becomes a target. The AI score tracked customer ratings during the hidden baseline, then diverged after it controlled evaluation and pay.
- Improvement on the scorecard did not mean better customer service overall. Delivery-related behaviors rose while problem diagnosis, solution quality and customer ratings declined.
- Organizations need an outcome check outside the AI evaluation system. Customer experience and other independent measures can reveal whether employees are improving the work or merely optimizing the score.
- Predicting which cases need attention does not identify the right intervention. A difficult case may warrant support, but that does not establish that a bonus will improve the response.
- Average incentive effects can hide meaningful differences. The bonus produced little overall change even though it improved some interactions and harmed others.
- Behavior and customer outcomes must be interpreted together. Faster work may indicate efficient diagnosis or a rushed answer, while slower work may reflect useful repair or unproductive delay.
- AI management systems require causal testing and continuing validation. Organizations should test how employees respond to AI-directed decisions and whether those responses benefit customers.
Validating Enterprise AI Retrieval Systems for Structured Analytics Data
Harini Devulapalli – Senior Manager, Chewy
Harini Devulapalli of Chewy argued that the reliability of enterprise AI assistants depends not only on choosing the right model and supplying accurate data, but also on how structured tables are prepared for retrieval. She illustrated the problem through “Maya,” an email marketer whose assistant returns one correct row alongside two plausible near matches when asked for an open rate. If the user must verify every answer manually, the assistant has not reduced the work. Worse, weak retrieval can select the wrong row, encourage the model to invent a number or add unnecessary token and processing costs.
The study compared four ways of representing the same tabular information: labelled fields, plain-language sentences, a combination of sentences and fields, and relations built from entities and links. The researchers applied these methods to three kinds of business data and tested 30 questions per dataset while holding the model and instructions constant. Only the data preparation method changed. The purpose was not to identify a universal winner, but to see how representation affected whether the correct supporting rows reached the model.
Each format offered a different tradeoff. Fields were the most dependable across table types and never collapsed in the tests, but they were slowest on questions containing several conditions and could still confuse similar values. Sentences produced the single best result yet performed poorly on coded or identifier-heavy tables and substantially increased the amount of text processed. Combining sentences and fields was fastest and performed best on the financial dataset but had the weakest average accuracy and required two pipelines. Relations were designed for questions spanning multiple entities but recorded the lowest recall and were the most expensive approach to build and host.
Question structure proved as important as data structure. Precise, multi-condition requests performed worse than simple questions because records satisfying only some conditions looked very similar to the true answer and crowded it out during retrieval. Across formats, narrowing down a list scored around 0.38 to 0.42, while asking for one number scored around 0.69 to 0.80—approximately twice as well. The results also exposed a speed tradeoff: fields slowed on harder questions, while sentence retrieval remained nearly constant. Overall averages therefore concealed the particular datasets and question types on which each method failed.
Harini recommended choosing a representation for the table and use case rather than enforcing a company-wide default. Organizations should identify the question type before retrieval, route different questions to the appropriate preparation method, display the source rows behind every answer and instruct the system to refuse when the evidence is insufficient. Validation should use known-right answers from the organization’s own tables and measure retrieval accuracy, latency, permissions, workflow fit and cost separately. She also noted that this initial study did not establish performance at millions of rows, test whether the model’s final generated answer was correct, or compare retrieval with text-to-SQL. The broader message was that data representation can move results more than a model change—and that enterprise reliability must be demonstrated in the actual operating context.
Key takeaways:
- Data representation is part of the AI system, not a pre-processing detail. Preparing the same table in a different form changed retrieval quality, speed and cost even when the model and instructions stayed constant.
- No representation won across every dataset and question type. Enterprise teams should select methods by use case instead of imposing a single organization-wide standard.
- Fields offered the most consistent performance. They did not collapse on any tested table, although complex conditions increased latency and similar values remained difficult.
- Sentence-based retrieval could excel but was fragile. It produced the study’s best individual result yet struggled with coded data and increased the amount of text processed.
- Hybrid and relational approaches introduced their own costs. Combining formats helped financial questions but required two pipelines, while relations had the lowest recall and highest implementation burden in these tests.
- Precise questions require separate validation. Multiple conditions created many plausible near matches, causing retrieval performance to fall to about half that of simple one-number questions.
- Overall averages can hide operational failure. Results should be segmented by dataset, question structure, latency and cost to reveal where each approach breaks.
- Trust requires visible evidence and refusal rules. Assistants should show the source rows supporting an answer and decline to guess when retrieval does not provide adequate evidence.
From Reviews to Actionable Insights: An LLM-Based Approach for Attribute and Feature Extraction
Khaled Boughanmi – Assistant Professor of Marketing, Cornell University, Johnson Graduate School of Management
Khaled Boughanmi of Cornell presented an LLM-based approach for converting customer reviews into specific insights that managers can act on. The research, conducted with Kamel Jedidi of Columbia University and Nour Jedidi of the University of Waterloo, addresses a limitation of traditional text analysis: ratings show whether customers are satisfied, and topic models can identify broad themes, but neither necessarily explains what a company should change. The framework distinguishes broad perceptual attributes, such as customer service, from concrete features, such as staff friendliness, order accuracy, or Wi-Fi speed.
The researchers analyzed 20,000 Yelp reviews covering 722 Starbucks locations in the United States and Canada, written by 10,264 unique reviewers between 2005 and 2022. Their process begins with an exploratory phase in which reviews are batched and the LLM identifies attributes and features. The resulting lists are manually consolidated to remove duplicates, merge related ideas, and standardize terminology. In a confirmatory phase, each review is divided into sentences. The model then assigns relevant attributes and features and scores the sentiment expressed toward each. This converts unstructured review text into a structured dataset that can support statistical analysis and dashboards.
The team validated the method against human coding on 300 reviews. Boughanmi reported that the human and GPT-4.1 mini distributions for attribute mentions and sentiment were statistically indistinguishable in the tests presented. The workflow was also substantially faster: a human coder required approximately six minutes per review, while the LLM approach took about two seconds, or 180 times less time. The researchers processed the full set of reviews overnight, while comparable continuous human coding would have required roughly six months.
The Starbucks application demonstrated why feature-level analysis matters. Customer service appeared in 88% of reviews, more often than any other attribute, with positive and negative sentiment nearly balanced at 46% and 42%. Within customer service, management and staff friendliness, expertise and professionalism were the most frequently mentioned features. A model using the extracted, interpretable features also performed strongly when predicting customers’ ratings. In the attribute-importance analysis, customer service accounted for 40% of rating importance, compared with 14% for coffee and beverages. Boughanmi summarized the managerial implication directly: Starbucks operates as much in the service business as in the coffee business.
Because the framework retains store, time, and feature information, managers can diagnose where problems occur rather than relying on a company-wide average. The analysis can track sentiment over time, compare locations, identify whether a store’s weakness stems from staff behavior, wait time or another feature, and position stores on broader dimensions such as the coffeehouse environment and the product offer. Khaled emphasized that these results are based on observational review data and are not causal. Simulated improvements, such as changing negative staff-friendliness mentions to positive ones, should therefore be treated as suggestive prioritization tools rather than forecasts of what an intervention will cause. The framework’s value lies in scaling interpretable voice-of-customer analysis while keeping the output grounded in specific review content.
Key takeaways:
- Customer reviews become more useful when broad themes are linked to actionable features. Knowing that customer service is a problem is less diagnostic than identifying staff friendliness, wait time or order accuracy as the source.
- The framework separates discovery from measurement. An exploratory phase builds the attribute-feature taxonomy, while a confirmatory phase applies it consistently across individual review sentences.
- Breaking reviews into sentences helps constrain the LLM. Smaller, clearly defined tasks reduce room for unsupported interpretation and make feature and sentiment assignments easier to validate.
- Human validation is essential. On the 300-review validation sample, the LLM produced attribute and sentiment distributions that closely matched those of human coders.
- LLMs can dramatically increase the scale of qualitative analysis. Processing took about two seconds per review versus roughly six minutes for a human coder, enabling 20,000 reviews to be analyzed overnight.
- Customer service was the dominant driver in the Starbucks application. It appeared in 88% of reviews and accounted for 40% of rating importance, far more than coffee and beverages at 14%.
- Structured review data can support local operating decisions. Store-level dashboards can reveal where specific issues occur and distinguish problems involving staff, speed, product quality, access or the physical environment.
- Observational insights should not be treated as causal effects. The results can guide priorities and hypotheses, but proposed interventions still require experiments or other causal validation.
Why Your AI Persona is Lying to You: The Danger of Data Overload in Synthetic Populations
Luiz G. Duarte, Ph.D. – Sr. Research Consultant, Wortya
Luiz G. Duarte (Wortya) analyzed whether a synthetic respondent could express the ideas and personalities as well as the purchasing behaviors of real humans. The team had built AI digital twins of real Peruvian families; however, the digital twin of a mother completely hallucinated and misrepresented her real concerns about her children leaving home. This disconnect triggered three months of rigorous empirical research as explained by Adriana Rocha via video to understand the circumstances when synthetic personas reflect real human behavior and when they are they telling untruths.
The Wortya researchers executed over half a million LLM interactions and 11,000 synthetic interviews across six commercial AI models. They pitted these AI models against two years of actual, ground-truth purchase data from 100 real U.S. households. This allowed the team to measure the exact correlation between what an AI respondent predicts and what real consumers actually put in their shopping carts.
A major finding directly challenged the market research industry’s habit of feeding AI models massive persona prompts. Adding more data degrades AI accuracy. In routine grocery predictions, adding personality data actually reduced predictive accuracy by up to 160%. Stripping away the personality text and providing only raw behavioral data significantly improved the AI’s accuracy.
LLMs are text-based narrative generators and are 2.5 times better at simulating choices tied to personal identity, like being an organic shopper, than predicting raw transactional data like basket size or store-switching frequency. LLMs have no narrative anchor and default to generic, inaccurate guesses.
The single biggest failure point for marketers is that LLMs cannot simulate true brand loyalty or loss aversion. If a competitor drops prices by ten cents, the AI persona switches instantly, even if the prompt explicitly tells it the household is 100% loyal to the original brand. Alignment training forces AI models to behave like hyper-rational economic agents.
The solution was to develop domain-aware cognitive routing, illustrated below. Instead of using a single prompt, a modern synthetic research platform must run a cognitive router. To test brand message resonance, route to the Identity Domain with OCEAN traits. To predict routine grocery purchases, route to the Operational Domain and feed the AI strictly raw behavioral numbers.
Synthetic market research is moving fast, but key areas still require rigorous study, including fusing proprietary first-party enterprise data into synthetic prompts without causing layer interference, building dedicated synthetic panel LLM architectures, and developing the ability to map synthetic exploratory sentiment directly to ground-truth retail sales.
Key takeaways:
- A more detailed AI persona is not necessarily a more accurate consumer. AI may be better at simulating how consumers think and talk about themselves than predicting exactly what they will buy.
- Different types of synthetic consumers may be appropriate for different research questions. Personality and identity information may be useful when researchers want to understand attitudes, but actual transaction data appear more important when the objective is to simulate concrete purchasing behavior.
- Marketers should treat synthetic panels as a filter for rapid, early-stage exploration.
- Use synthetic populations in a hybrid workflow. Test a large number of concepts synthetically and validate the final three with human panels.
- Invest engineering effort in prompt architecture not larger models.
- Prompt design explains 74% of accuracy, while the specific AI model accounts for less than a tenth of a percent.
Extracting Consumer Insight from Text: A Large Language Model Approach to Emotion and Evaluation Measurement
Peter Danaher – Professor of Marketing and Econometrics, Monash University
The goal of Peter Danaher of Monash Business School and his colleagues was to extract emotions and evaluations from online reviews. Danaher presented a large-language-model approach for converting unstructured customer text into standardized measures of consumer emotions and marketing evaluations. The research focused on extracting 16 consumer-relevant emotions, including joy, anger, discontent, contentment, and optimism, as well as four marketing evaluations: trust, commitment, likelihood to recommend, and sentiment related to the marketing mix. The goal was to turn online reviews and other customer-generated text into structured numeric variables that could be used to profile customer emotions, diagnose perceptions of brands, and predict outcomes such as ratings and purchases. Businesses value knowing how their customers perceive company performance and where pain points occur.
Rather than having outside coders read reviews and decide which emotions are present, Peter and his collaborators asked consumers to write about a product or service they had recently experienced and then rate their own text using established academic measurement scales. This approach provided the model with both the written review and the consumer’s own assessment of what that review expressed. The training data consisted of 2,523 survey responses from panelists in the U.S., U.K., Canada, and Australia. Respondents averaged 39 years old and 57% were women. They were asked to describe a product or service experienced during the previous month.
The researchers started with the open-source Llama 3.2 3B model and fine-tuned it using these consumer survey responses. The model was trained to identify whether individual emotions were present and to classify evaluations such as trust, commitment, recommendation, and sentiment as positive, negative, or neutral. The measures were anchored in established scales, including Richins for emotions, Morgan and Hunt for trust and commitment, Net Promoter Score for recommendation, and Gaski and Etzel for marketing evaluations. The resulting system, called Linguistic eXtractor (LX), was compared with widely used transformers and proprietary LLMs, capable of similar tasks, such as BERT, RoBERTa, and DeepSeek-V3.
Testing showed that LX achieved the strongest overall performance on the survey data. It also performed strongly when applied to separate Amazon and Yelp review datasets. The testing results demonstrated that the fine-tuned approach could identify marketing-relevant emotions and evaluations more accurately than the benchmark models tested.
The presentation also examined whether emotions contained in reviews related to actual purchasing behavior. The behavioral analysis used data from a large U.K. online retailer covering 197,605 unique panelists and approximately 540,000 clickstream observations over two months. Shoppers viewed products in home and garden or technology categories, with product pages containing information such as price, star ratings, review volume, and review text. The analysis linked review ratings, the emotional content of reviews, and purchase behavior through a mediation model.
Danaher concluded that large language models can summarize customer text in many ways, but the proposed method is specifically designed to extract marketing-relevant information using validated measurement scales and consumer-generated training data. This method outperforms the other tested models for emotion and evaluation prediction and identifies discontent and contentment in online reviews as having a significant impact on future purchases.
Key takeaways:
- LLMs have the capability to summarize online reviews and any textual data in multiple ways.
- The LX model is designed to capture aspects that are relevant to marketing using validated scales, and it is fine-tuned on reliable data.
- This method outperforms other LLMs for emotion and evaluation prediction and also shows promise as a verifiable and cost-effective way to extract information from customer feedback.
- Emotions of discontent and contentment expressed in online reviews have a significant impact on future purchases.
Closing the Innovation Forecast Gap: An AI/ML Approach to New Product Forecasting and Incrementality at Kenvue
JJ Huang – Director, Global Data Science, Kenvue
Raul Jurado – Principal and Planning Intelligence Lead, Cloverpop
Lana Klein – Practice Lead, Cloverpop
JJ Huang of Kenvue, Lana Klein and Raul Jurado, both of Cloverpop, presented an AI-enabled approach to improve forecasting for new product launches. Ninety percent of new product launches fail, and the typical forecast error for new product sales is 40-55%. This challenge can be attributed to three factors: new products lack sales history, important characteristics such as benefits, claims, and positioning are difficult to quantify, and human judgment can introduce optimism bias. The current methods of assessing innovation, such as concept testing, trial & repeat models, and look-alike analogs, do not provide one reusable model that brings product attributes, launch execution, and incrementality together.
The Kenvue/Cloverpop approach, CloverPlan AI™, was designed to bring product characteristics, historical experience, and launch execution together while answering two separate questions: how much a new product will sell and how much of those sales will be incremental to the business rather than cannibalizing existing sales. The platform contains two connected forecasting engines: one estimates total new-product sales and the other estimates incremental sales. The models consider what the product is, its attributes and type of innovation, the brand launching it, and how the launch will be executed. The team assembled information from previous products and launches, converted product descriptions into measurable variables and trained machine-learning models to identify which combinations of characteristics were associated with sales outcomes. Random forests and gradient-boosting models were used to capture complex relationships among the variables, while Optuna was used to tune the models for forecast accuracy.
Generative AI was used to create data that traditional market datasets do not contain. Standard sales data may show units and price but not detailed information such as product benefits, ingredients, or intended consumers. The team used AI to collect and formalize attributes, benefits, and target-consumer information, then appended those variables to sales data. Human judgment remains part of the process through assumptions and model guardrails.
To test performance, out-of-sample validation using historical launches was conducted, which tested the model on launches that were not used to build it. The outcome revealed approximately 85% accuracy across the markets and categories examined, and this framework has been deployed across six markets and 30 market/category combinations. The system does not replace human judgment but provides teams with a common evidence base for decision making. The speakers also identified limitations: the model is difficult to apply to genuinely disruptive products, performs less well where few comparable products exist, and does not explicitly model organic social activity or predict virality.
Key takeaways:
- Innovation is the most expensive guess in consumer goods: $160B+ is spent every year on R&D and innovation that does not translate into commercial returns.
- None of the current methods of assessing innovation provided one reusable model that combined product attributes, launch execution, and incrementality.
- However, Kenvue and Cloverpop created the needed model that supports earlier go/no-go decisions.
- Concepts can be refined before launch. Users can change pricing, product, and execution assumptions and evaluate how those changes affect projected performance.
- Distribution is the single biggest driver of new-product sales.
- Scenario planning changes the forecasting discussion. Teams can focus on debating assumptions rather than debating only the final forecast number.
- Forecasts can become part of an ongoing learning system. Organizations can compare forecasts with actual outcomes and close the learning loop.
- Organizational adoption remains a major challenge. Companies need to demonstrate model performance, earn users’ trust, and establish processes to maintain capability over time.
DAY 2
Opening Remarks
Scott McDonald, PhD – CEO & President, ARF
GEO in the Loop: A Framework for Integrating Generative Engine Optimization Into Marketing Analytics Workflows
Manish Hira – Data Analysis Manager, Capital One; Ph.D. Scholar, Virginia Commonwealth University
Manish Hira (Capital One) described Generative Engine Optimization (GEO), also called answer engine optimization, as an emerging marketing analytics channel created by the growing use of generative AI for consumer discovery. Unlike traditional search, where brands compete to appear among search-engine links, generative AI systems synthesize information into direct answers tailored to a user’s question. As a result, this process changes what marketers need to measure because AI-generated answers are non-deterministic: brands can be absent, inaccurately represented, or differently represented when the same question is asked multiple times.
Manish’s presentation included a synthesis of research from academia and industry. The combined research covered approximately 350,000 AI responses across 12+ commercial LLMs, 100 brands, and 52 categories. These findings are organized into a framework designed to incorporate GEO into an existing marketing analytics system rather than treating it as an entirely separate discipline.
The research focused on whether brands are included in AI answers, whether the information is accurate, and whether the answers are reliable over repeated queries. Manish proposed extending the traditional marketing analytics stack to GEO through four areas: measurement, diagnostics, attribution/forecasting, and governance. Measurement covers descriptive indicators such as rank or share of voice. Diagnostics identify changes or instability in those indicators. Attribution and forecasting connect GEO activity with broader marketing measurement, while governance addresses risks such as inaccurate or unsafe brand representations. Five GEO measures that can fit within this framework were identified: mention rate, share of voice, first mention rate, position within an answer, and description accuracy.
Manish also describes GEO as a funnel rather than a single visibility metric. A brand first needs to be available to the AI system through its existing knowledge or information retrieved from the web. It then needs to appear within the generated response. The answer may include a citation, the citation may generate a click, the click may lead to adoption, and that activity may ultimately produce an economic benefit. Therefore, marketers need an integrated view of performance across these stages rather than relying on any one GEO metric. Existing analytics systems can be expanded to accommodate GEO measures rather than rebuilt from scratch.
Key takeaways:
- All brands are struggling with GEO issues.
- AI engines are an active customer acquisition channel. Brands should measure them with the same rigor as search, social, and paid media.
- GEO metrics integrate into existing marketing KPIs and decisions.
- The predictive frontier is designed. However, linking answer share to forecasting and MMM is an empirical, testable question.
- Marketers have the framework for GEO, but alignment is needed for GEO to prove its value.
- Limitations include multi-engine prompt sensitivity, which requires continuous re-baselining.
- GEO measurement requires repeated testing rather than one-time querying. The same question may not produce the same brand representation each time.
- Marketers should develop a content strategy for both GEO and SEO. SEO is still important.
- Consumer query wording materially affects brand visibility. Explicitly naming a brand in the question produced a 62% probability that the brand would appear in the answer while generic questions might produce no brand appearance.
- Content substance is more important than cosmetic formatting. Adding topical relevance and verifiable information, such as dates or explicit pricing generated a 43% visibility gain, compared with only a one percent gain from cosmetic or formatting changes.
Who Chops the Onions? The Hidden Ingredients in Every Profitable AI Agent
Dr. Adithya (Adi) Pattabhiramaiah – The Sharon A. and David B. Pearce Associate Professor of Marketing and Faculty Director, Center for AI in Business at the Scheller College of Business, Georgia Institute of Technology
Adithya Pattabhiramaiah (Scheller College of Business, Georgia Institute of Technology) stated that AI agents are moving into business processes much faster than firms are learning how to evaluate them. Additionally, expected profits have been slower to result than expected. Although McKinsey has reported that 62% of organizations are experimenting with AI agents, Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027. This presentation analyzed where agentification stands today and provided a framework for evaluating when and how companies should deploy agentic AI. Adi examined six stages, from deciding whether agents belong in a workflow at all to measuring what they actually returned. At each stage, examples from the field were paired with academic and industry evidence to demonstrate where AI initiatives most often succeed, stall, or leak value. This approach is important because simply investing more in AI does not necessarily produce business value.
Six key stages of agentification: (Industry and academic data were provided for each stage.)
- Stage 1: Why use an agent? The first decision is to specify the task, the baseline, and determine why an agent is the right tool.
- Stage 2: Define the return. Choose the success metric and time horizon before deployment.
- Stage 3: Design and measurement. Can the value in the workflow be measured? Build accountability and task-level measurement into the workflow.
- Stage 4: Deployment. Does it work in live operations? Test the full operating model at real volume, including hidden human work.
- Stage 5: People and incentives. Will people use it effectively? Align incentives, monitor skill loss, and verify how people actually use the system.
- Stage 6: Measure the return. Measure incremental return and explain any gap in the metric, timing, workflow, or adoption.
Adi also discussed the research program, consisting of three projects which study theory, observed behavior, and causal effects relating to AI agents. CARMA is a project examining how marketing relationships change when firms and customers delegate decisions to AI agents, with more exchange occurring agent-to-agent. It proposes five governance mechanisms: calibration, accountability, reciprocity, mediation, and adaptivity and translates the framework into six measurable KPIs:
The CARMA field test examines observed governance behavior among AI-native founding teams in a live venture competition, using experiment logs from identified teams rather than relying only on self-reporting. The India RCT is a randomized field experiment with a large Indian NGO comparing three onboarding approaches for instructors serving underserved classrooms: human-only training, agent-only training, and agent-assisted human training in which a master trainer curates and extends AI interactions.
Key takeaways:
- The hardest part of introducing agentic-AI deployment is demonstrating measurable returns.
- Marketers should start with the business decision, not the model. Define the task, baseline, metric, and time horizon before scaling.
- Agentic-AI performance should be evaluated at the individual task level. Aggregate claims about AI value can obscure which specific tasks actually generate benefits or failures. Additionally, self-reported productivity can differ from observed performance.
- Organizational adoption cannot be assumed simply because the technology works. Employee and union resistance may impact deployment and adoption. Also, pilot results may not predict operating requirements at scale.
- AI assistance may create different values for different users. Expert users obtained relatively little benefit from chatbot assistance while people with less expertise benefited more.
- The underlying evaluation questions may remain useful even if individual empirical findings about AI may become outdated quickly.
- Treat human/AI roles as a design choice.
Operationalizing Causal Machine Learning to Optimize Online Ads Parameterization Using Brand Lift Studies Data
Audrey Poinsot – Research Lead, Ekimetrics
Audrey Poinsot (Ekimetrics) explained how causal AI can be integrated into marketing effectiveness measurement while keeping human expertise central to the process. She distinguished predictive AI from causal AI, arguing that predictive AI is poorly suited to marketing effectiveness questions. Predictive AI assumes that the relationship learned from historical data will remain stable when predicting new situations. However, actionable insights require causal reasoning. Marketing effectiveness analysis, by contrast, asks what would happen if a marketer took a different action, which changes the conditions being analyzed. Ekimetrics combines causal inference, which provides a theoretical framework for reasoning about interventions and counterfactuals, with AI and machine learning, which can process large volumes of complex data efficiently. A three-stage, human-driven causal AI framework is used. First, experts explicitly define the causal assumptions behind the analysis, often using a causal graph. Expert knowledge identifies issues such as data quality, noise, and selection bias, while the data are used to challenge assumptions that may be inaccurate. The team iterates between assumptions, usefulness, and statistical checks until it has an acceptable model, typically requiring several iterations. Second, the estimation stage identifies an appropriate causal criterion and framework, then selects machine-learning models that fit the business objective and the characteristics of the data. Models are trained using techniques such as hyperparameter tuning and cross-validation. Because causal effects do not have observable “true labels” against which prediction error can be directly tested, Ekimetrics also quantifies uncertainty and sensitivity to assumptions. Third, humans analyze and validate the resulting effects, including the average estimated effect, differences across groups or situations and the uncertainty surrounding each estimate.
Audrey illustrated the framework with a case study for an international consumer-goods client that had a vast brand lift database. The primary objective was to build actionable global guidelines for designing social media campaigns, depending on specific branding objectives. Although the client could determine which campaigns produced lift, it could not determine which specific campaign characteristics were responsible. Campaign drivers across five stages of the purchase funnel, recall, awareness, consideration, favorability, and purchase intent and two dimensions, effectiveness and cost efficiency were analyzed.
The scale of the analysis and strong correlations among campaign variables made a simple predictive approach inappropriate, according to Audrey, while AI enabled the team to process the large number of estimates required. The analysis identified which campaign characteristics had the largest estimated effects for each funnel stage and then compared different configurations of those characteristics. One example examined format against both lift and cost per lifted user, allowing the client to identify options that balanced effectiveness with efficiency. The resulting guidance was subsequently used by the client’s marketers worldwide. The analysis also identified questions that the available observational data could not resolve, leading the client to obtain additional brand-lift measurements and design-targeted A/B tests.
Key takeaways:
- Human judgment determines whether a causal model is useful enough for a specific business decision. The balance between statistical accuracy and practical usefulness cannot be determined automatically.
- Start with causal inference and use AI as an estimation and scalability tool rather than allowing AI to determine causality. The causal assumptions and identification framework come first, with machine learning subsequently used to estimate the required statistical relationships at scale.
- Observational causal analysis has limits that must be made explicit to decision-makers. Observational studies cannot provide the same guarantees as experiments and transparency about uncontrolled confounders, assumptions, and approximations are essential.
- Unstructured creative data must generally be converted into structured features before applying formal causal inference. Generative AI can extract features from video, text, or images, and those features can enter the causal analysis.
- Heterogeneous effects can reveal where additional optimization is possible. A positive average effect may conceal substantial differences among individuals, regions, or other subgroups, prompting more granular analysis.
- Validation combines multiple checks rather than relying on a single statistical test. Potential checks include placebo treatments, random-cause perturbations, randomness checks, comparisons across estimators and consistency with previous client analyses, followed by weighing those checks according to the application.
- Uncertainty is itself an actionable result. When uncertainty is too high to support a conclusion, that result can be used to identify where additional experiments, data, or specifically designed A/B tests are required.
- The collaboration of experts with different backgrounds contributes to a project’s success.
When Search Stops Sending Traffic: The Impact of Google AI Overviews on Organic Traffic Monetization in E-Commerce
Qiwei Han – Assistant Professor, Nova School of Business and Economics, Portugal
AI Overviews (AIOVs) create a summary of web sources with source links based on the customer’s query. The implication for merchants is that consumers can complete information gathering, comparison, and screening before visiting an external site. Google’s AIOVs change what happens before a shopper ever reaches a merchant website. There is a decline of 6.8% organic impressions related to search exposure and a decline of 2.8% website sessions from referred traffic. Qiwei Han of Nova School of Business and Economics and his colleagues investigated where the missing traffic went and its commercial value.
The staggered global rollout of AIOVs created an opportunity for a natural experiment. Newly treated markets could be compared with markets that were not yet treated. The analysis combined Google Search Console, which tracks search impressions, rankings and clicks, with Google Analytics, which captures website sessions, transactions, and revenue. This allowed the researchers to follow performance from Google search exposure through downstream commercial outcomes across more than 4,600+ e-commerce domains in 130+ markets. Qiwei and colleagues used a staggered difference-in-differences design, comparing changes in countries after AIOV rollout with countries where it had not yet arrived, and examined effects over multiple post-launch weeks. The researchers also conducted alternative specifications, samples, placebo tests, and other robustness analyses. The team found that search exposure and referral traffic fell substantially more than completed transactions and revenue. Losing traffic did not necessarily mean losing customers. Transactions and revenue were statistically unchanged. AI search may send fewer visitors to the brand’s site without sending proportionally fewer customers.
Beyond overall traffic, the study examined acquisition productivity. Conversion rates increased by 2.5% and revenue per session increased by 2%, while average order value did not change significantly. These results are consistent with a possible filtering mechanism in which AI Overviews resolve some information needs before consumers visit merchant sites, leaving a smaller group of visitors who are more likely to purchase.
Key takeaways:
- AI Overviews reduce Google organic traffic, while completed transactions and total revenue remain largely unchanged. A two-three percent traffic decline does not imply a two to three percent loss of commercial demand.
- Marketers should measure acquisition productivity, not visits alone. Track conversion and revenue per session, alongside clicks and sessions.
- Search ranking matters. The study found little change for merchants appearing in the top three organic positions, while websites ranked fourth or lower experienced traffic declines after the AI Overview rollout.
- The impact of AIOVs varies by search intent. Traffic declines were concentrated in informational searches, while the researchers did not observe comparable effects across most other search-intent categories examined, including navigational, commercial investigational, and transactional categories.
- “Which customer journeys still need to reach our website?” is the question marketers should ask—not “How much traffic do we get?”
- Paid search was outside this study’s scope. Cost-per-click, advertising performance, and other paid-search measures were not evaluated because this research focused specifically on organic traffic.
Foundational Relational MMM: Zero-Shot Generalization and Row-Level Attribution
Mike Finnerty – U.S. President, Mutinex
Mike Finnerty (Mutinex) presented an R&D study that compared hierarchical Bayesian regression with relational deep learning (RDL) fitted over the marketing mix modeling (MMM) context. The study examined whether RDL could improve on the hierarchical Bayesian time-series regression that Mutinex was using as the core estimation approach in its MMM. Mutinex was already using AI for data labeling, extract, transform and load (ETL), quality assurance, taxonomy, workflow orchestration, analysis, and deliverable creation but not for the MMM estimation itself, which remained based on hierarchical Bayesian regression.
The limitations of conventional MMM include extensive data preparation before modeling. Every MMM imposes structure before it sees the data. Data from campaigns, publishers, creatives, formats, and other commercial drivers are typically normalized, aggregated and transformed through processes such as adstock, saturation curves, and feature selection. These steps impose assumptions before estimation and can remove granularity from the underlying data. Additionally, marketing data tends to be highly dimensional and sparse, with many potential relationships being estimated from relatively few weekly observations.
A champion-challenger test was conducted to compare its traditional hierarchical Bayesian approach with an RDL model using the same real-world business data and basic modeling rules. Rather than pre-aggregating or transforming the data, the RDL approach mapped raw, record-level inputs directly to sales outcomes while defining relationships among variables and requiring information to move forward through time. The researchers evaluated the models primarily on their ability to predict sales during periods excluded from model estimation, using average percentage error, whether the model correctly predicted the direction of sales changes and whether it captured the magnitude of those changes. In this test, Finnerty reported that the RDL model was more accurate compared to hierarchical Bayesian MMM baseline on average percentage error, performed better when predicting whether sales would rise or fall, and was more predictive on the amplitude of those movements.
Additionally, when the relational model was applied to products, channels, and markets not previously seen, its average percentage error was not materially different from its performance on areas included in estimation.
Mutinex also explored attribution by simulating what the model predicted with individual factors present versus the absence of those factors. The resulting contributions from paid media, pricing, calendar events, and seasonality appeared reasonable, although this attribution work was not directly benchmarked against hierarchical Bayesian results.
Mutinex is not replacing its production hierarchical Bayesian MMM at the present time, in part because the RDL model was strongest at identifying large signals while additional work is needed to capture smaller, more nuanced relationships.
Key takeaways:
- The Relational Deep Learning (RDL) model was tested as an alternative to traditional MMM.
- RDL outperformed on every metric, including the Pearson Correlation Coefficient, when compared to a Bayesian MMM baseline. For example, RDL produced better sales forecasts.
- The model can estimate the contribution of individual business drivers.
- RDL is generalizable and can drive decisions. This model was accurate on predictions for brands it had never modeled.
Closing Remarks
Scott McDonald, PhD – CEO & President, ARF