Where A-B Testing Goes Wrong: How Divergent Delivery Affects What Online Experiments Cannot (and Can) Tell You About How Customers Respond to Advertising – MSI

Working Paper

Where A-B Testing Goes Wrong: How Divergent Delivery Affects What Online Experiments Cannot (and Can) Tell You About How Customers Respond to Advertising

Michael Braun

Southern Methodist University

Eric Schwartz

University of Michigan

Jun 27, 2024

Develops a conceptual model and offers prescriptive guidance to experimenters on how to manage the fact that online platforms’ A-B testing tools do not randomize ad exposure in ways that allow causal inferences about user response.

Topics

research recap

How Online Platforms’ A-B Test Tools Confound Causal Inference

A-B testing tools on online ad platforms do not randomize ad exposure in ways that allow causal inferences about user response because targeting algorithms serve each ad to different mixes of users optimized for each ad. This paper offers prescriptive guidance to experimenters based on their decision-making goals for A-B ad tests.

By using MSI.org you agree to our use of cookies as identifiers and for other features of the site as described in our Privacy Policy.