Working Paper
Fields of Gold: Web Scraping for Consumer Research
Jan 24, 2021
Outlines the key challenges, state-of-the-art remedies, best practices, and corresponding standards for evaluation for web scraping in consumer research. Provides a structured workflow designed to achieve consistency and standardization in how web scraping is conducted, documented, reported, and evaluated in both the research and peer review process. The authors propose four interdependent facets necessary for generating credible scientific findings from web scraping-based research: (1) design transparency, (2) analytic reproducibility, (3) analytic robustness, and (4) effect replicability and generalizability.