Measuring the Unmeasured: Combining Technology and Survey Design to Filter Noise in Self-Reported Business Outcomes

In low-income countries, the majority of the poor either run or work in a micro-enterprise. Understanding the dynamics of these enterprises, and whether policies aimed at improving firm productivity and the incomes of the self-employed are effective, relies on the availability of accurate measurements of micro-enterprise sales and profits. However, these data are plagued by high levels of noise (measurement error), making it difficult to detect whether the millions of dollars being invested into macro and micro level policy programs is having the intended effect (or any at all).

To address these measurement challenges, this research team has designed a new electronic survey tool, combining an electronic approach with a novel survey design, which reduces noise by triangulating on firm sales, aggregating within and across firm cost categories, and adjusting estimates iteratively to narrow in on a more accurate and less noisy estimate of firm profits. During initial fieldwork, this tool has been shown to decrease the coefficient of variation (standard deviation over sample mean) for sales, cost and profit estimates, as well as increase the autocorrelation of firm outcomes over multiple survey rounds. 

While these initial results are encouraging, the best way to evaluate the effectiveness of this new electronic surveying approach is to conduct a randomized controlled trial (RCT) with an independent sample of micro-entrepreneurs who are not already participating in a research project. The researchers will carry out this experiment on a sample of 600 micro-entrepreneurs (50% female) in greater Accra, Ghana. They will randomly assign half of the entrepreneurs into a treatment group (who will be administered the new electronic survey tool) and half into a control group (who will be administered a comparable paper survey tool). The team’s analysis will evaluate whether the new electronic surveying approach works better and, if so, then how it actually helps to improve the precision of estimates. Outcomes to examine include the coefficient of variation for each estimate, as well as the adjustment factor within each estimate - i.e. the extent to which an estimate changes from its pre-adjusted initial value (from unaided recall) to its post-adjusted final value, obtained after a surveyor took the respondent through multiple adjustment steps to iteratively improve the accuracy of their response. 

Authors

Stephen Anderson

Stanford University

Bilal Zia

World Bank