Declining response rates in surveys have inarguably led to disproportionate attention to nonresponse reduction in surveys. Yet other sources of survey error may make greater contribution to bias in survey estimates. Even if the impact of other errors are not larger than that of nonresponse, they may be less susceptible to adjustment. That is, nonignorable bias in survey estimates may be dominated by undercoverage of the target population, for example.
In landline RDD telephone surveys, it is unknown for any given study whether nonresponse or coverage error lead to greater bias in survey estimates. Surveys have limited resources and the magnitudes of these errors are needed in order to optimize the data collection design. In order to evaluate these sources of error, one needs a design in which: (1) adjustments for nonresponse and coverage error are created, and (2) data are collected from nonrespondents and those who are excluded from the sampling frame.
To obtain direct estimate of nonresponse bias in a landline RDD survey, we drew a subsample of nonrespondents and subjected them to a more effective protocol - shorter survey and higher incentive. To obtain an estimate of coverage bias, we also selected a sample of cell phone numbers and conducted interviews with those who did not have a landline telephone.
Our results (Peytchev, Carley-Baxter, and Black, 2011) show that both sources of error, nonresponse and undercoverage, can be substantial and postsurvey adjustments are not sufficient. Also after adjustments, they do not seem to cancel each other out, as one may hope without targeting each error source. Addressing at least one of the two sources on nonobservation error is found to lead to a reduction in error. We argue for the consideration of total survey error and measuring its components, to inform the survey design.
