Thursday, May 2, 2013

Are we choosing the wrong survey design because of goals expressed in number of interviews?

We could be choosing a suboptimal survey design (with respect to bias in estimates, for example) because of pursuit of higher response rates. But this is another topic.

Survey objectives are often expressed as X,XXX number of interviews. A general survey design is selected that can achieve that number within given budget constraints, balancing cost and credibility in some fashion. For example, a Random-Digit-Dial telephone survey using mostly landline numbers (lower cost) and some, although relatively few, cell phone numbers. This is not unlike a decade or two ago when an analogous split was made between the listed and list-assisted RDD sampling frames. So the design above would produce unbiased estimates to the extent that adults with only cell phones are represented.

What is not obvious is that this design, while cost-driven, may be very cost-inefficient.

With relatively few and still declining number of people with only landlines and near complete coverage of the population through cell phones, the design effect in the above design can be tremendous. The effective sample size may be much lower than if the overall sample size is reduced and a larger proportion of the interviews conducted from the cell phone frame. Possibly at a lower cost.

It seems there is sometimes a disjoint between sampling theory and practice. The causes are certainly understandable - it is easier to set requirements in number of interviews than in effective sample size. But ultimately, it is the effective sample size that is of interest.