I just listened to a pollster describe polls relying increasingly more on art than science. Decisions on how to identify likely voters, how to weigh the data, etc. And it made me think: could we not quantify the Art part so that it becomes Science?
Probability-based surveys and polls rely on sampling theory, and ultimately, on a sampling distribution. This is certainly the case when frequentist-based inferential methods are used. That is, if the survey is repeated again and again, there will be a sampling distribution of the survey estimate of interest. Based on this distribution, we expect that approximately 95% of the replications of the survey will fall within two standard deviations of its mean.
I can think of two other sampling distributions that are ignored. The different estimates that this pollster would have if he made other choices (e.g., higher turnout among minority groups) can also be seen as having a sampling distribution, although it is unlikely to be Normal. But if one goes through the exercise to build one, essentially this will quantify the Art part of his job. This in turn can reflect uncertainty in the estimates that result from his more subjective decisions. Ideally, this would include even aspects of the design such as number of days of data collection, but many of these aspects will be observed in only one way. To some degree these unobserved results will be included in the next sampling distribution.
A third sampling distribution is across data collection organizations. Gallup could get very different estimates from Pew, and both can be different from his estimates. Many of the differences are nearly impossible to attribute to a
particular design feature, but can be observed as a "package" of
differences in design that we can call the survey or poll's protocol. This "house" effect creates another sampling distribution, that of estimates from different survey organizations. This is what people like Nate Silver have capitalized upon--building a single estimate from the distribution of estimates from polls from different organizations. That, as Frank Newport has pointed out, does not mean that we have less value for individual polls. Instead, it means that there is value in incorporating and using this third distribution.
Personally, I think there is much to be done studying the second sampling distribution above, particularly in polls where many arbitrary and sometimes subjective decisions need to be made.
