Post developed by Catherine Allen-West and Ozan Kuru.
ICYMI (In Case You Missed It) the following work was presented at the 2016 Annual Meeting of the American Political Science Association (APSA). The presentation, titled “Motivated Reasoning in the Perceived Credibility of Public Opinion Polls,” was part of the session “Surprises: A Magical Mystery Tour of Public Opinion and Political Psychology” on Saturday, September 3, 2016.
Polls have been an integral part of American democracy, political rhetoric, and news coverage since the 1930s. Today, there are new polls reported constantly, showing public opinion on a range of issues from the President’s approval rating to the direction of the country. Polls remain relevant because numbers and statistical evidence have always been regarded as sound evidence to support one’s beliefs or affirm their affiliations; similarly, polls are supposed to provide relatively objective information in politics.
However, despite their importance and ever-increasing prevalence, polls are often heavily criticized, both by the public and politicians, especially when they fail to predict election outcomes. Such criticisms and discounting of poll credibility is important, because people’s perceptions of polls matter. In such an environment, the perceived credibility of polls becomes an important issue for the public’s reception of poll findings, which then determines the likelihood of any meaningful impact of their results.
For the public, the greatest antagonism comes not from election prediction failures (something that is for experts and pundits to figure out), but from the conflict between results of polls and people’s own positions. Political elites and the public at large both discredit poll results very frequently by claiming that the results are manipulated and/or saying that those holding the opponent view or political party have been disproportionately oversampled in a survey. This then leads to efforts such as “unskewing” the polls or severe cynicism in the public evaluations (Chia & Chang, 2015).
Current research documents how motivational biases – a discrepancy motivated by one’s personal position – in poll perceptions play out in the context of poll reports in the media and examines whether explaining the methodology of the polls can counteract the biases of individuals. We contacted 1,211 respondents from a demographically representative national sample using Qualtrics. We presented respondents with poll reports on gun control and abortion in which we manipulated the results (by either showing a majority support for or against gun control), source (a conservative vs liberal news source), and reporting style (methodological details disclosed or not). Respondents then answered a series of questions to evaluate their perceived credibility of the poll and to measure their own attitudes and political (issue) knowledge.
Our results showed that people frequently discredit polls that they disagree with. Moreover, in line with motivated reasoning theories, those who are more politically sophisticated actually discredit the polls more. That is, as political knowledge increases, the credibility drops substantially for those who disagree with the poll result. On the other hand, for those who are not in disagreement, the credibility stays about the same across the knowledge variance (see Figure 1). This finding is somewhat counter-intuitive in that it defies the idealistic notion that more knowledge should result in less biased judgments. However, it is in line with motivated reasoning theory (Lodge & Taber, 2013) and some other recent research on the impact of political knowledge and numeracy on political judgments and evaluations (Kahan, Peters, Dawson, & Slovic, 2013; Miller, Saunders, & Farhart, 2015). This asymmetric motivational bias was persistent against any source effects as well (conservative vs liberal sources presenting the poll reports) and suggests that people focus on the poll results alone, not the media outlets presenting them.
Figure 1. Disagreement by Political Knowledge Predicting Poll Credibility
Next, we did something more novel. We compared poll reports that report only the results, and those that also include the relevant methodological information about the sample, sample size, response rate, margin of error etc. This constitutes a first test of the impact of efforts in disclosing more methodological information regarding polling (see AAPOR Transparency Initiative) in terms of public evaluation of polls. Such methodological information could improve the credibility of polls. Our results showed that the presence or absence of details did not matter by itself and did not interact with political knowledge. However, we also employed a poll methodology quiz to our respondents, to evaluate their knowledge of key methodological concepts such as response rate and margin of error. Those who disagreed with the poll results seemed to discredit poll reports more when they had higher methodological knowledge. This shows that methodological disclosure served as a way to attack the polls when people were already motivated to dislike them. This again points to motivated reasoning instead of a corrective potential of methodological disclosures.
These results speak to an interesting relationship between numbers and statistics in politics. What we take as evidence, not surprisingly, depends on what we want to see and selectively attend and assign importance to. These initial results show that transparency does not guarantee less biased judgments, even for highly knowledgeable people. Our paper on this study is forthcoming in Public Opinion Quarterly. In the meantime, we are conducting further research in order to understand people’s engagement with poll reports and methodological disclosure within the context of the 2016 election polls in the U.S.
Chia, S. C., & Chang, T.-K. (2015). Not My Horse: Voter Preferences, Media Sources, and Hostile Poll Reports in Election Campaigns. International Journal of Public Opinion Research, edv046.
Kahan, D. M., Peters, E., Dawson, E. C., & Slovic, P. (2013). Motivated Numeracy and Enlightened Self-Government (SSRN Scholarly Paper No. ID 2319992). Rochester, NY: Social Science Research Network.
Lodge, M., & Taber, C. S. (2013). The rationalizing voter. New York: Cambridge University Press.
Miller, J. M., Saunders, K. L., & Farhart, C. E. (2015). Conspiracy Endorsement as Motivated
Reasoning: The Moderating Roles of Political Knowledge and Trust. American Journal of Political Science.