Post developed by Catherine Allen-West and Ozan Kuru.
ICYMI (In Case You Missed It) the following work was presented at the 2016 Annual Meeting of the American Political Science Association (APSA). The presentation, titled “Motivated Reasoning in the Perceived Credibility of Public Opinion Polls,” was part of the session “Surprises: A Magical Mystery Tour of Public Opinion and Political Psychology” on Saturday, September 3, 2016.
Polls have been an integral part of American democracy, political rhetoric, and news coverage since the 1930s. Today, there are new polls reported constantly, showing public opinion on a range of issues from the President’s approval rating to the direction of the country. Polls remain relevant because numbers and statistical evidence have always been regarded as sound evidence to support one’s beliefs or affirm their affiliations; similarly, polls are supposed to provide relatively objective information in politics.
However, despite their importance and ever-increasing prevalence, polls are often heavily criticized, both by the public and politicians, especially when they fail to predict election outcomes. Such criticisms and discounting of poll credibility is important, because people’s perceptions of polls matter. In such an environment, the perceived credibility of polls becomes an important issue for the public’s reception of poll findings, which then determines the likelihood of any meaningful impact of their results.
For the public, the greatest antagonism comes not from election prediction failures (something that is for experts and pundits to figure out), but from the conflict between results of polls and people’s own positions. Political elites and the public at large both discredit poll results very frequently by claiming that the results are manipulated and/or saying that those holding the opponent view or political party have been disproportionately oversampled in a survey. This then leads to efforts such as “unskewing” the polls or severe cynicism in the public evaluations (Chia & Chang, 2015).
Current research documents how motivational biases – a discrepancy motivated by one’s personal position – in poll perceptions play out in the context of poll reports in the media and examines whether explaining the methodology of the polls can counteract the biases of individuals. We contacted 1,211 respondents from a demographically representative national sample using Qualtrics. We presented respondents with poll reports on gun control and abortion in which we manipulated the results (by either showing a majority support for or against gun control), source (a conservative vs liberal news source), and reporting style (methodological details disclosed or not). Respondents then answered a series of questions to evaluate their perceived credibility of the poll and to measure their own attitudes and political (issue) knowledge.
Our results showed that people frequently discredit polls that they disagree with. Moreover, in line with motivated reasoning theories, those who are more politically sophisticated actually discredit the polls more. That is, as political knowledge increases, the credibility drops substantially for those who disagree with the poll result. On the other hand, for those who are not in disagreement, the credibility stays about the same across the knowledge variance (see Figure 1). This finding is somewhat counter-intuitive in that it defies the idealistic notion that more knowledge should result in less biased judgments. However, it is in line with motivated reasoning theory (Lodge & Taber, 2013) and some other recent research on the impact of political knowledge and numeracy on political judgments and evaluations (Kahan, Peters, Dawson, & Slovic, 2013; Miller, Saunders, & Farhart, 2015). This asymmetric motivational bias was persistent against any source effects as well (conservative vs liberal sources presenting the poll reports) and suggests that people focus on the poll results alone, not the media outlets presenting them.
Figure 1. Disagreement by Political Knowledge Predicting Poll Credibility
Next, we did something more novel. We compared poll reports that report only the results, and those that also include the relevant methodological information about the sample, sample size, response rate, margin of error etc. This constitutes a first test of the impact of efforts in disclosing more methodological information regarding polling (see AAPOR Transparency Initiative) in terms of public evaluation of polls. Such methodological information could improve the credibility of polls. Our results showed that the presence or absence of details did not matter by itself and did not interact with political knowledge. However, we also employed a poll methodology quiz to our respondents, to evaluate their knowledge of key methodological concepts such as response rate and margin of error. Those who disagreed with the poll results seemed to discredit poll reports more when they had higher methodological knowledge. This shows that methodological disclosure served as a way to attack the polls when people were already motivated to dislike them. This again points to motivated reasoning instead of a corrective potential of methodological disclosures.
These results speak to an interesting relationship between numbers and statistics in politics. What we take as evidence, not surprisingly, depends on what we want to see and selectively attend and assign importance to. These initial results show that transparency does not guarantee less biased judgments, even for highly knowledgeable people. Our paper on this study is forthcoming in Public Opinion Quarterly. In the meantime, we are conducting further research in order to understand people’s engagement with poll reports and methodological disclosure within the context of the 2016 election polls in the U.S.
Ozan Kuru is a PhD Candidate, Communication Studies, Josh Pasek is an Assistant Professor, Communication Studies and CPS Faculty Associate and Michael Traugott is a CPS Research Professor.
Chia, S. C., & Chang, T.-K. (2015). Not My Horse: Voter Preferences, Media Sources, and Hostile Poll Reports in Election Campaigns. International Journal of Public Opinion Research, edv046.
Kahan, D. M., Peters, E., Dawson, E. C., & Slovic, P. (2013). Motivated Numeracy and Enlightened Self-Government (SSRN Scholarly Paper No. ID 2319992). Rochester, NY: Social Science Research Network.
Lodge, M., & Taber, C. S. (2013). The rationalizing voter. New York: Cambridge University Press.
Miller, J. M., Saunders, K. L., & Farhart, C. E. (2015). Conspiracy Endorsement as Motivated
Reasoning: The Moderating Roles of Political Knowledge and Trust. American Journal of Political Science.
Interesting result! One question: what if the more politically sophisticated people doubt the credibility of the polls due to their own knowledge of what the actual national sentiment is on that issue? For instance, if a highly sophisticated pro-life voter is well-aware of the state of the national polls on abortion, showing a result where 65% of the country supports abortion might cause that person to doubt the credibility of the poll. In contrast, a politically sophisticated pro-life voter might be more subject to motivated reasoning and agreeing with the 65% result in that case. A potential follow-up might examine whether or not these sophisticated voters have a better idea of where the actual non-hypothetical polling stands (it looks like this might be possible based on the data gathered).
Thanks a lot for your interesting suggestion. In this current study, unfortunately, we did not ask their knowledge about the national support levels for the issues. This could be an important future research, although in the political issue context, it might be hard to measure and incorporate such poll knowledge of respondents in an experimental set up. Although they might have a sense of where the majority’s position might be at, these issues and related polling information are more complex than an experimental set up we have here. For example, when we talk about public support for gun control, what do we mean or what do respondents have in mind? – support for restriction on semi-automatic guns, support for restriction for ammunition sales, and support for strict background checks etc. However, in a related follow-up study in the election context, we also ask respondents their own predictions regarding the support levels for the presidential candidates.