Post developed by Nicholas Valentino and Katherine Pearson
Survey research is an ever-evolving field. Technology has increased the number of ways to reach respondents, while simultaneously reducing response rates by freeing people from the constraints of one land-line telephone per household. Surveys remain an essential tool for making inferences about societal and political trends, so many survey researchers offer incentives to survey respondents in order to ensure a large and representative sample. Financial incentives to complete surveys, in turn, entice some people to respond to a large number of online surveys on a regular basis, essentially becoming professional survey respondents.
Survey methodologists have carefully considered the ways that survey modes may impact the way people answer questions. Talking to a real person is different than answering questions online. But less is known about how individual factors bias participation in surveys in the first place. For example, might personality traits shape your willingness to agree to answer a survey online versus someone who comes to your door? New work from researchers at the University of Michigan and Duke suggests in fact this is the case.
In order to examine the personality traits of survey respondents, the research team used data from the 2012 and 2016 American National Election Studies (ANES). During these two study periods, the ANES ran parallel and face-to-face surveys. In both years, the ANES included the 10-item personality inventory (TIPI), which consists of pairs of items asking respondents to assess their own traits. Based on the responses, respondents build a profile of “the Big Five” personality traits: openness to experience, conscientiousness, extraversion, agreeableness, and emotional stability.
Big Five traits with corresponding TIPI qualities
Trait
TIPI Qualities
Coding
Openness to experience
Open to new experiences, complex
Conventional, uncreative
Original
Reversed
Conscientiousness
Dependable, self-disciplined
Disorganized, careless
Original
Reversed
Extraversion
Extraverted, enthusiastic
Reserved, quiet
Original
Reversed
Agreeableness
Critical, quarrelsome
Sympathetic, warm
Reversed
Original
Emotional Stability
Anxious, easily upset
Calm, emotionally stable
Reversed
Original
Researchers were able to compare responses to the TIPI with measures of political predispositions and policy preferences, based on responses to questions on the ANES. These include partisanship, liberal–conservative ideology, issue self-placements, and other measures of political orientation.
Based on these data, the authors found that respondents in the online samples were, on average, less open to experience and more politically conservative on a variety of issues compared to those responding to face-to-face surveys. They also found that the more surveys a respondent completed, the lower they scored on measures of openness. Given that professionalized survey respondents comprise the majority of online survey samples, these results suggest caution for those who would like to generalize results to the population at large. It is not enough to balance samples on simple demographics. Attitudinal and personality based differences might also lead online sample estimates to diverge from the truth.
It is difficult to say whether online survey respondents or face-to-face respondents are more representative of personality traits in the general population. If personality is a factor in whether someone will participate in a survey, that might bias both types of samples. However, the authors note that the data suggest that professional online samples are the outlier. They find “that samples based on fresh cross-sections, both face-to-face and online, yield better population estimates for personality and political attitudes compared to professionalized panels.” While it may be possible to mitigate the potential sampling bias of personality traits, it is important that survey researchers understand the role that personality traits play in professional online samples.
In every U.S. presidential election since 1948, the American National Election Studies (ANES) has conducted pre- and post-election surveys of a large representative sample of American voters. ANES participant interviews looked different in 2020 than they did in the past; the COVID19 pandemic made traditional face-to-face interviews impractical and risky. The study team began planning for the extraordinary circumstances in March, without any idea what the conditions would be when interviews began in August. The team pivoted nimbly to redesign the study even as the onset of data collection approached.
The majority of interviews in 2020 were completed as web surveys, some following an online format similar to one used in 2016, and others using an innovative mixed-mode design. Respondents to the mixed-mode surveys were randomly assigned either to complete the questionnaire by themselves online, or to take the survey with a live interviewer via a Zoom video link. Few surveys conduct live video interviews, but the ANES study team felt that it was critical to explore the use of this technology as a potential means of balancing issues of cost, continuity, and data quality.
To answer online surveys, respondents must have reliable access to the Internet and comfort using computers. Under normal circumstances, people without access to computers or the Internet in their homes can gain access in public settings like libraries or at their workplace. With many of these places closed due to the pandemic, online access became a bigger challenge. In mixed-mode cases where it was difficult to complete a web or video interview, interviewers contacted the respondents to secure a phone interview. Providing phone interviews helped the team strengthen sample quality by reaching respondents without access to the Internet as well as those who are less comfortable using computers.
Data collection for the 2020 surveys, out of necessity, departed significantly from the practices of the past 70 years of the ANES. The study team will continue to monitor and address the implications of these changes. In the end, the team was pleased to field a very high quality survey with relatively high response rates, thoroughly vetted questions, and the largest sample in the history of ANES.
Pre-election surveys
Pre-election interviews began in August 2020. The pre-election questionnaire is available on the ANES website. The questionnaire includes time series questions dating back to the earliest days of the ANES survey, as well as new questions that reflect more recent developments in the study of American politics. The ANES team must always be prepared to add a few questions late in the design process to capture substantial developments in the presidential campaign or American society. In 2020 the survey added questions about election integrity, urban unrest, and COVID-19, among other topics.
The investigators, ANES staff, and their survey operations partners at Westat monitored the data collection closely, in case further adjustments in procedures or sample were required. The final pre-election sample consists of over 8,200 complete or sufficient-partial interviews. This includes a reinterview panel with the respondents from the ANES 2016 Time Series. Over 2,800 respondents from the 2016 study were reinterviewed, more than three quarters of the original group.
Post-election surveys
Post-election interviews began on November 8, 2020, and will be completed on January 4, 2021. This post-election effort includes additional respondents who took part in the 2000 study of the General Social Survey (GSS). Due to the pandemic-altered timing of the GSS data collection, it was not possible to interview these individuals prior to the election. However, these respondents completed nearly all of the ANES post-election interview, plus almost ten minutes of critical questions that appeared on the ANES pre-election interview, and several additional questions suggested by the GSS team.
ANES staff will continue to review and clean the data into the new year, including checks of respondent eligibility that may alter the final sample in modest ways. Pending this review, the team expects response rates to come in slightly below the 2016 web response rates.
Overall, despite the challenges of this past year, the ANES study team was able to gather robust data from a large probability sample of Americans, extending the longest-running, most in-depth, and highest quality survey of US public opinion and voting behavior, at a critical juncture for American society and democracy. The team will continue to share updates, here and on the ANES website, as data from this survey become available.
Stuart Soroka began his presentation by showing additional data from The Breakthrough, which assigns a sentiment to each response about the candidates, in order to track trends in the sentiment over time. This is not the sentiment of news content itself, but rather it’s the sentiment of the content that respondents remember. Soroka noted that the data show stability over time with a slightly positive average for Biden and a slightly negative average for Trump.
Sentiment of Recalled News about the Candidates
Amongst Republicans, Trump gets more positive recollections than Biden. Over the past month or so recollections of Trump have been a little more positive, albeit with a slight and statistically significant decline over the last month. Soroka said “That’s an interesting result because it runs contrary to what media observers might suggest. And contrary to what the Trump campaign’s own claims about negative coverage of Trump and mainstream media.”
Sentiment of Recalled News about the Candidates
The Impact of COVID-19 on the Sentiment of Recalled News about Trump
Next, Soroka focused on the impact of COVID-realted content on recollections of news related to Trump for Republicans, Democrats, and independents. The data show that as COVID-related content is associated with negative sentiment among Democrats, it’s associated with increasingly positive sentiment among Republicans. That trend is not just the product of Trump’s quicker recovery in the week of October 11th. It’s evident in weeks prior to that as well. “In short,” Soroka said, “partisans see the same information in fundamentally different ways.”
The Impact of COVID-19 on the Sentiment of Recalled News about Trump
Switching to analyses of news content, Soroka showed data on the ratio of Republican candidate mentions to Democratic candidate mentions in the first week of September across 11 major newspapers for every election back to 1980. Incumbents get more coverage than challengers, and this has been equally true for both parties. We see this for Reagan, we see it for Clinton, we see it for Obama. That imbalance tends to be roughly a 20-percentage point gap. If there are ten candidate mentions, six of those are for incumbent, and four for the challenger. There are two outliers, first in 2016 Trump got incumbent style coverage in terms of imbalance although he was not the incumbent. After the election there was lots of discussion in the Academy, but publicly amongst journalists and the media as well, especially expressing concern that the media had focused too much attention on Trump.
Balance of Coverage, 1980-2020
Now that Trump is incumbent, Soroka said that Trump is getting outsized proportions of coverage. This year, for every ten-candidate mentions, seven to eight of them are about Trump, and two to three of them are about Biden. Are there differences across newspapers? Soroka said these differences are relatively slight, and have barely changed over time. He also noted that public opinion data show that partisans see information in different ways; researchers cannot reasonably assume that coverage of Trump is viewed as negative or positive on average.
Overwhelming coverage of Trump makes it very difficult for citizens to learn about Biden. In other words, Soroka pointed out, “the nature of media coverage that I have just reviewed makes it possible for voters to decide on whether or not to vote for Trump, but it gives very little information that helps voters decide whether or not to vote for Biden.” The imbalance of coverage is evident across media outlets throughout this campaign, and at a rate that is roughly twice as large as we’ve seen for the last 40 years.
Soroka concluded by highlighting the areas in which the media can play a critical role in fostering the well-informed citizenry. He directed the audience to https://mediafordemocracy.org to find a set of recommendations on how to improve coverage of this election, as well as various possible outcomes of this election. Soroka is one of the signatories of an open-letter, signed co-authored and signed by more than 60 professors, at more than 40 universities.They hope this will be a useful resource for people interested in the role that media can play in American democracy both leading up to and following the election.
Josh Pasek said there are several ways that a scholar would look at the 2020 presidential election. One is looking at the structural features that set up the landscape in which the current election is taking place. Another is to look at the messaging that people are getting in the context of this campaign. Yet another is to think more broadly about the American political system and what that might imply. Pasek examined each of these perspectives in turn.
Structural explanations
Pasek noted that there has never before been a U.S. president with a negative net favorability rating for his entire term, but this is true of Trump. A majority of individuals who have ever been asked whether they approve or disapprove of the president’s job have said that they do not approve. This is very unusual, said Pasek. Usually approval is strongly correlated with reelection chances, and an approval rate of 44 percent would not be associated with reelection. But it’s also an unusual way to target one-self as a sitting president, who typically tries to represent all the American people and to gain favorability among all of them.
The economic situation which is another of the major indicators to look at. The COVID-19 pandemic hurt the U.S. economy, and the economic growth in the last two quarters has been strongly negative. Pasek pointed out that strongly negative numbers tend again to be something that point against reelection of an incumbent president.
Messaging explanations
Pasek emphasized the incredible stability of polling in this election cycle. There has also been relatively stable attention to issues across the campaign. Pasek, Traugott, and Soroka contribute to a project called The Breakthrough, which asks people what they’ve seen, read, or heard about Biden and Trump over the past week. The survey has found that people are hearing essentially the same things about the candidates throughout the campaign.
This moment in American Politics
Pasek suggested there is another story that needs attention, which is this moment in American politics. He said we are in a moment “where there is increasing animosity in American politics.” Tying this sentiment to an increasing erosion of key Democratic norms, Pasek said that Americans are less likely to reject political violence than they were in 2017.
There are also issues that are more concerning, Pasek said, such as questions about whether the president is willing to accept the election results or engage in a peaceful transition of power if he does lose the election. He points to evidence that groups have been recruiting people to show up at the polls armed, which could be sort of a widespread attempt at voter suppression.
“Maybe that all points to a situation where things are a little bit more abnormal, and we really do need to worry about whether the models are right, not just because the models sometimes have error, but because they’re actually is enough distrust in the system that people may work against it,” said Pasek. If we were looking at a normal election year, Pasek said he expect this to be a change election. People tend to be not particularly happy when they don’t like the president overall, and whether the economy is poor, and polling fits in line with that pretty clearly. But given some of these unique contextual features of the campaign, he concluded that it is hard to know how this election cycle will end.
Michael Traugott began by noting that this is a very interesting and unusual election. When Donald Trump ran for President in 2016, he had no prior experience in governing, and he ran explicitly as an outsider. “He is typically focused on his base,” said Traugott. “Unlike any other first-term president, who takes office and thinks about how to enlarge his coalitions.” In the 2020 presidential election, Trump is now an incumbent with a record to defend. His campaign strategy before the COVID-19 pandemic was to run on a strong economy, and to turn out his base. However, the onset of the COVID-19 pandemic has altered the course of the campaign dramatically.
Division of partisanship in the constituency
Traugott emphasized that it is important to recognize that there are more Democrats in the adult population in the U.S. than there are Republicans. Data from the Pew Research Center show that there are about as many independents as Democrats, both of these groups outnumber the Republicans.
Furthermore, there are substantial racial and ethnic differences in party support. Among whites, the Republicans are the favored party. Among Blacks, Democrats outnumber Republicans by eight to one. Among Hispanics, Democrats outnumber Republicans by a little more than two to one, and among Asian Americans, about five to one. The demographics of the U.S. are shifting, and whites are expected to be a minority in the population by the mid-2040s. Traugott points out that there is a significant question about what it means to be a Republican or a Democrat in today’s society and how that is likely to change across the next 20-25 years.
Candidate Trait Assessments
Traugott presented data on assessments of the candidates’ traits, collected in a recent survey by the Pew Research Center. Respondents to this survey were more likely to see Joe Biden as compassionate than Donald Trump, with a 33 percentage point advantage. Biden was also seen as a good role model, with a 26 percentage point advantage, and honest, an 18 percentage point advantage. Trump stands out as energetic to respondents, with a 16 percentage point advantage.
Policy Differences Between the Candidates
Traugott presented the results of a survey that asked respondents about their confidence that either Biden or Trump would make good decisions in various policy areas. Respondents rated the candidates evenly on making good decisions about economic policy. Across a series of social issues, including law enforcement and criminal justice, nominees for the Supreme Court, and foreign policy, Biden has a growing advantage.
Polling Data and Voter Turnout
Taking a national view of the polls, Biden currently has a 10.6 percentage point lead over Trump according to FiveThirtyEight. However, Traugott points out that in two of the last five U.S. presidential elections, the candidate receiving the most votes did not win the election because of the electoral college. Traugott instead encouraged the audience to pay attention to who is leading in the battleground states. He noted that Biden has an advantage in seven of these states, whereas Trump has an advantage in Texas, and Georgia is currently too close to call.
Among people who say that they’re going to vote in-person on election day, Trump is ahead by a margin of two to one. However, Biden has an advantage among those who are going to vote by mail and those who will vote in-person before election day. Traugott noted that as of October 14, there were about 10 million votes already cast in this election.
The Democrats are emphasizing turnout voting in large numbers, whereas the Trump administration has been sowing doubt about the integrity of the electoral system. Traugott presented data collected by YouGov for Yahoo News, which show the partisan differences in attitudes about whether the election will be rigged or not. Republicans are much more likely to believe that there will be problems with vote counting and irregularities. Democrats are more likely to believe election security experts who say the U.S. presidential election cannot be rigged. Traugott concluded by saying “I think the most disheartening part of this is when the respondents were asked do you think this year’s presidential election will be free and fair, only about a quarter of the respondents said yes.”
This post looks at the opinions of Twitter users surrounding the first Presidential Debate. We look at content containing at least one debate hashtag, shared immediately before, during, and after the debate; and we determine the “stance” or opinion (for or against) of each tweet towards Biden and Trump.
The figure below shows the average proportion of expressed support or opposition for the candidate every minute of the debate from 8pm (20:00) to 11:30pm (23:30). A score above zero indicates a net positive stance towards the candidate. A score below zero indicates a net negative stance.
Presidential Debate 1: Stance of Candidates on Twitter
We see that in the hour before the debate begins, both candidates have a net negative stance. In other words, more opinions against each candidate are being shared than are opinions for each candidate. At around the 11 minute mark in the debate (roughly 21:11), pro-Biden expressions begin increasing, and continues to increase until the overall stance is in support of Biden. In contrast, around the same time, stance towards Trump decreases and continues to decrease for the first 10 minutes.
Over the course of the debate there are specific moments that help and hurt each of the candidates. When there is perceived bickering, there is usually a decline in stance for both candidates, although there are exceptions. The moment in which Trump received the most support was when he spoke about judges. Biden’s best moment was when he discussed race relations and the need to support black Americans.
By the end of the debate, the stance of Twitter discussion towards Biden had increased by 0.5 – a striking shift. He clearly benefited from the debate, at least in the short term amongst Twitter users. In contrast, the stance of Twitter discussion towards Trump decreased by approximately 0.2. Even as there was a good deal of opposition towards Trumps expressed immediately before the debate, there was even more negativity towards him at the end of the debate.
It is worth noting that within an hour of the debate the expressed stance towards Trump returned to pre-debate levels. These are decidedly negative, of course; but the additional negative impact of the debate on Twitter discussion of Trump may have been short-lived. The same is not true for Biden. The hours surrounding the debate saw a marked shift in expressed stance towards Biden, from by-minute averages that were anti-Biden to clearly pro-Biden. The shift is evident only 10 minutes into the 90-minute debate, and durable for the hour following the debate as well.
Twitter is by no means an accurate representation of public opinion more broadly – we must be sure to interpret these results as indicating the debate impact on Twitter discussion, not the public writ large. That said, where Twitter is concerned it seems relatively clear that Biden ‘won’ the debate.
Information about the analysis:
This analysis was conducted using approximately 1.3 million tweets that contained at least of the debate hashtags. We collect posts using the Twitter Streaming API. We use the core debate hashtags for this analysis, e.g. #debates2020, #presidentialdebate2020, etc. We determine if the tweet showed support, opposition, or neither for each candidate. For each minute, we compute an aggregate stance score as follows: Stance Score = (# Support – # Oppose) / (# of tweets that minute having a stance). To determine the stance itself, we trained a BERT fined tune model with a single layer on 5 million posts related to election 2020. We also had three people label 1000 tweets with stance to further improve our model.
This analysis was conducted by the Political Communications Election 2020 project of the Social Science and Social Media Collaborative (S3MC). The faculty involved in that project include Ceren Budak (University of Michigan), Jonathan Ladd (Georgetown University), Josh Pasek (University of Michigan), Lisa Singh (Georgetown University), Stuart Soroka (University of Michigan), and Michael Traugott (University of Michigan). The work is funded in part by National Science Foundation awards #1934925 and #1934494 and the Massive Data Institute. This project is a collaborative effort by the University of Michigan and Georgetown University to address how to harness the abundance of data from social media in order to understand social and political trends better. For the latest updates about this group’s research related to the 2020 Election, visit the project website: https://s3mc.org/political-communication/election-2020-project/. For information about the interdisciplinary methodology being developed by this group, visit: http://smrconverge.org/home/methodology/
Post developed by 
Pasek noted that there has never before been a U.S. president with a negative net favorability rating for his entire term, but this is true of Trump. A majority of individuals who have ever been asked whether they approve or disapprove of the president’s job have said that they do not approve. This is very unusual, said Pasek. Usually approval is strongly correlated with reelection chances, and an approval rate of 44 percent would not be associated with reelection. But it’s also an unusual way to target one-self as a sitting president, who typically tries to represent all the American people and to gain favorability among all of them.
Division of partisanship in the constituency
Policy Differences Between the Candidates
