“Not which ones, but how many” is a phrase used in list experiments instruction, where researchers instruct participants, “After I read all four (five) statements, just tell me how many of them upset you. I don’t want to know which ones, just how many.” In retrospect, I was surprised to see that this phrase encapsulates not only the key research idea, but also my fieldwork adventure: not which plans could go awry, but how many. The fieldwork experience could be frustrating at times, but it has led me to uncharted terrain and brought insights into the research contexts. The valuable exposure would not have been possible without support from the Roy Pierce Award and guidance from Professor Yuki Shiraito.
Research that I conducted with Yuki Shiraito explores the effect of behavior on political attitudes in authoritarian contexts to answer the question: does voting for autocracy reinforce individual regime support? To answer this question, two conditions need to be true. First, people need to honestly report their level of support before- and after- voting in authoritarian elections. Second, voting behavior needs to be random. Neither situation is probable in illiberal autocracies. Our project addresses these methodological challenges by conducting a field experiment that combines a list experiment and a randomized encouragement design in China.
In this study, list experiments are used instead of direct questions to measure the respondents’ attitudes towards the regime in the pre- and post-election surveys. The list experiment is a survey technique to mitigate preference falsification by respondents. Although the true preference of individual respondents will be hidden, the technique allows us to identify the average level of support for the regime within a group of respondents. In addition, we employ a randomized encouragement design where get-out-the-vote messages are randomly assigned, which help us estimate the average causal effect of a treatment. For effect moderated by prior support for the regime, we estimate the probability of the prior support using individual characteristics and then estimate the effect for the prior supporters via a latent variable model.
While the theoretical part of the project went smoothly and the simulation results were promising, the complication of fieldwork exceeded my expectation. For the list experiment survey, the usually reticent respondents started asking questions about the list questions immediately after the questionnaires were distributed. Their queries took the form of “I am upset by option 1, 2, and 4, so what number should I write down here?” This was not supposed to happen. List experiments are developed to conceal individual respondents’ answers from researchers. By replacing the questions of “which ones” with the question of “how many,” respondents’ true preference is not directly observable, which makes it easier for them to answer sensitive questions honestly. Respondents’ eagerness to tell me their options directly defeats the purpose of this design. Later I learned from other researchers that the problem I encountered was common in list experiment implementation regardless of research contexts and types of respondents.
The rationale behind respondents’ desire to share their individual options despite being given a chance to hide them is thought-provoking. Is it because of the cognitive burden of answering a list question, which is not a familiar type of questions to respondents? Or is it because the sensitive items, despite careful construction, raise the alarm? Respondents are eager to specify their stance on each option and identify themselves as regime supporters: they do not leave any room for misinterpretation. To ease the potential cognitive burden, we will try a new way to implement the list experiment in a similar project on preference falsification in Japan. We are looking forward to seeing if it improves respondents’ comprehension of the list question setup. The second explanation is more concerning, however. It suggests the scope condition of list experiments as a valid tool to elicit truthful answers from respondents. Other more implicit tools, such as endorsement experiments, may be appropriate in those contexts to gauge respondent’s preference.
Besides the intricacies of the list experiment, carrying out encouragement design on the ground is challenging. We had to modify the behavioral intervention to adapt needs from our local collaborators, and the realized sample size was only a fraction of the negotiated size initially. Despite the compromises, the implementation is imbued with uncertainty: meetings were postponed or rescheduled last minutes, instructions from local partners are sometimes inconsistent and conflictual. The frustration was certainly real. But the pain makes me cognizant of judgment calls researchers have to make in the backstage. The amount of effort required to produce reliable data is admirable. And as a consumer of data, I should always interpret data with great caution.
While the pilot study does not lead to a significant finding directly, the research experience and the methods we developed have informed the design of a larger project that we are currently doing in Japan.
I always thought of doing research as establishing a series of logical steps between a question and an answer. Before I departed for the pilot study, I made a detailed timeline for the project with color-coded tasks, flourish-shaped arrows pointing at milestones of the upcoming fieldwork. When I presented this plan to Professor Shiraito, he smiled and told me that “when doing research, it is generally helpful to think of the world in two ways: the ideal world and the real world. You should be prepared for both.” Wise words. Because of this, I am grateful for the Roy Pierce Award for offering the opportunity to catch a glimpse of the real world. And I am indebted to Professor Shiraito for helping me see the potential of attaining the ideal world with intelligence and appropriate tools.
Since its establishment in 2013, a total of 137 posts have appeared on the Center for Political Studies (CPS) Blog. As we approach the new year, we look back at 2017’s most-viewed posts. Listed below are the posts that you, our dear readers, found most interesting on the blog this year.
There are political issues and then there are moral political issues. Often cited examples of the latter include abortion and same sex marriage. But what makes a political issue moral?An extensive literature already asserts a moral vs. not moral issue distinction. Yet, there is no consensus in how to distinguish between moral and non-moral political issues. Further, trying to sort issues into these categories proves challenging.
By closely investigating all known cases of state-backed cross-sector surveillance collaborations, our findings demonstrate that the deployment of mass surveillance systems by states has been globally increasing throughout the last twenty years. More importantly, from 2006-2010 to present, states have uniformly doubled their surveillance investments compared with the previous decade.
In 2012, Barack Obama received 93% of the African American vote but just 39% of the White vote. This 55% disparity is bigger than vote gaps by education level (4%), gender (10%), age (16%), income (16%), and religion (28%). And this wasn’t about just the 2012 or 2008 elections, notable for the first appearance of a major ticket African American candidate, Barack Obama. Democratic candidates typically receive 85-95% of the Black vote in the United States. Why the near unanimity among Black voters?
Both parties moving toward ideological poles has resulted in policy gridlock (see: government shutdown, debt ceiling negotiations). But does this polarization extend to the public in general? To answer this question, Iyengar measured individual resentment with both explicit and implicit measures.
The enthusiasm for both Trump and Sanders’ messages about the influence of money in politics brings up an important question: Is policy driven by the rich, or does government respond to all? Political scientists have long been interested in identifying to what degree wealth drives policy, but not all agree on it’s impact.
Exploring the Tone of the 2016 Election by U-M undergraduate students Megan Bayagich, Laura Cohen, Lauren Farfel, Andrew Krowitz, Emily Kuchman, Sarah Lindenberg, Natalie Sochacki, and Hannah Suh, and their professor Stuart Soroka (2017)
Political economists often theorize about relationships between politics and macroeconomics in the developing world; specifically, which political or social structures promote economic growth, or wealth, or economic openness, and conversely, how those economic outcomes affect politics. Answering these questions often requires some reference to macroeconomic statistics. However, recent work has questioned these data’s accuracy and objectivity. An under-explored aspect of these data’s limitations is their instability over time.
In a recent piece in the Washington Post, we addressed some common misconceptions about what the Swedish crime data can and cannot tell us. However, questions about the data persist. These questions are varied but are related to two core issues: (1) what kind of data policy makers need to inform their decisions and (2) what claims can be supported by the existing data.
Ryan’s overarching hypothesis boils non-compromise down to morals: a moral mindset orients citizens to oppose political compromises and punish compromising politicians. There are all kinds of issues for which some citizens seem resistant to compromises: tax reform, same-sex marriage, collective bargaining, etc. But who is resistant? Ryan shows that part of the answer has to do with who sees these issues through a moral lens.
Since 1948, the ANES- a collaborative project between the University of Michigan and Stanford University- has conducted benchmark election surveys on voting, public opinion, and political participation. This year’s polarizing election warranted especially interesting responses.
The effectiveness of America’s system of democratic representation, in practice, turns on broad participation. Yet only about 60 percent of voting eligible Americans cast their vote in presidential elections. This number is nearly cut in half in off-year elections (about 36 percent), and participation in local elections is even lower. This lack of electoral engagement does not fall equally across racial and ethnic subgroups. Latinos, for one, are particularly underrepresented at polling booths across the country. In 2016, eligible Latino voters were about 20 percentage points less likely to vote than their White counterparts, and about 13 percentage points less likely to vote than their Black counterparts.
This fall, a group of 24 University of Michigan undergraduate students sought to explore this disparity and pinpoint what, if anything, works to increase Latino political participation. In the class, entitled The Politics of Latinidad, CPS Faculty Associate and U-M Political Science ProfessorMara Ostfeld taught her students how to measure public opinion and challenged them to analyze the factors that affect Latino political participation.
Today, more than 50,000 Latinos live in Detroit and a majority of them reside in City Council District 6 in Southwest Detroit which is precisely where this course focused. The students began by studying the history of Latinos in Southeast Michigan and exploring how Latinos played critical roles in the city’s development dating back to before World War I. They analyzed broad trends in Latino public opinion, and considered how and why these patterns might be similar or different in Detroit. Students then designed their own pre-election polls to take into the field.
In order to understand what affects voter turnout, students surveyed over 300 residents of Southwest Detroit to measure the issues that were most important to them.
Students pictured here: Storm Boehlke, Mohamad Zawahra, Alex Tabet , Hannel So, Sion Lee.
The results illustrate some powerful patterns. Among the issues that the residents found most important, immigration and crime stood out. Forty-nine and 45 percent of Latinos listed immigration and crime, respectively, as issues of particular concern, with only 31 percent of residents saying that they felt safe in their own home.
Latinos in Southwest Detroit feel extremely high levels of discrimination. Seventy percent of Latinos surveyed said they felt Latinos face “a great deal” of discrimination. This significantly exceeds the roughly half of Latinos nationwide who say they have experienced discrimination.
Student Alex Garcia visits residents in Detroit.
Local issues were also at the forefront of residents’ minds. Latinos had mixed views on the city’s use of blight tickets to combat housing code violations, with one third of respondents supporting them and one third opposing them.
As local organizations, like Michigan United, continue trying to get a paid sick leave initiative on the ballot in 2018, they can expect strong support among Latinos in Southwest Detroit. About two out of every three Latinos in the area indicated they would be more likely to support a candidate who supports the paid sick leave requirement.
The students then followed up with the residents a month later to see if they planned to vote in the upcoming city council election. At this point, the students implemented some interventions that have been used to increase political participation like, evoking emotions that have been shown to have a mobilizing effect, framing voting as an important social norm, and speaking with voters immediately before an election. With the election now over, students are back in the classroom analyzing the effectiveness of these interventions and will use their first-hand experience to better understand public opinion and political participation.
By undergraduate students Megan Bayagich, Laura Cohen, Lauren Farfel, Andrew Krowitz, Emily Kuchman, Sarah Lindenberg, Natalie Sochacki, and Hannah Suh, and their professor Stuart Soroka, all from the University of Michigan.
The 2016 election campaign seems to many to have been one of the most negative campaigns in recent history. Our exploration of negativity in the campaign – focused on debate transcripts and Facebook-distributed news content – begins with the following observations.
Since the advent of the first radio-broadcasted debate in 1948, debates have become a staple in the presidential campaign process. They are an opportunity for voters to see candidates’ debate policies and reply to attacks in real-time. Just as importantly, candidates use their time to establish a public persona to which viewers can feel attracted and connected.
Research has accordingly explored the effects of debates on voter preferences and behavior. Issue knowledge has been found to increase with debate viewership, as well as knowledge of candidates’ policy preferences. Debates also have an agenda-setting effect, as the issues discussed in debates then tend to be considered more important by viewers. Additionally, there is tentative support for debate influence on voter preferences, particularly for independent and nonpartisan viewers. While debate content might not alter the preferences of strong partisans, it may affect a significant percentage of the population who is unsure in its voting decision. (For a review of the literature on debates effects, see Benoit, Hansen, & Verser, 2003).
Of course, the impact of debates comes not just from watching them but also from the news that follows. The media’s power to determine the content that is seen and how it is presented can have significant consequences. The literatures on agenda setting, priming, and framing make clear the way in which media shape our political reality. And studies have found that media’s coverage of debates can alter the public’s perception of debate content and their attitudes toward candidates. (See, for instance, Hwang, Gotlieb, Nah & McLeod 2006, Fridkin, Kenney, Gershon & Woodall 2008.)
This is true not just for traditional media, but for social media as well. As noted by the Pew Research Center, “…44% of U.S. adults reported having learned about the 2016 presidential election in the past week from social media, outpacing both local and national print newspapers.” Social media has become a valuable tool for the public to gather news throughout election cycles, with 61% of millennials getting political news from Facebook in a given week versus 37% who receive it from local TV. The significance of news disseminated through Facebook continues to increase.
It is in this context that we explore the nature of the content and coverage of the presidential debates of 2016. Over the course of a term-long seminar exploring media coverage surrounding the 2016 presidential election, we became interested in measuring fluctuations in negativity across the last 40 years of presidential debates, with a specific emphasis on the 2016 debates. We simultaneously were interested in the tone of media coverage over the election cycle, examined through media outlets’ Facebook posts.
To test these hypotheses, we compiled and coded debate transcripts from presidential debates between 1976 and 2016. We estimated “tone” using computer-automated analyses. Using the Lexicoder Sentiment Dictionary (LSD) we counted the number of positive and negative words across all debates. We then ran the same test over news articles posted on Facebook during the election cycle, taken news feeds of main media outlets including ABC, CBS, CNN, NBC, and FOX. (Facebook data are drawn from Martinchek 2016.)
We begin with a simple measure of the volume of tone, or “sentiment,” in debates. Figure 1 shows the total amount of sentiment – the total number of positive and negative words combined, as a percentage of all words – in all statements made by each in candidate across all debates. In contrast with what some may expect, the 2016 debates were not particularly emotion-laden when compared to past cycles. From 1976 through to 2016, roughy 6.9% of the words said during debates are included in our sentiment dictionary. Hillary Clinton and Donald Trump’s speeches were essentially on par with this average; neither reached the peak of 8% (like 2004) or the low of 6% (like 2012).
Figure 1: Total Sentiment in Debates, 1976-2016
Figure 2 shows the percent of all sentiment words that were negative (to be clear: negative words as a percent of all sentiment words), and here we see some interesting differences. Negativity from Democratic candidates has not fluctuated very much over time. The average percent of negative sentiment words for Democrats is 33.6%. Even so, Hillary Clinton’s debate speeches showed relatively high levels of negativity, at 40.2%. Indeed, Clinton was the only Democratic candidate other than Mondale to express sentiment that is more than 40% negative.
Figure 2: Negativity in Debate Speeches, By Political Party, 1976-2016
Clinton’s negativity pales in comparison with Trump’s, however. Figure 2 makes clear the large jump in negativity for Donald Trump in comparison with past candidates. For the first time in 32 years, sentiment-laden words used by Trump are nearly 50% negative – a level similar to Reagan in 1980 and 1984. Indeed, when we look at negative words as a proportion of all words, not just words in the sentiment dictionary, it seems that nearly every one in ten words uttered by Trump during the debates was negative.
The 2016 debates thus appear to be markedly more negative than most past debates. To what extent is the tone of debate content reflected in news coverage? Does negative speech in debates produce news coverage reflecting similar degrees of negativity? Figure 3 explores this question, illustrating negativity (again, negative words as a proportion of all sentiment words) in the text of all Facebook posts concerning either Trump or Clinton, as distributed by five major news networks.
What stands out most in Figure 3 are the differences across networks: ABC, CNN, and NBC show higher negativity for Trump-related posts, while Fox shows higher negativity for Clinton-related posts. CBS posts reflect a more neutral position.
Figure 3: Negativity in Facebook News Postings by Major Broadcasters, By Candidate, 2016
Clearly, political news content varies greatly across news sources. Trump’s expressed negativity in debates (and perhaps in campaign communications more generally) does not necessarily translate to more negative news content, at least by these measures. For instance: even as Trump is expressing more negative sentiment than Clinton, coverage in Fox is more positive towards Trump. Of course, news coverage isn’t (and shouldn’t be) just a reflection of what candidates say. But these make clear that the tone of coverage for candidates needn’t be in line with the sentiment expressed by those candidates. Expressing negative sentiment can produce negative coverage, or positive coverage, or (as Figure 3 suggests), both.
This much is clear: in line with our expectations, the 2016 presidential debates were among the most negative of all US presidential debates. The same seems true of the campaigns, or at least the candidates’ stump speeches, more generally. Although there was a good deal of negativity during debates, however, the tone of news coverage varied across sources. Depending on citizens’ news source, even as candidates seem to have focused on negative themes, this may or may not have been a fundamentally negative campaign cycle. For those interested in the “tone” of political debate, our results highlight the importance of considering both politicians’ rhetoric, and the mass-mediated political debate that reaches citizens.
This article was co-authored by U-M capstone Communication Studies 463 class of 2016, which took place during the fall election campaign. Class readings and discussion focused on the campaign, and the class found themselves asking questions about the “tone” of the 2016 debates, and the campaign more generally. Using their professor Stuart Soroka as a data manager/research assistant, students looked for answers to some of their questions about the degree of negativity in the 2016 campaign.
As an undergraduate student at the University of Michigan, Xhensila (Janie) Valencia was interested in participating in the University’s Undergraduate Student Research Opportunity Program (UROP). Through the program, she sought a research project that would allow her to build work experience. “I interviewed for several interesting projects, but CLEA fit best with my majors in political science and international studies and sounded the most promising in terms translating into work skills,” she says. She could not have guessed at the time how helpful CLEA would be in that regard.
CLEA is a repository of detailed results from lower house elections from around the world. CLEA provides opportunities for students to be involved at all stages of the data collection process, providing valuable experience and training for them. Working on research projects can be an excellent way for students to explore whether they would like to further their career in research and academia. Many of CLEA’s alumni have gone on to attend graduate school and obtain research-oriented jobs.
Janie remembers her most interesting work with CLEA data: “I’m originally from Albania and immigrated to Michigan with my family at the age of 5, so when I saw that there was a data file on Albania, I immediately volunteered for it” she says. Because she is fluent in Albanian, and is familiar with its political history, she found that file interesting and easier to work with than some of the others, specifically because she could recognize the names of parties in both English and Albanian without having to overcome some of the usual language barriers that sometimes arise when working with the data.
She also found that focusing on the data from specific countries allowed her to learn interesting things about the political history and mood of a country. In particular, Poland stood out to her because they went from having few parties after the fall of communism to many parties, including the Beer Lover’s Party, whose platform was to promote cultural beer drinking in the country.
Janie credits her work with CLEA for helping her land an internship in the U.S. Senate, and later a job at a company called Congressional Quarterly / Roll Call, a subsidiary of the Economist Group that provides congressional research and reporting to subscribers. She was told that it was specifically her work with CLEA that made her uniquely qualified for the researcher position right out of college. She had been working there for about a year and a half when an opportunity arose for a new position that would allow her not only to work with data, but also to broaden her experiences to interpret the data and write about her results.
Janie sees many parallels between her current position and her work with CLEA. “I think it’s vital to provide free accessible information about elections and public opinion for both research purposes and the public good.” She has allowed the notion to carry her into her current job. “Being able to contribute in a way that makes information accessible to the public, is what I do now, and it is also one of the great things about CLEA,” she added. Being cited by news outlets for her research is both exciting for her and satisfying because it means that the public is directly benefiting from data she helped collect and analyze.
Josue Gomez was raised in a farming community in southern Idaho, the son of a Mexican-American farm worker. The current debate on immigration policy, especially the relationship between the U.S. and Mexico, piqued his interest in politics. He enrolled at Boise State University — the first in his family to attend college — and majored in political science. As part of the McNair Scholars program supporting under-represented students, Josue was required to complete a summer research program prior to graduating. Josue and his advisor, Ross Burkhart, identified the University of Michigan as a good place to apply, and Josue was accepted as part of the Student Research Opportunity Program (SROP) at Michigan. Through the SROP program, he joined the Constituency-Level Elections Archive (CLEA) project as a research assistant in the Summer of 2012.
CLEA is a repository of detailed election results from around the world which collects outcomes from lower house elections. CLEA provides opportunities for students to be involved at all stages of the data collection process, providing valuable experience and training for them. Working on research projects can be an excellent way for students to explore whether they would like to further their career in research and academia. Many of CLEA’s alumni have gone on to attend graduate school and obtained research-oriented jobs.
As part of his responsibilities on the CLEA project, Josue was assigned to work on Latin American and a few European countries. Given his fluency in Spanish and natural inclination to learn about these countries, Josue grew a strong connection to the project. Among the countries he was assigned to work on, he was encouraged to choose one to study in more depth. Chile was the largest country in Latin America that was not yet represented in CLEA, and Josue decided that it would be valuable for CLEA to include it. As Josue studied the intricacies and results of elections in Chile, he became interested in a broader research agenda concerning political parties and democratization. Prior to the military dictatorship of Augusto Pinoche, Chile had held elections After the dictatorship was removed, Chile began to hold elections again. In his studies, Josue began to wonder how relationships between parties and an old regime (in Chile’s case, the dictatorship) influence the performance of the parties in elections during and following the transition to democracy.
At the time, Josue was a senior at Boise State University. In his work on CLEA he identified what became a fundamental question for him: How do parties succeed in foundational elections? CLEA also helped him begin to answer this question.
As a McNair Scholar, Josue published a short article based on his work with CLEA. In the paper, Josue lays out a spectrum of parties that exist in new democracies, in order to help understand why some parties are more successful than others. Focused on Latin America, Josue finds a relationship between party alignment in older regimes and success in new elections.
Josue notes that political scientists and other researchers are always looking for reliable data like that provided by CLEA. By examining the electoral rules and election results from countries around the world, researchers can discover what electoral systems work better in certain regions and in certain time frames, investigate how political parties developed or declined, and seek to understand whether and why the democratic experience is working or not.
Josue’s experience in building social science infrastructure and his own research skills in his work with CLEA laid the foundation for his McNair Scholars paper, and has also influenced his academic path. The experience has led him to pursue a Ph.D.; he is currently enrolled as a graduate student in the Department of Political Science at The Ohio State University