Online volunteer laboratories for human subjects research

(with Austin Strange, Mark Hill, and Amy Lakeman), 2019, PLOS ONE 14(8): e0221676

Once a fixture of research in the social and behavioral sciences, volunteer subjects are now only rarely used in human subjects research. Yet volunteers are a potentially valuable resource, especially for research conducted online. We argue that online volunteer laboratories are able to produce high-quality data comparable to that from other online pools. The scalability of volunteer labs means that they can produce large volumes of high-quality data for multiple researchers, while imposing little or no financial burden. Using a range of original tests, we show that volunteer and paid respondents have different motivations for participating in research, but have similar descriptive compositions. Furthermore, volunteer samples are able to replicate classic and contemporary social science findings, and produce high levels of overall response quality comparable to paid subjects. Our results suggest that online volunteer labs represent a potentially significant untapped source of human subjects data.

Supporting Information

Can Violent Protest Change Local Policy Support? Evidence from the Aftermath of the 1992 Los Angeles Riot

(with Aaron Russell Kaufman and Melissa L. Sands), 2019, American Political Science Review

Violent protests are dramatic political events often credited with causing durable and significant changes in public policy.  Scholarly research usually treats violent protests as deliberate acts, undertaken in pursuit of specific policy goals. However, due to a lack of appropriate data and difficulty in causal identification, there is little evidence of whether riots accomplish these goals. We collect unique electoral measures of policy support before and after the 1992 Los Angeles Riot---one of the most high-profile events of political violence in recent American history---which occurred just prior to an election. Contrary to some expectations from the academic literature and the popular press, we find that the riot caused a marked liberal shift in policy support at the polls. Investigating the sources of this shift, we find that it was likely the result of increased mobilization of both African American and white voters. Remarkably, this mobilization endures over a decade later.

Supporting Information | Replication Material

Exclusion and Cooperation in Diverse Societies: Experimental Evidence from Israel

(with Noam Gidron), 2018, American Political Science Review 112(4): 742–757

It is well-established that in diverse societies, certain groups prefer to exclude other groups from power and often from society entirely. Yet as many Western societies are diversifying at an increasingly rapid pace, the need for cross-group cooperation to solve collective action problems has intensified.  Do preferences for exclusion inhibit the ability for individuals to cooperate and, therefore, diminish the ability for societies to collectively provide public goods?  Some scholarship suggests this may not be the case, since preferences are often not diagnostic of behavior.  Turning to Israel, a society with multiple overlapping and politically salient cleavages, we use a large-scale lab-in-the-field design to investigate how much preferences for exclusion among the Jewish majority predict discriminatory behavior toward the Arab minority.  We establish that such preferences appear to be symbolic attitudes, are held especially strongly by low-status members of the majority group, and are strongly predictive of costly non-cooperation.  This preferences-behaviors relationship appears unaffected by mitigating factors  proposed in the intergroup relations literature such as outgroup stereotypes and repeated interactions. The influence of symbolic attitudes on directly observed behavior, which has not been empirically demonstrated before,  calls for a closer examination of the social roots of exclusionary preferences.

Supporting Information


(with Bryce J. Dietrich and Maya Sen), 2018, Political Analysis

Abstract: Do Justices telegraph their preferences during oral arguments? We demonstrate that Justices implicitly reveal their leanings during oral arguments, even before arguments and deliberations have concluded. Specifically, we extract the emotional content of over 3,000 hours of audio recordings spanning 30 years of oral arguments before the Court.  Using only the level of emotional arousal in each of the Justices’ voices during these arguments, as measured by their vocal pitch, we are able to accurately predict many of their eventual votes, while using none of the text or substantive content. These predictions are statistically and practically significant and robust to including a range of controls. Our findings suggest that mannerisms that may be subconscious, such as vocal pitch, carry information that basic legal, political, and textual information do not, and can be used to predict the decisions of even elite political actors.

Supporting Information

The Effect of Segregation on Intergroup Relations

(with Christopher Celaya), 2018, Journal of Experimental Political Science, 5(1): 26–38.

Inter-ethnic residential segregation is correlated with intergroup bias and conflict, poorly functioning states and civil societies, weak economic development, and ethno-centric political behavior. Because of these correlations, segregation has been a subject of long-standing interest. However, segregation has not been assigned in randomized controlled trials, so the observed correlations between segregation and these outcomes may be spurious and the mechanism behind these correlations is poorly understood.  In two experiments we randomly assign segregation in a laboratory and demonstrate that segregation a ects perceptions of other people and causes intergroup bias in costly decision-making. These experiments include the randomized assignment of in-person subjects to the experience of a spatially segregated environment. Rather than segregation merely inhibiting intergroup contact, we demonstrate that segregation directly affects perception and thus can affect intergroup relations even when holding contact constant.

Supporting Information

The Negative Effect Fallacy: A Case Study of Incorrect Statistical Reasoning by Federal Courts

(with Anthony Fowler and Chris Havasy), 2017, Journal of Empirical Legal Studies, 14(3): 618–647.

This article examines the negative effect fallacy, a flawed statistical argument first utilized by the Warren Court in Elkins v. United States. The Court argued that empirical evidence could not determine whether the exclusionary rule prevents future illegal searches and seizures because “it is never easy to prove a negative,” inappropriately conflating the philosophical and arithmetic definitions of the word negative. Subsequently, the Court has repeated this mistake in other domains, including free speech, voting rights, and campaign finance. The fallacy has also proliferated into the federal circuit and district court levels. Narrowly, our investigation aims to eradicate the use of the negative effect fallacy in federal courts. More broadly, we highlight several challenges and concerns with the increasing use of statistical reasoning in court decisions. As courts continue to evaluate statistical and empirical questions, we recommend that they evaluate the evidence on its own merit rather than relying on convenient arguments embedded in precedent.

Context, Perception, and Intergroup Relations

2016, Psychological Inquiry 27(4):294-296.

Comment on Xiao, Coppin, and Van Bavel (2016).


(with Anthony Fowler), 2016, Political Science Research and Methods 6(4): 733–751 (first published online 18 May 2016).

To what extent do political campaigns mobilize voters? Despite the central role of campaigns in American politics and despite many experiments on campaigning, we know little about the aggregate effects of an entire campaign on voter participation. Drawing upon inside information from presidential campaigns and utilizing a geographic research design that exploits media markets spanning state boundaries, we estimate the aggregate effects of a large-scale campaign. We estimate that the 2012 presidential campaigns increased turnout in highly targeted states by 7-8 percentage points, on average, indicating that modern campaigns can significantly alter the size and composition of the voting population. Further evidence suggests that the predominant mechanism behind this effect is traditional ground campaigning, which has dramatically increased in scale in the last few presidential elections. Additionally, we find no evidence of diminishing marginal returns to ground campaigning, meaning that voter contacts, each likely exhibiting small individual effects, may aggregate to large effects over the course of a campaign. 

Supporting Information | Replication Material


(with Noam Gidron), 2016. Journal of Politics 78(3):851--867

Abstract: Why are the negative effects of social diversity more pronounced in some places than in others? What are the mechanisms underlying the relationship between diversity and discriminatory behaviors and why do they vary in prevalence and strength across locations? Experimental research has made advances in examining these questions by testing for differences in behavior when interacting with individuals from different groups. At the same time, research in American and comparative politics has demonstrated that attitudes toward other groups are a function of context. Uniting these two lines of research, we show that discriminatory behaviors are strongly conditioned by the ways in which groups are organized in space. We examine this claim in the context of intra-Jewish conflict in Israel, using original data compiled through multi-site lab-in-the-field experiments and survey responses collected across 20 locations.

Supporting Information | Replication Material


(with Eitan Hersh), 2017, British Journal of Political Science 47: 501–519 (first published online 22 September 2015)

Abstract: Partnering with twenty-five state parties and the Obama 2012 campaign, we surveyed nearly 4,000 campaign staffers from 200 races during the election season, asking about the expected voteshare in their races. Political operatives’ perceptions of electoral closeness can affect how they engage in campaigning and in the representation of citizens, but their perceptions may be wildly inaccurate: incumbent campaigns may irrationally fear close contests; conversely, campaign elites may be over-optimistic of their chances. Building on models of uncertainty, we assess perceptions of closeness. Findings indicate that elites are more optimistic about their chances than they are fearful, and that better-educated elites and elites associated with incumbent and higher-office campaigns are more accurate at assessing their chances. While the public may be better served by politicians fearing defeat, campaigns are typically staffed by workers who are over-confident in their abilities, which may limit the purported benefits of electoral competition.

Supporting Information | Replication Material

What the Demolition of Public Housing Teaches Us About the Impact of Racial Threat on Political Behavior

2016, American Journal of Political Science 60(1):123–142 (first published online 24 February 2015)

Abstract: How does the context in which a person lives affect his or her political behavior? I exploit an event in which demographic context was exogenously changed, leading to a significant change in voters' behavior, and demonstrating that voters react strongly to changes in an outgroup population. Between 2000 and 2004, the reconstruction of public housing in Chicago caused the displacement of over 25,000 African Americans, many of whom had previously lived in close proximity to white voters. After the removal of their African American neighbors, the white voters' turnout dropped by over ten percentage points. Consistent with psychological theories of racial threat, their change in behavior was a function of the size and proximity of the outgroup population. Proximity was also related to increased voting for conservative candidates. These findings strongly suggest that racial threat occurs because of attitude change rather than selection.

Supporting Information | Replication Material

Party Activists as Campaign Advertisers: The Ground Campaign as a Principal-Agent Problem

(with Eitan Hersh), 2015. American Political Science Review 109(2):252–278

Abstract: As a key element of their strategy, recent Presidential campaigns  have recruited thousands of workers to engage in direct voter contact. We conceive of this strategy as a principal-agent problem. Workers engaged in direct contact are intermediaries between  candidates and voters, but they may be ill-suited to convey messages to  general-election audiences. By analyzing a survey of  workers fielded in partnership with the 2012 Obama campaign,  we show that in the context of the campaign widely considered most adept at direct contact, individuals who were interacting with swing voters  on the campaign's behalf  were demographically unrepresentative, ideologically extreme, cared about atypical issues, and misunderstood the voters' priorities.  We find little evidence that the  campaign was able to use  strategies of agent control to mitigate its principal-agent problem. We question whether individuals typically willing to be volunteer surrogates are productive agents for a strategic campaign.

Replication Material

Pivotality and Turnout: Evidence from a Field Experiment in the Aftermath of a Tied Election

(with Anthony Fowler) , 2014, Political Science Research and Methods 2(02):309-319.

Abstract: Many citizens abstain from the political process, and the reasons for this abstention are of great interest and importance. Most scholars and pundits assume that greater electoral competition and the increased chance of pivotality will motivate citizens to participate. We test this hypothesis through a large-scale field experiment which exploits the rare opportunity of a tied election for major political office. Informing citizens that an upcoming election will be close has little mobilizing effect. To the extent that we do detect an effect, it is concentrated among a small set of frequent voters. Our evidence suggests that increased pivotality is not a solution to low turnout, and the predominant models of turnout focusing on pivotality are of little practical use.

Supporting Information | Replication Material

Causal Effect of Intergroup Contact on Exclusionary Attitudes

2014, Proceedings of the National Academy of Sciences of the United States of America 111(10): 3699–3704

Abstract: The effect of intergroup contact has long been a question central to social scientists. As political and technological changes bring increased international migration, understanding intergroup contact is increasingly important to scientific and policy debates.  Unfortunately, limitations in causal inference using observational data and the practical inability to experimentally manipulate demographic diversity has limited scholars’ ability to address the effects of intergroup contact. Here, I report the results of a randomized controlled trial testing the causal effects of repeated intergroup contact, in which Spanish-speaking confederates were randomly assigned to be inserted, for a period of days, into the daily routines of unknowing Anglo-whites living in homogeneous communities in the United States, thus simulating the conditions of demographic change. The result of this experiment is a significant shift toward exclusionary attitudes among treated subjects. This experiment demonstrates that even very minor demographic change causes strong exclusionary reactions.  Developed nations and politically liberal subnational units are expected to experience a politically conservative shift as international migration brings increased intergroup contact.

Supporting Information | Replication Material | Correction

van Hoorn letter to editor | Enos reply to van Hoorn


Increasing Inequality: The Effect of GOTV Mobilization on the Composition of the Electorate

(with Anthony Fowler and Lynn Vavreck), 2014, Journal of Politics 76 (1):273-288  

Abstract: Numerous get-out-the-vote (GOTV) interventions are successful in raising voter turnout. However, these increases may not be evenly distributed across the electorate and may actually increase the differences between voters and non-voters. This phenomenon is particularly notable given the many GOTV strategies that explicitly aim to reduce inequalities in representation. By analyzing individual level-data, we reassess previous GOTV experiments to determine which interventions mobilize under-represented versus well-represented citizens. We develop a generalized and exportable test which indicates whether a particular intervention reduces or exacerbates disparities in political participation and apply it to 27 previous experimental interventions. Despite raising mean levels of voter turnout, more than two-thirds of the interventions in our sample widened disparities in participation. On average, voter mobilization strategies tend to increase the participation gap, thereby exacerbating representational inequality. We conclude by discussing substantive implications for political representation and methodological implications for experimenters.

Supporting Information | Replication Material

Does Public Financing Chill Political Speech? Exploiting a Court Injunction as a Natural Experiment

(with Conor M. Dowling, Anthony Fowler, and Costas Panagopoulos), 2012. Election Law Journal 11(3):302--315.

Abstract: In 2011, the Supreme Court struck down the matching provisions in Arizona's campaign finance law on the grounds that they violate free speech by chilling private spending. In this article, we explicitly test the effects of Arizona's matching provisions in two ways. First, we find that privately funded state legislative candidates do not strategically cluster their spending below the threshold that would trigger money to their opponents. Second, we exploit a 2010 Court injunction as a natural experiment. When Arizona's matching provisions were removed, private spending did not increase relative to other states. Contrary to the view of the Court, we find no empirical evidence that campaign finance laws chill private political speech. More generally, our analysis demonstrates the value of exploiting court injunctions as natural experiments to assess the causal effects of laws.

Replication Material

Related Amicus brief, McComish v Bennett


(with Matthew Atkinson and Seth J. Hill), 2009. Quarterly Journal of Political Science 4 (3), pp 229-249. 

Abstract: We estimate the effect of candidate appearance on vote choice in congressional elections using an original survey instrument. Based on estimates of the facial competence of 972 congressional candidates, we show that in more competitive races the out-party tends to run candidates with higher quality faces. We estimate the direct effect of face on vote choice when controlling for the competitiveness of the contest and for individual partisanship. Combining survey data with our facial quality scores and a measure of contest competitiveness, we find a face quality effect for Senate challengers of about 4 points for independent voters and 1 to 3 points for partisans. While we estimate face effects that could potentially matter in close elections, we find that the challenging candidate's face is never the difference between a challenger and incumbent victory in all 99 Senate elections in our study.

Corrigendum to Atkinson, Enos, and Hill (2009).

Replication Material

Conservatism and Fairness in Contemporary Politics: Unpacking the Psychological Underpinnings of Modern Racism

(with Riley K. Carney)

The study of intergroup attitudes is a central topic across the social sciences.  While there is little doubt about the importance of intergroup attitudes in shaping behavior, both the psychological underpinnings of these attitudes and the tools used to measure them remain contentious.  Modern racism scales, which are the most common way to measure anti-Black prejudice in political science, were created in response to a shift in the attitudes of white Americans toward African Americans, and reflect a mix of social conservatism and anti-Black affect. Using experiments, we offer evidence that modern racism scales measure attitudes toward any group, rather than African Americans alone. In the spirit of the original motivation behind modern racism scales, which were created to capture changing public opinion about race, we suggest this property of modern racism may reflect a change in how stereotypes about low work-ethic are applied across groups and that the target of resentment for white Americans, especially for political conservatives, has broadened beyond African Americans. Our results suggest that modern racism scales reflect a general set of attitudes about fairness and that new instruments may be needed to measure group-specific prejudice.  

Intrinsic Motivation at Scale: Online Volunteer Laboratories for Social Science Research

(with Austin Strange, Mark Hill, and Amy Lakeman)

Once a fixture of empirical social science, volunteer subjects are now only rarely used in human-subjects research. Yet volunteers are a potentially valuable resource, especially for research conducted online. We argue that online volunteer laboratories, by relying on intrinsically motivated subjects, are able to produce high-quality data and may avoid some of the challenges associated with paid online crowd-sourcing. Furthermore, we argue that volunteer labs are scalable and able to produce large datasets for multiple researchers, while imposing little or no financial burden. Using a range of original tests, we show that volunteer and paid respondents differ in their motivations for participation yet have similar descriptive composition. Volunteer samples are able to replicate classic and contemporary social science findings, and produce high levels of overall response quality relative to paid subjects. Our results suggest that volunteer labs represent a potentially significant untapped source of human-subjects data in the social sciences.