This week, I analyzed the data collected in another survey completed by my undergraduates.
Once again, my goal was to replicate existing research in a relevant topic, and show the students how we build a research design and find similar results.
The Assignment
The survey described two different scenarios. First, they read about a conflict that involves US interests, and the president making a promise regarding American involvement. In one condition, the promise is to "Stay out" of the conflict. In the other condition, it is an "Empty threat" - a promise to send military forces, but then backing-down and deciding not to intervene. The respondents are asked to state their level of approval for the decision (using a 5-point scale items ranging from "Approve strongly" to "Disapprove strongly").
The figure below plots their aggregated responses. I create two categories based on the responses (approve or disapprove) and remove the middle category ("Neither"). The figure displays the proportion of both options, separated by type of policy choice – "Stay out" or "Empty threat".
These results replicate the work by Tomz (2007) who demonstrated that when the president chooses to back-down from a promise, the degree of disapproval is much higher, compared to a decision not to intervene. These findings provide evidence for the microfoundations of the "audience costs" theory of international relations.
In the second experiment, respondents read about another conflict scenario, and the American response. In both cases, the president violates an early promise. First, the "Empty threat" option as described in experiment 1 above (a decision to "Back-down" after initial threat). Second, a “Backed-in” option in which the president promises to “stay-out”, but then chooses to send forces and intervene in the conflict. I measure the proportion of approval for either decision. Below are the responses based on a 5-point scale.
This first-cut of the responses shows that respondents are more critical of an "Empty threat" (approximately 75% disapproval when backing-down) compared to a decision to "Back-in" (which gains about 48% approval).
This experiment included another element - the role of new information. The question is what are the effects on approval when the president provides new information to justify the decision to "back-down" or "back-in"?
Similar to experiment 1, I aggregate the responses to categories of "Approve" or "Disapprove" (and remove the middle category of "Neither"). The figure below displays the proportion of approval for either type of 'broken' promise, also accounting for the information condition.
What can we learn from this analysis?
First, as the previous figure shows, respondents disapprove of an "Empty threat", and are less concerned from the other type of broken promise (“Backed-In”). Also, the new information condition is pretty powerful - providing justification for the president's choice can reduce the disapproval for "Backing-down", and enhance the approval when "Backing-in".
These findings replicate two existing studies that test the public angle of audience costs theory (Fearon 1994). First, the other type of broken promise - "Backing-in", is explored in Levy et al. (2015). The effects of new information on approval are introduced by Levendusky and Horowitz (2012).
Finally, the survey collected data to evaluate how the choice to violate a promise affects views of the president's competence, as well as how it affects the perceived reputation and credibility of the US. The responses for these items are based on 1-4 or 1-5 scales. I collected the negative responses (i.e. leader is incompetent, damages to reputation etc.) and compared them for each of the policy options ("Back-down" and "Backed-in").
In all three options, the decision to "Back-down" is viewed much more negatively. It harms the US credibility and reputation a lot more, and the president is seen as more incompetent in such a scenario.
If you are interested in the R code and associated document that describe the experiment procedure and analysis described here, check out my Github page.