Against Dyadic Design

As I begin I should note that I’m on Team Cranmer and Desmarais (2016) all the way. I was at UNC at the same time as they were, and Skyler was on my dissertation committee. That gave me the opportunity to argue about these and other issues over the course of several years, which I did. (Sorry for being difficult, Skyler; it’s how I learn.) In fact, I have made many of the same arguments that Paul Poast (2016) and Paul Diehl and Thorin Wright (2016) make in their contributions to this excellent symposium, in person. And it is because I have already lost those arguments that I can say, with confidence and humility, that I am on Team Cranmer and Desmarais.

 

At the same time I am surprised by the extent to which Poast (2016) and Diehl and Wright (2016) are also on Team Cranmer and Desmarais. Their responses to the Cranmer and Desmarais critique are tepid. They are not tepid enough, I argue below, but they both accept all of Cranmer and Desmarais’s main points at least partly (and many of them in full). I am surprised because there has been a parallel debate in international political economy (IPE) circles regarding many of the same issues, and in IPE the issue appears to be far more contentious. While I was very disappointed to see zero references in this symposium to the IPE discussion – which hasn’t exactly been hidden: it has taken place since 2011 in journals such as International Organization and Perspectives on Politics, at conferences like the annual meetings of APSA and ISA, and on prominent weblogs and social media – I was encouraged that all of the participants are willing to accept the need for reflection and critique.

 

The first step, as they say, is to admit you have a problem. The problem that Cranmer and Desmarais articulate is that dyadic research designs conducted on relational systems (as are all international political and economic systems) must make two strong assumptions about hyperdyadic dependence, i.e. interdependence beyond the “level” of the dyad: first, that it does not exist; second, that if it exists it is not important or interesting. They rightly claim that hyperdyadic dependence will usually exist, and I would go further to claim that there are very few theories (as opposed to detached hypotheses) in which hyperdyadic dependence does not play an important role. In fact, I struggle to think of a single theory in international relations in which that is the case.

 

Cranmer and Desmarais also claim that hyperdyadic dependence is often important and is usually interesting. To drive this point home they distinguish between hypothetical expectations about covariance, which are often modest and narrow, and models of the entire data-generating process. These are by necessity less modest, less narrow, and require systemic thinking. When I say “systemic” I do not intend to bring to mind functionally undifferentiated billiard balls colliding on a bed of pure anarchy. I mean what Herbert Simon described in 1962 as a collection “of a large number of parts that interact in a nonsimple way” (p. 468). A dyadic interaction contains the smallest number of parts that can interact. And, in a regression framework, it is difficult to specify a model containing nonsimple effects.

 

So in a sense what Cranmer and Desmarais are asking is this: “Do you think the world is simple, or complex?” They believe it is complex, and I believe that most other IR scholars do too. But our commonly-used models are too simple to capture this complexity, so we too often make less-interesting theoretical propositions than we otherwise might. Cranmer and Desmarais characterize this state of affairs as “theoretical myopia driven by the availability of data” rather than any principled theoretical choice. But the problems do not end with theory. Indeed, dyadic designs can also lead to poor statistical inferences when hyperdyadic dependence exists.

 

Thus, Cranmer and Desmarais’s charge is that IR has long committed sins of omission, and that we remain in sin. They call us to repent, with haste, and change our ways. Poast (2016) and Diehl & Wright (2016) do not share Cranmer and Desmarais’s sense of urgency despite accepting nearly the whole of their critique. Their responses sometimes recall Augustine of Hippo: “Grant me chastity and continence, but not yet!” I believe this is a mistake for reasons I’ll describe below, but first let me try to characterize the two types of objections that PP and DW primarily make: first, that in practice the problems Cranmer and Desmarais identify are not (always) that big of a deal; second, that even in the face of potentially-severe problems, a dyadic design can make sense if it is chosen for reasons of utility or theoretical parsimony.

 

For example, Diehl & Wright (2016) argue that dyadic designs have generated many knowledge gains in the past. Of course they have, and Cranmer and Desmarais say the same thing in their initial critique. But the question is whether it is appropriate now to continue in our old ways, given that all of the low-hanging and most of the higher-hanging fruit has already been picked from the dyad tree?

 

Diehl & Wright (2016) argue that it might be, if an outcome of interest occurs at the level of the dyad. I appreciate Zinnes (1980) as much as the next guy, but referencing a nearly forty-year old address does not on its own constitute a justification of a practice in the face of powerful theoretical and statistical arguments that the practice is suboptimal in the present day. They seem to know that dyadic hypotheses can be tested in a hyperdyadic context, but I’d like to reinforce the point: an exponential random graph model (ERGM) can test the probability of a relationship forming between i and j given a set of covariates just as well as a dyadic regression can. In fact, Cranmer and Desmarais provide (in this paper and their others) the statistical proof that dyadic regressions are simply special cases of the more general, and flexible, ERGM. Thus it is worth repeating: there is really nothing you can do with a dyadic regression that you can’t do with an ERGM; the reverse is not true.

 

Thus, it would require a very brave theory to force us to choose a dyadic regression over an ERGM given relational data, regardless of what we think about “levels of analysis”.

 

Unlike Diehl & Wright, Poast (2016) admits the inferiority of dyadic models whenever there is any multilateral process, but argues that the assumption of independence of dyads is not so problematic. I would like to make two points about this, briefly. The first is that it is unknowable, ex ante or ex post, how big of a problem it is to assume independence when there is interdependence. There is no statistical test for this, nor is there any robustness check or post-estimation procedure that can tell us how much our estimates are biased when we wrongly assume independence. But what we do know for certain is that there is bias, every time, in an unknown direction, to an unknown degree. I don’t know about you but this makes me uncomfortable.

 

The second defense that Poast (2016) makes against Cranmer and Desmarais’s critique is that the bias from wrongly assuming independence can be partially mitigated by engaging in extreme statistical exertions. I’m not sure of the sequence in which the authors exchanged their drafts, but Cranmer and Desmarais actually discuss several of these in their piece – the spatial approach proposed by Neumayer and Plumper (2010) and the k-adic approach developed by Poast (2016). While certainly a major improvement over prior practice these are partial fixes, are limited in some key respects (including unidimensionality), and are far more labor-intensive than simply switching to a model that does not assume independence in the first place.

 

There are other problems that Cranmer and Desmarais did not mention, or barely mentioned. I have already blown past my suggested word limit so I will briefly discuss only one. That is that dyadic regressions multiply data, which artificially inflates the number of observations, which artificially shrinks standard errors. From the perspective of inference this is problematic. In a world of 200 states there are nearly 40,000 dyadic observations per year, which are treated independently in dyadic regression. This strikes me as a very strange way to think about the world at any particular point in time, but it also makes me skeptical of IR papers with around a million observations in which almost every variable has three stars next to its coefficient. Again, one might propose this or that statistical tweak to partially mitigate this problem, but I agree with Cranmer and Desmarais that there is a better way forward.

 

So I conclude where I began: I am on Cranmer and Desmarais. I also needed to be convinced, but now I believe their critique is much-needed, long-overdue, and should be taken seriously by scholars, reviewers, and editors. There may be times when data limitations or some other suboptimality force us to use models that we know are not the best of all possible worlds. But whenever we have dyadic data we can, and should, do more with them than we typically have done.

 

 

References:

 

Neumayer, Erik, and Thomas Plumper. 2010. “Spatial Effects in Dyadic

Data,” International Organization 64(1): 145–66.

 

Simon, Herbert A. 1962. “The Architecture of Complexity,” Proceedings of the American Philosophical Society 106(6): 467-482.

 

Zinnes, Dina. 1980. “Three Puzzles in Search of a Researcher,”

International Studies Quarterly 24(3): 315–42.

 

 

 

Discuss this Article
There are currently no comments, be the first to post one.
Start the Discussion...
Only registered users may post comments.
ISQ On Twitter