On the appropriate use of QCA in environmental management research: A comment on Hossu et al.

Comment to: Hossu, C.A. et al. 2018. Factors driving collaboration in natural resource conflict management: Evidence from Romania. Ambio. 10.1007/s13280-018-1016-0.

Qualitative comparative analysis (QCA) is an innovative and powerful method in the field of environmental governance and environmental management (Dressler et al. 2017; Ide 2018). But if QCAs are not conducted in an appropriate way, the results can be misleading, hence undermining the credibility of both the research field and the method. The study of Hossu et al. (2018) recently published in this journal suffers from such issues. Hossu and colleagues conduct a multi-value QCA of the factors necessary and sufficient for the initiation of collaborative natural resource management in Romania and the USA. The study suffers from four main methodological problems.

First, it uses four causal conditions (~ explanatory factors) in an analysis of eight cases. This poses a “too many variables/too few cases” problem. Marx and Dusa (2011) have shown that when using more than two or three causal conditions for eight cases, the QCA algorithm is likely to identify robust patterns in arbitrary data. While their analysis focuses on crisp-set QCA, the problem holds when employing multi-value QCA.

Second, there is too little variation within the variables. In the sample of Hossu et al., seven out of eight cases are characterised by some form of leadership, and only one experiences no collaborative resource management. By contrast, a variation of around 20% or more is considered appropriate in the QCA literature (Schneider and Wagemann 2012). In addition, the condition “interdependence” does occur in none of the Romanian and in 75% of the US cases, and hence might functions as a proxy for political, economic and cultural differences between both countries. This is especially problematic as all US cases show some collaborative resource management.

Third, a bad conditions-to-cases ratio and little variation combine to produce a problem of limited diversity (Schneider and Wagemann 2012). This is a serious issue given that Hossu et al. choose to use the intermediate solution, which also uses empirically unobserved cases for logical minimisation when they are in line with established theoretical knowledge. I am not arguing that using the intermediate solution is per se inappropriate in small- to medium-N QCAs. But in this case, the solution formula is derived from eight empirically observed cases (characterised by little variation of the relevant conditions) and 28 cases (with more relevant variation) judged on theoretical grounds. This causes a confirmation bias as the vast majority of cases used to derive the solution formula is coded based on the theoretical framework to be tested rather than on empirically observed data (although the QCA algorithm does not allow the intermediate solution to contradict the latter).

Fourth, it is important to use the right type of QCA. Hossu et al. opt for a multi-value QCA, which is preferable if a (gradual or fuzzy) dichotomisation of the data is not possible because multiple “extreme points” exist (Thiem 2013). But Hossu et al. employ a calibration procedure based on (gradual) differences between two ideal types, and hence should opt for a fuzzy-set rather than for a multi-value QCA. For the condition of uncertainty, for instance, they distinguish between cases where parties have no information about the conflict and where they have good information, with an intermediate category of parties having partial information about the conflict.

I have re-run the analysis as a fuzzy-set QCA, using either Hossu et al.’s statements in the paper (version 1) or procedures to address limited variation (version 2) to re-calibrate the data. Full results can be found in Electronic Supplementary Material S1. Both versions of the fuzzy-set QCA detect no necessary conditions. For version 1, both the complex solution (which I prefer in such analyses) and the parsimonious solution (advocated, for instance, by Baumgartner and Thiem 2017) largely contradict the expectations of Hossu et al. The same is true for the complex solution yielded by version 2. The parsimonious solution produced by the version 2 QCA (The solution is: interdependence* (uncertainty + ~ leadership) + consequential incentives * leadership → collaboration) largely supports the results of Hossu et al., but should also be treated with caution due to the limited diversity problem (see above).

Rich insights can certainly be gained from the impressive material collected by Hossu et al. But due to methodical weaknesses and non-robust results, the QCA results are unlikely to be “useful to other post-communist countries when they want to explore the drivers that might lead to the initiation of collaboration” (Hossu et al. 2018, p. 12). More broadly speaking, scholars of environmental management and environmental governance are encouraged to take great care and follow established good practices to produce robust insights when utilising QCA.

Electronic supplementary material

Below is the link to the electronic supplementary material.