Reply to Inbar: Contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology

We find that contextual sensitivity correlates with the reproducibility of 100 psychology studies from the Reproducibility Project (1). This relationship remains after adjusting for several methodological factors believed to account for reproducibility (e.g., sample size, effect size) and does not differ across subdisciplines (social vs. cognitive psychology). We also report a strong relationship between subdiscipline and context (1): social psychology studies are more likely to examine contextually sensitive topics than cognitive psychology studies. We conclude that context might play a role in reproducibility across (rather than within) subdisciplines.

Inbar (2) notes that the reproducibility rate of social psychology (28%) is lower than cognitive psychology (53%). He also reports that the association between context and reproducibility becomes nonsignificant when adjusting for subdiscipline. On this basis, Inbar argues that the relationship between context and reproducibility is spurious. This claim is conceptually misguided and overlooks a more important issue: contextual sensitivity helps explain the reproducibility gap between social and cognitive psychology.

The authors of the Reproducibility Project: Psychology (RP:P) argued that the lower reproducibility rate of social psychology is a result of weaker statistical power and effect sizes (3). But subdiscipline predicts reproducibility (P = 0.044) even after adjusting for methodological factors (e.g., sample size, effect size; see model 2 in ref. 1). However, subdiscipline no longer predicts reproducibility once context is added to the model (P = 0.453; see https://osf.io/cgur9/ for details) (Table 1). Although he does not report it, Inbar’s (2) own model reveals that the association between subdiscipline and reproducibility becomes nonsignificant when adjusting for context. In sum, context appears to explain the reproducibility gap better than these methodological factors.

Table 1.

Regression coefficients predicting reproducibility and variance explained in two steps of a hierarchical regression

Variable	b	SE	P	R2
Step 1				0.60
Effect size, original study	2.62	1.77	0.200
Surprisingness of original study	−0.67	0.37	0.072
Power of replication	8.34	4.18	0.046
Surprisingness of replication study	−2.04	0.51	<0.001
Replication similarity	0.18	0.30	0.552
n of original study	<0.001	<0.001	0.899
Subdiscipline	−1.45	0.72	0.044
Step 2				0.62
Effect size, original study	1.78	1.83	0.330
Surprisingness of original study	−0.67	0.39	0.087
Power of replication	10.86	4.68	0.020
Surprisingness of replication study	−2.26	0.59	<0.001
Replication similarity	0.23	0.30	0.448
n of original study	<0.001	<0.001	0.929
Subdiscipline	−0.66	0.88	0.453
Contextual sensitivity	−0.60	0.40	0.132

R² values refer to Nagelkerke R². Bold text indicates the relationship between subdiscipline (social vs. cognitive psychology) and reproducibility adjusting for other variables. Subdiscipline predicts reproducibility over-and-above several methods' variables (step 1), but becomes nonsignificant once contextual sensitivity is added to the model (step 2).

The fact that context and subdiscipline become nonsignificant when entered simultaneously in a regression demonstrates multicollinearity and makes it inappropriate to categorize one as a confound (4). Before claiming a spurious relationship, one needs to rule out a mediating relationship. It seems more likely that subdiscipline affects reproducibility through contextual sensitivity, rather than the other way around. Contextual sensitivity cannot cause a change in subdiscipline. Thus, Inbar’s (2) implied conceptual model appears to be backward.

There are other problems with prioritizing subdiscipline. First, no coding scheme for subdiscipline was reported in the RP:P (3). It is therefore impossible to know if the RP:P coders used context as a heuristic for categorizing studies as social vs. cognitive. Second, the binary coding scheme imposes a false dichotomy between social and cognitive psychology that cannot account for fields like social cognition (5). These coding limitations might explain why several studies appear to have been miscoded in the RP:P [as we noted in our paper (1)].

Our analysis suggests that different rates of reproducibility between social and cognitive psychology partially stem from differences in contextual sensitivity. This will come as little surprise to social psychologists: The notion that human psychology is shaped by the social context has been the central premise of the field for nearly a century (6). We expect that same principle applies across the social sciences.