Jumping to Conclusions About the Beads Task? A Meta-analysis of Delusional Ideation and Data-Gathering

Abstract

It has been claimed that delusional and delusion-prone individuals have a tendency to gather less data before forming beliefs. Most of the evidence for this “jumping to conclusions” (JTC) bias comes from studies using the “beads task” data-gathering paradigm. However, the evidence for the JTC bias is mixed. We conducted a random-effects meta-analysis of individual participant data from 38 clinical and nonclinical samples (n = 2,237) to investigate the relationship between data gathering in the beads task (using the “draws to decision” measure) and delusional ideation (as indexed by the “Peters et al Delusions Inventory”; PDI). We found that delusional ideation is negatively associated with data gathering (r _s = −0.10, 95% CI [−0.17, −0.03]) and that there is heterogeneity in the estimated effect sizes (Q-stat P = .03, I ² = 33). Subgroup analysis revealed that the negative association is present when considering the 23 samples (n = 1,754) from the large general population subgroup alone (r _s = −0.10, 95% CI [−0.18, −0.02]) but not when considering the 8 samples (n = 262) from the small current delusions subgroup alone (r _s = −0.12, 95% CI [−0.31, 0.07]). These results provide some provisional support for continuum theories of psychosis and cognitive models that implicate the JTC bias in the formation and maintenance of delusions.

Introduction

In a now classic study, the “beads task”¹ was adapted to examine the relationship between delusions and data-gathering.² Participants were shown 2 jars of beads, a mostly pink jar (85 pink beads; 15 green beads) and a mostly green jar (85 green beads; 15 pink beads). The jars were then hidden and participants were shown a sequence of beads apparently being drawn from 1 of the 2 jars (the sequence was actually prespecified by the experimenter). After each draw, participants were asked if they were ready to make a decision about which jar the beads were being drawn from or if they would like to see another bead. This study found that people with delusions made a decision about which jar the beads were being drawn from on the basis of significantly fewer beads than controls. This study has inspired a large empirical literature, and primarily on the basis of evidence from studies using the beads task paradigm it has been argued that people with delusions show a “jumping to conclusions” (JTC) bias: they are willing to accept hypotheses on the basis of less evidence than non-delusional people.^3–5

It has long been argued that the positive symptoms of psychosis—delusions and hallucinations—lie at the extreme end of a continuum of similar subclinical phenomena in the general population.^6–10 The existence of a “psychosis continuum” is supported by 2 recent meta-analyses.^11,12 Furthermore, it has been argued that the syndrome-based diagnostic categories of psychiatry impede progress in understanding the aetiology of mental illnesses and research should be reoriented to focus on vulnerability traits and symptoms across diagnostic categories and within the general population.^13–15 Notably, the American National Institute of Mental Health (NIMH) has recently released a strategic plan that proposes abandoning syndrome-based classifications of the Diagnostics and Statistical Manual of Mental Disorders (DSM) to examine “the full range of variation, from normal to abnormal, among the fundamental components [dimensions] to improve understanding of what is typical versus pathological.”¹⁶

A dimensional approach that examines variation in the general population could provide insight into clinical delusions. In particular, evidence for an association between the JTC bias and delusional ideation in the general population would provide support for influential cognitive models that implicate the JTC bias in the aetiology of clinical delusions.^17–23 The Peters et al Delusions Inventory (PDI)²⁴ is by far the most widely used measure of delusional ideation in the beads task literature. It is a self-report questionnaire that asks people if they have ever had particular delusion-like experiences. For example, one item asks, “Do you ever feel as if there is a conspiracy against you?” For each item endorsed people are asked to rate the degree of associated distress, preoccupation, and conviction separately on 5-point Likert scales. The original PDI has 40 items,²⁴ but a 21-item version with comparable psychometric properties is also widely used.²⁵ On average, patients diagnosed with delusions report higher PDI scores than healthy controls^24,25; on average, members of new religious movements report PDI scores that fall between those of delusional and general populations²⁶; and PDI scores correlate moderately strongly with observer-rated delusions using a structured clinical interview.²⁷ Such findings suggest that the PDI is a valid measure of thoughts that lie on a continuum with delusional beliefs.

Studies of the association between the JTC bias and delusions do not always provide consistent results, casting doubt on cognitive models implicating the JTC bias in delusion formation and maintenance. Consequently, careful evaluation of the overall weight of evidence is needed. A 1999 systematic review by Garety and Freeman⁵ argued that 7 of 8 beads task studies provided evidence that the JTC bias is associated with delusions. A 2013 follow-up by Garety and Freeman⁴ reviewed 53 new beads task studies (and 8 studies using other probabilistic reasoning tasks) and concluded that “the clear majority of these studies … confirmed that JTC is characteristic of people with delusions” (p. 328). Although this “vote counting” approach is commonly employed in systematic reviews, it is known to have significant limitations.^28,29 First, it does not take into consideration features of studies (such as sample size) that ought to result in some studies being weighed more heavily than others. Second, it does not test or control for publication bias, selective reporting bias, or other biases that can inflate the evidence for an effect. Third, it does not quantify effect sizes.

The limitations of systematic reviews make meta-analysis a crucial tool for integrating evidence across studies.^29–32 A 2007 meta-analysis by Fine et al³ examined 12 studies. They found that 1 measure of the JTC bias reached statistical significance (“draws to decision”), but another 3 measures did not. Although useful, this meta-analysis has limitations. First, it used multiple effect sizes from the same samples (47 effect sizes from 22 samples) to estimate a single underlying construct (the JTC bias), which violates the assumption of statistical independence.^29,32 Second, the Stouffer method of meta-analysis was used,³³ which makes the problematic assumption that all studies are sampled from a single population.²⁹ Third, the possibility of publication bias and selective reporting bias was not examined. Fourth, effect sizes were not quantified.

Taylor et al³⁴ recently preregistered a rationale and protocol for a meta-analysis that will examine whether the JTC bias is associated with clinical delusions. This protocol is methodologically sophisticated and promises to address limitations of the earlier systematic reviews and meta-analysis. Nevertheless, because this protocol focuses on between-group differences it will not speak directly to the hypothesis that delusions lie at the extreme end of a continuum of subclinical phenomena within the general population.^6–12

It has been argued that a variety of questionable research practices are prevalent in the social sciences, and an anonymous survey suggests that selective reporting of results is particularly widespread.³⁵ Selective reporting bias can result in false-positives well above the nominal rate of 5%,³⁶ even when researchers strive to report their results dispassionately and honestly.³⁷ Selective reporting bias can be curtailed by direct replication of experiments, ideally with preregistered protocols, and open access to raw data. However, due to the value placed on innovation in the social sciences, direct replication is extremely rare.³⁸ In this respect, the beads task literature fits the typical profile of social science research. We were unable to identify any beads task study that directly replicated an earlier study, or used a preregistered protocol, or provided open access to raw data. Consequently there is considerable scope for selective reporting bias.

In the present meta-analysis, we tested the hypothesis that delusional ideation is negatively associated with data-gathering in the general population and clinical populations. We did not use the effect sizes reported in publications. Instead we acquired raw data for each study that met our inclusion criteria and calculated the precise effect size of interest for each sample: the association between draws to decision on the beads task and PDI scores. This “individual participant data” approach offers numerous advantages over conventional meta-analysis and is considered to be the “gold standard” of systematic review.^39–41 Two advantages are particularly salient when considering the beads task literature. First, beads task studies typically report differences between samples only. That is, variation within delusional samples and within nondelusional samples, which is crucial for testing continuum models, is not always reported. Using individual participant data we were able to examine this crucial, but neglected, variation. Second, by consistently applying the same screening criteria and statistical tests to samples from different studies we were able to circumvent selective reporting bias and related biases.

Methods

Search Strategy

We used 2 strategies for identifying studies for possible inclusion in our meta-analysis. First, we assessed for eligibility studies tabulated in the 2013 systematic review by Garety and Freeman.⁴ They reported using 3 search techniques. First, they searched the Web of Science and PubMed databases using the following search terms: “jump to conclusions” and “delusions”; “jump to conclusions” and “schizophrenia”; “jump to conclusions” and “psychosis”; “jump to conclusions” and “paranoia”. Second, they consulted 5 widely cited review articles on delusions.^3,5,22,42,43 Third, they manually searched “early view” articles in the journals Schizophrenia Bulletin, Schizophrenia Research, British Journal of Clinical Psychology, Behaviour Research and Therapy, Journal of Behavioural Research and Experimental Psychiatry, Psychological Medicine, and Journal of Abnormal Psychology. In addition, we searched for articles published from 2011 to the present and “early view” articles using the same search techniques as Garety and Freeman⁴ to identify studies that might have been published after they completed their search.

Second, we used Google Scholar’s cited reference search functionality to identify studies that had cited the article that introduced the 40-item PDI²⁴ or the 21-item PDI from 2011 to the present.²⁵ Of the articles identified by Google Scholar, we inspected for possible inclusion those that had titles that indicated that they might include the beads task. Our literature search was completed July 10, 2014, and is inclusive of studies published up to that date.

Inclusion Criteria

We categorized studies as eligible for inclusion in our meta-analysis if they met 2 inclusion criteria. First, the study used either a 40-item or 21-item PDI. We included all studies that used a PDI, even if the PDI had been modified. This meant that we included 1 study that used a version of the PDI that measured preoccupation only⁴⁴ and 3 studies that did not use the 3 PDI subscales but used only the initial “yes/no” question.^45–47 Second, the study used a standard 2 jar draws to decision version of the beads task. We did not include studies that used “beads task-like” data-gathering paradigms (such as “emotional beads task” or “fishing task” paradigms) because we wanted tasks to be as directly comparable as possible.

We are interested in delusion ideation across syndrome-based diagnostic categories, so we did not exclude clinical groups that did not have a diagnosis of schizophrenia. We emailed the authors of all eligible studies with a request for raw data from their published and unpublished studies. We succeeded in sourcing raw data for all eligible published studies (bar one⁴⁸) and one currently unpublished study (R. Ephraums and R. P. Balzan, unpublished data). All studies we sourced that met our 2 inclusion criteria were included in the meta-analysis.

Data Coding

When possible, we calculated total PDI scores for each participant by adding the number of “yes” responses to scores from the 3 subscales. This was possible for 27 samples (see table 1 for exceptions). Twenty-five samples were tested using the 21-item PDI^{44–46,49–59} (including R. Ephraums and R. P. Balzan, unpublished data) and 13 using the 40-item PDI.^{47, 60–65} When calculating total PDI scores for samples that used the 40-item PDI, we included only those 21 items that appear in the 21-item PDI. Raw data obtained for 6 samples that used the 40-item PDI did not include scores for individual items^{47, 60}; in these cases we used the total PDI score for all 40 items. Eight samples used a version of the PDI that did not include the subscales^45–47; in these cases we used “yes” responses to the initial questions to calculate the PDI total score. For the sample that used a version of the PDI that measured preoccupation only,⁴⁴ we used the preoccupation score to calculate the PDI total score.

Table 1.

Characteristics of Samples Included in the Meta-analysis

Sample	Subgroup	Participants	Mean Age	% Female	% Easy	Trials	Note
Bensi et al49	General population	140	25.82	55	50	2
Bentall et al47 (a)	General population	63	56.32	63.49	0	3	1,2
Bentall et al47 (b)	Current delusions	83	49.22	45.78	0	3	1,2
Bentall et al47 (c)	Previous delusions	27	34.7	37.04	0	3	1,2
Bentall et al47 (d)	Anxiety or depression	55	63.49	60	0	3	1,2
Broome et al60 (a)	General population	22	24.87	N/A	50	2	1
Broome et al60 (b)	At risk	27	24.56	N/A	50	2	1
Cafferkey et al50 (a)	General population	133	22.86	68.42	100	1
Cafferkey et al50 (b)	General population	136	26.88	71.32	0	1
Colbert and Peters51	General population	68	41.21	55.88	100	1
Ephraums and Balzan	General population	99	23.38	75.76	50	2
Jacobsen et al52 (a)	General population	16	34.19	56.25	50	2
Jacobsen et al52 (b)	Current delusions	16	39.5	43.75	50	2
Jacobsen et al52 (c)	OCD	32	35.66	62.5	50	2
Keefe and Warman61	General population	132	21.42	78.79	0	4
Langdon et al45 (a)	General population	34	32.03	23.53	100	1	2
Langdon et al45 (b)	Current delusions	29	35.1	34.48	100	1	2
Langdon et al46 (a)	General population	19	20.79	10.53	100	1	2
Langdon et al46 (b)	Current delusions	17	20.59	0	100	1	2
Lim et al53 (a)	General population	63	23.95	74.6	50	2
Lim et al53 (b)	Current delusions	25	24.6	36	50	2
Lim et al53 (c)	NRM	32	31.03	37.5	50	2
Lincoln et al62 (a)	General population	68	33.76	38.24	50	6
Lincoln et al62 (b)	Current delusions	44	35.48	31.82	50	6
Lincoln et al62 (c)	Previous delusions	27	30.59	29.63	50	6
McKay et al54	General population	57	20.96	63.16	100	1
Menon et al55	General population	121	31.05	64.46	0	1
Ochoa et al44	General population	57	45.07	40.35	50	2	3
Peters and Garety56 (a)	General population	36	27.72	50	100	1
Peters and Garety56 (b)	Current delusions	18	32.22	11.11	100	1
Peters and Garety56 (c)	Anxiety or depression	21	41.19	47.62	100	1
Rodier et al57	General population	78	29.24	57.69	100	1
So et al58 (a)	General population	30	20.07	66.67	50	2
So et al58 (b)	Current delusions	30	21.6	56.67	50	2
Warman and Martin63	General population	199	21.11	77.39	100	4
Warman et al64	General population	59	21.39	74.58	0	4
White and Mansell59	General population	39	19.44	84.62	50	2
Ziegler et al65	General population	85	24.31	58.82	100	3

Note: OCD, obsessive-compulsive disorder; N/A, information not available; NRM, new religious movement. Note 1 = 40-item Peters et al Delusions Inventory (PDI) for total PDI score; Note 2 = initial "yes/no" question for total PDI score; Note 3 = preoccupation for total PDI score.

Samples varied with respect to the maximum number of beads participants were able to request before making a decision about which jar the beads were being drawn from. Participants were not told what this limit was. If they reached this limit they were asked to make a decision. Following standard practice, we retained data from participants who reached the limit. Samples also varied with respect to the number of beads task trials presented to each participant. For analysis, we calculated a mean draws to decision score across trials for each participant. We coded beads task trials with a ratio of beads of 60:40 as “difficult” and beads task trials with ratios of beads of either 85:15 or 80:20 as “easy,” which we used to calculate the percentage of trials that were “easy” for each sample.

Inspection of raw data occasionally revealed instances of typographic errors. When we identified such errors we attempted to infer correct values. When this was not possible we recoded erroneous values as missing data. Participants who had missing data for PDI score, draws to decision, age, or gender were removed prior to analysis. The single exception was the study by Broome et al⁶⁰ that did not code for gender; we included participants from this study in all analyses that did not include gender as a variable. In total, 58 participants (2.5% of participants) were removed due to missing data.

Statistical Analysis

Inspection of PDI scores and mean draws to decision revealed that neither variable was normally distributed, so we used the nonparametric Spearman’s rank correlation coefficient (r _s) for analysis. As per the method advocated by Hedges and Olkin,³¹ we converted r _s scores to their associated Fisher’s z-scores for estimating uncertainty in effect sizes and back-transformed Fisher’s z-scores to r _s scores for interpretation.

Meta-analysis was conducted using the software OpenMEE^66,67 and the R package Metafor.^68,69 We used a random effects model^30,32 and examined heterogeneity in estimates of r _s for the overall group of samples and diagnostic subgroups using the Q statistic⁷⁰ and the I ² index.⁷¹ The Q statistic can be underpowered,⁷² so we paid some attention to I ² indices even in cases of nonsignificant Q statistics. We used the Sidik–Jonkman method for estimating heterogeneity because it provides more accurate estimates than the more widely used DerSimonian–Laird method.⁷³ To examine potential moderators of effect sizes, we used random-effects meta-regression⁷⁴ with a separate meta-regression for each potential moderator. We assessed the evidence for publication bias using a funnel plot and Egger’s regression test for funnel plot asymmetry.^75,76

Results

Meta-Analysis

Figure 1 shows a forest plot for the random-effects meta-analysis. The analysis indicates that there is a negative association between draws to decision and PDI score (r _s = −0.10, 95% CI [−0.17, −0.03], n = 2,237, k = 38; see the dark gray diamond in figure 1). We found moderate heterogeneity (Q-stat P = .03, I ² = 33), which suggests that the precise magnitude of the overall effect size should be interpreted with some degree of caution.

Fig. 1.

Forest plot of random effects meta-analysis showing effect sizes (r _s) for the association between draws to decision and Peters et al Delusions Inventory. The black squares show effect sizes for each sample and are drawn proportional to the relative weighting of each sample in the analysis. The error bars show the 95% CI for each sample. The dark gray diamond shows the overall 95% CI. The light gray diamonds show the 95% CI for each subgroup. The broken line shows the overall mean effect size estimate.

Subgroup Analysis

We found a negative association between draws to decision and PDI when considering the general population subgroup alone (r _s = −0.10, 95% CI [−0.18, −0.02], n = 1,754, k = 23), but not in the current delusions subgroup alone (r _s = −0.12, 95% CI = −0.31, 0.07, n = 262, k = 8), the previous delusions subgroup alone (r _s = 0.05, 95% CI [−0.53, 0.63], n = 54, k = 2), or the anxiety or depression subgroup alone (r _s = −0.04, 95% CI [−0.28, 0.19], n = 76, k = 2; see respective light gray diamonds in figure 1).

We found moderate heterogeneity within the general population subgroup (Q-stat P = .03, I ² = 40) and substantial heterogeneity within the previous delusions subgroup (Q-stat P = .03, I ² = 78), but we did not find statistically significant heterogeneity in the current delusions subgroup (Q-stat P = .11, I ² = 40) or the anxiety or depression subgroup (Q-stat P = .78, I ² = 0).

Meta-Regression

Because there is evidence for heterogeneity in effect sizes we performed exploratory moderator analysis. Figure 2 shows that the number of trials is a statistically significant predictor of r _s (β = 0.07, SE = 0.02; 95% CI [0.02, 0.11]; z = 3.02, P < .01; α = −0.26, SE = 0.06; 95% CI [−0.38, −0.13]; z = −4.09, P < .01). Visual inspection of figure 2 suggests that the 3 samples with 6 trials (all of which came from the same study) might have a large influence on the regression. To test the robustness of this association we re-ran the meta-regression with these 3 samples removed and found that the number of trials is no longer a statistically significant predictor of r _s (β = 0.06, SE = 0.03; 95% CI [0.00, 0.13]; z = 1.92, P = .06; α = −0.25, SE = 0.08; 95% CI [−0.40, −0.10]; z = −3.27, P < .01), which suggests that some caution is warranted when interpreting this association.

Fig. 2.

Regression plot showing the random-effects meta-regression examining the relationship between number of beads task trials and effect sizes. Effect sizes (r _s) are plotted as circles with the size of the circles drawn proportional to the relative weight of the sample in the analysis.

Other potential sample-level moderators were not found to be statistically significant predictors of effect size: mean age of participants (β = 0.00, SE = 0.00; 95% CI [−0.01, 0.01]; z = −0.59, P = .56; α = −0.04, SE = 0.11; 95% CI [−0.25, 0.18]; z = −0.32, P = .75); percentage of females (male = 0, female = 1; β = 0.00, SE = 0.00; 95% CI [0.00, 0.01]; z = 0.84, P = .40; α = −0.19, SE = 0.12; 95% CI [−0.42, 0.04]; z = −1.60, P = .11); and percentage of trials using an easy version of the beads task (β = 0.00, SE = 0.00; 95% CI [−0.00, 0.00]; z = −1.07, P = .28; α = −0.04, SE = 0.06; 95% CI [−0.16, 0.08]; z = −0.71, P = .48).

Publication Bias

Figure 3 shows a funnel plot. A negative effect size is predicted, so publication bias would be expected to manifest as a gap in the bottom right region of the funnel plot. There is no obvious gap here or elsewhere. In addition, the plot appears to be relatively symmetric and Egger’s regression test for funnel plot asymmetry was nonsignificant (z = −0.74, P = .46). Overall, we found no evidence for publication bias. Nevertheless, absence of evidence should be interpreted with some caution, because even when publication bias is present it can be difficult to identify.^76,77

Fig. 3.

Funnel plot showing effect size (r _s) plotted against standard error.

Discussion

Summary of Findings

Overall, these results suggest that people with higher PDI scores tend to request fewer beads before making a decision about which jar beads are being drawn from than people with lower PDI scores. The overall effect size is small, and exhibits moderate heterogeneity. This negative association between PDI scores and draws to decision is present when considering the general population subgroup alone, but not when considering the current delusions subgroup alone. We found no evidence to suggest that effect size estimates have been inflated as a result of publication bias.

Discussion of Findings

Overall, these results provide some provisional support for continuum theories of psychosis^6–12 and cognitive models that implicate the JTC bias in the formation and maintenance of delusions.^17–23 That we did not find evidence for an association in the current delusions subgroup could be considered to be of concern, as cognitive theories aim to account for clinical delusions not mere delusional ideation in the general population. Nonetheless, as figure 1 shows, the confidence intervals for the general population subgroup and the current delusions subgroup overlap substantially and have almost identical means (in fact, the mean for the current delusions subgroup is slightly more negative than the mean for the general population subgroup). We suggest that the low statistical power of subgroup analysis²⁹ is a more plausible interpretation of this counter-intuitive result than proposing that the association is present in the large general population subgroup (n = 1,754; k = 23) but not the small current delusions subgroup (n = 262; k = 8).

Although these results provide evidence for an association between delusional ideation and data gathering, it is important to consider the small effect sizes. One possible interpretation of the small effect sizes is that the JTC bias plays only a minor role in delusion formation and maintenance, despite the many published studies reporting that the JTC bias is more common in those with current delusions than controls.^4,5 It may even be the case that the JTC bias would explain no additional variation in delusional ideation if other important dimensions of individual variation were taken into account. Such a possibility is consistent with one of the larger beads task studies included in our meta-analysis, which found no evidence for an association between paranoia (in the context of paranoid delusions) and jumping to conclusions once the association between paranoia and general cognitive performance had been controlled for.⁴⁷ We anticipate that the upcoming meta-analysis by Taylor et al³⁴ will help clarify because they aim to quantify differences in data gathering between groups with delusions and control groups while exploring a variety of potential moderating variables.

Another possible interpretation of these small effect sizes is that widely used versions of the beads task paradigm might not be well suited to examining the JTC bias. This possibility is consistent with claims that studies that have used the beads task may have significant methodological limitations. First, if a participant asks to see only a small number of beads it is not always clear that they are jumping to conclusions on the basis of insufficient evidence.⁷⁸ In many instances, very few beads need to be drawn for the posterior probability of one of the jars to be very high (eg, when the ratio of colours is 85:15 and the first 2 beads are the same colour, the posterior probability of the corresponding jar is 0.97). Second, evidence from a “graded estimates” variant of the beads task suggests that people with delusions misunderstand task instructions more often than controls.⁷⁹ Third, the beads task is rarely incentivised and motivation might explain some differences in performance.^80,81 We suggest that further progress in determining whether people with delusions jump to conclusions could be made by using meta-analysis to investigate between-group differences in beads task performance,³⁴ exploring new data-gathering paradigms that might overcome methodological limitations of the beads task,^80–86 examining more closely what evidence is available to participants at the point when they decide to stop drawing beads,⁸⁷ controlling for important aspects of individual variation,⁴⁷ and revisiting the original—and methodologically sophisticated—beads task paradigm.¹

Funding

This work was supported by an Australian Research Council (ARC) Centre of Excellence in Cognition and its Disorders grant (grant number CE110001021 with M.C., R.L., and R.M. being Chief or Partner Investigators), salary support to R.L. from an ARC Future Fellowship (FT110100631), and a MQRES PhD scholarship awarded to R.M.R. by Macquarie University.