Table 1. Main features of individual projects.
| RPP | EERP | ML2 | SSRP | Pooled data | |
|---|---|---|---|---|---|
| Replication Study | Ref [12] | Ref [6] | Ref [10] | Ref [7] | |
| Forecasting Study | Ref [19] | Ref [25] | |||
| Field of study | Experimental Psychology | Experimental Economics | Experimental Psychology | Experimental Social Science | |
| Source Journals | JPSP, PS, JEP (2008) | AER, QJE (2011–2014) | Several psychology outlets, including JEP, JPSP, PS (1977–2014) | Science, Nature (2010–2015) | |
| Replicated Findings | 40 | 18 | 24 | 21 | 103 |
| Successful replications | 15 (37.5%) | 11 (61.1%) | 11 (45.8%) | 13 (61.9%) | 51 (49%) |
| Mean beliefs—Prediction Market | 0.556 | 0.751 | 0.644 | 0.634 | 0.627 |
| Correct–Prediction Markets (%) | 28(70%) | 11 (61%) | 18 (75%) | 18 (86%) | 76 (73%) |
| Mean Absolute Error–Prediction Market | 0.431 | 0.414 | 0.354 | 0.303 | 0.384 |
| Mean beliefs—survey | 0.546 | 0.711 | 0.647 | 0.605 | 0.610 |
| Correct—Survey (%) | 23 (58%) | 11 (61%) | 16 (67%) | 18 (86%) | 68 (66%) |
| Spearman Correlation–Prediction Market and Survey beliefs | 0.736 | 0.792 | 0.947 | 0.845 | 0.837 |
| Spearman Correlation–Replication Outcomes and Prediction Market | 0.418 | 0.297 | 0.755 | 0.842 | 0.568 |
| Spearman Correlation–Replication Outcomes and Survey beliefs | 0.243 | 0.516 | 0.731 | 0.760 | 0.557 |
This table contains key characteristics and summaries of the datasets and the pooled data. In calculations of correct forecasts by the prediction market and survey, we interpreted a final price of 0.50 or greater as prediction of a successful replication; if the final price is lower than 0.50, we interpret this as prediction of a failed replication. Overall, the actual replication rate was 49%, indicating that the forecasters were overconfident with the average market price being 0.627. Prediction Markets tend to outperform survey’s when forecasting replication success when considering overall accuracy– 73% compared to 66%.