An ensemble method utilising multiple thinking styles that boosts the wisdom-of-inner-crowd effect

Abstract

Previous studies have demonstrated that individuals can utilize the wisdom of crowds, known as ‘the wisdom of the inner crowd’. This requires answering the same question multiple times and averaging the estimates. Although several methods have been proposed to achieve more accurate estimates, its efficacy remains relatively low. Therefore, this study proposes a method that assembles multiple independent methods to simulate the wisdom of the inner crowd effect. Particularly, our method instructs participants to provide estimates five times. Through a behavioural experiment, we confirmed that our method can produce the wisdom-of-innercrowd effect. Moreover, we found that our method produced more accurate estimates than a method that required participants to estimate five times without specific instructions. Furthermore, mathematical modelling demonstrated that the effectiveness of our method was greater than that of 1.5 persons. In sum, this study proposes a method to improve daily estimates.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-14740-3.

Introduction

In daily life, people often need to estimate unknown events (e.g., the number of people attending a conference or the price of a car). How can people make accurate estimates for these types of questions? One promising approach is that of ‘the wisdom of the crowds’^1–12. In other words, the average estimate of a crowd of individuals yields surprisingly accurate estimates. Researchers have investigated this topic for over 100 years.

However, it is also well known that the wisdom of crowds has an underlying problem: difficulty in collecting estimates from several people. Many studies have addressed this issue. Specifically, they have shown that even an individual could use the wisdom of the crowds (called ‘the wisdom of the inner crowd’^13–33. In these studies, individuals were instructed to produce different estimates (mainly twice) for a single question. In other words, they were expected to produce quasi-crowd estimates. The estimates were then averaged. The results showed that the average estimate was more accurate than the individual estimate (i.e., the first estimate).

Thus, the inner crowd’s wisdom can improve daily life estimates. However, it also has a fundamental problem in that its efficacy is relatively low. Specifically, previous studies^13,16 reported that the average estimate’s accuracy corresponded to only 1.1–1.4 persons’ first estimates. Therefore, this study aims to propose a method that can ‘boost’^33–37 the wisdom-of-inner-crowd effect.

To do so, we combined multiple methods proposed in previous studies. The first refers to using the forgetting power. For a single question, a previous study¹³ gave participants a timespan (two weeks) between two estimates. Subsequently, they discovered that the wisdom of the inner crowd emerged. The second approach uses the power of dialectics. Previous studies^14,19,20 asked participants to consider the opposite in their second estimates and showed that people could utilise the wisdom of the inner crowd (called ‘dialectical bootstrapping’). The third method involves perspective-taking^38,39. It is well known that taking others’ perspectives changes various forms of cognition (e.g., stereotypic biases⁴⁰; preferential values²⁸; and egocentric thinking⁴¹. Based on these findings, previous studies have asked participants to consider others’ perspectives in their second estimates. For accessing the perspectives of others, one study³⁰ used general crowds, and another^29,31,32 used a person who disagreed with the participants. Both previous studies reported that people can produce the wisdom of the inner crowd.

As illustrated above, these methods differ in instructing participants to produce estimates. Therefore, it seems possible that, by combining these methods, an individual can boost the wisdom-of-inner-crowd effect. The purpose of this study was to test this hypothesis.

Note that, there is only one previous study showing that ensembles multiple methods could enhance the wisdom of the inner crowd, to the best of our knowledge⁴². The summary of the study is as follows: Participants performed correlation assessments (e.g., the height and weight of male students, SAT section scores). Six methods were used for the assessments. For example, one method focused on the Strength of Relationship (e.g., ‘How would you characterize the strength and nature of the relationship between height and weight for male MBA students?’), while another involved estimating the Correlation (e.g., ‘What would you estimate the correlation between height and weight for male MBA students to be?’). They found that adding both experts and methods could improve accuracy.

The important point for our study is that the previous study⁴² has shown that even a single person can improve accuracy by combining these methods—that is, through the wisdom-of-inner-crowd effect by an ensemble method. Thus, these two studies share similar findings. However, there are key differences between our study and the previous one: The task in the previous study⁴² was limited to correlation assessments. In contrast, our method can be applied to all types of estimation tasks in which participants respond in percentages.

Overview of the experiments

The primary objective of this research is to propose and validate an ensemble method designed to enhance the accuracy of individual estimates by leveraging ‘the wisdom of the crowds’. In a pilot study, we initially examined whether averaging intuitive and deliberative judgments from the same individual could improve estimation accuracy compared to using either judgment alone. This provided preliminary evidence that combining distinct cognitive approaches might be beneficial.

Building upon this pilot, the subsequent experiment systematically tested our proposed ensemble method in a controlled setting. This ensemble method integrates multiple cognitive strategies: intuitive judgment, deliberative reasoning, dialectical thinking, general crowd perspective, and the perspective of a disagreeing individual. Results from this experiment robustly supported the effectiveness of our ensemble method, demonstrating significantly greater accuracy compared to repeated judgments without specific instructions.

Pilot study

Along with the above methods, this study also proposes a new method. For a single question, the method makes people think intuitively (called ‘Intuition’; for the full instruction, see Table 1) in the first estimate and think deliberately in the second estimate (‘Deliberation’). The two estimates are then averaged (see S1 in the Supplementary Information) for optimal weighting.

Table 1.

Type of estimate and detailed instructions.

Type of estimate	Instruction
Intuition	Please do not think deliberately. Answer quickly what comes to mind intuitively. Please answer within 8 s.
Deliberation	Please ignore your intuition. Think deliberately before answering. Utilize your knowledge and experience, while being aware of the basis of the estimate. There is no time limit. Please use enough time to think about the following question.
Dialectic	First, assume that your last estimate is off the mark. Second, think about a few reasons why that could be. Which assumptions and considerations could have been wrong? Third, what do these new considerations imply? Was the second estimate too high or too low? Fourth, based on this new perspective, make an alternative estimate. (The computer display shows the second estimate).
General crowd’s perspective	How do you think people, in general, estimate the following question? Think and answer how people, in general, estimate this. Please do not answer you own estimate.
Disagree-other’s perspective	Now, picture a friend whose views and opinions are very different from yours. To illustrate, when discussing politics, societies, and daily affairs, you often find yourself disagreeing on various issues. How would he or she answer the following six questions? Answer these questions now as this friend. Please do not answer your own estimates.

This method is based on the findings related to rule selection in judgment⁴⁰. Researchers discuss judgments that involve multiple rules and suggest that successful judgment requires sufficient processing capacity. For example, rule selection depends on factors such as processing motivation^43,44 and mental fatigue⁴⁵. Importantly, the need for cognition⁴⁶ and cognitive closure^47–49 also influence rule selection. In our context, intuitive and deliberate judgments may engage in different rule selection processes, resulting in non-redundant errors, although our estimation task did not involve explicit rules.

Only the Dialectic estimate required the second (i.e., Deliberation) estimate to be displayed, while the other estimates did not. All instructions were translated into Japanese.

In the Pilot study, we examined the effectiveness of this method for the general knowledge questions (Table 2; see Methods section for more detail). Participants made an estimate in each of Intuition and Deliberation for each question. We calculated MSE (Mean Squared Error) for Intuition, Deliberation, and the average of the two estimates (as for analysis of the Mean Absolute Error, see S4 in the Supplementary Information). We found that the average of the two estimates was significantly more accurate than the Intuition estimate (t = 2.97, p = 0.0084, Cohen’s d = 0.38 [0.73, 0.03]; Fig. 1). No significant effects were observed between Intuition and Deliberation (t = 0.67; p = 0.78, Cohen’s d = 0.06 [0.40, − 0.29])

Table 2.

Questions and correct answers used in the experiments.

Number	Question	Answer (%)
1	What percent of the world’s roads are in India?	9.7
2	What percent of the world’s telephone lines are in China, USA, or the European Union?	52.0
3	Saudi Arabia consumes what percentage of the oil it produces?	72.1
4	What percent of the world’s population is between 15 and 64 years old?	65.2
5	What percent of the world’s population is Christian?	31.4
6	What percent of the worldwide labour force works in the service sector?	50.6
7	What percent of the worldwide gross domestic product (GDP) is re-invested?	25.2
8	What percentage of Japanese adult males smoke?	27.1
9	What percentage of Japanese households have a fixed-line phone?	69.0
10	What percentage of the world’s countries have a higher life expectancy than the United States?	22.6

Fig. 1.

Results of pilot study. Average indicates the average of the estimates in Intuition and Deliberation. As the figure shows, the estimate in average was (marginally) significantly accurate compared to those in Intuition and Deliberation.

Although we cannot find the significant effects between the average of the two estimates and the Deliberation estimate, (t = 3.75, p = 0.054), we also found a small effect size which indicated the accuracy of the average of the two estimates (Cohen’s d = 0.35 [0.70, 0.00]).

Consequently, we can consider this method as one that may lead to accurate estimates. Therefore, combining the estimates based on Intuition and Deliberation with other estimates (including one’s own estimate) could boost the wisdom-of-inner-crowd effect.

Thus, we found that the average of the Intuition and Deliberation estimates was significantly more accurate than the Intuition estimate. This experiment is related to the study by Keck and Tang¹⁰, wherein participants were instructed to provide estimates either intuitively or deliberately. Their findings revealed that the combination of intuition and deliberation enhances the wisdom-of-crowds effect; In particular, the group in which half of the participants thought intuitively and the other half thought deliberately recorded higher accuracy than the groups in which all the participants thought intuitively or deliberately. Therefore, the Pilot study partially replicated the results of Keck and Tang¹⁰ at an individual level (but see also the Experiment section).

Note that we can also connect this study to previous studies on social circles^50–54. These studies indicate that by using an individual’s knowledge of their social circles (e.g. friends), we can improve predictions such as those referring to the results of a political election⁵³. We can assume that a prediction based on knowledge of the social circles is similar to the estimate from different perspectives. However, the aims of the studies were different. In contrast to previous research, our method aims to obtain an accurate average estimate. In other words, for our method, the estimates are not necessarily accurate.

We checked all the answers on 2022/10/27. We used the answers in The CIA World Factbook⁵⁵ World Bank Open Data⁵⁶ and the data from Japan’s Ministry of Internal Affairs and Communications⁵⁷ based on previous studies^19,30. The Pilot study used all the questions. The Experiment used Questions 2, 3, 4, 5, 6, and 10. All questions were translated into Japanese.

Experiment

Based on these findings, we propose a new method that combines the methods above. This method consists of making five estimates in response to a single question (Fig. 2; Table 1): (1) making an estimate intuitively, (2) making an estimate deliberately, (3) considering the opposite (i.e. dialectical bootstrapping), (4) taking the general crowd’s perspective, and (5) taking the disagree-other’s perspective. The estimates are then averaged (for optimal weighting, see S2 in the Supplementary Information). Because this study aimed to propose a method that could be used for everyday estimation, we excluded methods that required a two-week timespan¹³.

Fig. 2.

An illustration of our method. For a single question, a participant estimates five times. The order of the estimates is shown in this figure.

Importantly, previous studies have attempted to develop methods that consist of making five estimates for a single question. In Rauhut and Lorenz¹⁶, participants made five estimates without specific instructions. In Fujisaki et al.³⁰, participants first provided their estimates and then four estimates from the perspective of general crowds. However, the results indicated that these methods were ineffective. In particular, the average of the five estimates was not more accurate than that of the first two estimates.

Therefore, the key question in this research is whether our method (hereafter, ‘Ensemble method’) could produce accurate averaged estimates. In the following sections, through a behavioural experiment, we confirm that the Ensemble method can produce estimates whose accuracy increases monotonically (i.e., over the two estimates). Moreover, we compared the Ensemble method condition with the control condition, which asked participants to estimate five times without specific instruction (‘Repeated condition’). We found that the Ensemble method was more effective than the method used in the control condition.

The experimental settings were similar to those of meta-prediction methods^7,58–61 which instructed participants to predict what others would predict. For example, Prelec et al.⁷ examined this method. In particular, they proposed an algorithm that selected the answer that was more popular than people had predicted. The results showed that this method can correct the bias in crowds and enhance the wisdom-of-crowds effect. As illustrated above, the proposed method differs from meta-prediction methods in several respects. First, our method aims to improve an individual’s (not a crowd’s) estimate. Second, our method averages the estimates. Nevertheless, we assume that our method and the meta-prediction methods together demonstrate that we can enhance the wisdom of (the inner) crowd by regulating people’s thinking.

Note that we conducted mixed-effects analysis for the statistical test (including the analysis in the Supplementary Information). We selected the best model and computed all the statistical values using the step() function in R for the full model with random participants and stimulus (i.e., question) intercepts (see ‘mixed-effects analysis’ section in Methods).

Results

Main results

Figure 3 presents the results of this analysis. The mean squared error (MSE) was used as an index of estimate accuracy (as for Mean Absolute Error, see S4 in the Supplementary Information). We then calculated the MSE for each participant for all questions. To do so, we averaged the estimates as follows: Let us define the correct answer as θ. For example, when the number of estimates was three, we calculated the MSE as follows:

Fig. 3.

Results of experiment. When the number of estimates was 1, the Ensemble method condition had larger MSE than the repeated condition. However, the larger the number of estimates, the smaller the MSE in the Ensemble method condition. In contrast, the MSE in the Repeated condition remained flat. As a result, the Ensemble method condition had a smaller MSE than the Repeated condition when the number of estimates was 5 (p = 0.016; comparing the average estimates when the number of estimates was 5).

Subsequently, we manipulated the number of estimates from one to five. As shown in Fig. 3, the MSE in the Repeated condition remained flat, as reported in previous studies^16,30. In other words, the accuracy did not tend to increase even if the number of estimates increased. In contrast, in the Ensemble method condition, the larger the number of estimates, the more the MSE decreased, although the slope tended to be slightly attenuated.

First, we confirmed whether the Ensemble method condition reproduced the wisdom of the inner crowd. We could assume the first estimate in the Repeated condition as the participants’ ‘own estimate’. Therefore, we used this estimate in our analysis. We found that the MSE of the average of the five estimates in the Ensemble method condition was lower than that of the first estimate in the Repeated condition (t = 2.19, p = 0.029, Cohen’s d = 0.54 [1.00, 0.073]). The results suggest that the Ensemble method could elicit the wisdom of the inner crowd.

Second, and more importantly, the MSE of the average of the five estimates in the Ensemble method condition was lower than that of the average of the five estimates in the Repeated condition (t = 2.51, p = 0.012, Cohen’s d = 0.59 [1.06, 0.13]). Thus, the Ensemble method condition was more effective than the Repeated condition.

Moreover, the results showed that the Ensemble method produced estimates with monotonically increasing accuracy. We applied the Page’s trend test and found that the MSE in the Ensemble method condition decreased monotonically as the number of estimates increased (z = -5.15, p < 0.001). By contrast, we confirmed that the MSE in the Repeated condition neither increased nor decreased monotonically (z = 1.09, p = 0.28).

Comparison with the wisdom of (the outer) crowds

Further analyses were performed to address the effectiveness of the Ensemble method. In particular, we compared the Ensemble method condition (and the Repeated condition) with the wisdom of the crowds. In the Repeated condition, participants first provided their own estimates. Therefore, to collect these estimates, we got the wisdom-crowds-effect (in a general sense) (hereafter called ‘Repeated-first group’).

We fitted a nonlinear mixed-effects model using Bayesian parameter estimation (for more details, see S10 in the Supplementary Information). We assumed that the relationship between the number of estimates (T) and MSE could be represented by a hyperbola as follows:

where a represents the magnitude of the wisdom-of-inner-crowd effect and b represents the error when the number of estimates is infinite.

Figure 4 presents the results of this analysis. In the Repeated-first group, the larger the group size, the smaller the MSE. Thus, we confirmed that the wisdom-of-crowds effect emerged in the Repeated-first group. Compared to the Repeated-first group, the Ensemble method condition had a more gradual slope. In other words, the Ensemble method condition had a weaker wisdom-of-crowds effect than the Repeated-first group.

Fig. 4.

Results of the model fitting. The points indicate actual values and the solid lines represent the estimated hyperbolas. To compute T_T, we first projected the MSE in the Ensemble method condition onto the hyperbola of the Repeated-first group. Subsequently, we observed the x value and regarded it as T_T. To compute T_T, we used parameter b since b represents the MSE when the number of estimates was infinite. Subsequently, we projected b onto the hyperbola of the Repeated-first group and observed T_∞.

Subsequently, we performed a quantitative comparison. We used T_T, which represented the number of people in the Ensemble method condition corresponding to the Repeated-first group. For example, when T_T = 1.3, the Ensemble method condition corresponded to 1.3 persons in the Repeated-first group.

Table 3 presents the results of the analysis. When T was 5, T₅ was over 1.5 (i.e., 1.51). In other words, the effect of the Ensemble method was larger than 1.5 persons of the Repeated-first group. To the best of our knowledge, this value is larger than that of the method tested in previous studies, especially those focusing on the general knowledge question^13,16. However, even if the number of estimates was infinite, T_T could not exceed two in the Repeated-first group. Thus, the effectiveness and limitations of the Ensemble method can be observed.

Table 3.

Results of T_T.

	Repeated	Ensemble method
T 2	1.02	1.18
T 3	0.99	1.33
T 4	0.99	1.40
T 5	0.96	1.51
T ∞	0.97	1.90

In the Ensemble method condition, T_T was over 1.5. On the contrary, in the Repeated condition, the values were around 1 irrespective of T_T.

Decomposition of the error

How does the Ensemble method condition produce more accurate estimates than the Repeat condition? It is well known that the wisdom-of-crowd effect can be represented mathematically⁵. Note that we translated the equation into our context (i.e., the wisdom of the inner crowd). Let us define Ei as the estimate of group member i, < Ei > as its average over an individual’s estimates, and θ as the correct answer. Then, the equation is:

Here, we refer to the left side of the equation as the collective error, which indicates the MSE. Therefore, a lower collective error value indicates an accurate estimate. We refer to the first term on the right-hand side as the expected squared error. A lower value of the expected squared error represents an accurate estimate. The second term on the right side represents diversity. As the equation shows, higher diversity leads to better collective performance.

Table 4 presents the results of the analysis. As mentioned above, the Ensemble method condition showed a lower collective error than the Repeated condition. Importantly, the Ensemble method condition showed a higher expected squared error than the Repeated condition. In other words, the estimate in the Ensemble method condition tended to be less accurate than that in the Repeated condition (see also S6 in the Supplementary Information). However, the Ensemble method condition had significantly larger diversity than the Repeated condition (see S7 in the Supplementary Information). Consequently, the Ensemble method condition had a lower collective error (i.e., better performance) than the Repeated condition.

Table 4.

Results of the decomposition of the error. We conducted bootstrapping⁶² based on 10,000 sampling with replacement.

	Collective error	Expected squared error	Diversity
Ensemble method	225.8 [184.9, 270.4]	438.8 [391.7, 486.7]	212.9 [202.8, 223.3]
Repeated	315.0 [259.9, 374.2]	359.0 [303.1, 419.4]	43.9 [39.5, 48.6]
Repeated-first	108.2 [90.2, 127.0]	300.3 [275.4, 325.8]	192.0 [183.8, 200.2]

Table 4 also displays the results of the Repeated-first group analysis. For this group, we defined Ei as the first estimate of group member i and < Ei > as its average overall group members. As shown in the table, the Repeated-first group exhibited a lower expected squared error than the Ensemble method. In addition, the Repeated-first group had approximately the same diversity as the Ensemble method. As a result, the Ensemble method was not more effective than the Repeated-first group.

Discussion

This study proposes a method that boosts the wisdom-of-inner-crowd effect. Our method requires participants to provide five estimates in response to a question:1) making an estimate intuitively (Intuition); 2) making an estimate deliberately (Deliberation); 3) considering the opposite (Dialectic); 4) taking the general crowds perspective (General crowd’s perspective); and 5) taking the disagree-other’s perspective (Disagree-other’s perspective). It then averaged the five estimates.

We first confirmed that our method recorded higher accuracy than the first estimate and the average of the five estimates in the control condition (i.e. the Repeated condition). Moreover, the results show that our method produced estimates whose accuracy increased monotonically with the increasing number of estimates. Furthermore, through mathematical modelling, we found that the estimation accuracy of our method was higher than 1.5 persons’ estimates.

We do not claim that this is the only method that boosts the wisdom-of-inner-crowd effect by combining multiple methods.

We conducted a post-hoc analysis of the Intuition-Deliberation method. Specifically, we compared the reduction of MSE from the first to second guesses between the Ensemble and Repeated conditions. The results showed that the Ensemble condition had a larger reduction in MSE compared to the Repeated condition (t = 3.148, p = 0.002, Cohen’s d = 0.71 [1.19, − 0.23]). However, when comparing the MSE obtained from the second guess, we found no significant difference between the conditions (t = 0.85, p = 0.40, Cohen’s d = 0.1 [0.57, − 0.36]), although the average MSE was lower in the Ensemble condition (270.09) than in the Repeated condition (301.06). In other words, there is no clear evidence that combining Intuition and Deliberation yields more accurate results than simply having individuals provide two repeated estimates. Thus, regarding the replication of Keck and Tang’s¹⁰ findings, our results constitute only a partial replication, and further research on this issue is warranted.

Therefore, in the future, it would be important to explore other methods based on the power of combining. Note that all combinations of the five estimates (including instructions, order, and time gaps) should be explored, but here we will give one example, for five estimations: (1) first guess, (2) dialectical reasoning, (3) general crowd estimation, (4) disagree-other, and (5) a last guess. Between the initial guess and the final guess, a moderate timespan would be allowed, as tested by Vul and Pashler¹³. This approach can be regarded as a timespan-based ensemble method.

Another promising approach is to encourage participants to provide different estimates directly. For example, through the anchoring paradigm, a previous study⁶³ successfully prompted participants to give lower and higher estimates, thereby increasing the diversity of estimates and enhancing the wisdom-of-crowds effect. Based on these findings, we could potentially improve the wisdom-of-inner-crowd effect using similar instructions.

Since Vul and Pashler¹³ and Herzog and Hertwig¹⁴ demonstrated that individuals could use the wisdom of crowds, many studies have focused on this. However, its shortcomings have been highlighted. Rauhut and Lorenz¹⁶ pointed out that people cannot produce accurate estimates by increasing the number of estimates. Nevertheless, for over a decade, effective methods to overcome this shortcoming have not been proposed. In this paper, we propose an effective method that combines multiple methods proposed in previous studies. By utilising this method (i.e. the Ensemble method), we can significantly enhance the accuracy of our estimates.

Limitations of the study

Finally, we describe the limitations of this study. We developed the Ensemble method, especially the fixed order of the five estimates, for the following reasons. First, we could not set the Dialectic as the first estimate method because it would make an individual ‘re-consider’ the previous estimate. Second, we considered that we should set the Intuition as the first estimate because it was the estimate by which an individual answered what came to mind first. However, we do not claim that the order of the estimates in our method is the only one which can boost the wisdom-of-inner-crowd effect. Therefore, further studies using this methodology should be conducted. For example, we excluded the timespan method proposed by Vul and Pashler¹³ from our method because it required participants to commit for a long time (two weeks). Thus, by adding the timespan method to our method, an individual may enhance the wisdom-of-inner-crowd effect.

Moreover, in the Experiment, we compared our method with a control condition (i.e. Repeated condition) that did not include specific instructions. However, we did not directly compare our method with those used in previous studies^13,14,30. Therefore, further studies comparing these methods are warranted.

Another limitation of this study is the type of questions. We used questions with the percentage of correct answers based on previous studies^{13,17,18,22,29,30}. However, other types of questions exist. For example, some studies used the years of historical events^14,17 whereas others used numerical estimation tasks^16,17,21,29. A review study¹⁸ reported that the wisdom-of-inner-crowd effect was maintained across different types of questions. However, the manner in which our method works for other types of problems remains unclear. Notably, our method may also enhance the accuracy of choice tasks. Because it requires participants to answer five times, it is possible that aggregation rules, such as majority rules, work effectively. For example, in the binary choice task, even if Intuition and Deliberation were incorrect, we could perform accurate inference by majority rule (i.e. when Dialectic, General crowd’s perspective, and Disagree-other perspective were correct). Further research should be conducted to generalise the findings of this study.

Moreover, there are several methodological limitations that must be acknowledged. First, participants were not provided with accuracy-based incentives, and the descriptions of these incentives were inconsistently applied across different conditions. Future research should explicitly test whether accuracy incentives influence the effectiveness of the Ensemble method, ensuring that such incentives are consistently described and uniformly implemented. Second, the Experiment introduced a 30-minute interval between two rounds of estimation in the Ensemble condition, whereas this interval was not included in the Repeated condition. This discrepancy introduces a potential confound and should be addressed in future studies by standardizing the timing of estimations across all experimental conditions.

Additionally, another promising approach to further enhance the wisdom-of-inner-crowd effect is to integrate new methods into our existing approach. In our experiment, the performance of our method did not surpass the accuracy achieved by averaging two separate individuals’ estimates. Specifically, even if an infinite number of estimates were combined using our approach, the accuracy (T_T) would not exceed that of two individuals in the Repeated-first group. Recently, new methods⁶⁴ have been proposed that effectively boost the wisdom of the inner crowd. Incorporating these innovative approaches into our current method might enable us to achieve accuracy levels that surpass those obtained from averaging two distinct individuals’ estimates.

Methods

Two experiments were conducted using Qualtrics software. All the participants provided informed consent before participating in the study. The experimental protocol was approved by the Research Ethics Committee of University of Tokyo and was conducted in accordance with the 2013 version of the Declaration of Helsinki.

Details of the Pilot study

Participants: The participants were 64 undergraduate and graduate students (24 women, 39 men, and one did not want to respond; M_age = 21.25 and SD_age = 2.24). After the experiment, they received a flat fee of 1,000 Japanese yen (approximately $9.17 at the currency rate at the time) for participation.

Stimulus: Based on previous studies^18,30, we prepared ten questions about general knowledge (Table 2).

Procedure: We set only one condition in this experiment, in which all participants provided two estimates for each question (Sets 1–2).

In Set 1, participants answered all ten questions intuitively. In this set, we also set a time limit of eight seconds, based on previous studies^65,66 that manipulated participants’ thinking styles. After answering each question, participants rated their confidence (see S3 for Supplementary Information). Between Sets 1 and 2, we set a 30-minute time interval because of the experimental design. During this period, the participants performed an irrelevant task. In this task, participants were instructed to make a binary choice as for consumer products.

In Set 2, participants answered all ten questions deliberately. In this set, we did not set any time limits. After answering all the questions, participants answered questions about their thinking styles in this set.

We randomised the order of the questions for each participant. The randomised order of the questions remained constant across the (two) sets.

Details of the Experiment

Participants: The participants were 76 Japanese undergraduate students. In this experiment, we set two conditions: the Ensemble method condition and the Repeated condition. Participants were randomly assigned to one of the two conditions (Ensemble method condition: n = 38, M_age = 19.58, SD_age = 0.86, 24 women, 14 men; Repeated condition: n = 38, M_age =19.74, SD_age = 0.92, 23 women, 14 men, and one did not want to respond).

Stimulus: We prepared six questions on general knowledge (Table 2) based on those used in the Pilot study.

Procedure: In the Ensemble method condition, the participants answered each question five times based on the instructions. They performed Intuition, Deliberation, Dialectic, General crowd’s perspective, and Disagree-others’ perspective estimates in this order. We randomised the order of the questions for each participant. Across the five estimates, the randomised order of the questions remained constant. Note that in the Ensemble method condition, we set a 30-min time interval between the Dialectic and General crowd conditions. During this period, the participants performed an irrelevant task. In this task, participants were instructed to make binary choices related to consumer products.

This experiment did not include a participation fee because it was conducted as part of a psychology class. In the Repeated condition, based on the previous study, we made participants imagine that this experiment would reward themselves depending on the accuracy of each estimate (i.e., not averaged estimates). Note that, at the start of the experiment, we explicitly informed participants that the payment was only hypothetical; they did not actually get paid. In the Ensemble condition, we did not tell participants this because they should complete the estimate based on the instruction (i.e., Dialectic) as mentioned above.

Based on the instructions, the participants answered each question five times. We randomised the order of the questions for each participant. Across the five estimates, the randomised order of the questions remained constant. Note that in the Repeated condition, participants performed the same irrelevant task as in the Ensemble method condition for 30 min after completing the estimation task.

Mixed-effects analysis

All mixed-effects analyses (see also previous studies²⁹⁾ were conducted using the R(4.1.1) packages lme4 and lmerTest. We selected the best model and computed all the statistical values using the step() function in R for the full model with random participants and stimulus intercepts⁶⁷. All multiple comparisons were performed using the R packages lsmeans and pbkrtest⁶⁸.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

IF, LY, and KU developed the study concept and contributed to the study design. LY and YT collected data. IF, LY, and YK analysed the data. IF wrote the manuscript, with feedback from YK and KU. KU and YK won funding.

Data availability

The R-code and the three datasets analysed during the current study are available in the Mendeley Data: https://data.mendeley.com/datasets/yx7xscnxg8/1.

Declarations

Competing interests

The authors declare no competing interests.