Estimating the cost of regulating genome edited crops: expert judgment and overconfidence

ABSTRACT

Experts are often called on to inform decision makers with subjective estimates of uncertain events. Their judgment serves as the basis for policy-related decision-making. This paper analyzes survey results used to collect experts’ opinions of the likely cost to bring genome edited crops to market. We also examine the effect of expertise (scientific experts versus social scientists in plant biotechnology) and possible knowledge mis-calibration, both in terms of overconfidence (i.e., when subjective knowledge is inflated) and under-confidence (i.e., when subjective knowledge is deflated), on the estimation of cost involved in the development and commercial release of genome edited crops. We found that the expected costs of genome edited crops are case specific and depend on whether crops will likely be regulated as genetically modified or accepted as conventional varieties and not subject to any regulatory oversight by federal regulators. While cost evaluation of genome edited crops did not vary among scientific and social experts, it did vary among domains of knowledge. Hence, expert’s performance can be described as task-specific in the context of this study.

INTRODUCTION

Climate change and the skyrocketing world population have increased pressure on many, if not all, natural resources. Food security and sustainable agriculture have become significant global challenges. Advances in agriculture biotechnologies offer opportunities to address the burning issues of ensuring food security without destroying environmental resources. ¹ According to Abah et al.²: “Innovative technologies have to be exploited in order to enable sufficient food availability in the future.” The latest of these plants breeding strategies is genome editing, with the best-known example being CRISPR (Clustered Regularly Interspaced Short Palindromic Repeats). These techniques yield ‘customized’ crops by adding favorable traits or deleting undesirable traits using molecular scissors to change the genetic code of the plant (i.e. engineered nucleases). These changes might involve the insertion of small endogenous DNA (genome editing class 1 and 2 site-directed nucleases, SDN, or the introduction of foreign genetic material, SDN3). Owing to their apparent unprecedented accuracy and scope, genome editing tools can be described as qualitatively ‘superior’ to traditional random and untargeted transgenic and conventional mutagenic breeding methods. So far, genome editing has been applied in more than fifty different crops and plants for basic research as a poof-of-concept in addition to several market-oriented traits (enhanced agronomic characteristics, improved food quality, herbicide tolerance, etc.) .³ The scientific community agrees that genome editing tools enable targeted, gene-specific research on more traits and more crop species at lower costs and in shorter times.^4–6

While some genome edited products are already on the market¹ (Cibus’ sulfonylurea (SU) herbicide-tolerant canola, waxy corn with enriched amylopectin, Calyxt high oleic soybean oil known as Calyno^TM7) and others are coming to the market (e.g. fish⁸, CRISPR/Cas9 edited tomatoes), there is little publicly available or accessible information on the costs involved in the overall process of discovering, developing and authorizing a genome edited crop through to commercialization. According to Phillips McDougall⁹, there is currently no accepted standard for the costs and time involved in the development of a new trait as costs vary among companies and among crop species. Using an online survey, we asked international experts in agricultural biotechnology to estimate the cost and time for bringing genome edited products to markets. The cost evaluation is based on experts’ knowledge in the subject, which could be the result of expertise (tacit knowledge acquired through training, skills, and experience) and/or expert judgment (opinions, predictions, estimates) .¹⁰ With increasingly complex technological processes, limited time and scare resources, expert judgment has been recognized as an (if not the only) effective source of good information.^11–13 Expert judgment deemed to be ‘good’ is expected to be well calibrated (i.e. close to reality) and informative (i.e. precise and confident).^11,12 However, it is possible that expert judgment could be biased, mis-calibrated or self-serving.¹⁴ This paper examines the effect of expertise (scientific experts versus social scientists) and knowledge mis-calibration in terms of both over-confidence (i.e., when subjective knowledge is inflated) and under-confidence (i.e., when subjective knowledge is deflated) on the estimation of costs involved in the development and commercial release of genome edited crops. A better understanding of experts’ judgment of confidence might help policymakers arrive at better decisions when setting biotechnology-related policy. The current results are present opinions of experts regarding the scientific, regulatory, and market uncertainty about the development and commercialization of genome edited products.

The remainder of the paper is structured as follows. First, we briefly present a literature review focusing on expert judgment and confidence. Next, we describe the method and survey design, followed by results and discussion. The final section provides conclusions.

EXPERT JUDGMENT AND CONFIDENCE

Expert opinions—usually in the form of subjective probabilistic judgments—are widely used in fields (e.g. science, technology) where empirical data are lacking, and for new, rare, complex or poorly understood problems.^10,15 As experts hold extensive technical or scientific information on certain topics, they are deemed most likely to provide insights into future events. Their judgments (internal beliefs) can help support timely, informed decision-making.¹⁶ However, experts, like other human beings, are known to be subject to biases in decision-making.

People unconsciously use heuristics to make judgments about uncertain events, yet they are subject to several sources of cognitive bias in making assessments, which might lead to systematic errors.¹⁷ One of these sources is over-confidence, usually defined as excessive certainty that one knows the truth.¹⁸ Three types of over-confidence exist in the literature: over-estimation of one’s actual performance, over-placement of one’s performance relative to others, and excessive precision in one’s beliefs—known also as mis-calibration.¹⁹ This paper explores confidence as mis-calibration. To illustrate, an individual who indicates that he is 95% confident in his responses to a list of questions will often realize that less than 95% of his answers actually are correct. Such poor calibration occurs because individuals do not know the boundaries of their knowledge.²⁰ Kahneman²¹ asserts that over-confidence arises because individuals are often blind to their own blindness. On the other hand, proper calibration happens when the confidence level closely matches the accuracy level. Over-confidence appears to be the most ubiquitous bias in studies of calibrated judgments about risks and uncertainties.¹⁵ In this paper, experts are invited to assess the uncertain costs of modern plant biotechnologies and disclose their confidence in their judgments.

While both experts and laymen fall prey to over-confidence, expertise (i.e. being more knowledgeable) does not necessarily reduce over-confidence. There is abundant experimental evidence that the phenomenon is more pronounced for knowledgeable individuals such as medical doctors (e.g.²²), physicists (e.g.²³), economists (e.g.²⁴), and financial professionals (e.g.²⁵). On the other hand, there is little evidence about the differences between experts and novices, specifically whether experts are just over-confident or are more over-confident than others.²⁶ There are a number of studies that challenge the generalization that experts are systematically over-confident, focusing on the methods used.^27,28 It is widely reported that over-confidence measured with binary questions and confidence-range judgments is consistent and substantial (see ref.^29,30)

Experts might be more susceptible to over-confidence because of either information processing bias or unbiased judgmental errors.²⁷ Motivational factors are involved in information search strategies, such that moral or professional responsibility, legal liability or peer credibility might influence expert judgment.³¹ Under-confidence occurs when overly conservative decisions are made by an expert who feels ethically or professionally responsible for the outcomes of his/her predictive judgment (self-protecting). Over-confidence happens when an expert’s response is too certain, in an effort to bolster one’s credibility or as the expert pursues their own self-serving interests (e.g.^32,33)

People tend to be variably over-confident. A battery of previous studies has found that the degree of judgmental over-confidence within general-knowledge questions increases with the task complexity.^29,34,35 In short, over-confidence is more pronounced for hard questions; in contrast, experts often project inappropriately low confidence for easy ones (e.g.^29,36). Thus, the selection of questions might lead to biased responses, as difficult questions might produce spurious over-confidence and their exclusion, under-confidence. However, the complexity of the question cannot be the sole cause of confidence bias as some studies found over-confidence even when asking seemingly easy questions.^37,38 Corrective and frequent feedback—in the form of comprehensive evaluation of the assessor’s responses—was found to play a role in reducing some over-confidence and improving calibration, but not always.^35,39,40

Based on this review, we tested for the effects of expertise and judgment of confidence (i.e. over- and under-confidence) on estimating costs of genome edited crops. We did not formulate explicit hypotheses as findings in the literature are mixed. Our analysis is thus exploratory rather than confirmatory. According to Klayman et al.²⁷ over-confidence depends on how, what and whom you ask.

METHOD

An online survey² was designed to gather data between October 2017 and January 2018 on the cost and time involved in the research and development (R&D) process of genome edited crops. It mainly focused on quantifiable input for which limited or no existing data are available using domain expert elicitation. The survey was emailed to a panel of 523 international experts (scientists, government officials, agribusiness professionals, etc.) with related backgrounds and experiences in biotechnology. The survey instrument is part of a multi-year survey project investigating expert opinions regarding agricultural innovation, particularly the application of new plant technologies such as genome editing. The expert panel was obtained from a contact database that was constructed using emails of participants for a number of conferences on biotechnology organized by the researchers over the past 15 years, and of experts from online searches (e.g. university, research institution, biotech company, and government websites). Recruiting a large panel of international experts online is a challenging task, and this method allowed us to reach a large number of international experts in the field of study.

The survey included two parts (see appendix 1). The first part asked the respondents to offer estimates of the likely cost of developing and commercializing genome edited crops. It collected and evaluated cost (US$ millions, anchored between 0 and 999) and time (values of years anchored between 0 and 99) involved in each of the activity stages in the overall process of discovering, developing and authorizing a genome edited crop for commercialization. Two cases were considered: first, the possibility that genome edited crops would not be regulated as genetically modified (GM), and then the case where genome edited crops are regulated as GM. Participants were also asked to reveal their “subjective” confidence level on a 4-point Likert scale. It is fair to note that we are not presuming that all genome edited crops are going to be regulated either as GM or as non-GM. Results from a previous survey on the regulatory uncertainty of new breeding technologies show that the majority of experts indicate that some genome edited crops should be regulated as GM when they involve gene insertions or substitutions.⁴¹ DNA-free genome edits are usually indistinguishable from natural mutations. We focused on genome editing as the most likely technological innovation in crop breeding because the results of our earlier surveys indicated that these techniques are the most likely for the future of crop improvement.⁴²

The second part of the questionnaire is a calibration test measuring over-confidence. Psychological research offers two techniques to measure calibration: making probability judgments about discrete propositions, and the calibration of probability density functions for uncertain quantities (the fractile method) .²⁹ We measured over-confidence with a number of discrete propositions (see justification below) where subjects were presented with a series of multiple-choice questions. For each question, they were requested to select one answer and prompted to state how confident they were that their answer is correct on a 100-point scale (0% means no confidence and 100% means total confidence). A comprehensive measure of individual over-confidence was constructed to compare the cost-related questions among two groups of experts: those who tend to be over-confident (higher confidence levels) and those who do not (lower confidence levels). The over-confidence test was developed with the following assumptions in mind. First, 18 questions tested two domains/tasks: general knowledge/factual information (nine questions) and science-related knowledge/scientific information (nine questions). Second, as psychological research shows that over-confidence is more pronounced for hard questions, we made use of the balanced to hard-easy effect tests to avoid high levels of under-/over-confidence.^27,29 As recommended by Pulford and Colman⁴³, both sets of questions included three difficulty categories: three easy, three medium and three hard questions.³ Unlike a typical calibration experiment where subjects select one of the two answers, three possible answers to each question were proposed in this study with only one of the choices being correct. The multiple-choice discrete propositions’ task format is less likely to produce substantial over-confidence levels compared to binary questions and subjective confidence intervals.^27,29 When choosing the questions, we tried to avoid regional (and gender) bias: no questions that could be easier for Americans (males) than Europeans (females) were asked and vice versa. A typical example of the questions asked was: “The first crop plant genome sequenced was _____?” Rice, Barley or Wheat (Correct answer: Rice). The order of the questions in each domain was randomized for each participant. Finally, as the survey was conducted via the Internet, participants were asked to answer the questions using their own knowledge and to not check other sources to find answers. They were instructed to guess any answers they did not know.

RESULTS AND ANALYSIS

Ninety-nine participants completed the survey, a response rate of 19%. The sample is dominated by males (69%), aged between 45 and 65 years (70%). Fifty-four percent are from North America, 25% from Europe, 8% from Africa, 5% from Asia, 5% from Oceania and 3% from Central and South America. Sixty-two percent identified themselves as scientific experts, and 38% as social scientists (business managers, lawyers, etc.). Forty percent work for industry, 26% for university, and 20% for government.

Costs of Genome Edited Crops

Panelists reported their estimates of cost in US dollars (77%) or in euros (23%) and of time in years.⁴ Estimates in euros were converted to USD for the analysis using the exchange rate of 1.13 prevailing on March 27, 2019. When respondents were asked about their level of confidence in answering the cost question (Mdn = 2.00; SD = .9), 36% were not confident, 38% were slightly confident, 22% were moderately confident and only 4% were very confident. According to the psychology research on over-confidence, extreme confidence (levels approaching 100%) indicates over-confidence (e.g.^29,34,37). The low confidence can be explained by the nature of the assignment: predicting economic variables such as the cost is a challenging task, especially when various phases are in play. In fact, economic forecasts are often vague and ambiguous.²⁴ The cost and time estimates were adjusted taking into account the confidence level self-declared by the participants. Values were weighted to midpoints of the 4-point Likert scale: .12; .37; .62 and .87 for not confident (0–24%), slightly confident (25–49%), moderately confident (50–74%) and very confident (75–100%), respectively.

The responses were not normally distributed (i.e. tails of the variable’s statistical distribution were not balanced), rather they were skewed to the right (i.e. long tail points right).⁵ The outlying values were provided by those who were not confident or were slightly confident in their answers due to lack of knowledge regarding the costs of genome edited crops in one or more phases of the R&D process (depending on his/her expertise, an expert might better predict upstream costs than downstream costs, and vice versa). Most social science researchers choose to eliminate or alter suspected responses to reduce their deleterious impact on statistical inferences.^44,45 Following this protocol, we removed extreme values for each variable in the dataset. Outliers (observations lying far away from the majority of other data points) in the sample were detected using boxplots with the interquartile range, a measure of the dispersion of values as reported by SPSS Statistics (Statistical Package for the Social Science). Compared to the mean and the standard deviation (SD), the median and quartile range are better statistics as they are less sensitive to outliers.⁴⁶

Mean results are reported in Table 1, and show that experts believe genome edited crops might be able to reach the market at lower costs (US$ 10.5 M) and in a shorter time (5 years) compared with innovations regulated as GM (case 2). The APHIS regulatory database confirms that at least in the US, genome edited varieties with genes knocked out have been summarily judged as not requiring further regulatory oversight; those candidates exempted from the full regulatory process sometimes can be reviewed in a matter of months rather than the years required to review a GM trait. If genome edited varieties are to be regulated as GM, respondents estimated the costs and time could be US$ 24.5 M and 14 years, respectively. Both costs and timeframes for all phases are higher in case 2 compared to case 1. In case 1, the majority of the costs associated with genome edited crops arose from upstream R&D, adding up to US$ 6.5 M (62% of total costs). In case 2, regulatory approval and the commercial launch represent, together, US$ 14 M (57% of total costs). These results were expected. Unlike transgenic crops, site-specific genome edited technologies (SDNs 1 and 2) can avoid “concomitant, undesirable mutations requiring additional steps for their removal via segregation” that would trigger greater oversight.⁴⁷ We note that the R&D timeframe is extended to 6 years when foreign DNA is introduced into the plant genome: costs rise to US$ 9 M (representing 37% of the total cost) and duration of the process elongates (5 years representing 36% of the total time) due to greater regulatory approval. Indeed, DNA-free genome edited crops are judged as not requiring the full GM review process in the US and Japan.⁴⁸ The shorter time and lower cost is the greatest advantage of bringing site-specific genome edited crops to market.

TABLE 1.

Estimated cost and time involved in getting genome edited crops to market (mean values).

	Case 1: Not likely regulated						Case 2: Likely regulated
	Cost			Time			Cost			Time
R&D stages	US$ M	SD	%	Years	SD	%	US$ M	SD	%	Years	SD	%
Research/Discovery/Conception	3	4.11	29	1.5	1.10	30	4	5.45	17	2	1.16	14
Development/Implementation	3.5	4.75	33	1.5	1.27	30	6.5	8.90	27	4	2.03	29
Regulatory activity/Authorization	2	3.25	19	1	.88	20	9	11.17	37	5	3.03	36
Launch/1st commercial sale	2	2.79	19	1	.89	20	5	5.71	20	3	2.22	21
Total	10.5		100	5		100	24.5		100	14		100

For simplicity, costs and time values were rounded to the nearest half or whole.

Several studies estimate the costs related to GM crops, with most reporting financial costs of regulatory compliance, while excluding R&D costs.^49–54 For instance, Kalaitzandonakes et al. (2007) identified compliance costs for herbicide-tolerant maize ranging between US$ 6.2–14.5 million per country. In 2011, Phillips McDougall conducted a consultancy study on the costs and time involved in the discovery and development of a new plant biotechnology-derived trait for Crop Life International. We considered this study the closest—qualitatively—to compare with. While Phillips McDougall’s evaluation involved firms self-identifying their costs of R&D and regulatory compliance, our valuation was based on expert judgment. The earlier study included all pre-launch costs, while our estimation includes additional costs of commercialization. Finally, Phillips McDougall asked respondents to focus on a single event for a single crop, while we did not focus on specific crops or on a particular trait as different genes can be the targets of gene editing tools.

Phillips McDougall⁹ found the mean cost associated with discovery, development, and authorization of a new biotech-derived crop trait introduced between 2008 and 2012 to be US$136 M. The magnitude of this cost is far higher than our findings in cases 1 and 2. A substantial part of the cost (74%) found by Phillips McDougall arises in the upstream research on discovery opportunities and product development. Only a quarter of the cost is allocated to regulatory activity. According to McDougall⁹, the high upstream cost is attributed to the high number of traits (6,204 events) processed by companies between 2008 and 2012, before they converge on a narrower set of commercial events that are targeted for selection. There is evidence that applications for stacked traits took longer to assess (and by implication generated higher costs) compared with single trait category.⁵⁴ Our results in case 2 (excluding stage 4) show that upstream research costs and authorization represent 54% and 46% of the total cost, respectively. As explained earlier, it takes longer to bring a transgenic crop to market compared to its non-GM genome edited counterpart due to additional scientific testing and regulatory procedures. The economic advantage of genome edited crops can be explained by the technological progress in molecular biology and the advanced knowledge in genomics that allow targeted or point-specific changes in the plant genome. Unlike random mutations, such desired accuracy offers a great potential to shorten R&D time, in addition, to lower the cost to produce or improve certain traits, especially in the case of genome edited crops free from foreign DNA.⁵⁵ According to Mohanta et al.⁵⁶: “Genome edited tools have been efficiently used for trait discovery and for the generation of plants with high crop yields and resistance to biotic and abiotic stresses” (p. 399). In addition, a country’s regulatory costs tend to decrease over time as experience is gained (i.e. regulatory cost and time become lower for already approved products) .⁵² Smart et al. (2017) found that the EU approval time for some GM crops decreases by 7–8% yearly.⁵⁴ In other words, regulatory systems governing GM put in place over two decades ago are expected to become more efficient (with the experience from research and commercialization of transgenic crops) in terms of the cost and time of the approval process.^53,57

We acknowledge that the results of our study and those of Phillips McDougall⁹ on the duration of the overall R&D process might be slight over-estimates as the different stages overlap in real time. According to Phillips McDougall, the actual overall time to the commercialization of a single trait is 13 years, on average—which is in line with our finding of 14 years. While our results are aggregated region-wide, we acknowledge potential regional differences in the perceptions of the time and cost to bring a novel biotech product to the market. Table 2 compares the estimated cost and time involved in getting genome edited crops to market between North America (Canada and US) and Europe. For both cases, results show different cost estimates between both regions yet similar timeframes.

TABLE 2.

Estimated cost and time involved in getting genome edited crops to market (mean values): NA vs. Europe.

	Case 1: Not likely regulated				Case 2: Likely regulated
	Cost (US$ M)		Time (Years)		Cost (US$ M)		Time (Years)
R&D stages	NA	Europe	NA	Europe	NA	Europe	NA	Europe
Research/Discovery/Conception	4	1.5	1.5	1.5	5	1	2	1.5
Development/Implementation	5	1.5	1.5	1.5	9	3	4	4
Regulatory activity/Authorization	3	1	1	1	12	7	5	6.5
Launch/1st commercial sale	3	1	1	1	7	3	3	3
Total	15	5	5	5	33	14	14	15

The regulatory systems in the US use a product-based trigger, and substantial equivalence to triage risks while the EU uses a process-based trigger and the precautionary principle. One interesting result comes from Smart et al. (2017) who found that from 1996 to 2015 the approval time estimated was 1,321 days (3.6 years) and 2,467 days (6.75 years) in the EU and US, respectively. Smart et al. assert this somewhat counterintuitive result is due to a rise in applications in the US after 2006, and a shortage of staff; another explanation is that a much narrower range of traits is reviewed in the EU—most first-round reviews (including stacks) are undertaken in the US first, and then only prosecuted in the EU if there is some prospect for them being approved. Most of the more complicated and controversial traits are never submitted to the EU system.

While we tempted to make ‘direct’ comparison between our results and those of Phillips McDougall and Smart et al., we acknowledge that these studies differ in their settings, timeframes, and assumptions (endogenous inconsistencies). For instance, the type of crop, the number of traits included in the application for approval and country’s regulatory system are among the key factors determining the length—and by implication—the cost of the authorization.

Our findings largely support numerous studies agreeing that genome editing might be less expensive than transgenic practices and offer a superior alternative to conventional breeding by random mutagenesis.^58–61 Scientists agree that targeted breeding allows for the introduction of more types of genetic change in a highly specific manner and with greater precision. The game-changing tool CRISPR provides many advantages in terms of simplicity, speed and cost, over other genome edited methods such as ZFN (Zinc Finger Nucleases) and TALEN (Transcription Activator-Like Effector Nucleases) .^58,62,63 While we report an average cost and time for genome edited crops in a general way, we admit that these variables (i.e. time and cost) are likely to differ among genome editing tools as they are applied to different genetic constructs and crop types.

Sample Knowledge Characteristics

Confidence

Confidence has been used as a measure of cognitive performance in the decision sciences to map the correspondence/difference between people’s judgments and reality (e.g. the accuracy in forecasting weather, in predicting the outcomes of sport games or the proportion of correct answers in a test) .⁶⁴ Table 3 compares our subjects’ judgment of confidence for the three difficulty levels of questions. On average, respondents had the highest confidence for answering easy questions—83% for general knowledge questions and 94% for scientific knowledge questions. The average confidence level for the medium questions was 76% and 55%, and for the hard questions—52% and 51%. It appears that disclosed confidence decreases with the difficulty of the question, regardless of the type of knowledge task.

TABLE 3.

Characteristics of the three levels of question difficulty (average values).

	General knowledge			Scientific knowledge
Difficulty level	Easy	Medium	Hard	Easy	Medium	Hard
Confidence level (%)	83	76	52	94	55	51
Accuracy (%)	88	69	56	97	68	57
Bias score*	−5	7	−4	−3	−13	−6
Total	−2			−22

Note: Bias score = average % confidence – average % correct

Accuracy

The complexity of the task has been identified in the literature as one of the factors that can influence the accuracy–confidence relationship. Typically, confidence is positively correlated with choice accuracy.^65,66 According to Pulford and Colman⁴³, total accuracy (i.e. the proportion of correct answers) ranges between 0–33%, 34–66% and 67–100% for hard, medium difficulty and easy questions, respectively. Table 3 shows that the average accuracy for medium-level difficulty questions is marginally within the suggested range whereas accuracy for hard questions is significantly out of range. In fact, 83% of the hard questions in both domains of knowledge have fallen in the category of medium questions.⁶ The advanced knowledge of our expert panel compared to laypeople could explain these findings. In fact, previous studies by Arkes, Christensen³⁵ and Lichtenstein and Fischhoff³⁷ found a negative correlation between accuracy and over-confidence where more knowledgeable subjects exhibited less over-confidence.

The Pearson correlation coefficient, r, was computed to test for the accuracy–confidence relationship. There was a moderate positive correlation between accuracy and confidence in the general knowledge task (r = .428, p < .01) and a weak positive association between the variables in the scientific knowledge task (r = .312, p < .01). Hence, our results show that higher confidence is associated with higher accuracy, yet the magnitude of the association is task-specific (i.e. depends on the nature of the task). Lin and Bier¹⁵ found considerable variation among the average calibration scores not only for questions within different fields but also among questions within a particular field.

Over-Confidence

Calibration can be tested through various measures. We calculated the bias score as a convenient proxy that enables discrimination between under- and over-confidence. The bias score is obtained as the difference between the mean subjective level of confidence and accuracy. A positive bias score represents over-confidence (high confidence), and a negative bias score reflects under-confidence (low confidence). In Table 3, for the medium level questions on general knowledge, participants, on average, assessed the probability that they have chosen the correct answer to be 76%, but only got 69% correct, over-confidence is 7. For the remainder of the questions, the expert panel is under-confident (Total score = −24). The finding is in line with recent empirical studies that show individuals are not generally over-confident.²⁷ In fact, over-confidence depends on how, what and whom you ask.²⁷ When comparing bias scores in both tasks, experts appeared to disclose much less confidence in their domain of expertise (bias score = |22|) than in the general knowledge task (bias score = |2|). As mentioned earlier, experts might make conservative decisions—under uncertainty—for professional reasons related to the outcomes of their predictive judgment.

For each participant, an average bias score was generated for each domain. Based on the sign of the average score (positive or negative), a participant was categorized as generally over-confident or under-confident. Visual inspection of scores reveals that judgments of confidence were not consistent across domains: some panelists were over-confident when answering general knowledge questions and under-confident when answering scientific questions, and vice-versa. This shows that over- and under-confidence vary systematically with the domain of questions.

Table 4 reports contingency table analysis for expertise and judgment of confidence, in both domains of knowledge. Expertise includes two groups: scientific experts (62%) and social scientists (38%). The judgment of confidence includes two clusters: over-confident participants representing 55% of the total sample, and under-confident participants at 45%. Contingency table analysis cross-tabulates the levels of the nominal independent variable (i.e. expertise) with the levels of the dichotomous dependent variable (over-/under-confidence). The cross-tabulation is a joint frequency distribution of cases based on two or more categorical variables that can be analyzed with the Chi-square statistic (χ²), which determines whether the variables are statistically independent or associated. If the calculated p-value of Chi-square is lower than the critical value of .05, then there is evidence against the null hypothesis that the two variables (expertise and the confidence type) are independent. In the domain of general knowledge, scientific experts and social scientists made statistically different judgments of confidence (p-value <.001). The plurality of scientists (45% representing 73% of the 55% over-confident participants) were over-confident while the plurality of social scientists (28% representing 74% of the 45% under-confident participants) were under-confident. There was no evidence that experts diverged with respect to their judgment of confidence in the domain of scientific knowledge (p-value > .05). In fact, a plurality of social (22%) and scientific (33%) experts were overconfident in the scientific task.

TABLE 4.

Judgment of confidence by domain of knowledge among experts.

		General knowledge			Scientific knowledge
Judgment type		Under-confident	Over-confident	Total	Underconfident	Over-confident
Expert group	Scientific	17	45	62	29	33
	Non-scientific	28	10	38	16	22
Total		45	55	100	45	55
Chi-square statistic		χ2 = 19.823; df = 1; p < .001			χ2 = .279; df = 1; p = .597

Effect of Judgment of Confidence on Cost Estimation

In this section, we report differences in estimated costs of genome edited crops among under-confident and over-confident groups for each domain of knowledge. Table 5 displays results within the general knowledge domain. Results show that total cost estimates reported by the under-confident group were higher than those reported by the over-confident group. In case 1 (case 2), under-confident panelists estimated the total cost to get genome edited crops to market to be US$14M (US$ 33.5M), while over-confident respondents offered estimates at nearly half that value: US$ 7.5 (US$ 18M).

TABLE 5.

Estimated costs of genome edited crops among groups within general knowledge domain (mean values).

	Case 1: Not likely regulated				Case 2: Likely regulated
Judgment type	Under-confident		Over-confident		Under-confident		Over-confident
R&D stages	US$ M	SD	US$ M	SD	US$ M	SD	US$ M	SD
Research/Discovery/Conception	4	4.77	2	3.45	5.5	7.18	2.5	3.96
Development/Implementation	4	4.26	3	4.99	9	10.41	5	7.71
Regulatory activity/Authorization	3	3.81	1.5	2.95	13	13.99	7	8.30
Launch/1st commercial sale	3	3.99	1	1.68	6	6.58	3.5	4.89
Total	14		7.5		33.5		18

As panelists are presumably either over-confident or under-confident; a respondent cannot be one of each condition (i.e. a respondent can either be over- or under-confident on his/her answer for a particular question). Therefore, judgment type (expertise type) is an independent variable with independent levels. Thus, a two-sample independent t-test was conducted to compare the means of costs (continuous variable) among groups of under-confident and over-confident (scientific experts and social scientists) respondents.

The independent t-test indicates a significant statistical difference in the mean costs of all the four stages of R&D process in case 2 (p-value <.05). In case 1, only the research (t₍₆₄₎ = 2.142, p = .045.) and commercial launch (t₍₅₆₎ = 2.314, p = .003) stages were statistically different among both groups. These results confirm that judgment of confidence in the general knowledge domain has an effect on cost estimation, as both groups reported different costs of genome edited crops regardless of their regulatory status.

Regarding the scientific knowledge domain results presented in Table 6 show that cost estimates reported by both under-confident and over-confident groups are comparable. Results of t-test were not statistically significant: the p-values for all four stages were greater than .05 in both cases. Unlike in the general knowledge domain, the judgment of confidence in the domain of scientific knowledge did not appear to have an effect on cost estimation, as both groups reported similar costs for developing genome edited crops, regardless of their regulatory status. In other words, overconfidence varies with one’s background/expertise (e.g. a person can have high overconfidence in one domain and low confidence in another domain) which in turn shapes evaluation/decision-making. People were found to be overconfident in their predictions in fields where they have self-declared expertise⁶⁷; that is, knowledge level and overconfidence are positively associated. Our results show that experts offered similar opinions on costs within their own (scientific) domain of knowledge (Table 5), yet their predictions diverged for tasks unrelated to their core expertise (general knowledge domain).

TABLE 6.

Estimated costs of genome edited crops among groups within scientific knowledge domain (mean values).

	Case 1: Not likely regulated				Case 2: Likely regulated
Judgment type	Under-confident		Over-confident		Under-confident		Over-confident
R&D stages	US$ M	SD	US$ M	SD	US$ M	SD	US$ M	SD
Research/Discovery/Conception	2	3.13	3.5	4.58	3.5	6.37	4	4.88
Development/Implementation	3	4.52	3.5	4.96	5	6.79	7	6.96
Regulatory activity/Authorization	2	3.33	2	3.25	9.5	12.46	9	10.51
Launch/1st commercial sale	2	3.85	2	1.74	4	5.58	5	5.80
Total	9		12.5		22		25

Effect of Expertise on Cost Estimation

Tests of the differences in estimated costs of genome edited crops among expert groups are reported in Table 7. Regardless of their field of specialization, experts estimated the costs of genome edited crops to be at least US$ 22M if they were subject to the more intensive reviews of GM events. Scientific experts were more optimistic, pegging the slightly lower total cost of genome edited crop (US$ 8.5M in case 1 and US$22.5M in case 2) compared to social scientists, who estimated costs of US$ 12M in case 1 and US$25M in case 2. Yet, the difference is not statistically significant as shown by the independent t-test yielded a non-significant statistical difference (p-value > .05) for all four R&D stages and for both cases. These results clearly show that expertise did not have an effect on predicting the costs of genome edited crops. Both scientific and social experts converged to similar cost evaluation for both the high and low regulatory cases. This result is in line with some other studies that found experts who perform best on the judgment task are not always with the most experience in their fields.^11,15

TABLE 7.

Estimated costs of genome edited crops among experts (mean values).

	Case 1: Not likely regulated				Case 2: Likely regulated
Expert group	Scientist		Non-scientist		Scientist		Non-scientist
R&D stages	US$ M	SD	US$ M	SD	US$ M	SD	US$ M	SD
Research/Discovery/Conception	2	3.62	4	4.59	3	5.57	4.5	5.21
Development/Implementation	3	5.17	4	3.78	6	8.57	7	9.65
Regulatory activity/Authorization	2	3.49	2	2.72	9	10.71	9	12.29
Launch/1st commercial sale	1.5	2.42	2.5	3.43	4.5	5.73	4.5	5.81
Total	8.5		12.5		22.5		25

Table 8 summarizes key differences in cost evaluation of genome edited crops among domains of knowledge (general knowledge versus scientific knowledge) and among experts (scientific versus social). The type of judgment of confidence (over-/under-confidence) did have a significant statistical effect on expert’s cost estimations when tested in the general knowledge task but not when tested in the scientific knowledge task or against expertise. Hence, expert judgments are domain-specific.

TABLE 8.

Summary of the statistical significance of the results.

		Genome edited crops not likely regulated	Genome edited crops likely regulated
Judgment of confidence	General knowledge domain	Significant at .05	Significant at .05
	Scientific knowledge domain	Insignificant	Insignificant
Expertise		Insignificant	Insignificant

CONCLUSION

One of the objectives of this study was to establish a value for the relative cost of the development and commercial release of genome edited crops. Our valuation is based on a survey of an international panel of experts in plant biotechnology. Predicting economic variables such as costs of novel crops is a tough task especially when several, different and overlapping stages are involved in the process, information is lacking and uncertainty is high. Experts have been shown to predict badly (the so-called process-performance paradox)⁶⁸, partly because they are poor at combining multiple sources of information to come up with a single predictive judgment.^26,69

The results of this study indicate that the overall cost to bring a genome edited crop to market is, on average US$ 10.5M within 5 years if regulated as a conventional crop and US$ 24.5M within 14 years if regulated as GM. If realized, it will be more efficient—cheaper and quicker—to bring DNA-free genome edited crops to market compared to GM genome edited alternatives. Such economic advantage mainly lies in the absence of additional technological and regulatory testing of the foreign genetic material in the genome. Nevertheless, it still appears to be cheaper and faster to produce GM crops using genome editing techniques than traditional transgenic approaches owing to the desired accuracy of genome edited innovations—gained from the advanced knowledge in genomics—and to the accumulated regulatory experience.

While the current estimates are reported at an aggregate level (i.e. combined expert opinion via weighted average estimates of cost and time), we showed in Table 2 that these predicted values are likely to vary by region given the divergent politics and the heterogeneity in national regulatory frameworks governing plant biotechnology. For instance, the steps in the American and EU regulatory approval systems differ in length and type and thus direct (statistical) comparison might not be rationally doable.⁵⁴

Significant statistical differences in the evaluation of the total costs were found with respect to the judgment of confidence within the general knowledge domain but not within the scientific knowledge task or within the expertise. Experts tend to make a similar decision within their field than under different domains.

The experts in our panel hold the belief that if genome edited crop varieties are not regulated as GM crops, their development and commercialization will occur more rapidly and for less cost. This will improve investment as uncertainty will be reduced, ultimately contributing to improving food security as more new crop varieties will be able to reach the market faster than was previously the case. Conversely, our experts believe that if genome edited crop varieties are regulated as GM crops, the cost and time for approval will rise, reducing investment and frustrating improvements in food security.

Appendix 1. Costs of NBTs Survey

Consent

Dear participant,

We appreciate your participation in our sixth quarterly survey that includes questions related to the costs of breeding innovation, in addition to general knowledge questions. The questionnaire is part of a three-year project on risk decision-making regarding NBTs. You have already completed at least one survey with us, and your responses have been invaluable in moving the project forward.

The multi-year survey project is investigating risk preferences among knowledgeable experts regarding innovative technology applications in the agri-food industry. The lead researchers for this project are Dr. Stuart Smyth (stuart.smyth@usask.ca, (306) 966 2929) and Dr. Peter Phillips (peter.phillips@usask.ca, (306) 966 4021). They can be contacted should you have any questions or comments. Any questions regarding your rights as a participant may be addressed to the University of Saskatchewan Research Ethics Office ethics.office@usask.ca; (306) 966–2975. Out of town participants may call toll free (888) 966–2975.

As an expression of our gratitude, we will ensure you are granted access to all publications, reports and press releases prior to their publication.

This survey is hosted by Voxco, a Canadian-owned and managed company whose data is securely stored in Canada. Please consider printing this page for your records.

There are no known risks to participating in this survey; however, as with any online activity, the risk of breach of confidentiality is always possible.

In order to complete this survey, you may be required to answer certain questions; however, you are never obligated to respond and you may withdraw from the survey at any time by closing your internet browser.

By selecting the next and completing this questionnaire, your free and informed consent is implied and indicates that you understand and accept the above conditions of participating in this study.

1. We are interested in the costs and time involved in getting a gene-edited plant to market. We focus on gene editing techniques as previous survey results show they are likely to be the most promising biotechnologies used in the future agricultural development.

   We are interested in the breakdown of investment costs to bring a gene-edited product (crop, plant) to market in two cases: when the product is likely not to be regulated (deemed as conventional variety), and when it is likely to be regulated as genetically modified (GM).

   Could you provide an estimate for all costs associated with the stages below, including the time required to bring a successful product to the market?

   Please provide estimates of costs in US dollar ($ million) or in Euros (€ million) and time in months and years.

1.1. Case 1: Gene-edited plant is not likely to be regulated as genetically modified (GM)

Activity Stage	Cost ($ million, € million)	Time (Months, Years)
Research/Discovery/Conception
Development/Implementation
Regulatory activity/Authorization
Launch/1st commercial sale
Total

• 1.2. How confident/sure are you in your answers?

  • Not confident
  • Slightly confident
  • Moderately confident
  • Very confident

1.3. Case 2: Gene-edited plant is likely to be regulated as genetically modified (GM)

Activity Stage	Cost ($ million, € million)	Time (Months, Years)
Research/Discovery/Conception
Development/Implementation
Regulatory activity/Authorization
Launch/1st commercial sale
Total

• 1.4. How confident/sure are you in your answers?

  • Not confident
  • Slightly confident
  • Moderately confident
  • Very confident

• 2. Below you will be presented with a set of science-related and general knowledge questions.

  • (i) Please choose the correct answer from the three given alternatives. Only one of them is correct. Please use your own knowledge for answering the questions and do not consult other sources such as the Internet, books, etc.
  • (ii) When you have made your choice, we would like to know how sure/confident you are that your answer is correct. Please select a percentage between 0% and 100%.
  • (iii) Please answer all questions, even if you have to guess everything.

    2.1 What is ascorbic acid? Apple vinegar Vitamin C Vitamin A

    How confident are you that your answer is correct? _[dropdown 0%-100% for all] %

    2.2. What falling object is said to have inspired Isaac Newton's theories about gravity? Apple Orange Plum

    How confident are you that your answer is correct? _______ %

    2.3. Which nutrient contains more calories per gram Fat protein alcohol

    How confident are you that your answer is correct? _______ %

    2.4. The first crop plant genome sequenced was _____? Rice Barely Wheat

    How confident are you that your answer is correct? _______ %

    2.5. Golden rice is a transgenic crop with the following improved trait Insect resistance high vitamin A content high lysine content

    How confident are you that your answer is correct? _______ %

    2.6. Which is the most abundant metal on Earth? Iron aluminum copper

    How confident are you that your answer is correct? _______ %

    2.7. Cybrids are: Nuclear hybrids cytological hybrids cytoplasmic hybrids

    How confident are you that your answer is correct? _______ %

    2.8. Which is NOT a plant derived alkaloid? Menthol nicotine codeine

    How confident are you that your answer is correct? _______ %

    2.9. The starch content of potatoes can be increased by using a bacterial gene, known as: sucrose phosphate synthase gene ADP glucose pyrophosphorylase gene polygalactouranase gene

    How confident are you that your answer is correct? _______ %

    2.10 Which enterprise did Bill Gates help create and lead? Intel Microsoft Apple

    How confident are you that your answer is correct? _______ %

    2.11. What is an unknown person known as? stranger ignorant ideologue

    How confident are you that your answer is correct? _______ %

    2.12. Which one is a hot chili sauce? Tabasco Curacao Macao

    How confident are you that your answer is correct? _______ %

    2.13. In which year was the technology company Google founded? 1997 1998 1999

    How confident are you that your answer is correct? _______ %

    2.14. In which country will the 2018 Winter Olympics be held? South Korea Canada France

    How confident are you that your answer is correct? _______ %

    2.15. What is the name of the Greek Goddess of wisdom? Athena Nike Penelope

    How confident are you that your answer is correct? _______ %

    2.16. How many Apollo missions landed men on the moon? 5 6 9

    How confident are you that your answer is correct? _______ %

    2.17. Who is the founder of Snapchat? Dong Nguyen Kevin Systrom Evan Spiegel

    How confident are you that your answer is correct? _______ %

    2.18. Which planet is named after the Roman god of war? Mars Neptune Venus

    How confident are you that your answer is correct? _______ %

    What falling object is said to have inspired Isaac Newton‘s theories about gravity? Apple Orange Plum

    How confident are you that your answer is correct? _______ %

    Which nutrient contains more calories per gram Fat protein alcohol

    How confident are you that your answer is correct? _______ %

    The first crop plant genome sequenced was _____? Rice Barely Wheat

    How confident are you that your answer is correct? _______ %

    Golden rice is a transgenic crop with the following improved trait Insect resistance high vitamin A content high lysine content

    How confident are you that your answer is correct? _______ %

    Which is the most abundant metal on Earth? Iron aluminum copper

    How confident are you that your answer is correct? _______ %

    Cybrids are: Nuclear hybrids cytological hybrids cytoplasmic hybrids

    How confident are you that your answer is correct? _______ %

    Which is NOT a plant-derived alkaloid? Menthol nicotine codeine

    How confident are you that your answer is correct? _______ %

    The starch content of potatoes can be increased by using a bacterial gene, known as: sucrose phosphate synthase gene ADP glucose pyrophosphorylase gene polygalactouranase gene

    How confident are you that your answer is correct? _______ %

    Which enterprise did Bill Gates help create and lead? Intel Microsoft Apple

    How confident are you that your answer is correct? _______ %

    What is an unknown person known as? stranger ignorant ideologue

    How confident are you that your answer is correct? _______ %

    Which one is a hot chili sauce? Tabasco Curacao Macao

    How confident are you that your answer is correct? _______ %

    In which year was the technology company Google founded? 1997 1998 1999

    How confident are you that your answer is correct? _______ %

    In which country will the 2018 Winter Olympics be held? South Korea Canada France

    How confident are you that your answer is correct? _______ %

    What is the name of the Greek Goddess of wisdom? Athena Nike Penelope

    How confident are you that your answer is correct? _______ %

    How many Apollo missions landed men on the moon? 5 6 9

    How confident are you that your answer is correct? _______ %

    Who is the founder of Snapchat? Dong Nguyen Kevin Systrom Evan Spiegel

    How confident are you that your answer is correct? _______ %

    Which planet is named after the Roman god of war? Mars Neptune Venus
    How confident are you that your answer is correct? _______ %

• 3. As some of our panelists might have changed their country of residence and/or their field of work since the time they registered in our project in fall 2015, we would like to ask you a couple of questions.

• 3.1. Where do you currently reside?

  • Africa
  • Asia
  • Europe
  • Central & South America
  • North America
  • Oceania
• 3.2 Do you identify yourself as:

  • A scientist?
  • A non-scientist (manager, lawyer, etc.)?

• 4. Do you have any suggestions feedback on this survey?

• 5. While this questionnaire surveys the cost of NBTs, the next survey will deal with their benefits. Do you have any suggestions or questions you would like to see in the next questionnaire?

Funding Statement

This work was supported by the Canada First Research Excellence Fund.