Table 1.

Top 20 Prioritized Research Questions Including Weighted Research Priority Scores, Criterion Scores, and Average Expert Agreement (AEA)

	Proposed research question	Weighted RPS	Domain	Usefulness	Answerability	Effectiveness	Potential for translation	Effect on diagnostic safety	AEA*
1	How do we better develop the evidence base of diagnostic decision support tools (e.g., differential diagnosis generators, decision support for test selection and interpretation, etc.) in terms of effectiveness and implementation? i.e., how can we effectively integrate diagnostic decision support into clinician and patient workflows?	90.00	Technology	94	81	89	90	82	0.81
2	How can EHRs and patient portals be optimized (through local preferences or EHR vendor changes) to most effectively manage abnormal test results, such as incidental findings or test results that come back after transitions of care (e.g., discharge from ED or hospital)?	88.71	Technology	94	89	84	89	77	0.79
3	What are effective strategies to include nurses and other health professionals in optimizing the diagnostic process and identifying and preventing potential harmful diagnostic situations?	88.42	Teamwork	95	88	86	83	79	0.77
4	How can we best bring expert knowledge about diagnostic test selection and result interpretation to ordering providers at the point of care?	88.42	Teamwork	93	86	89	83	79	0.79
5	How do different forms of health IT and associated information content, information displays and health IT-human interactions impact clinical decision-making and the diagnostic process? Different forms of health IT include EHRs, telehealth, portals, apps. Information content broadly includes decision support, use of coded data and documentation. Information displays includes all types of visualization modalities. Different forms of interactions could include clinician-patient interactions affected by computers, use or scribes.	86.13	Technology	92	83	81	79	83	0.74
6	How do we develop and evaluate performance of diagnostic trigger tools that can be used to identify or prevent diagnostic errors across the care continuum?	85.95	Measurement	91	84	81	84	78	0.71
7	How can systematic feedback be given to providers in different settings/specialties to improve metacognition (including calibration between confidence and accuracy) and improve diagnostic processes and outcomes without increasing over-testing and overdiagnosis?	85.42	Cognition	96	84	79	82	77	0.76
8	How do work system factors such as workload (and work compression) time-pressure and interruptions affect the frequency and types of diagnostic errors?	85.33	Epidemiology	91	82	82	75	84	0.76
9	What types of EHR design and functionality can effectively and efficiently summarize important historical patient context and new clinical findings to facilitate the making of an otherwise unrecognized diagnosis?	83.76	Technology	86	75	81	80	81	0.70
10	Understand how AI can be used effectively to augment diagnostic decision-making, including probabilistic decision-making; identify which AI-based tools and techniques are useful to improve diagnostic accuracy and how AI can be best integrated into the clinician's diagnostic process-related workflow.	83.60	Technology	90	83	79	80	75	0.69
11	What are the effective strategies in which to include patients, families and caregivers in preventing diagnostic errors (e.g., by using patient feedback to increase learning and to create safety nets)?	83.14	Teamwork	88	79	83	78	76	0.68
12	What are the barriers and enablers to effective diagnostic teamwork observed in various situations (e.g., by practice settings, different diagnostic time courses, different team configurations, etc.). How can we leverage methods and theories from cognitive psychology and human factors to examine and support effective teamwork?	82.88	Teamwork	85	76	78	81	80	0.71
13	How do we best use patient input and feedback to identify diagnostic error in a reliable and valid fashion?	82.79	Teamwork	94	79	80	79	72	0.70
14	In what conditions can team-based approaches to diagnosis (such as use of collective intelligence or other methods leveraging distributed models of cognition especially through use of technology), significantly increase diagnostic accuracy in real world clinical settings?	81.54	Teamwork	95	76	81	76	70	0.67
15	How can we use IT-based tools and techniques to better capture, analyze, visualize, represent and share clinical decision making related to the diagnostic process? These would include decision-making processes related to uncertainty, watchful waiting, differential diagnosis, Bayesian reasoning.	80.77	Technology	87	67	79	77	78	0.67
16	What are the most effective methods to leverage existing electronic data to do real time (or quasi “real time”, meaning a clinically meaningful timeframe) measurement of diagnostic error? Provide actionable feedback of diagnostic accuracy at the individual clinician level in “real time”?	80.73	Measurement	91	78	78	74	72	0.66
17	Diagnostic accuracy/expertise depends on experiential knowledge—what are the most effective strategies in medical education for improving experiential knowledge prior to independent practice? Can we jump start the acquisition of experience via simulated diagnostic experiences?	80.00	Cognition	87	79	76	81	67	0.65
18	Can we improve diagnostic safety by facilitating shared decision making in the diagnostic process, i.e., by discussing the risks and benefits of watchful waiting vs. additional diagnostic testing and treatment options?	79.89	Cognition	92	74	78	76	69	0.66
19	How can we effectively use near real time second review considering factors such as case selection (random or systematic), specialty (within specialty or multidisciplinary) to impact calibration, knowledge, and error reduction?	78.91	Measurement	84	85	76	69	72	0.68
20	Are diagnostic errors more or less likely in specific patient population? For example, certain demographics (race/ethnicity), certain socioeconomic or social determinants of health factors or other factors (prison, homelessness, migrant etc.) may lead to disparities with respect to diagnostic delays and errors.	78.81	Epidemiology	87	85	72	77	66	0.65

*The AEA is the Average Expert Agreement, where 0 means there is no agreement and 1 represents full agreement. The closer the number is to 1, the more the experts agreed on the prioritization scores for the question