Objectives
A lack of consensus around definitions and reporting standards for diagnostic errors limits the extent to which healthcare organizations can aggregate, analyze, share, and learn from these events. In response to this problem, the Agency for Healthcare Research and Quality (AHRQ) began the development of the Common Formats for Event Reporting for Diagnostic Safety Events (CFER-DS). We conducted a usability assessment of the draft CFER-DS to inform future revision and implementation.
Methods
We recruited a purposive sample of quality and safety personnel working in 8 U.S. healthcare organizations. Participants were invited to use the CFER-DS to simulate reporting for a minimum of 5 cases of diagnostic safety events and then provide written and verbal qualitative feedback. Analysis focused on participants’ perceptions of content validity, ease of use, and potential for implementation.
Results
Estimated completion time was 30 to 90 minutes per event. Participants shared generally positive feedback about content coverage and item clarity but identified reporter burden as a potential concern. Participants also identified opportunities to clarify several conceptual definitions, ensure applicability across different care settings, and develop guidance to operationalize use of CFER-DS. Findings led to refinement of content and supplementary materials to facilitate implementation.
Conclusions
Standardized definitions of diagnostic safety events and reporting standards for contextual information and contributing factors can help capture and analyze diagnostic safety events. In addition to usability testing, additional feedback from the field will ensure that AHRQ’s CFER-DS is useful to a broad range of users for learning and safety improvement.
Key Words: diagnostic safety, usability testing, patient safety, healthcare quality improvement, measurement
Diagnostic errors are common and often represent learning opportunities to improve the safety and quality of healthcare delivery.1 Current understanding of diagnostic errors is based largely on single-institution studies, and national analysis of diagnostic safety events lags far behind other types of patient safety events. A lack of consensus around definitions and reporting standards for diagnostic errors limits the extent to which healthcare organizations can aggregate, analyze, share, and learn from these events.2
Existing tools and taxonomies used for analysis of diagnostic safety events3–6 have been used in research and clinical settings but were not primarily designed for event reporting and data aggregation across organizations. Recent initiatives such as the Primary-Care Research in Diagnosis Errors Learning Network demonstrate the value of collecting and sharing deidentified diagnostic error cases,7 but to our knowledge, there are no formal initiatives that collect structured diagnostic safety data suitable for national learning and improvement efforts.
More than a decade ago, as authorized by the Patient Safety and Quality Improvement Act of 2005 (Patient Safety Act), the Agency for Healthcare Research and Quality (AHRQ) began creating “Common Formats for Event Reporting” (CFER). The AHRQ CFER provides common definitions and reporting formats to facilitate the reporting, aggregation, and analysis of patient safety event data for learning and improvement. The AHRQ makes the CFER specifications and tools available in the public domain to facilitate their widespread adoption and implementation. Any provider may use the AHRQ Common Formats, but to take advantage of the Patient Safety Act’s Federal privilege and confidentiality protections, a provider must choose to work with a federally listed Patient Safety Organization (PSO) and develop the information as patient safety work product. The national Network of Patient Safety Databases, which may only receive nonidentified patient safety data, currently accepts data voluntarily submitted by federally listed PSOs.8,9
In 2019 the AHRQ began the development of the Common Formats for Event Reporting—Diagnostic Safety (CFER-DS). The CFER-DS is intended to facilitate the collection of a basic set of meaningful data about diagnostic safety events that can be aggregated and analyzed for learning and improvement at the local, regional, and national levels. As with all AHRQ CFER, the CFER-DS is expected to be made available in the public domain.
Usability is a prime consideration for any national data gathering initiative focused on diagnostic safety. Unlike the types of safety events defined by existing Common Formats, diagnostic safety events tend to be more ambiguous and may evolve over time, locations, and providers. In collaboration with the AHRQ, we conducted a usability assessment of a preliminary draft of the CFER-DS to inform its further development and implementation. Because terminology and concepts related to diagnostic safety are relatively novel in the context of safety event reporting, we were primarily concerned with users’ perceptions of content validity and ease of use. Therefore, our overall aim was to assess whether users found the CFER-DS items valid, appropriate in scope, and adequate to the task of encoding details of diagnostic safety events. We also explored users’ perceptions of feasibility and potential for adoption by healthcare organizations.
METHODS
Rationale and Overview
Prior studies have evaluated the content and usability of previously developed Common Formats using various qualitative and quantitative methods.10–13 Qualitative methods, such as interviews and focus groups, have been used to evaluate other safety event reporting systems.14–16 Structured items should be able to capture the dynamic complexity of diagnostic safety events but also be easily understood by the range of individuals who capture these events. We thus assessed the usability and validity of the CFER-DS by soliciting feedback from quality and safety personnel in U.S. healthcare organizations.
Participants were invited to use the CFER-DS to simulate reporting for a minimum of 5 diagnostic safety events. To protect the privacy and confidentiality of patients and providers, we collected information on user experiences with completion of the CFER-DS but did not collect information about the events themselves. The first author’s institution served as the coordinating site and housed the evaluation team who were primarily responsible for developing evaluation procedures, interviewing participants, and collecting and analyzing data. All procedures were approved by the institutional review board at the coordinating institution and conducted between August 2020 and December 2020.
Initial Draft of the CFER-DS
The draft of the CFER-DS adapted items from previously developed Common Formats and from 2 existing frameworks for conceptualizing diagnostic errors, the Safer Dx Framework17 and the Diagnostic Evaluation and Education Research taxonomy.4,5 The draft was revised with input from 3 subject matter experts (M.G., G.D.S., H.S.) with extensive experience in measurement and classification of diagnostic errors. The CFER-DS also offers a definition of a diagnostic safety event using concepts proposed in 2 prior diagnostic error definitions (one by the National Academy of Medicine,1 and the other by Singh18), as follows:
Diagnostic safety event: one or both of the following occurred, whether or not the patient was harmed:
Delayed, wrong, or missed diagnosis: There were one or more missed opportunities to pursue or identify an accurate and timely diagnosis (or other explanation) of the patient’s health problem(s) based on the information that existed at the time.
Diagnosis not communicated to patient: An accurate diagnosis (or other explanation) of the patient’s health problem(s) was available, but it was not communicated to the patient (includes the patient’s representative or family as applicable).
The test version of the CFER-DS was a paper form with 2 main components: a brief narrative report designed to be completed by a clinician with knowledge of the event and a series of structured items with multiple choice, yes/no answer choices, and/or free-text fields to capture detailed aspects of the event. Only the “form” version of the CFER-DS was subjected to usability testing, so participants experienced it as a questionnaire or survey tool.
Site Recruitment
We recruited a purposive sample of personnel from 8 U.S. healthcare organizations to provide feedback about the usability of the CFER-DS and potential for implementation. Participants held operational roles in their organizations, had expertise in healthcare quality and safety, and were involved in local and/or national organizational initiatives in quality and safety. Participants were encouraged to consult other team members from the same organization at their discretion.
Procedure
All evaluation procedures were conducted remotely. First, participants attended a teleconference during which they were oriented to the purpose of the project and the data collection procedures. The evaluation team then emailed an electronic copy of the draft CFER-DS to each participant. Participants were asked to complete the CFER-DS to simulate event reporting for 5 cases of diagnostic safety events that were familiar to them (e.g., events that occurred at their organizations, published or hypothetical events) and to email written feedback within approximately 4 weeks. The evaluation team held an interim “office hours” teleconference to answer questions while sites worked with the CFER-DS. Finally, participants engaged in individual postcompletion interviews with the evaluation team to follow up on written feedback and answer additional questions about usability and feasibility.
Data Collection
We developed a semistructured interview guide for postcompletion interviews that included questions about usability; the design, content, and sequence of items; and future implementation prospects. A qualitative methodologist (U.S.) led the postcompletion interviews; other members of the team took notes and occasionally prompted for elaboration or clarification. Interviews were scheduled for 1 hour, and participants were invited to include colleagues from their respective organizations as desired. We also developed a structured usability questionnaire to solicit multiple choice and open-ended feedback on ease of completion, burden, item clarity and flow, overall length, and importance of items. Finally, we invited participants to make comments and suggested revisions directly on a blank electronic copy of the CFER-DS (formatted in Microsoft Word). Thus, our evaluation data comprised participants’ written feedback and our team’s notes from the interviews and office-hour calls. Participants were reminded to maintain privacy and confidentiality of involved patients and providers, and no protected health information or other case details were disclosed during data collection.
Data Analysis
The interviewer conducted the initial data analysis by reviewing all team members’ notes, responses to the usability questionnaire, and participants’ annotations to the CFER-DS and creating a spreadsheet to categorize the detailed feedback from each participant (deidentified). The evaluation team met on multiple occasions to review and discuss participant feedback before sharing it with AHRQ. Frequently occurring issues, usability-related themes, and conceptual/definitional issues were captured in writing and are summarized in the following section. Separately, we consulted a psychometrician with expertise in survey design to comment on the design, clarity, and usability of the CFER-DS. The psychometrician was briefed on our methodology and on key themes from participants’ feedback.
RESULTS
Participants
Participants were located in the northeastern, southern, midwestern, and western regions of the United States. Participants were predominantly physicians (n = 6); one held a background in nursing, and another had a nonclinical professional background. All 8 participants held roles in quality and safety programs within their respective organizations. Their job titles included Chief Quality Officer, Medical Director of Quality, Associate Safety Officer, Director of Patient Safety, Clinical Quality and Regulatory Programs Director, and Professor and [Department] Vice-Chair. All participants completed and evaluated the CFER-DS in collaboration with other professionals at their organizations. Participants completed all evaluation procedures, with the exception of one who did not complete the usability questionnaire. All postcompletion interviews were completed in 60 minutes or less.
General Feedback and Impressions
Initial Modifications to the CFER-DS
Based on consistent feedback from the first 2 participants, we made interim modifications to streamline data collection, and AHRQ personnel made additional changes to the CFER-DS form including 2 major revisions:
A “brief narrative” section was moved from the beginning to the end of the form so that users provided this narrative summary or overview after they had a chance to answer detailed questions about the event.
The form was restructured to allow respondents to report details of several interrelated diagnostic encounters (i.e., multiple missed diagnostic opportunities related to the same event) using optional additional pages.
Length and Burden
Estimated average completion time for all CFER-DS items ranged from 30 to 90 minutes per case. Several participants noted faster completion times as they completed successive reports. Nonetheless, most participants shared concerns about burden and usability. For example, 2 participants anticipated that users might check “unknown” on many items because of the high burden of finding the information needed to answer them. Another anticipated possible “overfilling” if users make assumptions about the case without firsthand knowledge or adequate documentation. Generally, concerns about burden were greatest for information that is not reliably documented in the medical record (e.g., whether a trainee was involved in the case). Suggestions for reducing burden included the following: (1) to make some items optional and (2) to “frontload the form with higher priority questions.” Similarly, recommendations from the psychometric consultant included reducing overall length and burden and prioritizing items with the most critical or actionable content.
Item Clarity and Comprehension
Participants shared generally positive feedback about the clarity of items and adequacy of response choices. There were no recommendations from participants for major revisions to improve clarity. Psychometric recommendations included modifying response choice order (e.g., listing the most frequent or typical response choices first) and formatting and modifying instruction sets for clarity.
Content Validity and Coverage
There was good agreement among participants that the content coverage was adequate, and there were relatively few suggestions for additional content. A few participants noted opportunities to expand or elaborate answer values related to patient factors, including social determinants of health that may affect various aspects of the diagnostic process. Other suggestions pertained to specifics about the timing of communicating the diagnosis to the patient, involvement of ancillary diagnostic services, and patient and family engagement. For other items, some participants observed that having more or more specific answer values was not necessarily helpful. Some commented on the challenges of retrospectively characterizing communication, cognition, and other processes that may not be reliably documented or otherwise possible to discern. One participant commented that it would be helpful to accommodate the use of other commonly used standardized harm scales.
Appropriateness for Inpatient and Outpatient Settings
A few participants indicated that the items seemed to be a better fit for outpatient than inpatient care, referencing the use of the term “encounters” in the draft CFER-DS, and answer values that did not clearly accommodate more rapidly evolving inpatient and critical care scenarios. Suggestions for revising the form included having separate sets of items (e.g., using branching logic) or separate forms for inpatient versus outpatient settings.
Potential for Use in Operational Settings and Appropriate User Qualifications
We solicited feedback about usability of the CFER-DS as a reporting tool or questionnaire to be completed in its current form. Several participants suggested that use of the CFER-DS could fit into existing institutional quality and safety activities (e.g., used in the course of routine investigations triggered by incident reports). There was no consensus about who in the organization should complete the form, and some participants expressed doubt that end-user clinicians would do so routinely. However, all agreed that at least some clinician input was necessary. One participant stated, “Most likely it would be assigned to a review nurse or one of the risk management nurses, and I think they could do a lot of it, but some of these questions…might boil down to a judgment exercise,” and later noted, “The nuances of a missed opportunity might…be beyond nursing.” Another participant suggested, “…may want to minimize questions that need to be answered by frontline providers and encourage Quality, Safety, or Risk staff to complete bulk of the form.”
Other participants were more optimistic that clinicians could and should be heavily involved in reporting. One participant felt strongly that a clinician (physician or advanced practice provider) should complete the entire tool to appropriately answer questions reflecting diagnostic reasoning and clinical judgment. However, participants acknowledged that clinicians may not have the required time to do so and institutional commitment would be necessary to make this feasible.
Finally, one site noted potential for reporting only severe cases if, for example, reporting is coordinated through a specific program or office, such as risk management.
Conceptual Issues
Definition of Diagnostic “Encounter”
A recurring theme was lack of clarity about the meaning of “encounter,” a term used in the form but not defined. Diagnostic events usually involved multiple encounters between patients, clinicians, and the health system. Most participants found it challenging to report multiple related or successive encounters and/or missed opportunities, including multiple encounters that occur over the course of an inpatient stay. All participants recommended more instruction and/or examples.
Use as an Investigative Versus Reporting Tool
When choosing cases to simulate reporting using the Common Formats tool, participants generally selected cases that had already been reviewed in detail through an existing process such as root cause analysis. One participant noted that prior review “made it a lot easier” to complete the form. Another participant shared, “I picked cases that either I or my team member were involved in investigating… Most of the cases I completed the tool for had at least 20 hours of investigation.” Participants were receptive to the idea of a 2-stage process where the CFER-DS is completed at the end of an investigation. However, some participants saw potential for the CFER-DS to be used as a tool not only for reporting but also to facilitate the internal analysis of a diagnostic safety event. For example, one expressed interest in using the tool to help enable classification of errors: “If you knew where you had recurring problems, you might be able to construct a tool that helps you seek them out.”
Role of the Brief Narrative Report
The brief narrative was viewed as a helpful opportunity for the respondent to summarize the case review (e.g., one participant said it “helped bring the whole case together”). It was also described as a useful place to note contextual information not easily captured by structured items. One participant suggested that the brief narrative could provide an opportunity to explain the user’s interpretation of an item’s prompt or instructions.
Revisions After Usability Testing
Usability testing reinforced the variability in diagnostic processes that arises from differences in clinical situations, settings, specialties, and contextual factors. Furthermore, the patient, family, and different clinicians may perceive the facts of any particular diagnostic safety event quite differently. Feedback from participants made it clear that the draft CFER-DS needed more clarity as to scope, perspective, and terminology. It also confirmed that design of the CFER-DS must minimize the burden of data collection, encourage users to resist hindsight bias, and facilitate a learning and improvement perspective on diagnostic safety over the traditional sharp-end focus on diagnostic error. To address these challenges, AHRQ eliminated, revised, and reorganized several items in the original draft. The word “encounter” was replaced with a set of more generic and flexible concepts. New terminology was created to help users standardize how to outline the broad boundaries of a diagnostic safety event for CFER-DS purposes and then how to choose where to focus data collection.
DISCUSSION
Standards for diagnostic safety event reporting are necessary to advance national-level data collection and analysis for learning and improvement. Even the mere singling out of diagnostic case reporting can signal to organizations the need to highlight this important, previously underreported domain of patient safety. The present evaluation provides early evidence that standardized definitions of diagnostic safety events, contextual information, and contributing factors can be used by clinicians and other professionals to capture and report these complex events. Participants were able to complete all CFER-DS items for each of the 5 cases they reviewed (30 unique cases in total). They found the CFER-DS comprehensible and generally appropriate in scope. They also identified opportunities to improve usability by reducing overall length and burden, limiting required information to that which is most readily available through chart review, and refining certain items to fit with setting (i.e., inpatient versus outpatient). Participants reported that using the CFER-DS to describe especially complex events (e.g., multiple missed diagnostic opportunities within an episode of care) would pose a greater challenge.
Our evaluation focused mainly on issues of content validity and subjective ease of translating the details of a known case into a standardized set of responses. Although participants found items comprehensible, they also expressed concerns about the need for sensitivity to time and workload constraints in operational settings. Further development of Common Formats should aim to maximize efficiency and reliability while retaining essential data for learning and improvement. For example, task analysis methods can be used to identify high-workload items and obstacles in the reporting process and their potential effects on reporter behavior, missing data, and data fidelity. We were limited in our ability to address these issues because of a small selective sample and constraints on access to the data used to populate the CFER-DS items during simulated reporting.
Beyond initial development, the next challenge will be designing innovative ways to facilitate implementation of the CFER-DS as part of an evolving diagnostic safety improvement infrastructure at the local and national levels. One important consideration for healthcare organizations is allocating the appropriate personnel. Participants emphasized the importance of involving clinicians in this work. Diagnostic excellence requires a shift in perspective, from considering diagnostic safety improvement work as diverting time from clinical effort to seeing it as a critical component of clinical practice. Organizational support for the diagnostic team is essential.19 One possible solution is the development of specialized roles for clinicians to support quality improvement infrastructure. When linked to clear organizational needs and objectives, such roles can be a resource-effective means of enhancing patient safety outcomes.20
As of August 2021, more than 2 million inpatient safety events were aggregated in the Network of Patient Safety Database using Common Formats for Event Reports—Hospitals.21 A database of diagnostic safety events that is even a fraction of this magnitude could transform knowledge of diagnostic safety. Moreover, further evidence that the collection of this type of data is feasible could spur greater investment of national resources to promote diagnostic excellence.
Version 0.1 of the CFER-DS, which incorporated what was learned from usability testing, was reviewed recently by the Federal Patient Safety Work Group and posted for public comment (sample pages shown in Fig. 1). An Expert Panel convened by the National Quality Forum reviews the public comments, and the AHRQ will finalize and formally release CFER-DS version 1.0 in 2022 after considering the Expert Panel’s recommendations. As with all CFER, the CFER-DS will be posted on the PSO Privacy Protection Center Web site and will include a User Guide and materials to facilitate implementation. The AHRQ will continue to accept public comments on an ongoing basis and will make periodic improvements to the CFER-DS after its initial dissemination.
FIGURE 1.
Common Formats for Event Reporting—Diagnostic Safety, version 0.1.
CONCLUSIONS
Development and initial evaluation of the CFER-DS provided support for its content validity and usability, an important first step toward implementation of a new and comprehensive set of data elements for national aggregation of diagnostic safety event information. Further evaluation and refinement efforts should examine the implementation of the CFER-DS in a variety of healthcare settings for a variety of cases. This will ensure that AHRQ’s CFER-DS is relevant to a broad range of users for learning and safety improvement.
ACKNOWLEDGMENTS
The authors wish to thank Christiane Spitzmueller, PhD, for expert review of the draft CFER-DS.
Footnotes
This paper was funded under Contract No. HHSP233201500022I/75P00119F37006 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services. The authors are solely responsible for this article’s contents, findings, and conclusions, which do not necessarily represent the views of the AHRQ. Readers should not interpret any statement in this product as an official position of the AHRQ or of the U.S. Department of Health and Human Services.
Dr Singh is funded in part by the Houston Veterans Administration Health Services Research and Development Center for Innovations in Quality, Effectiveness and Safety (CIN-13-413), the Houston Veterans Administration Health Services Research and Development Service (IIR17-127 and the Presidential Early Career Award for Scientists and Engineers USA 14-274), and the AHRQ (R01HS27363).
The authors disclose no conflict of interest.
Contributor Information
Andrea Bradford, Email: Andrea.Bradford@bcm.edu.
Umber Shahid, Email: Umber.Shahid@bcm.edu.
Gordon D. Schiff, Email: gschiff@bwh.harvard.edu.
Mark L. Graber, Email: graber.mark@gmail.com.
Abigail Marinez, Email: Abigail.Marinez@bcm.edu.
Paula DiStabile, Email: Paula.Distabile@ahrq.hhs.gov.
Andrea Timashenka, Email: Andrea.Timashenka@ahrq.hhs.gov.
Hamid Jalal, Email: Hamid.Jalal@ahrq.hhs.gov.
Hardeep Singh, Email: hardeep.singh@va.gov;hardeeps@bcm.tmc.edu.
REFERENCES
- 1.Institute of Medicine (U.S.). Committee on Diagnostic Error in Health Care . Improving diagnosis in health care. In: Balogh E, Miller BT, Ball J, eds. Quality Chasm Series. Washington, DC: The National Academies Press; 2015:xxvii, 444 pages. [PubMed] [Google Scholar]
- 2.Zwaan L, Schiff GD, Singh H. Advancing the research agenda for diagnostic error reduction. BMJ Qual Saf. 2013;22(Suppl 2):ii52–ii57. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med. 2005;165:1493–1499. [DOI] [PubMed] [Google Scholar]
- 4.Schiff G Kim S Abrams R, et al. Diagnosing diagnosis errors: lessons from a multi-institutional collaborative project. In: Henriksen K Battles J Marks E, et al., eds. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville, MD: Agency for Healthcare Research and Quality; 2005. [PubMed] [Google Scholar]
- 5.Schiff GD Hasan O Kim S, et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med. 2009;169:1881–1887. [DOI] [PubMed] [Google Scholar]
- 6.Singh H Khanna A Spitzmueller C, et al. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6:315–323. [DOI] [PubMed] [Google Scholar]
- 7.Betsy Lehman Center for Patient Safety . Improving diagnosis in primary care one case at a time. 2021. Betsy Lehman Center for Patient Safety. Available at: https://betsylehmancenterma.gov/news/case-reports-anchor-a-learning-network-for-better-diagnosis-in-primary-care. Accessed August 1, 2021.
- 8.Agency for Healthcare Research and Quality . Network of Patient Safety Databases (NPSD). 2021. Available at: https://www.ahrq.gov/npsd/index.html. Accessed August 1, 2021.
- 9.Clancy CM. Common formats allow uniform collection and reporting of patient safety data by patient safety organizations. Am J Med Qual. 2010;25:73–75. [DOI] [PubMed] [Google Scholar]
- 10.Collins S Couture B Dykes P, et al. Implementation, evaluation, and recommendations for extension of AHRQ Common Formats to capture patient- and carepartner-generated safety data. JAMIA Open. 2018;1:20–25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Hua L, Gong Y. Usability Evaluation of a Voluntary Patient Safety Reporting System: Understanding the Difference between Predicted and Observed Time Values by Retrospective Think-Aloud Protocols. Berlin, Heidelberg: Springer; 2013:94–100. [Google Scholar]
- 12.Hua L, Gong Y. Toward user-centered patient safety event reporting system: a trial of text prediction in clinical data entry. In: Sarkar I, Georgiou A, Mazzoncini de Azevedo Marques P, eds. MEDINFO 2015: eHealth-Enabled Health Proceedings of the 15th World Congress on Health and Biomedical Informatics. Amsterdam, the Netherlands: IOS Press; 2015. [PubMed] [Google Scholar]
- 13.Williams T Szekendi M Pavkovic S, et al. The reliability of AHRQ Common Format Harm Scales in rating patient safety events. J Patient Saf. 2015;11:52–59. [DOI] [PubMed] [Google Scholar]
- 14.Daniels JP King AD Cochrane DD, et al. A human factors and survey methodology-based design of a web-based adverse event reporting system for families. Int J Med Inform. 2010;79:339–348. [DOI] [PubMed] [Google Scholar]
- 15.Schneider EC Ridgely MS Quigley DD, et al. Developing and testing the health care safety hotline: a prototype consumer reporting system for patient safety events. Rand Health Q. 2017;6:1. [PMC free article] [PubMed] [Google Scholar]
- 16.Sun O, Chen J, Magrabi F. Using voice-activated conversational interfaces for reporting patient safety incidents: a technical feasibility and pilot usability study. In: Cummings E, Ryan A, Schaper L, eds. Enhance the Practitioner and Consumer Experience in Healthcare: Selected Papers from the 26th Australian National Health Informatics Conference (HIC 2018). Amsterdam, the Netherlands: IOS Press; 2018. [PubMed] [Google Scholar]
- 17.Singh H, Sittig DF. Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework. BMJ Qual Saf. 2015;24:103–110. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Singh H. Editorial: helping health care organizations to define diagnostic errors as missed opportunities in diagnosis. Jt Comm J Qual Patient Saf. 2014;40:99–101. [DOI] [PubMed] [Google Scholar]
- 19.Agency for Healthcare Research and Quality . Leadership to Improve Diagnosis: A Call to Action. 2021. Available at: https://www.ahrq.gov/patient-safety/reports/issue-briefs/leadership.html. Accessed August 1, 2021.
- 20.Padula WV Nagarajan M Davidson PM, et al. Investing in skilled specialists to grow hospital infrastructure for quality improvement. J Patient Saf. 2021;17:51–55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Agency for Healthcare Research and Quality . NPSD Chartbook 2020. Available at: https://www.ahrq.gov/npsd/data/chartbook/index.html. Accessed August 1, 2021. [DOI] [PubMed]