Abstract
Objectives
A clear definition of delayed and missed coronary artery disease (CAD) diagnosis is lacking, hampering research on the epidemiology and determinants of delayed and missed CAD diagnosis. We engaged healthcare professionals, patients and healthy citizens to develop a consensus-based definition of delayed and missed diagnosis in two CAD domains: acute coronary syndrome (ACS) and chronic coronary syndrome (CCS).
Design
Modified Delphi method.
Setting
Primary and secondary care.
Participants
The study was conducted among an expert panel consisting of cardiologists (n=19), general practitioners (n=35), patients (n=55) and healthy citizens (n=34).
Results
42 definitions for delayed and missed ACS and 30 definitions for delayed and missed CCS were developed during the first 2 Delphi rounds. The expert panel graded these definitions in rounds 3 and 4 by Likert scale (from 1 (strongly disagree) to 5 (strongly agree)). Useful definitions for research purposes were recommended based on predefined criteria: (1) IQR to assess consensus, (2) median to evaluate the level of support for the definition and (3) convergence of the group to assess stability of opinions across rounds. For delayed ACS, 2 definitions were classified as highly recommended, 13 as recommended and 6 as considerable. For missed ACS, none met the threshold for highly recommended; 5 were recommended and 12 were classified as considerable. For CCS, 2 definitions for delayed diagnosis were highly recommended, 13 recommended and 7 considerable. For missed CCS, 1 definition was highly recommended, 6 were recommended and 7 were considerable.
Conclusions
We were unable to establish a one-size-fits-all definition for delayed and missed ACS/CCS diagnosis, reflecting the complexity and context-dependency of CAD diagnostic pathways. However, the developed consensus-based framework and recommended definitions provide a valuable basis for future research, enabling study comparisons across different diagnostic settings. This will ultimately enhance understanding of delayed and missed ACS/CCS diagnosis.
Keywords: Acute Coronary Syndrome, Angina Pectoris, Epidemiology, Quality of Health Care
WHAT IS ALREADY KNOWN ON THIS TOPIC
Delayed and missed diagnosis of coronary artery disease (CAD) remains prevalent.
There is no unambiguous definition for delayed and missed CAD diagnosis.
The absence of a standardised definition hampers research into the underlying determinants contributing to these diagnostic challenges.
WHAT THIS STUDY ADDS
This is the first study that attempted to develop an unambiguous and consensus-based definition of delayed and missed diagnosis in two CAD domains: acute coronary syndrome (ACS) and chronic coronary syndrome (CCS).
Our study established a clear framework for defining delayed and missed ACS/CCS diagnosis, providing recommended definitions for researchers in the field.
We demonstrate that there is no one-size-fits-all definition that is applicable in all clinical contexts.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The consensus-based framework provides a practical tool to standardise the definition of delayed and missed ACS/CCS diagnosis across different diagnostic settings.
The standardisation enables more consistent comparisons across studies and strengthens epidemiological and clinical research.
Improved comparability can facilitate more robust identification of determinants of diagnostic delays in ACS/CCS diagnosis, ultimately leading to better targeted interventions to reduce this.
Introduction
Coronary artery disease (CAD) is the leading global cause of mortality and loss of disability-adjusted life-years.1 Accurate and timely diagnosis is therefore essential. However, patients with CAD are still missed or receive a delayed diagnosis, leading to increased risk of adverse cardiovascular events, morbidity and mortality.2,4
CAD is most commonly characterised by epicardial atherosclerotic plaque accumulation in the coronary arteries, but may also be caused by coronary spasm or microvascular disease.5 It is a dynamic disease, consisting of long, stable periods and acute atherothrombotic events. This dynamic nature can be categorised as either chronic coronary syndrome (CCS) or acute coronary syndrome (ACS).6 ACS results from a sudden reduction in coronary blood flow, manifesting as unstable angina pectoris (myocardial ischaemia without rise in troponin levels) or acute myocardial infarction (myocardial necrosis with rise in troponin levels). In CCS, the coronary arteries are partially obstructed by the gradual development of atherosclerotic plaques. CCS also includes microvascular disease and coronary vasospasm.6 7
Previous studies on delayed and missed CAD diagnosis have used varying definitions. Definitions of missed ACS range from postdischarge detection of raised cardiac enzymes to retrospective identification of pathological Q-waves on the ECG or regional wall motion abnormalities on (stress)echocardiography, in patients without clear episodes of chest discomfort.8,12 Moreover, none of these studies were able to assess missed unstable angina pectoris. Defining a delayed ACS diagnosis is even more challenging. Most studies report time to hospital arrival (‘time to door’) or to percutaneous coronary intervention (‘time to needle’), considering these as prehospital and in-hospital delays, respectively, without further specification of cut-off values.13 14 In addition, studies on delayed and missed diagnosis of ACS have primarily been performed in the hospital setting, and contemporary evidence regarding missed ACS diagnosis in general practice is lacking.2 15 For CCS, several clinical guidelines have been published with recommendations on referral and diagnostic tests, but these do not provide definitions for missed or delayed diagnosis.6 7
Because of these varying definitions and methodological heterogeneity, meaningful comparison and synthesis of existing evidence remains limited. This is reflected in the wide variation in reported incidence and prevalence estimates of missed CAD diagnoses.15 For missed and/or unrecognised ACS, the incidence and prevalence rates with all ACS cases in the denominator ranged from 2.1% up to 30%, and from 0.18% to 3.1% when either (1) patients suspected of ACS or (2) the general population were used in the numerator, respectively.8 16 17
A standardised, widely accepted definition is therefore needed to enable meaningful comparison between studies, identify determinants of diagnostic delay and develop strategies to improve early recognition and referral.15 Given the complexity of defining missed and delayed CAD diagnosis, with differing perspectives of general practitioners (GPs), cardiologists, patients and healthy citizens, the Delphi method was chosen to systematically gather the judgements of these different expert groups. This study aims to develop a consensus-based definition for delayed and missed diagnosis in two CAD domains: ACS and CCS, using a modified Delphi method, for use in research in the Netherlands and countries with comparable, primary care-based healthcare systems.
Methods
Design
The current study used a modified Delphi method to reach consensus about the definition of delayed and missed ACS and CCS diagnosis for research purposes. A Delphi method is a structured scientific communication technique that seeks to obtain consensus within a group of experts by using a series of questionnaires.18 19 The study was conducted between March 2022 and January 2023. The Delphi process was designed and reported in accordance with published methodological guidance, including the Conducting and REporting of DElphi Studies and the ACcurate COnsensus Reporting Document.20 21 The study protocol was not prospectively registered. The Delphi process was directed by the research team consisting of two PhD’s students (executive researchers), two GPs (of which one specialised in cardiovascular disease), and an assistant professor in psychocardiology. The research team was responsible for designing the Delphi rounds, developing the initial statements, selecting and inviting panel members, and analysing responses between rounds. Members of our research team did not participate in the Delphi rounds.
Expert panel
We aimed to gather opinions from all relevant members of the diagnostic process for establishing CAD. To this end, four relevant groups of experts were identified: (1) GPs, including those with emergency or cardiovascular specialisation, (2) cardiologists with various subspecialties working in academic or peripheral hospitals, (3) patients with a CAD diagnosis (either CCS or ACS) and (4) healthy citizens (ie, without CAD). These expert groups were selected to obtain a heterogeneous group of panellists, with different viewpoints, and therefore opinions.22 23 Patients and healthy citizens were included to ensure the definitions also represent their opinions and experiences on delayed and missed ACS/CCS diagnosis. Furthermore, a heterogeneous sample allows for group comparison. It was assumed that this selection of expert groups could lead to more credibility for the results of the study. Inclusion criteria were age >18 years and sufficient proficiency in Dutch. The exclusion criterion was age >75 years, because of the online format and anticipated questionnaire burden. Based on previous literature, it was decided that at least 7–10 members per group were required for reliable outcomes.24 25 Considering the attrition of participants between rounds, at least 20 panellists per group were invited. In total, 35 GPs, 20 cardiologists, 55 patients and 34 citizens were approached.
Expert panel recruitment
Eligible GPs and cardiologists (furthermore, referred to as healthcare providers (HCPs)) were identified based on the inclusion criteria, recommendations of our research team, and following consultation of the working group of cardiological centres in the Netherlands (WCN), the GP and emergency care expert group (SpoedHAG), and the society of GPs with cardiovascular specialisation (HartVaatHAG). HCPs were approached via email by the executive researchers (DLVS and ASMD), and an information letter was sent. Patients were approached by two national patient communities for cardiovascular diseases, either via email (Harteraad) or a call for participation on their Facebook page (Vrouwenhart.nl). When interested, patients could give consent to be contacted either via an online form regulated by the patient community (Harteraad) or by sending an email to the executive researchers (Vrouwenhart.nl). Healthy citizens were approached through the professional and personal networks of the executive researchers and via snowball effect. An information letter was sent by email to patients and healthy citizens. Patients and healthy citizens are hereafter referred to as non-HCPs. Panellists were not asked to suggest other panel members. For all eligible panellists, 2 weeks were given to consider participation and ask questions. After agreeing to participate, the questionnaire was sent via an online data management programme called Castor. Online informed consent was required before the questionnaire was started. To encourage participation, panellists were reimbursed for their time investment by receiving a gift voucher after completion of all four Delphi rounds.
Delphi rounds and analysis
The study consisted of four anonymous online questionnaire rounds. Panellists participated anonymously in the study and were not informed about the other panellists. They were aware of the four included expert groups. For each round, two versions of the questionnaire were created, one for HCPs and one for patients and healthy citizens in layman’s language. The first questionnaire for patients and healthy citizens was tested and evaluated by three patient representatives to ensure readability. Their feedback was used to improve comprehensibility of the questionnaire in layman’s language. Non-responders received up to two reminders per round, and all panellists were reinvited for subsequent rounds regardless of prior participation.26 After each round, the data obtained were analysed and presented to all panellists in the next round (controlled feedback).
Round 1
The first round consisted of six open-ended questions to generate first ideas about how to define delayed and missed CAD diagnosis. Furthermore, demographical information was gathered (eg, sex, age, profession, years of work experience). The questionnaire was developed by DS and AD based on published evidence, guidelines and clinical experience, and further evaluated by the research team. It was decided to differentiate between two domains of CAD; namely ACS and CCS.
Before answering the questions, the following situation was outlined for ACS:
A patient presents with acute chest discomfort (pain, pressure, or tightness in the chest). If it turns out to be an acute coronary syndrome…
The following situation was outlined for CCS:
A patient presents with long-standing chest discomfort. If it turns out to be a chronic coronary syndrome (CCS)…
Then, for both ACS and CCS, the following three questions were asked: (Q1) When is the diagnosis of ACS/CCS made in time? (Q2) When is the diagnosis of ACS/CCS delayed? (Q3) When is the diagnosis of ACS/CCS missed? It was specified that panellists should indicate time or frequency (eg, minutes, hours, number of contact moments). There was an option to comment in free-text boxes. Data from round 1 was analysed using a thematic analysis. This is a step-by-step standardised process for analysing and reporting themes within qualitative data and is further described by Braun & Clarke.27 In the first step, 50% of the data (answers) was coded independently by the DS and AD using Nvivo (released in March 2020). Thereafter, the first codes were discussed between DS and AD until a consensus on the codes was reached. Similar codes were grouped, and all codes were provided with a clear definition resulting in a codebook. The codebook was then further applied to the rest of the data. Inter-coder reliability was enhanced by setting up weekly meetings to discuss additional codes and coding differences. Once all answers were coded, codes were organised into potential overarching themes representing different ways to define timely, delayed and missed ACS/CCS diagnosis.
Round 2
In the second round, the resulting themes and draft definitions were returned to panellists for ‘member checking’ to optimise construct validity. The goal of member checking in qualitative research is to verify whether answers were correctly interpreted and summarised by the researchers and if there are any missing ideas.28 The panellists were asked whether the themes and list of definitions were clear and complete. If it was incomplete according to panellists, they were requested to formulate the missing theme or definition. Additional suggestions were incorporated using the same thematic analysis approach as in round 1.
Round 3
The objective of round 3 was to assess agreement on the refined definitions of round 2. First, based on the results from round 2, the definitions were further refined. Subsequently, the redefined definitions were evaluated in an expert panel discussion consisting of our research team and additionally a cardiologist. The final definitions were checked for completeness and accuracy by comparing them with the raw data from round 1. Lastly, all panellists were asked to what extent they agreed with a definition on a 5-point Likert scale (1-strongly disagree, 2-disagree, 3-neutral, 4-agree, 5-strongly agree). See online supplemental Tables S1 and S2 for an overview of all definitions for ACS and CCS, respectively.
Round 4
In the fourth and final round, panellists presented the same definitions as in round 3; no definitions were excluded based on the scoring in round 3. For each definition, panellists were shown the mean group score from the previous round as controlled feedback. With this additional information, panellists were again asked to rate their level of agreement with each definition using a 5-point Likert scale.
Statistical analysis rounds 3 and 4
The quantitative data from rounds 3 and 4 were analysed using descriptive statistics to determine consensus according to predetermined criteria, based on the literature review on consensus measurement in Delphi studies (see table 1).29
Table 1. Predefined consensus criteria.
| Support for definition | Consensus | Stability | |||
|---|---|---|---|---|---|
| Median | Degree | IQR | Degree | Relative IQR (CG) | Degree |
| ≥4 | High | IQR<1 | High | ≤0.5 | High |
| 3<M<4 | Medium | 1≤IQR≤1.5 | Medium | ||
| ≤3 | Low | IQR>1.5 | Low | 1 | No stability |
.
CG, convergence of group.
Predefined criteria for consensus, support and stability
The aim was to provide recommendations on the most appropriate definitions to use in research on missed and delayed ACS/CCS diagnosis. To this end, the rated definitions were classified based on predefined criteria, specifically: (1) the IQR to assess consensus, (2) the median to evaluate the level of support for the definition and (3) convergence of the group (CG) to assess the stability of opinions across successive rounds. The IQR is a measure of the dispersion of the median, with an IQR lower than 1 meaning that more than 50% of the opinions are within 1 point on the 5-point Likert scale. The median score indicates the level of support for a given definition on a 5-point Likert scale. A lower median score indicates a lower degree of support, whereas a higher median indicates a higher level of support for the rated definition. The CG is calculated by the relative IQR ((CG= IQRRound3 −IQRRound4)/IQR Round3)) and indicates the convergence of group opinions for a definition over rounds. In other words, it indicates how stable opinions are over the subsequent questionnaire rounds. A CG near 1 indicates a high degree of convergence of opinions; conversely, a lower CG means less convergence and thus stability of opinions between the rounds. Consensus was defined as an IQR below 1, a definition was highly supported if the median was equal to or above 4, and stability over rounds was reached if the CG was equal to or below 0.5 (see table 1).29 30
Classification of recommended definitions
Based on these criteria, definitions were classified as ‘highly recommended’, ‘recommended’, ‘to be considered’, or ‘not recommended’ (figure 1). The primary aim of the study was to develop standardised definitions suitable for use in clinical and epidemiological research. During rounds 1 and 2, HCPs and non-HCPs perspectives were equally incorporated in generating and refining candidate definitions. In rounds 3 and 4, the predefined classification criteria (median, IQR and CG) were first applied to the overall sample (HCPs and non-HCPs combined). Given that HCPs have specific expertise in clinical reasoning, diagnostic pathways and research methodology, we additionally examined whether applying the same criteria to the HCPs-only sample would yield different classification outcomes. During the analysis phase, differences in scoring patterns between HCPs and non-HCPs became apparent. Therefore, a definition was highly recommended if there was high support for the definition (median≥4), consensus (IQR<1) and stability (CG≤0.5) in the overall sample or in the HCPs-only sample. A definition was recommended if there was high support (median≥4), consensus (IQR<1), but no stability (CG>0.5) in both the overall sample and in the HCPs-only sample; it was classified as considerable if there was high support, consensus, but no stability in either sample. All other definitions were not recommended. This additional conditional decision rule ensured that definitions supported from a clinical and research expertise perspective were not excluded or too highly recommended. Notably, applying this approach led to more downgrades than upgrades, indicating that the procedure functioned as a stricter rather than a preferential weighting mechanism. To assess the impact of this conditional decision rule, we examined which definitions changed recommendation category when applying the criteria to the overall sample compared with the HCPs-only sample. To further examine differences between expert groups, independent samples t-tests were conducted comparing mean agreement levels between HCPs and non-HCPs and between cardiologists and GPs. Analyses were conducted using IBM SPSS Statistics, V.27.
Figure 1. Definition recommendation criteria. *Overall sample meaning: healthcare providers (cardiologists and GPs) and non-healthcare providers (patients and healthy citizens). **There was no clear distinction between definitions, meaning that the results showed consensus for defining a missed ACS diagnosis if the patient reports symptoms to a healthcare provider after more than 2 hours, more than 4 hours and more than 6 hours from symptom onset. With consensus on all three time frames, it is not possible to recommend a specific duration for defining a missed ACS diagnosis. ACS, acute coronary syndrome; HCP, healthcare provider.
Results
Study timeline
The study was conducted between 9 March 2022 and 12 January 2023. The time between rounds was longer than the anticipated 3 weeks (2 weeks for panel members to fill out the questionnaire and 1 week for researchers to analyse the data), because (1) we sent multiple reminders to encourage panellists to fill in the questionnaire, (2) analysis of the data took more time due to the extensive amount of information gathered, especially the first two rounds. See figure 2 for an overview of the study process.
Figure 2. Flow chart of the Delphi study.
Characteristics of participants
The response rates for Delphi rounds 1, 2, 3 and 4 were 79.9%, 58.8%, 59.3% and 58.5%, respectively (figure 2). Six panellists were deleted from the analysis because of multiple outliers, indicating that the questionnaire was not completed adequately. The characteristics of the panellists are shown in table 2. The mean age of the total sample was 51.9 years, and 58% were female. Females accounted for 37% of the cardiologists, 64% of GPs, 61% of patients and 58% of healthy citizens.
Table 2. Demographics of panellists who received and completed the questionnaire of round 1.
| Panellists | Cardiologists | GPs | Patients | Healthy citizens |
|---|---|---|---|---|
| n (%)/mean (SD) | n (%)/mean (SD) | n (%)/mean (SD) | n (%)/mean (SD) | |
| Number of participants | 19 (14) | 34 (25) | 49 (36) | 33 (25) |
| Demographics | ||||
| Mean age | 46.5 (SD 7.0) | 47.6 (SD 9.2) | 62.8 (SD 8.4) | 50.8 (SD 12.0) |
| Females | 7 (36.8) | 22 (64.7) | 30 (61.2) | 19 (57.6) |
| Males | 12 (63.2) | 12 (35.3) | 19 (38.8) | 14 (42.4) |
| Having a partner* | 39 (79.6) | 22 (66.7) | ||
| College education or higher† | 39 (79.6) | 21 (63.6) | ||
| Paid work (vs other)‡ | 7 (14.3) | 18 (54.5) | ||
| Subspecialty GP | ||||
| GP | 15 (44.1) | |||
| GP specialised in cardiovascular diseases | 14 (41.2) | |||
| GP specialised in acute care | 5 (14.7) | |||
| Subspecialty cardiologist | ||||
| Cardiologist–general | 4 (21.1) | |||
| Cardiologist–heart failure | 2 (10.5) | |||
| Cardiologist–cardiac imaging | 7 (36.8) | |||
| Cardiologist–coronary intervention | 1 (5.3) | |||
| Cardiologist–female-specific | 4 (21.1) | |||
| Cardiologist–congenital cardiology | 1 (5.3) | |||
| Years of working experience | 12.9 (SD 9.6) | 15.0 (SD 8.1) |
This information was only collected for patients and healthy citizens.
This information was only collected for patients and healthy citizens.
This information was only collected for patients and healthy citizens. Paid work includes both part-time and full-time paid work. No paid work includes unemployment, running the household, being retired and sick leave).
GP, general practitioner.
In total 80% of the patients and 64% of the healthy citizens had a college education or higher. Among patients, 14% were currently working, and of those with unpaid work, the majority were retired or incapacitated for work. The average working experience for cardiologists was 12.9 (SD 9.6) years, and for GPs 15.0 (SD 8.1) years.
Delphi round 1
Based on thematic analysis, we identified 40 codes that could be organised into four themes with corresponding subthemes (online supplemental Figure S3). The themes represented four different ways of defining delayed and missed CAD diagnosis. The first theme, number of contacts, included contacts with the GP and with the specialist (eg, cardiologist). The second theme, course of time, comprised time after first symptom presentation, time after first symptom onset and patient delay—the interval between symptom onset and the patient’s decision to seek medical care. The third theme, the degree of cardiac damage, included myocardial damage, death and quality of life. The fourth theme was the combination of the previous themes (eg, the number of contacts+the course of time, the number of contacts+the degree of (cardiac) damage, the course of time+the degree of (cardiac) damage).
After removing duplicate definitions, 18 definitions for delayed ACS, 12 for missed ACS, 16 for delayed CCS and 16 for missed CCS remained.
Delphi round 2
In total, 80% (n=63) of panellists agreed with the clarity and 72% (n=57) with the completeness of the four initial themes for defining delayed and missed ACS/CCS diagnosis (online supplemental Figure S3). Based on panel feedback, the themes and subthemes were revised: the fourth ‘combination’ theme was removed, the framework was simplified, and additional definitions were added for patient delay and quality of life. The final framework consisted of three main themes, hereinafter referred to as ways to define a delayed or missed diagnosis: (1) the number of contacts, (2) the course of time and (3) the degree of (cardiac) damage (see figure 3). Definitions identified in round 1 were resubmitted to assess completeness (member checking). Overall, 51 panellists (65%) reported no missing definitions for delayed ACS diagnosis, and 62 (79%) reported none for missed ACS diagnosis. For CCS, 56 panellists (71%) identified no missing definitions for delayed diagnosis and 63 (80%) for missed diagnosis (online supplemental Table S4).
Figure 3. Three ways to define missed and delayed ACS and CCS diagnoses. ACS, acute coronary syndrome; CCS, chronic coronary syndrome; GP, general practitioner.
In response to feedback, several definitions were added or simplified, including removal of contextual details (eg, reasons for delay or missed diagnosis). In total, 42 ACS definitions and 30 CCS definitions were scored in rounds 3 and 4 (see online supplemental Tables S1 and S2, respectively).
Delphi round 3
The support and consensus levels per definition for rounds 3 and 4 are shown in table 3 for ACS and in table 4 for CCS. Definitions were grouped under the three main themes of the final framework (figure 3). For ACS (table 3), the percentage of definitions that reached high support (median≥4) in round 3 was 95% (n=40) for delayed ACS and 74% (n=31) for missed ACS. However, no definition reached high support and high consensus (median≥4, IQR<1).
Table 3. Results of the Delphi questionnaire round 3 and 4 for defining missed and delayed acute coronary syndrome diagnosis in the overall sample.
| Theme | Item number* | Delphi round 3 (n=80) | Delphi round 4 (n=79) | Stability | ||||
|---|---|---|---|---|---|---|---|---|
| Mean (SD) | Median | IQR3 | Mean (SD) | Median | IQR4 | CG | ||
| Question: A patient presents with acute chest discomfort. If it turns out to be an acute coronary syndrome, when was the diagnosis delayed? | ||||||||
| 1. The number of contacts | 1 | 3.76 (1.26) | 4 | 2 | 3.84 (0.75) | 4 | 0 | 1 |
| 2 | 3.68 (1.45) | 4 | 2 | 3.85 (1.02) | 4 | 0 | 1 | |
| 3 | 3.50 (1.61) | 4 | 3 | 4.05 (1.17) | 4 | 1 | 0.7 | |
| 4 | 3.70 (1.36) | 4 | 2 | 3.82 (0.94) | 4 | 0 | 1 | |
| 5 | 3.66 (1.40) | 4 | 2 | 3.84 (0.93) | 4 | 0 | 1 | |
| 2. The course of time | 6 | 3.23 (1.17) | 3 | 2 | 3.21 (0.73) | 3 | 1 | 0.5 |
| 7 | 3.63 (1.17) | 4 | 2 | 3.61 (0.85) | 4 | 1 | 0.5 | |
| 8 | 3.71 (1.26) | 4 | 2 | 3.83 (0.83) | 4 | 0 | 1 | |
| 9 | 3.78 (1.41) | 4 | 2 | 3.92 (0.94) | 4 | 1 | 0.5 | |
| 10 | 3.20 (1.12) | 3 | 2 | 3.12 (0.67) | 3 | 1 | 0.5 | |
| 11 | 4.05 (1.10) | 4 | 1 | 3.85 (0.64) | 4 | 0 | 1 | |
| 12 | 3.80 (1.31) | 4 | 2 | 3.81 (0.98) | 4 | 0 | 1 | |
| 13 | 3.89 (1.16) | 4 | 2 | 3.96 (0.82) | 4 | 0 | 1 | |
| 14 | 3.74 (1.40) | 4 | 2 | 4.11 (0.89) | 4 | 1 | 0.5 | |
| 15 | 3.48 (1.54) | 4 | 3 | 3.74 (1.23) | 4 | 1.5 | 0.5 | |
| 16 | 3.60 (1.11) | 4 | 1 | 3.84 (0.56) | 4 | 0 | 1 | |
| 17 | 3.64 (1.31) | 4 | 2 | 4.02 (0.75) | 4 | 0 | 1 | |
| 18 | 3.67 (1.47) | 4 | 3 | 3.93 (1.15) | 4 | 1 | 0.7 | |
| 19 | 3.54 (1.49) | 4 | 3 | 3.91 (1.28) | 4 | 1 | 0.7 | |
| 20 | 3.72 (1.12) | 4 | 1.25 | 3.85 (0.66) | 4 | 0 | 1 | |
| 21 | 3.82 (1.24) | 4 | 2 | 3.83 (0.80) | 4 | 0 | 1 | |
| 22 | 4.00 (1.22) | 4 | 1 | 3.93 (0.89) | 4 | 0 | 1 | |
| 23 | 3.93 (1.13) | 4 | 2 | 3.91 (0.70) | 4 | 0 | 1 | |
| 24 | 4.00 (1.23) | 4 | 2 | 3.85 (0.93) | 4 | 0 | 1 | |
| 25 | 3.93 (1.28) | 4 | 2 | 3.84 (1.07) | 4 | 1 | 0.5 | |
| 26 | 3.84 (1.39) | 4 | 2 | 3.89 (1.06) | 4 | 1 | 0.5 | |
| 27 | 3.93 (1.01) | 4 | 2 | 3.93 (0.84) | 4 | 0 | 1 | |
| 28 | 3.85 (1.23) | 4 | 2 | 3.96 (0.87) | 4 | 0.25 | 0.9 | |
| 29 | 3.77 (1.34) | 4 | 2 | 3.80 (1.23) | 4 | 2 | 0 | |
| 30 | 3.56 (1.19) | 4 | 1.25 | 3.85 (0.46) | 4 | 0 | 1 | |
| 31 | 3.76 (1.18) | 4 | 2 | 3.89 (0.51) | 4 | 0 | 1 | |
| 32 | 3.74 (1.31) | 4 | 2 | 3.88 (0.55) | 4 | 0 | 1 | |
| 33 | 3.76 (1.35) | 4 | 2 | 3.92 (0.52) | 4 | 0 | 1 | |
| 34 | 3.77 (1.41) | 4 | 2 | 4.04 (0.55) | 4 | 0 | 1 | |
| 35 | 3.89 (0.87) | 4 | 1 | 3.85 (0.60) | 4 | 0 | 1 | |
| 36 | 3.98 (1.06) | 4 | 1 | 3.98 (0.50) | 4 | 0 | 1 | |
| 3. The degree of (cardiac) damage | 37 | 3.85 (1.17) | 4 | 1 | 3.89 (0.69) | 4 | 0 | 1 |
| 38 | 3.82 (1.24) | 4 | 1.25 | 3.98 (0.65) | 4 | 0.5 | 0.6 | |
| 39 | 3.54 (1.42) | 4 | 2.5 | 4.00 (0.84) | 4 | 1 | 0.6 | |
| 40 | 3.49 (1.55) | 4 | 3 | 4.02 (0.95) | 4 | 0 | 1 | |
| 41 | 3.95 (1.30) | 4 | 1 | 4.02 (0.77) | 4 | 0 | 1 | |
| 42 | 4.02 (1.03) | 4 | 1 | 3.95 (0.72) | 4 | 1 | 1 | |
| Question: A patient presents with acute chest discomfort. If it turns out to be an acute coronary syndrome, when was the diagnosis missed? | ||||||||
| 1. The number of contacts | 1 | 3.86 (1.23) | 4 | 2 | 3.88 (0.78) | 4 | 0 | 1 |
| 2 | 4.21 (1.08) | 5 | 1 | 4.39 (0.81) | 5 | 1 | 0 | |
| 3 | 4.21 (1.22) | 5 | 1 | 4.66 (0.48) | 5 | 1 | 0 | |
| 4 | 4.24 (1.02) | 5 | 1 | 4.57 (0.67) | 5 | 1 | 0 | |
| 5 | 4.13 (1.10) | 4 | 1 | 4.18 (0.83) | 4 | 1 | 0 | |
| 2. The course of time | 6 | 2.88 (1.29) | 3 | 2 | 2.72 (0.85) | 3 | 1 | 0.5 |
| 7 | 3.24 (1.34) | 3 | 3 | 3.19 (0.86) | 3 | 1 | 0.7 | |
| 8 | 3.62 (1.28) | 4 | 2 | 3.89 (0.88) | 4 | 0 | 1 | |
| 9 | 4.05 (1.19) | 4 | 1.25 | 4.15 (0.65) | 4 | 1 | 0.2 | |
| 10 | 2.85 (1.31) | 3 | 2 | 2.93 (0.68) | 3 | 0 | 1 | |
| 11 | 3.49 (1.39) | 4 | 3 | 3.68 (0.86) | 4 | 1 | 0.7 | |
| 12 | 3.67 (1.22) | 4 | 2 | 3.84 (0.81) | 4 | 0 | 1 | |
| 13 | 3.50 (1.14) | 4 | 1.25 | 3.77 (0.91) | 4 | 0 | 1 | |
| 14 | 3.74 (1.25) | 4 | 2 | 4.07 (0.86) | 4 | 1 | 0.5 | |
| 15 | 4.20 (1.13) | 5 | 1 | 4.47 (0.87) | 5 | 1 | 0 | |
| 16 | 2.95 (1.25) | 3 | 2 | 3.18 (0.90) | 3 | 1 | 0.5 | |
| 17 | 3.47 (1.27) | 4 | 2.5 | 4.05 (0.73) | 4 | 0 | 1 | |
| 18 | 3.75 (1.33) | 4 | 3 | 4.24 (0.79) | 4 | 1 | 0.7 | |
| 19 | 3.83 (1.31) | 4 | 2 | 4.28 (0.82) | 4 | 1 | 0.5 | |
| 20 | 3.31 (1.27) | 3.5 | 2 | 3.46 (0.83) | 3 | 1 | 0.5 | |
| 21 | 3.21 (1.29) | 3 | 2 | 3.13 (0.98) | 3 | 1 | 0.5 | |
| 22 | 3.82 (1.25) | 4 | 2 | 3.98 (0.81) | 4 | 0 | 1 | |
| 23 | 3.61 (1.28) | 4 | 2.5 | 3.80 (0.10) | 4 | 0.25 | 0.9 | |
| 24 | 3.77 (1.25) | 4 | 2 | 3.93 (0.10) | 4 | 0.25 | 0.9 | |
| 25 | 3.95 (1.19) | 4 | 2 | 4.13 (0.86) | 4 | 1 | 0.5 | |
| 26 | 4.06 (1.20) | 4.5 | 1.25 | 4.26 (0.65) | 4 | 1 | 0.2 | |
| 27 | 3.62 (1.11) | 4 | 1 | 3.94 (0.90) | 4 | 0 | 1 | |
| 28 | 3.79 (1.23) | 4 | 2 | 3.96 (0.93) | 4 | 1 | 0.5 | |
| 29 | 4.24 (1.08) | 5 | 1 | 4.42 (0.88) | 5 | 1 | 0 | |
| 30 | 2.65 (1.18) | 2 | 2 | 2.28 (0.74) | 2 | 1 | 0.5 | |
| 31 | 2.91 (1.26) | 3 | 2 | 2.79 (0.82) | 3 | 1 | 0.5 | |
| 32 | 3.22 (1.37) | 3.5 | 2 | 2.96 (0.77) | 3 | 1 | 0.5 | |
| 33 | 3.37 (1.43) | 4 | 3 | 3.40 (0.87) | 4 | 1 | 0.7 | |
| 34 | 3.38 (1.43) | 4 | 3 | 3.62 (0.89) | 4 | 0.75 | 0.8 | |
| 35 | 3.02 (1.22) | 3 | 2 | 3.10 (0.83) | 3 | 2 | 0 | |
| 36 | 3.51 (1.26) | 4 | 3 | 3.72 (0.77) | 4 | 0.5 | 0.8 | |
| 3. The degree of (cardiac) damage | 37 | 3.92 (1.06) | 4 | 1 | 4.11 (0.58) | 4 | 0 | 1 |
| 38 | 3.85 (1.22) | 4 | 2 | 4.05 (0.76) | 4 | 0 | 1 | |
| 39 | 3.97 (1.09) | 4 | 2 | 4.13 (0.85) | 4 | 1 | 0.5 | |
| 40 | 4.36 (0.87) | 5 | 1 | 4.63 (0.55) | 5 | 1 | 0 | |
| 41 | 4.04 (1.16) | 4 | 2 | 4.20 (0.73) | 4 | 1 | 0.5 | |
| 42 | 3.70 (1.15) | 4 | 2 | 4.03 (0.78) | 4 | 0 | 1 | |
.
Corresponding definitions can be found based on theme and item number in online supplemental Table 1.
CG, convergence of group.
Table 4. Results of the Delphi questionnaire rounds 3 and 4 for defining delayed and missed chronic coronary syndrome diagnosis in the overall sample.
| Theme | Item number* | Delphi round 3 (n=80) | Delphi round 4 (n=79) | Stability | ||||
|---|---|---|---|---|---|---|---|---|
| Mean (SD) | Median | IQR3 | Mean (SD) | Median | IQR4 | CG | ||
| Question: A patient presents with long-standing chest discomfort. If it turns out to be a chronic coronary syndrome, when was the diagnosis delayed? | ||||||||
| 1. The number of contacts | 1 | 3.50 (1.13) | 4 | 1 | 3.59 (0.83) | 4 | 1 | 0 |
| 2 | 3.65 (1.22) | 4 | 2 | 3.81 (0.64) | 4 | 0 | 1 | |
| 3 | 3.51 (1.41) | 4 | 3 | 3.82 (0.90) | 4 | 0 | 1 | |
| 4 | 3.41 (1.10) | 4 | 1 | 3.72 (0.78) | 4 | 0.5 | 0.5 | |
| 5 | 3.49 (1.12) | 4 | 1 | 3.93 (0.66) | 4 | 0 | 1 | |
| 6 | 3.45 (1.35) | 4 | 2.25 | 4.05 (0.82) | 4 | 1 | 0.5 | |
| 7 | 3.53 (1.10) | 4 | 1 | 3.86 (0.54) | 4 | 0 | 1 | |
| 8 | 3.63 (1.23) | 4 | 2 | 3.93 (0.74) | 4 | 0 | 1 | |
| 9 | 3.50 (1.48) | 4 | 3 | 3.98 (0.94) | 4 | 1 | 0.7 | |
| 10 | 3.53 (1.64) | 4 | 3 | 4.00 (1.05) | 4 | 1 | 0.7 | |
| 11 | 3.78 (1.20) | 4 | 2 | 3.85 (0.85) | 4 | 0 | 1 | |
| 2. The course of time | 12 | 3.31 (1.23) | 4 | 2 | 3.66 (0.80) | 4 | 0.75 | 0.6 |
| 13 | 3.31 (1.36) | 4 | 2.5 | 3.79 (0.68) | 4 | 0 | 1 | |
| 14 | 3.40 (1.46) | 4 | 3 | 3.76 (0.78) | 4 | 0 | 1 | |
| 15 | 3.66 (1.26) | 4 | 2 | 3.95 (0.62) | 4 | 0 | 1 | |
| 16 | 3.81 (1.41) | 4 | 2 | 4.00 (0.84) | 4 | 0.75 | 0.6 | |
| 17 | 3.70 (1.58) | 4 | 3 | 4.15 (0.95) | 4 | 1 | 0.7 | |
| 18 | 3.50 (1.74) | 4 | 4 | 4.10 (1.04) | 4 | 1 | 0.8 | |
| 19 | 3.55 (1.16) | 4 | 1 | 3.81 (0.60) | 4 | 0 | 1 | |
| 20 | 3.48 (1.31) | 4 | 3 | 3.93 (0.60) | 4 | 0 | 1 | |
| 21 | 3.59 (1.49) | 4 | 3 | 3.98 (0.71) | 4 | 0 | 1 | |
| 22 | 3.52 (1.03) | 4 | 1 | 3.78 (0.61) | 4 | 0 | 1 | |
| 23 | 3.81 (1.19) | 4 | 2 | 3.90 (0.51) | 4 | 0 | 1 | |
| 24 | 3.79 (1.35) | 4 | 2 | 3.97 (0.63) | 4 | 0 | 1 | |
| 3. The degree of (cardiac) damage | 25 | 3.60 (1.26) | 4 | 1.25 | 3.86 (0.64) | 4 | 0 | 1 |
| 26 | 3.61 (1.38) | 4 | 2.5 | 3.82 (0.90) | 4 | 0 | 1 | |
| 27 | 3.61 (1.25) | 4 | 2 | 3.90 (0.74) | 4 | 0 | 1 | |
| 28 | 3.48 (1.37) | 4 | 2 | 3.97 (0.76) | 4 | 0 | 1 | |
| 29 | 3.77 (1.08) | 4 | 2 | 3.98 (0.68) | 4 | 0 | 1 | |
| 30 | 3.72 (1.22) | 4 | 2 | 3.90 (0.85) | 4 | 0 | 1 | |
| Question: A patient presents with long-standing chest discomfort. If it turns out to be a chronic coronary syndrome, when was the diagnosis missed? | ||||||||
| 1. The number of contacts | 1 | 2.95 (1.27) | 3 | 2 | 2.95 (0.86) | 3 | 0 | 1.0 |
| 2 | 3.37 (1.31) | 4 | 2 | 3.66 (0.82) | 4 | 1 | 0.5 | |
| 3 | 3.78 (1.24) | 4 | 2 | 3.97 (0.64) | 4 | 0 | 1.0 | |
| 4 | 2.95 (1.26) | 3 | 2 | 3.07 (0.98) | 3 | 1 | 0.5 | |
| 5 | 3.47 (1.14) | 4 | 2 | 3.76 (0.84) | 4 | 0 | 1.0 | |
| 6 | 3.91 (1.16) | 4 | 2 | 4.19 (0.66) | 4 | 1 | 0.5 | |
| 7 | 3.16 (1.24) | 3 | 2 | 3.32 (0.78) | 3 | 1 | 0.5 | |
| 8 | 3.70 (1.12) | 4 | 2 | 4.02 (0.47) | 4 | 0 | 1.0 | |
| 9 | 4.05 (1.20) | 4 | 1 | 4.34 (0.58) | 4 | 1 | 0.0 | |
| 10 | 4.28 (1.18) | 5 | 1 | 4.59 (0.53) | 5 | 1 | 0.0 | |
| 11 | 3.78 (1.25) | 4 | 2 | 3.95 (0.78) | 4 | 0 | 1.0 | |
| 2. The course of time | 12 | 2.64 (1.34) | 2 | 2 | 2.59 (0.84) | 2 | 1 | 0.5 |
| 13 | 3.07 (1.39) | 3 | 2 | 3.13 (0.83) | 3 | 1 | 0.5 | |
| 14 | 3.26 (1.41) | 4 | 3 | 3.74 (0.81) | 4 | 0 | 1.0 | |
| 15 | 3.14 (1.32) | 3 | 2 | 3.30 (0.90) | 3 | 1 | 0.5 | |
| 16 | 3.79 (1.37) | 4 | 2 | 4.02 (0.74) | 4 | 0 | 1.0 | |
| 17 | 4.02 (1.28) | 4 | 1 | 4.33 (0.70) | 4 | 1 | 0.0 | |
| 18 | 4.26 (1.22) | 5 | 1 | 4.52 (0.74) | 5 | 1 | 0.0 | |
| 19 | 2.91 (1.20) | 3 | 2 | 3.20 (0.88) | 3 | 1 | 0.5 | |
| 20 | 3.19 (1.28) | 3 | 2 | 3.38 (0.89) | 3 | 1 | 0.5 | |
| 21 | 3.47 (1.40) | 4 | 3 | 3.84 (0.84) | 4 | 0 | 1.0 | |
| 22 | 2.67 (1.28) | 2 | 2 | 2.41 (0.76) | 2 | 1 | 0.5 | |
| 23 | 3.02 (1.29) | 3 | 2 | 2.98 (0.81) | 3 | 1 | 0.5 | |
| 24 | 3.36 (1.40) | 3.5 | 3 | 3.34 (0.75) | 3 | 1 | 0.7 | |
| 3. The degree of (cardiac) damage | 25 | 3.78 (1.14) | 4 | 2 | 3.87 (0.61) | 4 | 0 | 1.0 |
| 26 | 4.03 (1.15) | 4 | 1 | 4.19 (0.62) | 4 | 1 | 0.0 | |
| 27 | 3.87 (1.09) | 4 | 2 | 4.08 (0.61) | 4 | 0 | 1.0 | |
| 28 | 4.10 (1.03) | 4 | 1 | 4.15 (0.63) | 4 | 0.75 | 0.3 | |
| 29 | 3.76 (1.10) | 4 | 2 | 4.15 (0.49) | 4 | 0 | 1.0 | |
| 30 | 3.97 (1.03) | 4 | 1 | 4.15 (0.64) | 4 | 1 | 0.0 | |
Corresponding definitions can be found based on theme and item number in online supplemental Table S2.
CG, convergence of group.
In total, 26% (n=11) of the definitions for delayed ACS diagnosis and 29% (n=12) for missed ACS diagnosis reached high support and medium consensus (1≤IQR≤1.5).
For CCS (table 4), the percentage of definitions that reached high support (median≥4) in round 3 was 100% (n=30) for delayed CCS and 63% (n=19) for missed CCS. However, no definition reached high support and high consensus (IQR<1). In total, 23% (n=7) of the definitions for delayed CCS diagnosis and 23% (n=7) for missed CCS diagnosis reached high support and medium consensus (1≤IQR≤1.5).
Delphi round 4
For ACS (table 3), the percentage of definitions that reached high support (median≥4) in round 4 was 95% (n=40) for delayed ACS and 76% (n=32) for missed ACS. The percentage of definitions with high support and high consensus (IQR<1) increased in round 4 from 0% to 67% (n=28) for delayed ACS definitions and from 0% to 33% (n=14) for missed ACS definitions.
For CCS (table 4), the percentage of definitions that reached high support (median≥4) in round 4 was 100% (n=30) for delayed CCS and 63% (n=19) for missed CCS. The percentage of definitions with high support and high consensus (IQR<1) increased in round 4 from 0% to 80% (n=24) for delayed CCS definitions and from 0% to 37% (n=11) for missed CCS definitions. The same analyses were performed in the HCPs-only sample (cardiologists and GPs) and are reported in online supplemental Tables S5 and S6. The results are shortly addressed in the additional analyses section.
Recommended definitions for research purposes
Definitions were classified into levels of recommendation according to the criteria in figure 1. For delayed ACS, 2 definitions were classified as highly recommended, 13 as recommended and 6 as considerable. For missed ACS, none met the threshold for highly recommended; 5 were recommended, and 12 were classified as considerable.
For CCS, 2 definitions for delayed diagnosis were highly recommended, 13 recommended and 7 considerable. For missed CCS, 1 definition was highly recommended, 6 were recommended and seven were considerable. An overview of all recommended definitions for missed and delayed ACS and CCS is provided in tables5 6. The definitions that were not recommended based on the criteria in figure 1 were excluded from tables5 6.
Table 5. Recommended definitions of ACS for research purposes based on the predefined criteria (see figure 2).
| Recommendation level | Item number | Theme* | Definition |
|---|---|---|---|
| Definitions for a delayed ACS diagnosis | |||
| Highly recommended | 7 | 2 | If after initial symptom presentation to the general practitioner (or triagist, assistant) it takes >30 min before referral to the cardiac emergency department. |
| 20 | 2 | If the diagnosis is not made within 4 hours after onset of symptoms at the (cardiac) emergency department. | |
| Recommended | 1 | 1 | If at the first contact, action is not taken according to the guideline for suspected ACS (ie, referral to a cardiologist). |
| 2 | 1 | If at the first two contacts, action is not taken according to the guideline for suspected ACS (ie, referral to cardiologist). | |
| 8 | 2 | If after initial symptom presentation to the general practitioner (or triagist, assistant) it takes >1 hour before referral to the cardiac emergency department. | |
| 11 | 2 | If after initial symptom presentation, it takes >1 hour before the patient is assessed by a healthcare professional. | |
| 12 | 2 | If the patient is presented at the hospital >6 hours after onset of symptoms. | |
| 16 | 2 | If it takes >2 hours before the diagnosis is made after the onset of symptoms. | |
| 21 | 2 | If in a patient with a STEMI it takes >90 min from the presentation of the patient at the hospital to the start of the percutaneous coronary intervention (door-to-needle time). | |
| 23 | 2 | If a STEMI is not diagnosed within 90 min after symptom onset. | |
| 24 | 2 | If a STEMI is not diagnosed within 3 hours after symptom onset. | |
| 37 | 3 | If avoidable damage to the heart has occurred. | |
| 38 | 3 | If the emergency intervention (percutaneous coronary intervention, emergency coronary artery bypass) was delayed, causing more damage (eg, a regional wall movement disorder) than if the treatment had been carried out immediately. | |
| 41 | 3 | If the patient has consequences from the ACS that could most likely have been prevented with earlier treatment. | |
| 42 | 3 | If the recovery after the ACS most likely takes longer than if treatment had been provided sooner. | |
| Considerable | 4 | 1 | If during the first contact with the cardiologist, the additional examination (eg, ECG, troponin determination) is misinterpreted. |
| 5 | 1 | If a patient with an ACS is admitted to a non-cardiac ward at the first contact with the hospital, with a wrong diagnosis and no appropriate follow-up examination and treatment for ACS is performed. | |
| 13 | 2 | If the GP waits after the first contact and >6 hours later still refers to the cardiologist. | |
| 17 | 2 | If it takes >12 hours before the diagnosis is made after the onset of symptoms. | |
| 22 | 2 | If a STEMI is not diagnosed within 90 min after hospital entry. | |
| 27 | 2 | If an NSTEMI is not diagnosed within 12 hours after symptom onset. | |
| Definitions for a missed ACS diagnosis | |||
| Highly recommended | – | – | None met the criteria |
| Recommended | 17 | 2 | If it takes >12 hours before the diagnosis is made after the onset of symptoms. |
| 22 | 2 | If a STEMI is not diagnosed within 90 min after hospital entry. | |
| 37 | 3 | If avoidable damage to the heart has occurred. | |
| 38 | 3 | If the emergency intervention (percutaneous coronary intervention, emergency coronary artery bypass) was delayed, causing more damage (eg, a regional wall movement disorder) than if the treatment had been carried out immediately. | |
| 42 | 3 | If the recovery after the ACS most likely takes longer than if treatment had been provided sooner. | |
| Considerable | 1 | 1 | If at the first contact, action is not taken according to the guideline for suspected ACS (ie, referral to a cardiologist). |
| 9 | 2 | If after initial symptom presentation to the general practitioner (or triagist, assistant), the patient is referred to the cardiac emergency department the next day. | |
| 12 | 2 | If the patient is presented at the hospital >6 hours after onset of symptoms. | |
| 13 | 2 | If the GP waits after the first contact and >6 hours later still refers to the cardiologist. | |
| 23 | 2 | If a STEMI is not diagnosed within 90 min after symptom onset. | |
| 24 | 2 | If a STEMI is not diagnosed within 3 hours after symptom onset. | |
| 27 | 2 | If an NSTEMI is not diagnosed within 12 hours after symptom onset. | |
| 34 | 2 | If the patient reports his or her symptoms to a healthcare provider a day after the onset of symptoms. | |
| 36 | 2 | If the patient has not been assessed within 6 hours after the onset of symptoms. | |
| 8 | 2 | If after initial symptom presentation to the general practitioner (or triagist, assistant) it takes >1 hour before referral to the cardiac emergency department. | |
| 39 | 3 | If emergency intervention can no longer take place (the suitable time window has expired) | |
| 41 | 3 | If the patient has consequences from the ACS that could most likely have been prevented with earlier treatment. | |
Themes: 1=the number of contacts; 2=the course of time; 3=the degree of (cardiac) damage.
ACS, acute coronary syndrome; GP, general practitioner; NSTEMI, non-ST-elevation myocardial infarction; STEMI, ST-elevation myocardial infarction.
Table 6. Recommended definitions of CCS for research purposes based on the predefined criteria (see figure 2).
| Recommendation level | Item number | Theme | Definition |
|---|---|---|---|
|
Definitions for a delayed CCS diagnosis | |||
| Highly recommended | 2 | 1 | If a CCS diagnosis is not considered at the first two contacts and the patient is not referred to the cardiologist. |
| 4 | 1 | If a CCS diagnosis is not considered at the first contact and no trial treatment with anti-anginal medication is initiated. | |
| Recommended | 3 | 1 | If a CCS diagnosis is not considered at the first three contacts and the patient is not referred to the cardiologist. |
| 7 | 1 | If during the first contact, the (working) diagnosis of CCS is not established. | |
| 11 | 1 | If the cause of the symptoms was first attributed to a different pathology and a CCS diagnosis was discovered by chance at a later time. | |
| 14 | 2 | If after initial symptom presentation with stable angina, a patient is not clinically assessed within 3 days. | |
| 15 | 2 | If it takes >1 month to diagnose CCS after initial symptom presentation. | |
| 16 | 2 | If it takes >3–4 months to diagnose CCS after initial symptom presentation. | |
| 21 | 2 | If the patient is seen by a cardiologist >6–7 weeks after referral. | |
| 25 | 3 | If avoidable damage to the heart has occurred. | |
| 26 | 3 | If a CCS has become an ACS because the diagnosis was not made and no treatment was started. | |
| 27 | 3 | If the symptoms progress to unstable anginal symptoms. | |
| 28 | 3 | If the patient dies. | |
| 29 | 3 | If the quality of life is reduced because the condition has not been treated in time. | |
| 30 | 3 | When optimal prognosis is no longer possible (lower exercise tolerance than if treatment had been started earlier). | |
| Considerable | 5 | 1 | If a CCS diagnosis is not considered at the first two contacts and no trial treatment with anti-anginal medication is initiated. |
| 6 | 1 | If a CCS diagnosis is not considered at the first three contacts and no trial treatment with anti-anginal medication is initiated. | |
| 8 | 1 | If during the first two contacts, the (working) diagnosis of CCS is not established. | |
| 16 | 2 | If it takes >3–4 months to diagnose CCS after initial symptom presentation. | |
| 19 | 2 | If the patient is seen by a cardiologist >3 weeks after referral. | |
| 20 | 2 | If the patient is seen by a cardiologist >4–5 weeks after referral. | |
| 22 | 2 | If the patient does not immediately go to the doctor after the onset of symptoms. | |
| Definitions for a missed CCS diagnosis | |||
| Highly recommended | 28 | 3 | If the patient dies. |
| Recommended | 3 | 1 | If a CCS diagnosis is not considered at the first contact and the patient is not referred to the cardiologist. |
| 8 | 1 | If during the first two contacts, the (working) diagnosis of CCS is not established. | |
| 16 | 2 | If it takes >3–4 months to diagnose CCS after initial symptom presentation. | |
| 25 | 3 | If avoidable damage to the heart has occurred. | |
| 27 | 3 | If the symptoms progress to unstable anginal symptoms. | |
| 29 | 3 | If the quality of life is reduced because the condition has not been treated in time. | |
| Considerable | 5 | 1 | If a CCS diagnosis is not considered at the first two contacts and no trial treatment with anti-anginal medication is initiated. |
| 6 | 1 | If a CCS diagnosis is not considered at the first three contacts and no trial treatment with anti-anginal medication is initiated. | |
| 11 | 1 | If the cause of the symptoms was first attributed to a different pathology and a CCS diagnosis was discovered by chance at a later time. | |
| 14 | 2 | If after initial symptom presentation with stable angina, a patient is not clinically assessed within 3 days. | |
| 21 | 2 | If the patient is seen by a cardiologist >6–7 weeks after referral. | |
| 26 | 3 | If a CCS has become an ACS because the diagnosis was not made and no treatment was started. | |
| 30 | 3 | When optimal prognosis is no longer possible (lower exercise tolerance than if treatment had been started earlier). | |
Themes: 1=the number of contacts; 2=the course of time; 3=the degree of (cardiac) damage.
ACS, acute coronary syndrome; CCS, chronic coronary syndrome.
Additional analysis
In the additional analyses of the HCPs only sample (online supplemental Tables S5 and S6), we found the combination of high support and consensus in 52% (n=22) of the definitions for delayed ACS diagnosis and in 19% (n=8) of the definitions for missed ACS diagnosis (online supplemental Table S5). For CCS (online supplemental Table S6), the percentage of definitions that reached high support and high consensus for delayed CCS diagnosis was 66% (n=20), and 33% (n=10) for missed CCS diagnosis.
To assess the effect of the conditional decision rule, we examined how tables5 6 would appear when applying the classification criteria to the overall sample only and compared this with the final tables5 6 incorporating the HCPs-only sample to identify which definitions changed recommendation category. It showed that for delayed ACS, 11 definitions (26%) changed classification. Two definitions were upgraded from recommended to highly recommended (items 7 and 20) and one from considered to recommended (item 42). Eight definitions were downgraded from recommended to considerable (items 4, 5, 13, 17, 22, 27, 28 and 40). For missed ACS, 12 definitions (29%) changed classification. Three definitions were upgraded from not recommended to considerable (items 9, 39 and 41). Nine definitions were downgraded from recommended to considered (items 1, 8, 12, 13, 23, 24, 27, 34 and 36). For delayed CCS, 8 definitions (27%) changed classification. One item was upgraded from not recommended to considerable (item 6), and one from highly recommended to recommended (item 2). Six definitions were downgraded from recommended to considerable (items 5, 8, 16, 19, 20 and 22). For missed CCS, 7 items (23%) changed classification. Three definitions were upgraded from not recommended to considerable (items 6, 26 and 30). Four items were downgraded from recommended to considerable (items 5, 11, 14 and 21). Overall, incorporating the conditional decision rule based on the HCPs-only sample resulted in a limited number of classification shifts, with a predominance of downgrades, indicating a stricter classification than when based solely on the overall sample.Second, we tested for differences in agreement levels between HCPs and non-HCPs for all definitions (online supplemental Tables S7 and S8). The results indicate that when HCPs and non-HCPs differed in their level of agreement, HCPs more often had a lower mean score (indicating lower support). Except for three ACS definitions (items 4, 5, 39) for which HCPs scored higher, HCPs were more discriminative, while non-HCPs more often scored in the middle of the 5-point Likert scale.
Lastly, it was tested whether scoring between GPs and cardiologists differed per definition (online supplemental Tables S9 ad S10). Only mean score differences for definitions for which high consensus (IQR <1) was found will be further outlined. For missed CCS diagnosis definitions, significantly higher mean scores were found in the GP group compared with the cardiologists for definitions 8, 29 and 30. For all other definitions, the mean score did not differ between GPs and cardiologists.
Discussion
In this Delphi study, we aimed to develop a clear, consensus-based definition of delayed and missed diagnosis for acute and CCS to enable comparability across studies and improve research on the incidence, prevalence and determinants of diagnostic delays. To our knowledge, this is the first study to systematically address this need. The study produced a structured framework and a set of recommended definitions for use in future research. A key finding was that there is no single definition applicable across all clinical contexts. What constitutes a delayed or missed diagnosis depends on the patient’s position in the healthcare pathway and the availability of diagnostic tests at that point. Differences in perspectives among cardiologists, GPs, patients and healthy citizens further highlight the difficulty of establishing a universal definition. Importantly, the distinction between a delayed and a missed diagnosis is not always clearly binary, which explains the conceptual overlap between some definitions. Although our findings indicate that a fully context-independent definition is not achievable, the framework and recommended definitions developed provide a transparent and practical foundation for research into diagnostic delay in ACS/CCS diagnosis. A summary figure illustrating how researchers may apply the framework is presented in figure 4. Definitions based on the number of contacts (theme 1) are well suited for longitudinal routine (primary care) electronic health records. For example, for studying diagnostic delay in CCS, definition item 2 of theme 1 can be used. This could be operationalised by identifying a cohort of patients with a confirmed CCS diagnosis and retrospectively searching their electronic health records to quantify the frequency of consultations for chest discomfort prior to the referral to a cardiologist. Definitions based on the course of time (theme 2) are suited for datasets containing timestamps, such as hospital records. To assess delays in ACS diagnosis, researchers could apply the recommended definition (theme 2, item 21) by calculating the interval between hospital arrival and start of the percutaneous coronary intervention and determining whether this interval exceeds 90 min. Finally, definitions based on the degree of (cardiac) damage (theme 3) can be operationalised by linking diagnostic trajectories with clinical outcome data. For example, within the domain of missed CCS, definition item 28, theme 3 could be applied by linking national mortality registries with electronic health records to identify patients whose cause of death is attributed to ischaemic heart disease but lack any preceding recording of CAD in their medical history.
Figure 4. Example of how researchers may select definitions from the framework based on available data sources. ACS, acute coronary syndrome; CAD, coronary artery disease; CCS, chronic coronary syndrome; EHR, electronic health record; GP, general practitioner.
Comparison with literature
The framework proposes three ways of defining delayed and missed ACS/CCS diagnosis: (1) the number of contacts between the patient and the HCP, (2) the course of time subdivided into delays in the healthcare system and patient delays and (3) the degree of (cardiac) damage, including heart damage, death, and the patient’s quality of life. For the first two ways, we distinguished between the GP and specialist (eg, cardiologist) to acknowledge differences in perspectives and the availability of diagnostic tests. These ways can be used individually or in combinations by researchers, depending on their study objectives and available data.
In current literature, a wide range of definitions for delayed and missed ACS diagnosis are used, complicating comparisons of incidence, determinants and outcomes between studies. In the review by Kwok et al, all 15 studies in their review used different definitions of ‘missed ACS,’ most of which fit within the three approaches identified in our framework.15 The most frequently used definitions consider time (days, weeks or hours) and the number of contacts. However, two definitions were based on malpractice claims, which do not fit into our framework. Although malpractice claims can offer valuable insights into individual cases of medical negligence or error, they are often limited by underreporting, selection bias and lack of generalisability to the broader population, making them unsuitable for epidemiological research.31
A second major outcome of this study are the recommendation tables derived from consensus in Delphi rounds 3 and 4 (tables5 6). These tables identify which definitions the expert panel considered most appropriate for research purposes. Interestingly, only a limited number of definitions achieved the highest level of recommendation: two for delayed ACS, two for delayed CCS and one for missed CCS diagnosis. No definitions for missed ACS diagnosis reached this threshold. This may reflect the strict consensus criteria applied, since we only classified definitions with high support (median >4) and no definitions with medium support (>3 median <4). Definitions with medium support may still be useful, although maintaining predefined criteria is generally advised to preserve methodological rigour in Delphi studies.32
Among the recommended definitions, one states that a ST-elevation myocardial infarction (STEMI) diagnosis is delayed if the time from hospital presentation to percutaneous coronary intervention exceeds 90 min (theme 2, definition 21). Similarly, a STEMI is considered missed if it remains undiagnosed 90 min after hospital arrival (theme 2, definition 22). However, inconsistencies were evident. For example, high-support definitions for delayed ACS diagnosis used differing time intervals—90 min, 2 hours and 3 hours after symptom onset (theme 2, definitions 12, 16, 24). This highlights the complexity of the subject and, consequently, the varying nature of the responses of the expert panel in our Delphi study. Additional research, such as additional Delphi rounds or face-to-face discussions, is warranted to refine and standardise these definitions for use in research and clinical practice.
The recommended definitions should also be considered in light of European Society of Cardiology (ESC) guidelines. The 2023 ESC guidelines for the management of ACS recommend that patients with suspected ACS should have an interpreted ECG within 10 min after first medical contact and high-sensitivity cardiac troponin result within 1 hour.33 If a patient has a STEMI, primary percutaneous intervention (PCI) should follow within 120 min after the first medical contact. For patients with a non-STEMI, coronary angiography is recommended within 24 hours of diagnosis.33 Our expert panel recommends a cut-off value of 90 min from hospital arrival to start of the PCI (Theme 2, definition 21), therefore a shorter timeframe than recommended by the 2023 ESC guideline on ACS.
The 2024 ESC guidelines for managing CCS recommend a stepwise diagnostic approach, starting with a clinical evaluation by the GP, including ECG and basic laboratory testing. If CCS remains suspected, referral for further cardiac assessment, including echocardiography, is advised.6 However, the 2024 CCS guidelines do not specify timeframes for referral, diagnostic confirmation or specialist assessment. Consequently, comparison with our time-based definitions of delayed and missed CCS diagnosis is not possible. More broadly, ESC guidelines primarily focus on timelines after the first in-hospital medical contact and may not encompass delays occurring at the patient level or within the primary care setting. As a result, they do not encompass all dimensions needed for research definitions of delayed and missed ACS/CCS diagnosis. Our framework, therefore, complements existing guideline-based timelines by providing broader, research-oriented criteria applicable across the full diagnostic pathway.
Strengths and limitations
This study has several strengths. To our knowledge, it is the first to systematically develop a consensus-based framework for defining delayed and missed ACS/CCS diagnosis, addressing an important gap in the literature. The use of a structured Delphi method allowed iterative feedback and consensus building among a diverse panel of stakeholders, including cardiologists, GPs, patients and healthy citizens. The framework accommodates multiple approaches: (1) number of contacts, (2) time intervals and (3) degree of cardiac damage, allowing researchers to select definitions suited to their study objectives and available data. Nevertheless, our results should be interpreted in the context of certain limitations. As expected, attrition across Delphi rounds occurred. It is unclear whether this had an impact on our findings. A comparison between participants who completed all rounds and those who missed at least one round showed similar characteristics, but the numbers were too low to perform statistics (see online supplemental table S11). Although we included a diverse panel of experts, including cardiologists, GPs, patients and healthy citizens, we may not have sufficiently included researchers in the field of delayed and missed ACS/CCS diagnosis. This could potentially limit the suitability of the proposed definitions in current and future datasets.18 34 Interventional cardiologists were underrepresented in the expert panel. This likely did not affect our findings, as we focused on the diagnostic pathway in which all types of cardiologists are involved, certainly when on call. There was selection in the recruitment of patients and healthy citizens favouring those with higher health literacy and greater engagement with healthcare systems than those with a lower health literacy in the general population. This may have introduced selection bias by affecting the prioritisation of certain definitions. Furthermore, our definitions for a delayed and missed ACS/CCS diagnosis were generated from the context of a symptomatic patient. For both ACS and CCS, we outlined a situation for the panellists with a patient with acute and long-standing chest discomfort, respectively. This makes our study results less applicable to studies investigating silent myocardial infarction, eg, in patients without signs of angina or chest discomfort.
The additional analyses revealed noteworthy differences in the level of support (median scoring) for various definitions among healthcare professionals (HCPs) and non-HCPs. These variations may indicate differences in understanding, awareness and knowledge regarding the subject matter. Despite the questionnaire being translated into layman’s terms, patients and healthy citizens reported difficulty in comprehending it. The responses may have been impacted by the complexity of the questionnaire, potentially leading to unreliable results if they misunderstood the questions. This could explain why non-HCPs more frequently scored average on the Likert scale. Lastly, we applied an additional conditional decision rule based on the HCPs-only sample in the final classification of definitions. Although this may have influenced the final recommendations by giving relatively more weight to the scoring by the HCPs, its actual impact was limited. Only a small proportion of definitions shifted category, and more downgrades than upgrades were observed, indicating a stricter rather than preferential effect.
Implications for research and/or practice
By providing a structured, consensus-based framework, this study offers researchers a practical tool to standardise the definition of delayed and missed ACS/CCS diagnosis, enabling more consistent comparisons across studies and strengthening epidemiological and clinical research. Improved comparability may enhance identification of determinants of diagnostic delays, aiding development of targeted interventions aimed at reducing delayed or missed ACS/CCS diagnosis. While outcome-based definitions (theme 3) are useful for identifying diagnostic errors, caution is needed to avoid circular reasoning in epidemiological studies. If the aim is to quantify the impact of diagnostic delay on clinical outcomes, contact-based (theme 1) or time-based definitions (theme 2) should be prioritised to maintain a clear distinction between the diagnostic process and its subsequent clinical consequences. The generalisability of our findings is influenced by variations in healthcare systems across countries. Our study was conducted in the Netherlands, a primary care-based healthcare system characterised by GP gatekeeping and restricted direct access to specialist care. Consequently, components of our framework that rely on the number of healthcare contacts and referral thresholds are applicable to healthcare systems with a similar healthcare structure, eg, Scandinavian countries and the UK.35 However, several components of the framework are independent of the healthcare system structure and therefore more widely applicable. Definitions based on the course of time and patient delay are time-based, centred around symptom onset, presentation and diagnostic confirmation. This is independent of health care organisation but based on measurable time intervals. For example, the highly recommended definition for delayed ACS diagnosis (theme 2, item 20): ‘If the diagnosis is not made within 4 hours after onset of symptoms at the (cardiac) emergency department’ is applicable to most of the Western healthcare systems. Also, definitions belonging to theme 3: ‘degree of (cardiac) damage’, are universally relevant because of their focus on patient outcomes. To enhance international validity, similar Delphi studies should be conducted in other healthcare settings.
Conclusions
We were unable to establish a one-size-fits-all definition for delayed and missed ACS/CCS diagnosis, reflecting the complexity and context-dependency of CAD diagnostic pathways. However, the developed consensus-based framework and recommended definitions provide a valuable basis for future research, enabling study comparisons across different diagnostic settings. This will ultimately enhance understanding of delayed and missed ACS/CCS diagnosis.
Supplementary material
Acknowledgements
We would like to thank all participants for participating in the Delphi rounds and sharing their opinions. This manuscript is based on work previously presented in the doctoral thesis of DL.36
Footnotes
Funding: This work was funded by the Dutch Heart Foundation (de Hartstichting) (grant number: 2020B004).
Provenance and peer review: Not commissioned; externally peer reviewed.
Patient consent for publication: Consent obtained directly from patient(s).
Ethics approval: The study was approved by the Medical Ethics Review Committee Utrecht, the Netherlands (reference number 22-081/DB) and complied with the Declaration of Helsinki. Participants gave informed consent to participate in the study before taking part.
Data availability free text: The data can be made available for researchers whose proposed use of the data has been approved at the request of the corresponding author, with a signed data access agreement.
Collaborators: This manuscript was written on behalf of the IMPRESS consortium.
Data availability statement
Data are available on reasonable request.
References
- 1.Ralapanawa U, Sivakanesan R. Epidemiology and the Magnitude of Coronary Artery Disease and Acute Coronary Syndrome: A Narrative Review. J Epidemiol Glob Health. 2021;11:169–77. doi: 10.2991/jegh.k.201217.001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kwok CS, Mallen CD. Missed acute myocardial infarction: an underrecognized problem that contributes to poor patient outcomes. Coron Artery Dis. 2021;32:345–9. doi: 10.1097/MCA.0000000000000975. [DOI] [PubMed] [Google Scholar]
- 3.O’Neal WT, Shah AJ, Efird JT, et al. Subclinical myocardial injury identified by cardiac infarction/injury score and the risk of mortality in men and women free of cardiovascular disease. Am J Cardiol. 2014;114:1018–23. doi: 10.1016/j.amjcard.2014.06.032. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.De Bacquer D, De Backer G, Kornitzer M, et al. Prognostic value of ischemic electrocardiographic findings for cardiovascular mortality in men and women. J Am Coll Cardiol. 1998;32:680–5. doi: 10.1016/s0735-1097(98)00303-9. [DOI] [PubMed] [Google Scholar]
- 5.Hwang D, Park S-H, Koo B-K. Ischemia With Nonobstructive Coronary Artery Disease: Concept, Assessment, and Management. JACC Asia . 2023;3:169–84. doi: 10.1016/j.jacasi.2023.01.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Vrints C, Andreotti F, Koskinas KC, et al. ESC Guidelines for the management of chronic coronary syndromes: Developed by the task force for the management of chronic coronary syndromes of the European Society of Cardiology (ESC) Endorsed by the European Association for Cardio-Thoracic Surgery (EACTS) Eur Heart J. 2024;45:3415–537. doi: 10.1093/eurheartj/ehae177. [DOI] [PubMed] [Google Scholar]
- 7.Bouma MD, De Vries H, Loogman Mc, et al. Stabiele angina pectoris 2019. [3-Dec-2025]. https://richtlijnen.nhg.org/standaarden/stabiele-angina-pectoris#volledige-tekst Available. Accessed.
- 8.Pope JH, Aufderheide TP, Ruthazer R, et al. Missed Diagnoses of Acute Cardiac Ischemia in the Emergency Department. N Engl J Med. 2000;342:1163–70. doi: 10.1056/NEJM200004203421603. [DOI] [PubMed] [Google Scholar]
- 9.van der Ende MY, Juarez‐Orozco LE, Waardenburg I, et al. Sex‐Based Differences in Unrecognized Myocardial Infarction. JAHA. 2020;9 doi: 10.1161/JAHA.119.015519. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Margolis JR, Kannel WB, Feinleib M, et al. Clinical features of unrecognized myocardial infarction—silent and symptomatic: eighteen year follow-up: the Framingham study. Am J Cardiol. 1973;32:1–7. doi: 10.1016/S0002-9149(73)80079-7. [DOI] [PubMed] [Google Scholar]
- 11.Feringa HHH, Karagiannis SE, Vidakovic R, et al. The prevalence and prognosis of unrecognized myocardial infarction and silent myocardial ischemia in patients undergoing major vascular surgery. Coron Artery Dis. 2007;18:571–6. doi: 10.1097/MCA.0b013e3282f08e86. [DOI] [PubMed] [Google Scholar]
- 12.van Oortmerssen JAE, Zuo L, Tilly MJ, et al. Long-term prognosis of unrecognized myocardial infarction in women and men from the general population: the Rotterdam Study. Eur J Prev Cardiol. 2025 doi: 10.1093/eurjpc/zwaf689. [DOI] [PubMed] [Google Scholar]
- 13.Smolderen KG, Spertus JA, Nallamothu BK, et al. Health care insurance, financial concerns in accessing care, and delays to hospital presentation in acute myocardial infarction. JAMA. 2010;303:1392–400. doi: 10.1001/jama.2010.409. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Nguyen HL, Saczynski JS, Gore JM, et al. Age and sex differences in duration of prehospital delay in patients with acute myocardial infarction: a systematic review. Circ Cardiovasc Qual Outcomes. 2010;3:82–92. doi: 10.1161/CIRCOUTCOMES.109.884361. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kwok CS, Bennett S, Azam Z, et al. Misdiagnosis of Acute Myocardial Infarction: A Systematic Review of the Literature. Crit Pathw Cardiol. 2021;20:155–62. doi: 10.1097/HPC.0000000000000256. [DOI] [PubMed] [Google Scholar]
- 16.Valensi P, Lorgis L, Cottin Y. Prevalence, incidence, predictive factors and prognosis of silent myocardial infarction: a review of the literature. Arch Cardiovasc Dis. 2011;104:178–88. doi: 10.1016/j.acvd.2010.11.013. [DOI] [PubMed] [Google Scholar]
- 17.Sigurdsson E, Thorgeirsson G, Sigvaldason H, et al. Prevalence of coronary heart disease in Icelandic Men 1968-1986: The Reykjavik Study. Eur Heart J. 1993;14:584–91. doi: 10.1093/eurheartj/14.5.584. [DOI] [PubMed] [Google Scholar]
- 18.Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol. 2021;11:116–29. doi: 10.5662/wjm.v11.i4.116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Dalkey N, Helmer O. An Experimental Application of the DELPHI Method to the Use of Experts. Manage Sci. 1963;9:458–67. doi: 10.1287/mnsc.9.3.458. [DOI] [Google Scholar]
- 20.Jünger S, Payne SA, Brine J, et al. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017;31:684–706. doi: 10.1177/0269216317690685. [DOI] [PubMed] [Google Scholar]
- 21.Gattrell WT, Logullo P, van Zuuren EJ, et al. ACCORD (ACcurate COnsensus Reporting Document): A reporting guideline for consensus methods in biomedicine developed via a modified Delphi. PLoS Med. 2024;21:e1004326. doi: 10.1371/journal.pmed.1004326. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Page S. The difference: how the power of diversity creates better groups, firms, schools, and societies. 2025
- 23.Hong L, Page SE. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. Proc Natl Acad Sci USA. 2004;101:16385–9. doi: 10.1073/pnas.0403723101. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.HA L. In: Handbook of Futures Research. J F, editor. Esport: Greenwood Press; 1978. The delphi technique; pp. 271–300. [Google Scholar]
- 25.Mullen PM. Delphi: myths and reality. J Health Organ Manag. 2003;17:37–52. doi: 10.1108/14777260310469319. [DOI] [PubMed] [Google Scholar]
- 26.Boel A, Navarro-Compán V, Landewé R, et al. Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome. J Clin Epidemiol. 2021;129:31–9. doi: 10.1016/j.jclinepi.2020.09.034. [DOI] [PubMed] [Google Scholar]
- 27.Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101. doi: 10.1191/1478088706qp063oa. [DOI] [Google Scholar]
- 28.Howitt D, Cramer D. Introduction to qualitative methods in psychology: Pearson Education Limited. 2016. Ensuring qualitative research; pp. 224–447. [Google Scholar]
- 29.Gracht HA. Consensus measurement in Delphi studies: review and implications for future quality assurance. Technol Forecast Soc Change. 2012;79:1525–36. [Google Scholar]
- 30.Ray PK, Sahu S. Productivity Management in India: A Delphi Study. International Journal of Operations & Production Management. 1990;10:25–51. doi: 10.1108/01443579010005245. [DOI] [Google Scholar]
- 31.Balogh EP, Miller BT, Ball JR. Committee on diagnostic error in health care; board on health care services; institute of medicine; the national academies of sciences, engineering, and medicine. improving diagnosis in health care. improving diagnosis in health care. 2015 [PubMed]
- 32.Webbe J, Allin B, Knight M, et al. How to reach agreement: the impact of different analytical approaches to Delphi process results in core outcomes set development. Trials. 2023;24:345. doi: 10.1186/s13063-023-07285-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Byrne RA, Rossello X, Coughlan JJ, et al. ESC Guidelines for the management of acute coronary syndromes: Developed by the task force on the management of acute coronary syndromes of the European Society of Cardiology (ESC) Eur Heart J Acute Cardiovasc Care. 2023;13:55–161. doi: 10.1093/ehjacc/zuad107. [DOI] [PubMed] [Google Scholar]
- 34.Niederberger M, Spranger J. Delphi Technique in Health Sciences: A Map. Front Public Health. 2020;8:457. doi: 10.3389/fpubh.2020.00457. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Steeman L, Uijen M, Plat E, et al. Out-of-hours primary care in 26 European countries: an overview of organizational models. Fam Pract. 2020;37:744–50. doi: 10.1093/fampra/cmaa064. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Van Schalkwijk DL. Beyond the Heart: Exploring Patients’ and Healthcare Providers’ Perspectives in Cardiovascular Care. Tilburg University; 2024. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Data are available on reasonable request.