Abstract
Objective
To develop and validate a symptom-based prediction rule for early recognition of acute coronary syndrome (ACS) in patients with acute chest discomfort who call out-of-hours services for primary care (OHS-PC).
Design
Cross-sectional study. A diagnostic prediction rule was developed with multivariable regression analyses. All models were validated with internal-external cross validation within seven OHS-PC locations. Both age and sex were analysed as statistical interaction terms, applying for age non-linear effects.
Setting
Seven OHS-PC in the Netherlands.
Participants
2192 patients who called OHS-PC for acute chest discomfort (pain, pressure, tightness or discomfort) between 2014 and 2017. Backed up recordings of telephone triage conversations were analysed.
Primary and secondary outcomes measures
Diagnosis of ACS retrieved from the patient’s medical records in general practice, including hospital specialists discharge letters. Performance of the prediction rules was calculated with the c-statistic and the final model was chosen based on net benefit analyses.
Results
Among the 2192 patients who called the OHS-PC with acute chest discomfort, 8.3% females and 15.3% males had an ACS. The final diagnostic model included seven predictors (sex, age, acute onset of chest pain lasting less than 12 hours, a pressing/heavy character of the pain, radiation of the pain, sweating and calling at night). It had an adjusted c-statistic of 0.77 (95% CI 0.74 to 0.79) with good calibration.
Conclusion
The final prediction model for ACS has good discrimination and calibration and shows promise for replacing the existing telephone triage rules for patients with acute chest discomfort in general practice and OHS-PC.
Trial registration number
NTR7331.
Keywords: primary care, myocardial infarction, telemedicine
Strengths and limitations of this study.
We could analyse the original and very first telephone conversation of patients with acute chest discomfort.
The developed prediction model can be well generalised to other out-of-hours services for primary care (OHS-PC) locations in the Netherlands, but also to similar OHS-PC settings in other countries or even emergency medical service settings.
A limitation is that a full external validation of the model in another OHS-PC was impossible because no other cohort similar to ours was available.
Another limitation is that the effects of the predictors were assumed to be similar for males and females while this is not exactly the case, but by incorporating a differential non-linear effect of age and interaction with sex in the analyses this effect is minimalised.
Introduction
Chest discomfort is among the top five reasons for telephone contact in out-of-hours services for primary care (OHS-PC) and concerns 5% of all cases at the emergency department (ED) in the USA.1 In the Netherlands, around 80% of patients with chest discomfort first call the general practitioner (GP) or OHS-PC, while 20% directly calls the emergency medical service (EMS, or ambulance dispatch centre) or are self-referrals to the ED.2 Adequate triage and early diagnosis in these patients is vital, because in case of an underlying acute coronary syndrome (ACS) early effective therapeutic interventions (‘time is muscle’) improve the patient’s outcome and prognosis.3 For the diagnosis of ACS, a 12-lead ECG and troponin testing is needed.3 However, before the patient is referred to an ED where these diagnostic tests can be done, patient selection is necessary based on symptom presentation retrieved by telephone triage.4 5 Symptom-based differentiation of ACS from other causes of chest discomfort is notoriously difficult.6 Symptom-based prediction rules for diagnosing ACS in general practice and other prehospital settings are—although highly needed—scarce.7–9 The efficiency and safety of telephone triage in OHS-PC was poor in a population with a prior chance of ACS of 8.3% in females and 15.3% in males; almost 50% of the males and females with chest discomfort received a high priority ambulance, while 11% diagnosed with an ACS did not receive a high urgency (ie, was seen within 1 hour).10
Most prediction rules for diagnosing ACS were developed in the ED setting and include results from ECG and troponin testing.11 Such prediction rules cannot be straightforward implemented for telephone triage in general practice because (1) in the latter setting these diagnostic tests are not available, (2) the prior chance of ACS is rather low and (3) on average disease severity is less than in those seen in the ED.12 13 The prevalence of ACS among patients with chest discomfort who call OHS-PC or EMS is about 10%–15%, and among those seen at the ED between 10% and 30%.10 11 13 14 Only one prediction rule was developed in primary care for diagnosing ACS; the modified Grijseels prediction rule, which had moderate discriminative ability (c-statistic of 0.66) after external validation.7 8 Five other primary care prediction rules were developed to predict CAD; for example, the Marburg Heart Score (MHS) and INTERCHEST prediction rule (International Working Group on Chest Pain in Primary Care).9 In these studies both patients with acute and non-acute chest discomfort were included and the prevalence of stable CAD showed to be 10.9%–12.6%, while that of ACS was only 1.5%–2.5%.9 Thus, these prediction rules have limited applicability for specifically diagnosing ACS.
In most OHS-PC and about half of the EMS in the Netherlands, triage nurses use the semi-automatic ‘Netherlands Triage Standard’ (NTS) as a decision support tool to classify the urgency of the patient’s condition. Triage nurses have to choose one out of 56 ‘main complaints’ and based on answers linked to the triage criteria, the NTS automatically proposes one out of six levels of urgency, that is, a certain time frame in which patients should be seen (U0-U5, online supplemental appendix table 1). The NTS is a modified and shortened version of the Manchester Triage Standard which was developed in the ED setting.15 Although the NTS was explicitly developed for telephone triage, it has not yet been validated against clinical outcomes even though it is already implemented on large scale. Recent research showed that the NTS had a poor sensitivity of 0.73 (95% CI 0.68 to 0.78) for telephone triage of patients with acute chest discomfort. The NTS recommended a low urgency (U3, U4 or U5) to 27% of patients who eventually showed to have an ACS or other life threatening event (LTE).10 The NTS’ specificity was also poor with 0.43 (95% CI 0.40 to 0.45); the NTS recommended a high urgency to 57% of the patients who eventually showed not to have an ACS or LTE. Given this poor safety and efficiency of the NTS, there is an urgent need for a better prediction model for patients with acute chest discomfort calling OHS-PC. In addition, there is a need for exploring sex-specificity of such a prediction rule as there is an ongoing debate on whether females differ from males in reporting symptoms of ACS.16 17
bmjopen-2022-064402supp001.pdf (152.2KB, pdf)
The aim of this study was to develop, and internal-external cross validate a symptom-based prediction rule for diagnosing ACS which is considerate to sex categories in male and female patients who call the OHS-PC for acute chest discomfort.
Methods
We performed a cross-sectional study among 2192 patients who called one out of seven participating OHS-PC in the Netherlands because of acute chest discomfort (pain, pressure, tightness or discomfort) during the period 2014–2017.18 These OHS-PC serve a total population of 1.5 million people and cover around 300 000 calls a year. We first selected calls based on the International Code for Primary Care (ICPC; a WHO world-wide code system for primary care) with ICPC-codes K01, K02, K03, K24, K74, K75, K76, K77, K93, L04, P74, R02, R98 and calls with keywords thoracic pain, chest pain, myocardial infarction, heart attack and their common abbreviations (figure 1). We included a broad variety of symptoms to capture the entire domain of patients that could be suspected of ACS. We listed all available calls of these patients and assigned random numbers with the Random Number Generator (RAND) function in Microsoft Excel to retrieve a random sample. Calls were excluded before relistening when the patients’ age was below 18 years or when the patient did not live in the surrounding area of the OHS-PC (in which case we could not retrieve the final diagnosis from the patient’s own GP). Calls were excluded during relistening when it did not concern a triage call (eg, intercollegial consultation) or when the recording was of poor quality (figure 1).
Figure 1.
Flowchart of study population. ACS, acute coronary syndrome; ICPC, International Code for Primary Care; OHS-PC, out-of-hours services for primary care.
Candidate predictors
Research team members (LTCMW, DCAE) and medical students listened to the call recordings, blinded for the outcome, to collect data about symptoms, medical history and urgency allocation. Patient (age, sex) and call characteristics (call time, call duration) were collected from the OHS-PC electronic medical files of the patients. As candidate predictors we included age and sex, the NTS triage criteria (see online supplemental appendix table 2), the ACS predictors from the modified Grijseels prediction model (male sex, radiation, nausea, sweating and history of CAD), the ‘CAD predictors’ from MHS and INTERCHEST prediction models (age, pain feels like pressure, CAD history or cardiovascular (CV) risk factors, patient assumes cardiac origin of pain), and—based on a recent own study in OHS-PC—the predictor ‘calling at night between 00:00 and 09:00’.7–9 19
Outcome
The primary outcome was the diagnosis ACS. The final diagnoses were retrieved from the patient’s GP and based on the GP’s electronic medical files which include ED and cardiologist discharge letters and notes from the OHS-PC contact. The diagnosis ACS was nearly always made by a cardiologist (96.0%) and included information on levels of (high-sensitivity) troponin and electrocardiographic results. We used medical information up to 30 days following the contact with the OHS-PC to allow us to include diagnoses of ACS that were initially missed because the patient was not referred to the cardiologist the same day of the OHS-PC contact. However, in none of the patients in the study we had evidence of a missed diagnosis of ACS.
Sample size calculation
We relied on the minimal sample size criteria for prediction model development proposed by Riley et al using the ‘pmsampsize’ package in R.20 Based on an ACS prevalence of overall 11% and a Cox-Snell R2 of 0.075 (a conservative value based on a model with age and sex) and a total number of 2192 observations we were allowed to assess 19 candidate predictors.21 Based on sample size calculations we concluded that development of separate models for males and females would require a significantly larger sample, therefore we analysed sex as a statistical interaction term.20
Statistical analyses
We developed three diagnostic models using multivariable logistic regression analysis. First, we developed a base model with only age and sex as predictors, where age was modelled using a restricted cubic spline function (four knots) and an interaction with sex. This resulted in a base model with seven predictor parameters (excluding the intercept). Second, we fitted a full model with an additional 12 preselected binary predictors (having chest pain, acute chest pain shorter than 12 hours, shortness of breath, sweating, retrosternal located chest pain, radiation, pressing heavy pain, stabbing pain, history of cardiovascular disease (CVD), history of CAD, someone else calling, calling at night). Third, we applied backward elimination, with a cut-off p value <0.20 for including predictors (a higher cut-off value to lower the chances of overfitting).21
We applied internal-external cross validation (IECV) for model validation using the seven different OHS-PC locations (online supplemental appendix table 3 for patient characteristics of different OHS-PC locations).22 We evaluated the IECV performance in terms of the area under the receiver operating characteristic curve (c-statistic), the calibration slope and calibration in the large. The IECV estimates of performance were combined by using random-effect model (DerSimonian-Laird estimator). Based on the IECV we also constructed flexible calibration curves and decision curves. In the decision curve analyses we compared the final model with the currently used NTS triage model in OHS-PC in the Netherlands.23 Finally, we created an illustrative table of diagnostic test accuracy for various model-based risk thresholds of the final model, following the example in Wynants et al.23 IECV estimates for risk threshold specific sensitivity and specificity, and we applied a bivariate model commonly used for diagnostic test accuracy meta-analysis.24
Missing data
For missing data we carried out multiple imputation using the Multivariate Imputation via Chained Equation package in R, with 30 imputation rounds and 30 iterations.25 We pooled the results following Rubin’s rules.26 Predictors with over 50% missing were excluded from consideration in the models (online supplemental appendix table 4 for details about the missing data). Characteristics were compared between patients with and without information on the medical outcome—because some GPs refused diagnosis retrieval from their files—to allow for assessment of differences in characteristics between these patient groups (online supplemental appendix table 5). There were no clinically meaningful differences in symptoms and patient or call characteristics between the 2192 patients with information on the outcome, and the 1012 patients about whom knowledge of the medical outcome related to the OHS-PC contact because of acute chest discomfort was missing.
All analyses were done in R V.4.0.3. (10 October 2020) with the Regression Modelling Strategies (‘rms’) package in R.27 We reported our study in accordance to the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis criteria (online supplemental file).28
Patient and public involvement
No patients were involved in defining the research question or the outcome measures. Neither they were involved in developing plans for design. However, they participated in the discussion on implications and the implementation strategy. In addition, they were asked to advise on interpretation and writing up of the results. Results will be shared and discussed in more detail with representatives of the Dutch national patient community of cardiovascular diseases (‘Harteraad’).
Results
Among the 2192 callers with acute chest discomfort (mean age 59.1 (SD 19.5) years and 55.3% females) 251 (11.5%) had a final diagnosis of ACS; 101 (8.3%) females and 150 (15.3%) males (table 1). Patients with ACS were older than those without (mean age 69.7 (SD 13.4) vs 57.7 (SD 19.8) years) and females with ACS were on average older than men with ACS (73.8 (SD 13.5) years vs 67.0 (SD 12.6) years).
Table 1.
Characteristics of 2192 patients who called OHS-PC with acute chest discomfort between 2014 and 2017, divided between females and males with and without ACS
| Characteristics | 1213 females (55.3%) | 979 males (44.7%) | ||
| ACS, n=101 (8.3%) | No ACS, n=1112 (91.7%) | ACS, n=150 (15.3%) | No ACS, n=829 (84.7%) | |
| Patient characteristics | ||||
| Mean age in years (SD) (n=2192) | 73.8 (13.5) | 58.0 (20.2) | 67.0 (12.6) | 57.2 (19.2) |
| Call characteristics | ||||
| Median call duration in min (IQR) (n=2192) | 5:27 (3:57–8:24) | 6:59 (5:06–9:47) | 6:04 (4:03–8:17) | 6:56 (5:10–9:23) |
| Mean introduction time in min (IQR) (n=2192) | 0:13 (0:09–0:18) | 0:17 (0:11–0:26) | 0:14 (0:09–0:21) | 0:17 (0:11–0:25) |
| Call during the night (00:00–09:00) (n=2192) | 34 (33.7) | 304 (27.3) | 62 (41.3) | 188 (22.7) |
| Triage nurse consulted the GP (n=2192) | 43 (42.6) | 580 (52.2) | 75 (50.0) | 449 (54.2) |
| Someone else called on behalf of patient (n=2192) | 69 (68.3) | 515 (46.3) | 98 (65.3) | 432 (52.1) |
| The person who calls expressed concerns (n=988) | 42 (95.5) | 507 (90.5) | 61 (96.8) | 378 (87.1) |
| Medical history and risk factors | ||||
| Cardiovascular disease or CV risk factors (n=1844) | 70 (81.4) | 552 (61.1) | 106 (78.5) | 464 (64.4) |
| History of coronary artery disease (n=1151) | 23 (47.9) | 131 (24.2) | 54 (56.3) | 181 (38.8) |
| Diabetes (n=893) | 14 (42.4) | 66 (14.3) | 22 (39.3) | 78 (22.0) |
| Hypertension (n=894) | 26 (72.2) | 162 (34.0) | 22 (51.2) | 113 (33.3) |
| Hypercholesterolaemia/statin use (n=826) | 10 (40.0) | 96 (22.6) | 27 (50.0) | 79 (24.5) |
| Cardiac arrhythmia (n=905) | 4 (14.8) | 125 (26.2) | 12 (25.0) | 89 (25.2) |
| Symptoms | ||||
| Chest pain (n=2116) | 95 (96.9) | 1007 (94.1) | 139 (93.3) | 758 (94.9) |
| Shortness of breath (n=1696) | 57 (71.3) | 559 (65.4) | 63 (61.2) | 415 (63.1) |
| Chest pain duration <12 hours (n=1910) | 74 (86.0) | 703 (72.3) | 113 (82.5) | 510 (71.3) |
| Severe pain (Numeric Rating Scale>7, range 1–10) (n=917) | 19 (61.3) | 184 (39.6) | 18 (25.4) | 116 (33.0) |
| Pressing/heavy chest pain* (n=1625) | 58 (81.7) | 525 (62.5) | 95 (81.2) | 345 (57.7) |
| Stabbing chest pain* (n=1625) | 8 (11.3) | 190 (22.6) | 9 (7.7) | 159 (26.6) |
| Retrosternal chest pain † (n=1565) | 36 (54.5) | 326 (40.0) | 52 (53.1) | 227 (38.7) |
| Chest pain left or right of thorax† (n=1566) | 19 (28.8) | 318 (39.0) | 28 (28.6) | 262 (44.6) |
| Radiation of chest pain to any location (n=1678) | 74 (86.0) | 575 (67.8) | 83 (65.4) | 347 (56.2) |
| Radiation to the arm‡ (n=1677) | 37 (43.0) | 218 (25.7) | 54 (42.5) | 143 (23.2) |
| Radiation to the shoulder blades‡ (n=1678) | 14 (16.3) | 190 (22.4) | 19 (15.0) | 103 (16.7) |
| Radiation to the jaws‡ (n=1678) | 10 (11.6) | 77 (9.1) | 4 (3.1) | 33 (5.3) |
| Sweating (n=1366) | 36 (52.9) | 279 (42.0) | 54 (51.9) | 190 (35.8) |
| Nausea or vomiting (n=987) | 24 (52.2) | 295 (56.6) | 31 (43.1) | 139 (39.9) |
| Pallor or ashen skin (n=673) | 22 (59.5) | 139 (44.3) | 36 (64.3) | 125 (46.8) |
| (Near) fainting (n=1951) | 8 (9.5) | 76 (7.7) | 9 (6.7) | 50 (6.7) |
| Palpitations (n=162) | 10 (100.0) | 183 (84.7) | 8 (50.0) | 83 (75.5) |
| Patient recognises symptoms from previous cardiac event (n=915) | 17 (35.4) | 100 (22.0) | 30 (46.9) | 103 (29.5) |
| Urgency allocation | ||||
| High urgency (U1 or U2) (n=2192) | 89 (88.1) | 740 (66.5) | 133 (88.7) | 534 (64.4) |
| U1 | 75 (74.3) | 443 (39.8) | 106 (70.7) | 350 (42.2) |
| U2 | 14 (13.9) | 297 (26.7) | 27 (18.0) | 184 (22.2) |
| Low urgency (U3 or U4 or U5) | 12 (11.9) | 372 (33.5) | 17 (11.3) | 295 (35.6) |
*Pain described by patient. Pressing heavy pain: pressing, heavy or tightening pain vs other types of pain (stabbing, burning, cramping, tearing). Stabbing pain: stabbing versus other types of pain (pressing, heavy, tightening, burning, cramping).
†Retrosternal location versus other pain locations. Left or right side of the thorax versus other pain locations.
‡Radiation location versus no radiation and radiation other pain.
ACS, acute coronary syndrome; CV, cardiovascular; GP, general practitioner; OHS-PC, out-of-hours services for primary care.
Over two-thirds of all callers (68.3%) received a high urgency allocation (seen within 1 hour; U1 or U2) and among the 251 patients who showed to have an ACS, 88.4% received a high urgency allocation. Calls of patients who had an ACS were shorter than calls in those without ACS (median call duration 6:34 (SD 3:38) vs 7:42 (SD 3:48) min).
Medical history and symptoms
Females and males with ACS had more often a history of CVD or CV risk factors than those without (females with ACS 81.4% vs females without ACS 61.1%, males with ACS 78.5% vs males without ACS 64.4%) (table 1). The majority of both females and males had chest pain (94.5%) and this was similar among those with and without ACS. Overall, presented symptoms among males and females calling the OHS-PC for chest discomfort were quite similar. Symptoms associated with ACS in both sexes were pressing/heavy chest pain (females with ACS 81.7% vs females without ACS 62.5%, males 81.2% vs 57.7%, respectively), retrosternal located chest pain (females 54.5% vs 40.0%, males 53.1% vs 38.7%), radiation of pain (females 86.0% vs 67.8%, males 65.4% vs 56.2%) and sweating (females 52.9% vs 42.0%, males 51.9% vs 35.8%).
Diagnoses
In total 251 patients were diagnosed with ACS and 65 with other LTEs, and of clinical relevance is that both critical events occurred significantly more in males than females (15.3% vs 8.3%, p<0.001 for ACS, and 3.8% vs 2.3%, p=0.04 for other LTEs, respectively). Of the 101 females with ACS, 22.8% had an ST-segment elevation myocardial infarction (STEMI), 46.5% a non-ST-segment elevation myocardial infarction (NSTEMI), 19.8% unstable angina pectoris (UAP) and 10.9% non-classified ACS. In 150 males with ACS, 33.3% had an STEMI, 36.7% an NSTEMI, 26.0% UAP and 4.0% non-classified ACS (table 2).
Table 2.
Diagnoses of 2192 males and females who contacted the OHS-PC for acute chest discomfort between 2014 and 2017, by sex
| Diagnosis, n (%) | Females n=1213 | Males n=979 | P value |
| Acute coronary syndrome* | 101 (8.3) | 150 (15.3) | <0.001 |
| STEMI | 23 (22.8) | 50 (33.3) | 0.071 |
| NSTEMI | 47 (46.5) | 55 (36.7) | 0.119 |
| UAP | 20 (19.8) | 39 (26.0) | 0.256 |
| Non-classified ACS | 11 (10.9) | 6 (4.0) | 0.033 |
| Life threatening events (LTEs) | 28 (2.3) | 37 (3.8) | 0.043 |
| Pulmonary embolism | 8 (28.6) | 10 (27.0) | 0.890 |
| Acute abdominal aneurysm | 2 (7.1) | 3 (8.1) | 0.885 |
| Thoracic aortic dissection | 1 (3.6) | 4 (10.8) | 0.278 |
| Other† | 17 (60.7) | 20 (54.1) | 0.591 |
| Non-urgent cardiovascular diseases‡ | 223 (18.4) | 191 (19.5) | 0.069 |
| Musculoskeletal pain | 245 (20.2) | 148 (15.2) | 0.039 |
| Non-cardiac chest pain, not further specified*§ | 191 (15.7) | 179 (18.3) | 0.012 |
| Psychogenic disorders | 165 (13.6) | 85 (8.7) | 0.005 |
| Gastrointestinal tract disorders | 89 (7.3) | 68 (6.9) | 0.776 |
| Respiratory tract disorders | 61 (5.0) | 56 (5.7) | 0.203 |
| Other non-urgent diagnoses¶ | 110 (9.1) | 65 (6.6) | 0.152 |
*Almost all patients (96.0%) were diagnosed by a cardiologist. Ten (4.0%) ACS patients were not diagnosed by a cardiologist; four died before arrival of the ambulance, one patient died after resuscitation at the ED (all these five were classified as acute cardiac death due to ACS), and in five patients the ACS diagnosis was solely based on the GP’s interpretation in patients who were not referred to the hospital after shared decision because of a short life expectancy due to cancer in a palliative stage.
†Stroke, severe COPD exacerbation, acute severe heart failure, sepsis, hypokalaemia, diabetic ketoacidosis, epileptic insult, bleeding from oesophageal varices, ovarian torsion, ventricular fibrillation.
‡Stable angina pectoris (including atypical chest pain), stable heart failure, arrhythmias, hypertension.
§Cardiac pathology unlikely after cardiologist’s or GP’s diagnostic work-up, but without differential diagnosis.
¶Among others: anaemia, carcinoma, vasovagal collapse, side effects medication, dermatological diseases (eg, herpes zoster infection).
ACS, acute coronary syndrome; COPD, chronic obstructive pulmonary disease; ED, emergency department; GP, general practitioner; NSTEMI, non-ST-segment elevation myocardial infarction; OHS-PC, out-of-hours services for primary care; STEMI, ST-segment elevation myocardial infarction; UAP, unstable angina pectoris.
Twenty-eight (2.3%) females and 37 males (3.8%) had another life-threatening event than ACS (eg, pulmonary embolism, thoracic aortic dissection, acute abdominal aneurysm). All other patients (85.6%) had non-urgent medical conditions such as non-urgent cardiovascular disease (18.9%), musculoskeletal pain (17.9%), non-cardiac chest pain (not further specified) (16.9%), psychogenic disorder (11.4%), gastrointestinal disorders (7.2%), respiratory disorders (5.3%) and other non-urgent diagnoses (8.0%).
Model development, performance and validation
The base model with sex and age had an apparent c-statistic of 0.72 (95% CI 0.70 to 0.75), and an internal-external validation based c-statistic of 0.72 (95% CI 0.68 to 0.75) (online supplemental appendix table 6). The basic model shows that the risk of ACS increases with age for both sexes, with a notable peak risk for men at an age near 60 years and a more gradual increase in risk of ACS for women (figure 2). The full model including all candidate predictors had an apparent c-statistic of 0.79 (95% CI 0.76 to 0.81) and an internal-external validation based c-statistic of 0.77 (95% CI 0.74 to 0.80) (online supplemental appendix table 7). The full model had optimal calibration (flexible line close to the 45 degree reference line) up to a predicted probability of ACS of approximately 0.2 (online supplemental appendix figure 1). Risks higher than 0.2 tended to be overestimated by the model, however, since any plausible risk threshold will be lower than 0.2 in the primary care setting, we find the calibration in the relevant range to be satisfactory. After backward elimination, the backward model had an apparent c-statistic of 0.79 (95% CI 0.76 to 0.81), and the internal-external validation c-statistic was 0.77 (95% CI 0.74 to 0.79) (Table 3) It had very similar calibration to the full model (Table 3, figure 3).
Figure 2.
Base model with age and sex for predicting diagnosis acute coronary syndrome. ACS, acute coronary syndrome.
Table 3.
Final model for predicting the diagnosis acute coronary syndrome
| Predictors | Regression coefficients (SE) |
| Intercept | −16.246 (3.527) |
| Age | 0.293 (0.081) |
| Age′ | −0.391 (0.125) |
| Age’″ | 1.063 (0.395) |
| Female gender | 2.504 (5.512) |
| Age*female gender | −0.096 (0.126) |
| Age′*female gender | 0.189 (0.195) |
| Age″*female gender | −0.556 (0.605) |
| Acute chest pain shorter than 12 hours | 0.290 (0.198) |
| Sweating | 0.457 (0.178) |
| Radiation of chest pain | 0.609 (0.176) |
| Pressing heavy pain | 0.747 (0.200) |
| Call during the night (00:00–09:00) | 0.504 (0.151) |
| Apparent c-statistic 0.79 (95% CI 0.76 to 0.81) Adjusted c-statistic 0.77 (95% CI 0.74 to 0.79) Calibration slope 0.826 (95% CI 0.658 to 0.994) Calibration −0.224 (95% CI −0.604 to 0.157) R2 0.106 Knots for cubic spline functions placed at 5, 35, 65 and 95 | |
Figure 3.
Callibration of the final model with internal external validation.
Decision curve analyses and risk thresholds
Both the full and backward model showed a high net benefit as compared with the currently used NTS model for telephone triage in OHS-PC (figure 4). There was no difference in net benefit between the full model and backward model across plausible risk thresholds. Based on this analysis we decided to choose the backward as the final triage tool model because; (1) with fewer predictors the prediction of ACS remained similar accurate and (2) no valuable time is lost during telephone triage by asking the patient about symptoms that do not contribute to a better prediction. The final model included besides age and sex, the five following predictors; (i) acute onset of chest pain lasting <12 hours, (ii) a pressing/heavy character, (iii) radiation of pain, (iv) sweating, (v) calling at night between 00.00 and 09.00. Finally, we evaluated the diagnostic performance of the final prediction model across risk thresholds that may be chosen to apply in clinical practice (table 4, figure 5).
Figure 4.
Decision curve analyses comparing the full and final models versus the currently used model and versus treat all patients. NTS, Netherlands Triage Standard.
Table 4.
Diagnostic accuracy for a range of risk thresholds of the final model
| Risk threshold | Sensitivity (95% CI) | Specificity (95% CI) | Positive predictive value | Negative predictive value |
| 0.001 | 0.98 (0.95 to 0.99) | 0.21 (0.18 to 0.24) | 0.14 | 0.99 |
| 0.010 | 0.98 (0.95 to 0.99) | 0.42 (0.40 to 0.45) | 0.18 | 0.99 |
| 0.020 | 0.97 (0.94 to 0.99) | 0.50 (0.47 to 0.54) | 0.20 | 0.99 |
| 0.050 | 0.93 (0.87 to 0.96) | 0.63 (0.59 to 0.67) | 0.25 | 0.99 |
| 0.075 | 0.88 (0.81 to 0.92) | 0.72 (0.68 to 0.76) | 0.29 | 0.98 |
| 0.100 | 0.81 (0.7 to 30.87) | 0.79 (0.76 to 0.82) | 0.33 | 0.97 |
| 0.115 (prevalence) | 0.76 (0.67 to 0.83) | 0.82 (0.79 to 0.85) | 0.36 | 0.96 |
| 0.150 | 0.64 (0.56 to 0.73) | 0.88 (0.85 to 0.90) | 0.41 | 0.95 |
| 0.200 | 0.46 (0.38 to 0.55) | 0.93 (0.91 to 0.94) | 0.46 | 0.93 |
Figure 5.
Runway plot of diagnostic accuracy measures of the final model.
Discussion
This is the first study that developed and internal-external validated a symptom-based prediction rule for telephone triage of ACS in male and female patients who contact OHS-PC for acute chest discomfort (pain, pressure, tightness or discomfort). ACS was present in 8.3% of the females and 15.3% of the males. The prediction rule is applicable for triage in the OHS-PC setting and consists of sex and age as statistical interaction terms, and five other symptom-based predictors. It had a good discriminative ability (adjusted c-statistic 0.77 (95% CI 0.74 to 0.79)) and was well calibrated up to an ACS risk of 0.2.
Strengths and limitations
The major strength of this study is that we analysed the original and very first conversations of patients with acute chest discomfort with primary healthcare providers and assessed these talks without knowledge of the diagnosis; the assessment of symptoms was therefore not affected by hindsight bias caused by knowledge of the final diagnosis.29 Furthermore, we could analyse a large sample (N=2192) of patients which allowed us to evaluate up to nineteen candidate predictors. We assessed the risk of selection due to missing outcome data, and our data suggest that this missingness was unlikely to bias our findings. Because we used data from seven different OHS-PC our results will be well generalisable to other OHS-PC in the Netherlands, but we anticipate the model might be applicable to similar OHS-PC setting in, for example, the UK and Scandinavian countries. Our results may also be generalisable to some EMS settings, because the prior chance of having an ACS among those calling for acute chest discomfort is rather similar in the EMS setting as in the OHS-PC setting.14 30
A limitation of the study is that a full external validation of the final model is not done yet. However, at the time of executing the study hardly any primary care research data were available to perform such validation. Importantly, the internal-external validation showed very good calibration up to an ACS risk of 0.2. Although we performed extensive internal-external validation making use of the datasets of nine sites with substantial differences in case mix, we will strive for formal external validation before it can be widely applied in everyday primary care practice. Another limitation is that the effects of the predictors were assumed to be similar for male and female patients while that might not be optimal for the predictions. However, development of separate models for males and females would require a significantly larger sample size than was available. Importantly, a differential non-linear effect of age was incorporated using a spline function and interaction with sex was incorporated, and the final internal-external validated model did have good overall performance.
Comparison with other studies
Our prediction model had a higher discriminating ability for ACS than the NTS (c-statistic of 0.58) and modified Grijseels prediction rule (c-statistic of 0.66).7 This may largely be explained by the addition of age, the strongest predictor of ACS. This is in line with the notion that the prevalence of ACS increases with age.7 9 31 Importantly, in our study among people aged below 40, only one (0.4%) male patient had an ACS (UAP). For males to the age of 55 we found a peak risk of ACS of around 20% and remaining at this level with further age increase onwards. For females we found a gradual increase of risk with age with a maximum ACS risk of around 18% for those aged over 80 years. Similar to the modified Grijseels prediction rule our prediction model includes sweating and radiation of pain, however, the modified Grijseels rule combined nausea and sweating to a single predictor (ie, nausea or sweating).7 8 Age and sex were predictors in our model, but also in the MHS and INTERCHEST prediction models.32 33 Also the INTERCHEST rule included pressing heavy chest pain as predictor.32 A new predictor is calling at night (between 00.00 and 09.00).19 Previous studies in the ED setting also showed circadian variability with an early morning peak for ACS patients.34 Finally, symptoms associated with ACS were rather similar between females and males, which is in line with recent sex-stratified studies of patients with chest discomfort who called the OHS-PC, but is in contrast with the prevailing opinion.16 17 35
We performed decision curve analyses to investigate what would be the optimal threshold of ACS predicted probability to initiate treatment, where ‘treatment’ in prehospital setting refers to urgent referral for hospital admission (U1, ambulance within 15 min). However, there is a principal difference between diagnostic probability and categories of urgencies. Risk prediction provides a continuous value of the probability of disease, while urgency level categorisation is based on the interpretation of how risk probability can be translated in urgency, and time within a patient should be seen and treatment delivered. Context is needed to determine optimal thresholds, which concerns what percentage of missing ACS is considered as acceptable by healthcare providers, patients and policymakers. This percentage is expected to be very low because a missed ACS can result in permanent cardiac impairment, heart failure, life-threatening arrhythmias in the early phase and death.36 Furthermore missing an ACS is the most common reason for malpractice claims worldwide.37 A survey performed among 1029 ED doctors in the USA, New Zealand and Australia showed that they considered an average missing rate between 0.1% and 1% (range 0%–10%) as acceptable.38 When we apply a maximum of 1% missing with our prediction rules, the threshold has to be set at a predicted risk of ACS of 0.05 (negative predictive value of 0.99, table 4), which means based on our data that the majority of patients needs urgent referral. This would result in over-crowded EMS and EDs, and with the available resources being limited, this may result in exceeding target triage times, which could compromise patient safety in another way.39 A possible alternative to consider may be applying different ‘treatments’ per thresholds, that is, dispatching an ambulance (U1) for the high predicted risk patients, and GP visit within 1 hour (U2) for the low predicted risk patients. During GP visit more clinical parameters (blood pressure, heart rate, overall clinical impression) can be gathered to improve ACS risk prediction, and in the future, there might be room for applying point-of-care high-sensitivity troponin testing, as these are nowadays only available in the ED setting.40 In order to determine the ideal threshold, external validation will be needed combined with clinical and management considerations. The development of this diagnostic model is the necessary first step towards an implementation study in which this model is adapted to urgency levels that can be applied by triage nurses during telephone triage at the OHS-PC. The diagnostic model needs to be ‘translated’ in simple yes/no questions that can be incorporated in the existing NTS and a personalised risk prediction for age and gender is generated. Some older questions will then be substituted. We are aiming to do so in an implementation study applying action research.
Implications for clinical practice and future research
This symptom-based prediction model for ACS has good discrimination and calibration and could readily be applied for telephone triage of patients with acute chest discomfort in primary care, notably the OHS-PC setting. The results of the decision curve analysis showed a large net benefit over a range of plausible risk threshold as compared with the currently used NTS model in the OHS-PC in the Netherlands. For future research, full external validation in other OHS-PC or EMS populations could further optimise and update the model. Furthermore, sex-specific prediction models could be developed for ACS, but given the overlap in symptoms between men and women, this would not result in major changes in predictors.
Conclusion
The final prediction model for ACS has good discrimination and calibration and shows promise for replacing the existing telephone triage rules for patients with acute chest discomfort in general practice and OHS-PC. However, future research with an external validation is needed to provide insights into how the prediction model can be applied in practice.
Supplementary Material
Acknowledgments
The authors would like to thank the OHS-PC foundation ‘Primair Huisartsenposten’ and all employees of the participating locations for or their cooperation in this study, notably for providing data and technical support.
Footnotes
Contributors: FHR and DLMZ are the lead investigators who conceived the research idea and methodology. Funding acquisition was done by FHR, DLMZ and RAMJD. LTCMW and DCAE conducted data acquisition. LTCMW performed the analyses and wrote the first draft of the manuscript. She was supervised by FHR and DLMZ, who critically revised the manuscript, and by MvS who was involved with the analyses. DCAE, EdG, EJMA, HMdR, AWH and RAMJD contributed to and approved of the final version of the manuscript. The guarantor (LTCMW) accepts full responsibility for the finished work and/or the conduct of the study, had access to the data, and controlled the decision to publish.
Funding: This study was funded by an unrestricted grant from (1) the Department of General Practice of the University Medical Centre Utrecht, (2) a personal promotion grant of D L Zwart, MD, PhD, (3) the foundation ’The Netherlands Triage Standard’ and (4) the foundation ‘Stoffels-Hornstra’. This study contributes to the 2020B004 IMPRESS study which is funded by the Dutch Heart Foundation/Dutch Cardiovascular Alliance (DHF/DCVA).
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Data availability statement
Data are available upon reasonable request. Data can be made available for researchers whose proposed use of the data has been approved at request of the corresponding author, with a signed data access agreement.
Ethics statements
Patient consent for publication
Not applicable.
Ethics approval
This study involves human participants and was approved by the Medical Ethics Committee Utrecht, the Netherlands (reference number WAG/mb/16/003208) and complied with the Declaration of Helsinki. A waiver of informed consent was given because our study had minimal risk to subjects and could otherwise not be carried out logistically. Personal and research data were handled and stored according to the European General Data Protection Regulation.
References
- 1.Rui P, Kang K. National Hospital ambulatory medical care survey: 2017 emergency department summary tables. Available: cdc.gov
- 2.Mol KA, Smoczynska A, Rahel BM, et al. Non-cardiac chest pain: prognosis and secondary healthcare utilisation. Open Heart 2018;5:e000859. 10.1136/openhrt-2018-000859 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Collet J-P, Thiele H, Barbato E, et al. 2020 ESC guidelines for the management of acute coronary syndromes in patients presenting without persistent ST-segment elevation. Rev Esp Cardiol 2021;74:544. 10.1016/j.rec.2021.05.002 [DOI] [PubMed] [Google Scholar]
- 4.Burman RA, Zakariassen E, Hunskaar S. Management of chest pain: a prospective study from Norwegian out-of-hours primary care. BMC Fam Pract 2014;15:51. 10.1186/1471-2296-15-51 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Deakin CD, Sherwood DM, Smith A, et al. Does telephone triage of emergency (999) calls using advanced medical priority dispatch (AMPDS) with department of health (DH) call prioritisation effectively identify patients with an acute coronary syndrome? an audit of 42,657 emergency calls to Hampshire ambulance service NHS trust. Emerg Med J 2006;23:232–5. 10.1136/emj.2004.022962 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Bruyninckx R, Aertgeerts B, Bruyninckx P, et al. Signs and symptoms in diagnosing acute myocardial infarction and acute coronary syndrome: a diagnostic meta-analysis. Br J Gen Pract 2008;58:e1–8. 10.3399/bjgp08X277014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bruins Slot MHE, Rutten FH, van der Heijden GJMG, et al. Diagnosing acute coronary syndrome in primary care: comparison of the physicians' risk estimation and a clinical decision rule. Fam Pract 2011;28:323–8. 10.1093/fampra/cmq116 [DOI] [PubMed] [Google Scholar]
- 8.Grijseels EW, Deckers JW, Hoes AW, et al. Implementation of a pre-hospital decision rule in general practice. triage of patients with suspected myocardial infarction. Eur Heart J 1996;17:89–95. 10.1093/oxfordjournals.eurheartj.a014697 [DOI] [PubMed] [Google Scholar]
- 9.Harskamp RE, Laeven SC, Himmelreich JC, et al. Chest pain in general practice: a systematic review of prediction rules. BMJ Open 2019;9:e027081. 10.1136/bmjopen-2018-027081 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Wouters LT, Rutten FH, Erkelens DC, et al. Accuracy of telephone triage in primary care patients with chest discomfort: a cross-sectional study. Open Heart 2020;7:e001376. 10.1136/openhrt-2020-001376 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Leite L, Baptista R, Leitão J, et al. Chest pain in the emergency department: risk stratification with Manchester triage system and HEART score. BMC Cardiovasc Disord 2015;15:48. 10.1186/s12872-015-0049-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Leeflang MMG, Rutjes AWS, Reitsma JB, et al. Variation of a test’s sensitivity and specificity with disease prevalence. CMAJ 2013;185:E537–44. 10.1503/cmaj.121286 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Buntinx F, Knockaert D, Bruyninckx R, et al. Chest pain in general practice or in the hospital emergency department: is it the same? Fam Pract 2001;18:586–9. 10.1093/fampra/18.6.586 [DOI] [PubMed] [Google Scholar]
- 14.Rawshani A, Larsson A, Gelang C, et al. Characteristics and outcome among patients who dial for the EMS due to chest pain. Int J Cardiol 2014;176:859–65. 10.1016/j.ijcard.2014.08.004 [DOI] [PubMed] [Google Scholar]
- 15.Kuriyama A, Urushidani S, Nakayama T. Five-level emergency triage systems: variation in assessment of validity. Emerg Med J 2017;34:703–10. 10.1136/emermed-2016-206295 [DOI] [PubMed] [Google Scholar]
- 16.van der Meer MG, Appelman Y, Rutten KHG, et al. Are there gender disparities in symptom presentation or triage of patients with chest discomfort at primary care out-of-hours services? An observational study. BMJ Open 2019;9:e031613. 10.1136/bmjopen-2019-031613 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Wouters LTCM, Zwart DLM, Erkelens DCA, et al. Gender-stratified analyses of symptoms associated with acute coronary syndrome in telephone triage: a cross-sectional study. BMJ Open 2021;11:e042406. 10.1136/bmjopen-2020-042406 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Erkelens DC, Wouters LT, Zwart DL, et al. Optimisation of telephone triage of callers with symptoms suggestive of acute cardiovascular disease in out-of-hours primary care: observational design of the safety first study. BMJ Open 2019;9:e027477. 10.1136/bmjopen-2018-027477 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Wouters LT, Zwart DL, Erkelens DC, et al. Chest discomfort at night and risk of acute coronary syndrome: cross-sectional study of telephone conversations. Fam Pract 2020;37:473–8. 10.1093/fampra/cmaa005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ 2020;368:m441. 10.1136/bmj.m441 [DOI] [PubMed] [Google Scholar]
- 21.van Smeden M, Moons KG, de Groot JA, et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Stat Methods Med Res 2019;28:2455–74. 10.1177/0962280218784726 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Steyerberg EW, Harrell FE. Prediction models need appropriate internal, internal-external, and external validation. J Clin Epidemiol 2016;69:245–7. 10.1016/j.jclinepi.2015.04.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Wynants L, van Smeden M, McLernon DJ, et al. Three myths about risk thresholds for prediction models. BMC Med 2019;17:192. 10.1186/s12916-019-1425-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Reitsma JB, Glas AS, Rutjes AWS, et al. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. J Clin Epidemiol 2005;58:982–90. 10.1016/j.jclinepi.2005.02.022 [DOI] [PubMed] [Google Scholar]
- 25.Jolani S, Debray TPA, Koffijberg H, et al. Imputation of systematically missing predictors in an individual participant data meta-analysis: a generalized approach using mice. Stat Med 2015;34:1841–63. 10.1002/sim.6451 [DOI] [PubMed] [Google Scholar]
- 26.Rubin DB. Multiple imputation for non response in surveys. John Wiley & Sons, Hoboken, NJ, 1987. [Google Scholar]
- 27.Harrell FE. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd edn. Cham, Switzerland: Springer International, 2015. [Google Scholar]
- 28.Moons KGM, Altman DG, Reitsma JB, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 2015;162:W1–73. 10.7326/M14-0698 [DOI] [PubMed] [Google Scholar]
- 29.Zwaan L, Monteiro S, Sherbino J, et al. Is bias in the eye of the beholder? A vignette study to assess recognition of cognitive biases in clinical case workups. BMJ Qual Saf 2017;26:104–10. 10.1136/bmjqs-2015-005014 [DOI] [PubMed] [Google Scholar]
- 30.Scott G, Clawson JJ, Gardett I, et al. 9-1-1 triage of non-traumatic chest pain: association with Hospital diagnosis. Prehosp Emerg Care 2017;21:525–34. 10.1080/10903127.2017.1302530 [DOI] [PubMed] [Google Scholar]
- 31.Canto JG, Rogers WJ, Goldberg RJ, et al. Association of age and sex with myocardial infarction symptom presentation and in-hospital mortality. JAMA 2012;307:813–22. 10.1001/jama.2012.199 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.International Working Group on Chest Pain in Primary Care (INTERCHEST), Aerts M, Minalu G, et al. Pooled individual patient data from five countries were used to derive a clinical prediction rule for coronary artery disease in primary care. J Clin Epidemiol 2017;81:120–8. 10.1016/j.jclinepi.2016.09.011 [DOI] [PubMed] [Google Scholar]
- 33.Haasenritter J, Donner-Banzhoff N, Bösner S. Chest pain for coronary heart disease in general practice: clinical judgement and a clinical decision rule. Br J Gen Pract 2015;65:e748–53. 10.3399/bjgp15X687385 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Ekelund U, Akbarzadeh M, Khoshnood A, et al. Likelihood of acute coronary syndrome in emergency department chest pain patients varies with time of presentation. BMC Res Notes 2012;5:420. 10.1186/1756-0500-5-420 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.van Oosterhout REM, de Boer AR, Maas AHEM, et al. Sex differences in symptom presentation in acute coronary syndromes: a systematic review and meta-analysis. J Am Heart Assoc 2020;9:e014733. 10.1161/JAHA.119.014733 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Bugiardini R, Ricci B, Cenko E, et al. Delayed care and mortality among women and men with myocardial infarction. J Am Heart Assoc 2017;6. 10.1161/JAHA.117.005968 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Ernesäter A, Winblad U, Engström M, et al. Malpractice claims regarding calls to Swedish telephone advice nursing: what went wrong and why? J Telemed Telecare 2012;18:379–83. 10.1258/jtt.2012.120416 [DOI] [PubMed] [Google Scholar]
- 38.Than M, Herbert M, Flaws D, et al. What is an acceptable risk of major adverse cardiac event in chest pain patients soon after discharge from the emergency department?: a clinical survey. Int J Cardiol 2013;166:752–4. 10.1016/j.ijcard.2012.09.171 [DOI] [PubMed] [Google Scholar]
- 39.van der Linden MC, Meester BEAM, van der Linden N. Emergency department crowding affects triage processes. Int Emerg Nurs 2016;29:27–31. 10.1016/j.ienj.2016.02.003 [DOI] [PubMed] [Google Scholar]
- 40.Pickering JW, Young JM, George PM, et al. Validity of a novel point-of-care troponin assay for Single-Test Rule-Out of acute myocardial infarction. JAMA Cardiol 2018;3:1108–12. 10.1001/jamacardio.2018.3368 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2022-064402supp001.pdf (152.2KB, pdf)
Data Availability Statement
Data are available upon reasonable request. Data can be made available for researchers whose proposed use of the data has been approved at request of the corresponding author, with a signed data access agreement.