The AFFORD Clinical Decision Aid To Identify Emergency Department Patients With Atrial Fibrillation At Low Risk For 30-Day Adverse Events

Abstract

There is wide variation in the management of emergency department (ED) patients with atrial fibrillation (AF). We aimed to derive and internally validate the first prospective, ED-based clinical decision aid to identify patients with AF at low risk for 30-day adverse events. We performed a prospective cohort study at a university-affiliated, tertiary-care, ED. Patients were enrolled from June 9, 2010 to February 28, 2013 and followed for 30 days. We enrolled a convenience sample of ED patients presenting with symptomatic AF. Candidate predictors were based on ED data available in the first two hours. The decision aid was derived using model approximation (preconditioning) followed by strong bootstrap internal validation. We utilized an ordinal outcome hierarchy defined as the incidence of the most severe adverse event within 30 days of the ED evaluation. Of 497 patients enrolled, stroke and AF-related death occurred in 13 (3%) and 4 (<1%) patients, respectively. The decision aid included the following: age, triage vitals (systolic blood pressure, temperature, respiratory rate, oxygen saturation, supplemental oxygen requirement); medical history (heart failure, home sotalol use, prior percutaneous coronary intervention, electrical cardioversion, cardiac ablation, frequency of AF symptoms); ED data (2 hour heart rate, chest radiograph results, hemoglobin, creatinine, and brain natriuretic peptide). The decision aid’s c-statistic in predicting any 30-day adverse event was 0.7 (95% CI, 0.65, 0.76). In conclusion, among ED patients with AF, AFFORD provides the first evidence based decision aid for identifying patients who are at low risk for 30-day adverse events and candidates for safe discharge.

Atrial fibrillation (AF) affects an estimated 33.5 million individuals worldwide and is associated with a 2-fold increased mortality.¹ The management of AF accounts for an estimated $26 billion in annual healthcare costs in the United States (US).² Hospitalizations constitute the majority of the AF healthcare expenses.^2,3 Compared to matched controls, AF patients reported more than twofold emergency department (ED) evaluations and hospitalizations.⁴ The ED is the gatekeeper for acute AF management as over 70% of patients hospitalized for AF are initially managed there.^5–7 Between 25–35% of these visits are for new AF diagnoses.^6,8 Hospitalizations following an ED evaluation for AF vary considerably with admission frequencies of 81%, 62% and 24% in the US, Australia and Canada, respectively.⁹ The lack of accurate risk stratification may contribute to substantial variation in ED dispositions.^5,6,9 Although AF increases an individual’s lifetime risk of death and stroke,¹⁰ the 30-day risk of stroke and death following an ED visit for primary AF is relatively low with a combined incidence of 1–3%.^8,11 The combination of a high hospitalization rate and low incidence of 30-day serious adverse events provides an excellent opportunity to deliver more standardized and resource efficient treatment. Decision aids are an important component of ED practice resulting in improved patient care of common presenting complaints.^12,13 We previously reported the first decision aid for estimating 30-day adverse event risk in a retrospective cohort of ED patients with AF.⁸ Our objective was to derive the first prospective AF decision aid that incorporated data available within the initial 2 hours of ED evaluation and accurately predicted risk for 30-day adverse events.

Methods

We conducted a prospective observational cohort study at Vanderbilt University Medical Center’s ED from June 9, 2010 to February 28, 2013. The adult ED is a university-affiliated, urban, tertiary care, referral center with an annual patient census of 70,000. The details of the Atrial Fibrillation and Flutter Outcome Risk Determination (AFFORD) study design have been previously reported.¹⁴ Briefly, the study team consisting of the principal physician investigator and trained clinical trial associates screened ED patients for potential eligibility (Figure 1). We enrolled a convenience sample of patients presenting with signs (e.g., tachycardia, dyspnea) or symptoms (e.g., palpitations, chest pain, shortness of breath, weakness, pre-syncope, or syncope) consistent with symptomatic AF that prompted the treating emergency physician to obtain an electrocardiogram. The diagnosis of AF or atrial flutter was based on the emergency physician’s interpretation of the electrocardiogram. All electrocardiogram interpretations were confirmed by an independent cardiologist review of the electrocardiogram within 24 hours of enrollment. Patients whose electrocardiograms were determined to be a rhythm other than AF, based on the cardiologist’s interpretation, were withdrawn from the study. Patients with acute life threatening conditions (i.e., stroke, myocardial infarction, sepsis) were not enrolled. Treatment and disposition decisions for patients enrolled in the study were determined by the treating physicians and were not influenced by this investigation. Our medical center’s institutional review board reviewed and approved this study. Patients provided written informed consent.

Figure 1.

AFFORD Study Flow Diagram

We included both patients whose ED evaluations were primarily for AF and patients with AF presenting with an alternative primary disease. To determine whether the evaluation was primarily for AF or whether AF was complicating an alternative acute process, a board-certified emergency medicine physician investigator reviewed the ED records of all enrolled patients. The principal investigator used a standardized protocol. A board-certified electrophysiologist investigator provided a second review of a randomly selected 31% subset of the enrollments. We measured the absolute agreement and Cohen’s kappa statistic with 95% confidence intervals (CI) between the 2 reviewers’ designation for the primary cause of ED visits.

Clinical trial associates collected data on consented subjects by direct questioning and a review of the electronic health record. Study data were collected and managed using Research Electronic Data Capture (REDCap).¹⁵ The principal investigator reviewed and confirmed the accuracy of the data recorded by the clinical trial associates. To minimize missing data, we performed laboratory testing using frozen, stored blood specimens when such tests were not done as standard ED treatment. We defined AF type in accordance with international guidelines.^16,17

The primary outcome was a 10-level ordinal outcome representing the most severe adverse event experienced within 30 days of the index ED evaluation. (Figure 2) The events of the outcome were ordered from least severe (no event) to most severe (AF-related). The hierarchy was determined by consensus among the investigators including 3 emergency physicians and 3 electrophysiologists. We reviewed the patient’s records and the Social Security Death Index for outcomes. Admitted patients were followed until hospital discharge. All patients received structured follow-up telephone communication at 5 and 30-days from the ED visit. Investigators used a standard telephone communication data collection form for all interviews. The investigators assessing the presence of each outcome event were masked to the predictor variables and vice versa. The accurate determination of whether the adverse events were related to AF was of utmost importance for this study. The principal investigator and an electrophysiologist co-investigator used a standardized protocol to determine whether an adverse event was AF-related.

Figure 2. Hierarchy Of Adverse Events For AFFORD Decision Aid.

The Figure presents the 10-level ordinal outcome representing the most severe adverse event experienced within 30 days of the index ED evaluation. The events of the outcome were ordered from least severe (no event) to most severe (AF-related).

A large pool of candidate predictor variables were considered based on established risk factors for AF, focusing on those likely to be recorded in the ED,^8,18 and chosen in accordance with established principles.^13,19 Development of the AFFORD decision aid followed established strategies that are detailed in Appendix A.^13,19 Briefly, model selection from the pool of candidate predictors was done using a model approximation method called preconditioning.²⁰ Preconditioning is suited for high dimensional data problems in which the number of predictors is large relative to the number of events. Proportional odds logistic regression was then used to fit the ordinal outcome against the selected predictors. An ordinal outcome was used rather than a binary outcome because the proportional odds model allows for parsimonious modeling of an ordinal outcome with increased power and precision as compared with a binary logistic model. The AFFORD decision aids was then obtained by assigning points for selected predictors based on their corresponding regression coefficients.

We quantified AFFORD’s predictive accuracy in predicting any 30-day adverse event using a concordance index (c-statistic).¹³ The calibration was assessed using a smooth nonparametric calibration curve comparing predicted and observed probabilities of any 30-day adverse event for the original model (Apparent) and the bootstrapped bias-corrected model (Bias-corrected). In general, the calibration curve illustrates bias in the predicted values obtained from the prediction model. We performed a strong internal validation using bootstrap resampling in order to estimate the likely performance of the decision aid on a new sample of patients from the same patient stream.¹⁹ All analyses were done using R programming language (Vienna, Austria: R Foundation for Statistical Computing). Standard sample size requirements, supported by simulation studies and expert opinion, include 15 subjects or events per degree of freedom (i.e. per regression coefficient examined or estimated). Our initial planned sample size of 430 patients accounted for the a priori determined predictors and an anticipated 5% loss to follow-up.¹⁹ We exceeded the planned sample size due to higher than expected ED volumes of eligible AF patients. The entire sample was used in model development.^13,19

Results

This study enrolled 519 patients from June 9, 2010 to February 28, 2013 with 15 withdrawn and seven lost to follow-up (Figure 1). The 15 withdrawals were primarily for failure to meet inclusion criteria when the ED ECG was subsequently interpreted as a rhythm other than AF or flutter by the independent cardiologist. A search of the Social Security death index did not find any reported deaths among these withdrawals within 30 days of their ED visit. Of the 4151 patients screened, only 39 (<1%) subjects were classified “missed potentially eligible”.

Table 1 reports the baseline characteristics. The characteristics of the 501 potentially eligible patients who refused to participate (n=399) or were missed were similar to enrolled patients. Of the 497 patients in the cohort, the principal investigator determined that 326 (65.6%) were for primary AF. An electrophysiologist co-investigator independently reviewed 163 (31.4%) records to determine whether AF was the primary reason for the ED visit. Of those 163 records, concordance was found in 135 (83%), corresponding to a kappa of 0.8 (95% CI, 0.7, 0.9).

Table 1.

Baseline Patient Characteristics (n=497)

Characteristic	Study Population (n=497)
Age (years)	68 (58, 78)
Female	177 (36%)
White	447 (90%)
Black	50 (10%)
Insurance status: Medicare	220 (44%)
Private/Group	244 (49%)
Federal (Medicaid/Veterans Administration)	12 (2%)
Self-pay	21 (4%)
ED visit For Primary Atrial Fibrillation	326 (66%)
Type of Atrial Fibrillation
New diagnosis	131 (26%)
Paroxysmal	183 (37%)
Persistent	61 (12%)
Permanent	122 (25%)
CHA2DS2-VASc	3 (2, 5)
Complaint in the Emergency Department
Chest pain	90 (18%)
Palpitations	44 (9%)
Shortness of breath	78 (16%)
Fatigue	332 (67%)
Triage blood pressure, systolic (mm Hg)	134 (117, 152)
Triage blood pressure, diastolic (mm Hg)	80 (69, 92)
Triage heart rate (beats per minute)	105 (82, 130)
Triage respiratory rate	18 (16, 20)
Triage oxygen saturation (%)	97 (95, 98)
ED supplemental oxygen requirement	58 (12%)
Duration of symptoms prior to ED presentation (hours)
< 12	213 (43%)
12–24	62 (12%)
24–36	24 (5%)
36–48	18 (4%)
>48	123 (25%)
Unknown	56 (11%)
Maximum pulse rate during initial 2 hours of ED management (beats per minute)	123 (91, 144)
Heart rate at 2 hours following placement in ED room 2-hour rhythm strip interpretation	93 (77, 116)
Atrial fibrillation	309 (62%)
Atrial flutter	81 (16%)
Sinus rhythm	58 (12%)
Other	12 (2%)
Home aspirin use	226 (45%)
Home beta-blocker use	259 (52%)
Home diltiazem/verapamil use	86 (17%)
Home sotalol use	36 (7%)
Home warfarin use	166 (33%)
Therapeutic INR (2–3.4) +	108 (65%)
Home statin use	223 (45%)
Home ACEI/ARB use	227 (46%)
Home clopidogrel use	52 (10%)
Home novel anticoagulant use	16 (3%)
Home thyroid replacement	79 (16%)
Current smoker	60 (12%)
Current alcohol drinker	132 (28%)
Prior cocaine use	38 (7%)
Prior coronary artery disease	169 (34%)
Prior COPD	78 (16%)
Prior hypertension	356 (72%)
Prior valvular heart disease	132 (27%)
Aortic Regurgitation	29 (6%)
Aortic Stenosis	24 (5%)
Mitral Regurgitation	91 (18%)
Mitral Stenosis	12 (2%)
Pulmonic Regurgitation	10 (2%)
Pulmonic Stenosis	1 (0.2%)
Tricuspid Regurgitation	49 (10%)
Tricuspid Stenosis	1 (0.2%)
Prior cerebrovascular accident	67 (13%)
Prior transient ischemic attack	62 (12%)
Prior heart failure	167 (34%)
Prior renal insufficiency	88 (18%)
Prior non-insulin dependent diabetes	66 (13%)
Prior insulin-dependent diabetes	62 (12%)
Permanent pacemaker	80 (16%)
Family history of atrial fibrillation	95 (19%)
Prior percutaneous coronary intervention	82 (16%)
Prior electrical cardioversion	19 (4%)
Prior cardiac ablation	44 (9%)
Frequency of irregular heartbeat:
Never	188 (38%)
Sometimes	92 (19%)
Usually	57 (11%)
Always	131 (26%)
Hemoglobin (g/dL)	14 (12, 15)
Blood urea nitrogen (mg/dL)	17 (13, 24)
Creatinine (mg/dL)	1.07 (0.86, 1.39)
Troponin I (ng/mL)	0.02 (0.01, 0.03)
Brain natriuretic peptide (pg/mL)	261 (114, 538)
ED disposition, admit	415 (83.5%)
Hospital length of stay (days)	2.4 (1.0, 4.4)

^*N equal total number of non-missing responses for each variable. Categorical variables presented as number followed by percentage in parentheses. Continuous variables are represented as the median with interquartile range in parentheses.

⁺The reported percentage is of the 166 patients who reported that they were taking warfarin at the time of ED evaluation. Abbreviations in table: ACEI - angiotensin converting enzyme inhibitor; ARB - angiotensin receptor blocker; COPD - chronic obstructive pulmonary disease

Of the 82 patients who were discharged home, 73 (89%) had no 30-day adverse events, six returned to the ED for an AF-reason but were not hospitalized and three required an unscheduled AF-related hospitalization. Among the 415 patients who were hospitalized following their index ED evaluation, 291 (70%) experienced no 30-day adverse events and 22 had an AF-related return ED evaluation but were not hospitalized. Table 2 reports the incidence of 30-day adverse events.

Table 2.

Frequency of 30-Day Adverse Events

30-Day Adverse Events	No. (%)(n = 497)	AFFORD Hierarchy Level
Death, AF-related	4 (<1%)	10
Death, all cause	30 (6%)	9–10
Cardiac arrest, resuscitated	13 (3%)	9
Stroke (thromboembolic or hemorrhagic)	13 (3%)	8
Cardiogenic shock	4 (<1%)	8
Acute coronary syndrome requiring emergent PCI	0	7
Intensive care unit admission, AF-related	12 (2%)	7
Vasopressor administration required	11 (2%)	7
Emergent pacemaker placement	1 (<1%)	7
Ventricular arrhythmia requiring emergent treatment	7 (1%)	6
Unscheduled intubation/ mechanical ventilation	12 (2%)	6
Acute decompensated heart failure	13 (3%)	5
Acute coronary syndrome not requiring emergent PCI	7 (1%)	5
Emergent blood product transfusion required	8 (2%)	5
Unscheduled hospitalization, AF-related	32 (6%)	4
Unscheduled inpatient pacemaker placement	6 (1%)	4
Unscheduled inpatient blood product transfusion required	13 (3%)	3
Continuous AV nodal blocker infusion required	17 (3%)	3
ED visit, AF-related	39 (8%)	2
Atrial arrhythmia requiring emergent treatment*	73 (15%)	2
Experienced ≥1 adverse event	133 (27%)	2–10

^*Defined as recurrent symptomatic atrial fibrillation or atrial flutter including atrial fibrillation with rapid ventricular rate requiring acute rate control therapy.

AF: Atrial Fibrillation; AV: Atrioventricular; ED: Emergency Department; PCI: Percutaneous Coronary Intervention

The AFFORD decision aids contains 17 variables including 12 components that are often already collected and easily obtained in the ED. Triage oxygen saturation (SaO2), systolic blood pressure, and hemoglobin were fit with restricted cubic splines. The odds ratios (95% CI) for AFFORD are presented in the Appendix Table. The AFFORD nomogram is presented in Figure 3. In predicting any 30-day adverse event vs. no event, the c-statistic was 0.7 (95% CI, 0.65, 0.76). Figure 4 presents the calibration curve showing that AFFORD performs very well at identifying the low risk patients whose risk of any 30-day adverse event is 10% or less. The discrimination and calibration in predicting other levels of adverse event are also assessed and are presented in Appendix. Table 3 provides ranges of AFFORD total point corresponding to different predicted risk thresholds for any 30-day adverse event. The predicted risk for any 30-day adverse event is less than 20% for the majority of the cohort.

Figure 3. Thirty-Day Adverse Event AFFORD Nomogram.

Points are assigned for each of the 17 predictors. The total points correspond to an absolute predicted risk for 30-day adverse events.

Figure 4. Calibration Curve for AFFORD Decision Aid.

This plot illustrates the calibration accuracy of the original model (“Apparent”) and the bootstrap model (“Bias-corrected”) for 30-day adverse events with locally weighted scatterplot smoothing used to model the relationship between actual and predicted probabilities. As can be seen, the model’s calibration function estimate is slightly nonlinear, with the corrected calibration showing good agreement with the apparent calibration.

Table 3.

AFFORD Decision Aid Predicted Risks and Corresponding Point Totals for 30-Day Adverse Events

Predicted Risk Categories Derived From the Original AFFORD model	Corresponding AFFORD Point Totals For Any 30-Day Adverse Event*	Corresponding AFFORD Point Totals For 30-Day Unscheduled Hospitalization or More Severe Event+	Corresponding AFFORD Point Totals For 30-Day Stroke Or DeathŦ
0–2.49%	0–130	0–145	0–176
2.50–4.99%	131–151	146–166	177–197
5.0–7.49%	152–163	167–178	198–210
7.50–9.99%	164–173	179–188	211–219
10.0–12.49%	174–180	189–195	220–227
12.5–14.99%	181–186	196–201	228–233
15.0–17.49%	187–192	202–207	234–238
17.5–19.99%	193–197	208–212	239–243
≥20.0%	≥198	≥213	≥244

^*In the entire AFFORD cohort, 133 people experienced a 30-day adverse event.

⁺This outcome category corresponds to an individual experiencing ≥1 of the outcomes listed in Levels 4–10 in the hierarchy presented in Figure 2. The corresponding c-statistic using the 4th level of the outcome as the cut-off (94 events) was 0.74 (95% CI, 0.69, 0.8).

^ŦThese outcome category corresponds to an individual experiencing ≥1 of the outcomes listed in Levels 8–10 in the hierarchy presented in Figure 2. The corresponding c-statistic using the 8th level of the outcome as the cut-off (41 events) was 0.81 (95% CI, 0.74, 0.88).

We examined the decision aid’s performance in the 326 patients whose ED visit was primarily for AF. For the prediction of any 30-day adverse event, the c-statistic was 0.72 (95% CI, 0.66, 0.78). The calibration curve is reported in Appendix and is similar to the curve for the entire cohort.

Discussion

AFFORD is the first clinical decision aid to predict 30-day adverse events in a prospective ED patient cohort with acute symptomatic AF. AFFORD consists of easily obtained variables that accurately identify low risk patients who might be safely discharged from the ED. The majority of these variables can be easily collected in triage and should be available within the first two hours of ED management. AFFORD may be incorporated into the disposition decisions for hemodynamically stable ED patients whose AF reverts to sinus rhythm, either spontaneously or following pharmacological or electrical cardioversion, and those who are adequately rate controlled and candidates for outpatient management.

We derived the AFFORD decision aid in accordance with the recommended biostatistical methodologies and performed a strong internal validation.^13,19 The AFFORD decision aid is well calibrated and accurately identifies patients with a predicted 30-day adverse event risk less than 10% who may be candidates for safe ED discharge. AFFORD’s c-statistic was similar to the c-statistics reported for the CHADS₂ (0.56–0.80) and CHA₂DS₂-VASc scores (0.59–0.79) which are the current standards for AF thromboembolic risk assessment.^16,17,21,22

More than 70% of ED patients with AF hospitalized in the UK and US.^5,9 This rate eclipses the frequencies in Canada, Spain and The Netherlands who hospitalize only 17–36%.^9,23,24 There continues to be considerable regional differences in hospitalizations across the globe and within the US.^5,9 Nearly half of these AF hospitalizations could be avoided,²⁵ thus potentially reducing hospital-associated adverse events and healthcare expenditures.²⁶ Decision aids that promote the delivery of more standardized disposition practices may provide a platform for improved treatment and healthcare resource utilization. AFFORD is clinically sensible and includes objective data that is routinely collected during ED evaluations.⁵ Many of these variables have been associated previously with elevated risks for thromboembolic events and death among AF patients.^8,17,25 Incorporating AFFORD into ED physician’s disposition decision would have helped identify admitted patients as very low risk and potentially resulted in reduced rate of hospitalizations. In the absence of an alternative serious precipitating illness, such low-risk patients who are adequately rate controlled should be excellent candidates for outpatient cardiology management.

The incidence of adverse events within 30-days following an ED evaluation for AF is not trivial. However, patients who experienced adverse events most often had significant associated comorbidities. Hospitalization and inpatient treatment is often necessary for these patients given their hemodynamic fragility. AFFORD is not intended to recommend discharge for such high risk patients as the ED physician’s clinical gestalt alone should direct these patients to inpatient management.

AFFORD might also be incorporated into a shared decision model facilitating ED disposition decisions. Shared decision making empowers patients to take an active role in their disease management and encourages a collaborative decision process between patients, caregivers and treating clinicians.^27,28 Such strategies have proven successful in the ED management of other common conditions.²⁸ Given its clear definition and well-defined treatment goals (i.e., rate control, rhythm management, and anticoagulation risk assessment),^16,17 AF is an ideal disease for a shared decision-making model regarding patient disposition.²⁹ We are actively investigating the impact of incorporating AFFORD and a shared-decision making model on reducing AF hospital admissions.

We limited our candidate predictors to data available to ED physicians within the first two hours of evaluation. We did so to optimize the clinical utility of the decision aid as many ED physicians strive to have a disposition decision within the first two hours. We included patients whose ED evaluations were for primary AF and individuals in whom AF was complicating an alternative primary complaint such as acute infection. We included both groups to maximize the generalizability and utility of AFFORD. The decision aid performed similar in both the primary AF subgroup and the entire cohort as evidenced by the comparable c-statistics and calibration curves. ED patients frequently have more than one complaint and asymptomatic new AF is often diagnosed during ED evaluations for alternative complaints.^5,8 Patients whose AF was incidental or a secondary complaint may represent a different phenotype and therefore alter the ED clinician’s disposition decision.³⁰

This study was conducted at a university-affiliated referral center with resident physicians, fellows and faculty physicians staffing the ED. As a referral center, there is the potential that our ED population of AF patients has more complicated medical comorbidities at higher risk for adverse events, thus requiring more extensive evaluation and frequent hospitalizations. We chose to include a comprehensive and conservative hierarchy of possible adverse events. Some might not agree a return ED visit within 30 days for recurrent AF with rapid ventricular rate that does not require hospitalization is an adverse event, but rather the natural course of the disease. We chose these conservative outcomes to identify the lowest risk patients who might be safely discharged. AFFORD does include laboratory and chest radiograph results that may not always be obtained. We intend the AFFORD aid to augment, not replace, clinical decision making. Patient disposition might have impacted our primary outcome. There is the potential that an inpatient intervention or management decision might impact the incidence of a 30-day adverse event. We originally intended to derive a decision aid to predict 5-day adverse events as we detailed in our clinical trials registration and methodology publication.¹⁷ However, the low incidence of 5-day adverse events precluded us from deriving a separate stable decision aid. Acknowledging these limitations, we believe that our results provide evidence that an opportunity exists to improve resource utilization by reducing hospital admissions.

Appendix A. Additional Detailed Methodology

Development of the AFFORD decision aid followed established strategies.^13,19 First, we evaluated descriptive statistics of all candidate predictors for degree of missingness and level of information provided. Predictors with high missingness (>90%) or little variation (>90% prevalence of one level for categorical variable) were removed from consideration. Descriptive statistics on potential predictors available within the initial two hours of the index ED visit were calculated using median (Interquartile Range [IQR]) or percentage (N), as appropriate. Missing data on remaining candidate predictors were imputed using single imputation allowing for nonlinear transformations on the data to make optimal use of partial information recorded for each subject.^13,19 The challenge of the development of AFFORD decision aid lies in the presence of a large pool of 122 candidate predictor variables which produces “noise” during traditional step-wise model selection. To mitigate the effects of ‘noisy’ feature in such high-dimensional setting, a modeling approximation method called preconditioning was used for model selection.²⁰ Preconditioning was developed to handle high dimensional data problems in which the number of predictors is large relative to the number of events. The method has 2 steps: 1) a continuous ‘pre-conditioned’ outcome characterizing the underlying distribution of the 10-level ordinal outcome was derived as linear combination of all 122 candidate predictors by fitting the proportional odds model on the ordinal outcome; 2) The best set of predictors was selected from the pool based on maximizing the Akaike Information Criterion (AIC) with backward model selection using the ‘pre-conditioned’ outcome.¹⁹ The AFFORD decision aid was derived by fitting a proportional odds model to the 10-level ordinal outcome using 17 variables selected in the second step of preconditioning method. Regression splines were used in the risk prediction model. Proportional odds assumptions of the final model were verified using plots of the mean of each candidate predictor across the levels of the ordinal outcome.

We performed a strong internal validation using bootstrap resampling in order to estimate the likely performance of the decision aid on a new sample of patients from the same patient stream.¹⁹ Critics might argue that our decision aid needs to be externally validated prior to use in managing AF patients. Even in the small subset of studies comprising truly external validations, it is a common misconception that the validation statistics are precise. Many if not most external validations are unreliable due to instability in the estimate of predictive accuracy. This instability comes from two sources: the size of the validation sample, and the constitution of the validation sample. The former is easy to envision, while the latter is more subtle. In one example, Harrell and colleagues analyzed 17,000 intensive care unit patients with 1/3 of patients dying, splitting the dataset into two halves - a training sample and a validation sample. The validation c-index changed substantially when the 17,000 patients were re-allocated at random into a new training and test sample and the entire process repeated. Thus it can take quite a large external sample to yield reliable estimates and to “beat” strong internal validation using resampling. Thus we feel there is great utility in using strong internal validation.

Appendix Table.

The AFFORD Decision Aid

Predictor	Predictor Level	Odds Ratio	95% CI	p-value
Age (years)*		1.43	(1.00, 2.06)	0.05
Triage systolic blood pressure (mm Hg)	90	0.57	(0.26, 1.25)	0.02
	130 (ref)	1.00
	150	0.53	(0.36, 0.80)
	200	0.84	(0.35, 2.00)
Triage respiratory rate*		1.13	(0.89, 1.44)	0.30
Triage oxygen saturation (%)	90	1.66	(1.06, 2.61)	0.07
	93	1.22	(1.02, 1.44)
	95 (ref)	1.00
	97	0.89	(0.77, 1.03)
Triage supplemental oxygen required		1.52	(0.98, 2.34)	0.06
Triage temperature (Fahrenheit)*		0.90	(0.70, 1.17)	0.44
Home sotalol use		0.43	(0.16, 1.16)	0.10
Prior heart failure		1.03	(0.62, 1.70)	0.92
Prior percutaneous coronary intervention		1.11	(0.63, 1.99)	0.71
Prior electrical cardioversion		0.45	(0.05, 3.80)	0.46
Prior cardiac ablation		0.78	(0.34, 1.80)	0.56
Atrial fibrillation frequency	Never	1.00		0.65
	Sometimes	1.15	(0.66, 2.01)
	Usually	1.59	(0.79, 3.21)
	Always	1.06	(0.62, 1.85)
2 hour heart rate*		1.29	(0.93, 1.79)	0.12
Chest radiograph	Normal	1.00		0.44
	Chronic Changes	1.07	(0.59, 1.92)
	Acute Changes	1.40	(0.78, 2.51)
Hemoglobin (g/dL)	7	4.07	(1.56, 10.61)	0.01
	9	2.46	(1.33, 4.53)
	11	1.49	(1.14, 1.94)
	13 (ref)	1.00
Creatinine (mg/dL)*		1.12	(1.00, 1.26)	0.05
Brain natriuretic peptide (pg/mL)*		1.21	(1.05, 1.40)	0.01

^*The Odds Ratios for continuous variables without splines represents an increase from the 25th to the 75th percentile

Appendix Figure A.1. Calibration Curve For AFFORD Decision Aid Using The 4th Level Of The Outcome Hierarchy As The Cut-Off (n= 94 Events).

This plot illustrates the calibration accuracy of the original model (“Apparent”) and the bootstrap model (“Bias-corrected”) for the subgroup of 30-day adverse events (i.e. atrial fibrillation-related unscheduled hospitalization or more severe event) with locally weighted scatterplot smoothing used to model the relationship between actual and predicted probabilities. As can be seen, the model’s calibration function estimate is slightly nonlinear, with the corrected calibration showing good agreement with the apparent calibration. The corresponding c-statistic (95% CI) using the 4th level of the outcome as the cut-off was 0.74 (0.69, 0.8).

Appendix Figure A.2. Calibration Curve For AFFORD Decision Aid Using The 8th Level Of The Outcome Hierarchy As The Cut-Off (n= 41 Events).

This plot illustrates the calibration accuracy of the original model (“Apparent”) and the bootstrap model (“Bias-corrected”) for the subgroup of 30-day adverse events (i.e. stroke or death) with locally weighted scatterplot smoothing used to model the relationship between actual and predicted probabilities. As can be seen, the model’s calibration function estimate is slightly nonlinear, with the corrected calibration showing good agreement with the apparent calibration. The corresponding c-statistic (95% CI) using the 8th level of the outcome as the cut-off was 0.81 (95% CI, 0.74, 0.88).

Appendix Figure A.3. Calibration Curve For AFFORD Decision Aid In Patients (n =326) Whose ED Visit Was For Primary Atrial Fibrillation.

This plot illustrates the calibration accuracy of the original model (“Apparent”) and the bootstrap model (“Bias-corrected”) for 30-day adverse events with locally weighted scatterplot smoothing used to model the relationship between actual and predicted probabilities. As can be seen, the model’s calibration function estimate is slightly nonlinear, with the corrected calibration showing good agreement with the apparent calibration. The c-statistic (95% CI) for the prediction of any 30-day adverse event was 0.72 (95% CI, 0.66, 0.78).