Published models that predict hospital readmission: a critical appraisal

Abstract

Introduction

The number of readmission risk prediction models available has increased rapidly, and these models are used extensively for health decision-making. Unfortunately, readmission models can be subject to flaws in their development and validation, as well as limitations in their clinical usefulness.

Objective

To critically appraise readmission models in the published literature using Delphi-based recommendations for their development and validation.

Methods

We used the modified Delphi process to create Critical Appraisal of Models that Predict Readmission (CAMPR), which lists expert recommendations focused on development and validation of readmission models. Guided by CAMPR, two researchers independently appraised published readmission models in two recent systematic reviews and concurrently extracted data to generate reference lists of eligibility criteria and risk factors.

Results

We found that published models (n=81) followed 6.8 recommendations (45%) on average. Many models had weaknesses in their development, including failure to internally validate (12%), failure to account for readmission at other institutions (93%), failure to account for missing data (68%), failure to discuss data preprocessing (67%) and failure to state the model’s eligibility criteria (33%).

Conclusions

The high prevalence of weaknesses in model development identified in the published literature is concerning, as these weaknesses are known to compromise predictive validity. CAMPR may support researchers, clinicians and administrators to identify and prevent future weaknesses in model development.

Strengths and limitations of this study.

• This study appraises readmission models, of which dozens are published and which have limited regulatory oversight, to ensure high quality and usefulness for health decision-making.

• Recommendations are specific to readmission models and were developed using a modified Delphi process. Critical appraisal was undertaken independently by two researchers.

• Recommendations are limited to model development and validation, and future revision will be necessary as the field advances and developers adopt more modern techniques.

Introduction

In 2013, the Centers for Medicare and Medicaid Services (CMS) Hospital Readmission Reduction Program (HRRP) began to financially penalise US hospitals with excessive 30-day readmission rates, with the goal of improving patient care. Subsequently, research on readmission risk prediction models increased exponentially,^{1 2} with two distinct goals: (1) to identify high-risk patients for targeted interventions, and (2) to standardise institutions’ readmission rates for use as a performance indicator. Preventable hospital readmissions cost CMS $17 billion each year,³ and CMS penalties for subpar readmission rates totalled $566 million in 2018.⁴ Readmissions have received considerable attention due to this financial burden, their impact on patient care and their use as a performance indicator.^{5 6} Consequently, efforts to research^{7 8} and market^{9 10} readmission models have increased rapidly, and these models are used extensively for health decision-making.

However, uncritically accepting the results of any single model has risks. These models can be subject to flaws in their development and validation, as well as limitations in their clinical usefulness. Given the hundreds of readmission models now available, distinguishing the highest quality, most clinically useful models can be challenging for clinicians, researchers and healthcare administrators. In this study, we address this gap for readmission models in the published literature by critically appraising them. To conduct the critical appraisal, we developed Critical Appraisal of Models that Predict Readmission (CAMPR), which lists 15 Delphi-based expert recommendations for high-quality, clinically useful readmission models. CAMPR focuses on the unique considerations of readmission modelling, such as purpose, competing risks, outcome timing, risk factor definitions, data sources and thresholding. This manuscript discusses the expert recommendations and subsequent critical appraisal in detail, and provides reference lists of eligibility criteria and risk factors to consider when assessing readmission models.

Methods

The Delphi method is a well-established structured communication technique for systematically seeking expert opinion on a specific topic.^11–13 Traditionally, the first round uses open-ended questions to generate ideas from participants, whereas subsequent rounds rely on more structured communication to achieve consensus. In this study, we conducted two rounds of online expert surveys, the first open-ended and the second semistructured, and a third round consisting of expert application to the published literature.¹¹

Round 1: development of CAMPR (open-ended survey)

Survey content

A Delphi expert used an iterative process to develop the initial survey in collaboration with four physician experts in readmission models. The survey collected personal information on the respondents’ institution(s) and relevant expertise, as well as information on models at the respondents’ institution(s). Then, the survey assessed perceived barriers to model development and implementation, as well as strategies to overcome barriers and recommendations to improve models. The complete survey is available as online supplemental appendix A.

Data collection

To ensure rapidity and anonymity, provide convenience and recruit individuals from diverse backgrounds and geographical locations, we invited experts to participate electronically using Qualtrics Survey Software. Expert panels for Delphi studies traditionally have 12–20 members.¹⁴ Electronic participation enabled us to include more than 20 members, as desired due to the complex nature of the problem and the probable diversity of opinions. We distributed our survey via personalised, individual emails to all corresponding authors of readmission prediction studies from two recent systematic reviews (2011, 2016).^{1 2} Additionally, we publicly distributed it to members of the American Medical Informatics Association.

Eligibility criteria

We included both model developers and implementers in our expert panel to capture a broad range of perspectives on the readmission prediction literature. We required that participants speak English and self-report involvement in (1) the development of one or more readmission models, or (2) the implementation of one or more readmission models at one or more institutions.

Data analysis

Two researchers conducted thematic analysis in NVivo V.11 (QSR International). First, the researchers independently read each response and defined codes in a dictionary for the remaining analysis. Then, the researchers independently coded all responses using codes corresponding to the dictionary and summarised themes that emerged. Together, the researchers reviewed and named common themes that emerged, and resolved conflicts by discussion. To enhance confirmability, we shared summaries of coded data with three participants and asked for their confirmation or revisions to interpretation.

Round 2: further development of CAMPR (semistructured survey)

Preliminary version

Based on the thematic analysis in round 1, the study team identified 48 preliminary recommendations to operationalise two quality dimensions of readmission models: (1) development and (2) implementation. Each preliminary recommendation addressed one of four key thematic domains for development and five key thematic domains for implementation, identified via expert consensus (table 1).

Table 1.

Quality dimensions of readmission risk prediction models

Dimension	Key domain	Description of barrier
Development	Validation	Inadequate validation causing poor model discrimination when generalised
	Features	Over-reliance on biomedical features from administrative and claims data
	Timeframe	Possibility that the 30-day timeframe is not optimal for accurate prediction
	Data access	Dependence on inaccessible, low quality, outdated or manually entered data
Implementation	Resources	Insufficient personnel, statistical expertise or financial resources
	Vision	Lack of leadership, purpose or policy priorities related to readmission prediction
	Clinical relevance	Unclear clinical utility, actionability or relevance
	Workflow integration	Poor integration into the clinical workflow
	Maintenance	Inadequate maintenance or continuous improvement

Survey content and data collection

For each preliminary recommendation, the second survey asked participants to score the usefulness and content validity. Free-text fields enabled participants to add additional comments on each individual recommendation, as well as CAMPR in its entirety. The study team reviewed the survey before electronic delivery using Qualtrics. We distributed the survey via personalised emails to all previous eligible respondents who had agreed to additional contact. The complete survey is available as online supplemental appendix B.

Data analysis

We conducted a quantitative descriptive analysis of usefulness and content validity using R-3.3.3. Preliminary recommendations with a usefulness and content validity below predetermined thresholds (<50% useful or valid) were excluded, unless the free-text commentary indicated that the usefulness and content validity would greatly improve with revision. The study team reviewed all free-text commentary and refined, reworded or combined recommendations accordingly.

Round 3: application of CAMPR

Modified preliminary version

After refinement and reduction in round 2, the modified preliminary version contained 34 recommendations. We identified 23 development-related recommendations for inclusion in CAMPR, which primarily reflect the four key development domains identified in round 1 (validation, features, timeframe, data access). The remaining 11 implementation-related recommendations, which primarily reflect the five key implementation domains (resources, vision, clinical relevance, workflow integration and maintenance), will be reported on separately.

Iterative validation

Two researchers applied CAMPR to all published readmission prediction models identified in all known systematic reviews (2011, 2016).^{1 2} First, the researchers independently applied CAMPR to one-third of studies, then revised it to improve clarity, resolve discrepancies in application and combine redundant recommendations, which reduced the total number to 15. Then, the researchers independently applied the finalised version to all studies (detailed in online supplemental appendix C) and assessed inter-rater reliability. Conflicts were resolved by discussion.

Data extraction

Manual data extraction occurred concurrently with the application of CAMPR, to better assess characteristics of included studies. Importantly, we extracted eligibility criteria and risk factors for each readmission model, to generate reference lists for future developers (available tables 2 and 3 in the Results section). Examples of other types of extracted data include the readmission timeframe used (relevant to recommendation #6) and the validation technique used (relevant to recommendation #13). Two researchers developed the data extraction tool based on the initial review of one-third of studies. One researcher extracted data from each study, and another reviewed all extractions for completeness and accuracy.

Table 2.

Barriers to developing and implementing readmission models

Domain	Barriers	Evidence*	Example†	Strategies to overcome barriers
Validation	Poor generalisability	Substantial	“[We are] questioning the generalization of the model to our population.” (33)	More advanced modelling strategies than regression Better access to data to improve prediction External rather than internal validation and calibration Account for different clinical subpopulations
	Low discriminative value	Limited	“[The biggest barrier is] having a good model to start with.” (48)
Features	SDH not included	Extensive	“We need to look well beyond SES, etc. to social support and healthcare beliefs and behaviors.” (2)	Consider SDH and HAF when developing models Place SDH and HAF in structured data fields Record better measures of SDH and HAF Improve natural language processing and extraction More research on the factors leading to readmission Assess the possibility of using shorter timeframes
	HAF not included	Substantial	“We need to look beyond academic status and for-profit status … to understand processes.” (2)
Timeframe	Timeframe not optimised	Limited	“30 days is probably too long to provide an accurate prediction.” (48)
Data access	Barriers to data access	Extensive	“No matter how complex and good the model is, it is only as good as the data it has.” (20)	Federal incentives for data sharing Federal sanctions for data blocking Aggregate data sets from multiple sources Include sources with multiple data types Better documentation practices and training Human resources for data entry and validation Better tools or interfaces for data collection Implement better structured data capture processes Reduce unnecessary documentation burdens on healthcare providers, so the quality can improve
	Inadequate interoperability	Substantial	“We need access to databases, especially linked primary and secondary care ones.” (9)
	Insufficient data	Substantial	“We don’t have the necessary data and we don’t know what the necessary data even are.” (27)
	Poor quality data	Substantial	“When we use routinely collected data in EHRs, the quality is less reliable.” (38)
	Lacks current information	Limited	“If and when the factors change, we don’t know what they are - the case managers do.” (34)
Resources	Lacks personnel or expertise	Substantial	“[We lack] staffing resources for adequate capture of data and analytics of accumulated data.” (43)	More support from the hospital administration Better financial support for model development Better packages in statistical software to help developers move beyond logistic regression
	Financial barriers	Substantial	“[There is] reluctance to make the necessary investments to access the EHR’s back end.” (1)
Vision	Competing priorities	Limited	“It [the model] is not a priority… they [the administration] have competing priorities. (14)	Engage key stakeholders and senior decision-makers Establish governance over model implementation Select a clinical or administrative champion Implement financial and other incentives for hospitals
	Lack of leadership	Very limited	“[There is] no operational leadership, so the model hasn’t been implemented.” (21)
Clinical relevance	Poor perceived relevance	Extensive	“Risk score doesn’t necessarily flag the patients in whom we can most usefully intervene.” (9)	Models must determine how providers can intervene More research on hospital processes and needs Invest more human resources to prevent readmission Feedback for providers if readmission averted (or not) Validate and calibrate model in the new clinical setting Ensure users receive the most up-to-date predictions
	Unclear usefulness	Substantial	“[Models must] fit into a workflow where an intervention can be made.” (16)
	Poor perceived accuracy	Substantial	“I’ve found that it [the model] is not accurate at the individual patient level.” (6)
Workflow integration	Poor workflow integration	Extensive	“Getting it inserted into the EHR in a way that requires little provider effort is tough.” (48)	Consider who will receive the results, when, and how Tie integration to physical risk-reduction interventions Train, orient, and support potential users of the model Plan to iteratively improve on integration over time
	Alert fatigue	Very limited	“Clinicians get alert fatigue and stop paying attention to the results.” (33)
Maintenance	Antiquated model or interface	Limited	“Our commercial partner no longer supports the front end they developed [for our model].” (9)	Integrate within system with long-term IT support Designate responsibility of updating model annually

*Extensive evidence (≥8 mentions); substantial evidence (4–7 mentions); limited evidence (2–3 mentions), very limited evidence (1 mention).

†Numbers in parentheses refer to participant study ID.

EHR, electronic health record; HAF, healthcare-associated factors; IT, information technology; SDH, social determinants of health; SES, socioeconomic status.

Table 3.

Critical Appraisal of Models that Predict Readmission (CAMPR)

Number	Recommendation	Studies following recommendation (n)			P value*
		Overall, n=81 (1985–2015)	Early, n=26 (1985–2010)	Recent, n=55 (2011–2015)
#1	Is the model’s purpose and eligibility criteria explicitly stated?	44 (54%)	11 (42%)	33 (60%)	0.14
#2	Does the model consider common patient-related and institution-related risk factors for readmission?	1 (1%)	1 (4%)	0 (0%)	0.33
#3	Does the model consider competing risks to readmission, particularly mortality?	68 (84%)	23 (88%)	45 (82%)	0.43
#4	Does the model identify how providers may intervene to prevent readmission?	4 (5%)	3 (12%)	1 (2%)	0.15
#5	Does the model consider recent changes in the patient’s condition?	39 (48%)	6 (23%)	33 (60%)	0.001**
#6	Is the model’s timeframe an appropriate trade-off between sensitivity and statistical power?	12 (15%)	4 (15%)	8 (15%)	0.92
#7	Does the model exclude either planned or unavoidable readmissions?	42 (52%)	12 (46%)	30 (55%)	0.49
#8	Is the model equipped to handle missing data and is missingness in the development datasets reported?	13 (16%)	4 (15%)	9 (16%)	0.91
#9	Is preprocessing discussed and does the model avoid problematic preprocessing, particularly binning?	27 (33%)	9 (35%)	18 (33%)	0.96
#10	Does the model make use of all available data sources to improve performance?	59 (73%)	15 (58%)	44 (80%)	0.05
#11	Does the model use electronically available data rather than relying on manual data entry?	55 (68%)	17 (65%)	38 (69%)	0.75
#12	Does the model rely on data available in sufficient quantity and quality for prediction?	58 (72%)	18 (69%)	40 (73%)	0.75
#13	Is the model internally validated using cross-validation or a similarly rigorous method?	4 (5%)	1 (4%)	3 (5%)	0.75
#14	Is the model’s discrimination reported and compared with known models where appropriate?	74 (91%)	21 (81%)	53 (96%)	0.07
#15	Is the model calibrated if needed and is calibration reported?	47 (58%)	14 (54%)	33 (60%)	0.61

*Significant at p=0.05; **significant at p=0.001.

*Comparison is between models published earlier (1985–2010) and more recently (2011–2015).

Data Analysis

All analyses were performed in R V.3.6.3. We used Cohen’s kappa and percentage agreement to measure inter-rater reliability. We stratified literature into recent (2011–2015) and early (1985–2010), using the year of CMS HRRP (2011) as our cut-off. We conducted bivariate analyses to assess whether adherence to each recommendation differed between recent and early literature, using an unequal variances t-test. Furthermore, we used Spearman’s rank correlation to examine whether overall adherence to recommendations differed by publication year. When classifying risk factors, we used the same classification as the first systematic review,¹ with the added category ‘institution-related’ as suggested by expert consensus.

Results

Development of CAMPR

Round 1

We successfully contacted 75 out of 81 corresponding authors who developed unique readmission models published from 1985 to 2015, of whom 14 (19%) completed our survey. An additional 49 respondents completed our survey after we publicly distributed it, of which we included 40 who had experience implementing readmission models. The final 54 eligible experts (14 developers and 40 implementers, characterised in online supplemental appendix D) represented 20 unique models, including well-known models such as LACE and CMS-endorsed models. Of 14 developers, only 7 (50%) reported that any institution currently used their model in any capacity. Table 4 reports expert-identified barriers to developing, validating and implementing readmission models, as well as strategies to overcome barriers.

Table 4.

Eligibility criteria for readmission prediction models*

Criterion	Studies (n, %)	Citations (see online supplemental appendix E)
Inclusion criteria
Age	48 (59%)
>18 years (adults only)	25 (31%)	37 40 43 45 47 49–51 55 56 65 67 68 73 74 78 80–82 85 86 88 92 93 95
>65 years (elderly only)	19 (23%)	17–19 21 25 27 28 32–34 38 48 52 60 66 76 77 79 90
Other	5 (6%)	23 36 42 62 93
Condition specific	45 (56%)
Heart failure	16 (20%)	18 21 26 29 33 46 47 67–69 71–75 86
AMI or other cardiovascular	11 (14%)	3 46–58 64–66 70
Pneumonia or other pulmonary	6 (7%)	19 46 47 62 76 77
Multiple	6 (7%)	23 24 28 31 54 63
Other	11 (14%)	80 84–93
Service specific	24 (30%)
Medicine	14 (17%)	35 36 38 40 44 45 78–83 87
Surgery	7 (9%)	57 60 63 89–92
Other	3 (4%)	52 94 95
Beneficiaries	22 (27%)
Medicare	15 (19%)	15 17–19 21 25 28 32 33 41 48 58 60 66 90
Veterans	7 (9%)	22 36 38 39 53 58 61
Exclusion criteria
Disposition	28 (35%)
Left against medical advice	11 (14%)	21 40 42 45 46 50 58 60 61 80 82
Skilled nursing or other care facility	10 (12%)	32 37 43 50 56 61 68 78 86 93
Hospice care	8 (10%)	32 43 49 61 78 85 86 88
Different healthcare system	4 (5%)	35 36 44 94
Transfers	26 (32%)
Transfer-out (to other institution or service)	24 (30%)	17–22 27 33 40 42 45 46 56 58 61–63 68 72 74 76 77 80 93
Transfer-in (from other institution or service)	3 (4%)	47 53 61
Service or condition	24 (30%)
Rehabilitation	10 (12%)	21 43 49–51 56 58 61–63 68 72 74 76 77 80 93
Psychiatric	8 (10%)	22 27 43 50 51 55 56 61
Obstetric/neonatal	8 (10%)	20 23 42 43 47 51 55 56 61
Transplant	4 (5%)	75 84–86
Other (ESRD, trauma, etc)	5 (6%)	15 34 36 42 86
Length of stay	22 (27%)
<24 hours (same-day discharge or procedure)	16 (20%)	17 21 23 27 45–47 51 56 61 66 78 80 92 94
>30 days	5 (6%)	40 57 63 90 91
Hospitalisation type	9 (11%)
Planned (aka elective)	3 (4%)	34 93
Non-acute (aka non-emergent)	3 (4%)	24 42 79
Observation only	2 (2%)	43 51
Study related	7 (9%)
Unavailable for follow-up (no telephone, etc)	5 (6%)	36 37 40 70 89
Cannot consent (poor cognition)	2 (2%)	37 67
Non-English speaking	2 (2%)	67 73
Non-residents (outside the primary service area)	7 (9%)	20 33 36 46 65 67 93
No insurance coverage (for part or all of the data extraction period)	5 (6%)	42 56 60 66 77
Data-related (not recent or validated)	3 (4%)	32 33 52
Readmitted for different condition	3 (4%)	17–19

An extended version of this table is available in online supplemental appendix D.

*This table does not cover exclusions based on mortality (see recommendation #3) or missingness (see recommendation #8).

AMI, acute myocardial infarction; ESRD, end-stage renal disease.

Rounds 2 and 3

We had permission to reconnect with 22 previous respondents, of whom 5 (23%) completed our second survey.

Application of CAMPR

We included 81 published readmission models in our critical appraisal.^15–95 We found that published models followed 6.8 out of 15 recommendations (45%) on average. Fifty-five out of 81 (68%) followed less than half the recommendations, and no study followed every recommendation, suggesting an opportunity for improvement. Table 5 presents the percentages of published readmission models following each recommendation, stratified by publication year. Models published recently (2011–2015, n=55, 68%) followed significantly more recommendations than models published earlier (1985–2010, n=26, 32%) (7.1 vs 6.1, p=0.03), and publication year weakly correlated with recommendations followed (ρ=0.27, p=0.02), suggesting slight improvement in model quality over time as the field developed. Model types included regression (77, 95%), random forest (3, 4%), neural network (3, 4%), decision tree (2, 2%), discriminant analysis (2, 2%), support vector machine (1, 1%) and unclear (1, 1%). We found moderate-to-high inter-rater reliability for applying CAMPR (Cohen’s kappa=0.76, agreement=88%). Here, we summarise each recommendation in CAMPR and present the critical appraisal results. Additional results are in online supplemental appendix D. The complete dataset is available on request. CAMPR is available as online supplemental appendix E.

Table 5.

Risk factors included in readmission prediction models

Risk factor	Studies (n, %)		Citations (see online supplemental appendix E)
	Overall (n=81)	General* (n=12)
Demographics
Age	38 (47%)	6 (50%)	15–22 24 25 27 28 31 32 39 41 42 50 56 60–63 65–67 69 70 72 74 76 77 79 80 82 90 93 95
Sex/gender	30 (37%)	5 (42%)	15–21 25 27 28 31 41 50 53 56 60–63 65–67 69 76 77 79 83 85 90 93
Race/ethnicity	12 (15%)	2 (17%)	15 16 26 27 31 41 46 52 57 63 71 72
Disease related
Comorbidities	50 (62%)	4 (33%)
Cardiovascular (heart failure, infarction, etc)	39 (48%)	4 (33%)	17–21 23 24 26 27 31–34 36 38 41 42 46 53 57 60–67 69 76–79 82 83 86 90 93 94
Oncological (metastatic, leukaemia, etc)	30 (37%)	4 (33%)	17–21 23 24 27 32 34 41 42 45 46 53 54 61–63 66 76–80 82 83 90 91 93
Pulmonary (COPD, pneumonia, asthma, etc)	29 (36%)	3 (25%)	17–21 23 24 26 27 31 38 42 53 57 60 61 63 65 66 76–79 82 83 86 90 91 93
Endocrine (diabetes, etc)	26 (32%)	2 (17%)	17–19 21 23 24 26 31–34 41 42 53 57 60 62–64 66 70 77 82 83 90 93
Genitourinary (renal failure, etc)	26 (32%)	3 (25%)	17–21 23 24 26 27 31 42 53 57 60 62–64 66 76 78–80 83 86 90 91
Psychiatric (alcohol/drug use, psychosis, etc)	21 (26%)	1 (8%)	18–21 23 24 29 31 39 46 52 53 61 62 76 77 82 90 93–95
Haematological (fluid disorder, anaemia, etc)	20 (25%)	2 (17%)	17–21 23 24 27 31 53 57 62–64 66 83 86 90 92 93
Neurological (stroke, paralysis, etc)	17 (21%)	3 (25%)	17–21 23 27 31 32 41 42 46 66 82 86 89 93
Gastrointestinal (cirrhosis, obstruction, etc)	16 (20%)	3 (25%)	18–21 23 24 27 42 46 53 77 83 86 88 90 93
End of life (dementia, malnutrition, cachexia, etc)	14 (17%)	3 (25%)	17–21 23 27 42 46 63 66 77 83 90
Musculoskeletal-dermatological (injuries, etc)	12 (15%)	2 (17%)	19–21 23 27 31 41 53 61 62 86 93
Infectious (sepsis, shock, etc)	10 (12%)	–	17–19 21 23 53 63 66 70 90
Obesity related (BMI, sleep apnoea, etc)	4 (5%)	–	62 63 78 90
Obstetric-gynaecological (pregnancy, etc)	3 (4%)	1 (8%)	20 23 53
Severity scores	29 (36%)	8 (67%)
Charlson Comorbidity Index	14 (17%)	7 (58%)	16 20 32 37 40 46 50 51 55 56 61 74 92 95
Other (Elixhauser, Tabak, SOI, PMC-RIS, etc)	10 (12%)	2 (17%)	22 25 28 29 38 47 52 53 56 80
Disease specific (GRACE, MELD, etc)	6 (7%)	–	70 84 85 90 91 93
Laboratory values	22 (27%)	1 (8%)	21 33 35 36 45 46 51 57 60 65 69–72 76 77 84–88 91
Complications (post-procedure, etc)	8 (10%)	–	57 63 85 87 89 90 92 94
Emergent admission (or acute admission)	6 (7%)	4 (33%)	37 38 42 55 56 95
Complexity (No of medical conditions)	6 (7%)	–	28 39 52 67 69 79
Signs or symptoms (dyspnoea, ascites, etc)	5 (6%)	–	23 63 73 88 90
Condition description (chronic, high risk, etc)	4 (5%)	1 (8%)	16 25 68
Functional ability
Functional status (or assistance with ADL)	5 (6%)	–	32 36 40 63 90
Mental status (MMSE, etc)	3 (4%)	1 (8%)	34 39 77
Dependencies (ambulation, ventilator, etc)	3 (4%)	–	39 63 94
Recent falls	2 (2%)	–	41 52
Other (bedridden, incontinent, sedentary, etc)	2 (2%)	–	41 67
Healthcare utilisation
Previous admissions	32 (40%)	7 (58%)	15 20 24 29 31–34 39–42 45–47 49 50 52 56 61 62 64 71 76 78–80 82 84 87 93 95
Length of stay	26 (32%)	6 (50%)	28 37 40 45–47 50 51 53–56 61–63 68 69 74 76 78 79 83 86 91 92 95
Previous emergency visits	11 (14%)	5 (42%)	35 37 46 49 50 55 56 74 79 87 95
Previous outpatient visits (specialist or primary care)	7 (9%)	–	29 41 54 75 78 86 95
Previous procedures	5 (6%)	–	26 60 64 65 89
Current admission	25 (31%)	3 (25%)
Disposition (home, SNF, rehab, home health, etc)	8 (10%)	–	26 41 57 63 85 86 92 94
Source (emergency, SNF, outpatient, transfer, etc)	7 (9%)	1 (8%)	16 54 61 63 65 79 86
Ward (medical, surgical, neurology, etc)	3 (4%)	1 (8%)	22 27 54
Reimbursement amount	3 (4%)	1 (8%)	15 16 88
Other (consultancies, timing, refusal, etc)	9 (11%)	2 (17%)	16 29 31 45 47 62 69 71 75
Current procedures	15 (19%)	2 (17%)
Performed vs not (binary)	9 (11%)	2 (17%)	15 20 22 45 46 51 53 60 77
Type (arthroplasty, splenectomy, etc)	6 (7%)	1 (8%)	26 46 51 57 60 77
Characteristic (high risk, operation time, open, etc)	6 (7%)	1 (8%)	20 57 63 64 90 92
Status (urgent, emergent, elective, etc)	4 (5%)	–	57 60 63 90
Medication related
Specific medications (in-hospital or home)	19 (23%)	1 (8%)
Steroid	5 (6%)	–	46 57 62 63 90
ACE inhibitor (or ARB)	3 (4%)	–	62 68 69
Other cardiac (beta-blocker, nitrate, etc.)	5 (6%)	1 (8%)	34 46 60 70 82
Immunosuppressant	3 (4%)	–	60 76 77
Antibiotic	3 (4%)	–	46 62 88
Opioid	3 (4%)	–	80 82 86
Other (statin, anticoagulant, insulin, etc)	7 (9%)	–	46 62 70 82 86 90 92
No of medications (polypharmacy)	7 (9%)	2 (17%)	34 46 50 52 80 82 85
Social determinants of health
Zip code (or home address)	11 (14%)	2 (17%)
Distance from hospital	4 (5%)	–	22 64 78 87
Socioeconomic status (based on zip code)	4 (5%)	2 (17%)	16 24 29 42
Rurality (urban, suburban, rural, etc)	3 (4%)	–	24 64 69
Insurance status	10 (12%)	2 (17%)	15 26 27 29 32 40 50 65 84 87
Living arrangement (alone, SNF, homeless, etc)	7 (9%)	–	39 41 67 69 76 77 87
Marital status	7 (9%)	2 (17%)	29 34 39 40 46 50 76
Disability status	6 (7%)	–
Disabled vs not (binary)	5 (6%)	–	22 31 32 41 78
Type (developmental, visual, cognitive, hearing, etc)	3 (4%)	–	31 32 41
Patient-generated health data	5 (6%)	–	32 36 40 41 61
Smoking status (current, former, etc)	3 (4%)	1 (8%)	34 63 90
Education level	2 (2%)	–	41 89
Annual income	2 (2%)	–	41 76
Institution related
Rurality (urban, suburban, rural, etc)	2 (2%)	–	15 26
Standardised admission ratios	2 (2%)	1 (8%)	16 31
Identification code	2 (2%)	1 (8%)	42 52

We could not extract factors from seven studies, either due to poor reporting or lack of feature selection.30 43 44 48 58 59 81 An extended version of table is available in online supplemental appendix D.

*'General' indicates models for the general hospitalised population, and is not restricted to any one target population.16 20 27 34 37 42 47 49–51 55 56

ACE, Angiotensin-converting enzyme; ADL, activities of daily living; ARB, Angiotensin receptor blocker; BMI, body mass index; COPD, chronic obstructive pulmonary disease; DNR, do not resuscitate; MMSE, Mini Mental State Exam; SNF, skilled nursing facility; SOI, severity of illness.

Recommendation #1: is the model’s purpose and eligibility criteria explicitly stated?

About the recommendation

Readmission models traditionally serve one of two purposes, or intended applications: (1) to identify patient candidates for targeted interventions to prevent readmission, or (2) to risk-adjust readmission rates for hospital quality comparison.¹ Developers should clearly state which purpose their model serves, one or both. Developers should also define the target population by specifying eligibility criteria for patient inclusion in model development. Specifying eligibility criteria is critical to ensure implementers understand when each model applies, as unjustified application is a major reason why predictions fail.^{96 97}

Critical appraisal results

Eighteen out of 81 studies (22%) did not define their model’s purpose. Of the remaining models, 46 (57%) were for preventing readmission, 15 (19%) were for hospital quality comparison and 2 (2%) were for both. Table 4 provides an abbreviated reference list of eligibility criteria for published readmission models (the full reference list is available in online supplemental appendix D). Twenty-seven models (33%) did not specify their eligibility criteria.

Recommendation #2: does the model consider common patient-related and institution-related risk factors for readmission?

About the recommendation

Developers should show that they considered risk factors or features that were included in previous models. Notably, institution-related factors such as hospital name should not be used in models for hospital quality comparison, as they can mask differences in hospital quality.

Critical appraisal results

Table 5 provides an abbreviated reference list of known risk factors for readmission and their frequency of inclusion in published models (the full reference list is available as online supplemental appendix D). Based on expert consensus and the existing literature, we identified seven categories of factors.¹ Categories included (1) demographics (included in 75 models or 93%), (2) disease related (80, 99%), (3) functional ability (21, 26%), (4) healthcare utilisation (66, 81%), (5) medication related (33, 41%), (6) social determinants of health (53, 65%), (7) institution related (16, 23%). Five studies (out of 15, 33%) mistakenly used institution-related risk factors in models for hospital quality comparison.

Recommendation #3: does the model consider competing risks to readmission, particularly mortality?

About the recommendation

Death is a competing risk to readmission and may substantially impact readmission prediction depending on the target population.^{63 67 68} A high mortality rate may reduce model discrimination because death and readmission share similar predictive features. Ignoring mortality may limit insight about risk factors, and unaccounted changes in mortality may cause model drift. Developers should indicate that they accounted for both in-hospital and post-discharge mortality, as well as other competing risks to readmission (eg, transfers).²⁸

Critical appraisal results

Thirteen models (16%) did not account for mortality, 40 (49%) accounted for in-hospital mortality only, 5 (6%) accounted for post-discharge mortality only and 21 (26%) accounted for both.

Recommendation #4: does the model identify how providers may intervene to prevent readmission?

About the recommendation

The expert group recognised that building actionable models, which identify where providers can intervene on risk factors to prevent readmissions, is critical to clinical usefulness. An actionable model may (1) identify modifiable risk factors on the individual level,^{36 62 90} or (2) identify which individuals will benefit most from intervention, which may not coincide with readmission risk. Notably, non-modifiable risk factors like age can obscure modifiable ones like polypharmacy or quality of care^{60 98 99}; therefore, managing collinearity¹⁰⁰ is important. In the future, predicting benefit will become easier as options for intervention become more well researched.¹⁰¹

Critical appraisal results

Four published models (5%) identified modifiable factors on the individual level. No models have predicted which individuals would benefit most from intervention.

Recommendation #5: does the model consider recent changes in the patient’s condition?

About the recommendation

A model that does not account for recent changes in the patient’s condition may give an outdated prediction, limiting its clinical usefulness and eroding trust in its predictions. The expert group recommended that models that give predictions near hospital discharge (ie, most current models) should account for changes during hospitalisation, including treatment effects, hospital-acquired conditions and social support status.

Critical appraisal results

Thirty-nine models (48%) accounted for changes during hospitalisation.

Recommendation #6: is the model’s timeframe an appropriate trade-off between sensitivity and statistical power?

About the recommendation

Researchers initially selected the 30-day timeframe as the optimal trade-off between statistical power and likelihood of association with the index admission.²⁸ As common data models and health information exchange support larger datasets for model development, shorter timeframes such as 7 days may enable greater sensitivity for readmissions associated with the index admission without loss in statistical power.¹⁰² Therefore, developers should consider assessing prediction accuracy using multiple timeframes, as relevant to the clinical context and dataset size, to determine the best trade-off between sensitivity and statistical power. Timeframes should begin at discharge (as standardised by CMS)¹⁰³ to prevent immortal person–time bias.⁶³

Critical appraisal results

Sixty-three models (78%) used the standardised 30-day timeframe adopted by CMS, while 2 (2%) used 7 days, 3 (4%) 28 days, 5 (6%) 60 days and 9 (11%) 1 year. Twelve studies (15%) considered more than one timeframe, of which 7 (9%) modelled readmission risk using hazard rates. Nine studies (11%) inappropriately defined timeframes as beginning at admission, rather than discharge, and 14 (17%) did not specify when their timeframe began.

Recommendation #7: does the model exclude either planned or unavoidable readmissions?

About the recommendation

Planned readmission is defined as non-acute readmission for scheduled procedures. Planned readmissions should be excluded, as consistent with the standardised definition of all-cause readmission.^{17–19 66} Unavoidable readmission is defined more broadly as readmission not preventable through clinical intervention.²⁸ As researchers develop standardised algorithms to more effectively identify unavoidable readmissions,^{47 61 104} using the broader definition may enable greater sensitivity and improve the relevance of predictions to the clinical setting. Therefore, developers should consider excluding unavoidable readmissions if it is useful, such as in multiple sclerosis, where the disease inevitably progresses and later readmissions become increasingly unavoidable. Notably, exclusion criteria can be highly complex and require third-party processing (eg, Vizient). Ideally, developers should publish their code. If not, the readmission outcome should be sufficiently defined to ensure transparency and reproducibility.¹⁰⁵

Critical appraisal results

Thirty-nine models (48%) did not explicitly exclude planned readmissions. The remaining models either excluded planned readmissions (38, 47%) or excluded unavoidable readmissions more broadly (4, 5%).

Recommendation #8: is the model equipped to handle missing data and is missingness in the development datasets reported?

About the recommendation

Developers should explicitly state whether their model handles missingness and how, such as designating a ‘missing’ category for categorical variables, or multiple regression imputation for continuous variables. Dropping individuals with excess missingness is problematic because it decreases models’ generalisability to future individuals with excess missingness and falsely increases model performance in cases of structural missingness. Developers should also report on missingness in the datasets used for model development, so that implementers can determine potential generalisability to their real-world datasets.

Critical appraisal results

Only 34 studies (42%) discussed how their model handled missingness. Of these, 20 (25%) used one or more inappropriate techniques, including (1) dropping individuals with excess missingness (17, 21%), and (2) binning or imputation which was done improperly (3, 4%).

Recommendation #9: is preprocessing discussed and does the model avoid problematic preprocessing, particularly binning?

About the recommendation

Developers should explain their data preprocessing methods, because problematic methods may produce models with less-than-optimal predictive performance.⁹⁶ One example of a problematic method is binning.¹⁰⁶ Originally intended to improve interpretability, binning can cause information loss, and is no longer justifiable given users’ need for accurate predictions and modern interpretability techniques. In particular, manual or arbitrary binning, without clustering or splines, may decrease performance and introduce noise.¹⁰⁷

Critical appraisal results

Only 27 studies (33%) discussed one or more data preprocessing techniques, despite mostly using regression models, which can be highly sensitive to small changes. Commonly discussed techniques included binning, interaction terms and transformations to resolve skewness, non-linearity and outliers.

Recommendation #10: does the model make use of all available data sources to improve performance?

About the recommendation

Developers should make use of publicly available data sources where possible and appropriate to the model’s purpose, such as the Social Security Death Index to determine post-discharge mortality (see recommendation #3) or curated public datasets to externally validate (see recommendation #13). Other data sources such as health information exchanges can help assess readmission at multiple institutions, which is desirable to better estimate the true readmission rate. When considering data sources from multiple institutions, such as with health information exchanges, developers should account for hospital-level patterns and clustering of readmission risk, which may occur because quality of care and data collection practices vary between institutions.^{36 61 68}

Critical appraisal results

In the literature, data sources included claims (19, 23%), administrative datasets (33, 41%), electronic health records (42, 52%), disease-specific registries (12, 15%), research datasets (11, 14%), death registries (9, 11%), health information exchanges or linkages (4, 5%), and surveys or patient-generated health data (3, 4%). Seventy-five studies (93%) assessed readmission at only one institution, likely underestimating the true readmission rate. Thirty-nine studies (48%) used a single data source (administrative datasets: 15, 19%; electronic health records: 17, 21%).

Recommendation #11: does the model use electronically available data rather than relying on manual data entry?

About the recommendation

Developers should incorporate risk factors that will be available electronically at the time of prediction and avoid manual data entry by providers or research assistants. Manual data entry may inhibit widespread implementation, by consuming human resources and preventing automated generation of predictions.

Data extraction results

Twenty-six models (32%) relied on manual data entry.

Recommendation #12: does the model rely on data available in sufficient quantity and quality for prediction?

About the recommendation

Developers should indicate whether data included in their model can be accessed in sufficient amounts and quality for development and implementation. ‘Sufficient’ is subjective and requires consideration of real-world missingness. Automated quality assurance, which identifies erroneous entries (eg, age>120 years) and incorrect data combinations (eg, former smoker YES, never smoker YES), may help to improve quality.

Critical appraisal results

Twenty-three studies (28%) identified problems with either data quantity (17 out of 23, 74%) or quality (8 out of 23, 26%).

Recommendation #13: is the model internally validated using cross-validation or a similarly rigorous method?

About the recommendation

The importance of using repeated k-fold cross-validation or a similarly rigorous method is well established. Split-sample validation is insufficient and may cause unstable and suboptimal predictive performance.^108–110 If the model is intended for generalised use at more than one institution, developers or implementers should confirm external validity using one or more external, representative and independent datasets, from another institution or source. Internal validation alone is insufficient to ensure generalisability.¹⁰⁸

Critical appraisal results

Ten models (12%) were not validated at all, 64 (79%) were internally validated only and 7 (9%) were internally and externally validated. For internal validation, 46 (57%) used random split-sample, 11 (14%) used split-sample by time, 12 (15%) used bootstrapping, 3 (4%) used cross-validation and 1 (1%) used out-of-bag estimates.

Recommendation #14: is the model’s discrimination reported and compared with known models where appropriate?

About the recommendation

It is commonly accepted practice to prominently and clearly report discrimination using appropriate and well-known measures beyond just the concordance (c) statistic.¹¹¹ Where possible, comparison with an established baseline is essential, because so many models already exist. Developers should compare performance using statistical tests with cross-validation or another method, and only compare models with similar eligibility criteria.

Critical appraisal results

Seven models (8%) did not report discrimination. Commonly reported measures included the c statistic (47, 58%), sensitivity or specificity (23, 28%), area under the receiver operating characteristic curve (19, 23%), negative or positive predictive value (18, 22%), integrated discrimination improvement (5, 6%) and net reclassification improvement (3, 4%).

Recommendation #15: is the model calibrated if needed and is calibration reported?

About the recommendation

Proper calibration is critical for sorting patients in descending order of readmission risk for making intervention decisions. It is commonly accepted practice to report calibration using calibration curves with no binning.^{112 113} Reporting the Hosmer-Lemeshow (HL) goodness-of-fit statistic is insufficient, as a non-significant HL statistic does not imply the model is well calibrated, and the HL statistic is often insufficient to detect quadratic overfitting effects common to support vector machines and tree-based models.¹¹²

Critical appraisal results

Thirty-seven models (44%) did not assess calibration. Commonly reported measures included the HL statistic (29, 36%) and observed-to-expected ratios (17, 21%).

Discussion

In this study, we critically appraised readmission models using 15 Delphi-based expert recommendations for development and validation. Interestingly, we found that many published readmission models did not follow the experts’ recommendations. This included failure to internally validate (12%), failure to account for readmission at other institutions (93%), failure to account for missing data (68%), failure to discuss data preprocessing (67%) and failure to state the model’s eligibility criteria (33%). The high prevalence of these weaknesses in model development identified in the published literature is concerning, because these weaknesses are known to compromise predictive validity. Identification of weaknesses in these domains should undermine confidence in a model’s predictions.

In our expert surveys, several lessons emerged, most notably about improving models’ relevance to clinical care and integration of predictions into clinicians’ workflows. In particular, experts expressed concern that models identified the highest-risk patients rather than the patients who might benefit most from intervention, which led to recommendation #4. Experts also noted that the published literature in existing systematic reviews and therefore our critical appraisal is focused on development and internal validation. This suggests that literature on external validation and implementation is less common. Additional efforts to research external validation and implementation could improve readmission models, by making them more applicable to a broader patient population.

In the future, CAMPR may be a convenient teaching aid for model implementers and users at healthcare institutions, such as clinicians and healthcare administrators, as well as for model developers in academic and commercial research. CAMPR does not explain the detailed logic and methods of developing and implementing predictive models, and those looking for comprehensive advice should consult other resources. Finally, CAMPR is not intended as a reporting standard for academic studies, and responses to CAMPR recommendations should not be used to derive an overall score. An overall score may disguise critical weaknesses that should diminish confidence in model predictions. Rather than generating an overall score, consider the potential impact of failing to follow each recommendation, and how that may interact with the use of that model in the given patient population.

We developed CAMPR using a modified Delphi process consisting of two online rounds, which we found faster and more practical than conducting the traditional in-person meetings and three rounds. Beyond readmission modelling, other predictive modelling domains in healthcare (eg, sepsis risk, mortality risk, etc) could benefit from similar guidance. Thinking beyond better modelling techniques is essential, or model predictions will remain of limited clinical use. This includes thinking about how to generate better datasets, thinking about model drift and maintenance over time,¹¹⁴ and thinking about how to clinicians should act on predictions.

Limitations

The study used a modified Delphi process, which may lack rigour compared with the traditional Delphi process. We used an ‘opt-in’ process to recruit experts, and this self-selection bias may have led to missed recommendations or opinions. Fewer participants responded to the second round than expected, although the number was sufficient for the Delphi process. Future revision of recommendations will likely be necessary as the field advances and developers adopt more modern techniques. CAMPR is not intended as a reporting standard, and a more formal evaluation of construct validity and generalisability would be needed before it could be used as such.

Data availability statement

Data are available on reasonable request. Please email the corresponding author.

Ethics statements

Patient consent for publication

Not required.

Ethics approval

The Columbia University Medical Center Institutional Review Board approved the studies (Protocol AAAR0148). We used online surveys to collect opinions about predictive models from expert researchers. In accordance with 45CFR46.101, the Columbia University Medical Center Institutional Review Board determined this to be a Category 2 exemption and did not require written informed consent.