Abstract
Background:
Urgent, unplanned hospital readmissions are increasingly being used to gauge the quality of care. We reviewed urgent readmissions to determine which were potentially avoidable and compared rates of all-cause and avoidable readmissions.
Methods:
In a multicentre, prospective cohort study, we reviewed all urgent readmissions that occurred within six months among patients discharged to the community from 11 teaching and community hospitals between October 2002 and July 2006. Summaries of the readmissions were reviewed by at least four practising physicians using standardized methods to judge whether the readmission was an adverse event (poor clinical outcome due to medical care) and whether the adverse event could have been avoided. We used a latent class model to determine whether the probability that each readmission was truly avoidable exceeded 50%.
Results:
Of the 4812 patients included in the study, 649 (13.5%, 95% confidence interval [CI] 12.5%–14.5%) had an urgent readmission within six months after discharge. We considered 104 of them (16.0% of those readmitted, 95% CI 13.3%–19.1%; 2.2% of those discharged, 95% CI 1.8%–2.6%) to have had a potentially avoidable readmission. The proportion of patients who had an urgent readmission varied significantly by hospital (range 7.5%–22.5%; χ2 = 92.9, p < 0.001); the proportion of readmissions deemed avoidable did not show significant variation by hospital (range 1.2%–3.7%; χ2 = 12.5, p < 0.25). We found no association between the proportion of patients who had an urgent readmission and the proportion of patients who had an avoidable readmission (Pearson correlation 0.294; p = 0.38). In addition, we found no association between hospital rankings by proportion of patients readmitted and rankings by proportion of patients with an avoidable readmission (Spearman correlation coefficient 0.28, p = 0.41).
Interpretation:
Urgent readmissions deemed potentially avoidable were relatively uncommon, comprising less than 20% of all urgent readmissions following hospital discharge. Hospital-specific proportions of patients who were readmitted were not related to proportions with a potentially avoidable readmission.
Urgent, unplanned hospital readmissions are increasingly being used to measure institutional or regional quality of care.1–4 The public reporting of readmissions and their use in considerations for funding suggest a belief that readmissions indicate the quality of care provided by particular institutions. However, urgent readmissions are an informative metric only if we know what proportion of them are avoidable. If they are rarely avoidable, they would be a poor gauge of the quality of patient care.
Current estimates of the proportion of urgent readmissions that are avoidable are unreliable. In a systematic review of 34 studies that reviewed how many readmissions were avoidable, 3 of the studies relied solely on combinations of administrative diagnostic codes, and most used undefined or subjective criteria.5 In addition, most of the studies were conducted at a single centre and used only one reviewer. The proportion of readmissions deemed avoidable varied widely, from 5.1%6 to 78.9%,7 which reflected in part the lack of standardized and reliable methods to identify avoidable readmissions.
We conducted a multicentre prospective cohort study to elicit judgments from multiple practising physicians who used standard implicit review methods to determine whether urgent readmissions were potentially avoidable. We analyzed these judgments using a latent class analysis. We also measured the proportion of readmissions deemed avoidable and compared hospital-specific proportions of all-cause and avoidable readmissions.
Methods
Study design
This was a secondary analysis of a multicentre prospective cohort study involving patients discharged to the community after elective or urgent hospital care between October 2002 and July 2006. Patients discharged from medical and surgical services of 11 hospitals (6 teaching, 5 community) in five cities in the province of Ontario were invited to participate if they had no cognitive impairment, had a telephone and provided written informed consent. We excluded patients if they were less than 18 years old, were discharged to a nursing home, or were unable to communicate with study staff because of language difficulties and did not have someone to translate.
Patient recruitment was broadly inclusive, with a median of five (interquartile range 3–6) patients recruited per hospital per day. Each day, the recruiting nurses checked with the senior resident or a staff person to identify which patients were likely to be discharged to the community within the next 72 hours. Patients were approached to participate in the study in an informally random order, with recruitment stopping at the end of the recruiter’s work day.
The study was approved by the Ottawa Hospital Research Ethics Board and the ethics board of each participating hospital. All physician and hospital services in Ontario are covered by the province’s universal health care plan.
Data collection
Each patient was interviewed in hospital by study personnel to identify their baseline functional status and chronic medical conditions. The patient’s chart and hospital discharge summary were reviewed to confirm chronic medical conditions and to record details of the index hospital admission, including diagnoses in hospital and medications at discharge. To maximize completeness of patient follow-up, we registered multiple contact telephone numbers for each patient, including numbers for a secondary contact person. We also mailed letters to patients after discharge, mailed letters to their physicians, and telephoned patients and physician’s offices.
We contacted patients by telephone at one, three and six months after discharge to identify all (urgent and nonurgent) readmissions to hospital. To be classified as urgent, the readmission had to have been unplanned at the time of discharge from the index hospital admission, or the delay between the decision to readmit and the actual readmission was less than four weeks. The indication or primary diagnosis for urgent readmissions did not need to be related to the index admission.
For urgent readmissions, we elicited the following information from the patient and the hospital record: date of readmission; events leading up to the readmission, including actions the patient took in response to those events (e.g., visit to family physician or emergency department, or no action); the health care system’s responses to, and interventions for, these events; and documentation in the chart about the reason for the readmission. Trained registered nurses and study personnel created case reports that summarized the patient’s diagnosis, the index hospital admission (care in hospital and treatments received), care and treatment received between the index admission and the readmission, events after discharge from the index admission, and the readmission (presentation, relevant history, course in hospital, treatments, diagnoses and disposition).
Physician reviews
We recruited a convenience sample of 35 Canadian and American board-certified physicians to review the summaries of the urgent readmissions. These physicians were recruited from participants in the Canadian Adverse Event Study8 and personal contacts. The physician reviewers provided written informed consent to participate in the study and received standardized training similar to that used by Baker and coauthors8 and Brennan and colleagues.9 Reviewers conducted their reviews using a Web-based application available to them through a secured Internet connection. All identifying personal health information was removed from each summary.
Reviewers used a six-point ordinal scale (used in most major adverse event studies10,11) to rate whether the readmission was an adverse event (poor clinical outcome due to medical care) and whether the adverse event could have been avoided (see Appendix 1, at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110400/-/DC1). A readmission given a rating above three in both domains was classified as potentially avoidable by that reviewer. Finally, each reviewer indicated which of the following factors contributed to the adverse event: medications; procedures; nosocomial infection; diagnostic error; management error; system error; surgical complication; or other. Definitions and examples of these categories were provided to reviewers during their training (see in Appendix 2, at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110400/-/DC1).
Forty-seven of the readmissions were reviewed by 30 of the reviewers. These readmissions were randomly selected for inclusion in another study (unpublished data) that required 30 reviewers per readmission. This number of reviewers was chosen to ensure that reviewer sensitivity and specificity could be measured over a wide range of reviewer types, given that 14% of reviewers are expected to be statistical “outliers” and to ensure that at least one outlier reviewer was included in the review with more than 95% certainty.12 For the remaining readmissions, we used four of the reviewers because that number is the minimum required to calculate goodness-of-fit statistics.13
Main outcome
All patients were followed up to the earliest of the first planned readmission, the first urgent readmission, death or six months after discharge from the index hospital admission. The main outcome was a potentially avoidable readmission. A readmission was deemed to be potentially avoidable if the probability that it was truly avoidable (determined from the latent class model described below) exceeded 50%. We chose the cutoff point of 50% because it represents the point at which the readmission was more likely than not to be a truly avoidable readmission.
Statistical analysis
The probability that a readmission was truly avoidable was quantified using latent class analysis. This type of latent variable model uses maximum likelihood methods to estimate the probability that a readmission was truly avoidable based on the independent classifications by multiple reviewers.14,15 Modelling was done with PROC LCA in SAS 9.2, in which the sensitivity and specificity of each reviewer was allowed to vary independently.16 Final output for the models was the posterior probability that each readmission represented a truly avoidable readmission.
To determine whether the proportion of readmissions deemed avoidable varied by their timing, we created a logistic regression model with all urgent readmissions that had “avoidable” status as the outcome and time from the index hospital admission as the primary independent variable. Fractional polynomials17–19 were used to identify any nonlinear associations in this model. We compared hospital-specific proportions of all-cause and avoidable readmissions using the Pearson correlation coefficient. We also compared hospital rankings based on these proportions using the Spearman correlation coefficient.
Results
Between October 2002 and July 2006, we enrolled 5035 patients from the 11 study hospitals. We excluded 223 patients because they refused to participate when contacted for follow-up (n = 112), they were lost to follow-up before the 30-day interview (n = 101), or they were admitted to a nursing home during the first month after discharge (n = 10). The remaining 4812 patients (95.6% of those enrolled) were included in the study.
Characteristics of the study cohort are reported in Table 1. Patients were middle-aged, and 93.3% were living independently without the need for support of activities of daily living. Most of the patients were free of serious comorbidities, with more than 75% having a Charlson Comorbidity Index of zero.20 Most of the index admissions (58.1%) were urgent, and almost half (44.9%) were to a medical service. The most common reasons for admission included acute coronary syndromes, cancer diagnosis or related complications and heart failure. Coronary artery bypass grafting and arthroplasty were the most common procedures.
Table 1:
Characteristics of 4812 patients discharged from hospital who were included in the study
| Urgent readmission; no. (%) of patients* |
|||
|---|---|---|---|
| Characteristic | Total no. (%)of patients* n = 4 812 |
Potentially avoidable n = 104 |
Not avoidable n = 545 |
|
Patient | |||
| Age at index admission, yr, mean (SD) | 61.3 (17.0) | 64.2 (17.0) | 65.8 (16.4) |
| Female sex | 2530 (52.6) | 58 (55.8) | 294 (53.9) |
| Lived alone before index admission | 1127 (23.4) | 19 (18.3) | 146 (26.8) |
| Dependent for ≥ 1 activities of daily living | 323 (6.7) | 8 (7.7) | 51 (9.4) |
| Charlson Comorbidity Index > 0 | 1128 (23.4) | 36 (34.6) | 219 (40.2) |
| Admitted to hospital in previous 6 mo | 1557 (32.4) | 48 (46.2) | 242 (44.4) |
| Visited emergency department in previous 6 mo | 1750 (36.4) | 48 (46.2) | 298 (54.7) |
| Has regular physician | 4580 (95.2) | 102 (98.1) | 53 (97.2) |
|
Index admission | |||
| Admission to medical service | 2160 (44.9) | 53 (51.0) | 354 (65.0) |
| Urgent admission | 2796 (58.1) | 63 (60.6) | 414 (76.0) |
| Emergent surgery during admission | 391 (8.1) | 6 (5.8) | 24 (4.4) |
| Length of stay, d, median (IQR) | 5 (2–8) | 5.5 (3–10.5) | 6 (4–10) |
| No. of discharge medications, median (IQR) | 4 (2–7) | 5 (3–9) | 6 (3–9) |
| New medication at discharge | 2340 (48.6) | 59 (56.7) | 274 (50.3) |
| Complication in hospital | 615 (12.8) | 15 (14.4) | 111 (20.4) |
Note: IQR = interquartile range, SD = standard deviation.
Unless stated otherwise.
Patients were observed for a median of 175 days (interquartile range 144–177). The most common reason for the end of follow-up was study completion (n = 3514 [73.0%]), followed by urgent readmission (n = 649 [13.5%]), elective readmission (n = 283 [5.9%]), loss to follow-up after the 30-day interview (n = 266 [5.5%]) and death (n = 100 [2.1%]).
Potentially avoidable readmissions
Summaries of the 649 urgent readmissions were independently reviewed and rated by either 4 (n = 602 [92.8%]) or 30 (n = 47 [7.2%]) physician reviewers.
Overall, 104 readmissions (2.2% of the 4812 index admissions, 95% confidence interval [CI] 1.8%–2.6%; 16.0% of the 649 urgent readmissions, 95% CI 13.3%–19.1%) were considered to have been potentially avoidable. In the latent class analysis, the model-based estimate for the prevalence of avoidable readmissions was 18.1% (95% CI 14.0%–22.3%). This corresponded closely with the proportion of readmissions with posterior probability exceeding 50% and indicated that the fit of the latent class model was excellent (see the section “Model fit” in Appendix 3, at www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110400/-/DC1). Figure 1 illustrates that the latent class model was broadly discriminative between readmissions that were and were not deemed avoidable. This is shown by most of the readmissions having a probability of being avoidable of less than 10% or greater than 90% (Figure 2).
Figure 1:
Distribution of probability that an urgent, unplanned readmission was avoidable. Of the 649 urgent readmissions reviewed, those that had a probability of more than 50% were classified as being potentially avoidable.
Figure 2:
Association between time since discharge from index hospital admission, plotted by decile, and the probability of readmission classified as potentially avoidable. This analysis was limited to the 649 urgent readmissions. Error bars = 95% confidence intervals.
Patients who were readmitted had more extensive comorbidities and previous hospital admissions than other patients in the cohort (Table 1). The potentially avoidable readmissions are summarized in Appendix 4 (www.cmaj.ca/lookup/suppl/doi:10.1503/cmaj.110400/-/DC1). They were causally and contextually extremely varied and fell into notable groups. Some were clearly due to avoidable adverse events (e.g., case 36). Others were less clear cut (e.g., case 19), occurred after a failed trial of conservative therapy (e.g., case 6) or were due to an early discharge (e.g., case 91).
The contributing factor most commonly cited by the reviewers was management error (n = 50 [48.1% of avoidable readmissions]) (Appendix 4). The next most commonly cited factors were surgical complications (n = 40 [38.5%]) and medication-related events (n = 34 [32.7%]). Nosocomial infection was identified in 19 avoidable readmissions (18.3%), system error in 16 (15.4%), diagnostic error in 11 (10.6%), procedure-related event in 7 (6.7%) and “other” in 9 (8.6%).
Among the 35 physician reviewers, the average sensitivity was 53.1% (range 4.5%–90.5%) and the average specificity was 89.0% (range 68.6%–100%), respectively (see the section “Operating characteristics of reviewing physicians” in Appendix 3). Reviewers identified additional information that they felt would have helped them judge avoidability in 774 (20.3%) of 3818 reviews. Readmissions for which at least one reviewer requested additional information (n = 321) were significantly more likely to be classified as a potentially avoidable readmission than those for which additional information was not requested (n = 326) (19.9% v. 12.3%, χ2 = 7.0, 1 degree freedom, p = 0.01).
Secondary analyses
Readmissions that occurred soon after the index hospital admission were significantly more likely than later readmissions to be classified as avoidable. The relative odds that readmissions were classified as avoidable decreased by 32% with each additional month following discharge (odds ratio 0.68, 95% CI 0.58–0.81). Also, 21.9% of readmissions within 4 days after discharge were classified as avoidable, as compared with 6.2% of readmissions beyond 135 days after discharge (Figure 2). Almost half of the urgent readmissions (n = 317, 48.8%) occurred within a month after discharge, 70 of which (22.1%, 95% CI 17.6%–27.1%) were potentially avoidable.
The proportion of patients who had an urgent readmission varied significantly by hospital (Table 2) (χ2 = 92.9, 10 degrees of freedom, p < 0.001). In contrast, the proportion of readmissions deemed avoidable did not show significant variation (χ2 = 12.5, 10 degrees of freedom, p < 0.25). The proportion of patients who had an urgent readmission was not associated with the proportion of patients who had an avoidable readmission (Pearson correlation 0.294; p = 0.38). Finally, when we ranked the hospitals according to the proportion of patients with urgent or avoidable readmissions, the rankings were not significantly correlated with each other (Spearman coefficient 0.28; p = 0.41).
Table 2:
Proportion of 4812 patients discharged from hospital who had an urgent readmission and proportion of 649 patients whose urgent readmission was deemed potentially avoidable, by hospital
| Hospital | No. of patients | Urgent readmission, no. (%) of patients [rank] | Potentially avoidable readmission, no. (%)of patients [rank] |
|---|---|---|---|
| A | 516 | 102 (19.8) [10] | 12 (2.3) [7] |
| B | 453 | 102 (22.5) [11] | 13 (2.9) [10] |
| C | 491 | 46 (9.4) [5] | 18 (3.7) [11] |
| D | 378 | 32 (8.5) [2] | 7 (1.8) [6] |
| E | 476 | 43 (9.0) [3] | 7 (1.5) [3] |
| F | 371 | 28 (7.5) [1] | 6 (1.6) [4] |
| G | 388 | 58 (15.0) [8] | 7 (1.8) [5] |
| H | 403 | 58 (14.4) [6] | 11 (2.7) [9] |
| I | 482 | 75 (15.6) [9] | 6 (1.2) [1] |
| J | 475 | 69 (14.5) [7] | 12 (2.5) [8] |
| K | 379 | 35 (9.2) [4] | 5 (1.3) [2] |
Interpretation
Our multicentred, prospective cohort study showed that potentially avoidable readmissions after discharge from hospital were uncommon. Less than one-fifth of all urgent readmissions were deemed to be potentially avoidable, and hospital-specific proportions of patients who had an urgent readmission was not associated with the proportion of patients who had an avoidable readmission.
Strengths and limitations
Our study has several strengths. It had a large and representative cohort. The inclusion of all — rather than disease-specific — urgent readmissions avoided potential biases associated with assigning diagnoses. Multiple practising physicians reviewed the urgent readmissions using standard implicit review methods. We used latent class analysis to collate judgments into the probability a readmission was avoidable.
Our study has several limitations. First, it identified readmissions that were potentially, and not definitely, avoidable. The physician review determined whether the readmission represented an adverse event and whether that adverse event was avoidable. In such a case, the readmission itself may have been unavoidable. Future studies should explicitly determine whether the readmission itself was avoidable if the preventable adverse event had not occurred.
Second, we did not include a random sample of index hospital admissions and are uncertain what influence the exclusion of patients discharged to nursing homes and those unable to communicate with research personnel would have had on study results. However, our inclusion criteria were very broad, and we successfully recruited and followed a large proportion of patients who were discharged from hospital.
Third, we did not include a random sample of physician reviewers. Although this might have helped to ensure that the reviews were truly representative, our study included 35 practising physicians, a number that greatly exceeds the maximum number of 3 reviewers typically used, as noted in our systematic review of avoidable readmissions.5
Fourth, although this was a multicentred study that included both teaching and community hospitals, we cannot be certain how representative they are of other hospitals.
Fifth, our patient population was younger and healthier than the elderly patients included in studies that solely used Medicare data.21
Sixth, our broad inclusion criteria precluded diagnostic-specific analyses.
Finally, and most importantly, reviewers requested additional information for their judgment in 20% of the reviews, and this often resulted in a readmission being identified as potentially avoidable. Future studies in this area should focus on acquiring data collection that is as complete as possible. Getting input from clinicians (those involved in the index hospital admission and the readmission) and patients — as soon after the readmission as possible — will be essential to having the information required for accurate and reliable case reviews.
Conclusion
Urgent readmissions deemed potentially avoidable were relatively uncommon, comprising less than 20% of all urgent readmissions following hospital discharge. Hospital-specific proportions of patients who were readmitted were not related to proportions of those whose readmissions were deemed avoidable.
Given the variety of causes and circumstances of the potentially avoidable readmissions in our study (Appendix 4), interventions to decrease the risk of readmission need to be multifactorial in nature and malleable to be appropriately tailored for each situation.
Our study has important implications for research into the quality of hospital care. First, determining whether urgent readmissions were avoidable is a subjective judgment that requires detailed patient data, multiple reviewers and an analysis that accounts for differing reviewer accuracy when collating judgments. Such judgments cannot be determined accurately on the basis of administrative data alone, given the infinite combinations of patient, hospital, treatment and post-discharge factors that can influence urgent readmissions. Second, we found no association between hospital-specific proportions of all-cause and avoidable readmissions. Therefore, urgent readmissions should be used with caution to gauge the quality of hospital care.
Supplementary Material
Footnotes
Competing interests: None declared.
This article has been peer reviewed.
Contributors: Carl van Walraven and Alan Forster contributed to the conception and design of the study and the analysis and interpretation of the data. All of the authors contributed to the acquisition of data. Carl van Walraven drafted the article. All of the authors revised the manuscript critically for important intellectual content and approved the final version submitted for publication. Carl van Walraven had full access to all of the data in the study; he takes responsibility for the integrity of the data and the accuracy of the data analysis.
Funding: This study was funded by the Canadian Institutes of Health Research (grant nos. MOP68972 and FRN79977), the Physicians’ Services Incorporated (PSI) Foundation, and the Department of Medicine, University of Ottawa. The funding bodies did not play a role in the preparation of the article.
References
- 1.Government of Ontario 2010 Annual report — discharge of hospital patients. Toronto (ON): Office of the Auditor General of Ontario; 2010. p. 64–93 [Google Scholar]
- 2.Hospital compare. Baltimore (MD): Centers for Medicare & Medicaid Services; 2010 [Google Scholar]
- 3.Canadian Institutes of Health Information Health indicators 2010. Ottawa (ON): The Institutes; 2010 [Google Scholar]
- 4.Hospital outcome of care measures. Washington (DC): US Department of Health & Human Services; 2009 [Google Scholar]
- 5.van Walraven C, Bennett C, Jennings A, et al. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ 2011;183:E391–402 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.MacDowell NM, Hunter SA, Ludke RL. Readmissions to a Veterans Administration Medical Center. J Qual Assur 1985;7:20–3 [Google Scholar]
- 7.Halfon P, Eggli Y, Pretre-Rohrbach I, et al. Validation of the potentially avoidable hospital readmission rate as a routine indicator of the quality of hospital care. Med Care 2006;44:972–81 [DOI] [PubMed] [Google Scholar]
- 8.Baker GR, Norton PG, Flintoft V, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 2004;170:1678–86 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Brennan TA, Localio RJ, Laird NL. Reliability and validity of judgments concerning adverse events suffered by hospitalized patients. Med Care 1989;27:1148–58 [DOI] [PubMed] [Google Scholar]
- 10.Brennan TA, Leape LL, Laird NM, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991;324:370–6 [DOI] [PubMed] [Google Scholar]
- 11.Thomas EJ, Studdert DM, Burstin HR, et al. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000;38:261–71 [DOI] [PubMed] [Google Scholar]
- 12.Localio AR, Weaver SL, Landis JR, et al. Identifying adverse events caused by medical care: degree of physician agreement in a retrospective chart review. Ann Intern Med 1996;125:457–64 [DOI] [PubMed] [Google Scholar]
- 13.Walter SD, Irwig LM. Estimation of test error rates, disease prevalence and relative risk from misclassified data: a review. J Clin Epidemiol 1988;41:923–37 [DOI] [PubMed] [Google Scholar]
- 14.McCutcheon AL. Introduction: the logic of latent variables. In: Latent class analysis. Newbury Park (CA): SAGE publications, 1987. p. 5–10 [Google Scholar]
- 15.Formann AK, Kohlmann T. Latent class analysis in medical research. Stat Methods Med Res 1996;5:179–211 [DOI] [PubMed] [Google Scholar]
- 16.Collins LM, Lanza ST. Latent class and latent transitition analysis: with applications in the cosial, behavioral, and health sciences. Hoboken (NJ): John Wiley & Sons Inc.; 2010 [Google Scholar]
- 17.Royston P, Altman DG. Regression using fractional polynomials of continuous covariates: parsimonious parametric modeling. Appl Stat 1994;43:429–67 [Google Scholar]
- 18.Sauerbrei W, Royston P. Building multivariable prognostic and diagnostic models: transformation of the predictors by using fractional polynomials. J R Stat Soc [Ser A] 1999;162:71–94 [Google Scholar]
- 19.Sauerbrei W, Meier-Hirmer C, Benner A, et al. Multivariable regression model building by using fractional polynomials: description of SAS, STATA and R programs. Comput Stat Data Anal 2006;50:3464–85 [Google Scholar]
- 20.Charlson ME, Pompei P, Ales KL, et al. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373–83 [DOI] [PubMed] [Google Scholar]
- 21.Medicare Payment Advisory Commission Report to the congress: promoting greater efficiency in Medicare. Butler C, editor. Washington (DC): The Commission; 2007. p. 107–8 [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.