Nonlinear association of nurse staffing and readmissions uncovered in machine learning analysis

Abstract

Objective

Several studies of nurse staffing and patient outcomes found a curvilinear or U‐shaped relationship, with benefits from additional nurse staffing diminishing or reversing at high staffing levels. This study examined potential diminishing returns to nurse staffing and the existence of a “tipping point” or the level of staffing after which higher nurse staffing no longer improves and may worsen readmissions.

Data Sources/Study Setting

The Readiness Evaluation And Discharge Interventions (READI) study database of over 130,000 adult (18+) inpatient discharges from 62 medical, surgical, and medical‐surgical (noncritical care) units from 31 United States (US) hospitals during October 2014–March 2017.

Study Design

Observational cross‐sectional study using a fully nonparametric random forest machine learning method. Primary exposure was nurse hours per patient day (HPPD) broken down by registered nurses (nonovertime and overtime) and nonlicensed nursing personnel. The outcome was 30‐day all‐cause same‐hospital readmission. Partial dependence plots were used to visualize the pattern of predicted patient readmission risk along a range of unit staffing levels, holding all other patient characteristics and hospital and unit structural variables constant.

Data Collection/Extraction methods

Secondary analysis of the READI data. Missing values were imputed using the missing forest algorithm in R.

Principal Findings

Partial dependence plots were U‐shaped, showing the readmission risk first declining and then rising with additional nurse staffing. The tipping points were at 6.95 and 0.21 HPPD for registered nurse staffing (nonovertime and overtime, respectively) and 2.91 HPPD of nonlicensed nursing personnel.

Conclusions

The U‐shaped association was consistent with diminishing returns to nurse staffing suggesting that incremental gains in readmission reduction from additional nurse staffing taper off and could reverse at high staffing levels. If confirmed in future causal analyses across multiple outcomes, accompanying investments in infrastructure and human resources may be needed to maximize nursing performance outcomes at higher levels of nurse staffing.

What is known on this topic?

• Several studies have reported positive associations between hospital nurse staffing and patient outcomes that taper off or reverse at high levels of nurse staffing.

• Few explanations for these findings and no theoretical framework have yet been offered.

• No studies have examined the association of hospital nurse staffing with hospital readmissions.

What this study adds?

• The study demonstrates a novel application of machine learning methods for uncovering a nonlinear association, where traditional parametric regression may not be as effective.

• The study finds tapering off and reversal of the association of nurse staffing with readmissions.

• The study's findings are consistent with diminishing returns on readmissions from additional nurse staffing.

1. INTRODUCTION

When the United States (US) Center for Medicare and Medicaid Services implemented the Hospital Readmission Reduction Program (HRRP) in 2012, US hospitals intensified efforts to reduce readmissions. A small number of analyses have pointed to the potential benefits of monitoring and correcting low nurse staffing levels to reduce readmissions ¹ , ² , ³ and avoid HRRP penalties, ⁴ adding to a broadly accepted body of evidence that higher nurse staffing levels are associated with improved quality and safety outcomes (mortality, failure to rescue, and other patient outcomes ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² ) and patient satisfaction with hospital nursing care. ¹²

Yet, depending on the skill mix of nursing personnel and the existing level of nurse staffing in an organization, increasing the amount of nursing hours assigned to patient care may not always be uniformly beneficial. In a recent study from the United Kingdom, a uniform dose–response reduction in mortality on patient wards was associated with higher professional nurse staffing, but a U‐shaped relationship with nursing assistant staffing—at low nursing assistant staffing levels, mortality decreased with additional nursing assistant staffing, but at high staffing levels, additional nursing assistant staffing was associated with higher mortality. ⁷ This was not the first time a U‐shaped pattern was reported—nearly two decades ago, researchers reported higher rates of medication errors as registered nurse staffing increased past a certain level, ¹³ and in a subsequent multihospital study, a robust and consistent tapering‐off pattern of diminishing returns to increased nurse staffing was found across multiple outcome measures (mortality, hospital‐acquired infections, and pressure ulcers). ¹⁴ Two systematic reviews and meta‐analyses noted a “curvilinear” relationship with benefits of increased registered nurse staffing tapering off and even reversing at higher staffing levels in the US ⁹ and internationally. ¹⁵ While no such evidence has ever been reported for readmissions, a curvilinear pattern was found for nurse staffing and patient satisfaction with hospital nursing care and specifically with nurse communication, discharge information, and care transitions, in a large study of US hospitals. ¹² Most recently, Needleman and Shekelle ¹⁶ proposed that the tapering‐off effect may be a natural consequence of “diffusion of effort or responsibility” that may occur at high nurse staffing levels and conjectured that this phenomenon may occur across many types of patient outcomes.

No studies to date have examined the underlying empirical patterns behind the observed associations between nurse staffing and readmissions across a range of staffing levels. Therefore, the objective of this study was to examine the pattern of the association between nurse staffing and hospital readmissions, specifically examining the possibility that additional nurse staffing has diminishing returns, with the existence of a “tipping point” after which adding more nurse staffing no longer improves (and may worsen) readmissions. Evidence of a nonlinear association between nurse staffing and readmissions could inform future causal studies of the relationship between staffing and readmission, toward the ultimate goal of novel practice and policy recommendations for maximizing the value‐added contribution of nursing to patient and cost outcomes and hospital performance.

2. THEORETICAL FRAMEWORK

Most investigations of the relationship between nurse staffing and patient outcomes have been grounded in Donabedian's structure‐process‐outcomes model. ¹⁷ The model views patient outcomes as the result of many organizational structural factors catalyzed during the process of care delivery. Donabedian defines organizational structure as “the settings in which care is provided and the instrumentalities of which care delivery is a product.” Nurse staffing is one of many structural factors that jointly comprise organizational structure; others include the availability and qualifications of other clinical staff, adequacy of facilities and equipment, and the structure and operationalization of programs that support and direct the provision of care. ¹⁷ An extension ¹⁸ of Donabedian's model also identifies patient characteristics (specifically those directly contributing to the likelihood of an outcome, such as clinical risk factors and sociodemographic characteristics) as an additional input into the care delivery process.

In the field of economics, the theory of production ¹⁹ also views care delivery as a production process that utilizes health care resources as inputs to produce patient outcomes. Economic production theory categorizes structural inputs into two broad types: factors of production (“the who”) and technology (“the how”). Factors of production are the inputs that an organization employs to produce an output, and they are further categorized into labor and capital inputs. The labor input category concerns with the quantity (e.g., number of working hours and number of full‐time equivalents) and the quality (e.g., education, experience, expertise) of the human resources employed by the organization (nurses, physicians, etc.). The capital input category refers to the hospital's built environment and equipment, administrative and other support systems, type and organization of the electronic health records (EHRs), and other nonlabor resources that enable employees to deliver care. Technology encompasses all organizational rules and norms prescribing how the labor and capital inputs ought to interact during the care delivery process; it refers to the established care delivery components within the organization that form the organizational standard of care (such as protocols, programs, and practice guidelines). All structural factors (labor and capital inputs and technology) are part of a joint production process. As such, the incremental productivity of any single factor is determined by the quantity of the factor itself , the quantities of all other inputs, and the technology used in the production process. As such, according to the economic theory of production, the contribution of the nursing input to outcomes is not static, but rather it can be enhanced by increasing the availability of other types of labor, capital, and technology relevant to the delivery of nursing care.

Guided by these two theoretical frameworks, we conceptualized a patient's likelihood of a readmission as an outcome of the process of care delivery that uses an organization's labor and capital resources and the care delivery technology established in the organization. Doing so allowed us to view the relationship between readmissions and nurse staffing as a partial derivative, or partial dependence, of readmissions on nurse staffing specifically, while holding all other structural inputs (other types of labor, capital, and technology) constant as observed in the data.

Table 1 parallels the constructs and definitions of Donabedian's structure‐process‐outcome model, as expanded by Mitchell et al., ¹⁸ and the economic theory of production, ¹⁹ and shows the types of variables that we used as empirical measures for each construct.

TABLE 1.

Theoretical framework and empirical measures

Donabedian structure‐process‐outcome model a	Structural characteristics: The setting in which care is provided and the instrumentalities of which the process of care is the outcome.	Patient characteristics b : Patient‐level risk factors prior to care delivery that may guide how care is delivered and directly impact the outcome.	Process: The process of execution of clinical care delivery	Outcome: The outcome of medical care, in terms of recovery, restoration of function and of survival.
Economic theory of production c	Factors of production (labor and capital): inputs that are used to produce an output. Labor inputs: The quantity (e.g., number of working hours, number of full‐time equivalents) and quality (e.g., education, experience, expertise) of the human resources employed by the organization (nurses, physicians, etc.) Capital inputs: The hospital built environment and equipment, administrative and other support systems, type and organization of the electronic health records, and other non‐labor resources than enable its employees to deliver care. Technology: Established care delivery components that form the hospital or unit standard of care (such as protocols, programs, and practice guidelines) that prescribe how care is supposed to be delivered. c	[Not included]	Production function: The technological relation between quantities of inputs and the output	Output: The quantity of the product produced
Select variables available in the READI database d	Factors of production: Labor inputs: Unit‐specific: RN hours per patient day (HPPD); RN hours per patient day (HPPD) Non‐RN HPPD (nurse aides, licensed practical nurses, medical assistants); %BSN; % Specialty‐certified RNs; RN Experience; % Full/Part‐Time. Hospital‐specific: RN full‐time equivalents (FTEs); Non‐RN FTEs e Capital inputs: Unit‐specific: Bed size; Patient days; Average daily census; case mix index; Unit type (e.g., cardiac, medical, surgical). Hospital‐specific: Geographic region (East, West, Midwest, South); Bed size; Number of med., surg., or med‐surg units; Hospital Type (community/urban; non‐teaching/teaching); Type of EHR Technology: Unit‐specific: READI discharge intervention; RN case manager/social worker/discharge planner/coordinator/expeditor involved in discharge transition care; Prescriptions filled prior to discharge. Hospital‐specific: Discharge Care Model f ; Formal readmission risk screening imbedded in the EHR; Interdisciplinary care coordination rounds; Guidelines include interdisciplinary discharge rounds.	Demographic: Sex; Age; Race Ethnicity; Marital Status. Clinical: Admission type; Admission from (source); Service Type (medical/surgical); Payment type; Severity of Illness; Mortality risk; prior hospitalization (30 and 90 days); Elixhauser Comobidity Index; Length of Stay; ICU Stay	[Unobserved, not measured in this study]	Patient readmission within 30 days

^a Donabedian. ¹⁷

^b Mitchell et al. ¹⁸

^c Source: Adapted from Craig and Harris. ¹⁹

^d During the random forest recursive model selection process, some of these variables were eliminated from the final model. See Digital Supplement Figure A for the full list of variables included in the final model.

^e All hospital employees (physicians, pharmacists, therapists, administrative, laboratory, house‐keeping, etc.) on hospital payroll, per the American Hospital Association Annual Survey definition. ²⁰

^f Established programs of discharge care based on local, state, or national discharge transition improvement initiatives. ²¹

3. METHODS

3.1. Approach

To examine the relationship between nurse staffing and readmissions, this secondary analysis applied a machine learning (ML) approach ²² to a large multihospital dataset of adult inpatient discharges. ML methods “learn” data patterns from the data itself, without imposing restrictions on the functional form of the relationship among variables. Therefore, ML methods are capable of exposing the true data pattern behind the association of nurse staffing and readmissions, be it linear, curvilinear, a U‐shape, or virtually any other potentially unknown pattern.

A unique feature of ML is that, being purely atheoretical and nonparametric, the association between the dependent variable (readmission) and an explanatory variable (e.g., staffing) is revealed as a partial dependence plot (PDP). ²² , ²³ A PDP shows how the predicted probability of readmission varies along the observed range of nurse staffing in the sample, while holding all other variables constant as observed. In constructing a PDP, neither the direction nor the magnitude of the association between readmissions and staffing (traditionally measured by a regression coefficient) is derived as a parameter; instead, the local marginal effect (the derivative of readmission likelihood with respect to staffing) is represented by the slope of the PDP at each staffing level and can vary in sign and magnitude along the range of observed staffing values.

Like all observational (non‐experimental) methods, ML methods are vulnerable to unobserved and unmeasured variables and potential reverse causality. For example, a hospital's decisions on level of or budgeted nurse staffing are likely linked to the patient case mix (hospitals treating more complex patients tend to have higher nurse staffing levels), which could create a positive pattern of association between staffing and readmissions in the absence of a causal pathway. In an attempt to partially mitigate endogeneity, we used a unique proprietary dataset collected by a team of economists and clinicians to examine the impact of nursing care on readmissions. As described in the Section 3.2, the database includes all relevant structural input variables available in the EHR, a comprehensive set of technology variables capturing organizational practices relating to discharge, and an extensive set of patient characteristics. Nevertheless, our results should not be interpreted as causal, which is an overt limitation of our approach.

3.2. Data

The Readiness Evaluation And Discharge Interventions (READI) study [NCT01873118; 20] was a multihospital cluster‐randomized clinical trial, approved by [blinded for review] Institutional Review Board to test the impact of unit‐based implementation of a discharge readiness assessment intervention on readmission and emergency department use. The study team recruited hospitals through a call for interest to Magnet‐designated organizations coordinated by the American Nurse Credentialing Center of the American Nurses Association. The sample included nearly 145,000 adult (18+) inpatients discharged to home from 66 general medical, surgical, or medical‐surgical units. Thirty‐one US and two Saudi Arabia Magnet hospitals participated in the study between October 2014 and March 2017. Each hospital contributed 2 units that were randomly assigned to intervention or usual care control conditions. Data collection followed Donabedian's structure‐process‐outcomes model ¹⁷ and the economic theory of production. ¹⁹

The intervention involved augmenting the existing discharge standard‐of‐care protocol with a formalized discharge readiness assessment to inform patient preparation for discharge. The study was conducted in four phases over 19 months including a baseline phase followed by three intervention phases, each testing a different version of the discharge readiness assessment protocol. The intervention had high fidelity with more than 90% of nurses trained and more than 70% of patients assessed per study protocol on intervention units. Despite some evidence of potential effectiveness in units with high baseline readmission rates, the intervention was overall not effective in changing readmissions. We treated the READI intervention as a unit‐specific discharge process variable in the study along with other variables in the “technology” category.

3.3. Sample

We used 137,778 adult (18+) inpatients discharged from 62 units in 31 US hospitals in the READI study. Following the scientific standards for ML methods, ²² we split the sample into a 70% learning sample of 96,444 observations and a 30% testing sample of 41,334 observations.

3.4. Study variables and measures

The outcome was 30‐day all‐cause same‐hospital readmission, measured as a dichotomous variable (1 = patient had at least one readmission within 30 days post discharge; 0 = no readmissions). The primary exposure was unit nurse staffing, measured in hours per patient day (HPPD) with three continuous range variables: direct‐care registered nurse nonovertime (RN non‐OT HPPD), RN overtime (RN‐OT HPPD), and unlicensed nursing personnel (non‐RN HPPD). Nurse staffing was reported monthly by each study unit for a total of 1178 unit‐month observations (62 units × 19 months) and linked to patients by the discharge month. We included three READI intervention features: a dichotomous variable for the study units' assignments (1, if randomized to intervention condition and 0, if randomized to usual standard of care control condition), a dichotomous variable for the study patients' discharge protocol (1, if discharged per study protocol on an intervention unit and 0, if discharged not per study protocol or if discharged from a control unit), and a categorical variable for the study month at discharge (1–19). Structural input variables, technology, and patient characteristics are included in Table 1.

3.5. Statistical analyses

3.5.1. Model selection and performance

Using the learning sample and starting with the full set of structural inputs and patient variables available in the READI dataset, we performed a random forest recursive feature elimination process using the caret package V6.0‐84 ²⁴ with 10‐fold cross‐validation, 500 trees, and a minimum node size of six observations. We chose a random forest model as the best tool for learning about the relationships between the outcome and individual variables, given its robust classification power and easily interpretable learning mechanism. ²⁵ , ²⁶ , ²⁷ A random forest is a set of decision trees that each use explanatory variables as logical “if/then” splits leading down the paths, or “branches,” ending either in a predicted readmission or in no predicted readmission. Randomly reshuffling the values of each of the explanatory variables one at a time while holding all other variables constant at the values observed for each patient in the sample, the algorithm sorts through a large number of different combinations of the variables and selects a model with the highest predictive performance. We measured predictive performance by following two statistics: the random‐chance‐adjusted proportion of correctly predicted outcomes, or Cohen's Kappa, and the area under the curve (AUC). Cohen's Kappa (κ) varies between 0 and 1 and has a similar interpretation to the traditional R‐squared of the regression. ²⁸ We also evaluated the importance of the three nurse staffing variables relative to other explanatory variables in our model. We measured a variable's importance by the size of reduction in the model's Cohen's Kappa (Δκ) when the variable's values were reshuffled at random, with larger Δκs indicating more important variables. For our final model, we reported the Δκ and AUC in both the learning and the testing samples. All other results were presented for the testing sample only (results in the learning sample were similar, see the sensitivity analysis in section 4.3 below).

3.5.2. Sample descriptive characteristics

Once the variables selected into the model were determined, we calculated sample descriptive statistics using counts and sample proportions for all categorical variables, means and interquartile ranges for continuous unit‐level and hospital‐level variables, and means and SDs for continuous patient‐level variables.

3.5.3. Partial dependence plots

We built partial dependence plots (PDPs) for the three nurse staffing variables as predictors of readmission in the learning sample, using a 100‐point evenly spaced grid from the minimum value through the observed range of each variable. We then calculated 95% confidence bounds using the standard errors of the point predictions obtained after 100 bootstrap replications for each point.

3.5.4. Missing data

There were no missing values in the outcome variable. The nurse staffing variables were not reported by two units during the first study month, for a total of 236 patient observations (0.17%). Among the unit and hospital variables, missing data included: unit case mix index (12,508, 9.08% of patient observations), unit RN experience (10,180, 7.39% of observations), and unit nurse certification (6848, 4.97% of patient observations). These missing values were imputed using the missing forest algorithm in the R statistical computing environment. ²⁴ , ²⁵ , ²⁶ , ²⁷ Among the patient characteristics, several categorical variables had values coded as “unknown” in the hospitals' EHR (see Table 2); the unknown category was preserved in the analyses. No continuous patient variables had missing values.

TABLE 2.

Predictive accuracy of the model in the learning sample (n = 96,444) and the test sample (n = 41,334)

	Learning sample	Test sample
Kappa	0.23	0.13
Specificity	1.00	0.99
Sensitivity	0.15	0.08
Positive predictive value	1.00	0.86
Negative predictive value	0.89	0.89
Area under receiver operating characteristics curve	0.99	0.71

4. RESULTS

4.1. Model selection and performance

From the full set of 141 variables in the READI data, ML selected a subset of 60, including the three nurse staffing variables ( Supplemental File: Figure A ). The predictive accuracy of the final model was κ = 0.24 and AUC = 0.99 in the learning sample and κ = 0.13 and AUC = 0.71 in the testing sample. (Table 2).

In variable importance analysis, patient characteristics as a group were most important in predicting readmissions (joint Δκ = 0.23). The three nurse staffing variables as a group (joint Δκ = 0.04) were more important for predicting readmissions than other unit‐specific structure variables but less important than hospital characteristics (joint Δκ = 0.14). Individually, the nurse staffing variables ranked 15th (RN OT HPPD), 17th (RN non‐OT HPPD), and 19th (non‐RN HPPD) among the 60 variables in the model, and first, second, and seventh among the 32 labor, capital, and technology variables. Among the three READI study design features, unit assignment (intervention and control) and patient treatment status (per protocol and usual care) contributed very minimally to readmissions. ( Supplemental File: Figure A ).

4.2. Sample descriptive characteristics

The readmission rate was 12.2%. Patient characteristics of the sample are presented in Table 3; there were no significant differences between the learning and testing samples. The 62 study units were staffed with 10.3 h of nursing care per day (i.e., HPPD) including 6.77 RN non‐OT HPPD, 0.20 RN OT HPPD, and 3.35 non‐RN HPPD. The proportion of Bachelor of Science in Nursing (BSN)‐prepared nurses was 68%, and the average nurse experience on the unit was 6 years. (Table 4).

TABLE 3.

Select ^a descriptive characteristics of patients in the learning sample (n = 96,444) and test sample (n = 41,334)

Characteristic	Learning sample	Test sample
	n (%) or Mean (SD)	n (%) or Mean (SD)
Primary outcome: 30‐day readmission
No readmission	84 585 (87.70%)	36 333 (87.90%)
Readmission	11 859 (12.30%)	5001 (12.10%)
Patient characteristics
Patient sex
Male	47 265 (49.01%)	20 168 (48.79%)
Female	49 179 (50.99%)	21 166 (51.21%)
Patient age	60.10 (17.38)	60.09 (17.36)
Patient race
American Indian or Alaska Native	858 (0.89%)	348 (0.84%)
Asian	3320 (3.44%)	1416 (3.43%)
Black or African American	14 717 (15.26%)	6288 (15.21%)
Native Hawaiian or other Pacific Islander	305 (0.32%)	146 (0.35%)
White	65 847 (68.27%)	28 318 (68.51%)
Unknown	11 397 (11.82%)	4818 (11.66%)
Patient ethnicity
Not Hispanic	84 517 (87.63%)	36 129 (87.41%)
Hispanic	10 407 (10.79%)	4554 (11.02%)
Unknown	1520 (1.58%)	651 (1.57%)
Patient marital status
Not married	41 988 (43.54%)	18 197 (44.02%)
Married	43 968 (45.59%)	18 566 (44.92%)
Unknown	10 488 (10.87%)	4571 (11.06%)
Admission type
Emergency	52 613 (54.55%)	22 582 (54.63%)
Urgent	17 626 (18.28%)	7536 (18.23%)
Elective	17 451 (18.09%)	7543 (18.25%)
Unknown	8754 (9.08%)	3673 (8.89%)
Admission source
Physician referral	36 540 (37.89%)	15 822 (38.28%)
Clinic referral	6247 (6.48%)	2662 (6.44%)
Managed care referral	185 (0.19%)	79 (0.19%)
Transfer from hospital	3022 (3.13%)	1271 (3.07%)
Transfer from another healthcare facility	2199 (2.28%)	966 (2.34%)
Emergency department	19 605 (20.33%)	8356 (20.22%)
Unknown	28 646 (29.70%)	12 177 (29.46%)
Service type
Medical	69 385 (71.94%)	29 770 (72.02%)
Surgical	25 748 (26.70%)	11 025 (26.67%)
Unknown	1311 (1.36%)	539 (1.30%)
Payment type
Private insurance	29 763 (30.86%)	12 730 (30.80%)
Medicare	40 820 (42.33%)	17 485 (42.30%)
Medicaid	13 993 (14.51%)	6012 (14.54%)
Uninsured/unknown	11 868 (12.31%)	5107 (12.36%)
Severity of illness score
Minor	11 564 (11.99%)	5033 (12.18%)
Moderate	27 244 (28.25%)	11 657 (28.20%)
Major	25 287 (26.22%)	10 878 (26.32%)
Extreme	4883 (5.06%)	2052 (4.96%)
Unknown	27 466 (28.48%)	11 714 (28.34%)
Mortality risk score
Minor	23 559 (24.43%)	10 063 (24.35%)
Moderate	19 894 (20.63%)	8672 (20.98%)
Major	14 841 (15.39%)	6323 (15.30%)
Extreme	3441 (3.57%)	1464 (3.54%)
Unknown	27 466 (28.48%)	11 714 (28.34%)
Prior hospitalization within 30 days	11 907 (12.35%)	5009 (12.12%)
Elixhauser comorbidity index	7.04 (8.39)	7.08 (8.36)
Total length of stay, days	4.22 (4.22)	4.24 (4.26)
ICU stay	17 376 (18.02%)	7355 (17.79%)
Discharge disposition
Discharged to home/self‐care	77 120 (79.96%)	33 029 (79.91%)
Discharged to home/home health service	16 851 (17.47%)	7197 (17.41%)
Discharged to hospice care	1184 (1.23%)	535 (1.29%)
Left against medical advice	1289 (1.34%)	573 (1.39%)
Comorbidities
Renal failure	7691 (7.97%)	3344 (8.09%)
Weight loss	3171 (3.29%)	1310 (3.17%)
Hypertension	21 861 (22.67%)	9464 (22.90%)
Metastatic tumor	2406 (2.49%)	1035 (2.50%)
Electrolyte disorders	15 786 (16.37%)	6812 (16.48%)
Congestive heart failure	4796 (4.97%)	2146 (5.19%)
READI study status b
Patient from an intervention unit	49 195 (51.01%)	21 224 (51.35%)
Patient treated per protocol c	25 365 (51.56%)	10 934 (51.52%)

^a See Supplement Table A for the full set of descriptive statistics.

^b Study month not shown, see Supplement Table A.

^c Patients from intervention units only; the percentage includes the baseline period, see Supplement Table B.

TABLE 4.

Select ^a descriptive structural input characteristics of units (n = 62) and hospitals (n = 31)

Input variable	n (%) or Mean (Interquartile Range)
Labor inputs
Unit‐specific nursing labor inputs
Quantity: nursing hours per patient day a
RN nonovertime	6.77 (5.98, 7.42)
RN overtime	0.20 (0.13, 0.25)
Non‐RN total HPPD	3.35 (2.47, 3.68)
Quality: education, expertise, experience, flexibility:
% BSN‐prepared RNs	67.65 (55.66, 80.78)
% Certified RNs	29.35 (10.98, 40.98)
Average RN experience (on the unit)	6.18 (4.50, 8.00)
% Full‐time RNs b	78.28 (71.25, 93.18)
Hospital‐wide labor inputs:
RN full‐time equivalents	1101.84 (486.91, 1271.35)
Non‐RN full‐time equivalents (all combined) c	2774.88 (570.91, 3259.22)
Capital inputs
Hospital facility and resources
Hospital bed size	540.87 (315.00, 607.50)
Hospital type:
Rural, nonteaching	1 (3.23%)
Community, nonteaching	11 (35.48%)
Community, teaching	7 (22.58%)
Urban, nonteaching	4 (12.90%)
Urban, teaching	3 (9.68%)
Academic medical center	5 (16.13%)
Unit‐specific facility and resources
Unit type:
Cardiac care	11 (17.74%)
Medical	8 (12.90%)
Surgical	2 (3.23%)
Medical/surgical	17 (27.42%)
Medical with telemetry	17 (27.42%)
Surgical with telemetry	3 (4.84%)
Neurology/neurosurgery	2 (3.23%)
Orthopedics	1 (1.61%)
Respiratory	1 (1.61%)
Average daily patient census b	26.49 (22.19, 28.70)
Patient case mix index b	1.76 (1.42, 1.80)
Technology for readmission avoidance
Discharge care models d
Transitional care model	1 (3.23%)
Care transitions model	6 (19.35%)
Re‐engineered discharge	2 (6.45%)
Institute for Healthcare Improvement/State Action on Avoidable Readmissions	2 (6.45%)
Centers for Medicare/Medicaid Services	3 (9.68%)
State hospital association initiative	3 (9.68%)
Local/regional collaborative initiative	4 (12.90%)
None of the above/other	18 (58.06%)

Abbreviations: BSN, Bachelor of Science in Nursing; HPPD, hours per patient day; RN, registered nurse.

^a See Supplement Figure A and Table A for the full set of structural input variables.

^b Measured monthly (19 data points); otherwise measured annually (two data points).

^c All hospital employees (physicians, pharmacists, therapists, administrative, laboratory, house‐keeping, etc.) on hospital payroll, per the American Hospital Association Annual Survey definition. ²⁰

^d Categories are not mutually exclusive.

4.3. Partial dependence plots

The PDPs for nurse staffing variables (RN non‐OT HPPD, RN OT HPPD, and non‐RN HPPD) revealed a common quasi‐parabolic data pattern with readmission likelihood first falling as staffing increased from the lowest staffing levels, reaching a minimum point, and then increasing as staffing levels increased. (Figure 1). The tipping points were observed at 6.95 HPPD of RN non‐OT staffing, 0.21 HPPD of RN OT staffing, and 2.91 HPPD non‐RN staffing.

FIGURE 1.

Partial dependence plots of the relationship between nurse staffing and readmissions, the learning sample (n = 96 444). Note: For each of the staffing variables, the figure shows: (1) the PDP for the relationship between the corresponding staffing hours variable and the predicted probability of readmission (PDP, solid black curve) and the 95% confidence interval (pink sleeve); (2) the relative frequency distribution of the patient sample by nursing hours per patient day in 25 increments (gray‐shaded bars); and (3) the first and third quartile of the unit‐level distribution of nursing hours per patient day (dashed vertical lines) [Color figure can be viewed at wileyonlinelibrary.com]

In sensitivity analyses, the quasi‐parabolic patterns were also evident: (1) in the testing data sample, supporting the robustness of the model in out‐of‐sample performance ( Supplemental File: Figure B ); (2) using restricted models that eliminated various features, reducing concerns about overfitting ( Supplemental File: Figure C ); (3) in a subsample of patients discharged from control units, eliminating potential confounding from the intervention ( Supplemental File: Figure D ); and (4) after casewise deletion, instead of imputation of missing data prior to estimation ( Supplemental File: Figure E ).

5. DISCUSSION

In our sample of medical, surgical, and medical‐surgical units in 31 US Magnet hospitals, we found that hospital readmissions are related to nurse staffing via a robust U‐shaped data pattern consistent with a tipping point past which additional staffing was associated with increasing readmission rates. Adding to several prior studies reporting diminishing returns to nurse staffing, ⁷ , ⁹ , ¹² , ¹³ , ¹⁴ , ¹⁵ our findings suggest that when it comes to optimizing delivery of care for maximum outcomes, a “more is better” approach may not be a universally applicable principle to guide organizations in workforce planning.

From the lens of the economic theory of production, the observed U‐shaped relationship between staffing and readmissions can be explained by the law of diminishing returns. ¹⁹ , ²⁹ It states that the production of additional output will decrease as more of a single factor of production (e.g., nursing labor) is incrementally added while the amounts of all other factors (other types of labor and capital) and technology stay the same. The law of diminishing returns is a direct derivative from the economic theory of production because the contribution of any one input, such as labor, depends on the amounts of the other inputs and technology; adding more labor alone, without simultaneously increasing the levels of the other inputs or improving technology, will eventually diminish the productivity (or returns) of labor. For example, as more autoworkers are hired by an automobile manufacturer, without also modifying the production floor for safety, installing additional equipment, and hiring more training and management personnel, the productivity of the expanding workforce will eventually decline.

Applied to nursing, the law of diminishing returns predicts that increasing a nursing unit's staff will lead to initial performance gains. Eventually, however, economic theory predicts that, without additional investments in all relevant structural factors that enable the delivery of high‐quality care by nurses, adding more nursing hours alone will, first, reduce, then eliminate, and may eventually even reverse the initial productivity gains. One explanation could be what Needleman and Shekelle (2019) called “the diffusion of effort and responsibility” ¹⁶ —simply increasing the number of direct care nurses may lead to missed, duplicative, or low‐value nursing care that fails to improve and may even worsen outcomes. ³⁰ , ³¹ Specific to readmissions, diffusion of effort and responsibility may manifest itself as poor nurse–patient communication during discharge planning, coordination, and teaching, ¹² thus inhibiting a successful patient transition from hospital to home and increasing readmissions. ³² This conceptual framework and our results align with the previous studies showing tapering off and a U‐shaped relationship between nurse staffing and adverse patient outcomes. ⁷ , ¹² , ¹³ , ¹⁴ , ¹⁵

The downward‐trending part of the U‐shape is consistent with a large body of staffing literature reporting lower rates of readmission and other adverse outcomes with increasing RN staffing. ¹ , ² , ³ , ⁴ , ⁵ , ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ Higher nurse staffing and lower patient–nurse ratios can afford more direct nursing care time for assessment of patients' readiness for discharge, discharge teaching, and discharge coordination, thus reducing risk of readmission. ³² Higher nurse staffing can also contribute to reducing readmissions indirectly, by affording more time for professional development and unit‐based nursing governance, which can improve nurse job satisfaction and reduce burnout and turnover. ³³ , ³⁴ For units with nurse staffing levels below their tipping point, retention and recruitment of a larger unit nurse workforce can be foundational to reducing readmissions.

It is important, however, not to wrongfully infer that the observed tipping point is the 'optimal' staffing point for benchmarking unit staffing levels; nor does our study imply that relatively well‐staffed units, those on the upward sloping part of the U‐shaped pattern, should be cutting back on nurse staffing. Firstly, our observational study design limits causal interpretation of our findings. More importantly, though, is to understand that, even if our findings truly reflect a causal diminishing returns relationship, the tipping point would occur, theoretically, only when the organization fails to deploy other structural resources to support a larger nursing staff, thus effectively resulting in an inefficient substitution of nurse staffing for other hospital inputs (commonly known as “plugging a hole”). Therefore, to derive continuing productivity gains from nurse staffing, well‐staffed organizations should be not reducing nursing staff, but instead increasing investments in other structural variables (factors of production and technology) that enable nurses to deliver high‐quality discharge care. Depending on the context of each specific organization, these may include an increased supply of labor (e.g., shift managers or discharge coordinators, planners, flow coordinators, expeditors), capital (e.g., an EHR system with capacity for real‐time aggregation of discharge‐relevant information from multiple entry points), or technology (e.g., implementation of new organizational processes to improve communication and care coordination, such as interdisciplinary team discharge rounds). Currently, little is known about various organizational approaches to preventing and reversing diminishing returns; this area of future research presents an exciting new opportunity for informing continued outcomes improvement in already well‐performing organizations.

Our study was one of the first to examine the importance of nurse staffing with other structural and patient variables interacting in a complex nonparametric model. Not surprisingly, patient characteristics were the most important predictors of patient readmission risk in our study. However, nurse HPPD had the highest predictive association among most other structural factors, including nurse skill mix, education, experience and expertise, and hospital‐ and unit‐specific discharge care variables. While a patient's risk of readmission attributable to patient‐specific factors may be difficult to identify and modify, nurse staffing strategies are well within an organization's domain of influence and should continue to be a focus of organizational outcomes improvement efforts.

Our study was the first to apply ML methods to study nurse staffing and readmissions. To date, empirical studies of the relationship between nurse staffing and patient outcomes (including readmissions) have been performed using a parametric regression analysis approach, which requires the regression equation to be chosen by the researcher prior to estimation process. Parametric regression binds the data to a preconceived (by the researcher) notion of the relationship, then finds the coefficients, or parameter estimates, that best support the a priori chosen equation in a given set of data. Previous studies that reported diminishing returns to nurse staffing were designed a priori to look for particular nonlinear shapes: a piecewise linear regression (ie, a “V”) ¹³ and a higher‐order polynomial for staffing (ie, a “U”). ⁷ , ¹² , ¹⁴ Once chosen, the parametric methods in these studies were only able to confirm or reject the shape specified a priori by the researchers but unable to reveal any other data patterns. By not constraining our data to any functional form prior to estimation, our findings provide further credence to the idea that the previously reported U‐shapes were likely real and not an artifact of the previously selected parametrizations. Interestingly, when we attempted to replicate the previously used parametric approaches in our data ( Supplemental File: Figures F1‐F3 ), there was little agreement regarding the shape of the pattern between methods and types of nurse staffing. One explanation is the possibility that the underlying empirical pattern in our data was not consistent with either a cubic or a linear spline model, resulting in poor fit. The other possibility lies in the different ways that parametric regression and ML methods form predictions—while traditional regression models calculate predictions by setting all over covariates at their means, ML predictions are simulated while keeping other covariates at their observed values. ²³ The relative performance of ML methods versus parametric and semi‐parametric (e.g., quantile and fixed effects) regression should be examined in future studies.

Our study had a number of limitations. First, like all observational studies, our study is subject to confounding from unobserved variables and potential reverse causality. Although our study used one of the largest, richest sources of data collected specifically for analysis of readmissions in a structure‐process‐outcomes framework, and although the findings are robust across samples and in models with different sets of features in sensitivity analyses, our study design does not allow for causal interpretation. Second, although the READI intervention was modeled in our analysis as a unit‐specific technology variable (similarly to how we accounted for all other hospital and unit discharge practices) and although our findings are robust in a subsample of patients discharged from control units in sensitivity analyses, it is difficult to know for sure to what extent our results apply to an intervention‐independent sample. Third, although our sample has similar nurse staffing levels ³⁵ , ³⁶ and readmission rates ³⁷ to national studies of US hospitals (including Magnet hospitals ³⁶ ), in general, Magnet hospitals tend to be larger, have higher levels of nurse staffing, invest more in nurse staff development and education, and deliver higher‐quality care including discharge care, ³⁶ , ³⁸ , ³⁹ , ⁴⁰ further limiting the generalizability of our findings. Fourth, we estimated the overall association of staffing with readmissions across 31 hospitals. Even though the actual tipping points in HPPD are likely context and patient‐population specific, the ML method we demonstrated can be applied across contexts. Fifth, we only examined a single patient outcome—readmission; further research is needed to test whether these findings are equally true for other outcomes. Last, we studied three nurse staffing variables (RN non‐OT HPPD, RN OT HPPD, and non‐RN HPPD) separately, and the association of each with readmissions was obtained holding the other two constant. Future studies should examine nurse staffing variables as dynamic and interdependent to fully understand the optimal staffing strategy to produce desired patient outcomes.

6. CONCLUSION

While the idea of diminishing returns to labor is intuitive to most economists, it has not yet influenced health care policy makers or administrators who tend to subscribe to a linear “more is better” thinking. Our findings suggest that increasing nurse staffing alone may not always yield continuing improvements in readmissions; after a point, accompanying investments in infrastructure and other resources may be needed to support further nurse performance improvement and outcome gains. In complex health care delivery systems where relationships among staffing variables and patient outcomes are interdependent, subject to organization‐specific factors, and are not directly observed, ML methods may offer an advantage of exposing the tipping point and informing proactive organizational action to support continued high returns from nurse staffing to patient outcomes.

Supporting information

Appendix S1 Supporting information

Yakusheva O, Bang JT, Hughes RG, Bobay KL, Costa L, Weiss ME. Nonlinear association of nurse staffing and readmissions uncovered in machine learning analysis. Health Serv Res. 2022;57(2):311–321. 10.1111/1475-6773.13695