Assessment of the effectiveness of protein in critical illness: the role of statistical shortcomings in explaining discrepancies between observational studies and randomized controlled trials

Abstract

Background

Until 2023, recommendations for prescribing protein to critically ill patients were largely based on observational studies and non-randomized interventions. Several large, multicentre non-randomized studies of intervention conducted between 2009 and 2017 consistently reported that higher protein intake during the acute phase of critical illness was associated with lower mortality. However, major randomized controlled trials published in recent years (the EFFORT Protein trial in 2023, the PRECISe trial in 2024, and the TARGET Protein trial in 2025) have found no clinical benefit of increased protein administration. The reasons for these conflicting results are unclear, but they may be related to statistical shortcomings in observational studies.

Main body

The authors identified numerous types of statistical bias in observational nutrition research. These include violations of the time axis, confounding by indication, failure to account for a wash-in period, inappropriate methods for variable selection and modelling of numerical confounder variables, transfer bias, sample-size bias, competing-risk bias, bias by data not missing at random, immortal time bias, and collider bias. To improve reliability, modern observational studies should emulate the design of randomized controlled trials. Advanced statistical approaches, including competing-risk models, time-varying analyses, nonlinear modelling and lag-time adjustments, can better account for the complexity of nutrition therapy in critical illness.

Conclusions

Numerous types of bias may have exaggerated the apparent benefits of higher protein intake in older observational studies possibly explaining discrepancies to newer randomized trials. Rigorous statistical planning, transparent analysis protocols, use of advanced statistical approaches and expert statistical review are required to ensure that observational nutrition research produces valid, clinically meaningful evidence.

Graphical abstract

Introduction

Until 2023, the evidence base for recommendations regarding the prescription of protein for critically ill patients was almost entirely derived from large observational studies and non-randomized intervention studies. Between 2009 and 2017, five large, influential, uncontrolled studies were published, each analyzing more than 1,000 patients [1–5]. In [1–3, 5], the reference body weight for the protein prescription was either the actual body weight (BMI 18–30 kg/m²) or the ideal body weight (BMI > 30 kg/m²). Zusman et al. [4] did not state the reference body weight.

These multicentre studies all found that higher protein intake during the acute phase of critical illness was associated with better outcomes, typically defined as 60-day hospital mortality (see Table 1). In four of these studies [1–3, 5], the association was linear when daily protein intake was categorized in increments of 30 g or analyzed as a continuous variable (percentage intake of the protein target, which mostly varied between 0.9 and 1.6 g/kg/day in the ICUs that participated). In 2016 and 2022, A.S.P.E.N. eventually recommended a protein intake of 1.2–2.0 g/kg/day for critical illness [6, 7].

Table 1.

Large observational studies (> 1000 participants) examining the association between a different protein intake and outcomes in critically ill patients

Study	Number of patients	Design	Inclusion criteria	Variable of interest	Outcome	Results
Alberda et al. 2009 [1]	2772	Retrospective multicenter	MV, ICU LOS > 3 days	Protein intake (EN/PN) between days 1 and 12 (mean) (per Δ = 30 g per day)	P: 60-day hospital mortality S: Ventilator-free days	Sign. negative linear association with (P) only if BMI < 25 or ≥ 35; no association with (S)
Elke et al. 2014 [2]	2270	Retrospective multicenter	MV, ICU-LOS > 3 days	Protein intake (EN) between days 1 and 12 (mean) (per Δ = 30 g per day)	P: 60-day hospital mortality S: Ventilator-free days	Sign. negative linear association with (P) Sign. positive linear association with (S)
Nicolo et al. 2016 [3]	2828	Retrospective multicenter	MV, ICU LOS > 3 days	Protein intake (EN/PN, not volitional) between days 1 and 12 (mean of % of target per day): ≥80% of target vs. <80%; Duration of EN/PN ≥ 4 days	P: 60-day hospital mortality S: Time until hospital discharge (survivors)	Sign. negative association with (P), no association with (S)
Zusman et al. 2016 [4]	1171	Retrospective monocentric	ICU LOS > 4 days	Protein intake (EN/PN) during ICU stay: mean of % of target per day	P: 60-day mortality	Sign. negative non-linear association with (P)
Compher et al. 2017 [5]	2853	Prospective multicenter	Duration of MV > 3 days, low vs. high NUTRIC score	Protein intake (EN/PN) between days 1 and 12 (mean) (per Δ = 10% of target)	P: 60-day hospital mortality S: Time until hospital discharge (survivors)	Sign. negative linear association with (P) and (S) at high NUTRIC score

MV; mechanical ventilation; ICU-LOS: length-of-stay in the intensive care unit; EN: enteral nutrition; PN: parenteral nutrition; P: primary outcome; S: secondary outcome; BMI: body mass index; Sign.: significant

The publication of the randomized controlled EFFORT Protein trial [8] in 2023, the PRECISe trial [9] in 2024 and the TARGET Protein trial [10] in 2025 brought about a landslide change in the existing evidence base. These trials compared protein intakes of 0.9–1.2 g/kg/day and 1.6–1.9 g/kg/day in the acute phase of critical illness and found no benefit, regardless of the type of outcome or the specific study design. After the EFFORT Protein trial, ESPEN published more conservative recommendations in 2023. These recommendations suggest delivering up to 1.3 g of protein equivalents per kilogram of body weight per day during the acute phase [1].

The discrepancy between controlled and observational studies is consistent with several other spectacular observational ‘failures’ that were not confirmed in subsequent high-quality randomized controlled trials (e.g. low-dose steroids for sepsis: CORTICUS trial, early intensive glucose control: NICE-SUGAR trial, pulmonary artery catheter (Swan-Ganz): PAC-Man trial; high-dose continuous renal replacement therapy (CRRT): RENAL trial).

To some extent, this phenomenon appears specific to research in critical care. A recent review summarized the results of methodological reviews that compared the effect estimates of observational studies with randomized controlled trials from evidence syntheses that addressed the same health research question [11]. The authors analyzed 2,869 RCTs involving 3,882,115 participants and 3,924 observational studies involving 19,499,970 participants. The key finding was that there was no difference, or only a very small difference, between the effect estimates of randomized controlled trials and observational studies. Consequently, factors other than study type alone must be considered when investigating reasons for discrepancies between the results of randomized controlled trials and observational studies concerning protein prescription in critical illness.

Formally, the results of randomized trials may be biased by limited external validity. In the EFFORT Protein trial, for instance, only 65.1% of eligible patients could be randomized [8]. This limited external validity is typically offset by high internal validity. The general validity of observational studies and registry analyses may be biased by limited internal validity related to database quality and analytical strategies, including those involving recent AI methods [12]. There are significant reasons to believe that the limited internal validity of older observational critical care nutrition studies is the main reason for the discrepancies with randomized trials.

In a purely observational study design, medical nutrition therapy for critical illness is a highly complex, time-varying variable. The effects of this therapy are not immediate and may also vary over time. Furthermore, and in significant contrast to other fields, a comparably large number of patients escapes observation early (either by in-ICU death or ICU-discharge alive). All these factors explain why observational studies in critical care nutrition are particularly vulnerable to bias. The goal of this narrative review is to discuss statistical biases in observational nutrition studies potentially limiting their internal validity.

Statistical bias in observational studies

The example of the observational study by Alberda et al.

In 2009, Alberda et al. published a prospective observational study examining the impact of energy and protein intake on the outcomes of 2,772 mechanically ventilated patients with an ICU length of stay exceeding three days. Due to its size, the study gained enormous influence rapidly and was considered a landmark study at the time [1]. Consequently, this study may serve as an illustrative example on how statistical bias may affect results in observational critical care nutrition studies.

The authors analyzed daily protein intake (ranging from 0 to 2.3 g/kg per day) during the acute phase of critical illness. The main result was that, for patients with a normal body weight, the risk of death up to 60 days after ICU admission decreased linearly by around 15% for each additional 30 g of protein taken up per day (approximately 0.4 g/kg).

The reasons for this striking discrepancy in the results of recent randomized controlled trials have never been discussed in detail. One potential reason may be various types of statistical bias.

Specific statistical bias in the study by Alberda et al.

In statistics, bias means a systematic distortion of a statistical result due to a factor not allowed for in its derivation. The Alberda study suffers from various such distortions:

• Bias by violation of the time axis

  The Alberda study failed to consider the temporal order of the effector and the event. For example, if one patient died on day 6 after ICU admission, while another patient’s protein intake was recorded up to day 10, it would be a serious error to combine information from these two patients in a statistical model assessing the association of average daily protein intake with death [13]. To overcome this type of bias, the statistical analysis plan must be designed so that the effect always precedes the event on the time axis.

• Bias by confounding by indication

  When analyzing nutrition in critically ill patients, confounding by indication means confounding by better nutritional therapy (e.g. higher (enteral) protein intake) in survivors and lower intake in the dying. Sicker patients have worse clinical outcomes, and the same forces that make them sicker work together to make them more difficult to feed or candidates for feeding restrictions as death approaches [14]. Consequently, protein intake during an ICU stay becomes a reciprocal indicator of disease severity, rather than an independent predictor variable. The Alberda study did not employ strategies to minimize the risk of bias. In the context of enteral nutrition, potential strategies to minimize this type of bias include adjusting for supplemental parenteral nutrition use and gastrointestinal function. The better the gastrointestinal function and the lower the frequency of parenteral nutrition use, the higher the chance of a positive outcome. Another strategy is to implement a lag time between the last day of a nutrition protocol and the day on which the outcome is recorded. Due to therapeutic restrictions, protein intake may be lower immediately before in-ICU death.

• Bias by ignorance of a wash-in phase

  A certain level of protein intake is neither acutely fatal nor life-saving, and a wash-in phase is required for it to potentially become effective. Wash-in effects are of special concern in dietary studies [15]. Macronutrient composition and dose affect the secretion of many hormones. These hormonal responses, in turn, affect myriad biological pathways relating to energy metabolism, growth, reproduction, inflammation, and apoptosis. Considering the profound nature of these biological actions, adaptation/response to a major change in diet may require several weeks in non-critically ill patients [15]. Due to additional, pharmaceutical macronutrient effects, wash-in time may be shorter in critical illness (presumably in the range of days [16], but still requires consideration when designing an observational study. Therefore, a time lag should be accounted for after a defined period of medical nutrition therapy. However, conventional logistic regression analysis, as used in the Alberda study, does not allow such lags to be incorporated into a multivariate model. Only Cox-type models allow for the incorporation of a lag time into the analysis.

• Modelling bias by inappropriate variable selection

  In order to identify the independent association between a certain level of protein intake and mortality, it is essential to adjust for potential confounding factors. How these confounders are defined, selected, and modelled individually is crucial for the validity of the results. The technical procedures employed in the statistical software used in the Alberda study to select variables for the final model — namely, forward selection, backward elimination, and stepwise selection — are now recognized as biased (the current standards, albeit not directly interchangeable, are directed acyclic graphs, the Akaike information criterion (AIC), the least absolute shrinkage and selection operator (LASSO) and machine learning procedures [17, 18]).

• Modelling bias by inappropriate modelling of individual variables

  The Alberda study assumed that there was a linear association between numerical confounding variables (e.g. age) and the outcome. In reality, the relationship between numerical variables and outcomes is not always linear. As no attempt was made to consider a potential non-linearity of age (i.e. to test smooth associations), the results of the multivariate analyses can be biased [19]. Non-linear, unadjusted models should have been tested to analyze the predictive power of numerical confounding variables before incorporating their linear models into a regression model. Generalized additive models (GAMs) are currently the preferred method for minimizing this type of bias.

• Incomplete outcome ascertainment

  The Alberda study used 60-day in-hospital mortality as the outcome measure. However, this approach is biased, as a significant proportion of patients who are discharged alive may actually die soon after being transferred to other institutions [20, 21]. In the context of the Alberda study, a time-to-event analysis including the analysis of competing risks (time to in-hospital death or hospital discharge alive) would have been required [22]. Fine-and-Gray models (also known as subdistributional hazard models) [23] are currently the preferred method for minimizing this type of bias and for determining the hazards related to the time until hospital discharge alive, given the risk of in-hospital death.

In summary, even if the relative importance of these different types of statistical bias are unknown, the results of the Alberda study are heavily biased for a number of statistical reasons poorly understood at the time when the study was conducted. Consequently, the results are historical and cannot inform current recommendations on how to prescribe protein during the acute phase of critical illness.

Data analysis in observational studies: considering statistical bias is key

In order to qualify as a high-quality observational study in the field of medical nutrition therapy, studies must take into account some additional risks of statistical bias that are currently recognized as being of great importance when analyzing observational nutrition data.

• Sample size bias

  Calculating an appropriate sample size is especially essential with comparatively small databases. For binary outcomes, the number of outcome events per predictor variable (EPV) determines the reliability of multivariate analyses [24]. For numeric or countable outcomes, the reliability parameter is the number of subjects relative to the number of confounders [25].

• Competing risks bias

  The acute mortality rate among critically ill patients is high by default. If death is not the only selected outcome, then any other outcome will be affected by death. This is relevant for binary outcomes where the event (e.g. the onset of acute kidney failure) could not have been ‘experienced’ by ‘early’ death, and for time-related numeric or countable outcomes (e.g. the duration of mechanical ventilation), which may be ‘artificially’ shortened by death. Estimating associations with binary outcomes or durations in the context of high mortality requires assessing competing risks [26] or using specific tools (e.g. multistate models) to account for death [27].

• Bias by information missing not at random (MNAR)

  For outcomes that are only registered at specific points in time (e.g. muscle thickness), drop-out is a common phenomenon. Missing information during an ICU stay (e.g. due to in-ICU death or discharge from the ICU alive) cannot be assumed to be missing at random, since patients with and without missing information may differ systematically. Methods such as mixed modelling and multiple imputation of patient-reported outcomes can be used to minimize this type of bias [28]. Missing not at random (MNAR) also causes significant bias in popular nutrition research methods such as classical group-based trajectory modelling (GBTM) and latent class trajectory modelling (LCTM), both of which assume that missing data are missing at random. Specific extensions are necessary to account for this bias [29, 30].

• Immortal time bias

  When using prolonged ICU length of stay as an inclusion criterion or when trying to identify trajectories, patients must remain in the ICU for a certain period of time. Therefore, a certain length of stay, and consequently survival time, is both a prerequisite and an outcome. Certain therapies may affect the length of stay, for example by reducing in-ICU deaths or resulting in a later ICU discharge. This scenario causes an immortal time bias [31].

• Collider bias

  Incorporating an exposure (e.g. high protein intake), a confounder (e.g. severe sepsis) and a third variable (e.g. acute kidney failure) into a model where the outcome is mortality would cause collider bias. The collider would be acute kidney failure, as this may be caused independently by severe sepsis and high protein intake [32].

• Modelling bias when analyzing numeric or countable outcomes

  Popular linear regression analyses assume a linear association between a numeric or countable outcome and a numeric or countable predictor variable, and that the outcome parameters are normally distributed. In reality, however, both of these assumptions are often incorrect, and statistical ignorance can lead to significant bias. To minimize this bias, outcomes must be transformed, for example using negative binomial or Poisson regression models [33], and the non-linearity of confounders must be accounted for using generalized additive mixed models [34].

Thus, the quality assessment of observational studies is inherently complex due to the numerous potential sources of bias. Identification of a specific bias is extremely challenging, requiring in-depth statistical knowledge. However, this knowledge is essential to identify those studies with relevant results due to low statistical bias among the myriad nutrition observational studies in critical illness. The adequacy of the statistical methods used is the primary indicator of quality in observational studies.

Although established tools such as TRIPOD [35] and PROBAST [36] address the reporting and risk of bias in diagnostic and prognostic models, they lack precise recommendations on how to assess the quality and adequacy of statistical analyses. The ROBINS-E tool can be used to assess potential biases in study-specific estimates of the effects of non-interventional types of exposures on outcomes [37]. For intervention models, the planned revision of the ROBINS-I tool (ROBINS-I V2) [38] will provide more detailed recommendations on how to assess statistical bias due to due to missing data or confounding factors in the classification of interventions and selection of participants, in the measurement of the outcome, and in the selection of the reported result. For obvious reasons, however, neither ROBINS-I V2 nor ROBINS-E can provide details on the appropriateness of specific statistical procedures.

Analytical strategy in high-quality observational studies: mimicking a randomized controlled trial is crucial

Historically, statistical biases specific to the Alberda study and those generally relevant to observational critical care nutrition studies have been nearly impossible to overcome, despite the use of highly sophisticated statistical analysis plans. One major reason for this was the lack of user-friendly, specialized software. Consequently, it has been concluded that only randomized controlled trials can provide reliable nutritional guidance for clinical practice [39].

It has only been in the last decade that these types of bias could be effectively addressed in statistical research due to significant progress in software availability. To minimize bias, competing risk models are available today where the hazard is a piecewise constant function depending on protein intake, for example, and confounding variables. Confounder models are flexible non-linear models with partially time-varying and partially time-constant associations. Non-linear associations are estimated using penalized splines, and piecewise exponential additive mixed models include lag and lead times. This proposed combination of established statistical models also enables the consideration of cumulative associations and the pairwise comparison of different nutrient intakes in a specific period of time [40–44].

Such highly complex analytical strategies can mimic the design and measurements of randomized controlled trials. This is a prerequisite if real-world studies are to reach similar conclusions to those of randomized controlled trials [45–47].

In fact, when we reanalyzed the Alberda study database (supplemented with more recent data from the same registry) using the complex analytical strategies described above to minimize the risk of statistical bias, we obtained results comparable to those of recent randomized trials [46, 48]. It should be noted that the results of the randomized trials were unknown when this registry reanalysis was performed, and the main results of the reanalysis [46] were published before those of the EFFORT Protein trial.

Furthermore, in the context of critical illness, these strategies allow a separate analysis of protein prescriptions made during different phases of the disease. This differentiation is important since the effectiveness of protein intake might vary according to disease phase [49].

Summary and future perspective

There are numerous types of statistical bias that are particularly relevant to research in critical care nutrition. The Alberda study illustrates how failure to consider the enormous statistical complexity inherent in an appropriate analysis of the data may explain why older observational studies addressing protein intake in critical illness obtained results that differ from those of randomized trials. Although we did not scrutinize the other studies presented in Table 1 for bias, it is highly likely that the various types of bias discussed above contributed to the apparent benefits to a significant degree. Four of the five studies were follow-ups to the Alberda study, all of which were conducted by the same group.

Addressing this statistical complexity requires high-level professional statistical guidance, similar to that required for randomized controlled trials. Particular attention must be paid to AI algorithms, as their flexibility makes them highly vulnerable to biases arising from the analysis of routinely collected data. Common challenges include time point alignment, interventions and tests not being random, and analytical variability due to a multiplicity of possible analysis strategies. This multiplicity requires rigorous internal and external validation [12].

A detailed statistical analysis plan should be published before the study is conducted to allow for high-quality statistical peer review, thereby optimizing the quality of the statistical analysis. Furthermore, such studies should undergo external professional statistical peer review after submission. It cannot be assumed that medical reviewers will recognize specific statistical shortcomings. Only this dual-track strategy may ensure the high quality of observational studies, which remain important due to their ability to examine real-world scenarios.

Author contributions

Conceptualization: MN, WH; Writing – Original Draft Preparation: WH, MN; Writing – Review & Editing: MN, WH. All authors read and approved the final manuscript.

Funding

None.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

Not applicable. This manuscript is an opinion paper (narrative review) and does not report original research involving human participants, animals, human tissue, or human cell samples. Therefore, ethics committee approval, informed consent, CONSORT reporting, and confirmations regarding compliance with experimental guidelines and regulations are not applicable.

Consent for publication

All authors critically reviewed and approved the final submitted version of the manuscript. All authors have agreed both to be personally accountable for the author’s own contributions and to ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Competing interests

The authors declare no competing interests.