Recommendations for the Assessment and Reporting of Multivariable Logistic Regression in Transplantation Literature

Abstract

Multivariable logistic regression is an important method to evaluate risk factors and prognosis in solid organ transplant literature. We aimed to assess the quality of this method in six major transplantation journals.

Eleven analytical criteria and four documentation criteria were analyzed for each selected article that used logistic regression.

A total of 106 studies (6%) out of 1,701 original articles used logistic regression analyses from January 1, 2005 to January 1, 2006. The analytical criteria and their respective reporting percentage among the six journals were: Linearity (25%); Beta coefficient (48%); Interaction tests (19%); Main estimates (98%); Ovefitting prevention (84%); Goodness-of-fit (3.8%); Multicolinearity (4.7%); Internal validation (3.8%); External validation (8.5%). The documentation criteria were reported as follows: Selection of independent variables (73%); Coding of variables (9%); Fitting procedures (49%); Statistical program (65%). No significant differences were found among different journals or between general versus subspecialty journals with respect to reporting quality. We found that the report of logistic regression is unsatisfactory in transplantation journals. Because our findings may have major consequences for the care of transplant patients and for the design of transplant clinical trials, we recommend a practical solution for the use and reporting of logistic regression in transplantation journals.

INTRODUCTION

“We live in a multivariable world” (1) and most events in life are multifactorial, i.e. many different factors (i.e. variables) are associated to a specific outcome. Multivariable logistic regression is one of the tools that help to determine the contribution of each of these factors to a single outcome. Multivariable logistic regression is an important statistical method commonly used in all fields of Medicine and Surgery, as well as in the solid organ transplant literature. This method is mainly used to either understand the effect of several variables (e.g. risk factors) that may ‘explain’ changes in specific outcomes (e.g. allograft rejection) – also called explanatory modeling, or that may ‘predict’ changes in specific outcomes (e.g. patient survival) - predictive modeling. Of note, while the explanatory or predictor variables being analyzed can be continuous -also called numerical due to the presence of multiple levels (e.g tacrolimus levels), or dichotomous, also called binary due to the presence of two levels (e.g. gender male vs. female), the outcome variable must be dichotomous for most logistic regression analyses.

Landmark studies evaluating this methodology in four major medical journals -NEJM, JAMA, ANNALS, and BMJ - have consistently demonstrated important analytic and reporting problems (2–4) concerning the reporting of multivariable regression analyses. These problems have important bedside implications considering that the regression analyses results from published studies may be inaccurate and lead to inappropriate application to the clinical or surgical care of transplant patients.

We performed an assessment of the quality of logistic regression models in the solid organ transplant literature. In order to perform this study, we conducted a systematic review of the literature in six major transplantation journals. Then, we used specific criteria to evaluate the quality of the studies which used logistic regression models. Finally, we analyzed the overall proportion of articles with adequate methodology within each assessment criterion, addressed the magnitude and clinical importance of the observed methodology inadequacies, and provided solutions for the better utilization of logistic regression models in the transplant literature.

METHODS

We developed our initial assessment criteria based on three studies (5–7) and one expert guideline by the American College of Physicians (8). Further modifications were made according to six other studies (2–4, 9–11).

a) Transplantation Journals Selected For Data Extraction

All original articles including the keywords “multiple, multivariable, multivariate, logistic, linear, regression, model, method, risks, prediction, rules”, and published from January 1, 2005 to January 1, 2006 were searched in PubMed for the following journals: General transplantation journals: American Journal of Transplantation; Transplantation; and Clinical Transplantation. Subspecialty journals: Nephrology Dialysis Transplantation; Liver Transplantation; and Journal of Heart and Lung Transplantation.

The reasons for choosing the year of 2005–2006 were the following: 1) to allow PubMed enough time to correct reporting mistakes commonly seen after articles’ release; 2) to not miss articles which were retroactively placed in PubMed; 3) to allow time for potential articles retrievals from each journal; 4) to allow time for the PubMed incorporation of errors discovered after final publications; and 5) to allow time for us to assess letters to the editor concerning each selected article.

It is of importance to remind the reader that what was reported in a manuscript may not have always reflected what was actually done in its entirety. However, reporting itself was the main focus of our study, thus we assumed that the published reports encompassed and represented most of what was performed for their regression analyses.

Both Fisher’s exact test and Chi-squared test were performed for all criteria comparisons between journals. A pre-determined alpha error threshold of 0.05 (two-tailed) was used. Due to the small sample size of each category and the expected frequency of some cells being less than 5, which makes Chi-squared test results less reliable, we only report results from Fisher’s exact test.

b) Analytical Criteria

1. Presence of linearity for continuous or transformed categorical variables:

   • Inadequate: no description of the linearity assumption

   • Adequate: description of the linearity assumption. Preferably some kind of justification is provided, e.g. prior clinical knowledge, exploratory plot of the data.

2. Presence of multiple regression equation or table:

   • Inadequate: no report of coefficients and associated standard errors.

   • Adequate: report of each coefficients and associated standard errors.

3. Test for interaction for independent variables:

   • Inadequate: use of interaction term without providing its statistical or clinical justification.

   • Adequate: use of interaction term providing its statistical or clinical justification.

4. Main effect estimates, 95%CI, and statistical significance:

   • Inadequate: odds ratio and 95%CI (or p values) for each variable included in the final model is not provided.

   • Adequate: odds ratio and 95%CI (or p values) for each variable included in the final model is provided.

5. Prevention of overfitting:

   • Inadequate: The ratio between the number of the less common outcomes and the number of independent variable parameters is less than 10.

   • Adequate: the ratio is equal or greater than 10.

6. Report of outlying data:

   • Inadequate: If present, no report about how these data were treated.

   • Adequate: If present, report about how these data were treated.

7. Goodness-of-fit procedures:

   • Inadequate: no reporting of goodness-of-fit test (e.g. Hosmer-Lemeshow test) or regression diagnostics.

   • Adequate: reporting of goodness-of-fit test (e.g. Hosmer-Lemeshow test) or regression diagnostics. Area under ROC curve should be provided if prediction is of interest.

8. Multicolinearity of predictive models:

   • Inadequate: If high correlation between two independent variables is discussed in the paper, but a test for colinearity is not provided.

   • Adequate: If high correlation between two independent variables is discussed in the paper and a test for colinearity is provided.

9. Internal validation of predictive models:

   • Inadequate: no data-splitting (including cross-validation), or bootstrapping, or jackknife.

   • Adequate: provision of at least one of these 3 procedures.

10. External validation of predictive models:

    • Inadequate: not prospectively studied in a different patient population.

    • Adequate: prospectively studied in a different patient population.

11. Evidence for clinical decision rules:

    • Level 1: At least one prospective validation in a different population and one impact analysis, demonstrating change in clinician behavior and improve patient outcomes.

    • Level 2: At least one prospective validation in a broad spectrum of patients and clinicians, or validated in several smaller settings that differ from one another.

    • Level 3: At least one prospective validation in a narrow spectrum of patients, similar to those evaluated in the derivation sample.

    • Level 4: Just internal validation (data-splitting, bootstrapping, jackknife) from retrospective data.

    • Level 5: Derivation only. No internal validation.

c) Documentation Criteria

1. Selection of independent variables:

   • Inadequate: no provision of the rationale for including each variable.

   • Adequate: provision of the rationale for including each variable, e.g. prior studies/clinical knowledge, exploratory analysis.

2. Coding of independent variables:

   • Inadequate: if variables included in the model can not have coding identifiable by our team.

   • Adequate: if variables included in the model have coding identifiable by our team.

3. Fitting procedure:

   • Inadequate: no provision of how the final model was selected.

   • Adequate: provision of model selection method used, such as automatic procedures (e.g. forward inclusion or backward elimination or stepwise or best subset, preferably with description of the criteria used), non-automated backward selection, or prior knowledge.

4. Statistical program:

   • Inadequate: no provision of software used.

   • Adequate: provision of software used.

RESULTS

a) Selected Studies

A total of 106 studies out of 1,701 original articles (6%) used logistic regression analyses in the six transplantation journals during the year of January 1 2005 to January 1, 2006: Forty-six articles were selected from general transplantation journals (American Journal of Transplantation [11 studies] (12–22); Transplantation [27 studies] (23–49); and Clinical Transplantation [8 studies] (50–57); and sixty from subspecialty transplantation journals (Nephrology Dialysis Transplantation [28 studies] (58–85); Liver Transplantation [20 studies] (86–105); and Journal of Heart and Lung Transplantation [12 studies] (106–117).

b) Results for the Analytical Criteria

Ten out of the twelve analytical criteria results are described in table 1. Criterion number 7 (report of outlying data) had zero reports out of 106 studies. Criterion number 12 (evidence for clinical decision rules) showed the following results: For general transplant journals (level 5 [39 studies]; level 4 [3]; level 3 [3]; level 2 [1]; level 1 [0]); For subspecialty transplant journals (level 5 [36 studies]; level 4 [4]; level 3 [16]; level 2 [4]; level 1 [0]).

Table 1.

Adequacy of Analytical Criteria: Number (%)

Transplant Journal	Report of Linearity	Report of β (SE)1	Report of Interaction Tests	Report of Main Estimates	Report of Overfitting Prevention	Goodness- Report of of-fit	Report of Multicolinearity	Internal Validation	External Validation
General2	12/46 (26%)	20/46 (43%)	7/46 (15%)	44/46 (96%)	36/46 (78%)	2/46 (4.3%)	1/46 (2.2%)	2/46 (4.3%)	3/46 (6.5%)
Subspecialty3	15/60 (25%)	31/60 (52%)	13/60 (22%)	60/60 (100%)	53/60 (88%)	2/60 (3.3%)	4/60 (6.7%)	2/60 (3.3%)	6/60 (10%)
P value4	1.00	0.44	0.46	0.19	0.19	1.00	0.38	1.00	0.73
Total	27/106 (25%)	51/106 (48%)	20/106 (19%)	104/106 (98%)	89/106 (84%)	4/106 (3.8%)	5/106 (4.7%)	4/106 (3.8%)	9/106 (8.5%)

¹Regression coefficient and standard error

²American Journal of Transplantation; Transplantation; and Clinical Transplantation.

³Nephrology Dialysis Transplantation; Liver Transplantation; and Journal of Heart and Lung Transplantation.

⁴P value for the comparison between general and subspecialty journals.

c) Results for the Documentation Criteria

All documentation criteria results are described in table 2.

Table 2.

Adequacy of Documentation Criteria: Number (%)

DOCUMENTATION CRITERIA RESULTS
Transplant Journal	Selection of Independent Variables	Coding of Independent Variables	Description of Fitting Procedure	Description of Statistical Program
General1	30/46 (65%)	4/46 (8.7)	20/46 (43%)	25/46 (54%)
Subspecialty2	47/60 (78%)	6/60 (10%)	32/60 (53%)	44/60 (73%)
P value3	0.19	1.00	0.33	0.06
Total	77/106 (73%)	10/106 (9%)	52/106 (49%)	69/106 (65%)

¹American Journal of Transplantation; Transplantation; and Clinical Transplantation.

²Nephrology Dialysis Transplantation; Liver Transplantation; and Journal of Heart and Lung Transplantation.

³P value for the comparison between general and subspecialty journals.

d) Results for the comparison among journals

All p values were greater than 0.05 for each criterion comparison between different journals, as well for each criterion comparison between general versus subspecialty transplantation journals (tables 1 and 2).

DISCUSSION

Our findings demonstrate that logistic regression analyses are unsatisfactorily reported in transplantation research, both in general and subspecialty journals. The report of main estimates and the prevention of model overfitting were the only analytical criteria reported by the majority of journals (84–98%), and the selection of independent variables was the documentation criterion with the highest reporting rate (65–78%). While 10 or more outcome events per variable is a traditional rule to prevent model overfitting (6, 118, 119), there is recent evidence that a minimum of 8 events per variable should produce models as well fit as the ones with 21 events per variable (120). These fitting rules are very helpful but are not perfect since relevant variables may end excluded by following these guidelines too strictly.

The criterion with the lowest reporting rate among all was the goodness-of-fit, which was present in just 4% of all papers. Goodness-of-fit tests are useful to understand how well the model fits the data, that is, how well the model reflects the data on which it was generated. If this test reveals a lack-of-fit, the final results of the logistic regression may be severely compromised and potentially unreliable. This is an area in which studies have definitely to improve.

The report of multicolinearity was low for all journals (4.7%). Multicolinearity is an important analytical criterion that determines if two or more variables are parallel or collinear, which implies that they are not independent from each other (an implicit assumption of all logistic models). For example, inappropriate low levels of tacrolimus post-transplant and poor patient adherence with medication intake in a model evaluating allograft rejection as the outcome (dependent) variable – both can increase the risk of rejection. If these non-independent variables are kept in the model, it is possible that the statistical association may be reported as absent despite the fact there is a statistically significant association with one of these variables. In other words, despite the fact that inappropriate low levels of tacrolimus is associated with higher rejection rate, the inclusion of this variable in addition to the poor compliance variable in the same logistic model may produce falsely negative results. If multicolinearity is detected, then one of the variables - usually the one with the weakest relationship - should be removed from the model.

Interaction tests should be a routine part of all logistic regression reports, but we found it present in only 19% of the publications in transplant journals. The reason for its importance is that the effect of one variable on the outcome would depend on the effects of a second variable. For example, we consider delayed kidney graft function (defined as the need for dialysis within the first week post transplant) and acute cellular rejection as variables for a logistic model evaluating allograft survival (outcome variable). Either variable may not show significant association with the study main outcome (or either may show a weak association), but when both are analyzed together as a single “interaction variable”, the results may change drastically. That is why interaction effects are also called “modification effects”. Opposite to the multicolinearity solution (exclusion of one variable), statistically significant interaction demonstrate that both variables should be kept together until the final regression model.

Interestingly, the report of external validation was low (8.5%) and this was expected because of the inherent logistic difficulties of applying regression results to other patient populations and institutions. However, the finding of an another population cohort to perform an external validation may not be possible due to inherent difficulties associated to recruiting the adequate number of subjects to perform another analysis, especially in single-center studies. On the other side, the quite low rate of internal validation (3.8%) was unexpected because this can be easily done even with a retrospective analysis from a single institution, which is a common setting for logistic regression analyses. The authors would have to either split the data or perform repeated simulations within the original population, then use procedures called data-splitting, bootstrapping, or jackknife to perform internal validation. This type of validation brings the original regression findings to another level of reliability because the original results have to stand despite the use of different subset of patients.

In general, documentation criteria were better reported than that of analytical criteria. Seventy three percent of the manuscripts mentioned the reasons for the selection of independent variables. This is an optimistic finding because the reader must know the rationale for all the variables entered into the model. Surprisingly, the description of statistical programs was seen only in 65% of the publications. This should be standard request from every journal not only because the data can be further verified by the use of the same software, but also because the peer-reviewers can request further information based on the capabilities of the program being used by the authors.

It is unfortunate that the fitting procedures were reported in 49% of the papers. There are specific technical indications for simultaneous or hierarchical (e.g. forward inclusion or backward elimination) procedures to be used. Different fitting procedures may result in substantially different study results, therefore, the clear specification of the type of procedure and the reasons for choosing it are essential for the quality of the regression analyses report. The coding of the independent variables was rare (9%), but we do not believe that the cumbersome addition of the coding for each variable would add substantial information to the physician. On the other hand, this could be provided as an electronic appendix.

One possible limitation of these data is that not all articles using regression analyses were included in this study; if for example, they did not have one of the key words to meet our search criteria. We don’t have data to estimate whether or not 6% of the publication is the correct fraction of studies using regression analysis for that given year, but we believe this number (106 studies) is representative of the majority of the articles using regression analysis during the selected period. Another argument against our findings could be based on the relatively low rate of use for this method −6%. However, we strongly believe that the results from these 6% have substantial influence in both patient care and future research because statistically significant risk factors by logistic regression are used not only in the direct assessment of transplant patients (e.g. which patients are at higher risk for cytomegalovirus infections), but also in the design of future clinical trials (e.g. which patients should be enrolled in a trial evaluating a new drug to prevent cytomegalovirus infections). The logistic regression findings concerning prognostic factors would have similar applicability in both clinical and research settings. Another limitation concerns to the possible redundant information provided by items 2 and 4 of the analytical criteria: both lead to the reporting of the odds ratio and confidence intervals, but the reporting of beta values and standard errors gives the reader an extra opportunity to more precisely evaluate the regression equation, and potentially apply this equation to his/her institution. Variables that depend on time were not evaluated in our study because the logistic regression model can not account for time to event or censored events (e.g. events that did not happened until the end of study follow up); other statistical methods such as the Cox proportional hazards regression analysis would be more appropriate for these variables. Last, we did not perform inter-rate reliability analyses, however, except for two disagreements resolved by consensus, authors agreed about everything else.

We propose two solutions in order to improve the report of regression analyses: 1) Transplantation journals should have a concise but obligatory list of requests for all manuscripts that use the logistic regression technique. Our suggestions for this list are described in table 3. If the requirements are not met at first submission, the paper should not be sent out for peer-review; 2) If one or more peer-reviewers still raises questions about the logistic regression report, the journal should ask for an independent statistician to perform a specific review of the technical aspects of the analyses. Also, journal editors should increase the emphasis on a more comprehensive description of study methods section; space limitation should not be an excuse for short methods’ section anymore since nowadays most journal websites can publish electronic appendices of any size.

Table 3.

Suggestions for Transplantation Journals “Minimal Requirements” for the Reporting of Logistic Regression Analyses

1) Report of Odds Ratio and 95%CI

2) Report of Overfitting Prevention Procedures

3) Report of Goodness-of-Fit Procedures

4) Report of Multicolinearity Tests

5) Report of Interaction Tests

6) Internal Validation (if sample size allows)

7) Report of the Selection of Independent Variables

8) Report of Statistical Program

In conclusion, we found that the report of logistic regression is unsatisfactory in transplantation articles. Because this finding may have major consequences for the care of transplant patients, we believe that journal editors should be more pro-active and specific about the requirements for the publication of studies with logistic regression analyses. We hope that through our dissection of the major reporting problems with logistic regression studies and our suggestions for possible resolutions, we may stimulate both authors and editors to further refine the report of their next logistic regression manuscript.