Letter to the Editor: Can Predictive Modeling Tools Identify Patients at High Risk of Prolonged Opioid Use After ACL Reconstruction?

To the Editor,

We read the study by Anderson et al. [1] with great interest and appreciate the authors for building prediction models to identify high-risk patients of prolonged opioid use after ACL reconstruction. Nevertheless, we have some concerns about the report that we would like to share.

First, Anderson et al. [1] noted that “the gradient boosting machine model was the most suitable for clinical application.” They developed four models in this study, but only one was plotted using decision curve analysis (Figure 7 in the original study). Decision curve analyses assess a model’s clinical utility [5], and preferably are used for comparisons among multiple models, such as comparing the net benefits across a particular threshold range [2]. Here, the plot with one single model only proves that the model’s clinical application gains are improvement over chance alone. In our view, to support such a statement, the authors should have incorporated all four models’ decision curves into the same figure.

Second, the authors abandoned the logistic regression model and chose the gradient boosting machine (GBM) as the final model, which we think is questionable for two reasons. First, the areas under the curve for the two models were the same (0.76 vs. 0.76), while the calibration curve of the logistic regression seemed better. According to Figure 3 in the original study, the GBM model overestimates the risks in both low- and high-risk populations, while underestimating the medium-risk. Additionally, the 95% CI of the calibration curve in the high-risk population is broad, indicating a lack of power. Secondly, clinical application is at the heart of developing prediction models. The machine learning model is more likely to be a “black box” [4], in that the algorithm behind the model is always too complicated for clinical use. Just as the authors stated, the clinical use of such models should be based on “a web-based application” [1]. In this case, the logistic regression model is easier to implement because it can be transformed into easy-to-use, paper-based tools like nomograms, score schemes, and decision trees [5].

Finally, the authors stated that “the next step of the model lifecycle is external validation of the model in other patient populations” [1]. However, it is unlikely that the predictors included in the final model will be broadly applicable. We believe the final model can only be used in certain geographical regions (in the United States), and only for military patients due to the inclusion of predictors of pharmacy ordering site, beneficiary region, and military rank. Given the source of the data for the current study, it does not seem appropriate to use it in other settings or in the general population. It would be helpful to know whether general factors that we have seen in non-military populations [3] have similar associations with the outcomes of interest in military populations. For example, low household income has been associated with prolonged opioid use after major surgery [3], and this could have been a large part of the observed finding in the study [1] that associated region/rank and prolonged opioid use. Replacing region-/rank-specific factors by general factors such as household income in the final model would facilitate external validation.