Reply: Developing Machine Learning Algorithms to Support Patient-centered, Value-based Carpal Tunnel Decompression Surgery

Sir,

We thank Mantelakis and Khajuria¹ for opening the opportunity to explain some of the more technical elements of our recent article.²

Black box algorithms are indeed potentially dangerous, but our algorithms were not black boxes. One of the strengths of our article was the use of Shapley additive explanations (SHAP) to estimate the contribution of each feature to each decision. Contrary to what they claim, reading our article and supplementary material confirms that the variables they are concerned about made very little contribution to the algorithms’ decisions.

A more useful discussion might have focused on the limitations of SHAP and similar procedures,³ approaches to variable selection under traditional statistical and machine learning paradigms, or how model parsimony may have improved generalizability.

Mantelakis and Khajuria’s understanding of data splitting, overfitting, and hyperparameter tuning is not entirely accurate. Through k-fold cross validation, the first dataset is iteratively partitioned into what they term “training” data (k-1-folds), and what they term “validation” data (1-fold, more commonly referred to as “test” data). Once hyperparameters are selected, parameters are fitted to the whole of the first dataset (k-folds). The second dataset acts as what Matelakis and Khajuria term a “test” set (usually known as the internal validation dataset).

A more valid criticism would be our use of internal validation only, rather than evaluating the models in an external sample. This is likely to overestimate performance, which we highlight as a limitation of our work. It would have been more insightful to challenge our use of data splitting in general, as this technique does not make the most of all available data. Other techniques, such as bootstrapping and adjusting for model optimism have been proposed, and these may have been preferable.⁴

Finally, the correspondence authors have confused area under the receiver operating characteristic curve with classification accuracy.

Our article certainly has limitations, and could be challenged by experts. For example, how appropriate is it to dichotomize patients into improved/not improved, based on distributionally estimated minimal important change statistics? How appropriate is it to dichotomize at all, and would a regression algorithm have been more appropriate than a classifier? Is class imbalance truly a problem? Our article lacks a calibration plot, or an assessment of net benefit and we could have perhaps shown closer adherence to the TRIPOD⁵ and PROBAST guidance.⁶

Constructive criticism has an important role to play in improving publishing standards, and we fully support it. It is hoped that we have taken this opportunity to provide helpful feedback. We welcome further criticism of the article, and look forward to seeing original research contribution from Mantelakis and Khajuria to this field in the future.