Fig. 1.

Overview of the prediction pipeline. For early surgery, we identified LDH/LSS patients if they have at least 2 diagnosis codes one year prior to LIRE enrollment and then identified out of these patients as having surgery if they had at least 1 decompression code within 2 months ahead. For late surgery, we identified LDH/LSS patients if they have at least 2 diagnosis codes one year prior to LIRE enrollment and then identified out of these patients as having surgery if they had at least 1 decompression code within 12 months ahead of a 2 month gap. For each prediction task, we collected patients’ demographics, diagnosis codes, procedure codes, drug names, and index image reports. For the multimodal deep learning architecture, the index image reports are passed into a CNN, the diagnosis and procedure codes and drug names are passed into a GRU, and the demographics are featurized. The output from each network are concatenated together along with the featurized demographics and then passed into a fully-connected layer and then to an output layer to make predictions. CNN, Convolutional Neural Network; GRU, Gated Recurrent Unit; LSS, Lumbar Spinal Stenosis; LDH, Lumbar Disc Herniation; LIRE, Lumbar Imaging With Reporting Of Epidemiology