An Online Evidence-Based Decision Support System for Distinguishing Benign from Malignant Vertebral Compression Fractures by Magnetic Resonance Imaging Feature Analysis

Abstract

Decision support systems have been used to promote the practice of evidence-based medicine. Computer-assisted diagnosis can serve as one element of evidence-based radiology. One area where such tools may provide benefit is analysis of vertebral compression fractures (VCFs), which can be a challenge in MRI interpretation. VCFs may be benign or malignant in etiology, and several MRI features may help to make this important distinction. We describe a web-based decision support system for discriminating benign from malignant VCFs as a prototype for a more general diagnostic decision support framework for radiologists. The system has three components: a feature checklist with an image gallery derived from proven reference cases, a prediction model, and a reporting mechanism. The website allows users to input the findings for a case to be interpreted using a structured feature checklist. The image gallery complements the checklist, for clarity and training purposes. The input from the checklist is then used to calculate the likelihood of malignancy by a logistic regression prediction model. Standardized report text is generated that summarizes pertinent positive and negative findings. This computer-assisted diagnosis system demonstrates the integration of three areas where diagnostic decision support can aid radiologists: first, in image interpretation, through feature checklists and illustrative image galleries; second, in feature-based prediction modeling; and third, in structured reporting. We present a diagnostic decision support tool that provides radiologists with evidence-based guidance for discriminating benign from malignant VCF. This model may be useful in other difficult-diagnosis situations and requires further clinical testing.

Background

Evidence-based medicine (EBM) has been defined as “the process of systematically finding, appraising and using contemporaneous research findings as the basis for clinical decisions”¹. Over the past few decades, EBM has emerged in response to perceived variability and complexity in clinical practice, leading to an emphasis on the use of clinical research in routine medical practice^2,3.At the same time, computer-based clinical decision support systems have continued to evolve in sophistication. Together, these trends have led to a range of systems designed to promote the application of research-based practice guidelines in areas such as management of chronic diseases and selection of antibiotics^4,5.

While awareness of EBM is widespread in some specialties, it has received relatively less attention in radiology^6,7. Barriers to a broader application of evidence-based radiology (EBR) include lack of time, unfamiliarity with how to translate published research results to clinical practice, and limited access to resources⁶, and while there have been technological developments in computer-aided radiology exam selection^8–12 and artificial intelligence for image interpretation^13–15, there continues to be an opportunity for further integration of evidence-based research within radiology decision support systems. Such integration has the potential to reduce the barriers to more widespread practice of EBR. In addition, the potential importance of EBM in radiology education has also been discussed¹⁶, and computer applications incorporating current knowledge may help to advance training in imaging as well.

In the realm of diagnostic decision support in radiology, approaches to computer-assisted image interpretation range from techniques in computer vision for image segmentation (i.e. computer-aided detection) to feature-based prediction modeling¹⁷. While the former may be considered more fundamental with regard to image understanding, the latter may be more practical in many situations, leveraging the radiologist’s skill in synthesizing feature detection and characterization. Feature-based prediction may utilize any of a number of modeling methods, including Bayesian networks, logistic regression, recursive partitioning, and neural networks^13–15, in order to derive potential diagnoses. One specific problem amenable to feature-based prediction modeling and diagnostic decision support is analysis of vertebral compression fractures (VCFs) on magnetic resonance imaging (MRI).

Vertebral Compression Fractures

Metastases to the vertebrae are present in 5% to 10% of all patients with malignancy¹⁸. Malignant VCFs occur in approximately 10% to 15% of patients with skeletal metastasis¹⁹. Another common cause of VCFs is osteoporosis, a benign skeletal disease characterized by low bone mass and micro-architectural deterioration of bone tissue, leading to enhanced bone fragility²⁰. In the US, there are approximately 700,000 osteoporotic VCFs per year resulting in about 115,000 hospital admissions. The lifetime risk of an osteoporosis-related VCF is approximately 16% for women and 5% for men; the latter is likely an underestimate²¹. Acute/subacute VCFs are often associated with bone marrow edema, regardless of etiology. In addition, several other imaging findings may be present, and there may be significant overlap in the imaging appearance of benign (osteoporotic) and malignant (typically metastatic but sometimes myeloma-related) VCFs. Differentiating benign and malignant spinal compression fractures is a common problem confronting radiologists.

Structured Reporting

Efforts to standardize the format and content of radiology reports have been underway for over a decade, with the goal of improving the clarity and efficiency of communication in radiology²². This has provided part of the impetus for the RadLex project²³, which is a controlled lexicon of radiological terms. In addition, the Radiology Reporting Committee of the Radiological Society of North America (RSNA) has recently described its work in identifying subspecialty-based best practices in radiology reporting²⁴. Decision support systems based on imaging features are well positioned to leverage both controlled lexicons and structured reporting templates, and MRI feature analysis of VCFs integrates well with the objectives and techniques of standardized reporting.

Evidence-based prediction modeling can help a radiologist to integrate the range of findings present in a given case and may serve to clarify the certainty of a particular diagnosis. In order to achieve these goals in a clinically useful manner, speed and ease of use are of critical importance²⁵. This work describes a web-based decision support system, using feature-based prediction modeling and a standardized reporting mechanism, for discriminating benign from malignant VCFs by MRI appearance.

Methods

Prediction Modeling

Feature-based prediction modeling for radiology interpretation is described in detail elsewhere¹⁵. Briefly, prediction model development includes feature identification, model selection, model derivation, followed by model validation. First, potentially relevant imaging features should be identified. In general, such imaging features may be dichotomous, non-dichotomous discrete, or continuous in nature. How any particular feature is treated may depend on the nature of the finding and the nature of the clinical question. For example, in the case of MRI features of VCFs, while vertebral body bone marrow edema might conceivably be graded on a continuous scale, such observations may be less reproducible compared to a discretization of this finding, with a dichotomous treatment (i.e., bone marrow edema either present or not) likely offering the greatest reproducibility. Given this, and because the clinical question with regard to VCFs is itself also dichotomous (i.e., benign or malignant cause of fracture), dichotomous treatment of variables was preferred in this work.

The nature of the imaging features identified also impacts the selection of a modeling method. Several methods have been used to construct prediction models, including logistic regression, neural networks, recursive partitioning, Bayesian networks, and case-based reasoning^14,15. While certain of these options (e.g., neural networks, recursive partitioning) may be more well suited for discerning possible non-linearity in the relationships between variables, there may be an associated computational penalty and/or loss of transparency as to the meaning of the derived model. Logistic regression is a relatively simple method appropriate for modeling dichotomous outcomes and, given the dichotomous nature of the VCF outcome noted above, logistic regression was selected as the modeling method for this application.

After candidate features have been identified and logistic regression selected as the model of choice, individual features are tested for diagnostic significance using univariate analysis. Correlation among features will in general cause certain features to be deemed insignificant. That is, if multiple features are found to be correlated with one another, then they are not independent predictors of the given outcome and would not be expected to contribute separately to the model¹⁵. Once the final set of significant features has been determined, logistic regression is performed, leading to a set of weighting factors which allow for a closed-form calculation of the predicted outcome probability.

After model derivation is completed, model validation may be considered. This validation may include both narrow forms, whereby the derived model is tested against cases which are similar to those in the training set, and broad forms, in which the model is applied across diverse clinical settings, patient populations, and institutions.

Web-Based Implementation

Platform-independent, cross-browser, web-based implementation of feature-based radiology prediction rules may be achieved using a combination of hypertext markup language (HTML) and JavaScript. In this work, HTML is used to provide a feature checklist, where each feature is accompanied by an annotated illustration for review. JavaScript is used to implement the probability calculation at the core of logistic regression modeling, which combines the features of a given case using the weighting factors described above. JavaScript is also used to generate a standardized report text based on the feature checklist input, describing pertinent positives and negatives in prose form. The block diagram in Figure 1 illustrates the functional organization of the system.

Fig 1.

Feature-based decision support systems for radiologic diagnosis include two primary components: a set of relevant imaging findings and a prediction model to aggregate these findings. Relevant findings in a given clinical case are summarized using a feature checklist. A gallery of reference images provides guidance in evaluating individual features. These features are used as input to a prediction model, which may be based on any of several modeling methods including logistic regression (LR), neural networks (NN), recursive partitioning (RP), Bayesian networks (B), and case-based reasoning (CR). Model parameters are derived using a set of training cases. The model may then be used to calculate predictions in clinical cases (e.g., the probability of a given outcome). In addition, the structured nature of the feature checklist lends itself to automatic generation of standardized report text.

Results

Using a combination of peer-reviewed literature, as well as other sources consisting of book chapters and local experts (i.e., practicing academic neuroradiologists and musculoskeletal radiologists), a total of 31 candidate MRI features were identified as possible contributors to a model for distinguishing between benign and malignant VCFs. Twenty-eight of these 31 features were treated as dichotomous. For the remaining three candidate features (i.e., pedicle involvement, enhancement pattern, and involvement of multiple vertebral levels), the available literature and expert opinion indicated that non-dichotomous discrete treatment would be most appropriate. For example, in the case of enhancement pattern, there is evidence that heterogeneous enhancement, as opposed to homogeneous appearance on post-contrast imaging, is suggestive of malignant VCF^26,27. As a result, enhancement pattern was treated in this work as a non-dichotomous discrete variable with several possible values: heterogeneous, homogeneous, non-enhancing, or unknown with no contrast given.

The 31 candidate features were assessed in a series of 128 pathologically proven cases of benign and malignant VCF by three radiologists who were blinded to the diagnosis, using a checklist-based questionnaire. The diagnostic value of each feature was assessed using univariate analysis, showing nine of the 31 candidate features to be significant. Using these nine significant features, logistic regression was performed, leading to a series of weighting factors.

The web-based implementation of this prediction model for VCF analysis is available at http://bricweb.partners.org/vcf. The feature checklist of the top-level display (Fig. 2) consists of checkboxes for dichotomous features and pop-up menus for non-dichotomous discrete features. While only a subset of the initial 31 candidate features was deemed to be diagnostically significant, all 31 features are included in the checklist. Annotated illustrations for each feature may be browsed using a gallery of thumbnail images (Fig. 3), accessed with the “image gallery” link towards the top of the main page, or illustrations for each individual feature may be displayed using the question mark icons adjacent to each item in the checklist (Fig. 4). Each illustration has an associated caption describing the findings which constitute the given feature.

Fig 2.

The primary screen of the vertebral compression fracture decision support website presents a feature checklist to the user. The majority of these features are dichotomous in nature, shown as checkboxes. A few are non-dichotomous discrete variables, shown as pop-up menus. If a particular feature is unknown to the user, clicking the adjacent question mark will display an annotated illustration demonstrating that feature (Fig. 4).

Fig 3.

MRI features of vertebral compression fractures are illustrated using a series of images. These may be browsed in a gallery format, shown here, accessed using the “image gallery” link toward the top of the main page (Fig. 2). Each thumbnail in this gallery is labeled with a feature description. Clicking a particular thumbnail leads to a larger, annotated image with text-based description (Fig. 4).

Fig 4.

A detailed, annotated image or set of images is available for each of the MRI features listed in the checklist of the main page (Fig. 2). A combination of image marks and text-based explanations summarize the findings which constitute a given feature, promoting a uniform understanding of these features and providing a learning resource for trainees.

When the checklist has been completed, clicking the “submit” button towards the bottom of the main page triggers the JavaScript-based prediction model probability calculation, as well as the construction of a standardized report text based on a reporting template (Fig. 5). The prediction result is shown as a probability of malignancy given the constellation of MRI features present in the case. The standardized report text is presented for possible incorporation into the user’s reporting system, such as by a cut-and-paste operation when the reporting software is running on the same machine as the user’s web browser.

Fig 5.

Once the feature checklist has been completed, clicking the “submit” button towards the bottom of the main page triggers the prediction model probability calculation and template-based report text generation, both shown below the checklist items. These results are displayed respectively as a probability of malignancy and as a block of text available for cut-and-paste incorporation into the user’s reporting system.

Discussion

The combination of evidence-based practice and clinical decision support systems promises to substantially improve patient care²⁸. Image interpretation guidance, feature-based prediction modeling, and standardized reporting may all contribute to this goal in the realm of radiology diagnostic decision support. For interpretation guidance, feature checklists serve to formalize and standardize the basis for reading an imaging examination. Accompanying image galleries, such as the one shown here, help to promote a uniform understanding of the relevant imaging findings. These illustrative images also serve as a learning resource, providing detailed yet easily accessible information which may be useful to trainees at the reading station. This type of feature-based reference image organization is based in part on prior work in image-oriented expert systems for decision support²⁹. Furthermore, while the reference images included in the VCF analysis tool focus on pathological findings, anatomical illustrations are also an important area where these tools have the potential to assist learners.

Inclusion of all candidate features in the input checklist, rather than only the nine features significant in the prediction model, allows for a more complete description of the lesion present in the case, and inclusion of all features may still promote overall characterization in the report text. In a robust, broadly validated model, all features will not be needed for sufficient discrimination and could be omitted in future versions to make the system more parsimonious and probably more acceptable to the user. Only limited, narrow validation of the model presented here has been performed, and broader testing would be required before such a model could be deemed appropriate for routine clinical use.

The objectives of evidence-based decision support also integrate well with quantitative research methods, such as the prediction modeling technique used here. The VCF analysis tool provides an example of how the complexity of prediction modeling techniques may be encapsulated within a straightforward web-based interface designed to complement the radiologist’s clinical workflow. However, the basis of EBM requires practitioners to be able to evaluate the validity of specific research results, and users of evidence-based decision support tools must engage in continual critical appraisal of the research underlying any such system. At the same time, use of such systems over time may also contribute to the validity of the models used. As pathological proof is obtained in unknown cases, these could be used to augment the initial training set (consisting of 128 cases in this system, as described above), with cumulative refinement of the model.

Decision support systems should provide actionable results³⁰, and another area for potential improvement of the VCF analysis tool relates to the interpretation of the probabilities produced by the prediction model. While results characterized by very high and very low probability are relatively easy to apply clinically, cases of intermediate probabilities may be more difficult for the radiologist to use in providing actionable recommendations. Management guidelines for cases of intermediate probability, such as recommendations for short-term follow-up imaging, suggested follow-up intervals, when to consider biopsy, and so on, may be helpful. Development of such guidelines would depend on further research and could be added to the application in order to provide a more comprehensive decision support tool. However, such systems remain tools to support rather than replace trained radiologists, who need to integrate imaging information with all available clinical, laboratory, pathological, and other data in clinical decision making³¹.

Future directions for this work include developing similar systems for other common imaging domains, such as routine spine MRI where there is documented substantial variability of interpretation for intervertebral disk and degenerative findings^32,33, as well as for emerging domains where many radiologists may not have much experience or knowledge, such as MR neurography. In addition, integration with knowledge management systems³⁴ may serve to advance the educational benefits of these tools. The broad, long-term goal is to create a generalized model and application framework for image interpretation decision support systems that could be a platform for a variety of interpretation tasks.

Conclusion

The practice of evidence-based medicine within radiology is growing, and decision support tools provide an important mechanism by which to facilitate further advancement of evidence-based radiology, with a goal of formalizing and standardizing image interpretation and results communication. The web-based tool presented here for distinguishing benign from malignant vertebral compression fractures by MRI appearance integrates image interpretation guidance, feature-based prediction modeling, and structured reporting.