Improving Automating Quality Control in Radiology: Leveraging Large Language Models to Extract Correlative Findings in Radiology and Operative Reports

Abstract

Radiology Imaging plays a pivotal role in medical diagnostics, providing clinicians with insights into patient health and guiding the next steps in treatment. The true value of a radiological image lies in the accuracy of its accompanying report. To ensure the reliability of these reports, they are often cross-referenced with operative findings. The conventional method of manually comparing radiology and operative reports is labor-intensive and demands specialized knowledge. This study explores the potential of a Large Language Model (LLM) to simplify the radiology evaluation process by automatically extracting pertinent details from these reports, focusing especially on the shoulder’s primary anatomical structures. A fine-tuned LLM identifies mentions of the supraspinatus tendon, infraspinatus tendon, subscapularis tendon, biceps tendon, and glenoid labrum in lengthy radiology and operative documents. Initial findings emphasize the model’s capability to pinpoint relevant data, suggesting a transformative approach to the typical evaluation methods in radiology.

1. Introduction

Shoulder MRI is a commonly utilized imaging modality for the evaluation of rotator cuff pathologies¹. Numerous studies have demonstrated the diagnostic utility of shoulder MRI in identifying a variety of conditions^2–4. Some of these studies have demonstrated varying diagnostic accuracy, sensitivity, and specificity depending on different underlying patient conditions. While these studies are useful in evaluating different diagnostic benchmarks, they are only able to evaluate a small sample of exams that are completed annually.

Correlation of the operative findings with the imaging report is a key quality control mechanism in radiology. Of particular importance in this process is a discrepancy between the radiology report and the operative report. Differences in these findings demonstrate value both in their educational opportunities as well as quality improvement. Currently, however, this involves the radiologist looking up the operative report and reviewing the operative findings. This process is tedious and time-consuming, currently requiring manual tracking of completed reports and constant searching within the medical record to see if surgery took place. Additionally, this correlative process only occurs on a subset of cases, as opposed to a broader selection of completed exams. Currently, there is no automated method for performing this in clinical practice.

We sought to develop a transformer-based NLP model to automatically extract relevant portions of the radiology and operative notes to more rapidly highlight discrepancies within the two reports. An automated system would provide a solution that could evaluate a larger number of cases, providing a greater educational and quality assurance opportunity. Ultimately this may lead to enhanced imaging protocols, more accurate diagnoses, and overall improved patient care.

Recently, Large Language Models (LLMs) such as GPT-4⁵ LLaMa2⁶, and Mistral⁷, etc. have shown incredible capabilities in various natural language tasks including summarization, sentiment analysis, language translation, text generation, and information retrieval^8,9. Training on extensive corpora empowers LLMs with foundational language understanding, while fine-tuning enables these pre-trained models to specialize further in particular tasks or domains^10,11. Fine-tuning, even with limited data, is especially important for the use of LLMs in a clinical context because medical data is most often confidential, and thus under-represented in the large public corpora that are used to train foundational LLMs¹². Fine-tuning a language model for specialized clinical annotation tasks requires example annotations generated by human experts¹³. In this study, we employ the Parameter-Efficient Fine-Tuning (PEFT) technique¹⁴ to adapt a Falcon LLM for a specific clinical application. This involves fine-tuning the model to identify pertinent details in radiology or operative reports when queried about the structural integrity of the shoulder.

2. Methods

This section describes the methodology employed in our study, including the data collection and preparation procedures, the fine-tuning of the Falcon Large Language Model (LLM), and the evaluation process.

2.1 Data

All data for this study were sourced from the Radiology Information System (RIS) system of Henry Ford Hospital in Detroit, Michigan. The dataset included operative and radiological reports of 483 patients who underwent shoulder imaging and subsequent operative procedures. The dataset was collected to enable a comparison between radiological findings and operative observations, focusing on specific anatomical structures of the shoulder. For each patient in the dataset, we had two reports:

1. Radiology: provide a detailed description of the MRI findings related to the patient’s shoulder; it provides the condition of various anatomical structures of the shoulder based on imaging results.

2. Operative: provides the operative observations made during the patient’s operative procedure; it describes the physical state of the shoulder structures, as seen by the surgeon(s).

In Table 1, we provide a summary statistics of the dataset. As seen in the Table, the patient ages spanned a wide range, (mean of 57.65 years and a standard deviation of 11.75 years), but the gender distribution was relatively balanced (52.16% female; 47.84% male). The lengths of the radiology and operative reports, both in terms of sentences and words, show distinct patterns: On average, radiology reports were 25 sentences (±6.05) while operative reports we nearly 4 times longer, averaging 89.94 sentences (±45.38).

Table 1.

Summary Statistic of the dataset. The Cosine similarity between the radiology and operative reports is calculated using a Term Frequency-Inverse Document Frequency (TF-IDF) tokenizer and a Bert-based tokenizer.

Number of patients	483
Patient age (mean±std)	57.65±11.75
Gender (%)	F	52.16
	M	47.84
Radiology report length (mean±std)	Sentence	25.00 ± 6.05
	Word	332.84 ± 81.32
Operative report length (mean±std)	Sentence	89.94 ± 45.38
	Word	1,414.00 ± 708.98
Cosine similarity	TF-IDF tokenizer	0.57 ± 0.09
	Bert-based tokenizer	0.90 ± 0.07

In this Table, the cosine similarity between the radiology report and the corresponding operative report is calculated using two different tokenizers: 1) Term Frequency-Inverse Document Frequency (TF-IDF) tokenizer and 2) Bert-based tokenizer. TF-IDF tokenizer focuses on the frequency of words in each document relative to their frequency across all documents while the Bert tokenizer is a more advanced model that understands the context of words in a sentence. A moderate similarity between the reports (TF-IDF tokenizer has a mean value of 0.57); this indicates some shared terminologies or descriptions but also highlights distinct differences in the narratives. The higher degree of Cosine similarity with Bert-based tokenizer implies that, despite their domain-specific language, the reports likely share contextual meaning.

Specialist-annotation: For both report types, a team of radiologists at the Henry Ford Hospital annotated segments of the text (contiguous sequences of tokens) that offered important insights into the integrity of five shoulder structures: (1) the supraspinatus tendon, (2) infraspinatus tendon, (3) subscapularis tendon, (4) biceps tendon, and (5) glenoid labrum. The annotated text segments were collected to facilitate fine-tuning of an LLM that generates the annotated text, given the report and the structure type; the details of the LLM model are described in subsequent sections.

In Figure 1 we show the textual embeddings of the radiology and operative reports related to the five key shoulder structures, projected into a two-dimensional space using the t-SNE technique¹⁵. A distinct observation from the plots is that points from the same report type (either radiology or operative) tend to cluster together, indicating a consistent narrative or terminology used within each report type. However, some overlap between the purple (radiology) and blue (operative) points in certain structures may suggest shared terminologies or descriptions between the two report types for those specific structures. It can be inferred that while both report types have their unique terminologies, there are instances of shared language, especially when detailing specific shoulder structures.

Figure 1.

The textual embeddings of the radiology (purple) and operative (blue) reports, based on five key shoulder structures, are projected into a two-dimensional space using t-SNE.

Preliminary inspection of the dataset reveals diverse findings across patients: radiological reports often contain detailed descriptions of the condition of the tendons and labrum, with mentions of tears, degeneration, and other anomalies. operative reports also describe the state of these structures, and in some cases, may use terminologies slightly different from those used in radiology, such as ‘complete tear’ versus ‘full-thickness tear’.

Standard tags: In addition to the annotated segments, a team of radiologists defined uniform tags for each shoulder structure to represent its condition. Taking the Supraspinatus tendon integrity as an example, the tags include intact, normal, completely intact, tendinosis, deep bursal, thickness tear, etc. These standard tags were used to evaluate the model’s output.

2.1.1 Data Preparation

To prepare the radiology and operative notes for subsequent use within the Large Language Model, we performed several preprocessing steps.

Text Normalization: The initial preprocessing step involved cleansing the texts to remove unwanted characters (such as $, •), textual artifacts (such as \n, cc: ), extra white space, and normalizing text cases.

Question-Answer Pairing: In the second preprocessing step, we structured sections of the radiology and operative reports into a series of question-answer (Q-A) pairs. This restructuring process was designed to pair a question about the integrity of the shoulder structure (e.g. “What is the integrity of the Supraspinatus tendon?”), with an answer sourced from the specialist-annotated reports (e.g. “There is a full-thickness tear of the distal supraspinatus tendon”). Training Data Creation: In the third step, we generated text samples for fine-tuning an LLM. More specifically, each text sample consisted of a report’s text followed by a question and the corresponding answer. This approach was designed to provide the model with sufficient context for generating accurate responses, given a question. To further enhance the model’s understanding of the input text’s structure, we employed specific tags such as ‘<question>’, ‘<answer>’, ‘<radiology report>’, and ‘<operative report>’. These tags served as markers to indicate different sections and types of text, thereby guiding the model in processing and responding to the information more effectively.

Table 2 provides an overview of the structured text sample for fine-tuning based on the report type and shoulder structure. In total, the dataset was converted to 4,830 ( 483 records × 2 report types × 5 structure types ) text samples used for model training, evaluation, and testing.

Table 2.

Overview of the structured text sample derived from radiological and operative reports, based on report type and specific shoulder structure. <question>, <answer>, <<radiology report>>, and <<operative report>> are the uniformed tags. {report content} is used to denote the actual content from the report, and {label content} represents the corresponding annotation.

	Report type
Structure	Radiology	Operative
Supraspinatus	<question>: <<radiology report>> {report content} what is the integrity of the Supraspinatus tendon? <answer>: {label content}	<question>: <<operative report>> {report content} what is the integrity of the Supraspinatus tendon? <answer>: {label content}
Infraspinatus	<question>: <<radiology report>> {report content} what is the integrity of the Infraspinatus tendon?<answer>: {label content}	<question>: <<operative report>> {report content} what is the integrity of the Infraspinatus tendon?<answer>: label content
Subscapularis	<question>: <<radiology report>> {report content} what is the integrity of the Subscapularis tendon? <answer>: {label content}	<question>: <<operative report>> {report content} what is the integrity of the Subscapularis tendon? <answer>: {label content}
Biceps	<question>: <<radiology report>> {report content} what is the integrity of the Biceps tendon?<answer>: {label content}	<question>: <<operative report>> {report content} what is the integrity of the Biceps | tendon?<answer>: {label content}
Labrum	<<radiology report>> {report content} what is the integrity of the Labrum?<answer>: {label content}	<question>: <<operative report>> {report content} what is the integrity of the Labrum?<answer>: {label content}

Tokenization: In the fourth step, the text samples were converted into token sequences. Given the variability in report lengths, the token sequences were uniformly managed by padding shorter sequences with a padding token “<pad>”.

Training, Validation, and Testing Set Partition: Finally, text samples were randomly divided into a training set (60%; 2890 Q-A pairs), a validation set (20%; 970 Q-A pairs), and a test set (20%; 970 Q-A pairs). The training set was used to train the model, the validation set was used to find the optimal setting of the model’s hyperparameters, and the test set was used to assess the model’s generalization capability on unseen data.

2.2. Method

2.2.1 Fine-tuning

We utilized the pre-trained LLM, Falcon,¹⁶ as our base model. Given Falcon’s extensive architecture (encompassing 7 billion parameters), traditional full-scale fine-tuning approaches were impractical on our computational equipment. To address this issue, we adopted the Parameter-Efficient Fine-Tuning (PEFT) strategy as proposed by Dettmers et al. (2022)¹⁴. Our approach involved two key steps: initially, we employed a 4-bit quantized model to downscale the precision of the model’s parameters, effectively reducing the overall size of the model. Subsequently, rather than tuning all the model’s parameters, we incorporated a sparse array of learnable low-rank adapter weights to the quantized LLM through the Low RAnk (LoRA) adaptation method, as introduced by Hu et al. (2021)¹⁷. This method, which involves freezing the original model and adjusting a concise set of trainable weights, ensures efficient fine-tuning while maintaining optimal performance.

2.2.2 Hyperparameters

Optimal performance was achieved with hyperparameters set to a rank of 64 for the update matrices (r) and a scaling factor (α) of 32 in LoRA. LoRA was applied exclusively to the query/value, dense, dense h to 4h, and dense 4h to h projections within the Transformer layers. The training was conducted over 3 epochs with a learning rate of 2e−4 and a batch size of 16. All other settings were maintained as defined in the original pre-trained model.

3. Evaluation

We employed both quantitative and qualitative methods for model evaluation: quantitatively, we used partial match accuracy and ROUGE-L metrics for assessing label accuracy and coverage, and qualitatively, we utilized specialist evaluation, where two radiologists independently rated model outputs against ground truth on a -1 to 1 scale, with Cohen’s Kappa to measure rating consistency. Additional details on the evaluation approach are described in the following subsections.

3.1. Quantitative assessment

We utilized ROUGE-L (Recall-Oriented Understudy for Gisting Evaluation using Longest Common Subsequence) as a measure of agreement between the desired answer (from clinical annotations) and the generated answers (from the LLM). ROUGE-L focuses on the longest common subsequence between two text sequences and is often used to evaluate how much of the content (in terms of structure and word sequence) in a reference text is covered by a generated text. This metric provides a detailed evaluation of how often the model’s predictions contain the desired labels.

In addition to ROUGE-L, we checked if the LLM-generated answers contained any standard tags (“intact”, “normal”, “absent”, “completely intact” etc.) that matched those generated by the radiologists. These tags were set to be compared against the ground truth to calculate metrics such as Accuracy, Precision, Recall, Specificity, and F1 Score, offering a quantitative measure of the model’s performance.

3.2. Qualitative assessment

Although transformer-based models have demonstrated impressive capabilities in diverse radiology tasks such as protocoling¹⁸, report generation and summarization¹⁹, and entity extraction²⁰; these models frequently encounter issues such as hallucinations, biases, and ethical concerns, underscoring the need for comprehensive evaluation, as highlighted by Khanmohammadi et al. 21. To address this, we conducted a comparison of the LLM-generated responses against ground truth labels through specialist evaluation. The evaluation follows a rating paradigm where the specialist’s annotation and model’s outputs are presented to two reviewers (radiologists), who evaluate the information independently. They’re tasked with rating their consensus on a scale from -1 to 1 for a subset of 100 randomly selected examples. A score of 1 signifies that the model’s output is both precise and concise, 0 means it’s accurate but not concise, and -1 denotes either incorrect or conflicting information. Cohen’s Kappa statistical test is used to measure the consistency of ratings between the two evaluators.

4. Results

4.1 Comparing generated labels with Specialist-Annotated Labels

Table 3 presents the ROUGE-L scores for each shoulder structure, broken down by report type (radiology and operative). The scores are divided into three categories: Recall, Precision, and F1. The results indicate that the model performs well with radiology reports, particularly in identifying and predicting information about the Labrum (72.14) and Biceps (F1:77.06) tendon. We can see that there is a notable drop in performance, indicating that the model struggles more with the language or structure of operative reports. This could be due to more varied terminology, less structured narratives, or other complexities inherent in operative documentation.

Table 3.

ROUGE-L scores of the model for each shoulder structure. The values indicate the recall, precision, and F1 of the longest common subsequence from the desired labels were present in the model’s predictions.

Report Type	ROUG-L	Supraspinatus	Infraspinatus	Subscapularis	Biceps	Labrum
Radiology	Recall	77.36	56.37	75.25	80.26	82.25
	Precision	73.36	54.55	69.09	72.68	76.34
	F1-score	71.33	51.41	69.63	72.14	77.06
Operative	Recall	42.18	38.71	39.29	47.13	45.33
	Precision	37.93	32.04	30.91	41.03	40.15
	F1-score	37.77	32.13	31.48	40.95	38.19

Table 4 provides a detailed evaluation of the model’s performance in predicting the integrity of various shoulder structures based on standardized tags. For radiology reports, the model demonstrated its highest accuracy and precision with the “Supraspinatus” structure, achieving scores of 0.90. In contrast, for operative reports, the accuracy and precision were relatively lower, with the “Supraspinatus” and “Biceps” structures both peaking at 0.69. The “Subscapularis” structure in operative reports, however, recorded the lowest accuracy and precision at 0.26 and 0.16 respectively. Notably, the model’s performance was generally stronger for radiology reports across all shoulder structures compared to operative reports.

Table 4.

Evaluation metrics for each shoulder structure based on radiology and operative reports. The metrics include Accuracy, Precision, Recall, and F1 Score, rounded to two decimal places.

Structure	Report Type	Accuracy	Precision	Recall	F1 Score
Supraspinatus	Radiology	0.90	0.90	0.90	0.91
	Operative	0.69	0.85	0.69	0.76
Infraspinatus	Radiology	0.62	0.47	0.62	0.52
	Operative	0.44	0.50	0.44	0.43
Subscapularis	Radiology	0.64	0.73	0.64	0.61
	Operative	0.26	0.16	0.26	0.18
Biceps	Radiology	0.69	0.70	0.69	0.68
	Operative	0.69	0.70	0.69	0.68
Labrum	Radiology	0.87	0.88	0.87	0.86
	Operative	0.54	0.67	0.54	0.57
		0.63	0.56	0.63	0.65

4.2 Human Evaluation

In our evaluation, two reviewers independently rated a subset of the model’s outputs (see section 3.2 for the approach), providing scores on a scale from -1 to 1. The distribution of these ratings, broken down by report type (radiology and operative), is visualized in Figure 2. The box plots in the figure highlight the median, quartiles, and potential outliers for each reviewer’s ratings. For both reports, the ratings from both reviewers cluster around similar medians, indicating a consensus. The level of agreement between the reviewers, as measured by Cohen’s Kappa, was κ=0.76 which indicates a high agreement between the two raters when evaluating the generated outputs.

Figure 2.

Boxplots illustrating the distribution of ratings by Reviewer 1 and Reviewer 2 for radiology (left) and operative (right) reports.

Figure 3 provides the distribution of ratings given by two independent reviewers alongside the average rating for each score category (-1, 0, 1). Both reviewers exhibit a predominant inclination towards rating 0 and 1, indicating a trend of considering the model’s output as accurate. The average rating trend closely mimics the individual reviewer distributions, reinforcing the consistency in their judgments. Notably, ratings of -1, which denote inaccurate or conflicting information, are substantially less frequent than ratings of 0 and 1, suggesting that the model’s outputs are generally aligned with expert opinion. The occurrence of neutral ratings (0) falls between the two extremes, reflecting instances where the output was accurate but not concise.

Figure 3.

Comparison of rating distributions across two reviewers and their average for each score category. The ratings are categorized as -1 (indicating inaccurate or conflicting information), 0 (indicating accurate but not concise information), and 1 (indicating both accurate and concise information). The bars represent the counts of each rating provided by the reviewers and their calculated average.

3. Discussion

We trained a large language model demonstrating the ability to extract relevant portions of radiology and operative notes for correlation. The LLM was trained using original radiology and operative notes and as such can be applied directly to clinical notes without the need for pre-processing. The model demonstrated the capability to extract relevant details regarding specific anatomic structures in both radiology and operative texts. The extracted data from these reports could then be compared to evaluate for discrepancies. Our model showed the highest performance metrics overall on radiology reports relative to operative notes across all of the anatomic structures evaluated.

Prompt feedback to radiologists of surgical findings in the context of a prior report provides both educational and quality improvement value. However, manual review of these cases is challenging for numerous reasons. The time from initial MRI examination to surgery varies by patient and there is not a consistent interval by which to check for new operative findings. While an algorithm-based solution to match radiology reports to operative notes would help with this issue (16) it does not provide a robust solution. In addition, the operative notes are lengthy in comparison to the radiology reports, as demonstrated in Table 1. In practice, the time required to locate the specific pathology and determine a level of agreement is not insignificant. Structured reporting feedback has been utilized in radiology before however, the free-text nature of operative notes proves to be challenging for a structured solution. Operative notes utilized came from numerous surgeons, each with a distinct dictation style. This variation requires a more robust, and flexible solution to be able to identify the correct portions of the text allowing for differences in writing styles. This aforementioned variance correlates with the model’s decreased performance on operative notes relative to the radiology dictations. The radiology reports were semi-structured and dictated using a standard department template. While the body of the report was still dictated by individual radiologists and residents, the headings of the report had a defined structure. This is in direct contrast to operative notes which were freely dictated by each respective surgeon.

To the best of our knowledge, our study is the first to evaluate the use of a large language model to perform automatic correlation between radiologic and surgical reports. NLP has been applied to this task before in regards to

correlating knee arthroscopy notes with MRI findings (17-18). When compared to prior studies on the knee, our model shows similar performance metrics regarding the supraspinatus tendon and glenoid labrum text extraction. Comparatively, our model shows decreased performance on the remaining elements of the shoulder.

Enabling an automatic analysis of radiology reports and operative notes for correlative analysis has numerous potential clinical implications. First, an automated system would alleviate the time required by the radiologist to manually find the salient details within radiology and surgical notes. Secondly, since the current radiology/surgical correlation process only involves a randomly selected subset of cases, an automated method would allow for more continuous quality improvement cycles. This would highlight the need for imaging protocol changes and/or increased education which would lead to rapid improvements in patient care.

Our study had numerous limitations that should be considered when evaluating the results. While the reports were generated by numerous radiologists and surgeons, it should be noted that they all originated from a single institution. The number of labeled entries during the training phase could have been more robust. However, even with a small sample size we were able to show the potential successful application of large language models given that usable results were generated, especially within the radiology reports.

4. Conclusion

Shoulder MRI is frequently employed in medical diagnostics, serving as a primary tool for assessing abnormalities and injuries related to the rotator cuff. The accuracy and comprehensiveness of radiology reports are paramount, serving as the bridge between imaging findings and clinical decision-making. In essence, the quality of a radiology report can significantly influence patient care pathways, diagnosis formulations, and treatment strategies. The robustness of these reports is ensured by cross-referencing them with operative findings. This process, though critical, is manual, tedious, and demands specialized knowledge, often limiting its frequency and scope. In this study, we explored the capabilities of modern Natural Language Processing (NLP), specifically the transformative potential of Large Language Models (LLMs), to automate this critical quality control step. Our model, trained on radiology and operative reports, demonstrated a promising ability to extract and correlate pertinent details, particularly from radiology reports. While our system exhibits the potential to reshape quality control mechanisms in radiology, it’s not without its challenges. Operative notes, with their unstructured nature and variability in dictation styles, posed a challenge for the model. Yet, the strengths of our approach lie in its scalability, adaptability, and the potential for continuous learning and improvement. With increasing data and further fine-tuning, the model’s proficiency can be enhanced to cater to a wider range of reports and clinical scenarios.

5. Ethical Approval

This research obtained approval for this retrospective study from the Institutional Review Board (IRB). It also granted a waiver of informed consent. This waiver was based on the retrospective design of the study, which involved the analysis of existing data that was anonymized and de-identified to ensure participant privacy and confidentiality.

6. Reproducibility

The code developed for this study can be accessed at the following GitHub repository: https://github.com/HAAIL/radiology-operative-correlation.