Evaluating Completeness of a Radiology Glossary Using Iterative Refinement

Abstract

A lay-language glossary of radiology, built to help patients better understand the content of their radiology reports, has been analyzed for its coverage and readability, but not for its completeness. We present an iterative method to sample radiology reports, identify “missing” terms, and measure the glossary’s completeness. We hypothesized that the refinement process would reduce the number of missing terms to fewer than 1 per report. A random sample of 1000 radiology reports from a large US academic health system was divided into 10 cohorts of 100 reports each. Each cohort was reviewed in sequence by two investigators to identify terms (single words and multi-word phrases) absent from the glossary. Terms marked as new were added to the glossary and hence was shown as matched in subsequent cohorts. This HIPAA-compliant study was IRB-approved; informed consent was waived. The refinement process added a mean of 288.0 new terms per 100 reports in the first 5 cohorts vs. a mean of 66.0 new terms per 100 reports in the last 5 cohorts; the difference was statistically significant (p < .01). After reviewing 500 reports, the review process found fewer than 1 new term per report in each of 500 subsequent reports. The findings suggest that 500 to 1000 reports is adequate to test the completeness of a glossary, and that the glossary after iterative refinement achieved a high level of completeness to cover the vocabulary of radiology reports.

Introduction

Patient- and family-centered care is focused on empowering and enabling patients and their family members to understand and participate in healthcare decisions [1, 2]. Electronic health record (EHR) systems play an increasingly important role as growing numbers of patients access their medical records through EHR patient portals [3, 4]. In a survey, 88% of patients indicated a strong interest to view their radiology reports online [5]. A cross-sectional study of more than 100,000 patients found that more than half of those who had access to radiology reports viewed them online [6].

The language of radiology reports poses a barrier for patients. It has been recommended that communications to patients be written at or below the 4th to 6th grade reading level [7, 8]. Radiology reports generally require at least 10th grade reading ability [9, 10]. The Patient-Oriented Radiology Reporter (PORTER) system has been proposed as a potential solution: it applies a glossary of medical terms to annotate online radiology reports with lay-language definitions and illustrations [11].

PORTER’s glossary has been shown to provide better coverage of terms in radiology reports and more readable definitions than other online resources [12]. In a study of 10,000 radiology reports, PORTER’s glossary covered significantly more of the vocabulary of radiology reports than MedlinePlus, an online health information resource from by the US National Library of Medicine. PORTER’s coverage was similar to that of RadLex, the radiology vocabulary developed by the RSNA; however, most RadLex terms lacked definitions, and those definitions it had were not well suited for lay readers.

The glossary, however, has not been evaluated for completeness. The current study sought to identify and tally “missing” terms: that is, those words and phrases that appeared in radiology reports but were absent from the glossary. We explored an iterative approach to identify missing terms and add them to the glossary. We tested the hypothesis that such an approach could reduce the average number of missing terms to less than 1 per report.

Materials and Methods

This HIPAA-compliant study was approved by the IRB; informed consent was waived. The PORTER glossary contained 3724 concepts with their lexical variants (e.g., plurals, adjectival forms, abbreviations) to comprise a total of 10,729 terms. We sampled 1000 reports at random from among 1 month of reports of a large, multi-site radiology department. The reports had been authored by more than 180 attending radiologists and radiology trainees at 5 hospitals and various outpatient facilities in both academic and community practice environments. We divided the reports into 10 cohorts of 100 reports each. Each cohort was reviewed in sequence to identify terms (single words and multi-word phrases) that appeared in the reports, but not in the glossary.

To aid manual review, terms that matched existing glossary entries were displayed as underlined. The algorithm displayed the longest matching string. An undergraduate premedical student and an experienced, board-certified radiologist reviewed the reports and edited the file to highlight new terms in the reports to add to the glossary (Fig. 1). Discrepancies were resolved by consensus. New terms included single words and multiword phrases. As shown in Fig. 1, the reviewers highlighted the term “sternum” as one not included previously in the glossary. New terms included completely new phrases and those where one or more words was already present in the glossary. For example, the new term “osseous metastatic disease” concatenated existing terms “osseous” (an adjectival form of the concept “bone”) and “metastatic disease”; the new term was defined as a form of the existing term “bone metastasis.”

Fig. 1.

An example of report text presented for review. Existing glossary terms are underlined. The reviewers have designated new terms by highlighting them in yellow

Each cohort of 100 reports was output as an HTML file in which matched terms were indicated by underlining. The reviewers used Microsoft Word to highlight new terms and saved each highlighted text file in HTML using Word’s “Web Page, Filtered” option. An automated script extracted and tallied the underlined (matched) and highlighted (new) terms from each HTML file and added the new terms to the glossary. The new terms were shown as matched in subsequent cohorts.

Results

The sample included reports of 456 radiographic exams, 201 CT exams, 127 MRI exams, 96 ultrasound exams, 44 mammograms, as well as PET, PET-CT, nuclear medicine, fluoroscopy, and dual-energy x-ray absorptiometry (DEXA) exams. Cohorts 1–5 contained a mean (± standard deviation) of 1172.4 ± 162.4 matched terms and 288.0 ± 185.5 new terms per 100 reports. Cohorts 6–10 contained 1328.0 ± 56.8 matched terms and 66 ± 20.7 new terms per 100 reports. The difference in the number of new terms was statistically significant (p < .01). In total, 1770 terms were added to the vocabulary (Fig. 2).

Fig. 2.

The number of matched and new terms in each cohort of radiology reports

Discussion

Overall, after reviewing and adding “missing” terms to the glossary from the first 500 reports, an average of less than 1 new term per report was identified in each of 500 subsequent reports. This finding suggests that even with a large vocabulary, such as with the 10,729 terms in the PORTER glossary, there is still a need to apply this type of test to determine the glossary’s completeness. Further, these findings show that iterative refinement of the glossary with a training set of 500 to 1000 reports sufficed to cover the vocabulary of radiology reports, and that the resulting glossary provided highly complete coverage of the content of radiology reports.

This study had several limitations. The number of reports was limited to 1000; this number was chosen for practicality, but the results suggest that they sufficed to demonstrate a high degree of completeness. All of the reports were from a single institution; however, the large number of radiologists likely provided sufficient diversity. Reports were analyzed by only two reviewers with highly diverse levels of experience in healthcare. The iterative review process described here provides an empirically based approach to test a glossary’s completeness and to incorporate new terms, and may be useful to assure coverage of subspecialty radiology reports, such as in breast imaging, and to accommodate differences in vocabulary among institutions.

Patient portals—which allow patients to access their healthcare information—have become increasingly widespread, and have driven patients’ interest to view their radiology results [13–15]. As Lee et al. have observed,

Many potential benefits may result from patients reviewing their radiology reports, including improvements in patients’ own understanding of their health, promotion of shared decision making and patient-physician communication, and, ultimately, improvements in patient outcomes. However, there may also be negative consequences, including confusion and anxiety among patients and longer patient-physician interactions. [16]

We currently are integrating PORTER into our health system’s EHR patient portal. When a patient views a radiology report through the portal and selects a link to PORTER, the system displays the radiology report with glossary terms underlined. When the user’s mouse hovers over one of the underlined terms, a “tooltip” balloon appears with the term and its lay-language definition [11]. Preliminary evaluation of a pilot version of PORTER yielded favorable results [17].

The complexity of radiology reports has engendered experiments to simplify report content [18]. Online educational resources offer an opportunity to educate patients and improve health literacy [19]. Such approaches are worth exploring to improve the visibility of radiologists to patients and referring providers [20]. All of these approaches have the potential to improve communication between radiologists and their patients. By helping patients better understand their radiology results, radiologists have an opportunity to improve health literacy and patient outcomes.

Conclusion

An iterative refinement approach identified terms missing from the glossary; review of 500 to 1000 reports was sufficient for this purpose based on the large number of terms already in the glossary. Moreover, the approach demonstrated the glossary to be highly complete, with an average of fewer than 1 missing term per report.