A Presentation System for Just-in-time Learning in Radiology

Abstract

There is growing interest in bringing medical educational materials to the point of care. We sought to develop a system for just-in-time learning in radiology. A database of 34 learning modules was derived from previously published journal articles. Learning objectives were specified for each module, and multiple-choice test items were created. A web-based system—called TEMPO—was developed to allow radiologists to select and view the learning modules. Web services were used to exchange clinical context information between TEMPO and the simulated radiology work station. Preliminary evaluation was conducted using the System Usability Scale (SUS) questionnaire. TEMPO identified learning modules that were relevant to the age, sex, imaging modality, and body part or organ system of the patient being viewed by the radiologist on the simulated clinical work station. Users expressed a high degree of satisfaction with the system’s design and user interface. TEMPO enables just-in-time learning in radiology, and can be extended to create a fully functional learning management system for point-of-care learning in radiology.

Introduction

The goal of “just-in-time” learning is to match educational resources with an individual’s immediate needs. In a clinical radiology practice, the widespread use of information technologies makes just-in-time learning a promising approach for continuing medical education (CME). Learning and self-assessment can be embedded into the radiologist’s daily workflow through links to clinical systems such as the radiology information system (RIS) and picture archiving and communication system (PACS).^1,2 Learning materials can be made available on demand at the point of care by linking them to the imaging procedure being evaluated, the body parts or organ systems of interest, and the radiologist’s educational needs, preferences, and level of expertise.

This report describes the development of the presentation component of TEMPO, a system for just-in-time learning in radiology. TEMPO displays educational modules and elicits the user’s responses to the modules’ self-assessment questions. The system uses information about the radiologist’s clinical context, such as the imaging procedure being viewed, to select appropriate learning modules for viewing.

Materials and Methods

Database of Learning Modules

The Human Research Review Committee of the Medical College of Wisconsin approved the research plan. We selected 74 articles from the American Journal of Roentgenology (AJR) related to imaging of the chest and abdomen. The journal’s publisher, the American Roentgen Ray Society (ARRS), granted permission to the investigators to adapt and incorporate these articles into learning modules.

The articles’ textual content, figures, tables, and references were retrieved from the on-line HTML version of the journal (www.ajronline.org) by using a set of software tools developed by the authors. We created a relational database for the articles and learning modules, and populated the database with information from the selected articles. Articles were segmented into sections by using the section headings as published in the articles. The database schema is illustrated in Figure 1. The database contained the 74 articles organized into 400 sections, with a total of 1,594 figures, 26 tables, and 1,422 references. The PubMed identifier (PMID) served as each article’s unique identifier.

Fig 1.

Entity-relationship diagram of TEMPO’s database of learning modules and journal articles. Each module comprises one or more sections of a single article.

We indexed each module by topic, imaging modality, and corresponding journal article’s Medical Subject Heading (MeSH) indexing codes. Topics, such as “Gastrointestinal Radiology” and “Computed Tomography,” were adapted from those used by the Radiological Society of North America to label published materials (Table 1). We created a set of modality codes (Table 2) to specify the imaging modalities that were relevant to each learning module. MeSH codes were obtained from the Entrez PubMed database by using the U.S. National Library of Medicine’s eQuery and eFetch utilities. An automated service was created to perform Internet-mediated queries of the database to obtain Medline information for each selected article using its PMID.

Table 1.

Topic codes used to index the learning modules

Code	Description
BR	Breast Imaging and Intervention
CA	Cardiac Radiology
CH	Chest Radiology
CT	Computed Tomography
ER	Emergency Radiology
GI	Gastrointestinal Radiology
GN	General Radiology
GU	Genitourinary Radiology
HN	Head and Neck Radiology
MK	Musculoskeletal Radiology
MO	Mammography
MR	Magnetic Resonance Imaging
NM	Nuclear Medicine
NR	Neuroradiology
OB	Obstetric/Gynecologic Radiology
PD	Pediatric Radiology
US	Ultrasound
VI	Vascular and Interventional Radiology

Table 2.

Modality codes used to index the learning modules

Code	Description
Angio	Angiography
CT	Computed Tomography
CTA	CT Angiography
MRI	Magnetic Resonance Imaging
NM	Nuclear Medicine
PET	Positron Emission Tomography
US	Ultrasound
Veno	Venography
XR	Radiography

From the set of articles, we constructed 34 learning modules. Each module was composed of several “learning objects,” such as text blocks, radiological images, drawings, and tables. A custom interface was created to allow content designers to easily enter and modify the learning modules. The authors augmented the articles by creating learning objectives and 34 multiple-choice test items. A simple, XML-based mark-up language was developed to allow authors to paginate the content and to insert the multiple-choice items into the module. The mark-up language provided flexibility so that a test item could appear anywhere within a learning module. This modularity of content also allows a test item to appear in more than one learning module, if so desired.

In crafting the modules, we sought to make them cohesive but brief. Our target, where possible, was 12 min projected completion time (0.20 CME credits) or less. Modules were assigned completion times ranging from 6 min (0.10 CME credits) to 30 min (0.50 CME credits) based on the number of words and illustrations. In several cases, one relatively large journal article served as the basis for three or more learning modules: each module comprised different sections of the article. Published CME materials in RadioGraphics and the New England Journal of Medicine—approved for 1 hour of Category 1 CME credit—were used as a baseline to estimate appropriate CME credit. No advertisements or commercial sponsorship was sought or included. The learning modules were neither reviewed nor approved for CME.

Simulated Radiology Work Station

We developed a panel of 12 clinical cases related to computed tomography (CT) and ultrasound imaging of the chest and abdomen. Personally identifiable information was removed; the cases were identified only by an accession number. Information for each case included the patient’s age, sex, imaging procedure, brief clinical history, and a fictitious patient name. Data were encoded by using CPT-4 codes for imaging procedures and ICD-9-CM codes for clinical history.

We created a web-based radiology work station to serve as a simulated clinical environment. Individual log-in is required; this information allows tracking of the imaging studies viewed by each user. After a user logs in, the work station presents a “worklist” of imaging studies to be viewed (Fig 2). Upon selecting an imaging study, the user is presented with four representative images from the study (Fig 3). Images can be viewed at full size by clicking on one of the displayed images. A navigation bar near the top of the work station display window allows users to log out, return to the worklist, or activate the just-in-time learning functions.

Fig 2.

Simulated radiology workstation, realized as a web-based display program. The worklist shows cases to be viewed. All patient names are fictitious.

Fig 3.

The radiology work station displays images of a 60-year-old man who presented with sudden back pain, radiating to the lower extremities. CT images show a type B aortic dissection. Radiologists can view an image at full size in a pop-up window by clicking on one of the images on the workstation; a full-size image is shown in the pop-up window in the bottom right of the figure. The “TEMPO Learning” hyperlink activates the just-in-time learning system.

Learning Presentation System

We developed a specialized web-based system, called TEMPO, to present learning modules. Users activated TEMPO by selecting the “TEMPO Learning” hyperlink from the radiology work station display. Metadata that described the radiologist’s “clinical context” were transmitted from the clinical work station to the TEMPO learning management system as form data by using Web services. The metadata included the user’s identification number, the patient’s age, sex, imaging procedure, and clinical indications for examination. Information about relevant organ systems and imaging modalities was mapped from the imaging procedure’s CPT-4 code.

A retrieval routine was developed that incorporated a scoring system by which learning modules were ranked by relevance to the age and sex of the patient and the imaging modality, organ system, and clinical indications of the study under review. As shown in Fig 4, TEMPO lists the six most relevant learning modules. Modules that the user has already completed are excluded from the list. The user can view the entire list of learning modules—including those already completed—by selecting the hyperlink at the bottom of the screen.

Fig 4.

Learning module selection screen. TEMPO lists the six most pertinent learning modules in descending order by relevance to the case being viewed on the rediology workstation, based on age, sex, imaging modality, body part (or organ system), and clinical indications. Modules that the user has already completed are excluded from the list.

After the user has selected a module, TEMPO presents a “home page” that describes each module, its learning objectives, source, and CME credits (Fig 5). The user can then select that module and navigate through its content. The TEMPO software manages the presentation of information in screen-sized pages, and allows users to view the figures, tables, and references associated with each learning module. The system automatically renumbers the figures, tables, and references so that their order matches that in the module, as a module may contain only a subset of the content of the article on which it was based.

Fig 5.

Start page for each learning module includes the learning objectives, number of CME credits, and reference to the journal article on which the module is based.

TEMPO presents multiple-choice items within the module (Fig 6). A pseudorandomization function was created to automatically shuffle the order of responses for each item. Although the order of questions differs by user and item, the choices are presented in a consistent order for each specific user and item. Once the user has selected the radio button for a particular choice, TEMPO records the user’s response and provides feedback (Fig 7). TEMPO tracks the user’s interactions with the display system, including login, logout, selection of an imaging study on the radiology work station, selection of a learning module, and viewing of learning module pages. The system maintains a record of the learning modules completed by each user and of the user’s responses to multiple-choice items (Fig 8).

Fig 6.

First page of learning module on aortic dissection. A multiple-choice test item is presented. The user can view the remainder of the module without answering the question, but will not be able to view the answers until he or she does so. By default, the first figure in the module appears in the sidebar on the left. (Image reprinted with permission from the American Journal of Roentgenology.)

Fig 7.

After user chooses a response to multiple-choice item, TEMPO displays the correct response (indicated by the shaded bar), the user’s choice (the filled radio button), and an explanation for each response. (Image reprinted with permission from the American Journal of Roentgenology.)

Fig 8.

Page 2 of the “Aortic Dissection” learning module. When the user clicks on the hyperlink for a figure or a reference, it appears in the sidebar on the left. (Image reprinted with permission from the American Journal of Roentgenology.)

We developed a scoring function to rank relevant learning modules based on the patient’s age and sex, the imaging modality, and organ system of the examination being viewed on the radiology work station. When a user selected the “TEMPO Learning” feature from the radiology work station, TEMPO listed learning modules that the user had not yet completed. Modules were listed in descending order by predicted relevance to the study being viewed on the radiology work station. The user also could view a list of all modules, including those already completed.

The radiology work station and TEMPO system were implemented by using PHP, an open-source scripting language that is well suited for Web development. The learning modules, self-assessment test questions, answers, and explanation were displayed as Web pages, which we generated in real-time from data in the system’s MySQL database. The presentation software underwent an “iterative design” approach, in which we solicited feedback from a small group of radiologists and modified the software accordingly.

Preliminary Evaluation

Radiology residents, fellows, and faculty members at the Medical College of Wisconsin (Milwaukee, WI, USA) were invited to participate in the evaluation of the system. Individual log-on IDs were created for each physician. After a brief overview of TEMPO’s concepts and functionality, they had an opportunity to explore the system. We employed the System Usability Scale (SUS) questionnaire,³ which asks 10 Likert-scaled items to measure overall user satisfaction (Table 3). SUS scores range from 0 (very little satisfaction) to 100 (very high satisfaction). Among a variety of well-known measures of user satisfaction, SUS has yielded the most reliable results across sample sizes⁴; sample sizes of at least 14 participants are needed to obtain reasonably reliable results. Average SUS scores are usually between 65 and 70.⁴

Table 3.

System Usability Scale (SUS)—a Ten-item questionnaire to assess user satisfaction

No.	Item
1	I think I would like to use the web site frequently
2	I found the web site unnecessarily complex
3	I thought the web site was easy to use
4	I would likely use this system to meet CME needs
5	I found the various functions in this web site were well integrated
6	I thought there was too much inconsistency in this web site
7	I would imagine that most people would learn to use this web site very quickly
8	I found the web site very cumbersome to use
9	I felt very confident using the web site
10	I need to learn a lot about this web site before I could effectively use it

Results

Eighteen physicians completed the evaluation of the system. Respondents’ SUS scores ranged from 57.5 to 85.0 (mean ± SD, 70.8 ± 8.0). As shown in Table 4, 16 respondents (89%) indicated agreement that the learning modules displayed by TEMPO were related to the case being reviewed (95% confidence interval [CI₉₅], 72–100%). Seventeen participants believed the length of the learning modules to be optimal (94%; CI₉₅, 81–100%). Comments included an interest in making the connection from the work station to TEMPO more explicit. The users requested a “close window” button on the start page and CME credit page. Several users indicated that they would prefer learning modules linked to a predefined curriculum. Overall, the users stated that they found TEMPO easy to use, and that the learning modules were on relevant topics that would be good for CME.

Table 4.

Responses to questions on the learning modules

Statement	SD	D	N	A	SA
I felt the articles were relevant to the case presented	0	0	2	9	7
I thought the length of time involved per article was optimal	0	1	0	13	4

Each item was answered using a Likert-scale response: SD = strongly disagree, D = disagree, N = neutral, A = agree, SA = strongly agree.

Discussion

Physicians must continue learning throughout their careers. There is growing concern that current approaches to medical education—particularly those for CME to support lifelong learning—are in need of change.⁵ Information technology has created new opportunities for CME. Internet-based programs have been shown to impart knowledge as effectively as traditional CME activities. ⁶ Because Internet-based CME programs support self-directed learning, they more closely meet physicians’ needs and motivations.⁷ Medical decision making is aided by targeted knowledge brought to the point of care.⁸ This “just-in-time” learning helps promote evidence-based radiology, where scientific knowledge is linked to radiology practice.⁹

Educational theory and empirical evidence support the use of “situated learning,” in which the learning experience is imparted where the knowledge is used.^10,11 Traditional approaches to CME may have little or no impact on the quality of medical care.¹² Many educational activities are removed in time, place, or context from the environment in which they can be applied with positive effect.¹³ A recent survey has found strong interest in the concept of just-in-time learning among radiologists; respondents indicated a preference for learning modules of 5–15 min. ¹⁴ Just-in-time learning may help improve the effectiveness of CME while allowing radiologists to maintain their clinical efficiency. TEMPO is designed to provide brief, context-sensitive, highly targeted learning objects that can be integrated into a radiologist’s daily workflow.

Future Work

The presentation system described here comprises only one component of the planned TEMPO learning management system. The system’s architecture includes components for content management, presentation, assessment, and feedback. The function of the content manager is to store and organize the system’s educational content. The content is organized by subject area, media type, and level of difficulty. The content manager also helps the developer specify a curriculum that lends structure to the available and planned content. It helps the developer define the curricular content, revise curricular goals as necessary, and track changes to the curriculum for future reference. It allows the learner to specify, within prescribed bounds, which educational materials are relevant and of interest, and to monitor educational progress. Learners can select areas of emphasis. The content manager provides the mechanism for storage and retrieval of the educational content within the system. As revisions to the content become necessary, this module applies those changes and maintains a history of modifications.

The assessment component is designed to score the learner’s responses to test questions for self-assessment and for subsequent analysis of the test questions themselves. The module provides self-assessment information to the radiologists. It scores exam questions and computes normalized scores. The feedback component allows the learner to evaluate the quality of the educational materials. It gathers and summarizes radiologists’ opinions to produce reports for users, system developers, and the authors of educational modules.

We plan to evaluate TEMPO to measure user satisfaction, educational benefits, and the impact on radiologists’ workflow. TEMPO tracks each user’s modules completed, items completed, and pages viewed. This information also will allow TEMPO to report a user’s completed CME activity on request. It will allow us to link learning resources with the user’s defined curriculum, areas of expertise, and preferred learning style.