Imaging Informatics Fellowship Curriculum: a Survey to Identify Core Topics and Potential Inter-Program Areas of Collaboration

Abstract

New consumer technologies and interoperability standards have dated the first standardized curriculum for imaging informatics fellowships suggested by the Society for Computer Applications in Radiology (SCAR) in 2004 (Journal of digital imaging 17(4):244-248, 2004). Last year, analysis from this institution characterized current state fellowship graduation requirements and broad curriculum topics for the first time in over a decade (SIIM Strategic Plan 2017–2020). However, an updated “core” curriculum has not yet been developed. Using the recent current state analysis as a baseline, we aimed to perform a focused assessment and propose that this would work towards an updated consensus “core” curriculum as outlined by the Society for Imaging Informatics in Medicine (SIIM, previously SCAR) strategic plan. A secondary aim was to identify individual program strengths and weaknesses to foster inter-program collaboration. Using sub-topics from the National Imaging Informatics Curriculum (NIIC), a week-long introductory course for residents, we expanded the original 29 broad curriculum categories identified in last year’s current state analysis into 114 sub-topics. We surveyed imaging informatics fellowship directors to identify sub-topic prioritization on a 5-item Likert scale, teaching methods for each sub-topic, cross-departmental partnerships, and individual program strengths and weaknesses. Only 8% of sub-topics (10/114) received a “definitely” rating with 100% agreement while the majority of sub-topics 77% (88/114) had mixed grading defined by two or fewer “definitely” ratings. These sub-topics mapped to only 4 of the original 29 broad fellowship curriculum categories including Standards, Programming/Development/Software, Infrastructure, and PACS/RIS/Reporting. Our plan is to use consensus topics to build a “core” informatics fellowship curriculum and initiate discussion surrounding mixed grading topics. Knowledge of individual program strengths and weaknesses can be used to foster inter-program collaboration.

Background

The need for standardization among imaging informatics fellowships has been ongoing since 2004, when the Society for Computer Applications in Radiology (SCAR) affirmed that “the scope of a dedicated Radiology Imaging Informatics fellowship and the specific curriculum elements for this training have not been identified.” [1] Although in 2004, SCAR outlined a radiology informatics curriculum incorporating the four broad topics of Information Technology, Clinical Informatics, PACS Administration, and Academics, the curriculum has not been updated in over a decade. The need for an updated curriculum is ongoing and recognized. The 2017–2020 Society for Imaging Informatics in Medicine (SIIM, previously SCAR) Strategic Plan outlines education and career development as a strategic goal, specifying action items of “supporting the development of an introduction to imaging informatics curriculum relevant for clinical residents” and “convening the community of imaging informatics fellowships to develop a common curriculum.” [2] However, in recent years, the focus of informatics education has been diverted to the compact and easily accessible National Imaging Informatics Curriculum (NIIC), a week-long course sponsored by the Radiological Society of North America (RSNA) and SIIM for senior radiology residents, consisting of recorded lectures and online small group discussions [3]. This course is open to all residents, including those who may not operate in imaging informatics roles later in their careers. The NIIC is an important and substantial accomplishment in increasing basic information technology literacy among radiology residents and was developed by key informatics leaders. Yet, to date, the NIIC has not been leveraged to facilitate a long-term consensus imaging informatics fellowship curriculum for those trainees who aspire to pursue more comprehensive informatics training.

A known barrier to consensus is the rapid pace of technological advancement. This has been well-outlined in both SCAR’s 2004 proposal and a Journal of Digital Imaging (JDI) debate-style commentary concerning informatics integration in medical schools [1, 4]. SCAR originally proposed overcoming this barrier by keeping an updated curriculum available online using “the very technology that drives most educational materials into obsolescence…to keep the curriculum current.” [1] Despite this sentiment, online SCAR curriculum maintenance was unsuccessful over the subsequent 15 years. Diversion of SIIM’s attention from fellowship curricula to general resident-level informatics education likely contributed to lack of SCAR curriculum maintenance. In addition, although imaging informatics fellowships are associated with ACGME-approved radiology programs, they are not ACGME-approved themselves. SIIM, in addition, does not govern imaging informatics fellowships but lists all imaging informatics fellowships that volunteer their information to SIIM on its website [5]. Both these aspects have allowed programs great flexibility, perhaps by choice, to dictate all aspects of their curriculum and graduation requirements but may have hindered educational coordination and online curriculum maintenance. Anecdotally, imaging informatics fellowships were updating their individual curricula to keep pace with the changing informatics environment during the 15 years since SCAR curriculum publication. Fellowship programs recognized that a static document was insufficient to meet their needs and had developed a workaround. This suggests that standardization efforts were missing a critical component conferring both momentum and durability in 2004. We propose this missing critical component is inter-program collaboration.

A living document based on living collaboration addresses the challenge of rapid informatics advancement and supplies a valuable tool to both informatics educators and to students considering a career in informatics. There are a limited number of imaging informatics fellowships, and only 10 are currently listed on the SIIM website [5]. These lean numbers can be advantageously leveraged for individual fellowship programs to exchange resources and share common sessions. We propose a sequential examination consisting of four discrete steps prior to inter-program collaborative exchange: (1) a current state analysis of areas of curriculum overlap, (2) expansion of broadly identified overlapping topics into sub-topics for clarity, (3) appraisal of whether each subtopic should belong in a core curriculum, and (4) identification of individual program areas of expertise as well as challenging subject areas where group efforts may be beneficial.

Existing work has supported the necessity of the first step in this proposed 4-step sequence, indicating additional work is necessary to clarify consensus elements in fellowship training [6–8]. This first step, current state analysis of areas of curriculum overlap, has already been completed [8]. However, information surrounding the subsequent steps is unknown. As a consensus core curriculum remains a SIIM strategic goal, such information can help inform future inter-program collaboration and work towards a living document. For these reasons, we aimed to leverage the NIIC to address steps two through four and characterize overlapping sub-topics, appraise subtopic belonging in a core curriculum, and identify program areas of expertise and challenge.

Methods

This institutional review board-exempt study was performed in a Health Insurance Portability and Accountability Act–compliant manner.

Using sub-topics from the NIIC, we expanded the 29 broad curriculum categories previously identified by Vey et al. into 114 sub-topics [8]. Some of the previously identified broad curriculum categories were covered consistently while others were unique to individual programs only. For completeness, all were included in our analysis. We chose to expand Vey et al.’s categories using NIIC sub-topics instead of other existing curricula such as the American Board of Imaging Informatics (ABII) Test Content Outline (TCO) as the NIIC to TCO content mapping revealed the NIIC delineated a greater number of topics when compared with the TCO [3, 9]. In addition to all TCO topics, the NIIC also covers the topic “Algorithms of Image and Non-Image Analytics” which includes the sub-topics basic image processing algorithms and methods, natural language processing, and machine learning. An important consideration fueling our choice to use NIIC subtopics was that the audience and scope of the NIIC and TCO differ, with the intended audience and material for the NIIC being medical residents with MD or DO degrees and the audience and material for the TCO being imaging informatics professionals with over 50 different types of non-medical degrees [10].

As delineated in methods published by Vey et al., we surveyed the directors of the imaging informatics fellowships listed on the SIIM website [5, 8]. Only fellowships focusing exclusively on imaging informatics offered to MD or DO applicants as opposed to biomedical or more general clinical informatics were included. We contacted fellowship program directors via email using Google Docs (Google LLC, Mountain View, CA) to house our survey. Survey questions asked program directors to identify demographic data such as institution and total number of program spots, curriculum sub-topic importance, teaching methods for each sub-topic, cross-departmental partnerships, and individual program strengths and weaknesses. Questions were a combination of questions based on a 5-item Likert-type scale, multiple choice format, and open-ended format with the option to provide additional commentary in an open-ended format at the end of the survey (Table 1).

Table 1.

Survey questions administered to Imaging Informatics fellowship directors.

Question	Answer type
Institution	Open-ended
Total number of program spots per year differentiating MD, DO, and PhD spots where applicable	Open-ended
Would you include this topic in a standardized imaging informatics curriculum?	5-Item Likert-Type Scale: Definitely, Probably, Neutral, Probably Not, Definitely Not, UNSURE
What topics were not listed above that should have been?
How is each topic taught?	Multiple Choice: Didactics, Required Reading, Small Group, Project, Online Course, Other
Since the first suggested Society of Computer Applications in Radiology (SCAR) curriculum was proposed over 15 years ago, many new topics such as Machine Learning, AI, DICOMweb, and FHIR have emerged. Are these topics covered? If yes, how are they covered?	Open-ended
Please describe any cross-departmental or cross-specialty partnerships. For example, with Department of Biomedical Informatics, Health Policy and Health Services, Department of Computer Science, or any others.	Open-ended
Which topic is your program especially strong in teaching? What aspects of your program or teaching methods makes that topic a strength?	Open-ended
Which topics would your program want help covering?	Open-ended
Are there any other comments about the topic of an imaging informatics curriculum you would like to make?	Open-ended

Finally, as an additional comparative measure, the frequency with which individual categories listed in the ABII TCO were tested was identified based on the number of allotted exam questions and compared against imaging informatics fellowship director appraisal of topic importance [9].

Results

From the eight fellowship directors contacted, five completed the survey corresponding to a 63% response rate. The institutions were all located along the East Coast and all trained exclusively MD/DO candidates. Table 2 shows the breakdown of the original 29 broad curriculum topics into 114 sub-topics.

Table 2.

Organization of 114 sub-topics from previously identified 29 broad curriculum topics.

Technical informatics	PACs/RIS/Reporting
	Archives, Viewers, Storage, Compression, Advanced 3D Visualization
	Downtime procedures
	Monitors, Hardware, and Perception
	RIS, Order Entry, Scheduling
	Dictation and Report Generation, Structured Reporting, Report Distribution
	Critical and Non-critical Results Reporting, Patient Portals
	Programming/Development/Software
	Machine Learning
	Natural Language Processing, Scripting, PHP, System Design, Image Segmentation
	Hardware, Software, Networks
	Active Directory, Service-oriented Architecture, Cloud Computing
	Standards
	HL7, DICOM, IHE
	ITIL, RadLex, CPT, ICD-10, SNOMED-CT, System Interoperability
	Data Science
	Information Visualization, ETL, Database Design, Business Intelligence
	Data Plumbing, Validation, Normalization, Access
	Enterprise Imaging
	Infrastructure (Computers, Networking)
	Storage Area Networks, Server Architecture
	High-Availability Design, CMM
	Other
	3D Printing, Ergonomics, Specialty PACs
	Human Factors Engineering, Social Media
	Security
	Layers (Physical, Device, Access, Policy, Networks)
	Hacking (Phishing, DOS Attacks, Password Management)
	PHI (DICOM Metadata, Eliminating “Burned In” Data)
	Department Infrastructure
	Image Acquisition Process, Critical Results
	Tech Feedback, Protocoling, Peer Review
	Study Management
Business and management	PMO (Program Management Office)
	Requirements Gathering, Usability Analysis, Workflow Modeling/Optimization, Change Management
	Implications of Regulations (MU, PAMA, HITECH)
	Business Management Skills
	Negotiation, Leadership
	HR, Organizational Design
	System Implementation
	Implementation and Upgrades of Clinical Systems
	System Evaluations
	Business Analytics
	RFP, Vendor Selection, COIs
	Contract Process, SLAs, Performance Reviews
	Communications
	Finance
	Informatics Funding, Purchasing, Procurement
	Revenue Cycle
	Meaningful Use
	Education
Quality and safety	Decision Support
	Quality
	Compliance/Regulatory
	HIPAA
	Patient Safety
	Radiation Dose
Research	IRB implications on informatics
	Quasi-experimental Study Design
	Surveying methods
	Evaluation Models/Methods
	Honest broker architectures

Only 8% of sub-topics (10/114) received a “definitely” rating with 100% agreement including PACS, Storage, Structured Reporting, Machine Learning/Artificial Intelligence, Natural Language Processing, Standards, HL7, DICOM, IHE, and Business Management Skills (Fig. 1). These sub-topics mapped to only 4 of the original 29 broad fellowship curriculum categories including Standards, Programming/Development/Software, Infrastructure, and PACS/RIS/Reporting. Seventy-seven percent of sub-topics (88/114) had mixed grading defined by two or fewer “definitely” ratings. There was greater agreement with 50% (57/114) for those topics rated probably or definitely.

Fig. 1.

Imaging informatics topics ranked on a 5-item Likert-type scale for likelihood to include in a standardized imaging informatics curriculum

All programs reported that their curricula include new topics emerging since the first suggested SCAR curriculum such as Machine Learning, Artificial Intelligence, DICOMweb, and FHIR. Three from the five programs engage in cross-departmental partnerships such as with the Division of Health Science Informatics, National Center for Human Factors in Healthcare, and Department of Biomedical Engineering. Program strengths and weaknesses were collected in order to begin discussions of partnerships. Program strengths included organizational psychology (change management, influence behavior, Six Sigma), data science (Python, Machine Learning/Artificial Intelligence), reporting, databases, and standards. Program weaknesses included storage, software development, enterprise imaging, revenue cycle, management, and Artificial Intelligence. In one case, one program’s strength could be paired with another program’s weakness (Machine Learning/Artificial Intelligence).

Two programs indicated that additional topics should be included, specifically Six Sigma, Lean, Python, and Alternate Reality/Virtual Reality. Teaching was overwhelmingly done in small groups with only Honest Broker Architecture and Finance taught through a combination of didactics and required reading. In the additional comments section at the end of the survey, one program responded, “We run the risk of creating an ambitious, comprehensive curriculum which exposes fellows in breadth to many topics but not in much depth. I think an important consideration as we develop a fellowship curriculum is to define the core curriculum and the supplemental topics.”

Image management, information technology, and medical imaging informatics were the most represented topics in TCO corresponding to 24, 18, and 17 questions, respectively, from 130 questions total (Fig. 2). Although, TCO and NIIC categories have multiple areas of overlap which don’t conform to exact category matching, the topmost tested category in the TCO (Image Management, 24 questions) could be most closely mapped to half of all NIIC categories (Standards, PACS and Archive, Life Cycle of A Radiology Exam, Radiology as Seen from Outside the Department, and The Business of Informatics). The 8% of sub-topics receiving a “definitely” rating with 100% agreement were all included in the top 3 most frequently tested categories in the TCO: image management (fellowship program director responses of PACS), information technology (Storage, Machine Learning/Artificial Intelligence, Natural Language Processing, DICOM), and medical imaging informatics (Structured Reporting, Standards, HL7, IHE).

Fig. 2.

Tested topics in the American Board of Imaging Informatics Test Content Outline (TCO) ranked by number of scored questions

Discussion

Using prior work establishing broad areas of imaging informatics fellowship curriculum overlap, we leveraged NIIC sub-topics for the first time to expand broad areas of overlap into sub-topics for clarity, appraise whether each subtopic should belong in a core curriculum, and identify individual program areas of expertise and challenging subject areas where collaboration may be beneficial. We were able to use curriculum documents from other well-recognized sources such as NIIC and the ABII TCO as a comparative measure to mitigate the deleterious effects of small sample size.

In stark contrast to the breadth of 114 sub-topics identified through mapping the NIIC to the TCO to previous current state assessment categories, only 8% (10/114) of sub-topics received a “definitely” rating with 100% agreement including PACS, Storage, Structured Reporting, Machine Learning/Artificial Intelligence, Natural Language Processing, Standards, HL7, DICOM, IHE, and Business Management Skills. The remaining 77% (88/114) of sub-topics had had mixed grading defined by two or fewer “definitely” ratings. There was greater agreement with 50% (57/114) for those topics rated probably or definitely. These results speak to a degree of disagreement between what should and should not be considered a core curriculum sub-topic. One potential reason for these differences could be that different programs have different objectives such as those inherent to an academic as opposed to a clinical focus.

However, there was consistency between fellowship director rating of topic importance and the TCO, as all 10 sub-topics receiving a “definitely” rating with 100% agreement were included in the top 3 most frequently tested categories in the TCO. These results highlight that while there is broad agreement on categories of teaching, such as those included in TCO (image management, information technology, and medical imaging informatics), there is great variability in incorporation of specific sub-categories into a core curriculum representing an opportunity for greater collaboration, discussion, and standardization.

Program director appraisal of relative sub-topic importance can be used to facilitate discussion on what should be included in a core curriculum. We have demonstrated that challenges will be identifying where the cut line should be for consensus, whether sub-topics receiving 50% “definitely” ratings should be included or whether this line should be drawn lower. One important consideration is the risk of creating an unwieldy comprehensive but superficial curriculum. This can be prevented by defining and prioritizing “core” sub-topics.

A key goal of this analysis was to identify program areas of collaboration. Program strengths and weaknesses did not follow a consistent pattern, except in one case where one program’s weakness in Artificial Intelligence could be paired with another program’s strength in the same area. Future discussion may include elaboration of such areas of collaboration and complementary program pairing. Additionally, once a consensus curriculum is established, those programs whose strengths were not identified as “core” sub-topics may designate these sub-topics as “supplementary,” thus differentiating themselves from other programs to potential applicants.

We have shown that programs were updating their internal curricula independent of the dated 2004 SCAR curriculum as every surveyed program included new topics emerging since 2004 including Machine Learning, Artificial Intelligence, DICOMweb, and FHIR. We believe that our work can be used to enable the natural process of curriculum sharing between sites. To facilitate this goal, an infrastructure of concrete pipelines for collaboration will be critical. For example, joint educational sessions between fellows at different institutions would both facilitate rapid information exchange and bolster diversity of educational resources.

As with any study of a niche field, ours was limited by a small sample size. Additionally, all institutions were located along the East coast while Midwestern institutions were not represented.

Curricular coordination and consensus may lead to a formal imaging informatics certification, such as the Clinical Informatics (CI) certification sponsored by the American Board of Preventative Medicine (ABPM), or the Imaging Informatics Professional (IIP) certification offered by the American Board of Imaging Informatics (ABII). Each of these certifications is broad, with CI covering clinical informatics in general as opposed to specifically imaging informatics and the IIP directed at both medical and non-medical professionals. For example, the IIP is offered to over 50 different types of eligible degrees and certifications including non-MDs [10]. An imaging informatics certification which defines time commitment and suggests funding mechanisms for that time could be a valuable future direction. To this end, next steps would include studying how much time is needed for a fellow to dedicate to each fellowship program, how the institution is funding the position, and how many faculty and resources are available for the informatics fellowships.

Additional future directions of study may be the potential impact of an informatics fellowship to an institution’s ability to attract clinical fellows as well as other potential benefits of an informatics fellowship to the institution outside those to the fellows themselves.

Conclusion

Leveraging the existing NIIC curriculum and the ABII TCO, we have shown that although there is variability in inter-program agreement between inclusion of surveyed sub-topics into a core curriculum, broad categories of agreement exist. Individual programs were updating their curricula and incorporating new topics emerging since the last published core curriculum in 2004 showing that perhaps the key barrier to an updated curriculum wasn’t the act of updating it itself, but coordination between programs. We have also outlined potential pitfalls as we approach a core curriculum such as creation of a curriculum that favors breadth over depth, which may be managed by careful consideration of the cut line for how much consensus is required before a sub-topic is considered part of the core. Finally, we have built the framework for inter-program collaboration by initiating a discussion on individual program strengths and weaknesses.