The Leaders in Informatics, Quality, and Systems (LInQS) Fellowship

Abstract

Background  Leaders in Informatics, Quality, and Systems (LInQS) is a non-ACGME (Accreditation Council for Graduate Medical Education)-accredited 2-year training program developed to enhance training in the fields of health care delivery, quality improvement (QI), clinical informatics, and leadership.

Methods  This single-institution 2-year longitudinal training program grounded in QI and informed by leadership and clinical informatics includes didactics, coaching, and mentorship, all centered around individualized QI projects. The program has been available to sub-specialty fellows, advanced practice providers, and physicians.

Results  From 2019 to 2023, 32 fellows have been accepted into the program with 13 graduates and 16 currently enrolled. Fellows have been predominately female, physicians, and from multiple specialties but predominantly hospital medicine. Fellows' evaluations of the fellowship are highly positive, rating the didactics and mentorship aspects of the curriculum most favorably. Most fellows' projects utilized informatics solutions including clinical decision support tools to increase quality of care, improve patient outcomes, and reduce costs of care resulting in manuscript publications, national presentations, and a national specialty society award. Since matriculation, 50% of fellows received certification as Epic Physician Builders and 34% received leadership positions in clinical informatics, quality, and education.

Conclusion  Our experience supports the need to provide health care providers more expansive training in the areas of QI, clinical informatics, and leadership for improving health care delivery. Additional in-depth knowledge and experience in these fields may produce and benefit leaders in these fields.

Background and Significance

Since publication of the Institute of Medicine's To Err is Human in 2000, there has been a shift in the way health care is delivered using available resources to offer optimal patient care to populations and delivering safer care by reducing patient harm. 1 During this time, medical education has evolved to include training in patient safety and quality improvement (QI) with the Accreditation Council for Graduate Medical Education (ACGME) incorporating these initiatives into its core competencies in 2003 and now enshrined in its Core Program Requirements. 2 There continues to be ongoing work on how to incorporate QI training into medical education given its importance in care delivery. 3 4 Physician leaders and their partnership with health systems are also essential for patient-centered QI, yet physicians are often unprepared for these roles due to lack of training. 5

Understanding health information technology (HIT) is also critical to the knowledge base and skill set of today's providers. The passing of the HITECH act in 2009 incentivized health care systems and providers to adopt electronic health records (EHRs). Since 2014, nearly all nonfederal acute care hospitals have adopted a certified EHR. 6 The promise of EHR adoption was to improve patient care through improved clarity of documentation, enhanced decision making through clinical decision support (CDS), and providing a framework for accountability for health care outcomes through data reporting and analytics. It therefore makes sense that HIT is critical for improving quality and patient safety. 7 Topics in HIT are slowly being incorporated into medical education at the Liaison Committee on Medical Education (LCME) and residency level. Although there are over 50 ACGME-accredited fellowships in clinical informatics (CI) and a national board certification in this specialty, pathways for narrower training in CI currently are limited. 8 9

These changes in health care delivery have created a growing need for more integrated and coordinated QI, CI, and leadership training for the provider workforce. In 2019, the University of Colorado School of Medicine, Department of Medicine created a non-ACGME accredited fellowship training program to develop transformational leaders in QI versed in CI. This is a descriptive study of the fellowship which involves didactic lectures on leadership, CI, and QI topics along with mentorship and project-based learning activities. The aims of this manuscript are to describe the education program and present an analysis of the outcomes including describing our matriculants, assessing their achievements, and evaluating the program.

Methods

Settings and Participants

The Department of Medicine at the University of Colorado School of Medicine is composed of 13 clinical divisions and includes over 1,000 faculty, 192 residents, and 200 fellows with affiliations at several urban academic medical centers. The Leaders in Informatics, Quality, and Systems (LInQS) Fellowship was established in 2019 to support junior faculty and fellows in sub-specialty training programs in the department of medicine to become leaders in CI, quality, safety, and health care system design all within a complex learning health system.

Fellow Selection Process

Fellows complete a personal statement and a structured application. They are chosen based on their experience with prior quality initiatives, their motivation to join the fellowship, ability to succeed, and the potential impact of their proposed projects. Projects must be conducted at the University of Colorado Hospital or affiliated clinics, which utilize Epic (Epic Systems Corporation) as the EHR. Proposed projects are rated on their importance, impact, feasibility, and alignment with health system priorities. Fellows are grouped into cohorts by their year of acceptance.

Faculty and Resources

Our LInQS core faculty of 11 physicians and 1 Advanced Practice Provider (APP) include board-certified informaticists, the Chief Medical Informatics Officer at the University of Colorado Hospital, medical directors, and quality experts. The resources including faculty time and administrative support required for development and administration of the LInQS Fellowship program are detailed in Supplementary Appendix 1 (available in the online version)

Curriculum

The LInQS Fellowship curriculum consists of 26 weeks of didactics and group work sessions led by expert faculty, known as the core curriculum. Sessions are held over 2 hours, bi-weekly in online and in-person formats. Didactics are provided by a mixture of LInQS core faculty and outside speakers. The core curriculum series focuses on informatics topics including the history of health informatics, EHR design and optimizations, EHR database structures and data extraction, CDS and user-centered design, and artificial intelligence in health care. Quality topics include complex health care systems, qualitative and mixed methods data analysis, intervention design, implementation and dissemination science, and high-performing teams (see Fig. 1 for learning objectives). The LInQS group work sessions include activities based on the Lean/Six Sigma Define-Measure-Analyze-Improve-Control (DMAIC) model. 10 During each session there is a brief instruction on the topic followed by the fellows completing assignments using DMAIC tools and concepts. Assignments include developing problem statement and aims, stakeholder analysis, metric design and tracking, root-cause analysis, and change management strategies. The curriculum also includes sessions on manuscript writing, curriculum vitae creation, and career counseling. In addition to the core curriculum, fellows attend a six-part workshop series focused on QI methodology and a 3-day workshop on becoming health care leaders offered by a local quality institute at the university.

Fig. 1.

LInQS Fellowship learning objectives. This is a schematic of the learning objectives of the LInQS Fellowship grouped into the core domains of the fellowship curriculum.

The core curriculum is centered around fellow-led individual projects targeted at improving health care delivery in an area of their choosing. Each fellow has an assigned faculty mentor as well as assistance from other fellows and faculty for project and problem-specific needs. Alongside the formal curriculum, fellows lead an individual project and participate in monthly mentorship meetings and quarterly report outs. All formal didactics and workshops are delivered in the first year of the fellowship. The second year of the fellowship is dedicated to project work with continued mentorship and quarterly report outs. Fellows are given 10% full-time equivalent (FTE) dedicated time to participate in the fellowship.

Outcomes Measured

We evaluated the LInQS Fellowship from a variety of aspects including matriculant demographics (gender identity, clinical specialty, position, clinical role, division in the department of medicine), scholarly activity (intra- or extra-mural funding, local or national presentations, manuscript publication), extra certifications—specifically Epic Systems Physician Builder, awards, and leadership positions attained. We also categorized project areas of focus and activities including use of EHR data and modifications including order changes, note templates, and incorporation of CDS. Finally, fellows evaluated the program after graduation in domains of perceived value of fellowship activities and their personal outcomes. We also asked graduates to rate the program as a whole and each activity including the core curriculum, mentorship, and QI and leadership training. Demographics, project activities, certifications, scholarly output, and new leadership positions are reported for all matriculants. Program evaluations were collected from graduates only ( N  = 13).

Collection and Analysis of Outcomes

Survey data are collected via Formstack (formstack.com), which has analytic functions to collate data automatically. Data were collated with Microsoft Excel (Microsoft Corporation 2022) and analyzed using descriptive statistics. Graduate survey data are compiled via Microsoft Excel in a compilation of individual fellow ratings. Survey answers were posed as a 5-point Likert scale from Strongly Disagree to Strongly Agree. We analyzed survey responses by assessing central tendency through looking at the distribution of responses and identifying the most frequent response. We categorized project topics and noted which included EHR-based interventions and captured specifics of the interventions.

Results

From 2019 to 2023, across five cohorts, 32 fellows have been accepted into the program with 13 graduates and 16 currently enrolled. Among the fellows, 12 (38%) are male and 20 (62%) are female. There are 14 (44%) assistant professors, 11 (34%) sub-specialty fellows, and 7 (22%) APPs. Fellows originate from 12 of the 13 clinical divisions in our department with the majority from hospital medicine (7, 22%), followed by gastroenterology and hepatology (4, 13%), cardiology (3, 10%), pulmonary and critical care (3, 10%), and others shown in Table 1 . In addition to matriculants from the department of medicine, three fellows matriculated from other departments including two sub-specialty fellows from pediatrics (pediatric endocrinology and pediatric hospital medicine) and one faculty member from obstetrics and gynecology (assistant professor). To date, three fellows (two from sub-specialty fellowships and one assistant professor) completed the curriculum but did not complete projects—a core pillar of the fellowship.

Table 1. Demographics of LInQS fellows.

Fellow gender	n (%)
Male	12 (38%)
Female	20 (62%)
Total	32
Fellow specialty	n (%)
Allergy and Clinical Immunology	1 (3%)
Cardiology	3 (10%)
Endocrinology, Metabolism and Diabetes	1 (3%)
Gastroenterology and Hepatology	4 (13%)
General Internal Medicine	2 (6%)
Geriatric Medicine	2 (6%)
Hematology and Bone Marrow Transplant	1 (3%)
Hospital Medicine	7 (22%)
Infectious Disease	2 (6%)
Medical Oncology	1 (3%)
Pulmonary and Critical Care	3 (10%)
Renal Medicine Disease and Hypertension (Nephrology)	2 (6%)
Rheumatology	0 (0%)
Outside Department	3 (9%)
Fellow position (at beginning of fellowship)	n (%)
Instructor	5 (16%)
Senior Instructor	1 (3%)
Assistant Professor	15 (47%)
GME Fellow	11 (34%)
Fellow role	n (%)
Physician	14 (44%)
GME Fellow physician	11 (34%)
Physician assistant	1 (3%)
Nurse practitioner	6 (19%)

Abbreviation: LInQS, Leadership, Informatics, QI, and Systems.

Note: List and percentages of LInQS Fellows' demographics including gender, specialty and position, and role.

Project areas of focus were grouped into categories including increasing guideline concordant care (12, 40%), access to care (4, 12%), reduction of adverse events (4, 12%), cancer screening/prevention (3, 9%), and other topics outlined in Table 2 . Projects that included an EHR-based intervention are noted in Table 2 and include electronic alerts (12, 38%), new or revised order panels (10, 31%), and note templates (6, 19%).

Table 2. Projects and outcomes of LInQS fellows.

Fellow project topics	n (%)
Guideline concordant care	12 (40%)
Access to care	4 (12%)
Antibiotic stewardship	1 (3%)
Transitions of care	2 (6%)
Reducing adverse events	4 (12%)
Advanced care planning	2 (6%)
Cancer screening/prevention	3 (9%)
Vaccine administration	1 (3%)
Medication reconciliation	1 (3%)
Resource utilization	2 (6%)
Fellow EHR-based interventions	n (%)
Order set	10 (31%)
Electronic alert	12 (38%)
Natural language processing	1 (3%)
Risk stratification tool	1 (3%)
Care pathway	2 (6%)
Problem-based charting	1 (3%)
Virtual health center management	1 (3%)
Note template/smartphrase	6 (19%)
Fellow leadership roles	n (%)
Any leadership role	11 (34%)
Informatics leader	5 (16%)
Quality leader	5 (16%)
Educational leader	1 (3%)
Clinical director	5 (16%)
Promoted position	3 (9%)
Fellow scholarly output	n (%)
Manuscript publication	4 (13%)
National presentation	4 (13%)
National recognition award	1 (3%)
Epic headquarters tool adoption	1 (3%)
Certified Epic Physician Builder	16 (50%)

Abbreviation: EHR, electronic health record.

Note: List and percentages of LInQS fellows including project areas of focus, EHR-based interventions, leadership roles attained, and scholarly output produced. Two projects included more than one EHR-based intervention.

Fellows' evaluation of the program is summarized in Table 3 . Furthermore, 11 of 13 graduates completed the survey. The core curriculum was rated the most valuable (46%, very valuable) for which 46% of fellows strongly agreed that the core curriculum gave them the required knowledge to carry out their QI projects. The fellows also valued the leadership training (46%, very valuable) and the QI workshops (39%, somewhat valuable). One-third of fellows felt the LInQS Fellowship guided their future career choices (31%, strongly agree) and more than two-thirds felt the fellowship was a valuable experience (69%, strongly agree). LInQS fellows rated the faculty mentorship highly (82%, strongly agree).

Table 3. Feedback from LInQS fellows.

Post-evaluation feedback		13 respondents
Chosen focus after graduation		n (%)
	Clinical informatics	4 (31%)
	Health care leadership	2 (15%)
	Learning health systems	1 (8%)
	Quality improvement	6 (46%)
Value of LInQS core meetings		n (%)
	Neutral	2 (15%)
	Somewhat valuable	5 (39%)
	Very valuable	6 (46%)
Value of QI workshops		n (%)
	Somewhat not valuable	3 (23%)
	Somewhat valuable	5 (39%)
	Very valuable	5 (39%)
Value of leadership training		n (%)
	Neutral	1 (8%)
	Somewhat valuable	6 (46%)
	Very valuable	6 (46%)
LInQS guided my future career choices		n (%)
	Neither agree nor disagree	2 (15%)
	Somewhat agree	7 (54%)
	Strongly agree	4 (31%)
LInQS was a valuable experience		n (%)
	Somewhat agree	4 (31%)
	Strongly agree	9 (69%)
LInQS didactic series gave me knowledge to carry out my QI project		n (%)
	Neither agree nor disagree	2 (15%)
	Somewhat agree	5 (39%)
	Strongly agree	6 (46%)

Abbreviation: QI, quality improvement.

Note: Distribution of feedback provided by LInQS fellows about the curriculum.

As detailed in Table 2 , one-third of fellows gained leadership positions (11, 34%), including informatics leadership roles (5, 16%), quality director roles (5, 16%), educational leadership roles (1, 3%), promotions in position (3, 9%), and promotions to clinical director (2, 6%). Half of the fellows (16, 50%) have been certified as Epic Physician Builders, demonstrating pursuit of extra training beyond the curriculum. Scholarly output includes peer-reviewed manuscript publications (4, 13%), national presentations (4, 13%), intramural grant awards (6, 19%), and one fellow was awarded a national specialty society award.

Discussion

To the best of our knowledge, this is the first report describing a non-ACGME fellowship combining QI, CI, and leadership training for medical providers. The LInQS Fellowship is a unique program based in QI training and informed by CI to develop leaders in the health care setting. Unlike traditional ACGME-accredited fellowships, the LInQS Fellowship attracts not only Graduate Medical Education fellows, but also junior faculty physicians and APPs including physician assistants and nurse practitioners.

The LInQS fellows have been primarily female medical providers, which, though nonintentional, is important as there is a documented dearth of women in CI, 11 particularly in leadership positions. We were also successful in recruiting fellows from across our department with fellows from every clinical division, save for rheumatology. We also had three fellows from external departments, demonstrating demand for this training beyond internal medicine and its sub-specialties.

The LInQS fellows' projects include a wide range of health topics including increasing guideline concordant care, access to care, and reduction of adverse events, which increasingly require EHR-based solutions such as groups of orders, pathways for diagnosis and treatment, and electronic alerts. Most fellows' projects utilized informatics solutions including CDS tools to increase quality of care, improve patient outcomes, and reduce costs of care. Our survey results show the LInQS core curriculum and mentorship to be the most valuable activity for fellows, giving them the required knowledge to carry out a QI project within the fellowship and in future efforts. Overall, the majority of the LInQS fellows strongly agreed the fellowship was a valuable experience that guided their future career choices. Several of the LInQS Fellowship graduates moved on to CI-focused leadership roles with over a third of fellows graduated and enrolled in leadership roles including directorships, educational roles, and promotions. Scholarly output is a core objective of the LInQS Fellowship, with nearly half of graduates publishing their project outcomes in peer-reviewed journals and presenting their work at national conferences shortly after graduation.

Lessons Learned

The LInQS Fellowship has transformed over the last 5 years with the addition of many new elements not originally included in the program. Learning from graduates and the three fellows who did not complete the fellowship, starting with Cohort 5, we implemented a new coaching model in which each fellow meets bi-weekly with one of our faculty for project coaching and timeline management. We have found this to be an invaluable addition to the structure of the program and hope to show increased scholarly output and improved fellowship completion. We believe this new coaching structure will ensure close contact with fellows who may struggle with teaching concepts, time commitment, or project barriers. Over the life of the program, we have become increasingly reliant on our informatics board-certified faculty to act as subject matter experts and mentor the fellows through challenges and barriers in their projects, specifically with stakeholder engagement and CDS implementation within the EHR. Additionally, we changed the didactics curriculum based on fellows' feedback, to be more tailored to their learning needs and career goals. In Cohort 5, we added several outside speakers to lecture on topics such as QI manuscript writing, careers in CI and QI, and navigating complex systems, payors, and priorities as a health care leader.

At the onset of the fellowship, we partnered with a local QI institute for a team-based QI leadership training, but fellows' feedback revealed the content to be duplicative and not well adapted for fellows working on individual efforts. Instead, beginning with Cohort 3, fellows attend a 3.5-day leadership focused workshop, with very positive feedback from fellows.

There are many established training programs in QI, CI, and leadership in health care, individually. However, a program like the LInQS Fellowship combining all three has not been described even though the combination of these activities has been recognized as important. For example, the CI board fellowship includes elements of QI and leadership as part of the core requirements. 12 Despite this, there are few options for providers interested in creating and leading lasting change in health care delivery who do not want to pursue a full CI board certification. As the fellowship progressed, we recognized the LInQS Fellowship is fundamentally a QI program anchored by knowledge and skills in CI and leadership. Training in QI requires a hands-on approach and project work with support through coaching and mentorship is integral to fellows' learning and potential for success. 1 Arguably, QI training must also include CI and leadership to ensure medical providers understand the systems in which they work and are equipped to implement effective and sustainable solutions. Interestingly, our results demonstrate many LInQS fellows pursued extra training beyond the curriculum (Epic Physician Builder certification) to manipulate the EHR and sought informatics leadership positions revealing the importance of informatics in the health care improvement space. 7

Limitations

First, when we started the LInQS Fellowship, we did not initially assess fellows' baseline skills, attitudes, or knowledge prior to matriculation. We attempted to assess learning after each activity but had poor response rates and nonstandardized questions from our faculty presenters. Despite this, we feel the outcomes described above including successful implementation of quality and CDS interventions demonstrate the acquisition of knowledge and skills. Second, with a project-based curriculum, it is difficult to standardize all experiences. Third, this program was designed for and largely included fellows from internal medicine and medical sub-specialties, thus we cannot infer how this curriculum would apply to other specialties. However, several of our fellows' projects included procedural documentation which may translate to surgical specialties. Fourth, we are highly dependent on local experts for much of our curriculum delivery, potentially making this curriculum hard to replicate in other settings. Fifth, differences in the demographics of our matriculating fellows may be more reflective of division-level FTE support and funding rather than true demand as all divisions within our department are funded differently. Finally, despite there being an established curriculum for CI fellowships with defined learning objectives and milestones, there are no consistent standards for QI or leadership training programs in health care, though we incorporated recommended standards for DMAIC methodology and utilized input from experienced faculty in each of our target areas.

Future Directions

We continue to incorporate fellows' feedback to tailor didactic topics, group work sessions, and the curriculum in general. Beginning in Cohort 4, we will compare pre- and post-program knowledge and attitude assessment. We have adapted our coaching and mentorship structure throughout the fellowship using this feedback and continue to adapt as needed. Our goal is to add more APP faculty to our LInQS core faculty to support our growing number of physician assistant and nurse practitioner fellows. We also plan to expand eligibility of the LInQS Fellowship to all faculty in the upcoming Cohort 6, where previously only junior faculty (instructors and assistant professors) and fellows were eligible for the fellowship.

There is an opportunity for more research to study the relationship between informatics training in a QI-based curriculum. While the LInQS Fellowship program has seen several fellows move on to successful leadership roles in CI, there is no literature studying the structure of joint QI, CI, and leadership-based programs and how to train providers in these careers. Anecdotally, reduction in clinical service to dedicate to the fellowship seems to be critical for meeting the curricular and project demands which should also be explored.

Conclusion

We believe the LInQS Fellowship is a novel and effective model for provider training as it combines QI, CI, and leadership elements which are traditionally taught in silos. The combination of these topics is important for providers hoping to transform health care delivery. Our fellowship demonstrates how a curriculum can combine these areas centered around QI project work, which is increasingly dependent on CI and leadership skills.

Clinical Relevance Statement

The LInQS Fellowship trains advanced practice providers, physicians, and sub-specialty fellows in quality improvement, clinical informatics, and leadership—domains critical for improving the delivery of health care. The implications of this program description are relevant to anyone trying to educate and train providers interested in clinical transformation of our health care systems.

Multiple-Choice Questions

1. Which of the following core content areas are included in the Leaders in Informatics, Quality, and Systems (LInQS) fellowship?

   1. Quality improvement using the DMAIC framework from Six Sigma

   2. Clinical Informatics

   3. Leadership

   4. All of the above

   Correct Answer: The correct answer is option d. All of the above. The LInQS Fellowship provides training in quality improvement, clinical informatics, and leadership.

2. What percentage of LInQS fellows was selected for any leadership role either during or after the fellowship?

   1. 10%

   2. 27%

   3. 34%

   4. 48%

   Correct Answer: The correct answer is option c. 34%. As detailed in Table 2 , one-third of fellows gained leadership positions (11, 34%) including informatics leadership roles (5, 16%), quality director roles (5, 16%), and educational leadership roles (1, 3%).