Where Are We and Where Are We Going? A GME Program Scorecard for Continuous Programmatic Improvement

ABSTRACT

Background

Graduate medical education (GME) leaders face challenges in evaluating program performance due to fragmented data systems and limited resources. Dashboards have been proposed, but few are scalable, participant-informed, and feasible without new infrastructure.

Objective

To develop and implement a centralized, participant-informed GME Program Scorecard (GPS) that enables continuous program evaluation and improvement using existing data systems without new data entry.

Methods

The GPS was developed at a large academic medical center in 2022-2023. A consensus development panel of 20 GME administrators identified 42 candidate metrics across 13 systems. Program directors (PDs) from the largest residency programs were surveyed (11; 9 responses, 81.8%) to rate each metric’s value. The 17 most supported metrics were integrated into a Tableau-based dashboard. The GPS was piloted with 4 programs (3 Accreditation Council for Graduate Medical Education [ACGME]-accredited, 1 non-ACGME-accredited), involving 8 PDs and program coordinators. Feedback was collected via structured forms and a focus group. Metrics were rated on a 1 to 5 scale, with lower scores indicating higher priority.

Results

Pre-intervention survey results guided prioritization, with 92.9% of metrics receiving a mean score <3. Pilot users reported high acceptability and identified design refinements incorporated before institutional launch. Development required 2920 analyst hours and $1,600 in licenses. Since launching in January 2024, 309 users (61.4% of 503 potential users) have accessed the GPS with 3933 total views.

Conclusions

The GPS demonstrated feasibility, requiring 2920 analyst hours and $1,600 in licenses, and with 61.4% of potential users accessing the dashboard within 22 months of launch.

Introduction

Sponsoring institutions with multiple graduate medical education (GME) programs often struggle to implement efficient, ongoing evaluation across diverse specialties and sites. For GME leaders and educators, the ability to access centralized, actionable data is essential to support continuous quality improvement and institutional oversight.

Traditional GME performance metrics—such as board pass rates and Accreditation Council for Graduate Medical Education (ACGME) citations—offer only limited insight into the broader educational environment. Early proposals for GME program report cards centered on quantitative outcomes such as match rates, fill rates, and board scores as primary markers of program quality.^1,2 Prior literature highlights persistent challenges in defining and tracking more nuanced indicators of program quality, including faculty engagement, resident experience, and compliance.^3-6 Barriers such as fragmented data sources, administrative burden, and lack of interoperability across systems have further limited the development of systems capable of offering holistic, institution-wide views of program quality.^2,4,7

Several institutions have piloted dashboards to support accreditation compliance and program benchmarking.^8-10 Dashboards are advantageous as they are dynamic and updateable; however, maintaining them can be resource-intensive and requires substantial data extraction or manual reporting, which limits scalability.⁴ Standardized report cards and educational scorecards have also been used to evaluate programs according to elements prioritized by the ACGME, such as program quality, resident education, graduate outcomes, faculty engagement, and alignment with institutional missions.^2,11 Unfortunately, these scorecards faced challenges with subjectivity, data validity, and administrative burden.^12,13 The literature underscores a critical gap: existing dashboards and scorecards struggle to integrate institutional oversight needs, accrediting body requirements, and participant priorities into a unified, low-administrative-burden process for monitoring program performance. The goal of this innovation was to develop a centralized GME Program Scorecard (GPS) that integrates accrediting body requirements, participant input, and institutional oversight needs, and to report feasibility and acceptability data from pilot users.

We developed the GPS, a centralized dashboard designed to integrate educational data across our large sponsoring institution. This article describes the first 2 phases of GPS development: (1) a participant-informed needs assessment and metric selection process, and (2) the design, implementation, and early evaluation of a dashboard to support real-time program monitoring and improvement.

KEY POINTS

What Is Known

Many graduate medical education (GME) leaders would like the ability to visualize and track program performance, especially across an institution, but dashboards can be challenging to create due to cost and data fragmentation.

What Is New

One large institution created a dashboard across its multiple GME specialties using existing data systems using Tableau; the article reports on cost and developer time needed to launch this project.

Bottom Line

A locally developed dashboard can be feasibly implemented and offers an opportunity for more streamlined outcomes evaluation and continuous quality improvement for GME programs.

Methods

Setting and Participants

This project was conducted at a large academic medical center with 246 residency and fellowship programs (118 ACGME-accredited) that include 1752 trainees across 7 clinical sites in 3 US states. The GPS was developed over the 2022-2023 academic year. Participants involved in development included all 20 nonclinical GME administrative staff, 11 program directors (PDs), and 4 program coordinators (PCs) representing ACGME- and non-ACGME-accredited programs of varying size and specialty.

Intervention: GPS Development

We used a consensus development panel methodology, grounded in constructivist theory, to design a participant-informed scorecard that reflects program-level educational quality.¹⁴ At a GME administrative retreat in February 2023, the 20 nonclinical staff, each with expertise in one or more education data systems, generated a list of 42 potential metrics across 13 institutional platforms. These spanned categories, including faculty and trainee performance, engagement, compliance, educational infrastructure, and graduate outcomes.

To prioritize metrics, we developed a survey instrument listing all 42 metrics, each rated on a 5-point anchored Likert scale (1=“Must be included,” 5=“Must be excluded”), along with open comment fields and optional prompts for suggesting additional metrics (provided as online supplementary data). The instrument was piloted with a program director not involved in the development process; no changes were recommended. The finalized survey was administered via Qualtrics (Qualtrics LLC) to the directors of the 11 largest programs at our institution. Large programs were targeted at this stage of the project, given the high data complexity inherent to managing more trainees. Smaller programs at our institution do not report as many challenges with data management as larger programs.

The GPS prototype was constructed in Tableau (Salesforce Inc), using the 17 most highly supported metrics and integrating data from 13 institutional sources via application programming interfaces (APIs) or manual feeds. The dashboard included expandable architecture with drill-down functionality for selected metrics (eg, faculty teaching scores).

The prototype was piloted in September 2023 with 4 program leadership teams (PDs and PCs), who were invited from the pool of survey participants who prioritized the metrics and selected for diversity in program size, specialty, and accreditation status. Small, large, ACGME-accredited, and non-ACGME-accredited programs were selected. Sampling procedure emphasized program diversity at this stage to ensure programs with different needs could provide input on the GPS that could be used by all programs. Each team received full access to their GPS dashboard and was asked to complete a structured feedback form (provided as online supplementary data), which had been developed by the research team based on existing usability instruments for website evaluation.¹⁵ All 8 invited users completed the evaluation, followed by an in-person focus group to elicit further feedback. Suggestions from the written evaluations and group discussion were incorporated into the GPS redesign.

The GPS was launched in January 2024. All users received training through a 1-hour live program leadership session at this time. This session included GME team members, PDs, PCs, educational administrators, and other GME leaders. GPS was provided to PDs, PCs, and educational oversight groups, including vice chairs of education, spanning all training programs. A written frequently asked questions sheet was provided for ongoing reference. No additional training sessions were required. Ongoing usage metrics have been collected continually via Tableau.

Outcomes Measured

Primary outcomes included: (1) the set of final metrics prioritized by participants; (2) qualitative feedback on GPS usability, content, and navigation; (3) user engagement following rollout, including unique users and page views; and (4) feasibility, assessed by tracking total developer hours and direct costs of the intervention.

This study was reviewed by the Cleveland Clinic Institutional Review Board and deemed exempt from formal review.

Results

Participant Demographics and Survey Response

Twenty nonclinical members of the GME administrative team participated in a consensus development panel held in February 2023. All 20 contributed to identifying an initial set of 42 candidate metrics across 13 institutional data systems.

To prioritize these metrics, we surveyed the PDs of the 11 largest residency programs at our institution. Of those invited, 9 PDs completed the survey (81.8% response rate). Survey respondents represented high-volume, ACGME-accredited training programs across procedural and non-procedural specialties.

Survey Results and Metrics Selection

The overall mean score for the 42 potential metrics was 2.15 (range, 1.22-3.44), with 1 being “Must be included” and 5 being “Must be excluded.” The 9 most highly prioritized metrics—all with mean scores ≤1.67—included board pass rates, faculty teaching scores, work hour violations, and resident survey results. Four additional metrics that were less prioritized were also included due to feasibility or alignment with institutional priorities.

• In addition to trainee publications (mean, 1.22; SD, 0.44), faculty publications (mean, 2.11; SD, 0.78) were included because their data were already integrated, imposing no added burden.

• Safety event reporting (mean, 2.13; SD, 0.64) patient feedback (mean, 2.56; SD, 1.01) were included due to alignment with hospital-wide goals around safety and quality.

• Percentage of total applicants who apply to the program (mean, 2.22; SD, 0.97) was included as a GME office priority metric.

These 13 metrics were extracted from 13 unique institutional data sources to populate the GPS dashboard. The final GPS prototype was implemented in Tableau and featured 5 years of longitudinal data with interactive dashboards for drilling down into metric details. All considered metrics are shown in the Table with corresponding mean scores and data sources. Figure 1 displays screenshots of the GPS dashboard’s drill-down architecture, using the faculty teaching scores as an example.

Figure 1.

Screenshots of GPS Dashboard’s Drill-Down Architecture (A) The GPS Main Dashboard for a Single Program (B) Faculty Teaching Scores for That Program (C) Faculty Teaching Scores by Individual in That Program (D) Faculty Teaching Scores for One Individual in That Program

Table 1.

Proposed Metrics for Inclusion in the GME Program Scorecard

No.	Metric	Mean Scorea	SD	Data Source
1	Publications by trainees	1.22	0.44	ACGME
2	Program faculty teaching scores	1.33	0.50	MedHub
3	Citations from accrediting bodies	1.33	0.50	ACGME, MedHub
4	Compliance with GME processes (PIP, APE, flu, MyLearning, etc)	1.33	0.50	GME
5	Program faculty feedback to trainees, % completed	1.44	0.53	MedHub
6	MedHub annual surveys of trainees/faculty	1.50	0.76	MedHub
7	ACGME annual surveys of trainees/faculty	1.50	0.76	ACGME, MedHub
8	Work hours compliance	1.56	0.53	GME
9	Board pass rates of graduates	1.67	0.71	GME
10	Morale of trainees	1.78	0.83	Resident/Fellow Annual Evaluation of a Clinical Training Program 2.0 (MD/DO) in MedHub
11	Wellness programs available	1.78	0.67	Resident/Fellow Annual Evaluation of a Clinical Training Program 2.0 (MD/DO) in MedHub
12	Percent of current trainees achieving expected ACGME Milestone progression	1.78	0.97	ACGME
13	Current trainee participation on CCF committees	1.89	0.60	Resident/Fellow Quality Improvement and Patient Safety Self-Evaluation 2.0 in MedHub
14	Percent of trainees who believe they are ready for independent practice at graduation	1.89	0.93	Alumni survey
15	Retention of graduates to CCF staff positions	1.89	0.60	OPSA, Alumni survey
16	Faculty development for education participation	1.89	0.60	Annual Faculty Evaluation of a Clinical Training Program 2.0 (MD/DO)
17	In-training examination score	2.00	0.87	MedHub
18	Academic positions held by program graduates	2.00	0.76	Alumni survey
19	Duration of program leadership	2.00	0.87	MedHub, ACGME
20	Quality of program faculty feedback to trainees	2.11	0.93	Evaluation of Faculty Teaching 2.0
21	Program faculty publications	2.11	0.78	ACGME, PubMed
22	No. of SERS reports filed by trainees	2.13	0.64	SERS
23	Percent of total applicants who apply to program	2.22	0.97	AAMC, ERAS, NRMP
24	Leadership positions among graduates	2.22	0.67	Alumni survey
25	Program efforts toward recruiting for diversity	2.22	0.83	No formal process
26	Procedural case log volumes	2.33	1.00	ACGME
27	Publications by graduates	2.33	1.00	Alumni survey
28	Faculty participation in the CCF education collaborative	2.38	1.19	EI
29	Presence of elected chief or other leadership positions for trainees	2.44	1.13	MedHub
30	Time since last update of goals and objectives	2.44	1.13	MedHub
31	Trainee awards received from outside CCF	2.56	0.53	MedHub
32	Presence of clinical specialist tracks (education, MSTP, etc)	2.56	0.73	MedHub
33	Patient feedback	2.56	1.01	HCAHPS
34	Current trainee participation on committees outside CCF	2.67	0.50	Resident/Fellow Quality Improvement and Patient Safety Self-Evaluation 2.0 in MedHub
35	Additional degrees earned by trainees during training	2.67	0.71	Updated in MedHub if trainee provides information/diploma
36	Community impact of graduates	2.67	0.50	Alumni Survey
37	Program faculty conference attendance	2.67	1.22	MedHub
38	Program faculty academic rank	2.67	1.12	OPSA
39	Percent of unfilled slots available for outside student rotators	2.78	1.30	CCLCM
40	Post-Match survey data	3.11	0.78	Program-initiated post-Match surveys
41	Adverse actions (malpractice) against program graduates	3.33	1.00	Alumni survey
42	Trainees from U.S. News & World Report Top 10 medical schools	3.44	1.24	NRMP

^aLikert-type scale: must be included (1), should be included (2), could be included or excluded (3), should be excluded (4), must be excluded (5).

Abbreviations: ACGME, Accreditation Council for Graduate Medical Education; GME, graduate medical education; PIP, program improvement plan; APE, annual program evaluation; CCF, Cleveland Clinic Foundation; OPSA, Office of Professional Staff Affairs; SERS, Safety Event Reporting System; AAMC, Association of American Medical Colleges; ERAS, Electronic Residency Application Service; NRMP, National Resident Matching Program; EI, Education Institute; MSTP, Medical Scientist Training Program; HCAHPS, Hospital Consumer Assessment of Healthcare Providers and Systems; CCLCM, Cleveland Clinic Lerner College of Medicine.

Note: Bolded metrics indicate those included in the final GME Program Scorecard.

Pilot Testing and Acceptability

A total of 8 participants (4 PDs, 4 PCs) reviewed the dashboard, completed a structured feedback form, and participated in an in-person focus group (100% response rate).

Pilot feedback indicated high acceptability. Users appreciated the consolidation of multiple data sources and features such as searchability and drill-down architecture. Still, they identified areas for improvement, including:

• Lack of clarity in headers, color coding, and metric definitions

• Desire for transparency in threshold logic and data provenance

• Concerns about interpretability, particularly when comparing across programs

Participants also requested additional features (eg, number of trainees, accreditation dates, educational activity listings) and proposed improved navigational supports (eg, frequently asked questions, tooltips). These suggestions were incorporated into the next GPS iteration.

Feasibility and Implementation

The GPS was developed using 2 Tableau Desktop licenses at $800 per person ($1,600 total), allocated to the 2 data analysts, already employed by the GME office, who were assigned to the project. At our institution, these licenses are maintained for broader educational data visualization purposes. However, institutions without existing Tableau infrastructure should anticipate this as both a startup and ongoing annual cost. The project was completed in weekly sprints over the course of 23 months, totaling approximately 2920 resource hours. Integration and dashboard development drew exclusively from existing data sources; no new data entry was needed from program directors or coordinators. The consensus development panel, surveys, and pilot study were conducted by the GME department and research team at no extra cost and without financial incentives provided to study participants. As development utilized internal resources, there were no additional direct costs. Rollout occurred in January 2024, accompanied by a faculty development session. GPS is accessible to 503 personnel across all 246 training programs at the institution. This includes 243 PDs, 110 PCs, 97 program improvement plan (PIP) faculty reviewers, 27 GME team members, 14 vice chiefs/chairs of education, 7 GME directors, 3 shared service team members, and 2 other leaders. Since launch, the GPS has been accessed by 309 unique users (61.4%) with 3933 total views over the first 22 months (Figure 2), indicating strong institutional uptake. Acceptability was demonstrated by full participation of pilot users and subsequent engagement by institutional leaders.

Figure 2.

GME Program Scorecard User Data Since Launch

Discussion

The GPS enables real-time visualization of educational metrics across all residency and fellowship programs within a large sponsoring institution. Notably, the GPS was built without requiring new data entry, leveraged existing infrastructure, and reflected metrics prioritized by a diverse group of participants—features that distinguish it from prior tools.

Prior efforts to assess GME program performance have often relied on static reports emphasizing board pass rates, Match data, and other parameters easily measured by narrow metrics.^1,2 While dashboards and report cards have been explored in GME, they are often institution-specific, resource-intensive, or limited to compliance monitoring.^8-10,12,13 Our GPS advances these efforts by integrating data from 13 systems and enabling both institutional and program-level benchmarking through dynamic visualization. Notably, several GPS metrics align with key performance measures identified through a national survey of designated institutional officials as most important for the Annual Institutional Review, including ACGME citations, scholarly activity, board pass rates, and faculty development,¹⁶ which suggests the GPS may also support institution-wide accreditation reporting needs in addition to program-level evaluation. Furthermore, participants shaped the design and content of the GPS, reinforcing its relevance and usability across diverse program types. While Educational Competency Committees (ECCs) typically require time-intensive reviews and rely on global performance judgments,^7,8 the GPS allows program leaders and administrators to access actionable, regularly updated data tailored to their roles. The GPS does not obviate the need for ECCs, but rather expedites and enhances their function.

Several limitations affect interpretation and generalizability of the GPS. First, our pilot and metric prioritization process focused on large programs, which may have different needs than smaller programs. Although the GPS has since been accessed by leaders from programs ranging from 1 to 168 trainees, the emphasis on scalable metrics may not fully address the contextual nuances of every training environment. Moderate engagement at 61.4% of potential users may reflect some resistance to using the new platform; we also suspect that some programs find their existing systems sufficient and do not wish to make changes to their current workflow. Second, while most GPS data sources update automatically via backend integration, some data—such as board pass rates and ACGME data—require manual updates. These limitations may influence timeliness and consistency. Third, although participant input was central to metric selection, preferences from our institution may not reflect those of other organizations. Additional studies are needed to explore how local priorities and system architecture influence dashboard design and impact as well as to establish the generalizability of this experience to other institutions and users.

Thus far, the GPS has been used by program leaders to complete several program evaluation processes: Program Evaluation Committee meetings, PIP process, Annual Program Evaluations, annual peer reviews of programs, and the major changes portion of the ACGME Accreditation Data System update. While the GPS can facilitate a more efficient and targeted program evaluation and reveal key areas for improvement, the next phase of this project aims to assess the downstream impact of the GPS on actual program improvements, such as whether underperforming metrics improve after targeted interventions. We continue to solicit feedback at meetings with PDs, PCs, and vice chairs of education about how to make the GPS more effective. Exploring how program leaders use the GPS in decision-making, accreditation preparation, or faculty development may also reveal use cases with the greatest institutional value. Additional studies comparing different implementation approaches across sponsoring institutions could support the development of GME dashboard best practices.

Conclusions

The GPS is a centralized, participant-informed dashboard that enables real-time monitoring of educational metrics across diverse residency and fellowship programs. Early usage data suggest that the tool is feasible, scalable, and responsive to user needs.

Author Notes

Funding: The authors report no external funding source for this study.

Conflict of interest: The authors declare they have no competing interests.

Disclosure: AI was not used in writing this article.

This work was previously presented at the Central Group on Educational Affairs meeting, April 11, 2025, Detroit, Michigan, USA.

Editor’s Note

The online supplementary data contains the survey used in the study and the GPS prototype review form.