Development of a Novel Competency-Based Evaluation System for HIV Primary Care Training: the HIV Entrustable Professional Activities

Abstract

Background

There is an anticipated shortage of primary care providers trained to care for patients with HIV. The Yale School of Medicine developed and implemented a novel HIV training track within our Primary Care Internal Medicine Residency Program. A set of 12 Entrustable Professional Activities (EPAs) were developed to guide curriculum development and resident assessment.

Aim

To describe the process of implementing a novel EPA-based curriculum for the HIV Primary Care Training Track including EPA-based trainee evaluation tools.

Participants/Settings

Two to three residents were enrolled annually from 2012 to 2017 (total n = 11). Training sites included the outpatient academic center HIV clinic and inpatient HIV ward.

Program Description

An expert panel developed 12 HIV-specific EPAs. These were mapped to curricular and reporting internal medicine milestones. Curricular activities and evaluation tools were developed to guide EPA progress.

Program Evaluation

Graduating residents were ready for unsupervised practice in 91% of EPAs at the end of the 3-year program.

Discussion

Development of HIV-specific training EPAs was effective for driving curricular development and resident evaluation, and served as an effective method to communicate expectations to resident participants. These HIV-specific EPAs could serve as a useful template to enhance HIV education in academic settings.

Electronic supplementary material

The online version of this article (10.1007/s11606-019-04956-1) contains supplementary material, which is available to authorized users.

INTRODUCTION

In response to a projected shortage of primary care physicians trained to effectively care for patients with or at risk for HIV, the Health Resources Service Administration (HRSA) offered competitive funding to develop residency training programs to train HIV primary care providers.¹ After successfully competing for this 4-year grant, the Yale Department of Internal Medicine embarked on curriculum planning for the HIV Primary Care Training Track. We sought to develop the curriculum based on the expected professional activities of an ideally trained primary care provider for HIV-infected patients. A separate manuscript describing the implementation and evaluation of the Yale HIV Primary Care training program structure has been published.²

The Accreditation Council for Graduate Medical Education (ACGME) currently requires residency training programs to evaluate trainees in six core competencies. In internal medicine (IM), the ACGME and American Board of Internal Medicine (ABIM) developed 22 reporting developmental milestones that contextualize these competencies.³ Because assessing some competencies require high inference by faculty, there have been recent efforts to integrate these competencies into discrete, directly observable activities that are critical for deeming a trainee entrustable for independent practice.^{4, 5} These Entrustable Professional Activities (EPA) are now being developed across a wide variety of specialties^5–7 as well as for undergraduate medical education.^{8, 9}

We report on the development of HIV-specific training EPAs and our experience with evaluation “bundles” developed to inform these EPAs, our early experience with tracking individual progress towards entrustment, and how the EPAs informed trainee and program evaluation. It is important to note that this HIV-specific training is embedded within the residents’ 3-year Primary Care Residency (parent) program requirement and is therefore a supplement to core residency training. The core program is also EPA based and shared competencies within the HIV training EPAs were included in the parent program biannual resident evaluations. The HIV Primary Care Training Track curriculum is accomplished during ambulatory immersion blocks, longitudinal clinic sessions, select HIV inpatient ward months, and a variety of other curricular activities.²

SETTING AND PARTICIPANTS

This educational innovation occurred within the Yale Primary Care Program with clinical and didactic experiences both in the inpatient setting at Yale New Haven Hospital and its embedded HIV ambulatory clinic (Nathan Smith Clinic). Eleven residents were recruited to the Yale Primary Care Program HIV Training Track over a 5-year period (2012–2017).

METHODS

We developed the HIV EPAs in an iterative process. The criteria for a clinical activity to be considered an EPA⁵ were disseminated to five Yale University School of Medicine AIDS Program faculty as well as to two Infectious Diseases faculty members. Twenty-five EPAs were identified after the first round of input. Four faculty members assigned to develop the HIV Training Track (DD, LB, MG, JT) winnowed the number by strictly applying the EPA criteria⁵, prioritizing the activities, and considering the number of EPAs manageable to track. This core group of faculty also determined and mapped the applicable curricular and reporting milestones for each EPA. The resulting document was programmed into our residency education management service (MedHub; Ann Arbor, MI) to generate a document that could automatically track current ACGME-required reporting milestones. Educational activities were selected to assist the resident in attaining entrustment in each EPA and were “bundled” with each EPA on MedHub for resident and advisor tracking and documentation.

RESULTS

• A.HIV EPAs—Twelve HIV Training Track EPAs were identified. The salient knowledge, skills, and attitudes required for each HIV EPA are summarized in Table 1 along with the associated activity and evaluation bundles.
• B.
  Activity bundles—For each EPA, a variety of activities were designed to help the trainee gain knowledge and skills/experiences necessary to progress towards entrustment within that activity. All content and resources were organized and available on the training track website (https://medicine.yale.edu/intmed/hivtraining/).

  1. EPA-specific HIV modules—Twenty-four topics were identified for case creation by the cohort of HIV training track residents. Residents were each responsible for writing two to three modules per year (formatting based on Yale Office Based Medicine, a nationally distributed curriculum with which the residents were very familiar) until all 24 were developed. Additionally, residents accessed computer-based interactive cases (International Aids Society-Cases on the Web and National HIV Curriculum) highlighting any additional content of each EPA not covered in resident-developed cases. After completing each case, residents were asked to document completion and comprehension by taking the post-test the successful completion of which generates a CME certificate (scanned into their educational portfolio).
  2. Deliberate practice with standardized patients (SP)—For two of the EPAs, scripts and SP roles were developed to exemplify (1) history-taking skills/difficult news-exploring risks for HIV acquisition emphasizing sex and drug-use history and giving positive results (EPA #1) and (2) initiating new medications (EPA #6 and others). These SP sessions were completed in the first 3 months of the first year with structured faculty observation and formative feedback/debriefing.
  3. HIV preclinic conference—Weekly HIV-related topics that directly relate to EPAs were discussed in a small group setting.
  4. Experiential—residents attended one half day per week in an HIV continuity clinic for 3 years with the aim of an eventual patient panel of 25–30 patients. In addition, residents had immersion blocks in which they spent 3 four-week blocks in the ambulatory setting. Residents were also assigned to the inpatient HIV ward 4–6 weeks per year. The details outlining activities and experiences afforded during the ambulatory immersion and inpatient HIV medicine blocks are outlined in our program description.²

• C.Evaluation

Table 1.

12 HIV EPAs and Activity and Assessment Bundles

EPA—description	Activity/assessment tools
1. Perform HIV counseling and testing—Residents need to incorporate knowledge of the performance characteristics of the HIV test used in the context of the prevalence of HIV in the population tested. Informing the patient about test results in a culturally competent manner including making the patient aware of the legal ramifications of a positive test is included in this EPA. Additionally, communication with the State Department of Public Health (DPH) for reporting and partner notification services is stressed.	• Yale HIV Testing and Screening Module • SP session #1—HIV testing-counseling/results • Framework—breaking bad news
2. Assess patients with newly diagnosed HIV infection—gain and demonstrate the knowledge necessary to stage HIV illness, determine the choice of antiretroviral treatment, opportunistic infection (OI) prophylaxis, and other health interventions such as immunization and screening for concomitant infections.	• Yale ART Initiation module • SP session #2—New Patient • Framework—information gathering
3. Counsel a patient on reducing high-risk behaviors—address initial and ongoing importance of risk reduction to decrease (1) transmission of HIV to uninfected sex or needle-sharing partners and (2) acquisition of new infections that can complicate HIV management such as STDs and/or viral hepatitis. Proficiency in behavioral counseling such as motivational interviewing and Stage of Change model is expected as well as ongoing culturally sensitive sexual and drug-use history taking.	• Motivational interviewing • Yale STI screening module • SP session #2—New Patient
4. Provide mental health and substance abuse management in clinical setting. Given the significant known prevalence of mental health and substance use comorbidity in the HIV-infected population, it is important to demonstrate effective utilization of case management and consultative psychiatry services, as well as interprofessional team management.	• SBIRT session and SP • IAS-USA COW#62, 8
5. Provide appropriate age-based and HIV-associated preventative services to patients. Includes activities that primary care practitioners include in the day-to-day outpatient care of patients additionally emphasizing where those preventive recommendations might differ for those with HIV infection.	• Yale cervical and anal cancer screening module • IAS-USA COW# 75, 77 • Smoking cessation
6. Initiate and monitor antiretroviral (ARV) treatment—Residents are expected to accurately recognize indications for and barriers to initiation of ARV. They should be able to construct and communicate an antiviral drug regimen based on current US guidelines, patient preference, and possible drug-drug interactions.	• Yale ART initiation module • IAS-USA COW#59, 47 • SP session #2—New Patient • Framework—information transfer/starting new med
7. Provide prophylaxis and treatment of opportunistic infections—Knowledge regarding OI risk relative to degree of immunosuppression, history and physical exam skills required to recognize and diagnose OI, and treatment and prevention of these OIs are encompassed	• Yale ART initiation/OI treatment (MAC, cryptococcal meningitis) modules • Chart stimulated recall (CSR)/chart audit
8. Assess and manage patients with HIV-associated metabolic disorders (lipids, HTN, CAD risk reduction, bone, renal, etc.)—recognition, diagnosis, and management of HIV metabolic disorders including diabetes, metabolic bone disease, and dyslipidemias.	• Preclinic Conferences • IAS-USA COW #23, 24, 64 • Chart Stimulated recall (CSR)/Audit
9. Provide effective end-of-life/palliative care to patients—Competence in palliative care/EOL here includes pain management, referral to palliative care services, family meetings, and utilization of services as patients become less independent towards the end of life.	• IAS-USA COW# 63
10. Manage HIV infection in setting of reproductive issues (infertility, pregnancy)—includes minimizing risk of HIV transmission to uninfected partners when trying to conceive, family planning/birth control options, management of the pregnant HIV-infected woman, and comanagement (in some settings) with high-risk obstetrical services.	• MFM presentation • IAS-USA COW #51, 42
11. Manage patients coinfected with HIV-HCV—effectively diagnose and manage the HIV-HCV-coinfected patient which requires knowledge of most recent indications for treatment, drug regimens, potential drug interactions, and indications for organ transplantation.	• HCV-HIV coinfection clinic • HIV transplant clinic • Yale HIV HCV module • IAS-USA COW #60,70
12. Provide care to GLBTQ patients with HIV—can incorporate recognition of disproportionate prevalence of STI, domestic/partner violence, drug and alcohol abuse, and mental health issues including suicidal tendencies. Additionally, competence in gender identity sensitivity, transgender care including hormonal replacement therapy and possible complications thereof, and referral to surgical services when requested are included in this domain.	• Yale sexual history taking/STI module • Transgender care preclinic conference

IAS-USA COW International AIDS Society-USA Case on the Web, ARV antiretroviral, SBIRT substance use brief intervention and treatment, SP standardized patient, MAC Mycobacterium avium complex, PJP pneumocystic jivecii pneumonia, TE toxoplasma encephalitis)

Progress towards and determination of entrustment for each HIV EPA were accomplished formatively by a combination of direct bedside/clinic setting observation by faculty preceptors (utilizing frameworks focused on patient-centered interviewing skills, general history-taking skills, physical exam skills, breaking bad news, and initiating new medications, Table 1), patient management discussion and chart review, and patient and staff feedback (surveys). Following clinical encounters, residents logged the case on MedHub which generated the appropriate corresponding EPA form. Faculty completed the evaluation form which placed the resident into the developmental framework and provided specific behavioral feedback and suggestions (action plan). Documentation of structured observations was therefore saved to the trainee MedHub portfolio. Formative feedback was given immediately after observing residents.

While the HIV-TT trainees were included in the larger Yale Primary Care residency biannual review and ACGME reporting, we instituted a parallel HIV Training Track Clinical Competency Committee (CCC) to more comprehensively review trainee progress in the 12 HIV EPAs. Results from this biannual review were used for formative feedback to the individual resident and were used for program evaluation. This committee consisted of the academic advisor, other HIV clinic faculty, nursing, and the Program Director (non-voting). The final meeting of the CCC (end of year 3) for that resident served as the summative determination of entrustment. The HIV CCC pooled together HIV EPA logs and direct observation forms by clinic preceptors to plot progress in each EPA using narrative milestones (Table 2). Not surprisingly, the number of logged observations varied widely by particular EPA and by PGY level. Most residents who completed training or were in their final year (n = 7) reported exposure and content experience with all 12 HIV EPAs. To date, seven residents have completed the HIV TT Primary Care residency and data for the two most recent graduating cohorts (n = 5) on EPA performance is shown in Fig. 1. The level determined to be consistent with being “ready for independent practice” (i.e., entrustable) was set at 4 with a 5 defined as “aspirational” (consistent with terminology and behaviors outlined in ABIM Internal Medicine Milestones³). One hundred percent of the graduating residents were entrustable or aspirational in 91% (i.e., 11 out of 12) of the EPAs.

Table 2.

Example HIV EPA Mapping and Behavioral Descriptors (See Appendix for Full Set)

Entrustable professional activity		Clinical deficiencies	Early learner	Advancing and demonstrating improvement	Ready for unsupervised practice	Aspirational
Assess patients with newly diagnosed HIV infection		Unable to recognize and assess newly diagnosed HIV infection despite guidance, the provision of a template, and orientation to the assessment	Identifies some signs and symptoms consistent with acute HIV infection or new diagnosis but does not accurately characterize entire syndrome. Unable to appropriately stage patient with necessary panel of blood tests or interpret baseline lab tests including HIV genotype testing	Understands when to initiate opportunistic infection prophylaxis and combined antiretroviral therapy but not able to choose the appropriate therapeutic regimen. Accurately interprets HIV staging tests	Develops plan to initiate OI prophylaxis and, using baseline genotype testing, develops a therapeutic approach to initiate combined antiretroviral treatment	Tailors newly diagnosed HIV assessment to particular patient presentation and clinical circumstances. Considers patient preferences and education level when devising a treatment plan
ACGME core competencies	IM milestones
PC, MK, IC	PC A1-4, B1, B3-4, C1-4, E1-2, F1-10; MK A1-9, B1-3; ICS A1-8, B2-3
Please give 2 or more examples of specific observed behaviors that exemplify this residents developmental level in this competency

Figure 1.

At the completion of the 3-year HIV Primary Care Residency, final assessment of competency (scale 1–5 as defined in the text) was determined for each of the 12 EPAs. Data is shown for the first two evaluable cohorts of residents (n = 5). The average number of observations per resident per EPA was 7.7 (range 2–15). There were no observations done for two graduating residents on one out of the 12 (8%) EPAs. With a competency level of “4” (ready for unsupervised practice) = entrustable

Information on gaps in skills and/or opportunities were identified through this process. The number of work-based assessment and mini-CEX events logged per resident per EPA was variable. Consistent with expected clinical work of routine HIV care, some EPAs were performed frequently (risk reduction, preventive care, ARV adherence/monitoring, mental health/substance use care) and some less (pregnant women with HIV, end-of-life/palliative care). Except for these latter EPAs (#9, 10) all five assessable residents who completed training had an average of 7.7 observations (2–15) per EPA.

DISCUSSION

We describe the first 5 years of our experience with trainee evaluation in a novel HIV Training Track residency program that is embedded in a northeastern US Primary Care Residency Program. The Entrustable Professional Activities construct gave us the opportunity to drive curricular planning to ensure that graduates of this HIV Training Track would possess the requisite skills to provide primary care for people living with HIV and/or those at risk. The strength of this Training Track is our a priori identification of these necessary professional activities (EPAs) to allow residents to gain these skills. Residents understood how learning activities (patient care, classes, modules, talks, SP sessions) contributed to the development of these EPAs and the basis by which they were assessed. Progress along the HIV EPAs provided formative feedback to residents and faculty to adjust and add experiences to assure adequate exposure. Faculty development needs were limited given the intuitive nature of EPAs, the availability of content, and the small core faculty precepting cohort had extensive prior experience with curriculum development, direct observation and feedback, and program evaluation.

A limitation to resident evaluation has been the cumbersome logging process that residents undertake to log patients (which generates the evaluation form sent to faculty who observed) by the EPA encountered after each visit. This limited our ability to consistently capture all of the EPAs that the residents actually performed. A more palatable smartphone platform for logging clinical experiences is in development as are the workplace-based assessments (WBA) for each. Strengthening this loop will afford more robust program evaluation and curricular adjustments going forward. Additionally, the small number of trainees each year does not allow for robust comparison of standardized test scores, passage of HIVMA certification test, and post-graduation competency.

With respect to EPAs, limited exposure to patients with HIV at the end of life may continue to affect our ability to train and assess competency and, because deaths due to HIV have thankfully plummeted, it may be removed as a specific EPA. Conversely, inconsistent exposure to HIV-infected women with reproductive issues has caused us to have our male residents comanage the pregnant women with HIV (who had requested female providers) when possible. Additionally, we note that prevention of HIV through the local development of a robust PrEP program had not been on the radar when developing the initial core EPA list and has subsequently become a significant clinical activity of great importance. We feel the knowledge and skills exemplified in EPA #3 (risk reduction counseling) and 2 (initiating ARV) encompass this new activity.

Despite not reaching 100% compliance with patient logs, the current system does allow faculty and participating residents to gain experience and track progress in most of the 12 HIV EPAs. Future plans include implementation of mobile patient logging and WBA, monitoring performance on summative knowledge assessments (HIVMA certificate exam), and post-graduate career choice and survey regarding how effective the program is in preparing graduates for a career in HIV primary care.

While patient outcomes were not explicitly evaluated, the residents increased clinical capacity and provided care comparable to the faculty providers in several performance measures and patient satisfaction.^{2, 10}

The implementation of this HIV EPA-based curriculum has enabled us to explicitly prepare our trainees to provide the clinical care that HIV-infected patients deserve.

Our early results support the continued use of these EPAs, allow us to strengthen gaps in the curriculum, and may provide a framework for other residency programs wishing to offer enhanced training in HIV primary care.

Funders

We acknowledge AIDS Education and Training Centers (AETC) grant support: H4HA22762.

Compliance with Ethical Standards

Conflict of Interest

DD—spouse holds stock in Pfizer, Inc.

Lydia Aoun Barakat served as a consultant for Gilead Sciences.

All other authors report no conflicts of interest.