Measuring the impact of an HIV rotation on trainees’ knowledge and confidence level: The importance of inviting recent graduates

Abstract

Background

The new Canadian Residency Accreditation Consortium (CanRAC) standards recommend surveying recently graduated trainees to target improvements in training programs. The goal of this study was to estimate the impact of a rotation in an HIV clinic on trainees’ related knowledge, confidence, and practice profile at the Université de Montréal.

Methods

An electronic survey was sent to practising physicians who completed the rotation between 2006 and 2016. Participants were asked to rate their agreement and level of confidence toward HIV- and HCV-related topics using 5-point Likert scales (0 to 4). Descriptive statistics and mean comparisons were calculated.

Results

Among invited participants, 27 of 45 (60%) completed the questionnaire. The majority of respondents were infectious diseases physicians (48%) or family physicians (37%) and had an outpatient caseload of <10 HIV patients/year (80%). For 37% of the respondents, the rotation had a large or very large impact on their career path. They considered that the rotation had increased their knowledge on the overall management of HIV (mean 3.2/4 [95% CI 2.9 to 3.4]), but less on pre-exposure prophylaxis (PrEP) (mean 1.5/4 [95% CI 1.1 to 2.0]) or HCV care (mean 1.9/4 [95% CI 1.4 to 2.3]). Participants felt less confident with genotyping interpretation (mean 2.6/4 [95% CI 2.2 to 2.9]) and PrEP (mean 2.4/4 [95% CI 2.0 to 2.8]).

Conclusions

These results suggest that a rotation in an HIV clinic improves knowledge related to HIV care. Feedback from past graduates helped us identify gaps in knowledge or level of confidence in PrEP and HCV care, which will feed curriculum improvement.

New standards from the Canadian Residency Accreditation Consortium (CanRAC) will be used for accreditation of residency programs starting July 2019 (1). These standards include a new domain focusing on continuous improvement and quality assurance of training programs. Among the updated standards are some exemplary measures, including one stating that “Feedback from recent graduates is regularly collected/accessed to improve the residency program” (1 p.20). It has been demonstrated on numerous occasions that such feedback provides valuable information to improve one’s curriculum (2,3). A common way to consider program evaluation is through Kirkpatrick’s four levels: 1) reaction (satisfaction), 2) learning (knowledge or skills), 3) behaviour (applying learning in practice), and 4) results (impact on society/health care) (4). In our institution, standard program evaluation collects many short-term data on a monthly basis targeted at satisfaction of trainees and teachers (level 1), but much less data regarding long-term outcomes, such as their practice profile (levels 3 and 4).

As lifespan of compliant patients living with HIV now resembles the one of any normal individuals, care has shifted to a chronic condition which increasingly requires care by non-infectious diseases (ID) physicians, such as family physicians and internists (5,6). Various methods have been proposed to try and improve training in HIV care for these physicians, either with a specialized track (7), electives (8), online modules, or short courses (9). Although HIV care training is usually mandatory for ID or medical microbiology trainees, such rotation remains optional for others, but may have an influence on their subsequent career choice or practice profile (10). The Centre Hospitalier de l’Université de Montréal (CHUM) is offering electives dedicated to HIV and chronic hepatitis care through a dedicated unit, called Unité Hospitalière de Recherche, d’Enseignement et de Soins sur le SIDA (UHRESS). Yearly, up to nine trainees are offered a 1-month rotation including HIV clinics with HIV-specialized physicians, but also discussions and contact with an interdisciplinary team comprising pharmacists, nurses, psychologists, social workers, and so forth.

As part of the evaluation of this HIV clinic rotation, we obtained recent graduates’ perceptions of their knowledge and practice choices regarding HIV medicine. The research questions were as follows:

• Does an HIV rotation improve trainees’ knowledge of HIV-related topics?

• How confident are young physicians in various HIV-related topics in practice?

• Does an HIV rotation have an impact on trainees’ career choice?

• Are there differences between ID trainees and other trainees regarding the impact of an HIV rotation?

Methods

Survey

This cross-sectional study was conducted in the form of an online survey using SurveyMonkey (www.surveymonkey.com). The survey was developed by the authors and was first piloted with trainees who were still in residency/fellowship in order to have feedback on the clarity of the questions and to ensure functionality of the electronic platform. A series of 35 questions and statements were written to assess respondent’s perception of: confidence toward various topics related to HIV care and management, perceived knowledge improvement after participating in the rotation and impact on career choice. These items were rated by participants using 5-point Likert scales (0 to 4) (questionnaire available in Supplemental Appendix 1). Demographic data and practice profile information were also collected. There were between 2 and 14 questions per page, and 13 pages in total. It was not possible for participants to go back between pages once they were completed. The only questions that could not be skipped were the one on informed consent, a question about being in active practice, and one confirming that they did the rotation. Adaptive questioning was used to reduce the number of questions for physicians not seeing patients in their regular practice. Answers to the questionnaire were anonymous, with the exception of a voluntary participation to a draw ($40 gift card for an online bookstore) for which e-mail addresses were collected and stored on a secured server at the CHUM. The study protocol was approved by the ethics committee of the CHUM. The first page of the survey provided the participants with all relevant information regarding the study. Participants gave informed consent by clicking on a designated button on that page.

Data collection

The respondents were former trainees of the UHRESS who completed the rotation between 2006 and 2016. Only physicians in active practice were included in the study. The list of potential respondents was determined from the UHRESS internship lists of previous years (available since 2011), lists of former medical microbiology and ID residents (since 2006), and data available in the documents of the UHRESS administrative officer and training supervisors (since 2006). Further research was conducted on the Québec Medical Microbiologists Association’s website to try to retrieve as many e-mail addresses as possible.

Invitations for the online survey were sent by e-mail on April 4, 2017, and the survey platform remained active for 4 weeks. Reminder e-mails were sent during the second and third weeks after the start of the response period. Each participant could complete the survey only once through an anonymous respondent ID linked to the participant’s IP address.

Data analysis

Only completed questionnaires were analyzed. Standard descriptive statistics were calculated. Likert scales were transformed into numerical scores (0 to 4) to allow further statistical analyses (11). The 95% confidence intervals for means were bootstrapped to account for the non-normal distribution of data. In addition, perceived knowledge improvement and confidence with relevant topics were compared between ID specialists and other participants using the Mann–Whitney U test. A non-parametric test was used because of the ordinal nature of the data and the small sample size. Analyses were performed using SPSS for Macintosh, version 20.0 (IBM Corp., Armonk, New York).

Results

Demographics and practice profile

Among the 45 participants who received the invitation, 27 were in active practice and completed the questionnaire (60.0%) (Table 1). The majority of respondents were female (59.3%) and 30–39 years old (70.4%). They mostly practised in the Montréal area (63.0%) in a teaching hospital (63.0%). Most of the former trainees (48.1%) were now ID physicians and medical microbiologists (which is usually a combined training program in Québec) or family physicians (37.0%). Two respondents (7.4%) were now in public health medicine and were not seeing patients as part of their practice. Among the 25 respondents who were currently seeing patients in their practice, 20 (80.0%) had an annual outpatient caseload of patients living with HIV of less than 10, and 17 (68.0%) had seen at least one HCV-infected patient in the last year. Only a third of the respondents (33.3%) are currently taking care of HCV-infected patients by themselves and only a quarter (25.9%) had already initiated pre-exposure prophylaxis (PrEP).

Table 1:

Participant demographics and practice profiles

Characteristic	No. of participants (%) (n = 27)
Male	11 (40.7)
Age, years
<30	6 (22.2)
30–39	19 (70.4)
40–49	2 (7.4)
Training program
Infectious diseases/medical microbiology	13 (48.1)
Family medicine	10 (37.0)
Public health	2 (7.4)
Internal medicine	2 (7.4)
Current practice environment*
Teaching hospital	17 (63.0)
Specialized HIV clinic	3 (11.1)†
Addiction clinic	5 (18.5)†
Specialized STD clinic	4 (14.8)†
Annual HIV outpatient caseload
None	9 (33.3)
<10	11 (40.7)
10–50	2 (7.4)
>50	3 (11.1)
Not applicable‡	2 (7.4)
Managed at least one occupational PEP in the last year	12 (44.4)
Managed at least one non-occupational PEP in the last year	16 (59.3)
Managed at least one PrEP in career	7 (25.9)
Number of HCV-infected patients seen last year
None	8 (30.0)
1–2	5 (18.4)
3–10	6 (22.1)
>10	6 (22.1)
Not applicable‡	2 (7.4)
Management of HCV-infected patients§
Refer to a colleague	7 (41.2)
Take care of the patient	9 (52.9)
Missing	1 (5.9)

^* Participants could choose more than one answer (total >100%)

^† All positive answers coming from family physicians

^‡ Public health specialists not seeing patients in their practice

^§ Responses available for 17 participants

STD = sexually transmitted disease; PEP = post-exposure prophylaxis; PrEP = pre-exposure prophylaxis

Satisfaction and knowledge

Most of the respondents were satisfied (37.0%) or very satisfied (55.6%) with their HIV clinic rotation. Respondents considered that the rotation had mostly increased their knowledge about the overall management of HIV-infected patients (mean 3.15/4 [95% CI 2.93 to 3.37]), but not that much on PrEP (mean 1.57/4 [95% CI 1.08 to 2.08]) or HCV care (mean 1.85/4 [95% CI 1.41 to 2.30]) (Table 2). It seems that former ID trainees considered the rotation significantly more useful to increasing their knowledge about antiretroviral therapy (ART) side effects (p = 0.019). On the other hand, trainees from other programs thought that they learned more about HCV management than their ID counterparts (p = 0.048).

Table 2:

Knowledge improvement about HIV-related domains

	Knowledge improvement attributed to the rotation*
Domain	Medical microbiology and ID program (95% CI) (n = 13)	Other programs (95% CI) (n = 14)	Overall (95% CI) (n = 27)	Mann–Whitney U test p-value
HIV screening criteria	2.15 (1.50–2.70)	2.79 (2.40–3.13)	2.48 (2.11–2.81)	0.094
HIV diagnostic tests	2.46 (2.00–2.90)	2.71 (2.33–3.08)	2.59 (2.30–2.85)	0.350
HIV genotype interpretation	3.15 (2.79–3.50)	2.50 (2.00–3.08)	2.81 (2.44–3.18)	0.128
Pre-exposure prophylaxis (PrEP)	1.54 (0.82–2.30)	1.50 (1.00–2.00)	1.57 (1.08–2.08)	1.000
Post-exposure prophylaxis (PEP)	2.25 (1.64–2.86)†	2.14 (1.60–2.60)	2.19 (1.77–2.54)	0.781
ART initiation	3.15 (2.83–3.47)	2.57 (2.21–3.00)	2.85 (2.56–3.11)	0.061
ART side effects	3.31 (2.75–3.71)	2.57 (2.17–2.92)	2.93 (2.59–3.22)	0.019
HIV overall management	3.38 (3.00–3.73)	2.93 (2.68–3.15)	3.15 (2.93–3.37)	0.085
Opportunistic infections treatment	2.23 (1.67–2.75)	2.43 (1.92–2.86)	2.33 (1.96–2.67)	0.583
HCV global management	1.46 (0.69–2.30)	2.21 (1.81–2.67)	1.85 (1.41–2.30)	0.048

^* Items were rated on a 5-point Likert scale (0 = none, 1 = a little, 2 = moderately, 3 = greatly, 4 = entirely)

^† Calculated with 12 respondents (one missing data)

ID = infectious diseases; ART = antiretroviral therapy

Confidence level

Topics that the participants felt the least confident about were genotyping interpretation (mean 2.59/4 [95% CI 2.30 to 2.96]) and PrEP (mean 2.41/4 [95% CI 2.04 to 2.81]). (Table 3) There were significant differences between physicians trained in ID and other programs with regard to some areas. Overall, ID physicians felt more confident about genotyping interpretation (p = 0.007), post-exposure prophylaxis (PEP) (p = 0.006), and management of opportunistic infections (p = 0.007).

Table 3:

Confidence level regarding HIV-related domains

	Confidence level*
Domain	Medical microbiology and ID program (95% CI) (n = 13)	Other programs (95% CI) (n = 14)	Overall (95% CI) (n = 27)	Mann–Whitney U test p-value
HIV screening criteria	3.69 (3.42–3.92)	3.36 (3.00–3.67)	3.52 (3.26–3.70)	0.220
HIV diagnostic tests	3.85 (3.62–4.00)	3.43 (3.15–3.69)	3.63 (3.44–3.81)	0.068
HIV genotype interpretation	3.08 (2.75–3.44)	2.14 (1.70–2.61)	2.59 (2.30–2.93)	0.007
Pre-exposure prophylaxis (PrEP)	2.46 (1.92–3.00)	2.36 (1.79–2.93)	2.41 (2.04–2.81)	0.830
Post-exposure prophylaxis (PEP)	3.46 (3.09–3.77)	2.36 (1.86–2.92)	2.89 (2.52–3.26)	0.006
ART initiation	3.31 (3.07–3.60)	2.93 (2.38–3.44)	3.11 (2.81–3.41)	0.458
ART side effects	2.92 (2.47–3.36)	2.43 (2.00–2.92)	2.67 (2.30–3.04)	0.169
HIV overall management	2.62 (2.08–3.13)	2.71 (2.38–3.06)	2.67 (2.37–2.93)	0.905
Opportunistic infections treatment	3.23 (2.92–3.55)	2.07 (1.47–2.64)	2.63 (2.19–3.00)	0.007

^* Items were rated on a 5-point Likert scale (0 = not confident, 1 = a little confident, 2 = neutral, 3 = confident, 4 = very confident)

ID = infectious diseases; ART = antiretroviral therapy

Practice choice impact

Retrospectively, 37.0% of the respondents considered that the HIV rotation had had a large (22.2%) or a very large (14.8%) impact on their practice choice. Overall, internists considered that the rotation had a large or very large impact on their career path, while most of the family physicians considered that the rotation had a medium impact (Figure 1).

Figure 1:

Impact on career choice according to training program

Discussion

This study describes a way to achieve one of the exemplary quality improvement requirements of the CanRAC; that is, by distributing a survey to trainees that participated in an HIV rotation across a 10-year time frame in order to get their feedback. Overall, trainees recalled the HIV rotation as a positive experience. Respondents considered that the rotation had some impact on their career profile. Whether this impact was positive or negative remains unclear, as many of them have a low caseload of HIV or HCV patients in their current practice. It has already been shown that the majority of family physicians in Ontario have a low-volume caseload (5) and the situation is likely to be similar in Québec. Overall, participants felt that the rotation helped them to improve their knowledge in many topics around HIV medicine. We identified some differences in perceived knowledge increase and confidence level between physicians trained in ID and physicians trained in other programs.

ID physicians felt more confident about some topics, particularly genotyping interpretation, PEP, and management of opportunistic infections. This is probably explained by their practice profile. Indeed, medical microbiologists and ID physicians usually have a hospital-based practice and are more often exposed to the management of opportunistic infections and occupational PEP. However, since PEP is also quite common in a non-occupational setting, we believe that it is important for non-ID physicians to be confident with this topic, especially since good national guidelines are now available (12). We therefore plan to put more emphasis on this topic during the rotation to ensure that general practitioners or internists participating in the rotation will feel confident about PEP.

Not surprisingly, gaps were identified for HCV treatment (knowledge increase) and PrEP (knowledge and confidence), two topics that have gone through significant developments over the last few years. As HIV-infected patients are disproportionately affected by HCV (13), an HIV rotation is a good opportunity for exposure to HCV-infected cases and treatments. It is very likely that, with the rapid development of direct-acting antivirals for HCV, trainees’ experience will improve, as they will be more often exposed to patients under treatment. It is interesting to note that ID physicians felt that the HIV rotation had less impact on their knowledge improvement about HCV. Whether this is educationally significant or not with our small sample remains unknown. However, one possible explanation is that ID trainees had a higher baseline knowledge regarding HCV and that the exposure in the rotation was perhaps too small to help them progress. Similarly, PrEP development has been very significant over the last few years, with results of trials such as iPrEx (14) and IPERGAY (15). Canadian PrEP guidelines have also recently been published (12). In the setting of a rotation at UHRESS, few non-HIV-infected patients are seen, and exposure to candidates for PrEP is low. However, considering the importance of this topic for HIV prevention, improvements of PrEP exposure in the rotation would likely need to be increased through collaboration with other clinics or formal teaching modules (16).

This study has several limitations that every small residency program is likely to face when trying to do a program evaluation. Our study was single-centred and had a relatively small number of respondents, making statistical analysis and interpretation limited. It is possible that some e-mail addresses that we had on file were not used anymore by the former trainees, limiting the pool of participants. It is also likely that a responder bias was present; that is, people who enjoyed the rotation the most, or for whom it had the greatest impact took time to fill out the survey. A bias can also result from the fact that the HIV rotation was mandatory for ID physicians and elective for most of the other respondents. Those who elected to do an HIV rotation might have had more interest at baseline for this topic and may have been more likely to consider it as a potential career path. Moreover, a social desirability bias remains possible, as with any self-administered questionnaire. Since questions were often asked many years after the rotation and data are self-reported, it is highly likely that a recall bias was also present. However, young physicians had the opportunity to consider the rotation in the context of their entire residency program and actual practice, which may help adjusting the content to what is relevant in real-life practice. The need to combine several cohorts to increase numbers added another bias: exposure to various topics and their management have likely changed over a 10-year period, especially regarding PrEP or HCV care. A similar challenge was faced by Curran et al (2) in an obstetrics and gynecology program, who suggested that a 5-year period would be more appropriate. However, this is difficult to achieve for small programs. The lack of control group also prevents us from concluding that learning or confidence was solely improved by our rotation, and not by other learning activities undertaken by the participants.

Conclusions

In conclusion, gathering feedback from past graduates is feasible and will become more important with the new Canadian accreditation standards for residency programs. It offers a good opportunity to identify gaps in knowledge or level of confidence in people who have a “real-life practice” (PEP, PrEP, and HCV care, notably, in our case). Input from young clinicians can therefore provide valuable information to help improve the overall curriculum and, as in our case, adjust it to the background of the trainee.

Funding Statement

VML is supported by a Chercheur boursier clinicien Junior 1 awards from the FRQ-S.

Competing Interests:

VML is supported by a Chercheur boursier clinicien Junior 1 awards from the FRQ-S.

Ethics Approval:

The study protocol was approved by the ethics committee of the Centre Hospitalier de l’Université de Montréal (CHUM). The first page of the survey provided the participants with all relevant information regarding the study. Participants gave informed consent by clicking on a designated button on that page.

Informed Consent:

Informed consent was obtained from the participants.

Registry and the Registration No. of the Study/Trial:

N/A

Animal Studies:

N/A

Funding:

VML is supported by a Chercheur boursier clinicien Junior 1 awards from the FRQ-S.

Peer Review:

This article has been peer reviewed.