Does integrated training in evidence-based medicine (EBM) in the general practice (GP) specialty training improve EBM behaviour in daily clinical practice? A cluster randomised controlled trial

Abstract

Objectives

Evidence-based medicine (EBM) is an important element in the general practice (GP) specialty training. Studies show that integrating EBM training into clinical practice brings larger benefits than stand-alone modules. However, these studies have neither been performed in GP nor assessed EBM behaviour of former trainees in daily clinical practice.

Setting

GP specialty training in the Netherlands.

Participants

All 82 third year GP trainees who started their final third year in 2011 were approached for inclusion, of whom 79 (96%) participated: 39 in the intervention group and 40 in the control group.

Intervention

Integrated EBM training, in which EBM is embedded closely within the clinical context by joint assignments for the trainee and supervisor in daily practice, and teaching sessions based on dilemmas from actual patient consultations.

Comparison

Stand-alone EBM training at the institute only.

Primary and secondary outcomes

Our primary outcome was EBM behaviour, assessed by measuring guideline adherence (incorporating rational, motivated deviation) and information-seeking behaviour. Our secondary outcomes were EBM attitude and EBM knowledge. Data were acquired using logbooks and questionnaires, respectively. Analyses were performed using mixed models.

Results

Logbook data were available from 76 (96%) of the participating trainees at baseline (7614 consultations), 60 (76%) at the end of the third year (T1, 4973 consultations) and 53 (67%) 1 year after graduation (T2, 3307 consultations). We found no significant differences in outcomes between the 2 groups, with relative risks for guideline adherence varying between 0.96 and 0.99 (95% CI 0.86 to 1.11) at T1, and 0.99 and 1.10 (95% CI 0.92 to 1.25) at T2, and for information-seeking behaviour between 0.97 and 1.16 (95% CI 0.70 to 1.91) and 0.90 and 1.10 (95% CI 0.70 to 1.32), respectively.

Conclusions

Integrated EBM training compared with stand-alone EBM training does not improve EBM behaviour, attitude or knowledge of (future) GPs.

Strengths and limitations of this study.

• To the best of our knowledge, this is the first randomised controlled trial among general practice (GP) trainees comparing integrated with stand-alone evidence-based medicine (EBM) training, and assessing both short-term and long-term effects on EBM behaviour, attitude and knowledge.

• We assessed EBM behaviour in daily clinical practice, using a large number and a wide variety of patient consultations, with a long-term follow-up. Prior studies frequently applied (retrospective) questionnaires, in contrast to ‘real-time’ measurements.

• The effects of EBM training on adherence of individual GP trainees to multiple guidelines for a wide range of management decisions have not been evaluated before.

• Motivated deviation from a guideline in individual patients was taken into account in assessing guideline adherence, which has never been done before.

• The choice to perform our study in the third year of the GP specialty training enabled us to assess the effects of the different EBM training programmes until after graduation.

• Large variability in the way some components of the intervention were delivered in the primary care practices may have diluted the effect of our intervention.

• The contrast between both groups may have been limited because our stand-alone EBM training could be considered as ‘integrated’ to some extent, because we do use clinical examples.

• Although we tried to limit the time investment needed for GP trainees to collect data in logbooks, by asking them to copy anonymised data from the electronic medical records, it was substantial. This may have reduced their commitment. Moreover, the copied data may not have adequately reflected guideline adherence, and misclassification may have occurred due to lack of relevant information.

• Although accounted for in our analyses, the high rates of loss to follow-up and difference in preferred type of logbook (digital or paper) at baseline between both groups may have affected the results.

Introduction

Evidence-based medicine (EBM), defined as integrating clinical expertise, patient values and the best available clinical evidence in daily clinical practice, is the cornerstone of today's medical practice.¹ General practice (GP) trainees in the Netherlands learn to work according to the principles of EBM during their 3-year competency-based specialty training.²

EBM is taught according to the five steps as defined in the Sicily Statement: ask, acquire, appraise, apply and assess.³ Despite its undisputed importance, evidence on the optimal method to teach EBM is lacking.⁴ Traditionally, EBM in the GP specialty training in the Netherlands is trained in theoretical, stand-alone educational sessions. This differs from integrated EBM training, in which EBM is trained in a clinical context, and teaching sessions are based on recent patient consultations in the trainees' practice. A review from 2004 reported that EBM knowledge among hospital trainees improved with both stand-alone and integrated EBM training, while skills, attitude and behaviour improved only with integrated EBM training. The authors concluded that “ teaching of evidence-based medicine should be moved from classrooms to clinical practice to achieve improvements in substantial outcomes”.⁵ However, the effectiveness of stand-alone and integrated EBM training was never directly compared in a (randomised) controlled trial; nor were the effects on EBM behaviour of former trainees assessed in daily clinical practice after graduation.^{5 6} Moreover, it is unknown whether integrated EBM training also improves EBM behaviour in GP trainees; in primary care, both the setting (one supervisor and one trainee) and the patient and disease spectrum differ substantially from the hospital environment.

We developed an integrated EBM training for GP trainees with the focus on the last two steps of EBM, that is, apply and assess (table 1). We also trained the supervising GPs, stressing the important role these supervisors have in integrating EBM training into clinical practice.¹¹ We report the results of a cluster randomised controlled trial among third year GP trainees comparing the effects of this integrated EBM training programme with a stand-alone EBM training programme.

Table 1.

Description of the EBM training programmes as taught to GP trainees from the GP specialty training in Utrecht, assigned to the intervention or control group

	Intervention	Control
	Integrated EBM training programme Theoretical basis: Hoes/Grobbee7 and Offringa8	Stand-alone EBM training programme Theoretical basis: Offringa8
Clinical practice (4 days a week*)	Tutorial dialogue with supervisor Every day in practice, 1 hour Integration of EBM in the ‘EBM tutorial dialogue’, once per week with the goal to integrate evidence into the dialogue, combining the clinical expertise of the supervisor with the EBM skills of the trainee6	Tutorial dialogue with supervisor ▸  Every day in practice, 1 hour No integration of EBM No critical appraisal together with the supervisor
	Critical appraisal of an article together with the supervisor, once a month. All trainees received an email every 3–4 weeks with suggestions for articles, and were encouraged to discuss other clinically relevant articles. These articles were mainly retrieved from ‘Behind the headlines’9 and the (last issue of the) national GP journal, Huisarts and Wetenschap.10
Institute (1 day a week)	EBM course, 5 days a year, 2½ hours Focus on the last 2 steps of EBM; translation from evidence into clinical practice Patient-related preassignments and postassignments to perform together with supervisor	EBM course, 4 days a year, 2½ hours Focus on the first 3 steps of EBM Preassignments about imaginary patients
	Exchange of last week's experiences in clinical practice 1 hour every week Integration of EBM for 15 min, with 1 trainee preparing a topic, based on a patient consultation, searching for the evidence base and chairing the session	Exchange of last week's experiences in clinical practice 1 hour every week No integration of EBM
	Coaching session about EBM attitude at the beginning of the year in which trainees were asked to discuss EBM needs and barriers, thus enabling us to tailor the content of the integrated EBM training programme to the needs of the trainees.	–
	Presentation of a CAT	–
Online	Possibility for e-learning expanded and improved	Possibility for e-learning
	Online coaching opportunity	–
Extra	Participation in research (PINET)	Possibility to participate in research
	Possibility for extra EBM course	Possibility for extra EBM course
Supervisors	2 days of EBM training at the institute with the same theoretical basis and focus as the programme of the trainees, including joint assignments for the trainee and supervisor in clinical practice, and with sessions about how to best integrate EBM in clinical practice, how to engage EBM in tutorial dialogues and the importance of being a role model	–
	4 times a year (1 hour) EBM training with before and after assignments for the trainee and supervisor in clinical practice	–

*Based on a full-time working contract.

CAT, critically appraised topic; EBM, evidence-based medicine; GP, general practice; PINET, Personalized INtegrated Evidence-based medicine teaching for Trainees in GP.

Methods

Design

The PINET study (Personalized INtegrated Evidence-based medicine teaching for Trainees in GP) was designed as a cluster randomised controlled trial among third (last) year GP trainees from six different groups (with 12 trainees each) starting at four consecutive time moments. Groups of trainees were randomly allocated to either an integrated (ie, new) EBM training programme (intervention group) or the regular stand-alone EBM training programme (control group).

Study population

The study was performed within the GP specialty training of the University Medical Centre Utrecht (UMCU) in the Netherlands between March 2011 and December 2013, a 3-year specialisation programme for general practitioners. The first and third years consist of in-practice training under the supervision of an experienced GP, while the second year consists of hospital rotations.¹² All 82 third year GP trainees who started their final third year in 2011 were approached for inclusion.

Allocation and data collection

In total, six trainee groups successively entered their third trainee year in 2011 (two groups in March, one in June, two in September and one in December). Since trainee groups of two consecutive time moments (two groups in March and one in June, and two in September and one in December) have joint educational programmes, we clustered the intervention in time, and randomly allocated the first three groups to the intervention arm and the last three groups to the control arm. Data were collected at baseline (T0), at the end of the third year (T1) and 1 year after graduation (T2). Baseline characteristics such as age, sex, medical school, clinical experience and score on the GP knowledge test, a national test that is used to monitor knowledge progress, were collected.¹³ Trainees were also asked to self-assess their EBM attitude and EBM knowledge at baseline on a five-point Likert scale ranging from 1 (very poor) to 5 (very good).

Intervention

The EBM training programme in the intervention and control groups is summarised in table 1. Essential skills, such as searching for evidence, critical appraisal of the literature for different research designs and basic analytic skills, are taught in accordance with the five steps of EBM training as described in the Sicily Statement, with the focus on clinical relevance as (future) GPs.³ The main difference between the stand-alone and integrated EBM training programme is the focus on the last two steps of EBM in the latter training programme. The theoretical basis of the programme was therefore adapted to emphasise the practical implication of research and to stress its clinical relevance.^{7 8} In addition, we changed four components of the (regular) stand-alone EBM training programme and incorporated two new elements in the programme to better link EBM theory to clinical practice (see table 1 for details). Finally, their GP supervisors in primary care practice were involved in the intensive EBM programme as well.

Outcomes

The primary outcome was EBM behaviour, measured as guideline adherence and information-seeking behaviour in the intervention and control groups during follow-up. Secondary outcomes were EBM attitude and EBM knowledge.

Instruments and measurements

EBM behaviour: guideline adherence and information-seeking behaviour

Guideline adherence was defined as the extent to which clinical management was in accordance with key recommendations in the professional GP clinical practice guidelines. This was assessed with a validated instrument that measures adherence with clinical practice guidelines on the¹⁴ most prevalent conditions in GP. The instrument assesses compliance on 59 different management decisions (diagnosis N=17, therapy N=20, referral N=22) as described in 27 different clinical practice guidelines covering the 23 most prevalent conditions (MF Kortekaas, MEL Bartelink, GJMG Van der Heijden, et al. Development and validation of a new instrument measuring EBM behaviour in clinical practice. submitted). Guideline adherence of the participants at T0 and T1 was scored by two independent researchers (YS and LW) for every decision in clinical management (diagnosis, therapy and referral). In case of disagreements, a third researcher (JvD) made a final decision. At T2, guideline adherence was scored by one researcher (JvD). Possible scores were non-compliant (−1), debatable (0) and full-compliant or motivated deviation.¹ Allocated scores were used to assess guideline adherence in five different ways, as shown in table 2.

Table 2.

Instruments and measurements used for EBM behaviour, EBM attitude and EBM knowledge

Outcome						Measurement
		Instrument	Data	Score per item		Outcome	Score	Numerator	Denominator
EBM behaviour	Guideline adherence	Validated instrument that assesses EBM behaviour of GP trainees by measuring rational guideline adherence during patient consultations, thereby allowing motivated deviation in individual patients on 59 different management decisions (diagnosis N=17, therapy N=20, referral N=22) for 23 conditions as described in 27 different Dutch GP clinical practice guidelines26	Copied from the EMRs: gender, age, reason for encounter, medical history, results of physical examination, diagnosis and treatment or referral	−1	Not in accordance with the guideline and no reason mentioned to deviate from it	Complete adherence in patient consultations	−1 to 1	All patient consultations with score 1 on all management decisions in that patient consultation	All patient consultations assessed with the instrument
						All decisions in patient management		All management decisions with score 1	All management decisions assessed with the instrument
				0	Debatable if in accordance with the guideline due to insufficient or contradicting information	Diagnostic decisions in patient management		All diagnostic management decisions with score 1	All diagnostic management decisions assessed with the instrument
						Therapeutic decisions in patient management		All therapeutic management decisions with score 1	All therapeutic management decisions assessed with the instrument
				1	In accordance with the guideline or not in accordance but with rational motivation	Referral decisions in patient management		All management decisions on referral with score 1	All management decisions on referral assessed with the instrument
	Information-seeking behaviour	Logbook	The presence of a clinical query, the presence of a search, resource used to search, and retrieval of an answer	1	Yes	Queries per patient	0–1	All clinical queries	All patient contacts
				0	No	Performed search per clinical query		All performed searches	All queries
						Performed search in evidence-based resources per clinical query*		All performed searches in evidence-based resources	All queries
						Retrieved answers per performed search		All retrieved answers	All performed searches
						Retrieved answers per performed search in evidence-based resources*		All retrieved answers	All performed searches in evidence-based resources
EBM attitude		Validated questionnaire with 7 questions on attitude (with a visual analogue scale from 0 to 100) of the translated version of the McColl questionnaire15–17		0–100		EBM attitude	0–100	Sum of all scores	All items scored×100
EBM knowledge		Validated questionnaire on EBM knowledge27		0–1		EBM knowledge	0–50	NA	NA

*Evidence-based resources: guidelines, PubMed and preappraised bibliographic databases (such as Clinical Evidence, TRIP, Cochrane).

EBM, evidence-based medicine; EMR, electronic medical record; GP, general practice; NA, not available.

Information-seeking behaviour was defined as the extent to which trainees used sources of medical information to address clinical queries they encountered in daily practice.¹⁸ A clinical query was defined as every consultation-related question that was a reason to search for an answer.¹⁹ Potential sources for EBM information included clinical practice guidelines, and preappraised and primary bibliographic databases (such as Clinical Evidence, Cochrane and PubMed). Information-seeking behaviour was quantified in five different ways, as shown in table 2.

Trainees were asked to collect data on guideline adherence and information-seeking behaviour in logbooks (see online supplementary appendix) during all face-to-face patient consultations, either at the GP office or the patient's home, during the measurement periods (excluding telephone consultations): at T0 (baseline) and T1 (at the end of their third year) during 8 consecutive days, at T2 (1 year after graduation) during 3 days (table 2). They were reminded to participate in person (T0 and T1 only) and by email. They were allowed to report more than one symptom per patient contact and more than one query during each consultation. Data were extracted from either paper or digital logbooks (T0), whereas during follow-up (T1 and T2) all registration was digital.

EBM attitude and knowledge

EBM attitude was defined as the mindset of trainees as to the principles of EBM. It was assessed using the translated version of the McColl questionnaire, consisting of seven questions, that was validated for the Dutch setting.^15–17 EBM knowledge was defined as the ability to answer questions on EBM, as taught in the EBM training programme, and was assessed with a newly developed, validated questionnaire (score 0–50), based on the content of the EBM training programme (MF Kortekaas et al, submitted). To minimise the impact of repeated measurements (ie, repeated use of the same questionnaire), no feedback on performance was reported to trainees after each questionnaire.

Evaluation of the intervention

Implementation of the (new) integrated EBM training programme was assessed by asking trainees to report their compliance to new elements of the EBM training programme anonymously, and by asking them for their opinion about the programme.

Sample size

The sample size calculation was based on the expected improvement in information-seeking behaviour. For an increase from 15% to 50%^20–22 at a significance level of 0.05 (two-sided) with a power of 90% and corrected for clustering (two clusters) with an intraclass correlation coefficient of 0.05, a total sample size of 33 trainees was needed (epicalc2000).²³ Since results from studies about information-seeking behaviour are generally heterogeneous, we doubled our sample size. Owing to the joint educational programmes, the total number of groups and thereby participants was substantially higher than needed based on the sample size calculations.

Statistical analysis

Trainee characteristics at baseline were reported as means with SDs, medians with IQRs or proportions. The outcome guidelines adherence and information-seeking behaviour for the intervention and control groups were reported as proportions. Relative risks (RRs) with 95% CIs were calculated to compare the two groups. For EBM attitude and knowledge, means with 95% CIs were estimated. The effect of the intervention on the outcomes was assessed using mixed models.^{14 24} A random intercept was included to account for cluster randomisation. An autoregressive residual (ie, GEE type) covariance matrix was included to correct for the associations between the outcomes at the different time points.²⁵ In addition, we adjusted for the preferred type of logbook and the national GP knowledge test score at the beginning of the third year. The analyses were based on the intention-to-treat principle. Missing data were not imputed, as multilevel analysis accounts for loss to follow-up. In a sensitivity analysis, we excluded trainees from the analyses when they had logged only a few patients at one or more measurements (the lowest decile, N=60 at T0, N=17 at T1 and N=47 at T2). Statistical analyses were performed using Statistical Package for the Social Sciences V.20.0 (SPSS, Chicago, Illinois, USA) and SAS V.9.2 (SAS Institute, Cary, North Carolina). Comparisons between groups were considered statistically significant at the p<0.05 level.

Ethical considerations

This study design was regarded as non-eligible for full informed consent. However, we obtained informed consent from the trainees for the use of data from the logbooks, questionnaires and educational tests during specialty training. Patient data were reported anonymously.

Results

Of the 82 trainees in the six groups approached, 79 (96%) participated; 39 were randomly allocated to the intervention group, and 40 to the control group (figure 1). Baseline characteristics in the two groups were comparable (table 3), except for the score on the national GP knowledge test (mean of 49 vs 40), guideline adherence (47% vs 68%; table 4), and the proportion of clinical queries (25% vs 17%; table 4). Seventy-six trainees (96%) kept logbooks at T0, 60 (76%) at T1 and 53 (67%) at T2. Of all the logbooks kept by GP trainees, 81% was digital. In total, trainees collected data about 15 894 patient consultations (mean 72, SD 33, range 6–169 per trainee): 7614 (48%) at T0, 4973 (31%) at T1 and 3307 (21%) at T2. At baseline, the preferred type of logbook differed significantly between both groups (16% digitally in the intervention group vs 79% in the control group, p<0.05). Questionnaires on EBM attitude and knowledge were completed by 79 (100%), 66 (84%) and 58 trainees (73%) at the three time points, respectively. More trainees were lost to follow-up in the intervention group than in the control group (41% vs 25% for logbooks, and 33% vs 20% for questionnaires). Baseline characteristics of trainees who did not complete all three measurements (N=34) did not differ from those who did (N=45).

Figure 1.

Flow chart of GP trainees from the GP specialty training in Utrecht, assigned to the intervention or control group. EBM, evidence-based medicine; GP, general practice.

Table 3.

Baseline characteristics of third year GP trainees from the GP specialty training in Utrecht, in 2011 (data collected from questionnaires)

	Intervention	Control
	N=39 (49%)	N=40 (51%)
Demographics
Age (median, IQR)	31 (29–35)	31 (30–35)
Male (N, %)	14 (36)	7 (18)
Full-time working (N, %)	29 (75)	25 (63)
Prior medical school
Same as the GP specialty training (N, %)	20 (51)	21 (53)
Experience
Time since graduation in years (median, IQR)	4 (3–6)	4 (3–6)
Experience as doctor in months (median, IQR)	18 (12–26)	24 (18–37)
Postdoctorate qualification (N, %)	1 (3)	1 (3)
EBM attitude
Self-reported, 5-point Likert scale (median, IQR)	4 (3–4)	3 (3–4)
EBM knowledge
Self-reported, 5-point Likert scale (median, IQR)	3 (2–3)	3 (2–3)
National GP knowledge test (mean, SD)	40 (10)	49 (8)

EBM, evidence-based medicine; GP, general practice.

Table 4.

EBM behaviour (95% CI) as recorded in logs by third year GP trainees from the GP specialty training in 2011

	T0, baseline* (N=76)			T1, end of third year† (N=60)			T2 1 year after graduation† (N=53)
	Intervention	Control	RR	Intervention	Control	RR	Intervention	Control	RR
	N=38	N=38		N=29	N=31		N=23	N=30
Information-seeking behaviour
Queries per patient	0.25 (0.21 to 0.29)	0.17 (0.15 to 0.20)	1.45 (1.15 to 1.83)	0.14 (0.13 to 0.16)	0.15 (0.13 to 0.17)	0.95 (0.80 to 1.12)	0.21 (0.19 to 0.23)	0.19 (0.16 to 0.22)	1.10 (0.92 to 1.32)
Performed search per clinical query	0.77 (0.72 to 0.82)	0.82 (0.75 to 0.91)	0.93 (0.83 to 1.05)	0.71 (0.64 to 0.79)	0.72 (0.68 to 0.76)	0.98 (0.87 to 1.11)	0.60 (0.59 to 0.61)	0.67 (0.57 to 0.79)	0.90 (0.76 to 1.07)
Performed search in evidence-based resources‡ per clinical query	0.32 (0.26 to 0.38)	0.30 (0.24 to 0.39)	1.05 (0.77 to 1.44)	0.27 (0.21 to 0.35)	0.24 (0.15 to 0.38)	1.16 (0.70 to 1.91)	0.27 (0.24 to 0.30)	0.30 (0.23 to 0.38)	0.90 (0.70 to 1.16)
Retrieved answers per performed search	0.84 (0.81 to 0.88)	0.81 (0.77 to 0.85)	1.04 (0.98 to 1.11)	0.81 (0.72 to 0.90)	0.79 (0.72 to 0.85)	1.03 (0.97 to 1.10)	0.82 (0.78 to 0.86)	0.78 (0.71 to 0.85)	1.05 (0.98 to 1.13)
Retrieved answers per performed search in evidence-based resources‡	0.91 (0.87 to 0.96)	0.88 (0.84 to 0.93)	1,03 (0.97 to 1.10)	0.87 (0.83 to 0.91)	0.90 (0.82 to 0.99)	0.97 (0.87 to 1.07)	0.86 (0.81 to 0.92)	0.96 (0.89 to 1.03)	0.90 (0.79 to 1.03)
Guideline adherence
Complete adherence on all decisions in a patient consultation	0.47 (0.42 to 0.53)	0.68 (0.62 to 0.74)	0.69 (0.60 to 0.80)	0.62 (0.59 to 0.66)	0.64 (0.60 to 0.70)	0.96 (0.86 to 1.08)	0.67 (0.60 to 0.74)	0.60 (0.56 to 0.65)	1.10 (0.97 to 1.25)
All decisions in patient management	0.69 (0.65 to 0.73)	0.85 (0.82 to 0.88)	0.81 (0.76 to 0.86)	0.83 (0.81 to 0.84)	0.83 (0.82 to 0.85)	0.99 (0.97 to 1.02)	0.83 (0.80 to 0.87)	0.80 (0.78 to 0.83)	1.04 (0.98 to 1.10)
Diagnostic decisions in patient management	0.66 (0.63 to 0.68)	0.88 (0.86 to 0.91)	0.74 (0.71 to 0.78)	0.87 (0.85 to 0.90)	0.88 (0.87 to 0.89)	0.99 (0.96 to 1.02)	0.85 (0.84 to 0.87)	0.83 (0.79 to 0.87)	1.03 (0.98 to 1.09)
Therapeutic decisions in patient management	0.58 (0.51 to 0.65)	0.71 (0.65 to 0.78)	0.81 (0.69 to 0.94)	0.66 (0.62 to 0.69)	0.66 (0.61 to 0.72)	0.99 (0.89 to 1.11)	0.72 (0.68 to 0.76)	0.66 (0.60 to 0.73)	1.09 (0.97 to 1.23)
Referral decisions in patient management	0.81 (0.79 to 0.84)	0.94 (0.93 to 0.96)	0.86 (0.83 to 0.89)	0.93 (0.92 to 0.95)	0.95 (0.94 to 0.96)	0.98 (0.96 to 1.00)	0.91 (0.86 to 0.97)	0.92 (0.89 to 0.94)	0.99 (0.92 to 1.07)

*Crude data, collected before the intervention at baseline.

†Adjusted for type of logbook (paper or digital), score on national GP knowledge test and repeated measures.

‡Evidence-based resources: Guidelines, PubMed and preappraised bibliographic databases (such as Clinical Evidence, TRIP, Cochrane).

EBM, evidence-based medicine; GP, general practice; RR, relative risk.

Intervention effects

Information-seeking behaviour and guideline adherence did not significantly differ between both groups, neither at the end of the third year (T1) nor 1 year after graduation (T2), with RRs for guideline adherence varying between 0.96 and 0.99 (T1) and 0.99 and 1.10 (T2), depending on the type of management decision, and for information-seeking behaviour varying between 0.97 and 1.16, and 0.90 and 1.10, respectively (table 4). GP trainees motivated why they deviated from guideline instructions in 10% of the patient consultations during training, and in 20% of their consultations after graduating. As for the secondary outcomes, mean scores for EBM attitude and knowledge at T1 and T2 did not significantly differ between the intervention and control groups (table 5). Exclusion of 14 trainees (9 in the intervention group and 5 in the control group) who had logged only a few patients at one or more measurements slightly increased the difference between the 2 groups in the proportion of retrieved answers from an RR of 1.05 (0.98 to 1.13) to a (statistically significant) RR of 1.09 (1.02 to 1.15), but this had no effect on the other outcome measures.

Table 5.

Estimated means (95% CI) of EBM attitude and EBM knowledge of third year GP trainees from the GP specialty training in 2011

	T0, baseline* (N=79)			T1, end of third year† (N=66)			T2, 1 year after graduation† (N=58)
	Intervention	Control	Difference	Intervention	Control	Difference	Intervention	Control	Difference
	N=39	N=40		N=33	N=33		N=26	N=32
EBM attitude
Validated McColl questionnaire, adjusted for Dutch GP trainees15–17 (score 0–100)	46.2 (42.9 to 49.6)	48.4 (45.6 to 51.1)	−2.1 (−6.5 to 2.3)	52.5 (49.8 to 55.2)	51.8 (49.1 to 54.4)	0.8 (−3.1 to 4.6)	59.5 (56.5 to 62.5)	58.9 (56.2 to 61.6)	0.6 (−3.6 to 4.7)
EBM knowledge
Validated EBM knowledge questionnaire (MF Kortekaas et al, submitted) (score 0–50)	29.9 (28.8 to 31.0)	29.7 (28.5 to 30.8)	0.2 (−1.4 to 1.9)	29.8 (28.3 to 31.3)	29.5 (28.0 to 31.0)	0.3 (−1.8 to 2.4)	30.3 (28.8 to 31.9)	29.7 (28.2 to 31.2)	0.6 (−1.6 to 2.8)

*Crude data.

†Adjusted for score on national GP knowledge test and repeated measures.

EBM, evidence-based medicine; GP, general practice.

Evaluation of the intervention

Seventeen trainees (44%) in the intervention group reported compliance to the new EBM training elements in daily clinical practice, and evaluated the training programme. All respondents reported having EBM tutorial dialogues with their supervisor, but only 12% did this once a week as suggested, the others less. The majority of responding trainees (53%) did not critically appraise articles with their supervisors, or only less than once every 2 months (29%). Nine trainees (53%) conducted one or more e-learning courses during the year, but almost none (12%) had used online coaching. Trainees rated the integrated EBM training programme with a mean score of 6 (scale 1–10). About half of the trainees (53%) considered the overall volume of EBM training as adequate. The integration of EBM into clinical practice was rated as sufficient by two-third (N=11). Half of the trainees (N=9) reported that their supervisors were not enthusiastic about the new EBM training elements in clinical practice, or that they were not aware of it (N=13). In contrast, more than half (N=10) felt supported by their supervisors to integrate the EBM training elements into clinical practice. Responding supervisors (N=15) frequently agreed with trainees' opinions.

Discussion

Our results show that an integrated EBM training programme in the final third year of the GP specialty training does not improve EBM performance as compared with stand-alone EBM training. EBM behaviour, with guideline adherence (RR 0.96 to 1.10) and information-seeking behaviour (RR 0.90 to 1.16), attitude and knowledge, did not differ between trainees in the two groups, neither during the last year of training, nor in the year following graduation. Overall, guideline adherence among trainees was high in all phases of their training, ranging from 69% to 95%, depending on the type of management decision. In more than half of the consultations, trainees adhered to the guideline on all management decisions. GP trainees had on average one clinical query per five patients, and succeeded in retrieving an answer to the vast majority of their questions.

To the best of our knowledge, this is the first study performing a randomised controlled trial among GP trainees comparing integrated with stand-alone EBM training, and assessing both short-term and long-term effects on EBM behaviour, attitude and knowledge.^{5 28–31} Prior studies assessing the effects of EBM training predominantly used non-controlled designs, and did not include a direct comparison between programmes.^{5 28 30} Moreover, integration of EBM is poorly defined in the literature, resulting in a large variety of training programmes included in studies, while these studies were performed in many different settings and populations.^{28 32} The fact that we use clinical examples in our stand-alone EBM training is considered ‘integrated’ by some already.³² Prior studies reporting effects of EBM training showed contradictory results. Outcomes most frequently reported were EBM knowledge, skills and attitude.^{5 30 33} The few studies assessing effects on EBM behaviour frequently used (retrospective) questionnaires, in contrast to ‘real-time’ measurements as in our study.³¹ This may have resulted in an overestimation of the small effects of EBM training observed in these earlier studies. The GP trainees' scores on EBM attitude in our study were low compared with those in earlier studies.¹⁷ The number of clinical queries was lower as well, but the GP trainees very frequently performed a (successful) search.³³ One year after graduation, the number of clinical queries in the intervention group was higher than at the end of the third year, but still low in comparison to earlier studies. Although the number of searches in the intervention group decreased after graduation, all GP trainees in our study still performed more searches compared with earlier studies.³³ The effects of EBM training on adherence of individual GP trainees to multiple guidelines for a wide range of management decisions have not been evaluated before. Moreover, we took motivated deviation in individual patients into account, and our results show that trainees appropriately motivated why they deviated from the guideline in 10% of the patient consultations during training, and in 20% of their consultations after graduating.^34–36

Strengths and limitations

We assessed EBM behaviour in daily clinical practice, using a large number and a wide variety of patient consultations, with a long-term follow-up. The choice to perform our study in the third year of the GP specialty training enabled us to assess the effects of the different EBM training programmes until after graduation. However, the high levels of guideline adherence and information-seeking behaviour at the beginning of the third year left little room for improvement, thereby making it very difficult to demonstrate any additional benefit of the intervention. At baseline, groups differed in the score on the national GP knowledge test and the preferred type of logbook. Since allocation to the intervention or control arm was random in our study, differences between the groups at baseline are random and it is not by definition obligatory to adjust for this.⁷ However, since the national GP knowledge measures knowledge of the professional guidelines and therefore may have been correlated to guideline adherence in clinical practice, we accounted for this in our analyses, but this did not materially change our findings. The differences in the preferred type of logbook at baseline had an important impact on the (quality of the) data reported by GP trainees, and thereby on the assessment of guideline adherence with our instrument. For this reason, we decided to account for the type of logbook as used at baseline in our analyses, and decided to only collect data digitally at the next two follow-up measurements in both groups. For integration in practice, we were dependent on the GP trainees and the GP supervisors, who are important role models. We know that insufficient skills and knowledge of supervisors, together with a negative attitude, are important barriers for them to use and to teach EBM.^{11 37} For this reason, we gave them training in EBM as well, and trainees and supervisors performed assignments together.⁵ An online survey after the intervention (with a response rate of 44%) showed that only half of these trainees thought that their supervisors were enthusiastic about their new EBM training. Large variability in the way some components of the intervention were delivered in the primary care practices may have diluted a potential effect of our intervention. Although such ‘lack of adherence’ may have reduced the effect of the intervention, such variability is likely to occur in any training programme in daily teaching environments. Furthermore, the contrast between both groups may have been limited because our stand-alone EBM training could be considered as ‘integrated’ to some extent, because we do use clinical examples.³² Comparison with ‘no EBM training’ could have provided more information about the effect(s) of EBM training. However, since EBM is one of the competencies obligatory in the GP specialty training in the Netherlands, this was no option. The time investment needed for GP trainees to collect data in logbooks was substantial. This may have reduced commitment from GP trainees and this may—to save time—have resulted in recording (too) little information on consultations (especially in the paper logbooks where they could not copy from the electronic medical record (EMR)), under-reporting, selective reporting of the number of clinical queries (eg, only when trying to pursue an answer), or higher rates of loss to follow-up. Bias due to the high rates of loss to follow-up cannot be fully excluded, although a multilevel analysis like the one we performed is very robust for (randomly) missing follow-up measurements.¹⁴ Moreover, while accounted for in our analyses, the differences between both groups in logbook preference at baseline may have affected the results. Using (copied) data from the EMRs may not have adequately reflected guideline adherence, and misclassification may have occurred due to lack of relevant information. Although guideline adherence was scored by different raters at the three different measurements, we do not expect this to affect the results as the instrument we used has been proven to be reliable, with inter-rater reliabilities ranging from moderate to excellent. Finally, for practical reasons, we randomly allocated the participating trainees at the group level, and we randomly assigned the integrated and stand-alone EBM intervention to the first and the last blocks of three groups. Although this may theoretically not be the optimal randomisation procedure, we do not think that the starting moment of the group was in any way related to clinical capacity of the trainees’ EBM behaviour.

Conclusion and implications for future

In a randomised controlled trial, we found no differences in EBM behaviour, attitude or knowledge between GP trainees who received EBM training through a stand-alone module or those who followed an integrated EBM training. We think that the main explanation is that the stand-alone programme adequately increases EBM skills, leaving little room for improvement. EBM skills generally develop well during both trainings: GP trainees in both groups showed consistently high levels of guideline adherence and frequently performed searches in case of clinical queries, which were usually (considered) successful. If EBM training is not yet incorporated in specialty training, we suggest using the integrated EBM training format, as this links to the clinical development of the trainee. Future studies could focus on better integration of EBM training into clinical practice, for instance by engaging supervisors more, as they are important role models for trainees. Moreover, assessment of EBM behaviour at point of care might be made easier, for example, by automatically extracting patient consultation data from EMRs.