Abstract
Background
GP appraisal is currently considered inadequate because it lacks robustness. Objective assessment of appraisal evidence is needed to enable judgements on professional performance to be made.
Aim
To determine GP appraisers' views of the acceptability, feasibility, and educational impact of external peer feedback received on three core appraisal activities undertaken as part of this study.
Design of study
Independent peer review and cross-sectional postal questionnaire study.
Setting
NHS Scotland.
Method
One of three core appraisal activities (criterion audit, significant event analysis, or video of consultations) was undertaken by GP appraisers and subjected to peer review by trained colleagues. A follow-up postal questionnaire elicited participants' views on the potential acceptability, feasibility, and educational impact of this approach.
Results
Of 164 appraisers, 80 agreed to participate; 67/80 (84%) submitted one of three appraisal materials for peer review and returned completed questionnaires. For significant event analyses (n = 44), most responders believed the peer feedback method was feasible (100%) and fair (92.5%) and would add value to appraisal (95.5%). Peer feedback on criterion audits (n = 15) was believed to be acceptable and fair (93.3%) and it was thought it would be a useful educational tool (100%). Completing a consultation video (n = 8) was perceived to be feasible as part of normal general practice (n = 5). It was unanimously agreed that assessment of videos by peers has educational impact and would help improve appraisal.
Conclusion
This group of GP appraisers strongly supported the role of external and independent feedback by trained peers as one approach to strengthening the existing appraisal process.
Keywords: appraisal, assessment, feedback, general practice, peer review
INTRODUCTION
Appraisal for GPs in NHS Scotland was introduced in 2003.1 This ‘non-judgemental’ process helps GPs to reflect on their working practices with a peer — a trained GP appraiser colleague. One of five core educational activities (Table 1) — each linked to the principles of Good Medical Practice,2 — is discussed in depth annually. Although appraisal has gained acceptance,3–5 the benefits and impact are limited for many GPs.6,7 When introduced, it was envisaged that the combined evidence from five appraisals would inform a proposed system of medical revalidation in the UK.8 The recent white paper makes clear, however, that the existing appraisal model is inadequate for this purpose because it lacks robustness and objectivity.9 Addressing this will require objective assessment of evidence to be included to allow judgements on professional performance.10
Table 1.
Core categories for GP appraisal in Scotland and alignment with the principles of the General Medical Council's Good Medical Practice.2
| Core category | Principle of good medical practice |
| Prescribing | Provide good clinical care |
| Referrals and peer review | Provide good clinical care |
| Clinical audit | Maintain and improve performance |
| Significant event analysis | Raise concerns about patient safety |
| Maintain and improve performance | |
| Communication skills | The doctor–patient partnership — good communication |
In the west of Scotland, a well-established educational model aims to provide GPs with informed, objective, confidential, and independent feedback from trained peers on the quality of three core appraisal activities: videoed consultations, criterion-based audit, and significant event analysis (SEA).11–13 The model is coordinated by NHS Education for Scotland (NES) and has been used on a voluntary basis by GPs as part of regional arrangements for continuing professional development. Although the concept of ‘peer review’ is well established in medicine, the novel aspect of this model is that it appears to be the only one reported in the literature that utilises GP peers who are specifically trained to review and provide developmental feedback on the educational activities outlined. To date, around 40% of west of Scotland GP principals have submitted at least one piece of work for peer review since its inception.
The underlying principles of the model are based on an adaptation of cognitive continuum theory.14 This framework aids understanding of the thinking used in performing a range of tasks. The aim is to improve the quality of reflection on particular tasks. How this is done is described in one of six ‘modes of practice’ ranging from the highly structured scientific experiment (mode 1) to intuitive judgements (mode 6). Peer review sits between modes 4 (system-aided judgement) and 5 (peer-aided judgement) and is designed to minimise the probability of a mode 6 judgement (self-assessment) leading to invalid conclusions on decisions made.
The model described is underpinned by a research base.11–13,15–23 However, further evidence from a range of GP groups as to its overall utility is necessary to inform future development.24,25 GP appraisers are a key group with important professional insights into the strengths and weakness of the appraisal system and how it may be enhanced. Against this background, this study aimed to evaluate GP appraisers' views of the acceptability, feasibility, and educational impact of independent, external peer feedback received on one of three core appraisal activities they undertook as part of this study. In this way, GP appraisers would gain experience of receiving feedback on specific aspects of their own appraisal activities as part of the NES model described. Appraisers could then judge the usefulness of this approach as a support mechanism for the GP appraisal system in NHS Scotland.
METHOD
Completion of core appraisal tasks
Sample and recruitment
All 164 GP appraisers in NHS Scotland were randomly split into three groups, and one of three appraisal activities (SEA, criterion audit, or videotaped consultations) was allocated to each. To maximise fairness for participants and encourage engagement with the study, group numbers were adjusted to account for the effort needed to perform each activity (for example, fewer individuals were allocated to video consultation than SEA because the former is much more onerous). Appraisers were invited to undertake the allocated activity within a 4-month timeframe and submit this to the NES peer feedback model. Informed consent was obtained by providing a full written description and purpose of the evaluation. A standardised re-imbursement fee was offered by the host organisation (NES) to cover back-fill costs for GP participation time.
How this fits in
GP appraisal is considered inadequate and needs to be more robust and objective. The quality of core appraisal activities undertaken by GPs is also known to be variable. Independent feedback by trained peers on these activities is proposed as a potential educational solution. This study provides evidence that GP appraisers who received peer feedback on appraisal activities found it acceptable and educationally useful. Furthermore, they agreed it would ‘add value’ in improving the quality of appraisal evidence.
The model involves the application of formative assessment instruments,11,15,16 by trained GP peers to facilitate developmental feedback for participants. For audit and SEA submissions, two GP peers (from a group of 20) are allocated to independently review submitted reports, with a single reviewer used for videos of consultations (from a group of 20). A GP peer review coordinator then collates this written feedback on performance, provides their own input, and compiles a short report which is sent to participants.
Postal questionnaire survey
Data collection
A questionnaire was developed to examine attitudes towards the feasibility, acceptability, and educational impact of external peer feedback on the relevant appraisal activity undertaken. The questionnaire was pre-tested with four GP appraiser colleagues. The survey took place in March 2008, 3 days after peer feedback reports were sent to participants. Non-responders were emailed reminders until all questionnaires were returned. Levels of agreement with attitudinal statements were indicated on a 7-point rating scale.
Data analysis
Data were entered into a Microsoft Excel spreadsheet. The number of rating scores ≥4 was calculated. A Wilcoxon signed-rank test and 95% confidence interval (CI) for estimated median differences were performed to compare before and after rating scores, which were calculated using Minitab (version 15).
RESULTS
Participants and appraisal materials
A total of 80/164 GP appraisers agreed to participate (49%). Of the 80, 67 (84%) submitted appraisal materials for external peer review. All 67 participants returned completed questionnaires (100%).
Participants' reported apprehension and knowledge and skill levels before and after peer feedback
Table 2 outlines statistical differences in participants' reported levels of apprehension about submitting SEA (P = 0.52), audit (P = 0.14), and video (P = 0.04) for external peer feedback before and after the study. The differences in appraisers' estimated knowledge and skills levels before and after receiving feedback on SEA (P<0.01), audit (P = 0.06) and video consultations (P = 0.18) are also detailed in Table 2.
Table 2.
Reported levels of apprehension about submitting educational materials for external peer feedback and estimated knowledge and skill ratings (before and after the study).
| Level of apprehension, continuous rating scale 1–7 (1 = not at all to 7 = extremely) | |||||
|---|---|---|---|---|---|
| CPD activity | Rating before the study, median (IQR) | Rating after the study, median (IQR) | Difference in rating,a median (IQR) | 95% CIc | P-value (H0: median = 0) |
| Significant event analysis (n = 44) | 2 (1–2) | 1 (1–2) | 0 (0–1) | −0.0 to 0.5 | 0.52 |
| Criterion audit (n = 15) | 3 (1–5) | 2 (1–3) | 0 (0–1) | −0.0 to 1.5 | 0.14 |
| Videoed consultations (n = 8) | 5 (2–5) | 3 (1–5) | 1 (0–2) | 0.5 to 2.0 | 0.04 |
| Knowledge and skill level rating, rating scale 1–7 (1 = poor to 7 = excellent) | |||||
|---|---|---|---|---|---|
| Rating before the study, median (IQR) | Rating after the study, median (IQR) | Difference in ratingb, median (IQR) | 95% CIc | P-value (H0: median = 0) | |
| Significant event analysis (n = 44) | 6 (5–6) | 6 (6–7) | 0 (0–1) | 0.5 to 0.5 | <0.01 |
| Criterion audit (n = 15) | 6 (5–6) | 6 (6–6) | 0 (0–1) | 0.0 to 0.5 | 0.06 |
| Videoed consultations (n = 8) | 5 (5–5) | 5 (5–6) | 0 (0–1) | 0.0 to 1.0 | 0.18 |
Difference = before – after.
Difference = after – before.
Wilcoxon signed-rank test and CI for median difference.
CPD = continuing professional development.
IQR = interquartile range.
Significant event analysis
A majority of potential participants allocated to the SEA group (91.2%; 44/48) submitted completed reports (Table 3). There was unanimous agreement (n = 44) that submitting one report every 5 years for external peer review would be feasible for most GPs. Responders believed the peer feedback received was fair (n = 42); would facilitate improvement in their SEA practice (n = 32); and would be acceptable to most colleagues (n = 42). The majority agreed (n = 42) that external feedback on SEA would add value to the existing appraisal system.
Table 3.
GP Appraisers' attitudes towards the feasibility, acceptability, and educational gain of peer-reviewed significant event analyses (n = 44).
| Level of agreement, rating scale 1–7a | ||
|---|---|---|
| Attitudinal statement | Median | Number of rating scores ≥4 |
| Feasibility | ||
| Undertaking a SEA using the method outlined is feasible as part of normal general practice | 7 | 44 |
| Submitting at least one SEA report for external peer feedback should be feasible for most GPs over a 5-year period | 7 | 44 |
| Acceptability | ||
| I was happy to have my SEA reviewed by trained colleagues | 7 | 44 |
| The feedback I received on my completed SEA was fair | 6 | 42 |
| I understood the process by which my SEA was peer reviewed | 6 | 42 |
| Submitting at least one SEA report for external peer feedback over a 5-year period would be acceptable to most GPs | 6 | 42 |
| Educational impact | ||
| The feedback helped me understand more about the process of completing a SEA | 4 | 33 |
| The feedback gave me specific advice on how to improve my significant event analyses | 5 | 36 |
| I have made (or will make) changes that have improved my ability to conduct significant event analyses in the practice | 4 | 32 |
| Assessment of SEA by peer feedback is a useful educational tool | 6 | 41 |
| Submitting at least one SEA report for external peer feedback over a 5-year period would have an educational impact for most GPs | 6 | 41 |
1 = strongly disagree to 7 = strongly agree. SEA = significant event analysis.
Criterion audit
Fifteen of 18 expected criterion audit reports (83.3%) were received (Table 4). Most responders were unconcerned about their audit report being reviewed by external peers (n = 14) and believed this would also be acceptable to GP colleagues (n = 15). Feedback received provided specific advice on how to improve audit skills (n = 15) and would facilitate improvements in related practices for the majority (n = 12). A majority agreed (n = 14) that the existing appraisal system would be enhanced by external feedback on criterion audits.
Table 4.
GP appraisers' attitudes towards the feasibility, acceptability, and educational gain of peer-reviewed criterion audit (n = 15).
| Level of agreement, rating scale 1–7a | ||
|---|---|---|
| Attitudinal statement | Median | Number of rating scores ≥4 |
| Feasibility | ||
| Undertaking a completed audit cycle using the method outlined is feasible as part of normal general practice | 6 | 13 |
| Submitting at least one completed audit report for external peer feedback should be feasible for most GPs over a 5-year period | 6 | 14 |
| Acceptability | ||
| I was happy to have my completed audit report reviewed by trained colleagues | 6 | 14 |
| The feedback I received on my completed audit report was fair | 6 | 14 |
| I understood the process by which my completed audit was peer reviewed | 6 | 15 |
| Submitting at least one completed audit report for external peer feedback over a 5-year period would be acceptable to most GPs | 5 | 15 |
| Educational impact | ||
| The feedback helped me understand more about the process of completing an audit project | 5 | 11 |
| The feedback gave me specific advice on how to improve my skills in audit | 5 | 15 |
| I have made (or will make) changes that have improved my ability to conduct audit in the practice | 4 | 12 |
| Assessment of criterion audit by peer feedback is a useful educational tool | 6 | 15 |
| Submitting at least one completed audit report for external peer feedback over a 5-year period would have an educational impact for most GPs | 6 | 13 |
1 = strongly disagree to 7 = strongly agree.
Videotaped consultations
A total of 8/14 appraisers (57.1%) submitted videos of consultations for review (Table 5). Five responders believed completing a consultation video was feasible as part of normal general practice. Peer feedback on consulting skills was perceived by all participant appraisers to be fair and educationally useful and to have the potential to have an impact on the consulting skills of most GP colleagues. There was a unanimous perception that peer-reviewed video consultations would add value to the appraisal process.
Table 5.
GP Appraisers' attitudes towards the feasibility, acceptability and educational gain of peer-reviewed video consultations (n = 8).
| Level of agreement, rating scale 1–7a | ||
|---|---|---|
| Attitudinal statement | Median | Number of rating scores ≥4 |
| Feasibility | ||
| Completing a video of six patient consultations is feasible as part of normal general practice | 5 | 5 |
| Submitting at least one video of six patient consultations for external peer feedback should be feasible for most GPs over a 5-year period | 5 | 6 |
| Acceptability | ||
| I was happy to have my video reviewed by trained colleagues | 6 | 7 |
| The feedback I received on my video of consultation skills was fair | 6 | 8 |
| I understood the process by which my video was peer reviewed | 6 | 8 |
| Submitting at least one video for external peer feedback over a 5-year period would be acceptable to most GPs | 4 | 6 |
| Educational impact | ||
| The feedback gave me specific advice on how to improve my consultation skills | 5.5 | 7 |
| I have made (or will make) changes that have improved my ability to consult with patients | 5 | 8 |
| Assessment of video by peer feedback is a useful educational tool | 6 | 8 |
| Submitting at least one video for external peer feedback over a 5-year period would have an educational impact for most GPs | 6 | 8 |
1 = strongly disagree to 7 = strongly agree.
DISCUSSION
Summary of main findings
This study found strong support from participants for the feasibility, acceptability, and educational gain associated with the external and independent review of three core appraisal activities by trained colleagues. Many expressed a certain level of apprehension about receiving external feedback on their appraisal activities, but this decreased slightly after participation, particularly for those submitting a video of consultations. Additionally, there was a general perception that review of core appraisal activities by trained colleagues would ‘add value’ to the current appraisal system in Scotland.
Peer feedback received on SEA and criterion audit reports helped responders to understand more about these techniques, and engendered a reported commitment to practise it more effectively. There was a perceived increase in estimated knowledge and skills levels associated with each activity after peer feedback had been received, particularly for SEA. However, a large minority experienced low-to-moderate education gains from participation. This could be explained by the fact that most GP appraisers should be experienced in SEA and audit. Related evidence and knowledge are also required for many educational and governance reasons,1 including training practice accreditation and to benefit from the Quality and Outcomes Framework.25 Additional knowledge will also be gleaned by appraisers when reflecting on SEA and audit reports with appraisees. It should also be acknowledged that a proportion of GPs will be highly effective in applying both techniques, and that peer feedback, while reinforcing excellent performance in these cases, may offer minimal educational gain. Given this background, there is a strong perception from appraisers that other GP colleagues would benefit from developmental feedback by trained peers on SEA and audit attempts, which reflects previous research evidence.17–20,22
A comparatively small number of appraisers submitted consultation videos for peer review. Although there was strong agreement that other GPs would benefit from this method of receiving feedback on their consultations skills, there was less agreement on whether this would be acceptable to most GPs. Indeed previous work has demonstrated that the process can be effectively undertaken by a GP without previous experience in this area,21 however, peer review of video consultation is often perceived to be much more challenging and stressful than participating in audit or SEA.26 The process can often take longer and be technically difficult to organise and execute, while patient consent is also required. Perhaps most importantly it is also perceived to be more professionally threatening than the other two activities.26 However, there was strong positive agreement on the potential value of external peer review of video consultations educationally and when linked to the GP appraisal system.
Strengths and limitations of the study
A large minority of all Scottish appraisers participated. However, the study population was self-selecting and potentially biased as the views and experiences of non-participating appraisers and other GP groups may differ from those reported. The study context is specific to NHS Scotland, but the underlying principle linking external peer review and appraisal is potentially relevant to other medical specialties and healthcare systems. Most appraisers who agreed to participate did so, but a substantial minority failed to submit video consultations, which points to a potential difficulty in linking this activity with the peer review model as part of normal general practice. Written feedback was provided by peer reviewers. However, in some situations, face-to-face feedback might be more desirable but this also creates a feasibility issue.
Comparison with existing literature
Given the increasing importance being accorded to the links between appraisal and revalidation,8–10 more attention will inevitably be paid to the standard of evidence being submitted. Ultimately, someone has to make a judgement as to whether it is good enough. Appraisers have consistently voiced their discomfort at being given this task.6,7 This external peer review system allows the judgements to be made prior to an appraisal with a built-in sampling process of ‘quality control’ ensuring fairness across the country, that is, a national standard. However, the feasibility of a national external peer review system to support appraisal is open to question. For such a system to function, additional resources will be required. One option is to consider the vision that all GPs are potential peer reviewers — with appropriate training — with local ‘experts’ taking on the role of quality assuring the process. If it is to be practicable, much wider discussion is needed. The status quo, however, may not be an option.
The literature on the limited value of self-assessment is clear.27 The benefit, therefore, of providing additional peer review should be not only desirable but — given the stakes — essential. The adaptation of the Hammond model of Cognitive Continuum Theory underpins this justification by encouraging a more rational (rather than merely intuitive) approach to decision making on material submitted for appraisal and inclusion in a revalidation folder.
A call for external verification of evidence will not be straightforward — GP appraisers are in the frontline and, in their opinion in this study, those who took part felt the advantages outweighed the disadvantages by suggesting it gave added value to the appraisal process. In their proposal to improve the appraisal process, the Royal College of General Practitioners suggests that over a 5-year period GPs should present evidence of a specific number of SEA and completed audit cycle reports for discussion during appraisals.28 This represents an increase in the quantity of evidence that is currently required. However, it is unclear how this overcomes the perceived inadequacy of the present approach, which is that the appraisal system lacks a robust method to verify the quality of evidence that is already known to be variable.17–20,22
Implications for future research
Future research on this model should focus on shedding light on the acceptability and feasibility of peer reviewing videos of consultations on a large-scale basis. More confirmatory work on strengthening the inter-rater reliability of video consultation peer review is also required. Overall, there is a need to explore with decision makers if and how independent peer review can make a meaningful contribution to improving the robustness of evidence required for appraisal in support of revalidation.
Acknowledgments
We wish to thank all GP appraisers and GP peer reviewers in NHS Scotland who participated. The following individuals helped to coordinate the study and we extend our grateful thanks to them, also: Marjorie McArthur, Kirsten Macdonald, Dr Ed Russell-Smith and Dr Gordon McLeay. We are also grateful to Dr Lilian Murray, University of Glasgow, for statistical advice.
Funding body
This study was funded by NHS Education for Scotland
Ethic approval
The study was judged by the authors to be service evaluation
Competing interests
The authors' employing organisation is responsible for managing the national GP appraisal scheme and also manages the peer review model outlined in the study. The views expressed reflect those of the authors and not NES
REFERENCES
- 1.Scottish Executive, NHS Education for Scotland, RCGP (Scotland) & BMA (Scotland) GP appraisal scheme: a brief guide. Edinburgh: Scottish Executive; 2003. http://www.sehd.scot.nhs.uk/publications/gpabg/gpabg-00.htm (accessed 2 Jun 2009) [Google Scholar]
- 2.General Medical Council. Good medical practice. London: GMC; 2006. http://www.gmc-uk.org/guidance/good_medical_practice/GMC_GMP.pdf (accessed 30 May 2009) [Google Scholar]
- 3.McKinstry B, Shaw J, McGillvray L, Skinner L. What do general practitioners think about annual appraisal? A questionnaire-based cross-sectional study in southeast Scotland. Educ Prim Care. 2002;13(4):472–476. [Google Scholar]
- 4.Jelley D, van Zwnanenberg T. Peer appraisal in general practice: descriptive study in the Northern Deanery. Educ Gen Pract. 2001;11(1):281–287. [Google Scholar]
- 5.Lewis M, Welwyn G, Wood F. Appraisal of family doctors: an evaluation study. Br J Gen Pract. 2003;53(491):454–460. [PMC free article] [PubMed] [Google Scholar]
- 6.Coltart I, Cameron N, McKinstry B, Blaney D. What do doctors really think about the relevance and impact of GP appraisal 3 years on? A survey of Scottish GPs. Br J Gen Pract. 2008;58(547):82–87. doi: 10.3399/bjgp08X264036. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Wakeling J, Cameron N, Lough M. What factors impact on general practitioners' engagement with appraisal? A qualitative evaluation in Scotland. Educ Prim Care. 2008;19(6):615–623. [Google Scholar]
- 8.General Medical Council. License to practise and revalidation for doctors. London: GMC; 2003. http://www.gmc-uk.org/doctors/documents/archive/licence_to_practise.pdf (accessed 2 Jun 2009) [Google Scholar]
- 9.Secretary of State for Health. Trust, assurance and safety — the regulation of health professionals in the 21st century. White Paper presented to Parliament by the Secretary of State for Health by Command of Her Majesty. London: the Stationery Office; 2007. http://www.gmc-uk.org/doctors/documents/archive/licence_to_practise.pdf (accessed 2 Jun 2009) [Google Scholar]
- 10.Department of Health. Good doctors, safer patients: proposals to strengthen the system to assure and improve the performance of doctors and to protect the safety of patients. A report by the Chief Medical Officer. London: Department of Health; 2006. http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_4137232 (accessed 2 Jun 2009) [Google Scholar]
- 11.Cameron N, McMillan R. Enhancing communication skills by peer review of consultation videos. Educ Prim Care. 2006;17(1):40–48. [Google Scholar]
- 12.Bowie P, McCoy S, McKay J, Lough M. Learning issues raised by the educational peer review of significant event analyses in general practice. Qual Prim Care. 2005;13(2):75–84. [Google Scholar]
- 13.McKay J, Bowie P, Lough M. Variations in the ability of general medical practitioners in applying two methods of clinical audit: a five-year study of assessment by peer review. J Eval Clin Pract. 2006;12(6):622–629. doi: 10.1111/j.1365-2753.2005.00630.x. [DOI] [PubMed] [Google Scholar]
- 14.Hammond KR. Towards increasing competence of thought in public policy formation. In: Hammond KR, editor. Judgement and decision in public policy formation. Boulder, CO: Westview Press; 1978. pp. 11–32. [Google Scholar]
- 15.Lough JRM, McKay J, Murray TS. Audit and summative assessment: a criterion-referenced marking schedule. Br J Gen Pract. 1995;45(400):607–609. [PMC free article] [PubMed] [Google Scholar]
- 16.McKay J, Murphy D, Bowie P, et al. Development and testing of an assessment instrument for the formative peer review of significant event analyses. Qual Saf Health Care. 2007;16(2):150–153. doi: 10.1136/qshc.2006.020750. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Bowie P, McKay J, Dalgetty E, Lough M. A qualitative study of why general practitioners may participate in significant event analysis and educational peer review. Qual Saf Health Care. 2005;14(3):185–189. doi: 10.1136/qshc.2004.010983. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.McKay J, Shepherd A, Bowie P, Lough M. Acceptability and educational impact of a peer feedback model for significant event analysis. Med Educ. 2008;42(12):1210–1217. doi: 10.1111/j.1365-2923.2008.03235.x. [DOI] [PubMed] [Google Scholar]
- 19.McKay J, Pope L, Bowie P, Lough M. Peer feedback in general practice: a focus group study of trained reviewers of significant event analyses. J Eval Clin Pract. 2009;15(1):142–147. doi: 10.1111/j.1365-2753.2008.00969.x. [DOI] [PubMed] [Google Scholar]
- 20.Murie J, McCrae J, Bowie P. The peer review pilot project: a potential system to support GP appraisal in NHS Scotland. Educ Prim Care. 2009;20:34–40. doi: 10.1080/14739879.2009.11493759. [DOI] [PubMed] [Google Scholar]
- 21.McMillan R, Cameron N. Factors influencing the submission of videotaped consultations by general practitioners for peer review and educational feedback. Qual Prim Care. 2006;14(2):85–89. [Google Scholar]
- 22.Bowie P, Cooke S, Lo P, et al. The assessment of criterion audit cycles by external peer review — when is an audit not an audit? J Eval Clin Pract. 2007;13(3):352–357. doi: 10.1111/j.1365-2753.2006.00704.x. [DOI] [PubMed] [Google Scholar]
- 23.Van der Vleuten CPM. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1996;1(1):41–67. doi: 10.1007/BF00596229. [DOI] [PubMed] [Google Scholar]
- 24.Evans R, Elwyn G, Edwards A. Review of instruments for peer assessment of physicians. BMJ. 2004;328(7458):1240–1243. doi: 10.1136/bmj.328.7450.1240. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Department of Health. New GMS contract 2006/7. London: Stationery Office; 2006. http://www.dh.gov.uk/en/Healthcare/Primarycare/Primarycarecontracting/GMS/DH_4125636 (accessed 2 Jun 2009) [Google Scholar]
- 26.NHS Modernisation Agency. Defining the evidence for revalidation — supporting the Royal College of General Practitioners. Collation of views from the NHS Clinical Governance Support Team Expert Group. NHS; 2004. http://www.gp-training.net/cme/revalidation/revalidation_evidence.pdf (accessed 2 Jun 2009) [Google Scholar]
- 27.Coltart A, Bagnall G, Evans A, et al. Best evidence in medical education (BEME). A systematic review of the literature on the effectiveness of self-assessment in clinical education. http://www.bemecollaboration.org/beme/pages/reviews/mckinstry.html (accessed 27 May 2009)
- 28.Royal College of General Practitioners. RCGP Guide to the Revalidation of general practitioners. London: Royal College of General Practitioners; 2008. http://www.rcgp.org.uk/PDF/PDS_Guide_to_Revalidation_for_GPs_April_2009_V1.0.pdf (accessed 30 May 2009) [Google Scholar]