Abstract
Background
Reducing delay in the primary care part of the cancer care pathway is likely to improve cancer survival. Identifying effective interventions in primary care would allow action by primary healthcare professionals and local commissioners to reduce delay.
Aim
To identify interventions that reduce primary care delay in the referral of patients with cancer to secondary care.
Design and setting
Systematic review in primary care.
Method
Eight electronic databases were searched using terms for primary care, cancer, and delay. Exclusion criteria included screening and the 2-week-wait referral system. Reference lists of relevant papers were hand searched. The quality of each paper was assessed using predefined criteria, and checked by a second reviewer.
Results
Searches identified 1798 references, of which 22 papers were found to meet the criteria. Interventions concerning education, audit and feedback, decision support software and guideline use, diagnostic tools, and other specific skills training were identified. Most studies reported a positive effect on their specified outcomes, although no study measured a direct effect on reducing delay.
Conclusion
There was no evidence that any intervention directly reduced primary care delay in the diagnosis of cancer. Limited evidence suggests that complex interventions, including audit and feedback and specific skills training, have the potential to do so.
Keywords: early diagnosis, intervention studies, neoplasms, primary health care
INTRODUCTION
Total delay in cancer diagnosis (from the first symptom noticed by the patient to a diagnosis or treatment) by more than 3 months has been found to have an adverse effect on survival in breast cancer.1 Evidence for other cancers, and which component of the patient pathway is most significant in impairing survival, is lacking.2 UK government policy to reduce diagnostic delay in all cancers has focused on public education, screening, and referral delay,3 the latter consisting of doctor delay (first consultation to referral by a doctor) and hospital delay (referral by a doctor to diagnosis).4 In 1999, the government introduced an initiative for breast cancer, extended to all cancers in 2000, known as the 2-week-wait (2ww) rule.5 This states that if a GP suspects a diagnosis of cancer, then they are required to send a referral within 24 hours and the secondary care provider to give an appointment for consultation within 2 weeks. In 2005, the National Institute for Health and Clinical Excellence (NICE) introduced referral guidelines for primary care concerning symptoms that are suspicious of cancer. These initiatives were prompted by studies suggesting that cancer survival in the UK was poor compared to other European countries,6 although the inference that care is ‘substandard’ continues to be questioned.7 In this review, the term ‘delay’ is used in a non-pejorative manner to represent a potential reduction in the time interval from presentation with a symptom or sign in primary care to referral to secondary care.
A review of audits of the government-initiated 2ww found that many audits were not well reported and their results appeared to show that the guidelines are not uniformly complied with.8 No conclusions were made concerning the effectiveness of the 2ww intervention in reducing diagnostic delay across the whole pathway. While arguments continue concerning the effectiveness of the intervention, NHS commissioners are obliged to follow government policy and take account of NICE guidance in their commissioning decisions.
Austoker et al9 conducted a systematic review of interventions to reduce patient delay (time from noticing a symptom to first doctor consultation),4 which covered both individual and community-level interventions. The authors found no studies that stated reducing delay as an explicit outcome, but found limited evidence for interventions to raise individual and community awareness and encourage early presentation, proxy measures for reduced delay.
Studies concerning medical care in general,10 and breast cancer specifically,11 found that good clinical outcomes may be achieved through improving knowledge, leading to an improvement in attitudes and change in behaviour. This implies that interventions to improve knowledge of particular symptoms or diagnostic skills of primary healthcare professionals (PHPs) may be used as proxy measures for reduced primary care delay. In this review, primary care delay is defined as the time interval from the patient first presenting to a PHP to referral to secondary care.
How this fits in
Primary care is often the first point of call for patients with symptoms that they think may be cancer related. Reducing delay in this section of the cancer care pathway may help to improve cancer survival. This review shows that there is only limited evidence for interventions in primary care.
Primary care has a very important role in the UK, as it is often the first point of call for a patient. Many parts of the consultation have the potential for interventions that would reduce delay in the cancer pathway. The aim of this review is to identify interventions that reduce primary care delay and have the potential to be commissioned locally or enacted by PHPs.
METHOD
Search strategy
Eight electronic databases were searched from inception to March 2010:
MEDLINE (from 1950)
Embase (from 1980)
Cumulative Index to Nursing and Allied Health Literature (CINAHL) (from 1981)
PsycINFO (from 1806)
Health Management Information Consortium (HMIC) (from 1919)
Web of Science (from 1970)
Education Resources Information Centre (ERIC) (from 1966)
Cochrane Library (from 1993).
The search terms included thesaurus and free-text terms for primary care, cancer, and delay. Specific types of intervention such as education and training were also included, as these were likely to give proxy measures for delay. An example of the search used in MEDLINE is given in Box 1. Once relevant articles from these databases had been identified, their reference lists were hand searched, and citation checking was performed to find any more papers that met the inclusion criteria. The reference lists of any relevant reviews were also checked for extra papers. Experts in the field were contacted for additional articles. All references were either downloaded or entered by hand into reference-management software.
Box 1.
“exp*. FAMILY PRACTICE OR exp. PRIMARY HEALTH CARE OR exp. PHYSICIANS, FAMILY OR exp. COMMUNITY HEALTH SERVICES OR “general practi*”† OR GP OR (primary ADJ2‡ care) OR (family ADJ2 physician*) OR (family ADJ2 practi*) OR (family ADJ2 doctor*) OR (community ADJ2 (service* OR care)) OR (general ADJ2 physician*)” |
AND |
“exp. NEOPLASMS OR neoplasm* OR cancer* OR carcinoma* OR malignan* OR lesion* OR tumo?§r” |
AND |
“educat* OR exp. EDUCATION, MEDICAL, CONTINUING OR skill* OR train* OR teach* |
AND |
(earl* ADJ2 diagnos*) OR (earl ADJ2 detect*) OR (delay* ADJ2 diagnos*) OR (delay* ADJ2 detect*) OR (suspicious OR suspect*) OR (earl ADJ2 recog*) OR (delay ADJ2 recog*) OR (present* ADJ2 symptom*) OR (delay* ADJ2 consult*)” |
exp. = exploded thesaurus term – picks up all articles that contain the particular terms as a keyword.
truncation – picks up various different word endings.
adj = adjacent – picks up words next to each other in a document (adj3 = picks up words that have up to 3 words in between them).
? = picks up alternative spellings.
Eligibility criteria
Any primary study that examined an intervention aimed at reducing primary care delay in the diagnosis of cancer was included. The study needed to have been carried out in a primary care setting, with an outcome of reducing delay, and had to include fully trained PHPs as participants. All study types were included. No limits were placed on date ranges or type of cancer.
Exclusion criteria
The following exclusion criteria were applied:
studies that looked to improve screening or self-examination rates, as these involve asymptomatic patients and the review aimed to examine studies of presented symptoms;
studies not carried out in a primary care setting;
studies of healthcare professionals other than fully trained general medical practitioners and practice nurses;
studies with populations in which the effect on primary care professionals was not analysed separately, as it was not possible to distinguish the PHPs from other participants such as medical students and secondary care professionals. Similarly, studies involving community, primary, and secondary care, in which it was not possible to separate the primary care component, were also excluded;
studies on the 2ww, as this service cannot currently be decommissioned by local PHPs and commissioners; and
non-English-language papers, due to resource constraints.
Study selection and assessment of bias
One reviewer screened firstly the titles of all downloaded references, and then the abstracts, in order to remove any papers that did not meet the inclusion criteria. The same reviewer then obtained full texts of all remaining papers. A second reviewer performed checks on 25% of the abstracts and all of the full-text articles, to ensure that the appropriate papers were being included. A 97% concordance rate was found for the abstract check, and an 88% concordance rate was achieved for the full-text articles.
The quality of the articles was assessed independently by both reviewers, using a modified version of the Newcastle–Ottawa Quality Assessment Scale (N–O QAS),12 adapted for use with randomised controlled trials (RCTs) and other studies. The N–O QAS is a generic tool that is split into three sections to cover selection of participants, comparability, and outcome. Stars are awarded depending on the answer given, and in this adapted version of the scale, an RCT could be awarded up to 11 stars and other studies up to 10. A score of ≥7 for an RCT and ≥6 for other studies was deemed to indicate good quality. Scores were then compared and discussed to achieve a consensus on how many stars a paper should be awarded.
Data extraction and analysis
Details of all included studies were entered into a standardised data-extraction form that had been constructed in Microsoft Excel (Table 1), which was filled in independently by two reviewers. The tables were then compared and the data discussed to reach consensus.
Table 1.
Reference | Country | HCP | Type of study | Cancer | Aim of intervention | Details of intervention | Outcome of study | Problems with study | QA score achieved by consensus |
---|---|---|---|---|---|---|---|---|---|
Argenziano et al, 200613 | Italy and Spain | Primary care physicians (PCPs) | RCT | Skin | To determine whether PCPs achieve greater accuracy when triaging skin lesions suggestive of skin cancer using dermoscopic evaluation and the 3-point checklist in addition to the standard clinical examination | Intervention components: the study was divided into four steps; a 1-day training course described the ABCD rule for clinical diagnosis and basic criteria for non-melanoma skin cancers and described the 3-point checklist (simple diagnostic algorithm); in step 2, participants were randomly assigned to assess lesions either by standard clinical examination or by a a hand-held dermatoscope, and consecutive patients asking for screening or exhibiting skin tumours were seen by the PCPs for routine examination; step 3 involved re-evaluation of the patient by dermatologists; and step 4 involved definitive diagnosis by dermatologists. Number of participants recruited: 88. Number of participants completing study: 73. Intervention delivered by: not stated. Length of intervention: 1 day. Time to follow-up: M/A. How intervention was assessed: patients assessed by the participants were re-evaluated by at least two melanoma experts—lesions lesions were evaluated and scored as benign or suggestive of skin cancer; all lesions suggestive of skin cancer were then excised and diagnosed histopathologically to assess participants' referral accuracy | The use of dermoscopy allowed PCPs to perform 25% better triage than naked-eye examination alone, and increased referral sensitivity and accuracy; increased negative predictive value (NPV)—participants were less likely to miss skin cancers | Selection bias; no method of randomisation; no participant details; patient population involved screened as well as presenting patients | 5 stars |
Bedlow et al, 200014 | England | GPs | Before-and-after study | Skin | To evaluate the use of an illustrated booklet supplemented by a didactic lecture on the diagnosis of benign, borderline, and malignant lesions by GPs | Intervention components: two sets of clinical slides showing 10 malignant, 10 benign, and 10 border line lesions were shown. A lecture on the clinical features of such lesions and a booklet that the lecture was based on was then given out following the lecture. Number of participants recruited: 23. Number of participants completing study: 17. Intervention delivered by: not stated. Length of intervention: not stated. Length of follow-up: 2 weeks. How intervention was assessed: GPs attending a seminar on skin cancer were assessed without priorwarning; 2 weeks later, at an unrelated seminar, another test of 30 slideswas shown | Significant improvement in diagnostic ability of GPs following the intervention; 81% of the GPs assessed read the booklet between tests | Selection bias; short follow-up; small sample size; no details on how test slides were assessed (multiple choice); difficult to gauge impact of booklet (only read by 81% of participants, no record of how often read, etc); no information on who presented the lecture | 0 stars |
Brochez et al, 200115 | Belgium | GPs | Before-and-after study | Skin (malignant melanoma) | To see if GPs' clinical ability to diagnose increased after amelanoma lecture (secondary outcome) | Intervention components: lectures on risk groups, clinical aspects, differential diagnosis, and management were given. Number of participants recruited: 160. Number of participants completing study: 146. Intervention delivered by: not stated. Length of intervention: not stated. Length of follow-up: none (immediate post-test). How intervention was assessed: test consisted of slides of different lesions with a brief patient history. GPs had to choose from seven possible diagnoses | Diagnostic accuracy of GPs improved from 49% to 56% following the course; 63% believed their diagnostic ability had improved as a result of the course | Selection bias; no further follow-up; small sample size; no other information such aswho carried out the lectures, what format they were given in, or how many therewere | 5 stars |
Carli et al, 200516 | Italy | GPs | Before-and-after study | Skin (malignant melanoma) | To evaluate the effect of a 4-hour training session on classification and management of lesions by family doctors | Intervention components: Training session consisted of instruction in basic melanoma triage, summary of melanoma and skin cancer epidemiology, prevention and counselling, presentation of images of lesions, and major clinical simulators. Number of participants recruited: 41. Number of participants completing study: 41. Intervention delivered by: two dermatologists experienced in melanoma screening. Length of intervention: 4 hours. Length of follow-up: none (immediate post-test). How intervention was assessed: the pre-test contained 15 images of lesions for which clinicians had to select the correct diagnosis, and the same test was administered after the intervention | Post-intervention, diagnostic accuracy increased from 46.8% to 76.2%. No change in referral rate but suggestion of more appropriate lesion selection | No follow-up; small sample size; selection bias | 5 stars |
Cockburn et al, 200117 | Australia | GPs | Before-and-after study | Breast | Evaluate the impact of guidelines, audit, education, and feedback on investigations of new breast symptoms | Intervention components: participating doctors completed an audit of the management of all women attending with a new (incident) breast symptom over a 12-week period before and after the intervention. The intervention consisted of feedback from the first audit from their own practice and grouped data from other practices, and a seminar on the guideline recommendations and evidence. Number of participants recruited: 227. Number of participants completing study: 104. Intervention delivered by: breast specialists and study authors. Length of intervention: 5–7 months. Length of follow-up: approx 6 months (unclear). How intervention was assessed: via feedback fromthe audits | Guideline adherence improved following the intervention, but already appeared to be quite high at the first audit; statistically significant improvement in five criteria | Selection bias; No control group; no details about who presented the audit findings orwhat format this took; authors acknowledge that participating GPs had more of an interest in breast health and only 18% completed both audits (high dropout rate); not consistent with UK guidelines | |
DeGannes et al, 200418 | Canada | Family physicians | RCT | Skin | To evaluate the effectiveness of a video intervention to promote early detection of skin cancer | Intervention components: participants were randomly assigned to the intervention or control group. The intervention group was given an educational video that participant could watch as of ten as they liked for 6 months. It contained a 12-minute video of skin-cancer, prevention and who is at risk. Nothing for control group. Number of participants recruited: 52. Number of participants completing intervention: 27. Intervention delivered by: video. Length of intervention: 12 minutes. Length of follow-up: 6 months. How intervention was assessed: both received the same internet-based multiple-choice questionnaire pre- and post-intervention, which contained knowledge, prevention, and case-management questions. A skin biopsy audit of all biopsies submitted by participating GP swas also conducted for 6 months pre- and 6 months post-intervention | No statistically significant improvement was observed between the intervention and control group | No follow-up; high dropout rate; the intervention group was already better at diagnosing lesions before the intervention; no record of how many times each participant watched the video - may have had an impact on outcome; the control group was sensitised to cancer through participating in the study | 8 stars |
Dolan et al, 199719 | US | Primary care physicians (internal medicine house staff and attending physicians with outpatient practices) | RCT | Skin | To test whether providing a brief education programme could change skin cancer control attitudes, improve knowledge, and increase counselling and examination among performance patients at moderate to high risk | Intervention components: physicians were stratified by training level and randomised to either the intervention or control group. The intervention group consisted of educational sessions on skin cancer control. One session concerned determining individual risk and targeting prevention strategies, and the other early detection. No details of control group. Number of participants recruited: 96. Number of participants completing intervention: 25. Intervention delivered by: a dermatologistand general internist. Length of intervention: 2 × 1-hour seminars. Length of follow-up: 1 month. How intervention was assessed: both groups filled in a questionnaire before the intervention and 1 month after, which consisted of skin cancer attitudes, beliefs, knowledge, and practice, previous dermatology training, risk factors, and lesion identification | Intervention had no statistically significant effect on skin cancer control attitude, beliefs, knowledge, or behaviours | Self-selected sample; high dropout rate; control groupmay have been sensitised by participation; difficult to determine whether participants were fully trained PHPs | 7 stars |
Gerbert et al 200020 | US | Primary care physicians | Before-and-after study | Skin | To determine whether decision-support software can help primary care physicians proficiently triage lesions that are suggestive of basal cell and squamous cell carcinoma | Intervention components: the software consisted of a clinical information form, decision tree, and support features. The algorithms for the decision tree incorporate the patient's clinical information. The test stimuliwere of 30 digitalised test images of cancerous, pre-cancerous, and benign lesions. Number of participants recruited: not stated. Number of participants completing intervention: 20. Intervention delivered by: developed by two dermatologists. Length of intervention: not stated. Length of follow-up: not stated. How intervention was assessed: participants completed a baseline survey, tests of their triage abilities when using and not using the software, and an exit survey | Participants made fewer incorrect decisions when using the software (36.7% versus 13.3%) | Only a pilot study; small sample; selection bias; no evidence of change in clinical practice | 4 stars |
Gerbert et al, 200221 | US | Primary care physicians | RCT | Skin | To determine whether a skin cancer triage intervention could be effectively delivered over the internet | Intervention components: tutorial modules: registration, pre-test, pre-test scores with individualised feedback, skin cancer instruction, post-test, post-test II, exit survey. participants had unlimited time to complete the modules but could not redo a module once completed. Number of participants recruited: 879. Number of participants completing study: 27 in the intervention group and 19 in the control group completed the whole programme. Intervention delivered by: online. Length of intervention: N/A, online. Length of follow-up: 8 weeks between post-tests I and II. How intervention was assessed: the pre-test module included 36 images of skin lesions (cancerous and non-cancerous), for which participants had to select a diagnosis and evaluation plan. Individualised feedback was then given. Post-test I contained a set of 25 different lesions. Post-test II contained the same lesions as given in post-test I | Intervention participants significantly improved on overall diagnosis and evaluation planning, and in diagnosing malignant melanoma; only malignant melanoma accuracy persisted at 8-week follow-up (correctly diagnosed 88% pre-test and 90% at post-test II) | High dropout rate; no set amount of time for participants to complete modules — may have been able to look up information, therefore giving false results; Problem with randomisation — differences between control and intervention participants; small clinical effect | |
Girgis et al, 199522 | Australia | Family doctors | RCT | Skin | To evaluate the effectiveness of a postgraduate training programme in improving family doctors' levels of knowledge and their clinical practice in relation to skin cancer diagnosis and management | Intervention components: split into three sessions; epidemiology, diagnosis and management (lecture session), melanoma unit (accompanying a specialist surgeon in follow-up examinations), and practical surgical procedures. Number of participants recruited: 65. Number of participants completing study: 41. Intervention delivered by: members of the local melanoma unit. Length of intervention: not stated—first two sessions were 3 hours each. Length of follow-up: not stated. How intervention was assessed: mailed survey assessing confidence and knowledge; a set of coloured slides and case histories to test diagnosis and management; patient self-report and pathology data were checked to assess increases in screening rates; and proportion of attempted diagnoses made | Significant improvements were made in the proportion of accurate diagnoses, correct management of test lesions, and confidence of doctors to provide advice to patients | Lack of detail about the length of time between intervention and follow-up; poor randomisation process; selection bias; no evidence of implications for practice; small sample size and evidence that the study is not representative of all GPs | 6 stars |
Jiwa et al, 200623 | UK | General practices (not clear if GPs only or other healthcare professionals were included) | RCT | Colorectal cancer | To examine whether the introduction of an electronic interactive referral pro forma or educational outreach visits by a colorectal surgeon to general practice can alter the case mix of patients referred to lower-bowel specialists | Intervention components: practices were randomised to receive either an educational outreach visit, an interactive electronic pro forma for processing referrals, both, or neither. The outreach visit involved a surgeon delivering short educational sessions tailored to the needs of the target audience, which summarised the features of significant colorectal disease and encouraged questions. The electronic forms requested information from drop-down menus for 15 signs and symptoms previously identified by GPs and surgeons as predicting colorectal disease. Once clinical data were entered, a referral letter was automatically produced, seeking an appropriate appointment at a hospital clinic. Number of participants recruited: 44 practices (180GPs). Number of participants completing study: not stated (assumed all recruited). Intervention delivered by: colorectal surgeons. Length of intervention: 6 months. Length of follow-up: 6 months. How intervention was assessed: an ‘assessment score’ was given for the quality of referral letters and the proportion of patients referred. Interviews were carried outwith participants who were assigned the electronic referral, but no information is given for the other groups | No evidence that either intervention was successful and did not increase the proportion of patients with pathology who were referred; pro forma documented better assessment of patients | Little information on assessment of participants; only 18% of GPs actually used the software; no a assessment of knowledge before and after the educational visit to assess impact | 7 stars |
Khan, 200924 | UK | GPs | Feasibility study | Colorectal cancer | To test the feasibility of a paper -based assessment tool incorporating the CAPER score (a clinical prediction rule for patients presenting to primary care with lower-gastrointestinal symtpoms) | Intervention components: a paper-based assessment tool was developed using the CAPER score and NICE guidelines for colorectal cancer. The GP had to record the patients' symptomhistory, clinical and rectal examinations, full blood count (FBC), and faecal occult blood (FOB) (if no overt bleeding), and then follow the referral advice. Number of participants recruited: 122. Number of participants completing study: 122. Intervention delivered by: GPs. Length of intervention: maximum of 3 months. Time to followup: 3 months. How intervention was assessed: three audits were conducted during and after the intervention to obtain consultation data and to check if the assessment tool was being correctly filled in. The final audit collected the clinical outcomes and final diagnoses of patients | Recruitment rateswere poor, with only 24% of the recruitment target being achieved. Rates were higher for GP than reception staff recruitment. As directed by the assesment tool, 93% of patients had a clinical examination but only 64% a rectal examination. 48% had a FBC and 38% FOB tests. In only 55% were the CAPER scores correctly calculated | Inadequate amount of follow-up; poor recruitment; selection bias; not powered to detect clinical effect | 4 stars |
Kirklin et al, 200725 | England | GPs and primary care nurses | RCT | Skin | To investigate whether the observational skills of doctors and nurses can beimproved by arts-based observational skills training | Intervention components: all practices received an on-site educational outreach programme. The intervention group received practical observational skills training delivered by an artist, while the control group had a lecture on the aetiology and management of psoriasis and acne and applying emollients, delivered by a dermatologist. Number of participants recruited: 68. Number of participants completing study: 68. Intervention delivered by: artist (intervention), dermatologist (control). Length of intervention: 90 minutes, plus 30 minutes for testing. Length of follow-up: not stated. How intervention was assessed: the test consisted of participants describing a series of three dermatological photographs (diagnosis was not an outcome) | The intervention appeared to have a significant effect on improving observational skills in the post-intervention test; the authors acknowledge that the study did not express how useful such skills would be in practice | Not clear if any follow-up was carried out; participants were not randomised; selection bias | 6 stars |
Logan et al, 200226 | England | General practices | RCT | Bowel | To assess the adequacy of investigation of iron deficiency anaemia and to establish whether a simple computer-generated prompt would increase the completeness of treatment and investigation of patients presenting in general practice | Intervention components: practices were randomised after being stratified by district and practice size. Patients were identified fromtheir blood indices—laboratory computers were programmed to print an appropriate prompt (control prompt: ‘consistent with iron deficiency: ?cause’, and intervention prompt: ‘consistent with iron deficiency:- ?cause, suggest treat with ferrous sulphate, 200mg t.i.d for 4 months but check response in 3–4 weeks. Simultaneously investigate cause. Consider barium enema to exclude colorectal problems’). Number of participants recruited: 603. Number of participants completing study: 431 patients included in analysis. Intervention delivered by:Hospital pathologists. Length of intervention: 12 months. Length of follow-up: 12 months. How intervention was assessed: patient records were analysed for adequate management of anaemia instigated within 3 months of symptom presentation, as defined by four criteria. | 47% of patients were managed adequately, but the prompt was not found to affect the level of investigation or adequacy of follow-up. There was an increase in iron therapy | No ‘no prompt’ control group; cross contamination (GPs in control and intervention groups may influence each other); outcome measure based on secondary care management guidelines | 9 stars |
Moreno-Ramirez et al, 200527 | Spain | GPs | Before-and-after study | Skin | To compare diagnosis and management options after the evaluation of clinical and dermatoscopic teleconsultations with a store-and-forward teledermatology screening system for pigmented lesions | Intervention components: all patients attending a participating primary care centre who met the eligibility criteria were enrolled into the study. Digital clinical pictures and dermatoscopic pictures were taken of the patients and submitted to the hospital. All patients underwent normal face-to-face consultations and skin biopsy. Number of participants recruited: 63 patients. Number of participants completing study: 61 patients. Intervention delivered by: GPs. Length of intervention: 1 month. Length of follow-up: 4 months. How intervention was assessed: two participant dermatologists from the PLC evaluated teleconsultations without dermatoscopic images, and completed a form listing the possible diagnoses, their confidence level, management decision, and whether the patient should be referred at two time points during the study. They then were re-assessed 2 months later using the same information with dermoscopic images | Dermoscopic teleconsultation yielded more relevant referrals than the clinical teleconsultation, and GPs spent more time on the dermoscopic teleconsultation. Without dermascopy referral was recommended in 47.5% of cases as opposed to 39.3% with dermascopy; no false negatives | Small sample size; short follow-up; selection not randomor consecutive; not felt to be cost-effective | 4 stars |
Nekhyludov et al, 200811 | US | Primary care clinicians (physicians, nurse practitioners, certified nurse midwives, and physician assistants) | Before-and-after study | Breast | Pilot test and evaluation of an office-based intervention aimed at improving outcomes by increasing the use of breast-symptom-related guidelines | Intervention components: sites were split into intervention and controls (no information about how this was done). At the intervention sites, at every visit of a female patient with breast symptoms, clinicians were sent a packet of materials containing two guidelines and a patient information sheet. Control sites were informed of the materials but were not given any directly. Number of participants recruited: 123. Number of participants completing intervention: 101. Intervention delivered by: not stated. Length of intervention: 8 months. Length of follow-up: not stated. How intervention was assessed: surveys to measure knowledge of breast symptoms based on information provided by the guidelines were administered before and after the intervention | The intervention was found to increase guideline use, but had no statistically significant effect on knowledge and attitudes | Short follow-up; not randomised—participants were mainly female which makes generalising the results more difficult; small sample size; no objective evidence that the guidelines were actually used | 4 stars |
Raasch et al, 200028 | Australia | Family physicians | RCT | Skin | Would practitioners' management of skin cancer improve if feedback highlighted specific deficiencies in their diagnosis and performance of skin excisions and grouped peer data for comparison? | Intervention components: doctors obtained patients' consent for an audit of excised skin lesions. Information from this was then fed back to the doctors and included an educational message (intervention group). The control group received no feedback. Number of participants recruited: 46. Number of participants completing intervention: 41. Intervention delivered by: not stated. Length of assessment: 9 months. Length of follow-up: 3 months. How intervention were assessed: audits were conducted before and after the feedback | Doctors in the intervention group improved their recordings of the clinical diagnosis on the pathology request form. There was no significant improvement in correct diagnosis; significant differences were found between the intervention and control group at baseline | High dropout rate; small sample size; possible contamination between intervention and control groups; poor randomisation | 9 stars |
Shah et al, 200629 | Egypt and Tunisia | Primary care physicians | Before-and-after study | Breast | To improve knowledge of epidemiology and management of breast cancer cases | Intervention components: participants were invited to a conference, which consisted of a pre-test session, a presentation of an educational module (30-minute presentation and question and answer session by a local oncologist) and post-testing. Number of participants recruited: 194. Number of participants completing intervention: not stated. Intervention delivered by: local oncologist. Length of assessment: not stated. Length of follow-up: none (immediate post-test). How intervention was assessed: the test questions covered topics included in the educational module, focusing on epidemiology, management, and referral in a true/false format | Significant improvement in knowledge after the intervention, but for many questions there was no significant change and Tunisian participants generally performed worse then Egyptian participants | Selection bias; not clear who presented the educational module (the oncologist or a member of the study team); no follow-up to see how long the effects lasted | 1 star |
Soliman et al, 200630 | Pakistan | Primary care physicians | Before-and-after study | Breast | To measure the effect of a continuing medical education breast cancer module on knowledge of primary care physicians | Intervention components: PowerPoint-based educational module that included breast cancer risk factors, early detection by clinical breast examand appropriate referral. Number of participants recruited: 183. Number of participants completing intervention: 133. Intervention delivered by: not stated. Length of intervention: not stated. Length of follow-up: none (immediate post-test). How intervention was assessed: tests were before and after the presentation, and included questions on current knowledge of several types of breast cancer, in a true/false format | Participants showed improvement for all knowledge questions following the intervention. The study controlled for postgraduate education and number of breast cancer cases seen in the previous year | No follow-up period to assess knowledge retention; small sample; selection bias | 1 star |
Westerhoff et al, 200031 | Australia | Primary care physicians | RCT | Skin (malignant melanoma) | To determine whether primary care physicians could improve their melanoma diagnosis using surface microscopy after a short educational intervention | Intervention components: the intervention group was given an atlas that described a method for diagnosing invasive melanomas, and a 1-hour presentation on surface microscopy that reviewed the diagnostic method in the atlas and included a quiz. Number of participants recruited: 74. Number of participants completing intervention: not stated. Programme delivered by: not stated. Length of intervention: not stated. Length of follow-up: median time 23 days. How intervention was assessed: the tests included 100 multiple-choice photos of skin lesions | Significant improvement in diagnostic ability post-intervention (pre-intervention 54.6% correct, post-education 62.7% correct, post-education and surface microscopy 75.9%) | Small sample size; selection bias; short follow-up; range of 2–54 days between completing intervention and completing post-test, no set time, may have affected results | 5 stars |
Wolters et al, 200432 | The Netherlands | GPs | RCT | Prostate | To examine the effects of a distance-learning programme on patient outcomes | Intervention components: the intervention group received amultifaceted distance learning programme: (1) a package for individual learning (PIL), (2) consultation support materials including a voiding diary, the International Prostate Symptom Score and Bother Score, (3) the Dutch lower urinary tract symptoms (LUTS) guideline summarised into a decision tree (4) 2 PILs (one on PSA testing and one on LUTS treatments). The PIL consisted of a knowledge test—answers were sent to a central institute and a set of standard correct answers was returned as feedback. Number of participants: 142. Number of participants completing intervention: 89. Intervention delivered by: N/A. Length of intervention: not stated. Length of follow-up: not stated. How intervention was assessed: via the questionnaire. | PSA testing requests were higher in the intervention group (not a desired study outcome); patients' fear of cancer was more of a motivating factor for GPs. Intervention did decrease the number of urology referrals. This was not expected, as the decision-support tree was felt to suggest a low rate of testing. | High dropout rate; small sample size | 8 stars |
You et al, 200733 | Australia | GPs | Before-and-after study | Skin | To evaluate a self-instructional education module with audit and feedback, designed to increase the skills of GPs in diagnosing melanocytic lesions and skin cancer | Intervention components: education package to assist in diagnosis and management of skin cancers and improve knowledge of the management and treatment of suspicious lesions, and procedures and guidelines for management and follow-up. Number of participants recruited: 16. Number of participants completing intervention: not stated (assumed all). Intervention delivered by: not stated. Length of intervention: 18 months. Length of follow-up: 6 months. How intervention was assessed: a baseline clinical self-audit of skin excisions was carried out over 6 months, followed by a feedback report and provision of the educational module over the next 6 months. A post-education self-audit was then redone for the last 6 months | Intervention appeared to improve the malignant: benign ratio for excised malanocytic lesions—double the number of malignant melanomas were diagnosed | Small sample size, not randomly selected; high dropout rate 3 stars |
N/A = not available. QA = quality assessment. PLC = pigmented lesion clinic.
RESULTS
The search for articles resulted in 1799 references, of which 22 were found to meet the inclusion criteria and were included in the review (Figure 1).
The majority of the studies included focused on skin cancer (n = 14), while the remaining papers looked at breast (n = 144), colorectal (n = 143), and prostate (n = 141) cancers. Ten of the 22 included studies were carried out in Europe, with five of these being carried out in the UK. Five of the remaining studies were conducted in Australia, four in the US, one in Canada, one in Egypt and Tunisia, and one in Pakistan. Studies were heterogeneous in terms of methodology and outcome, with no studies directly examining the primary outcome of reducing delay in the cancer care pathway. Instead, the outcomes included improved knowledge, improved observational skills or diagnostic accuracy, and improved referral rates. The types of intervention included were education (n = 149), audit and feedback (n = 143), decision-support software and guideline use (n = 145), diagnostic tools (n = 144), and other specific skills training (n = 141).
Overall, only six studies (all RCTs) scored above the study thresholds for good quality. Common problems across the articles included: selection bias, less than 1 year’s follow-up of participants, small sample size, high dropout rate, and a lack of detail as to how the intervention was administered.
Intervention type
Education
Four before-and-after studies used traditional lecture settings,14,15,29,30 and one used a training session16 to try to improve knowledge of risk factors and diagnostic ability among participants. None of these studies were rated as good quality. All five involved similar assessments (pre- and immediate post-test), and all showed some improvement in knowledge of risk factors and diagnostic ability. The RCTs included used a variety of educational interventions including lectures, videos, and an internet triage package.18,19,21,22 The interventions were compared to control groups that received no intervention. They were of mixed quality and provided inconsistent evidence. The two studies that reported a positive outcome of improved diagnosis and management did not see the improvement continue to the end of the study period.21,22 Overall, studies rated as high quality reported poorer outcomes than those rated as low quality, and positive outcomes were only seen in the short term.
Audit and feedback
Three studies attempted to use audit and feedback to try to decrease primary care delay. One high-quality RCT,28 where participants either did or did not receive feedback, reported improvement in the recording of clinical information but not in diagnostic accuracy. The low-quality before-and-after studies did show significant improvement in outcomes.17,33
Decision-support software and guideline use
Four studies, including three good-quality RCTs, tested computer-based interventions to change PHP behaviour.20,23,26,32 These studies were designed to improve the quality of referrals to secondary care as a proxy to reducing referral delay. Jiwa et al investigated the effectiveness of an educational outreach visit,23 an ‘electronic referral pro forma’, or both, on changing the type of colorectal referrals to secondary care. The control group did not receive an intervention. Neither intervention component was found to improve the appropriateness of referrals to secondary care, although the pro forma resulted in better documentation of those patients who were referred. Logan et al developed a prompt to improve the completeness of investigation and treatment of iron-deficiency anaemia (IDA).26 The control group received a prompt to investigate the possible cause of the patient’s IDA but no further information. This intervention was not shown to be effective in changing referral behaviour, but prescriptions of iron therapy increased in the intervention group. Wolters et al conducted a distance learning programme for prostate-specific antigen (PSA) testing and lower urinary tract symptoms.32 The authors found a decrease in urology referrals and a non-statistically significant rise in PSA testing, contrary to the outcome expected from the decision support tree.
In a low-quality before-and-after study, Gerbert et al devised software to improve triage of skin lesions, which included a decision tree, clinical information form, and support features.20 The authors found that participants made fewer incorrect decisions on clinical pictures when using the software.
A low-quality before-and-after study by Nekhlyudov et al tested an intervention to improve breast cancer outcomes by improving the use of breast-symptom-related guidelines.11 Although the study was found to increase guideline use, no effect on knowledge or attitudes was found.
Diagnostic and assessment tools
Four studies tried to improve particular skills or the use of equipment to increase diagnostic accuracy, but none were deemed to be good quality.
Moreno-Ramirez et al’s before-and-after study of skin cancer used teledermatology to investigate whether clinical examination or dermoscopy was more effective in improving the appropriateness of referrals to secondary care.27 Dermoscopic teleconsultations were found to lead to more relevant referrals. Khan conducted a feasibility study to explore whether an assessment tool using the CAPER score (a clinical prediction rule) for colorectal cancer would be useful in general practice.24 It was found that uptake of the tool was low.
Argenziano et al’s RCT13 involved PHPs first attending a 1-day training course describing the ABCD (Asymmetric, Border, Colour, Diameter) rule for skin cancer34 and a three-point checklist. They were then randomly assigned to assess patients with skin lesions, either by standard clinical examination or by dermoscopy, and these assessments were checked for accuracy, by dermatologists. Dermoscopy improved triage, referral sensitivity, and accuracy, and led to participants being less likely to miss skin cancers in comparison to standard clinical examination. Westerhoff et al’s RCT evaluated an educational intervention for improving melanoma diagnosis using surface microscopy.31 The control group did not receive the training. Participants’ diagnostic ability on clinical photographs was found to significantly improve in those who had received the microscopy training.
Other specific skills training
Kirklin et al investigated whether arts-based skills could help improve observational skills.25 This low-quality RCT involved the intervention group being taught observational skills by an artist, while the control group received a traditional lecture on skin diseases. Observational skills were found to improve significantly more in the intervention group than the control group, which may lead to more accurate diagnoses in practice.
DISCUSSION
Summary
None of the studies identified had reduction in primary care delay in cancer referral as the primary outcome, and all used proxy measurements, many of which were not of clinical behaviour change. The evidence for the effectiveness of these studies was overall mostly positive, with 15 out of the 22 interventions having an impact on improving at least one of the outcomes measured. In contrast, the high-quality RCTs failed to show an improvement in aspects of management and knowledge using education,18,19 diagnostic accuracy using audit and feedback,28 and referral accuracy and investigation using decision-support software.23,26 Mixed effects were found in aspects of management using educational material and a guideline.32 It was not always possible to pinpoint which particular aspects of the interventions led to particular improvements.
Strengths and limitations
Several problems were encountered in the studies included in this review. Some studies had a high risk of selection bias,13–17,19,20,22,24,25,27,29–31 with little information given (if any) about whether those who participated were different from those who chose not to take part. Almost all studies reported a follow-up period of less than 1 year;11,14,17–19,21,23,24,27,28,31,33 several tested participants straight after the intervention;15,16,29,30 and some did not include or state any follow-up period.20,22,25,32 Therefore, it cannot be determined whether any of the reported improvements persisted. Less than half of the 22 studies included recruited more than 100 participants,11,15,17,21,23,24,26,29,30,32 and several studies suffered from a high dropout rate,18,19,28,32,33 with only 18% of participants in one study17 completing the intervention. These studies have low internal validity and may suggest that the intervention was not accepted by the PHPs. Among the RCTs, the randomisation process of some studies was either poorly reported or inadequate.13,18,21,22,25,26,28 In some cases, the control group may also have been sensitised to cancer through participation in the study,18,19 in that their participation heightened their awareness of the topic. Generally, not enough detail was provided in the studies, especially with regard to information about the intervention and participants, to be able to assess the potential risk of bias.
Only papers published in English were included in this review, due to lack of resources for translation. Two papers the authors wished to examine in full could not be supplied by inter-library loan from the British Library. This led to four papers not being screened fully for inclusion. On closer examination of the article titles, abstracts, and journal titles of publication, it is unlikely these would have met the inclusion criteria and been included in the review.
Comparison with existing literature
In Austoker et al’s review of studies to reduce patient delay,9 the authors concluded that interventions that are tailored to the particular group involved are more likely to improve outcomes. The present review provides some support for this, in that the types of intervention that appeared to be most effective were audit and feedback studies and specific skills training. Giving participants information specific to their practice appears to be more useful than giving general information on cancer-referral management. Other reviews and studies have focused on various methods to improve knowledge, such as continuing medical education10,13,35 and audit and feedback,36 which have given mixed evidence as to the effectiveness of these types of interventions, although in general there appears to be a small positive effect in improving professional practice and healthcare outcomes for patients. Mansouri and Lockyer concluded that more-effective interventions tend to be more interactive, use a range of methods, and be used with small groups.35
Implications for research and practice
While evidence is limited, audit and feedback and specific skills training appear to be the most effective in improving outcomes other than short-term ones, and the success of these studies and comparison with other existing literature indicate that interventions are most useful when tailored to the needs of the population being studied. No studies were found that explicitly measured delay as an outcome. This could be because of the considerable difficulties in undertaking cancer research in primary care due to the rarity of the disease in this population. Future research that attempts to measure delay directly is needed to determine which type of intervention is most effective in reducing the time for referral from primary to secondary care. Such studies require large participant populations and associated funding. This, in combination with improved reporting of methods and results, longer follow-up periods, and attempts to increase retention of participants, will be necessary to increase confidence in the conclusions of these studies and subsequent improvement in health care.
Acknowledgments
We wish to thank Drs Giri Rajaratnam, Judith Bell, Zafar Iqbal, Max Kalsi, and Jackie Small, Professor Peter Croft, and Brian Dudley for commenting on the study design and draft paper.
Funding
NHS North Staffordshire; NHS Stoke; NHS Executive West Midlands R&D Office; North Staffordshire and Cheshire R&D Consortium.
Provenance
Freely submitted; externally peer reviewed.
Competing interests
The authors have declared no competing interests.
Discuss this article
Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss
REFERENCES
- 1.Richards MA, Westcombe AM, Love SB, et al. Influence of delay on survival in patients with breast cancer: a systematic review. Lancet. 1999;353(9159):1119–1127. doi: 10.1016/s0140-6736(99)02143-1. [DOI] [PubMed] [Google Scholar]
- 2.Neal RD. Do diagnostic delays in cancer matter? Br J Cancer. 2009;101(suppl 2):S9–S12. doi: 10.1038/sj.bjc.6605384. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Neal RD, Allgar VL, Ali N, et al. Stage, survival and delays in lung, colorectal, prostate and ovarian cancer: comparision between diagnostic routes. Br J Gen Pract. 2007;57(536):212–219. [PMC free article] [PubMed] [Google Scholar]
- 4.Macdonald S, Macleod U, Mitchell E, et al. Factors influencing patient and primary care delay in the diagnosis of cancer: a database of existing research and its implications for future practice. Report to the Department of Health, Cancer Symptom Profiles and Referral Strategies for Primary Care Research Programme 2004. Project Ref 1217522.
- 5.Department of Health. The NHS cancer plan. London: Department of Health; 2000. [Google Scholar]
- 6.Richards M. EUROCARE-4 studies bring new data on cancer survival. Lancet. 2007;8(9):752–753. doi: 10.1016/S1470-2045(07)70247-4. [DOI] [PubMed] [Google Scholar]
- 7.Souhami R. Are UK cancer cure rates worse than in most other European countries? Br J Gen Pract. 2010;60(571):81–82. doi: 10.3399/bjgp10X483102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Lewis R, Collins R, Flynn A, et al. A systematic review of cancer waiting time audits. Qual Saf Health Care. 2005;14(1):62–66. doi: 10.1136/qshc.2004.013359. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Austoker J, Bankhead C, Forbes LJ, et al. Interventions to promote cancer awareness and early presentation: systematic review. Br J Cancer. 2009;101(Suppl 2):S31–S39. doi: 10.1038/sj.bjc.6605388. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Forsetlund L, Bjorndal A, Rashidian A, et al. Continuing education meetings and workshops: Effects of professional practice and health care outcomes (review) Cochrane Database Syst Rev. 2009;(2):CD003030. doi: 10.1002/14651858.CD003030.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Nekhlyudov L, Nicola M, Jung I, Buechler E. Clinicians’ knowledge and attitudes about breast symptom management: Is there a use for clinical guidelines? J Womens Health. 2008;17(1):57–65. doi: 10.1089/jwh.2006.0296. [DOI] [PubMed] [Google Scholar]
- 12.Wells GA, Shea B, O’Connell D, et al. The Newcastle–Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp (accessed 29 Jun 2011) [Google Scholar]
- 13.Argenziano G, Puig S, Zalaudek I, et al. Dermoscopy improves accuracy of primary care physicians to triage lesions suggestive of skin cancer. J Clin Oncol. 2006;24(12):1877–1882. doi: 10.1200/JCO.2005.05.0864. [DOI] [PubMed] [Google Scholar]
- 14.Bedlow AJ, Cliff S, Melia J, et al. Impact of skin cancer education on general practitioners’ diagnostic skills. Clin Exp Dermatol. 2000;25(2):115–118. doi: 10.1046/j.1365-2230.2000.00590.x. [DOI] [PubMed] [Google Scholar]
- 15.Brochez L, Verhaeghe E, Bleyen L, Naeyaert J-M. Diagnostic ability of general practitioners and dermatologists in discriminating pigmented skin lesions. J Am Acad Dermatol. 2001;44(6):979–986. doi: 10.1067/mjd.2001.113442. [DOI] [PubMed] [Google Scholar]
- 16.Carli P, De G V, Crocetti E, et al. Diagnostic and referral accuracy of family doctors in melanoma screening: Effect of a short formal training. Eur J Cancer Prev. 2005;14(1):51–55. doi: 10.1097/00008469-200502000-00007. [DOI] [PubMed] [Google Scholar]
- 17.Cockburn J, Pit S, Zorbas H, Redman S. Investigating breast symptoms in primary care: enhancing concordance with current best advice. Cancer Detect Prev. 2001;25(5):407–413. [PubMed] [Google Scholar]
- 18.De Gannes GC, Ip JL, Martinka M, et al. Early detection of skin cancer by family physicians: a pilot project. J Cutan Med Surg. 2004;8(2):103–109. doi: 10.1007/s10227-002-0142-1. [DOI] [PubMed] [Google Scholar]
- 19.Dolan NC, Ng JS, Martin GJ, et al. Effectiveness of a skin cancer control educational intervention for internal medicine housestaff and attending physicians. J Gen Intern Med. 1997;12(9):531–536. doi: 10.1046/j.1525-1497.1997.07106.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Gerbert B, Bronstone A, Maurer T, et al. Decision support software to help primary care physicians triage skin cancer: a pilot study. Arch Dermatol. 2000;136(2):187–192. doi: 10.1001/archderm.136.2.187. [DOI] [PubMed] [Google Scholar]
- 21.Gerbert B, Bronstone A, Maurer T, et al. The effectiveness of an internet-based tutorial in improving primary care physicians' skin cancer triage skills. J Cancer Educ. 2002;17(1):7–11. doi: 10.1080/08858190209528784. [DOI] [PubMed] [Google Scholar]
- 22.Girgis A, Sanson-Fisher RW, Howe C, Raffan B. A skin cancer training programme: Evaluation of a postgraduate training for family doctors. Med Educ. 1995;29(5):364–371. doi: 10.1111/j.1365-2923.1995.tb00027.x. [DOI] [PubMed] [Google Scholar]
- 23.Jiwa M, Skinner P, Coker AO, et al. Implementing referral guidelines: lessons from a negative outcome cluster randomised factorial trial in general practice. BMC Fam Pract. 2006;7:65. doi: 10.1186/1471-2296-7-65. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Khan NF. Implementation of a diagnostic tool for symptomatic colorectal cancer in primary care: a feasibility study. Prim Health Care Res Dev. 2009;10:54–64. [Google Scholar]
- 25.Kirklin D, Duncan J, McBride S, et al. A cluster design controlled trial of arts-based observational skills training in primary care. Med Educ. 2007;41(4):395–402. doi: 10.1111/j.1365-2929.2007.02711.x. [DOI] [PubMed] [Google Scholar]
- 26.Logan ECM, Yates JM, Stewart RM, et al. Investigation and management of iron deficiency anaemia in general practice: a cluster randomised controlled trial of a simple management prompt. Postgrad Med J. 2002;78(923):533–537. doi: 10.1136/pmj.78.923.533. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Moreno-Ramirez D, Ferrandiz L, Galdeano R, Camacho FM. Teledermatoscopy as a triage system for pigmented lesions: a pilot study. Clin Exp Dermatol. 2005;31(1):13–18. doi: 10.1111/j.1365-2230.2005.02000.x. [DOI] [PubMed] [Google Scholar]
- 28.Raasch BA, Hays R, Buettner PG. An educational intervention to improve diagnosis and management of suspicious skin lesions. J Contin Educ Health Prof. 2000;20(1):39–51. doi: 10.1002/chp.1340200108. [DOI] [PubMed] [Google Scholar]
- 29.Shah NM, Soliman AS, Banerjee M, et al. Knowledge gained after a brief CME module on breast cancer diagnosis. J Cancer Educ. 2006;21(3):169–174. doi: 10.1207/s15430154jce2103_17. [DOI] [PubMed] [Google Scholar]
- 30.Soliman AS, Samadi S, Banerjee M, et al. Brief Continuing Medical Education (CME) module raises knowledge of developing country physicians. International Electronic Journal of Health Education. 2006;9:31–42. [Google Scholar]
- 31.Westerhoff K, McCarthy WH, Menzie SW. Increase in the sensitivity for melanoma diagnosis by primary care physicians using skin surface microscopy. Br J Dermatol. 2000;143(5):1016–1020. doi: 10.1046/j.1365-2133.2000.03836.x. [DOI] [PubMed] [Google Scholar]
- 32.Wolters R, Wensing M, van Weel C, Grol R. The effect of a distance-learning programme on patient self-management of lower urinary tract symptoms (LUTS) in general practice: a randomised controlled trial. Eur Urol. 2004;46(1):95–101. doi: 10.1016/j.eururo.2004.02.005. [DOI] [PubMed] [Google Scholar]
- 33.Youl PH, Raasch BA, Janda M, Aitken JF. The effect of an educational programme to improve the skills of general practitioners in diagnosing melanocytic/pigmented lesions. Clin Exp Dermatol. 2007;32(4):365–370. doi: 10.1111/j.1365-2230.2007.02414.x. [DOI] [PubMed] [Google Scholar]
- 34.Friedman RJ, Rigel DS, Kopf AW. Early detection of malignant melanoma: the role of physician examination and self-examination of the skin. CA Cancer J Clin. 1985;35(3):130–151. doi: 10.3322/canjclin.35.3.130. [DOI] [PubMed] [Google Scholar]
- 35.Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof. 2007;27(1):6–15. doi: 10.1002/chp.88. [DOI] [PubMed] [Google Scholar]
- 36.Jamvedt G, Young JM, Kristoffersen DT, et al. Audit and feedback: effects on professional practice and health care outcomes (review) Cochrane Database Syst Rev. 2006;(2):CD000259. doi: 10.1002/14651858.CD000259.pub2. [DOI] [PubMed] [Google Scholar]