Interventions to improve outpatient referrals from primary care to secondary care

Abstract

Background

The primary care specialist interface is a key organisational feature of many health care systems. Patients are referred to specialist care when investigation or therapeutic options are exhausted in primary care and more specialised care is needed. Referral has considerable implications for patients, the health care system and health care costs. There is considerable evidence that the referral processes can be improved.

Objectives

To estimate the effectiveness and efficiency of interventions to change outpatient referral rates or improve outpatient referral appropriateness.

Search methods

We conducted electronic searches of the Cochrane Effective Practice and Organisation of Care (EPOC) group specialised register (developed through extensive searches of MEDLINE, EMBASE, Healthstar and the Cochrane Library) (February 2002) and the National Research Register. Updated searches were conducted in MEDLINE and the EPOC specialised register up to October 2007.

Selection criteria

Randomised controlled trials, controlled clinical trials, controlled before and after studies and interrupted time series of interventions to change or improve outpatient referrals. Participants were primary care physicians. The outcomes were objectively measured provider performance or health outcomes.

Data collection and analysis

A minimum of two reviewers independently extracted data and assessed study quality.

Main results

Seventeen studies involving 23 separate comparisons were included. Nine studies (14 comparisons) evaluated professional educational interventions. Ineffective strategies included: passive dissemination of local referral guidelines (two studies), feedback of referral rates (one study) and discussion with an independent medical adviser (one study). Generally effective strategies included dissemination of guidelines with structured referral sheets (four out of five studies) and involvement of consultants in educational activities (two out of three studies). Four studies evaluated organisational interventions (patient management by family physicians compared to general internists, attachment of a physiotherapist to general practices, a new slot system for referrals and requiring a second 'in‐house' opinion prior to referral), all of which were effective. Four studies (five comparisons) evaluated financial interventions. One study evaluating change from a capitation based to mixed capitation and fee‐for‐service system and from a fee‐for‐service to a capitation based system (with an element of risk sharing for secondary care services) observed a reduction in referral rates. Modest reductions in referral rates of uncertain significance were observed following the introduction of the general practice fundholding scheme in the United Kingdom (UK). One study evaluating the effect of providing access to private specialists demonstrated an increase in the proportion of patients referred to specialist services but no overall effect on referral rates.

Authors' conclusions

There are a limited number of rigorous evaluations to base policy on. Active local educational interventions involving secondary care specialists and structured referral sheets are the only interventions shown to impact on referral rates based on current evidence. The effects of 'in‐house' second opinion and other intermediate primary care based alternatives to outpatient referral appear promising.

Plain language summary

Are there effective methods to improve the process of referring patients to specialised care?

Patients are referred to a specialist when more specialised care is needed. It has however been shown that the process by which patients are referred could be improved. Some patients may be referred to a specialist inappropriately or not be referred when they should have, or when they were referred have unnecessary tests or procedures.

This review found 17 studies that evaluated whether educating health care professionals about referrals, changing the organisation or system of referrals, and changing the fees or payments for referrals, could improve the referral process.

Education: The referral process will most likely improve when guidelines for referral are distributed with standard referral forms and when the health care professionals who are the consultants are involved in teaching about referring. But simply distributing guidelines and providing health care professionals with feedback about how they are referring may not improve the process.

Organisation: There is little evidence about organisational changes. But providing a second opinion before referring, or enhancing the services provided before a referral (e.g. providing access to a physiotherapist) may improve the referral process.

Financial: There is not enough evidence to draw firm conclusions about financial changes. Financial changes can change the number of referrals but it is not known whether they improve the quality or appropriateness of referrals.

Background

The primary‐secondary care interface is a key organisational feature of many health care systems. Primary care physicians provide primary health care and act as 'gatekeepers' with responsibility for defining which patients require secondary care. The referral system is the 'organisational structure for referring medical problems from generalists to specialists' (Coulter 1992). Some countries have a similar formal referral system, for example Denmark, the Netherlands and the UK, where primary care physicians provide health care and act as gatekeepers with responsibility for defining which patients require specialist care. Other countries have a less formalised referral system, for example France, Germany and the United States of America (USA) (Casparie 1988; Gervas 1994; Marinker 1988; Roland 1992).

Patients are referred to specialist care to obtain advice on diagnosis or management, to obtain a specialised procedure when investigation or therapeutic options are exhausted in primary care and more specialised care is needed, and to obtain a second opinion. During referral, there is 'a transfer of responsibility for some aspect of the patient's care' from primary to secondary care (McWhinney 1989). Referral has considerable implications for patients, the health care system and health care costs. However, there is considerable evidence that the process of referral is sub optimal. There are unexplained variations in referral rates (Wilkin 1992), suggesting that some patients are referred inappropriately, consuming health care resources which could have been used to provide other services, and that some patients are inappropriately managed in primary care settings who would benefit from specialist care. There is also evidence of inappropriateness of referral and poor communication at the time of referral (Roland 1992). As a result, patients may undergo unnecessary diagnostic or therapeutic procedures (including hospitalisation).

Despite the growing awareness of problems associated with referrals, there has been relatively little research evaluating interventions to improve referral behaviour compared with other types of behaviour (for example, prescribing). A systematic review of studies evaluating professional interventions to improve referral behaviour identified only four studies published between 1966 and 1995 (Grimshaw 1998). Mixed results were found; training plus structured assessment cards and joint consultation sessions were effective. However, development and dissemination of local consensus guidelines and the introduction of fundholding in UK primary care were found to have little effect. The review concluded that it was difficult to draw firm conclusions as a result of the limited number of rigorous studies identified and that further research was needed on interventions to improve the referral process. Since that review was undertaken, the NHS (Primary and secondary care interface programme) in the UK and other funders have commissioned a number of further studies.

This is an updated version of the Cochrane review published in 2005 (Grimshaw 2005).

Objectives

The aims of the review were: 
 (1) To identify which interventions have been evaluated to change primary care outpatient referral rates or improve referral appropriateness. 
 (2) To estimate the effectiveness of interventions to change primary care outpatient referral rates or improve outpatient referral appropriateness.

Methods

Criteria for considering studies for this review

Types of studies

Randomised controlled trials (RCTs), controlled clinical trials (CCTs), controlled before and after studies (CBAs) and interrupted time series (ITSs).

Types of participants

Primary care physicians, defined broadly as any medically qualified physician who provides primary health care. Primary health care provides 'integrated, easy to access, health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained and continuous relationship with patients, and practising in the context of family and community' (Vanselow 1995). Primary care physicians include general practitioners, family doctors, family physicians, family practitioners and other physicians working in primary health care settings who fulfil primary health care tasks (for example, general pediatricians in the USA).

Specialist physicians working in hospitals or community outpatient settings.

Types of interventions

The review focused on interventions to change outpatient referral rates or improve outpatient referrals appropriateness. During outpatient referral, there is 'a transfer of responsibility for some aspect of the patient's care' from primary to secondary care (McWhinney 1989). Referral is a management option in most diseases, therefore any intervention aiming to influence clinical behaviour could have indirect effects on the quality and quantity of referrals. Studies had to report explicitly that influencing referral was a primary objective of the intervention to be included. Interventions were classified according to the Cochrane EPOC taxonomy of interventions (see SCOPE in GROUP DETAILS).

Interventions to change or improve referrals for open access radiological or laboratory diagnostic investigations (eg radiology) were excluded.

Types of outcome measures

Objectively measured provider performance in a health care setting (for example, referral rates or appropriateness of referral) or health outcomes were included.

Search methods for identification of studies

(1) Cochrane Effective Practice and Organisation of Care Register 
 For the original review, we searched the specialised register and pending register of the Cochrane Effective Practice and Organisation of Care group using the terms: refer* and consultation* with the term outpatient*. The register is based upon retrospective and prospective sensitive searches of key bibliographic databases (including MEDLINE and CINAHL), hand searching of key journals and reference lists of published literature reviews (see SEARCH STRATEGIES FOR THE IDENTIFICATION OF STUDIES under GROUP DETAILS). Potentially relevant studies are entered into the pending register for assessment of the full text articles. Studies are included in the specialised register if they use RCT, CCT, CBA or ITS designs and evaluate interventions within EPOC's scope. Studies in the specialised register are coded by their design, type of intervention and type of targeted behaviour and include the full MEDLINE, EMBASE or Healthstar reference. The register was searched on February 2002 by the EPOC TSC for the initial version of the review. For the update, the EPOC register was searched up to October 2007 (refer to Appendix 01).

(2) MEDLINE 
 For the initial review, we conducted additional test searches of MEDLINE using a search strategy developed by CF. However, a search of the MEDLINE 1995 to 1999 database identified 6,000 records (after records identified by the existing EPOC search strategy were excluded). No additional potentially relevant studies were identified when we screened the first 500 records. As a result, we did not undertake further MEDLINE searches.

For this update, Medline was searched using the following search strategy:

Database: Ovid MEDLINE(R) <1950 to October Week 1 2007> 
 Search Strategy: 
 ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ 
 1 Family Practice/st (3894) 
 2 Primary Health Care/st (3454) 
 3 Physicians, Family/ (11246) 
 4 ((general or family) adj practi$).tw. (49353) 
 5 family physic$.tw. (7470) 
 6 primary care.tw. (38351) 
 7 primary health care.tw. (8717) 
 8 (gp or gps).tw. (21831) 
 9 exp Specialties, Medical/ (369884) 
 10 specialist?.tw. (33613) 
 11 Physician's Practice Patterns/ (23725) 
 12 ((secondary or speciali?ed) adj care).tw. (2153) 
 13 or/1‐12 (483483) 
 14 *"Referral and Consultation"/ (13844) 
 15 (referral? adj2 (practice? or rate? or appropriateness or improve$ or appropriate$ or method? or process$ or accuracy or pattern?)).tw. (4561) 
 16 *Gatekeeping/ (180) 
 17 gatekeeper$.tw. (1305) 
 18 (outpatient? adj3 referral?).tw. (417) 
 19 or/14‐18 (18547) 
 20 referral?.tw. (37828) 
 21 13 and 19 and 20 (3532) 
 22 randomized controlled trial.pt. (244391) 
 23 random$.tw. (389305) 
 24 intervention?.tw. (255306) 
 25 control$.tw. (1596990) 
 26 evaluat$.tw. (1257209) 
 27 effect?.tw. (2522982) 
 28 or/22‐27 (4629321) 
 29 21 and 28 (1469) 
 30 animal/ (4220069) 
 31 human/ (10034045) 
 32 30 not (30 and 31) (3194259) 
 33 29 not 32 (1469) 
 34 limit 33 to yr="1999 ‐ 2007" (924)

(3) UK National Research Register 
 We searched the UK National Research Register with the terms: (outpat* and refer*) and interface. We searched MEDLINE for published reports of completed projects by the name of the lead researcher.

Data collection and analysis

The review was conducted using standard EPOC methods (Bero 2008).

EG, CF, RT and CP screened the results of searches to identify potentially relevant papers. AA and AM screened the results of the updated searches. Two difficulties arose when identifying potentially relevant studies:

(1) Problems associated with definition of intervention ‐ We identified many studies which reported the effects of interventions on referral rates, however the majority of these focused on the general management of a clinical condition rather than on referral. As specified in our inclusion criteria we only considered interventions if they explicitly reported that a primary objective was to influence referrals. 
 (2) Problems associated with definition of primary care ‐ A significant number of potentially relevant studies were conducted in US ambulatory care clinics and it was difficult to determine whether the professionals targeted fulfilled our inclusion criteria. We contacted a number of experts in the US who advised us to consider family physicians and general internal medicine physicians as primary care physicians.

Two reviewers (EG and JG or CP or RT), independently selected the studies to be included in the review, and AA and AM selected studies for the update, with input from JG. A list of excluded studies can be obtained from the authors.

Data from each paper were abstracted independently by two authors (EG and at least one of CP, JG, AM, RT or RW) using the standard EPOC checklist (Bero 2008) and by three authors (AA, AM and MA) for the update. Data abstraction was checked and discrepancies were resolved through discussion by the relevant two authors. JG acted as arbiter for any unresolved discrepancies. If one of the authors of this review was involved in one of the reviewed studies, they did not participate in the abstraction of that study.

Given the substantial heterogeneity of interventions and methods across studies, it was not sensible to use meta‐analysis to pool the results of studies. Instead, we present the results of studies in tabular form and make a qualitative assessment of the effects of studies, based upon the quality, the size and direction of effect observed and the statistical significance of the studies. We report the following data (where available): pre intervention study and control data in natural units and statistical significance across groups, post intervention study and control data in natural units and statistical significance across groups, absolute and relative percentage improvement. If a unit of analysis error was present, we attempted to re‐analyse the study using data provided in the original paper. If this was not possible, we present the point estimates of effects without p‐values or 95% confidence intervals. If the study authors had stated the hypothesised direction of effect for any outcome variable, we noted whether the result favoured the study or control groups.

Results

Description of studies

See Table of Included Studies.

Seventeen studies were included in the original review, all but one study reported since 1990. 
 
 The search for the update identified 1058 hits, of which only 4 were considered potentially eligible. Upon further review, only one additonal study from the updated searches met the inclusion criteria. One study (Krasnik 1990) from the original review was removed. Krasnik 1990 was a CBA with only one intervention and one control group, which does not meet the revised EPOC inclusion criteria for CBA studies. 
 
 Characteristics of participants 
 Twelve included studies were based in the UK, two in the US and one each in the Netherlands, Palestine, and Finland

Characteristics of the intervention 
 Nine studies evaluated professional educational interventions (including 14 comparisons), four studies evaluated organisational interventions and four studies evaluated financial interventions (including five comparisons).

Professional education interventions 
 Seven studies evaluated different methods of disseminating and implementing referral guidelines. Two evaluated passive dissemination of local guidelines (Jones 1993; Grimshaw 1998). Jones (Jones 1993) evaluated the effects of disseminating consensus guidelines for management and referral of patients with dyspepsia. Grimshaw (Grimshaw 1998) evaluated the effects of disseminating locally developed guidelines for four common tracer conditions (low back pain, menorrhagia, suspected peptic ulcer, varicose veins) that accounted for 8% of total referrals. Five studies evaluated dissemination of referral guidelines with structured referral sheets ‐ checklists to be completed at the time of referral prompting the primary care physician about important elements of pre‐referral investigation and management (Abu‐Ramadan 2002; Bennett 2001; Emslie 1993; Morrison 2001; Thomas 2003). Two of these focused on general practitioner management and referral of infertility: Emslie's study (Emslie 1993) was based in a relatively simple referral setting (one referral hospital, traditionally good links between local general practitioners and the hospital), whereas Morrison's study (Morrison 2001) was set in a more complex referral settings (five referral hospitals in large city setting). Thomas' study (Thomas 2003) evaluated the effects of a guideline based, open access investigation service for two common urological conditions. General practitioners could refer patients to a fast track investigation service if they used a structured letter based upon referral guidelines. Abu‐Ramadan's study (Abu‐Ramadan 2002) evaluated the effect of a multifaceted intervention involving educational meetings, a new referral and reply sheet, new staff and changes in equipment and facilities. Bennett (Bennett 2001) included a risk factor checklist and a training video to train the practitioners.

Three studies evaluated secondary care provider‐led educational strategies (Banait 2000; Grimshaw 1998; Vierhout 1995). Vierhout's study (Vierhout 1995) evaluated the effects of joint general practitioner‐consultant sessions for patients whose general practitioners were uncertain of diagnosis or management and were considering orthopaedic referral. The sessions were held monthly for 18 months; each of four orthopaedic surgeons saw patients with the same three general practitioners. Grimshaw (Grimshaw 1998) evaluated GP‐consultant small group discussions for four common tracer conditions (see above). Banait (Banait 2000) evaluated consultant‐led education outreach visits to disseminate guidelines for the management and referral of patients with dyspepsia. Finally Grimshaw's study (Grimshaw 1998) also evaluated the effects of feeding back information about referral rate to general practitioners and discussions between general practitioners and an independent adviser about referral. Due to the complex design, Grimshaw (Grimshaw 1998) was able to explore the effects of the four interventions separately and in combination.

Organisational interventions 
 Bertakis (Bertakis 1987) evaluated the effects of the primary care physician's discipline (general internal medicine versus family physician) on referral behaviour. O'Cathain (O'Cathain 1995) evaluated the effect of providing primary care based physiotherapy services on referrals to orthopaedics and rheumatology. Kinnersley (Kinnersley 1999) evaluated the effects of requiring an 'in house' second opinion prior to referral. General practitioners considering referral arranged for the patient to see a different partner in the same practice for an independent assessment. Bridgman 2005 evaluated the effect of a new system where the number of appointment slots was allocated based on the size of the practices.

Financial interventions 
 Davidson (Davidson 1992) evaluated the effects of a change in remuneration system from a low cost fee‐for‐service system to either a high cost fee‐for‐service system or capitation‐based budgetary system (with some degree of risk sharing by the provider for secondary care provision) for the management of Medicaid eligible paediatric care. Two UK based studies evaluated the effect of fundholding on referral patterns (Coulter 1993, Kammerling 1996). Linnala (Linnala 2001) examined the impact of charging patients the same (lesser) rate to be seen by a private specialist as they would have been charged to see a hospital based specialist.

All of the professional educational interventions focused on a single or a small number of tracer conditions whereas the organisational and financial interventions focused on a broader range of conditions and problems.

Risk of bias in included studies

Study designs 
 Study designs included: eight cluster randomised trials (which randomised by professional or practice) (Banait 2000; Bennett 2001; Davidson 1992; Emslie 1993; Jones 1993; Kinnersley 1999; Morrison 2001; Thomas 2003); two patient randomised trials (Bertakis 1987; Vierhout 1995); one controlled clinical trial (which allocated by practice) (Grimshaw 1998); five controlled before and after studies (Bridgman 2005; Coulter 1993; Kammerling 1996; Linnala 2001; O'Cathain 1995) and one interrupted time series (Abu‐Ramadan 2002).

Quality assessments (Bero 2008) 
 There were eleven randomised or controlled clinical trials. Randomisation concealment was done in four of these studies and was not clear in the remaining seven studies. Adequate follow up of providers (greater than 80%) was done in five studies. Blinded assessment of outcomes (protection against detection bias) was present in one study, was not clear in eight studies and not done in two studies. Baseline measurement was done and no substantial differences were present across study groups in three studies, baseline measurement was not done or it was unclear whether there were significant differences across study groups in six studies and there were baseline differences likely to undermine the post intervention differences in two studies. Reliable outcomes were only used in three studies. Protection against contamination was done for nine studies, not clear for one study and not done for one study. Contamination could have occurred within the Vierhout (Vierhout 1995) study because participating general practitioners would have looked after both study and control patients.

There were five controlled before and after studies. Baseline measurement was reported and similar across study groups for one study, baseline measurement was not done or it was unclear whether there were significant differences across study groups in one study and there were baseline differences likely to undermine the post intervention differences in three studies. The characteristics of the second site were reported and similar for one study; it was not clear whether the characteristics of the second site were similar for the other studies. Follow up of providers was not specified in four studies (reported as NOT CLEAR) and not adequate (less than 80% follow up) in one study (reported as DONE). Blinded assessment of outcomes was done for three studies and not done for two studies. Reliable outcome measures were used in four studies and not clear in one study. All studies adequately protected against contamination.

There was one interrupted time series study. Abu Ramadan (Abu‐Ramadan 2002) monitored referral rates from general practitioners to an eye hospital monthly for 18 months prior and 30 months following the intervention. The intervention was multifaceted and included a new referral and reply sheet, new group and self training of staff and a monitoring system to evaluate and provide advice. The only data reported is the number of referrals, so it is impossible to assess if the data set is complete. There are sufficient points to include the study, but no formal test for trend. The intervention is apparently solely targeted at referrals, but because it is multifaceted, it is difficult to assess what aspect of the intervention is effective. 
 
 Other methodological characteristics 
 Proportion of eligible providers participating in the study 
 In eight studies, the majority of eligible providers participated in the study (Bennett 2001; Banait 2000; Davidson 1992; Emslie 1993; Jones 1993; Linnala 2001; Morrison 2001; Thomas 2003). In two studies less than 50% of the eligible providers were included (Coulter 1993; Grimshaw 1998). It was not clear what proportion of eligible providers participated in seven studies (Abu‐Ramadan 2002; Bridgman 2005; Bertakis 1987; Kammerling 1996; Kinnersley 1999; O'Cathain 1995; Vierhout 1995). Only 12 general practitioners were involved in Vierhout's study (Vierhout 1995).

Types of outcomes reported 
 Studies of professional educational interventions typically reported a combination of data relating to quantity of referrals, quality of referrals and other related outcomes (for example, impact on prescribing or subsequent hospital management). However, all but one (Linnala 2001) of the studies of organisational and financial interventions only reported data on quantity of referrals.

Unit of analysis errors 
 In cluster randomised trials, providers or groups of providers are randomised but data is collected at the patient level. A fundamental assumption of the statistics used to analyse patient‐randomised trials is that the outcome for an individual patient is completely unrelated to that for any other patient ‐ they are said to be 'independent'. This assumption is violated, however, when cluster randomisation is adopted, because patients within any one cluster are more likely to respond in a similar manner. The primary consequence of adopting a cluster randomised design is that it is not as statistically efficient and has lower statistical power than a patient‐randomised trial of equivalent size. Because of this lack of independence, sample sizes require to be inflated to adjust for the clustering effect, and special analytic techniques, such as multi‐level modelling need to be adopted, unless simple cluster‐level analysis is undertaken. If patient level analyses using standard statistical tests are used, the results are likely to be over precise with artificially extreme p‐values and over‐narrow confidence intervals, increasing the chances of spuriously significant findings and misleading conclusions. This is known as a unit of analysis error.

Unit of analysis errors were potentially present in six of the cluster randomised trials (Banait 2000; Bennett 2001; Davidson 1992; Emslie 1993; Jones 1993; Kinnersley 1999) and all but one (Bridgman 2005) of the controlled before and after studies.

Economic evaluation 
 Only two studies conducted an economic evaluation (Morrison 2001; Thomas 2003).

Effects of interventions

See Table 1.

1. Summary of Results.

Study	Measurement Period	Comparisons	Main Process Effect	Main Patient Outcome	Notes
Abu‐Ramadan 2002	Monthly observations for: 19 months before intervention 29 months following intervention		Patient Load on Eye Hospital Staff Pre Intervention: M =5961.89/month Post Intervention: M =3866.38/month Absolute Difference (M): 2095.51patients per month	Not assessed	EDUCATIONAL INTERVENTION ORGANISATIONAL INTERVENTION Multifaceted intervention was independent of other changes, but any effect of referrals component specifically is impossible to separate from other elements of the intervention. Other outcomes of interested were listed and measured, but only patient load results were actually reported in the article.
Banait 2000	6 ‐ 7 months following intervention		Referral rates (referrals per 10,000 patients) Post intervention: 1.42 (study) vs 0.92 (control)a Absolute difference (post): 0.50 referral per 10,000 patients Relative % difference (post): +54%b Appropriateness of upper gastrointestinal endoscopy: Post intervention: 62.5% (study) vs 50.8% (control) (p =*)a Absolute difference: +11.7% (study better) Relative difference:+23.0% Findings at endoscopy Normal: Pre intervention 37.8% (study) vs 39.1% (control) Post intervention 39.5% (study) vs 43.4% (control) (p =*)a Absolute difference post: ‐3.9% (study better) Relative difference post: ‐9.0% Absolute difference from baseline: 1.7% (study) vs 4.3% (control) Difference in absolute change from baseline: ‐ 2.6% Prescribing costs for acid suppressing drugs Overall expenditure (net ingredient cost per prescribing unit): Pre intervention: 4.10 (study) vs 4.08 (control) Post intervention: 4.43 (study) vs 4.16 (p =*)a Absolute different post: + 0.27 (study worse) Relative post difference: + 6.5% Absolute difference from baseline: 0.33 (study) vs 0.08 (control) Difference in absolute change from baseline: + 0.25 Requests for laboratory tests for Helicobacter Pylori Post intervention (median per practice): 4 (study) vs 0 (control) (p< 0.001 Mann‐Whitney test) (study better)	Not assessed	EDUCATIONAL INTERVENTION (a) Unit of analysis error (b) Hypothesised direction unclear
Bennett 2001	One year following intervention	Control (no intervention) vs Checklist vs Video vs Checklist and Video	PPV (%) ‐ Appropriateness of Referrals Pre Intervention Means Checklist Group: 35.57% Video Group: 42.37% Checklist & Video Group: 23.49% Control Group: 45.75% Post Intervention Means Checklist Group: 15.97% Video Group: 23.97% Checklist & Video Group: 51.59% Control Group: 15.15% Pre‐post change (M) Checklist Group: ‐19.6% [95% CI (‐44.8, 5.6)] Video Group: ‐18.4% [95% CI (‐35.2, ‐1.6)] Checklist & Video Group: +28.1% (ANOVA P=0.002) [95% CI (10.2, 46.0)] Control Group: ‐30.6% [95% CI (‐69.9, 8.66)]		EDUCATIONAL INTERVENTION Only 68% of practices included in complete study
Bertakis 1987	Mean length of follow up 2.1 years	Family medicine vs internal medicine	Number of primary care attendances per year: Post intervention 2.6 (family medicine FM) vs 3.2 (internal medicine IM) p<0.001 Absolute difference post: ‐0.6 attendances Relative percentage difference: ‐18.8% Proportion of patients with no emergency room attendances Post intervention: 67.5% (FM) vs 55.2% (IM) ( p < 0.01)a Absolute difference (post): +15.3% Relative percentage difference: +22.7% Proportion of patients with no acute care clinic visits Post intervention: 64.6% (FM) vs 57.0% (IM) (p = NS)a Absolute difference (post): +7.6% Relative percentage difference: +11.8% Proportion of patients with no non primary care clinic attendances Post intervention: 56.1% (FM) vs 38.6% (IM) (p < 0.001)a Absolute difference (post): +17.5% Relative percentage difference: +31.2% (Significantly fewer reductions to obstetrics and gynaecology, general surgery and dermatology clinic noted in family practice group) Median annual costs per patient of laboratory tests Post intervention: $64 (FM) vs $93 (IM) (p < 0.01 Wilcoxon rank sum test) Absolute difference (post): ‐$29 Relative percentage difference: ‐31.3%	Not assessed	ORGANISATIONAL INTERVENTION (a) Based on reanalysis of 2 x 2 Chi square no visit vs any visit using Arcus Biostat.
Bridgman 2005	18 months	New slot system vs control	Rate of referrals/10,000patients/month Baseline Intervention 9.40 (SE 0.41) Baseline Control 10.99 (SE 0.52) 12 months Intervention 7.29 (SE 0.31) Relative improvement 22% 12 months Control 9.90 ( SE 0.39) Relative improvement 10% 18 months Intervention 7.31 (SE 0.0.21) Relative improvement 22% 18 months Control 11.70 (SE 0.48) Relative improvement ‐6%	Not assessed	ORGANISATIONAL INTERVENTIOn
Coulter 1993	6 months pre‐intervention (‐6 ‐ 0 months) and 6 (7 ‐ 12 months) post‐intervention.	Fundholding vs control.	Standardised mean annual referral rates per 1000 population per year NHS Pre intervention: 109.7 (study) vs 97.5 (control) Post intervention:112.1 (study) vs 122.3 (control) (NS)a Absolute difference (post): ‐10.3 Relative percentage difference (post): ‐8.4% Absolute difference from baseline: +2.1 (study) vs + 24.8 (control) Difference in absolute change from baseline: ‐22.7 Private Pre intervention: 29.4 (study) vs 27.7 (control) Post intervention:26.6 (study) vs 28.8 (control) (NS)a Absolute difference (post): ‐2.2 Relative percentage difference (post): ‐7.6% Absolute difference from baseline: ‐2.8 (study) vs +1.1 (control) Difference in absolute change from baseline: ‐4.5	Not assessed.	PROFESSIONAL FINANCIAL INCENTIVES (a) Unit of analysis error. Reanalysed using T‐tests in Arcus Biostat.
Davidson 1992	6 months pre intervention, post intervention time period not specified	Capitation (I1) vs high fee for service (I2) vs low fee for service (control)	Mean annual number of primary care visits Pre intervention: 3.22 (I1) vs 3.68 (I2) vs 3.06 (control) Post intervention: 2.89 (I1) vs 3.71 (I2) vs 2.47 (control)a Absolute difference (post): I1 vs control +0.42 I2 vs control +1.24 I1 vs I2 ‐ 0.82 Relative percentage difference: I1 vs control +17.0% I2 vs control +50.2% I1 vs I2 ‐22.1% (relative to I2) Absolute difference from baseline: ‐0.33 (I1) vs +0.03 (I2) vs ‐0.59 (control) Difference in absolute change from baseline: I1 vs control +0.26 I2 vs control +0.62 I1 vs I2 ‐0.36 Mean annual number of non primary care visits Pre intervention: 0.62 (I1) vs 0.67 (I2) vs 0.61 (control) Post intervention: 0.57 (I1) vs 0.85 (I2) vs 0.80 (control) Absolute difference (post): I1 vs control ‐0.23 I2 vs control +0.05 I1 vs I2 ‐0.28 Relative percentage difference: I1 vs control ‐28.8% I2 vs control +6.25% I1 vs I2 ‐32.9% (relative to I2) Absolute difference from baseline: ‐0.042 (I1) vs ‐0.070 (I2) vs ‐0.031 (control) Difference in absolute change from baseline: I1 vs control ‐0.011 I2 vs control ‐0.039 I1 vs I2 +0.028	Not assessed	PROFESSIONAL FINANCIAL INCENTIVES Unit of analysis error
Emslie 1993	Post intervention measurement only ‐ during 9 months following intervention	Guidelines plus structured management/ referral sheet	Sexual history Knowledge of fertile period: Post intervention: 85% (Study) vs 73% (Control) (p = *)a Absolute difference (post): +12% (study better) Relative % difference (post): +15.7% Use of fertile period: Post intervention: 85% (Study) vs 69% (Control) (p = *)a Absolute difference (post): +16% (study better) Relative % difference (post): +23.2% Erectile problems: Post intervention: 86% (Study) vs 70% (Control) (p = *)a Absolute difference (post): +16% (study better) Relative % difference (post): +22.9% Ejaculatory problems: Post intervention: 86% (Study) vs 70% (Control) (p = *)a Absolute difference (post): +16% (study better) Relative % difference (post): +22.9% Dyspareunia: Post intervention: 86% (Study) vs 80% (Control) (p = *)a Absolute difference (post): +6% (study better) Relative % difference (post): +7.5% Pre referral examination and investigations ‐ female partner General examination: Post intervention: 68% (Study) vs 52% (Control) (p = *)a Absolute difference (post): +16% (study better) Relative % difference (post): 30.8% Pelvic examination: Post intervention: 67% (Study) vs 51% (Control) (p = *)a Absolute difference (post): +16% (study better) Relative % difference (post): +31.4% Full blood count: Post intervention: 37% (Study) vs 13% (Control) (p = *)a Absolute difference (post): +24% (study better) Relative % difference (post): +184.6% Progesterone: Post intervention: 72% (Study) vs 41% (Control) (p = *)a Absolute difference (post): +31% (study better) Relative % difference (post): +75.6% Rubella status: Post intervention: 64% (Study) vs 25% (Control) (p = *)a Absolute difference (post): +39% (study better) Relative % difference (post): +156% Pre referral examination and investigations ‐ male partner Seen by general practitioner: Post intervention: 50% (Study) vs 33% (Control) (p = *)a Absolute difference (post): +17% (study better) Relative % difference (post): +51.5% Genital examination: Post intervention: 39% (Study) vs 13% (Control) (p = *)a Absolute difference (post): +26% (study better) Relative % difference (post): +200% Semen analysis: Post intervention: 51% (Study) vs 41% (Control) (p = *)a Absolute difference (post): +10% (study better) Relative % difference (post): +24.4%	Not assessed	EDUCATIONAL INTERVENTION Unit of analysis error
Grimshaw 1998	4 months pre and 4 months post intervention	(1) Dissemination of guidelines. (2) GP‐consultant small group discussions (3) Feedback of referral rates (4) Discussion with independent medical adviser	Total number of referrals No significant changes detected with any intervention. Number of tracer referrals No significant changes detected with interventions 1, 3, 4. Intervention 2 ‐ mean number of referrals (b) Pre intervention: 1.67 (Study) vs 1.36 (Control) Post intervention: 1.62 (Study) vs 1.04 (Control) (p = 0.04)b Absolute difference (post): +0.58 referrals (study worse) Relative difference (post): +55.7% Absolute change from baseline: ‐ 0.05 (Study) vs ‐0.32 (Control) Difference in absolute change from baseline: ‐0.27 (study worse) Appropriateness of referral No significant changes detected with any intervention. Use of specialised hospital investigations No significant changes detected with any intervention. Use of specialised hospital treatments No significant changes detected with any intervention.	Not assessed	EDUCATIONAL INTERVENTION (a) Due to design it is not possible to present data simply. (b) Data analysed using general linear modelling.
Jones 1993	6 months pre and 6 months post intervention.	Consensus guidelines vs no intervention.	Medical and surgical referrals for upper gastrointestinal problems per GP: Pre intervention: 4.5 (Study) vs 3.92 (Control) Post intervention: 3.95 (Study) vs 2.85 (Control) (p = *)a Absolute difference (post): +1 referralb Relative % difference (post): ‐35.1% Absolute change from baseline: ‐0.55 (Study) vs ‐ 1.07 (Control) Difference in absolute change from baseline: + 0.52 Referrals for endoscopy per GP: Pre intervention: 1.54 (Study) vs 1.32 (Control) Post intervention: 1.86 (Study) vs 1.42 (Control) (p = *)a Absolute difference (post): +0.44 referral (study better) Relative % difference (post): +31.0% Absolute change from baseline: +0.32 (Study) vs +0.1 (Control) Difference in absolute change from baseline: +0.22 Referrals for upper gastrointestinal radiology investigations: Pre intervention: 3.59 (Study) vs 3.07 (Control) Post intervention: 2.99 (Study) vs 2.71 (Control) (p = *)a Absolute difference (post): +0.21 referrals (study worse) Relative % difference (post): +7.7% Absolute change from baseline: ‐0.6 (Study) vs ‐0.36 (Control) Difference in absolute change from baseline: ‐0.24 referrals (study better) Mean prescription costs of upper gastrointestinal drugs per GP (£): Pre intervention: £2634 (Study) vs £2254 (Control) Post intervention: £3215 (Study) vs £2545 (Control) (p<0.01)c Absolute difference (post): +£670b Relative % difference (post): +26.4% Absolute change from baseline: +£581 (Study) vs +£291 (Control) Difference in absolute change from baseline: +£290 Mean prescription costs of ulcer healing drugs per GP (£): Pre intervention: £1614 (Study) vs £1504 (Control) Post intervention: £2124 (Study) vs £1703 (Control) (p<0.01)c Absolute difference (post): +£421b Relative % difference (post): +24.7% Absolute change from baseline: +£510 (Study) vs +£199 (Control) Difference in absolute change from baseline: +£311	Not assessed	EDUCATIONAL INTERVENTION (a) Unit of analysis error (p‐values not reported) (b) Hypothesised direction unclear (c) Within group analyses reported in paper; post intervention across group analysis recalculated from summary statistics using Arcus Biostat.
Kammerling 1996	1 year pre and 1 year (T1) and 2 years (T2) post intervention	Fundholding vs control	Orthopaedic referral rates per 1000 population per year T1 Pre intervention: 7.96 (study) vs 8.23 (control)a Post intervention: 9.21 (study) vs 9.79 (control)b Absolute difference (post): ‐0.58 Relative percentage difference (post): ‐5.9% Absolute difference from baseline: +1.25 (study) vs +1.56 (control) Difference in absolute change from baseline: ‐0.31 T2 Pre intervention: 7.96 (study) vs 8.23 (control)a Post intervention:9.00 (study) vs 10.97 (control)b Absolute difference (post): ‐1.97 Relative percentage difference (post): ‐18.0% Absolute difference from baseline: (study) +1.04 vs +2.74 (control) Difference in absolute change from baseline: ‐1.70	Not assessed	PROFESSIONAL FINANCIAL INCENTIVES (a) Referral rates calculated from data presented using total populations covered. (b) Unit of analysis error.
Kinnersley 1999	Timescale not clear	In house referral vs control	Number of referrals 177 (study ‐ 8 practices) vs 145 (control ‐ 7 practices) Outcomes of in‐house referrals Study practices only ‐ 109 (63.0%) judged in need of immediate referral; 13 (7.5%) referred to same hospital speciality as their in‐house referral within 12 months; 17 (9.8%) referred to other hospital speciality within 12 months; 34 (19.6%) not referred within 12 months (missing data on 4 patients). Patients referred immediately were more likely to be older and have worse SF36 physical function score.	Patient satisfaction and SF36 scores: Data not presented in format allowing data abstraction. Authors report that patients referred in house were more likely to report themselves as satisfied compared to patients referred directly to hospital.	ORGANISATIONAL INTERVENTION
Linnala 2001	33 months following intervention	Control (no intervention) vs Patient incentives and Referrals list system	Referral Rate Pre Intervention G1 M = 4.4% p < 0.05 G2 M = 5.7% p < 0.05 Post Intervention G1 M = 5.5% p < 0.001 G2 M = 6.8% p < 0.001 Percentage relative change 1.1% p < 0.001 Referrals Sent to Private Services Pre Intervention G1 M = 8.8% G2 M = 5.7% Post Intervention G1 M = 5.6% p < 0.001 G2 M = 33.6% p < 0.001 Percentage relative change G1 ‐ 3.2% G2 27.9%	Not assessed	FINANCIAL INTERVENTION ORGANISATIONAL INTERVENTION
Morrison 1999	12 months post intervention		Referral rate per 1000 women aged 20 to 44 Post intervention: 3.25 (Study) vs 3.27 (Control) (p = *)a Absolute difference (post): ‐0.02 referrals (b) Relative % difference (post): ‐0.6% Pre referral management Proportion of couples receiving all appropriate investigations/advice Post intervention: 16.5% (Study) vs 6.9% (Control) (OR 1.324, 95%CI 1.001 to 1.752, p = 0.025)c Absolute difference (post): +9.6% (study better) Relative % difference (post): +139.1% Progesterone: Post intervention: 56.2% (Study) vs 48.4% (Control) (OR 1.464, 95%CI 0.927 to 2.312, p= 0.051)c Absolute difference (post): +7.8% (study better) Relative % difference (post): +16.1% Semen analysis: Post intervention: 37.3% (Study) vs 30.6% (Control) (OR 1.337, 95% CI 0.837 to 2.134, p = 0.112)c Absolute difference (post): +6.7% (study better) Relative % difference (post): +21.9% Cervical smear checked within previous 3 years: Post intervention: 85.9% (Study) vs 85.5% (Control) (OR 1.052, 95% CI 0.533 to 2.002 p = 0.438)c Absolute difference (post): +0.04% (study better) Relative % difference (post): +0.47% Advice given about folic acid: Post intervention: 57.0% (Study) vs 49.7% (Control) (OR 1.313, 95% CI 0.852 ‐ 2.023, p = 0.109)c Absolute difference (post): +7.3% (study better) Relative % difference (post): +14.7% Rubella immunity status checked: Post intervention: 44.6% (Study) vs 36.2% (Control) (OR 1.415, 95% CI 0.930 to 2.153, p = 0.052)c Absolute difference (post): +8.4% (study better) Relative % difference (post): +23.2% Post referral general practice management No significant difference in the number of consultations about infertility in the 12 months following referral (b) Hospital management Time from first appointment to management plan: Post intervention: 3.34 months (study) vs 2.98 months (control)(p = 0.24)d Absolute difference (post) = ‐0.36 months Relative % difference (post) = ‐12.1% Mean number of appointments before management plan reached: Post intervention: 1.92 (study) vs 1.91 (control) (p=0.84)d Absolute difference (post) = ‐0.01 appointment Relative % difference (post) = ‐0.005% Proportion of couples with management plan at one year Post intervention: 50.9% (study) vs 44.3% (control) (Odds ratio 1.24, 95% CI 0.87 to 1.77, p‐value = 0.24) Absolute difference (post): +6.6% Relative % difference (post): +14.9% Economic evaluation GP costs: Post intervention: £23 (study) vs £15 (control) (p = *) a, d Absolute difference (post): +£8 (study more expensive) Relative % difference (post): +53.3% Hospital costs Post intervention: £214 (study) vs £196 (control) (p = *)a,d Absolute difference (post): +£18 (study more expensive) Relative % difference (post): +9.2% Total NHS costs Post intervention: £251 (study) vs £215 (control) (p = *)a,d Absolute difference (post): +£36 Relative % difference (post): +16.7%	Pregnancy rates with 12 months of referral Post intervention: 32.1% (Study) vs 40.3% (control) (p=*)a,c Absolute difference (post): ‐8.2% Relative percentage difference: ‐25.5%	EDUCATIONAL INTERVENTION (a) Unit of analysis error. (b) Hypothesised direction unclear (c) Derived from multilevel model after correction for deprivation and referral hospital. (d) Further information sought from author
O'Cathain 1995	12 months pre (4/91 ‐ 3/92) and 12 month post (4/92 ‐ 3/93)a 9 month pre and 9 month post intervention (orthopaedic and rheumatology referrals)	General practice based physiotherapy service vs control	Physiotherapy contact rate (per 1000 practice population)b Pre intervention: 11.7 (study) vs 6.6 (control) Post intervention: 21.0 (study) vs 7.4 (control) Absolute difference (post): + 13.6 contacts per 1000 practice population Relative percentage difference: +183.8% Absolute difference from baseline: +9.3 contacts per 1000 practice population (study) vs + 0.8 (control) Difference in absolute change from baseline: +8.5 contacts per 1000 practice population Orthopaedic referral rate (per 1000 practice population)c Pre intervention: 10.6 (study) vs 12.1 (control) Post intervention: 8.8 (study) vs 11.0 (control) Absolute difference (post): ‐2.2 referrals per 1000 practice population Relative percentage difference: ‐20% Absolute difference from baseline: ‐1.8 referrals per 1000 practice population (study) vs ‐1.1 (control) Difference in absolute change from baseline: ‐0.7 referrals per 1000 practice population Rheumatology referral rate (per 1000 practice population)c Pre intervention: 2.3 (study) vs 2.0 (control) Post intervention: 1.1 (study) vs 1.3 (control) Absolute difference (post): ‐0.2 referrals per 1000 practice population Relative percentage difference: ‐15.4% Absolute difference from baseline: ‐1.2 referrals per 1000 practice population (study) vs ‐0.7 (control) Difference in absolute change from baseline: ‐0.5 referrals per 1000 practice population	Not assessed.	ORGANISATIONAL INTERVENTION (a) Intervention commenced 11/92 ie post intervention period only included 6 months of the intervention period. (b) Statistical testing not reported. (c) Within group analysis reported with probable unit of analysis error.
Thomas 2000	5 months pre and 10 months post intervention	Guideline based fast track open access investigation service vs control	Proportion of patients referred through fast track system Post intervention 48.2% of eligible patients referred through new system. Number and case mix of referrals No significant differences in referral rates or case mix (based on SF36 and condition specific outcome measures) of patients referred with tracer conditions. a (study equivalent) Compliance with referral guidelines (score out of 5) Pre intervention: 2.7 (study) vs 2.8 (control)a Post intervention: 3.4 (study) vs 3.0 (control) (Effect size 0.5; 95% CI 0.21 to 0.81)b (study better) General practitioner pre and post referral workload No significant differences in number of consultations prior to and 12 months following referral. a, c Waiting time from referral until first appointment (days) Pre intervention: 98 (study) vs 103 (control)a Post intervention: 39 (study) vs 59 (control) (Effect size 0.7; 95% CI 0.55 to 0.89) d, e (study better) Probability of management decision reached after one hospital appointment Pre intervention: 0.55 (study) vs 0.57 (control)a Post intervention: 0.73 (study) vs 0.56 (control) (Effect size 5.8; 95% CI 2.9 to 11.5)f (study better) Probability of discharge from consultant at 12 months Effect size 1.7; 95% CI 0.92 to 3.27) a, f (study better) Waiting time for all urology referrals Pre intervention: 24.3 weeks (g) Post intervention: 13.3 weeks Difference = 11 weeks (95% CI 7.1 to 15 weeks) Economic evaluation Total cost of intervention estimated to be £28,665 (annual cost £9,995 assuming 3 yearly guideline update cycle). Prostatism Post intervention mean general practice pre‐referral costs per patient £78.87 (study) vs £72.89 (control) Difference = +£5.98 (95% CI ‐£11.20 to +£21.85) (control cheaper) Post intervention mean general practice post‐referral costs per patient £126.35 (study) vs £178.13 (control) Difference = ‐£51.78 (95% CI ‐£160.52 to +£37.66) (study cheaper) Post intervention mean hospital costs per patient £158.88 (study) vs £239.24 (control) Difference = ‐£80.26 (95% CI ‐£150.00 to ‐£2.34) (study cheaper) Post intervention mean patient travel costs per patient £21.54 (study) vs £25.11(control) Difference = ‐£3.57 (95% CI ‐£12.60 to +£5.17) (study cheaper) Microscopic haematuria Post intervention mean general practice pre‐referral costs per patient £80.46 (study) vs £77.03 (control) Difference = +£3.43 (95% CI ‐£6.63 to +£13.29) (control cheaper) Post intervention mean general practice post‐referral costs per patient £21.85 (study) vs £18.70 (control) Difference = +£3.15 (95% CI ‐£10.05 to +£16.95) (control cheaper) Post intervention mean hospital costs per patient £222.94 (study) vs £267.73 (control) Difference = ‐£44.79 (95% CI ‐£70.14 to ‐£16.76) (study cheaper) Post intervention mean patient travel costs per patient £23.09 (study) vs £25.93 (control) Difference = ‐£2.84 (95% CI ‐£12.60 to +£5.17) (study cheaper)	Outcomes at 12 months No significant differences in SF36 MCS and PCS scores, HADS anxiety scale, AUA score (BPH only).	EDUCATIONAL INTERVENTION (a) Data analysed using multilevel model after correcting for pre intervention data and clustering of patients within practices. Reported data derived from multilevel models rather than crude data. (b) Effect size= difference in means (c) Hypothesised direction unclear (d) Effect size = ratio of means. (e) See text about pre post reduction in control group. (f) Effect size= odds ratio. OR>1 indicates benefit from intervention (g) Waiting times adjusted for number of available new appointments and clinic sessions and pre and post intervention differences tested using an unpaired t‐test. Data also presented as a time series graph suggesting effect largely due to intervention.
Vierhout 1995	12 months following recruitment	Joint consultation session vs control	General practitioner use of diagnostic tests a, b Laboratory tests: Post intervention: 1.4% (Study) vs 4.7% (Control) (p = NS) Absolute difference (post): ‐3.3% Relative % difference (post): ‐70.2% Radiography: Post intervention: 24.3% (Study) vs 22.7% (Control) (p = NS) Absolute difference (post): +1.6% Relative % difference (post): +7.0% General practitioner use of therapeutic measures a, b Medication: Post intervention: 22.2% (Study) vs 22.7% (Control) (p = NS) Absolute difference (post): ‐0.5% Relative % difference (post): ‐2.2% Injection therapy: Post intervention: 30.6% (Study) vs 11.7% (Control) (Chi square p < 0.001) Absolute difference (post): +18.9% Relative % difference (post): +161.5% Physiotherapy referral: Post intervention: 43.1% (Study) vs 42.2% (Control) (P = NS) Absolute difference (post): + 0.9% Relative % difference (post): +2.1% General practitioner referrals to orthopaedic surgeon a, b Post intervention: 35.4% (Study) vs 68.0% (Control) (P<0.001 Chi square test) Absolute difference (post): ‐32.6% Relative % difference (post): ‐47.9%	'No significant differences in between study and control groups in subjective or objective patients' variables including functional status.'b Disorder free after one year: 35.4% (study) vs 23.7% (control) (p<0.05).b	EDUCATIONAL INTERVENTION (a) Analyses recalculated using Arcus Biostat. (b) Explicit hypotheses not stated (c) Authors' analysis, raw data not presented.

Professional educational interventions 
 Two studies evaluated passive dissemination of locally developed consensus referral guidelines and neither observed changes in quantity or quality of referrals (Jones 1993; Grimshaw 1998). Jones and colleagues (Jones 1993) evaluated the development and dissemination of management guidelines for dyspepsia. They hypothesised that the guidelines would lead to an increase in the use of endoscopy compared with barium meals, fewer investigations of younger patients and changes in prescribing patterns of H2‐antagonists. They observed an increase in the number of referrals for upper gastrointestinal problems, referrals for endoscopy and upper gastrointestinal radiology investigations. Unfortunately, there was baseline imbalance for all of these outcomes which could undermine the post intervention differences. If differences in absolute change from baseline is considered, the direction of the observed effects support the authors' hypotheses. Unfortunately, because these outcomes also had unit of analysis errors which could not be re‐analysed, the statistical significance of these findings is uncertain. There were statistically significant increases in prescribing costs for upper gastrointestinal drugs and ulcer healing drugs. The authors concluded that the guidelines 'acceptance and adoption was variable and their measured effects on some aspects of clinical behaviour were relatively weak and not necessarily associated with either decreased costs or improved quality of care' (Jones 1993). Grimshaw (Grimshaw 1998) evaluated postal dissemination of referral guidelines for four tracer conditions; he observed no significant changes in referral patterns or appropriateness of referral.

Five studies evaluated dissemination of referral guidelines with structured management sheets and observed improved pre‐referral management of patients (Abu‐Ramadan 2002; Bennett 2001; Emslie 1993; Morrison 2001; Thomas 2003). Abu‐Ramadan (Abu‐Ramadan 2002) used an interrupted time series design to evaluate appropriate referrals for treatment of eye conditions. Prior to the implementation of the intervention, it was reported that general practitioners were not dealing with eye injuries and referring them too frequently. This resulted in increased workload for the opthamologists at the hospital, which in turn led to time limited consultations and increased expenses. Training programs were provided to 40 primary care physicians to educate them about screening, emergency issues and appropriate referrals. The trained physicians were then distributed to various sites to facilitate the teaching of the other practitioners. A new referral reply sheet was used, eye medications were made available and a monitoring, evaluating and advice system set up. The study reports an over 50% reduction in the number of referrals to the eye hospital post intervention. However, it is very difficult to interpret which aspect of the intervention contributed most to the change in the number of referrals.

Bennett (Bennett 2001) evaluated the use of a training video, checklist or both against a control group for referrals to specialists for otitis media with effusion (OME). There has been a wide range of general practice variation in referral rates for this condition and there are concerns about both over referral and under referral. There was no effect of any of the interventions on the referral rates. However, the authors analyzed data from 68% of the practices to determine 'quality of referrals'. They examined the percentage of children who at the time they were seen by a specialist, had a hearing loss greater than 20 dB in the better ear (an accepted indication for referral). In the group of physicians who saw the video and had access to the checklist, the percentage of appropriate referrals increased following the intervention; in all other groups including the control group it decreased.

Emslie and colleagues (Emslie 1993) evaluated dissemination of guidelines for general practitioner management and referral of infertile couples in the Grampian region of Scotland. The guidelines were disseminated with an infertility management package which included a structured referral sheet. They observed improvements in: eliciting five items of sexual history (median improvement in post intervention absolute difference +16.0%); undertaking five pre referral examinations and investigations in the female partner (median improvement in post intervention absolute difference +24%); and undertaking two pre referral examinations and investigations in the male partner (median improvement in post intervention absolute difference +18%). There was also an increase in the number of referrals in which the male partner had been seen prior to referral (improvement in post intervention absolute difference +17%). Unfortunately there was a unit of analysis error and the statistical significance of these findings is unclear. Furthermore, data from study and control groups were collected by different methods; data about referrals made by study general practitioners including the structured management sheet with the referral letter were abstracted from the referral document, whereas data from study general practitioners not using the structured referral document and control general practitioners were collected by computer assisted telephone interview.

Morrison and colleagues (Morrison 2001) evaluated a similar intervention in the Greater Glasgow Health Board area. They observed no difference in referral rates per 1000 registered women aged 20 to 44. There was little evidence of inappropriate referral; only 1.1% of patients were referred after less than 12 months of infertility without an indication for early referral. There were improvements in five items of pre‐referral investigations/advice, with a median improvement in post intervention absolute difference of + 7.3%, but none of the improvements in individual tests were statistically significant. However there was a statistically significant improvement in the proportion of couples receiving all appropriate investigations/advice (improvement in post intervention absolute difference +9.6%, Odds ratio 1.324; 95%CI 1.001 to 1.752, p=0.025). Following referral, hospitals commonly repeated investigations (100% of patients with a normal midluteal progesterone and 34.5% of patients with a normal semen analysis as reported in the referral letter received repeat investigations in hospital). There were no differences in the time from first appointment to establishing a management plan or in the proportion of couples with a management plan after one year of referral. There were 8% fewer pregnancies within 12 months in the study group. General practice and hospital costs were greater in the study group (unfortunately there was a unit of analysis error and the statistical significance of these findings is unclear).

Thomas and colleagues (Thomas 2003) evaluated the effect of a guideline based open access investigation service for two common urological conditions (prostatism and microscopic haematuria). Participating general practitioners were offered a two‐hour educational meeting and were mailed a guideline package (including a guideline booklet, quick reference flowchart and structured referral checklists). Under the existing system, patients usually attended an initial outpatient appointment and at least one further appointment for routine day case investigations. The open access investigation service allowed doctors to refer patients directly for day case investigations using the guidelines. Patients attending the open access service had all routine hospital‐based investigations and a management plan determined at this single consultation. Thomas and colleagues (Thomas 2003) hypothesised that the intervention would have little effect on general practitioners' referral patterns or patient outcomes, increase general practitioner compliance with referral guidelines, reduce patient waiting times, increase likelihood that patient would receive a management plan at first appointment and be discharged at 12 months. They observed that 48.2% of eligible patients were referred through the new system, freeing up the equivalent of 350 new outpatient slots over a 12 month period. There were no differences in referral patterns or case mix of the patients referred. Compliance with referral guidelines increased significantly. Waiting times for first appointments for the tracer conditions decreased (Ratio of means of waiting times 0.7 95% CI 0.55 to 0.89) although the post intervention difference is likely to underestimate the true effect of the intervention as there was a substantial fall in waiting times for control patients probably as a result of the freed up new outpatient appointments. There was also a reduction in waiting times for all urology patients of 11 weeks (95% CI 7.1 to 15 weeks). The probability of patients receiving a management decision at first appointment increased significantly (Odds ratio 5.8 95% CI 2.9 to 11.5). There was a non significant increase in the probability of discharge at 12 months (Odds ratio 1.7; 95% CI 0.92 to 3.27). The annual cost of the intervention was estimated to be £9555 (representing the total costs of guideline development and dissemination, however in many settings these costs would be subsumed into normal running costs). There were non‐significant reductions in post referral general practice costs (for prostatism only) and the travel costs of patients attending health services. There were significant reductions in the mean hospital management costs per patient of £80.26 for prostatism and £44.79 for haematuria.

Two of the three studies (Banait 2000; Grimshaw 1998; Vierhout 1995) evaluating secondary care provider‐led educational activities observed improvements. Vierhout and colleagues (Vierhout 1995) evaluated the effects of joint monthly consultant‐general practitioner workshops for patients with orthopaedic problems. There were no significant differences detected 12 months following recruitment in the number of patients receiving laboratory tests, radiography, medication or physiotherapy referrals. They did observe an increase in general practitioners' use of injection therapy (30.6% study versus 11.7% control p<0.001), a reduction in subsequent referral to orthopaedic surgeons (35.4% study versus 68.0% p<0.001) and an increase in proportion of patients disorder free after one year (35.4% study versus 23.7% control p<0.05).

Banait and colleagues (Banait 2000) evaluated the effects of consultant‐led, general practice‐based, small group educational workshops to disseminate guidelines for management of dyspepsia. They hypothesised that the intervention would increase appropriateness of referral for endoscopy, increase diagnostic yield of endoscopies, reduce expenditure on acid suppressing drugs and increase serological Helicobacter Pylori testing. Study practices had higher endoscopy referral rates, increased appropriateness of referral, marginally higher diagnostic yields, higher serology testing rates. However there was no reduction in prescriptions for acid suppressing drugs. Unfortunately there was a unit of analysis error and the statistical significance of these findings is unclear.

Grimshaw (Grimshaw 1998) evaluated small group general practitioner‐consultant workshops for four tracer conditions. He observed a significant increase in the number of tracer referrals following the intervention against the hypothesised direction of effect.

Grimshaw (Grimshaw 1998) also found no significant improvements in referral following feedback on referral rates or discussion with an independent adviser.

Organisational interventions 
 Bertakis and colleagues (Bertakis 1987) evaluated primary care provision for new patients in an Internal Medicine Clinic compared with a Family Practice Clinic. They observed fewer referrals and a lower annual per patient cost of laboratory tests for those patients seen in family practice. The difference in referral patterns was still noted whether the target of the referral was non‐primary care, obstetrics‐gynecology, general surgery or dermatology. There were fewer primary care attendances, fewer acute care visits and fewer emergency room visits for patients receiving primary care from family physicians. However, the authors report that the difference in emergency room visits may be due to access issues.

O'Cathain and colleagues (O'Cathain 1995) evaluated the effect of providing primary care based physiotherapy services. The aim of this service was to reduce orthopaedic referrals. They observed greater physiotherapy referrals, fewer orthopaedic referrals and fewer rheumatology referrals. Unfortunately, there was baseline imbalance for all of these outcomes which could undermine the post intervention differences. Furthermore, there was a unit of analysis error and the statistical significance of these findings is unclear.

Kinnersley and colleagues (Kinnersley 1999) evaluated the effects of an in‐house second opinion before outpatient referral. They found that approximately 70% of patients having an in‐house second opinion were judged to need referral to the same hospital discipline immediately (63.0%) or within 12 months (9.8%). Patients referred in‐house were more likely to report themselves as satisfied with their care.

Bridgman et al (Bridgman 2005) evaluated the effect of a 'slot system' designed to reduce waiting times by allocating a predetermined number of consultations to an orthopedic specialist. There were relative reductions in the monthly referral rate per 10,000 population of 22% in the intervention group and 10% in the control group. No significance levels were reported, but the authors state that multifactorial linear regression demonstrated a significant reduction in referrals from the intervention group. However, the intervention group had a 14% lower referral rate at baseline and it is unclear whether the analyses corrected for this. The imbalance raises concerns about the comparability of the intervention and control sites.

Financial interventions 
 One study evaluated the effects of changing remuneration systems. Davidson and colleagues (Davidson 1992) evaluated the effects of a change in remuneration system from a low cost fee‐for‐service system to either a high cost fee‐for‐service system or capitation‐based budgetary system. They observed a reduction in the number of non‐primary care referrals by providers receiving capitation based remuneration but little effect in providers receiving increased fee‐for‐service. 
 
 Two studies evaluated the effects of the general practice fundholding scheme within the UK (Coulter 1993; Kammerling 1996). Coulter and colleague (Coulter 1993) compared referral rates from 10 'first wave' fundholding with 6 non‐fundholding practices during the preparatory year (phase 1) and first year following the introduction of the scheme (phase 2). Referral rates during the preparatory year were used to set the budget for the first year. Referral rates were higher in fundholding practices during phase 1 but fundholding and non‐fundholding practices had similar referral rates during phase 2. In the original report, the authors state that fundholders referral rates had significantly increased, however there was a unit of analysis error and re‐analysis at practice level using T‐tests did not detect a significant difference. There was a significant increase in non‐fundholders referral rates from the pre to post intervention period (median pre intervention 95.9 annual referrals per 1000 population versus median post intervention 117.2 annual referrals per 1000 population, Mann Whitney test p<0.01). The authors also report referral rates for individual specialties. However, there was a unit of analysis error and insufficient data were presented to allow re‐analysis of the data. Surender and colleagues (Surender 1995) published a follow up study presenting data for a third time period; however, by this time, four control practices had become fundholders or shadow‐fundholders.

Kammerling and colleague (Kammerling 1996) evaluated the effects of fundholding on orthopaedic referral rates one and two years following referral in ten fundholding and twenty‐two non‐fundholding practices. There were reductions in the fundholding practices compared to controls but these were modest at best. There was a unit of analysis error and the statistical significance of these findings is unclear.

One study (Linnala 2001) evaluated the effect of providing access to private specialists or a hospital based specialist at the same cost to the patient. Normally, the cost for the private specialist was double or more the cost of the hospital based specialist. At baseline the rates of referral to a specialist were higher in the experimental group (5.7% age sex adjusted) than in the control group (4.4% age sex adjusted), increasing to 6.8% on the experimental group and 5.5% in the control group. The percentage of referrals sent to private specialist by the GPs in the control group increased from 5.7% to 33.6% in the experimental group but decreased slightly (8.8% to 5.6%) in the control group suggesting that there was a change in the destination rather than number of referrals.

Discussion

Despite the important role referral systems play in many health care systems, surprisingly few interventions have been rigorously evaluated. The majority of studies were conducted in the UK and the generalisability of these findings to other settings especially countries without a formal referral system is uncertain. As a result there is a limited evidence base to support policy decisions. Nevertheless it is possible to draw a number of preliminary conclusions. Passive dissemination of referral guidelines appears unlikely to lead to improvements in referral behaviour. This has implications for local or national referral guidelines; local dissemination and implementation activities appear necessary. The likely success of such local dissemination and implementation strategies appears to be increased if local secondary care providers are involved in educational activities.

Several studies observed improvements in the quality of referral when referral guidelines were disseminated with structured referral sheets which could be included in the referral letter. Structured referral sheets are checklists to be completed at the time of referral prompting the primary care physician about important elements of pre‐referral investigation and management. In all three studies, the use of these structured referral sheets led to improved pre‐referral investigation of patients ensuring that all appropriate examinations and investigations had been completed prior to referral. Whilst this is a potentially attractive intervention, general practices were only asked to use structured referral sheets for single conditions in each study and at best only about half of patients were referred with a completed sheet. There is a potential danger of overload if general practitioners are requested to use referral sheets for a wider range of conditions. In the future, this might be addressed by advances in informatics (for example, on‐line booking systems with embedded referral management sheets) but at present, these interventions should probably be used sparingly for referrals for common important conditions.

In the Thomas study (Thomas 2003), the guidelines and structured referral sheets were part of a complex intervention which included re‐organisation of the secondary care system to streamline the referral process. The results of the study suggest that this was successful; patients were seen and had a management decision more rapidly. It is likely that this non‐financial incentive was important for practitioners buy‐in to the intervention. In contrast, in the Morrison study (Morrison 2001) there was little evidence that secondary care management was influenced by the introduction of referral guidelines and that many investigations were unnecessarily repeated. The authors suggest that this might act as a disincentive to practitioners to comply with guidelines. Thus, it appears important that dissemination and implementation strategies for referral guidelines and similar interventions should consider the process of care across the primary ‐ secondary care interface ensuring that secondary care providers make appropriate changes in the content and organisation of care to optimise the efficiency of the referral system.

Relatively few organisational interventions have been evaluated. Enhancement of primary care capacity (for example, providing general practice based physiotherapy services) may be useful although the effects of on‐site mental health workers in primary care were uncertain (Bower 1999). One study evaluating the effects of in‐house second opinions prior to referral observed that approximately 30% of patients avoided subsequent referral. This is a potentially attractive intervention for primary care physicians working in group practices or multi‐practice organisations (for example, independent practitioner associations in New Zealand, primary care groups or trusts within the UK) and should be explored further. In this update, one study with a slot system intervention was included; however it is unclear if this new system has any benefit and further research is required.

Four of the studies of financial interventions observed modest reductions in referral rates, although none of the studies attempted to evaluate quality of care. At this time there is insufficient evidence to draw firm conclusions about the potential effects of financial interventions. There is a danger that financial interventions may lead to an unselective reduction in referral, both rational and non‐rational. A study allowing primary care physicians to refer patients to private specialist appeared to increase referrals to private specialists with little or no effect on overall referral rates. 
 
 The main weakness of the review is inevitably the limited number, methodological quality and limited evaluation of the identified studies. There have been relatively few evaluations of individual interventions, limiting ability to explore effect modifiers and confidence in the generalisability of the results to referrals for other conditions or in other settings. All of the studies had some methodological weaknesses, in particular, relatively few studies correctly analysed data from clustered randomised trials limiting statistical interpretation of the results. The studies demonstrate the complexity of undertaking research in this area. Interventions to improve referral may influence general practice management of non‐referred patients, referral behaviour (number and quality of referrals), secondary care management of patients, the flow of patients through the referral system, patient outcomes and satisfaction, and resource use. No individual study managed to evaluate all these aspects. There is little evidence about the relationship between referral rates and appropriateness. As a result, it is difficult to interpret studies which only reported effects of interventions on referral rates. Only two studies undertook an economic evaluation despite the important impact of referral on resource use; furthermore the likely resources required for the different interventions would vary considerably.

For the update, we revisited the studies that were included in the original review for inclusion consideration, particularly the CBA designs. The decision was made by the EPOC review group not to include CBA studies with only one control and one intervention group. This resulted in one study (Krasnik 1990) being removed for the update.

Faulkner and colleagues (Faulkner 2003) undertook a broader review of primary‐care based service innovations on the quality and patterns of referral to specialist secondary care. Some studies evaluated interventions that did not primarily aim to change outpatient referral rates or improve outpatient referrals appropriateness. Nevertheless their conclusions were broadly similar about the limitations of the current evidence base.

Authors' conclusions

Implications for practice.

Referral guidelines are more likely to be effective if local secondary care providers are involved in dissemination activities, structured referral sheets are used, secondary care management is responsive to changes in primary care behaviour as a result of the guidelines and if they reflect local circumstances and address local barriers.

There are a limited number of rigorous evaluations to base policy on. Nevertheless, passive dissemination of referral guidelines is unlikely to lead to improvements in referral practice. Referral guidelines are more likely to be effective if: local secondary care providers are involved in dissemination activities; structured referral sheets are used; secondary care management is responsive to changes in primary care behaviour as a result of the guidelines; and if they reflect local circumstances and address local barriers. There is little evidence on the effects of organisational interventions but the use of 'in‐house' second opinion and other intermediate primary care based alternatives to outpatient referral appear promising. Financial interventions can change referral rates but their effect on quality of referral is uncertain.

Implications for research.

Further research is needed to replicate the results of current evaluations (focusing on potentially effective interventions such as secondary care provider‐led educational activities, structured referral management sheets, enhancement of primary care and in‐house second opinions). Further research is also needed to explore a wide range of available interventions which do not appear to have been evaluated to date. In particular, it would be worth evaluating other intermediate primary‐care based referral systems. These evaluations should evaluate the effects of the intervention on the quantity and quality of referrals and include an economic evaluation.

What's new

Date	Event	Description
1 December 2010	Amended	Reference link corrected.

History

Protocol first published: Issue 2, 1999
 Review first published: Issue 3, 2005

Date	Event	Description
12 November 2008	Amended	Contact details corrected, format edits
12 June 2008	Amended	Converted to new review format.
15 February 2008	New search has been performed	New search, one new study, one removed
14 February 2008	New citation required but conclusions have not changed	New authors, 1 new study
13 May 2005	New citation required and conclusions have changed	Substantive amendment

Appendices

Appendix 1. Search strategy

Outpatient referrals from primary care to secondary care (Update) 
 16 October 2007 
 
 (refer* or consult*) and outpatient*

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Abu‐Ramadan 2002.

Methods	Design: ITS Completeness of data set: NOT CLEAR The intervention unlikely to affect data collection: DONE Sufficient data points to enable reliable statistical inference: DONE Formal test for trend: NOT CLEAR Blinded assessment : NOT DONE Reliable outcome measure(s): DONE The intervention is independent of other changes: DONE Type of data: CROSS SECTIONAL
Participants	PALESTINE 40 Physicians (GPs) in the Gaza Strip
Interventions	(1) Educational Meetings (2) New Referral and Reply Sheet (3) New Staff (4) Change in Equipment and Facilities (5) Quality Monitoring Mechanisms
Outcomes	Patient Load
Notes	EDUCATIONAL INTERVENTION ORGANISATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	D ‐ Not used

Banait 2000.

Methods	Design: RCT (Practice randomised 2 arm trial) Randomisation concealment: DONE (but see note) Follow up: Providers: DONE Patients: NOT CLEAR Blinded assessment: DONE/NOT CLEAR Baseline: DONE/NOT CLEAR Reliable outcomes: DONE/NOT CLEAR Protection against contamination: DONE
Participants	UK General practitioners in 114 general practices in Salford and Trafford Health Authority. Proportion of eligible providers who participated: 99.1% (114/115 practices) 33/57 (58%) of intervention practices received the intervention. Clinical area of interest: dyspepsia
Interventions	(1) Consultant led educational seminars plus paper copies of guidelines. Reinforcement practice visit at 3 months. (2) Paper copies of guidelines
Outcomes	Process: Appropriateness of referral for upper gastrointestinal endoscopy Findings at endoscopy Prescribing costs for acid suppressing drugs Requests for laboratory tests for Helicobacter Pylori
Notes	EDUCATIONAL INTERVENTION Allocation used minimisation based on practice size, fundholding status, previous expenditure on NSAIDs and previous involvement in a guideline initiative.
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Low risk	A ‐ Adequate

Bennett 2001.

Methods	Design: RCT (cluster randomized by practice) Protection against contamination: DONE Blinded assessment: DONE Reliable outcome measure(s): DONE Baseline measurement: DONE Follow‐up: NOT DONE
Participants	UK 177 Physicians (GPs) in 50 practices
Interventions	(1) Risk Factor Checklist (2) OME Training Video (3) Combination of Checklist and Video
Outcomes	Appropriateness of Referrals
Notes	EDUCATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	D ‐ Not used

Bertakis 1987.

Methods	Design: RCT (Patient randomised two arm trial) Randomisation concealment: NOT CLEAR Follow up: Providers: NOT CLEAR Patients: DONE Blinded assessment: NOT CLEAR Baseline: NOT CLEAR Reliable outcomes: NOT CLEAR Protection against contamination: DONE
Participants	US 520 patients attending US Medical Center Clinical area of interest: general management of problem
Interventions	(1) Primary care management in family practice clinic (2) Primary care management in internal medicine clinic
Outcomes	Process: Primary care consultations Emergency room and acute care clinic attendance's Specialist clinic attendances Costs of laboratory tests
Notes	ORGANISATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	B ‐ Unclear

Bridgman 2005.

Methods	Design: CBA Baseline measurements: NOT DONE Characteristics of second site: NOT DONE Follow up: Providers: NOT CLEAR Patients: N/A Blinded assessment: DONE Reliable outcome measures: DONE Protection against contamination: DONE
Participants	UK 36 practices; 12 intervention, 24 control
Interventions	Slot system, providing a limited number of spots for new orthopedic referrals
Outcomes	Process: Rate of new referrals to orthopedics
Notes	ORGANISATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	D ‐ Not used

Coulter 1993.

Methods	Design: CBA Baseline measurements: NOT DONE Characteristics of second site: NOT CLEAR Follow up: Providers: NOT CLEAR Patients: N/A Blinded assessment: NOT DONEa Reliable outcome measures: NOT CLEAR (see note) Protection against contamination: DONE
Participants	UK Ten fundholding (study) and six non‐fundholding (control) practices in Oxford region. During latter part of study four control practices were in preparation for fund holding. Proportion of participating practices: 40% (10/25) fundholding practices.
Interventions	(1) Fundholding scheme. (2) No intervention (during latter stage of the study, four control practices were in preparation for fundholding).
Outcomes	Process: Number of outpatient referrals that would incur a charge against fundholders' budgets.
Notes	PROFESSIONAL FINANCIAL INTERVENTION Data were collected by general practitioner or practice staff.
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	D ‐ Not used

Davidson 1992.

Methods	Design: RCT (2 arm physician randomised trial) Randomisation concealment NOT CLEAR Follow up Provider NOT CLEAR Patients NOT CLEAR Blinded assessment NOT CLEAR Baseline NOT CLEAR Reliable outcomes NOT CLEAR Protection against contamination NOT CLEAR Method of randomisation not clear Unit of analysis error
Participants	US 80 physicians in private office based practices who treated Medicaid children and more than $2000 in Medicaid billings in previous year. Medicaid eligible children receiving welfare benefit under Aid for Families with Dependent Children Program Proportion of eligible practitioners who participated: 57.1% (80/140).
Interventions	(1) Capitation (I1) (2) Fee for service (high rate) (I2) (3) Control ‐ fee for service low rates (see note)
Outcomes	Process: Mean number of primary care visits Mean number of non primary care visits Mean number of clinic/emergency department visits Mean number of hospitalisations
Notes	PROFESSIONAL FINANCIAL INTERVENTION Comparison group drawn from community based sample of patients who were recertified for Aid for Families programme .in the month before or after study recruitment.
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	B ‐ Unclear

Emslie 1993.

Methods	Design: RCT (Practice randomised 2 arm trial) Randomisation concealment: NOT CLEAR Follow up: Providers: NOT CLEAR Patients: DONE Blinded assessment: NOT DONE Baseline: NOT CLEAR Reliable outcomes: NOT CLEAR Protection against contamination: DONE
Participants	UK General practitioners in 82 general practices in Grampian region of Scotland. Proportion of eligible practices who participated: 95.3% (82/86 practices participated but nine individual general practitioners from participating practices declined to take part in study). Clinical area of interest: infertility
Interventions	(1) Locally developed guidelines plus structured record sheet (incorporating reminders) for general practice management and referral of infertile couples disseminated by mail. (2) No intervention (control GPs were informed that they would receive the guidelines at the end of the study).
Outcomes	Process: Compliance with guidelines in particular whether adequate sexual history was taken, whether couple were appropriately examined and investigated (see note).
Notes	EDUCATIONAL INTERVENTION Differences in data collection methods between study and control GPs.
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	B ‐ Unclear

Grimshaw 1998.

Methods	Design: CCT (Practice allocated 4x4 Latin square with embedded factorial design) Randomisation concealment: NOT CLEAR Follow up: Providers: DONE Patients: DONE Blinded assessment: NOT DONE Baseline: DONE Reliable outcomes: NOT CLEAR Protection against contamination: DONE
Participants	UK 116 general practitioners from the Grampian region. Proportion of eligible providers who participated: 38.8% (116/299). Clinical areas of interest: low back pain, menorrhagia, suspected peptic ulcer, varicose veins
Interventions	(1) Locally developed referral guidelines disseminated by mail. (2) Consultant‐general practitioner workshops (3) Feedback on referral rates (4) Discussion about referrals with an independent adviser Interactions between 1 + 2 and 3 + 4 also tested for. (5) No intervention
Outcomes	Process: Total number of referrals Number of referrals for tracer conditions Appropriateness of referral Use of specialised hospital investigations Use of specialised hospital treatments
Notes	EDUCATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	B ‐ Unclear

Jones 1993.

Methods	Design: RCT (Practice randomised 2 arm trial) Randomisation concealment: DONE Follow up: Providers: DONE Patients: N/A Blinded assessment: NOT CLEAR Baseline: NOT CLEAR Reliable outcomes: DONE/NOT CLEAR Protection against contamination: DONE
Participants	UK 179 general practitioners from 45 general practices. Proportion of eligible providers who participated: 70% (179/254). Clinical area of interest: dyspepsia
Interventions	(1) Local consensus meetings between general practitioners, surgeons, physicians and radiologists to agree guidelines for management of dyspepsia including choice and timing of investigations, when to refer for specialist advice and management. Guidelines distributed to study general practices. (2) No intervention.
Outcomes	Process: Medical and surgical referrals for upper gastrointestinal symptoms. Referrals for endoscopy. Referrals for upper gastrointestinal radiology investigations. Prescriptions of upper gastrointestinal drugs. Prescriptions of ulcer healing drugs.
Notes	EDUCATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Low risk	A ‐ Adequate

Kammerling 1996.

Methods	Design: CBA Baseline measurements: DONE Characteristics of second site: DONE Follow up: Providers: NOT CLEAR Patients: N/A Blinded assessment: DONE Reliable outcome measures: DONE Protection against contamination: DONE
Participants	UK 10 fundholding (study) and 22 non fundholding (control) practices Proportion of participating practices: not clear
Interventions	(1) Fundholding scheme. (2) No intervention
Outcomes	Process: Referral rates for orthopaedic problems.
Notes	PROFESSIONAL FINANCIAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	D ‐ Not used

Kinnersley 1999.

Methods	Design: RCT (Practice randomised 2 arm RCT) Randomisation concealment: NOT CLEAR Follow up: Providers: NOT CLEAR Patients: NOT DONE Blinded assessment: NOT CLEAR Baseline: NOT DONE Reliable outcomes: DONE Protection against contamination: DONE
Participants	UK General practitioners from 15 practices in East Cardiff and Gwent (see note a). Proportion of eligible practices who participated: Not clear Clinical area of interest: outpatient referrals for dermatological, gynaecological, opthalmological, ENT and musculo‐skeletal problems
Interventions	(1) Patients referred for an 'in house' second opinion prior to referral (2) No intervention
Outcomes	Process: Number of referrals to hospital (Study group only) Number of referrals immediately and within 6 months of in‐house referral Outcome: Medical Interview Satisfaction Scale SF‐36 health status measure Both measured at baseline, 6 months and 12 months (see note b).
Notes	ORGANISATIONAL INTERVENTION a. 16 practices were recruited but 1 withdrew following allocation to control arm. b. Baseline outcome questionnaires were administered after in house referral (study group) and after consultation when referral was made (control group).
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	B ‐ Unclear

Linnala 2001.

Methods	Design: CBA Baseline measurement: DONE Blinded assessment : DONE Reliable outcome measure(s): DONE Follow‐up: NOT CLEAR
Participants	FINLAND 14 Physicians (GPs) in 4 municipal health centers
Interventions	(1) Patient Incentives (2) Provided GPs with list system for referrals
Outcomes	Rate of Referrals to Public vs Private Sectors Referrals: Consult vs Care
Notes	FINANCIAL INTERVENTION ORGANISATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	D ‐ Not used

Morrison 2001.

Methods	Design: RCT (practice randomised) Blinded assessment : DONE Reliable outcome measure(s): NOT CLEAR Baseline measurement: NOT DONE Follow‐up: DONE
Participants	UK 598 Physicians (GPs) in 214 practices in Glasgow
Interventions	(1) Distribution of Locally Developed Guidelines (2) Structured record sheet checklist for GPs incorporating reminders (3) Educational Meetings
Outcomes	Rate of Referrals Appropriate pre‐referral investigations Hospital investigations after referral Time to management decision Cost of referrals to NHS
Notes	EDUCATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Low risk	A ‐ Adequate

O'Cathain 1995.

Methods	Design: CBA Baseline measurements: NOT DONE/NOT CLEAR Characteristics of second site: NOT CLEAR Follow up: Providers: NOT CLEAR Patients: NOT CLEAR Blinded assessment: DONE Reliable outcome measures: DONE Protection against contamination: DONE
Participants	UK 41 non fund‐holding general practices in Doncaster. Proportion of eligible practices who participated: Not clear Clinical area of interest: musculoskeletal conditions
Interventions	(1) Access to a primary care based physiotherapy service (2) No intervention (GPs had access to a hospital based physiotherapy service)
Outcomes	Process: Physiotherapy contact rates Orthopaedic referral rate Rheumatology referral rate
Notes	ORGANISATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	D ‐ Not used

Thomas 2003.

Methods	Design: RCT (Practice randomised 2 x 2 balanced incomplete block design) Randomisation concealment: DONE Follow up: Providers: DONE (see note) Patients: DONE Blinded assessment: NOT CLEAR Baseline: DONE Reliable outcomes: NOT CLEAR Protection against contamination: DONE
Participants	UK General practitioners in 76 general practices in the Grampian Region.A Proportion of eligible practices who participated: 84.4% Clinical area of interest: prostatism, microscopic haematuria
Interventions	(1) Locally developed referral guidelines, disseminated by educational meetings. General practitioners could refer patients to a fast track day case investigation service if they used a structured referral letter based on guidelines. (2) No intervention
Outcomes	Process: Proportion of patients referred through fast track system Number of referrals Casemix of referrals Compliance with referral guidelines General practitioner pre and post referral workload Waiting time from referral until first appointment Management decision reached after one hospital appointment Completed care within 12 months Waiting time for all urology referrals Outcomes: SF36 and condition specific measures at baseline and 12 months Economic evaluation
Notes	EDUCATIONAL INTERVENTION Data from 10 practices were excluded from analysis due to incomplete data capture that could lead to an overestimate of the effects of the intervention.
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Low risk	A ‐ Adequate

Vierhout 1995.

Methods	Design: RCT (Patient randomised 2 arm trial ‐ Zelen design) Randomisation concealment: NOT CLEAR Follow up: Providers: N/A Patients: DONE Blinded assessment: NOT CLEAR Baseline: NOT DONE Reliable outcomes: NOT CLEAR Protection against contamination: NOT DONE
Participants	Netherlands 272 patients with orthopaedic problems cared for by 12 Dutch general practitioners. Patient was selected if GP was uncertain about diagnosis or management and specialist referral was considered. Proportion of eligible providers who participated: not specified. Clinical area of interest: patients considered for orthopaedic referral
Interventions	(1) General practitioners and consultants operated a joint consultation session for patients with orthopaedic problems where the general practitioner was uncertain about diagnosis and management and referral was considered. (2) Routine general practitioner care.
Outcomes	Process (GP actions): Diagnostic actions. Therapeutic measures. Referrals to orthopaedic surgeon. Outcome: General health status (based on Netherlands Central Statistics Bureau Questionnaire). Level of industrial disability. Activities of daily living. Sickness impact profile.
Notes	EDUCATIONAL INTERVENTION
Risk of bias
Bias	Authors' judgement	Support for judgement
Allocation concealment?	Unclear risk	B ‐ Unclear

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Harris 2002	Contamination of control group Ceiling effect Only 11% follow‐up of professionals
Jaatinen 2002	Outcome not objectively measured: questionnaires, self‐report
Krasnik 1990	Only one intervention and one control group
Rosenheck 2000	Ineligible outcomes
Schulpen 2003	Insufficient baseline data
Wilson 2005	Ineligible outcomes

Contributions of authors

RW and JG developed the review protocol. CF developed the EPOC search strategy and additional Medline search strategies. EG undertook electronic searches. EG, JG, CF, CP and RW selected studies for inclusion in the review. EG, JG, RW, AM, CP and RT undertook data abstraction. JG drafted the paper. EG, CF, RW and RT commented on all drafts of the paper. For the update, AA and AM screened identified studies and JG was involved in the final selection. Data abstraction was done by AA, AM and MA.

Sources of support

Internal sources

• Centre for Quality of Care Research, Universities of Nijmegen and Maastricht, Netherlands.

• Diagnostic Centre of the Maastricht University Medical Centre, Netherlands.

• Ministry of Public Health, Netherlands.

• Health Services Research Unit, University of Aberdeen, UK.

• Institute of Population Health, University of Ottawa, Canada.

• The Ottawa Hospital, Canada.

• University of Ottawa, Canada.

External sources

• Department of Health Policy Research Program, UK.

Declarations of interest

JG is an author on several of the reviewed papers. RT is an author on one of the reviewed papers.

Edited (no change to conclusions)