Impact of referral letters on scheduling of hospital appointments: a randomised control trial

Abstract

Background

Communication is essential for triage, but intervention trials to improve it are scarce. Referral Writer (RW), a referral letter software program, enables documentation of clinical data and extracts relevant patient details from clinical software.

Aim

To evaluate whether specialists are more confident about scheduling appointments when they receive more information in referral letters.

Design and setting

Single-blind, parallel-groups, controlled design with a 1:1 randomisation. Australian GPs watched video vignettes virtually.

Method

GPs wrote referral letters after watching vignettes of patients with cancer symptoms. Letter content was scored against a benchmark. The proportions of referral letters triagable by a specialist with confidence, and in which the specialist was confident the patient had potentially life-limiting pathology were determined. Categorical outcomes were tested with χ² and continuous outcomes with t-tests. A random-effects logistic model assessed the influence of group randomisation (RW versus control), GP demographics, clinical specialty, and specialist referral assessor on specialist confidence in the information provided.

Results

The intervention (RW) group referred more patients and scored significantly higher on information relayed (mean difference 21.6 [95% confidence intervals {CI} = 20.1 to 23.2]). There was no difference in the proportion of letters for which specialists were confident they had sufficient information for appointment scheduling (RW 77.7% versus control 80.6%, P = 0.16). In the logistic model, limited agreement among specialists contributed substantially to the observed differences in appointment scheduling (P = 35% [95% CI 16% to 59%]).

Conclusion

In isolation, referral letter templates are unlikely to improve the scheduling of specialist appointments, even when more information is relayed.

INTRODUCTION

In some healthcare systems patients with cancer symptoms must consult a generalist, or ‘gatekeeper’, before they see a specialist.^1–4 The specialist appointment is determined by clinical urgency, as conveyed by the generalist’s referral letter.⁵ Although most patients referred from primary care are unlikely to have a life-limiting condition, such as cancer,⁶ if the generalist’s communication regarding these patients is not explicit, diagnosis and/or treatment by a specialist may be delayed.

A systematic review found no impact on patient outcomes attributable to the quality of referral letters from generalists to specialists.⁷ However, audits and qualitative studies suggest continued dissatisfaction among some specialists regarding limited information in referral letters.^8,9

A preliminary evaluation of the software program Referral Writer (RW) demonstrated that it can increase the quantity of relevant information in referral letters.¹⁰ This pilot study also suggested improvement in the specialist’s ability to prioritise cases based on the program. The study reported in this article explores this further.

METHOD

Design

The study involved a single-blind, parallel-groups, controlled design with a 1:1 randomisation of Australian GPs.

Materials

Guided by the UK’s National Institute for Health and Care Excellence (NICE) cancer referral guidelines,¹¹ 24 video vignettes were developed by six GPs, which comprised four videos for each of six specialties (Box 1). The vignettes comprised a 4-minute video monologue from an actor–patient accompanied by case notes containing the patient’s medical history, current medication, allergies, and previous consultations. The video included an off-camera commentary by an actor–doctor describing clinical signs to be found at this visit. Participation was via the internet and Qualtrics^©. The Qualtrics Research Suite is a user-friendly, feature rich, web-based survey tool which allows users to build, distribute, and analyse online surveys, collaborate in real-time, and export data in multiple formats. All GPs provided demographic data.

Box 1. Cases used in the video vignettes for GPs to make referral decisions.

Colorectal

Adrian Marshall, 65 years old, presents with history of rectal bleeding and diarrhoea for the past 6 weeks. Feeling fatigued. Grandfather died of bowel cancer 20 years ago. On examination found to have rectal mass.

Indications for referral: 6 weeks of unexplained rectal bleeding and rectal mass.

Upper gastrointestinal

Fred Jones, 66 years old. Very worried, tired, and generally unwell. Family have noticed that Fred can’t swallow solid food. Started with choking on steak, now can only eat small sips of soup. He has tried to keep the symptoms from his family, but has now lost 10 kg in 6 weeks. Son has insisted that he make an appointment with a doctor. He is a smoker. Used to have dyspepsia for years, took lots of antacids.

Indications for referral: dysphagia for solids.

Gynaecology

Freda Walton, 65 years old. Episode of heavy postmenopausal bleeding 6 weeks ago. Since then less severe bleeding and spotting every week. It started as heavy bleeding again last week. Sister advised her to make an appointment. Previously had brownish discharge sometimes, but it stopped long ago. Cervical smear test was normal last time. Hormone replacement therapy for 4 years after menopause; that was 6 years ago. Not on any medication now. No signs on clinical examination.

Indications for referral: postmenopausal bleeding.

Respiratory

Kevin Doyle, 49-year-old farmer. Had an insurance medical and chest X-ray recently. Has been asked to go and see his GP. Has been coughing a bit recently. One or two episodes of haemoptysis. Suspects he has lost some weight. Been a smoker most of his life. On examination has cervical lymphadenopathy and a hilar mass on chest X-ray.

Indications for referral: suspicious chest X-ray and haemoptysis.

Breast

Joanne Rammage, 35 years old. Had a baby 8 months ago. Discovered a 3 cm lump in her left breast 2 months ago. Lump is hard and feels different to other lumps in breast. Doesn’t change in size at menses. No pain/tenderness over lump and also no breast tenderness. No other lumps felt. No discharge/itching/ skin changes around nipple. Stopped breastfeeding 4 months ago. No discharge since. On examination skin dimpling over the lump.

Indications for referral: breast lump.

Genitourinary

Richard Cunningham, 55 years old. His brother was diagnosed with prostate cancer a year ago. Now worried because of symptoms of frequency, nocturia, hesitancy, terminal dribbling, and haematuria. Another doctor organised a prostate specific antigen (PSA) test. On examination has a large regular prostate gland. PSA is 22.

Indications for referral: high PSA and prostatic symptoms.

After viewing the video once, GPs chose to: prescribe medication; order a test; and/ or make a referral. Following their decision, GPs wrote the prescription, ordered the test, and/or made the referral.

Intervention

Initially trialled in the UK,¹² RW includes tick-box options for clinical data when making a referral to six specialties.¹³ RW extracts patient details from the practice’s database and uses algorithms to make a case for an urgent referral, as per the NICE guidelines.¹¹ These guidelines were selected because there are no equivalent Australian guidelines that specifically advise on the need for specialist referral based on defined high-risk symptom complexes. Further detail on RW can be sourced elsewhere.¹⁴

How this fits in

Very few trials have reported on interventions to improve the triage of appointments with reference to referral letters. Referral templates in isolation are unlikely to improve the clinical appropriateness of scheduling specialist appointments for patients with cancer symptoms. Agreed interdisciplinary guidelines may be required before the value of a referral template can be assessed. Agreement between specialists on the key information necessary to triage with confidence may also be required.

Sample size calculation

Typically, the intra-class correlation coefficient (ICC) in cluster-randomised studies in primary care range from 0.01 to 0.05.¹³ The unit of analysis was the individual referral, thus clustering among GPs had to be accounted for. A modest ICC of 0.03 was adopted to calculate the sample size, while adjusting for clustering. A sample size of 196 referrals yields 95% of power at the 5% level of significance to detect a 20% difference (70–90%) between the proportion of cases in which two specialists would rate themselves as ‘confident’ or ‘very confident’ to decide the timing of patient presentation. This is half the difference detected in the pilot study.¹⁰ Assuming each GP makes 10 referrals in any phase of the study, 196/10 or 20 GPs were required. For a clustered study, the number of GPs needed = 20 × design effect or 20 × (1 + [24 − 1] × 0.03) = 20 × 1.69 = 34. The cluster size is 24 as each GP will view a maximum of 24 vignettes. With a 30% attrition rate, the total number of GPs required is therefore 33.8/0.7 = 49.

Randomisation and masking

In the first of two phases, GPs who chose to refer the patient generated free-text referral letters. In phase 2, GPs were randomly assigned to one of two groups using computer-generated random numbers. The control group continued to write free-text letters, while the intervention group used RW.

Referral letters were anonymised and allocated a unique identifier. Letters were reviewed for quantity and content quality using a previously developed scoring system.^10,15 Two researchers, who were blind to the study aims, reviewed and scored the letters via consensus. Guided by three items with a 5-point Likert scale, two specialists from the six specialties, who were also blind to the study aims, triaged the referral letters.

Outcome measures

The outcome measures (OM) pertained to these hypotheses (H):

• H1: RW substantially increased the quality of information in the referral letter.

• OM: information quality score for each letter.

• H2: the specialists, working independently, were more likely to be ‘confident’ or ‘very confident’ about scheduling the patient’s specialist appointment based on the information presented in the RW letters than in the control group letters.

• OM: the proportion of letters in each group where the specialist was ‘confident’ or ‘very confident’ they had sufficient information to schedule the patient’s specialist appointment.

• H3: demographic and other participant variables, the specialists, and/or the cases influence appointment scheduling.

• OM: a random-effects logistic model to identify the factors that impact specialist confidence in using the referral letters to schedule appointments.

Statistical methods

The χ² test was used to assess the difference in proportion of referral letters triaged with confidence between the two groups at phase 2. The two-sample t-test was also used to compare the mean scores of referral letters between the two groups at phases 1 and 2. The pattern of specialist assessments on the effect of the intervention was evaluated by cross-tabulations. A random-effects logistic model was used to assess the influence of GP demographics, group (RW versus control), specialty, and specialist identity on the statement that the specialist was less than confident they had sufficient information to decide the timing of patient presentation. Specialist identity was entered into the model as a random-effect. The user-defined parsimonious models were constructed in a backward fashion from the full model, and only variables with P = 0.05 were retained in the final model. The variables of the intervention groups and clinical topics were included in the model regardless of significance. Kappa statistics were used to assess inter-rater agreement on whether there was adequate information in the referral letter and when the patient appointment should be scheduled. Cross-tabulations of Likert-scale responses revealed that in many cases, some categories had zero raters. Thus, weighted κ statistics, which adjusted the empty categories, were also reported. Stata build-in weighting matrix, ‘1−|i−j|/(κ−1)’ was used as the weights.

RESULTS

Participants

Between August 2011 and August 2012 (inclusive), 102 GPs were recruited via email and personal contact by the research team in seven Australian states and territories. Organisations approached for recruitment included Australian Divisions of General Practice (geographically defined organisations with comprehensive coverage of Australian GPs), university departments, research networks, and personal contacts. The geographical distribution of the recruits was as follows: Western Australia (n = 47); Victoria (n = 25); New South Wales (n = 13); Queensland (n = 7); South Australia (n = 7); the Australian Capital Territory (n = 2); and Tasmania (n = 1). GPs had to be currently practising, including registrars (vocational trainees), and have internet access. Of these, 87 GPs completed phase 1 and 15 withdrew for reasons that remain unknown (Figure 1). Participating GPs were instructed on how to access the videos and use RW. Researcher support was available if required. Following phase 1, 50.6% (n = 44) were randomised to the RW group and 39 completed the study. At phase 2, 44.8% (n = 39) remained in the RW group and 38 completed the study.

Figure 1.

Consort diagram.

There were no significant differences between the GPs randomised to the RW or control groups (Table 1). The participating GPs were younger, more likely to be registrars and more likely to practise in Western Australia than GPs in Australia generally.¹⁶ GPs were recompensed for their participation and could claim continuing medical education points. GP progress through the vignettes could be tracked online and reminders were sent after 2 weeks of inactivity for those who had not completed the study.

Table 1.

Participant demographic information [participants who entered the study and completed phase 1] (n = 102)

Attribute	Control group (n = 62)		Referral Writer group (n = 40)
	Mean	SD	Mean	SD
Age	44	11.8	41	11.7
Years after graduation	20	11.1	18	11.5
Years as GP	14	11.0	12	11.3
GPs at primary clinic	8	4.2	7	3.9
GP sessions worked/week	7	2.9	6	3.1
	n	%	n	%
Males	36	58.1)	22	55.0
Australian graduate	43	69.4)	30	75.0
GP registrar	14	22.6)	10	25.0
FRACGP	34	54.8)	24	60.0
GP position
Principal	13	21.0)	8	20.0
Non-principal	36	58.0)	27	67.5
Others	13	21.0)	5	12.5
GP geographical location
New South Wales	8	12.9)	5	12.5
Queensland	4	6.4)	3	7.5
Victoria	12	19.3)	13	32.5
South Australia	6	9.7)	1	2.5
Tasmania	0	0.0)	1	2.5
Western Australia	31	50.0)	16	40.0
Australian Capital Territory	1	1.6)	1	2.5
Regional location
Capital	29	46.8)	20	50.0
Other metropolitan	22	35.5)	16	40.0
Large rural	2	3.2)	3	7.5
Small rural	5	8.1)	1	2.5
Other rural	3	4.8)	0	0.0
Remote centre	1	1.6)	0	0.0
Remoteness of region
Major cities	45	72.6)	28	70.0
Inner regional	6	9.7)	9	22.5
Outer regional	8	12.9)	2	5.0
Remote	2	3.2)	1	2.5
Very remote	1	1.6)	0	0.0
Accredited practice	62	100.0)	39	97.5
Patients consulted/week
<100	24	38.7)	25	62.5
100–149	22	35.5)	8	20.0
150–199	13	21.0)	7	17.5
>199	3	4.8)	0	0.0
Direct patient care hours/week
<11	5	8.1)	6	15.0
11–20	11	17.7)	10	25.0
21–40	28	45.2)	19	47.5
41–60	15	24.2)	5	12.5
>60	3	4.8)	0	0.0
Non-English consultations
0	51	82.3)	33	82.5
<25%	10	16.1)	7	17.5
>50%	1	1.6)	0	0.0

FRACGP = Fellow Royal Australian College of General Practitioners.

Hypothesis 1

More vignettes were referred by the GPs after randomisation during phase 2 (vignettes = 880, 73.1%), relative to phase 1 (vignettes = 673, 66.0%; χ² test = 13.25, degrees of freedom [df] = 1, P<0.001) (Table 2). In phase 2, more referrals were made by the RW GPs (n = 418, 76.6%) than by the control group (n = 462, 70.2%; χ² test = 9.16, df = 1, P = 0.01). Furthermore, the quantity of clinically relevant information relayed by the RW GPs increased from a mean of 29.2 in phase 1 to 48.4, with a mean difference of 21.6 (95% confidence intervals [CI] = 20.1 to 23.2), P<0.001; Figure 2).

Table 2.

Kappa scores for agreement

Specialty	Kappa (% of agreement)		Standard weighted Kappaa (% of agreement)
	Phase 1	Phase 2	Phase 1	Phase 2
Breastb	0.12 (43.5)	0.02 (41.2)	0.12 (78.3)	0.05 (78.6)
Genitourinaryb	0.08 (67.3)	−0.02 (53.5)	0.18 (87.9)	0.01 (79.3)
Respiratoryb	−0.04 (22.0)	0.01 (38.7)	0.07 (72.4)	0.12 (82.8)
Upper gastrointestinalb	0.05 (34.9)	−0.01 (22.9)	0.08 (77.2)	0.06 (71.5)
Colorectalb	0.01 (6.0)	0.02 (35.8)	0.03 (62.3)	0.02 (62.6)
Gynaecologyb	0.03 (25.7)	0.07 (27.0)	0.04 (75.0)	0.17 (73.4)
Breastc	0.44 (71.0)	0.33 (97.9)	0.45 (85.8)	0.49 (99.0)
Genitourinaryc	−0.07 (29.1)	−0.05 (64.8)	−0.07 (64.6)	−0.02 (81.7)
Respiratoryc	−0.07 (84.6)	0.32 (78.7)	−0.07 (84.6)	0.33 (89.4)
Upper gastrointestinalc	0.21 (57.6)	0.10 (45.1)	0.21 (77.0)	0.15 (70.1)
Colorectalc	−0.01(1.2)	0.00 (20.6)	0.00 (45.8)	0.00 (57.1)
Gynaecologyc	−0.02 (19.7)	0.01 (29.1)	0.06 (72.7)	0.04 (69.8)

^aUsed Stata program prerecorded weights: 1−|i−j|/(κ−1), where i and j index the rows and columns of the rating by the two raters and κ is the maximum number of possible ratings.

^bAdequacy of information in referral letter.

^cAgreement on when the patient should be offered an appointment

Figure 2.

Quantity of clinically relevant information in the referral letters.

Hypothesis 2

There was little agreement between the specialists on the adequacy of information to schedule the patient appointment, and when to schedule the appointment (Table 2).

The proportion of referrals of which specialists were ‘confident’ or ‘very confident’ with the adequacy of the information to schedule the patient appointment was significantly higher in phase 2 (vignettes = 1329, 79.2%) than in phase 1 (vignettes = 827, 68.0%; χ² test = 46.94, df = 1, P<0.001). These differences were significant for gynaecology (P<0.001) and genitourinary referrals (P = 0.03).

In phase 2, there were no differences in the specialists’ assessment of letters written by the RW and control groups regarding how confident they were in the adequacy of information to schedule a patient appointment (RW group letters = 621, 77.7% versus control group letters = 708, 80.6%; χ² test = 2.03, df = 1, P = 0.16). However, for respiratory referrals, the specialists were more likely to be confident about the adequacy of information presented by the control group (letters = 135, 92.5%) relative to the RW group (letters = 108, 69.7%, P<0.001).

Hypothesis 3

A random-effects logistic model was generated to identify the factors that impact specialist confidence in using the referral letters to determine the timing of a patient appointment (Table 3). Specialists were not less confident in the information provided by either group for any specialty. Male GPs, GPs from inner regional areas, and GPs from larger practices were significantly more likely to provide information that reduced specialist confidence. The significant random-effect of specialists (P<0.001 in the likelihood-ratio test of ρ = 0) indicates that the panel-level variance component (specialist identity) contributed a significant proportion (35%, 95% CI = 16% to 59%) of the total variance.

Table 3.

Factors related to ‘less’ specialist confidence in the referral information to determine when patient should present (n = 1678)^a

Factor	Odds ratio (95% CI)	P-value
Group (RW)	1.26 (0.93 to 1.69)	0.130
Sex (male)	1.64 (1.22 to 2.19)	0.001
Inner regionalb	1.85 (1.26 to 2.72)	0.002
Number of GPs in the practicec	1.04 (1.01 to 1.08)	0.020
Specialty
Breast	1.00
Genitourinary	3.65 (0.23 to 58.27)	0.360
Gynaecology	27.2 (1.72 to 429.27)	0.020
Lower gastrointestinal	0.38 (0.02 to 6.85)	0.510
Respiratory	3.84 (0.24 to 60.58)	0.340
Upper gastrointestinal	1.98 (0.12 to 32.13)	0.630
Random-effect of specialists
Sigma_u	1.34 (0.81 to 2.19)
ρ	0.35 (0.17 to 0.59)	<0.001d

^aResults were derived from the random-effect logistic regression and specialist identity was entered as the random effect in the model.

^bBased on preliminary analysis, clinic remoteness was regrouped into two groups: major cities/outer regional/remote (reference group) and inner regional.

^cGP numbers (range: 0–21) were entered into the model as a continuous variable.

^dP-values were derived from the likelihood-ratio test of ρ = 0.

DISCUSSION

Summary

Specialists were not more confident about scheduling patient appointments based on letters prepared using the RW program. The most important factor influencing their confidence in the information was the identity of the triaging specialist. Differences in specialist decision making have been reported.^17,18 Due to such differences in opinion, patients are likely to experience marked variation in appointment times, depending on where they have been referred. This exacerbates current inequities in the delivery of quality health care.^19,20

The RW GPs referred more patients than their control group counterparts. This finding should be interpreted with caution due to the potential for social desirability bias,²¹ and/or novelty effects.²² Nevertheless, every vignette described patients with ‘obvious’ cancer symptoms and each merited a referral for specialist treatment. Therefore, if RW increased the quality of referral information and increased the likelihood that patients with cancer symptoms were referred sooner rather than later, this represents an important outcome. However, some patients were not referred. Australian GPs are able to refer patients for some investigations, which are not available to GPs in some other countries. However, the patients in this study would have been unlikely to benefit from such investigations nor would these investigations have obviated the need for specialist input. Previous Australian research suggests that some patients experience significant delay from presentation to diagnosis. For example, one study found that almost one-third of patients (28%) experienced a delay of over 2 months.²³

Another unexpected finding was the preference for some referrals in free-text form. This was particularly the case for referrals to respiratory specialists, where the specialists were more likely to be confident about control group referral letters. Although the reasons for this are unclear, the physicians may have identified ‘illness scripts’ in the story the free-text allowed.^24,25 Furthermore, the specialists reported being more confident about scheduling appointments in gynaecology and genitourinary referrals in phase 2 of the study, regardless of the group referring patients. This may have been an effect of being involved in the study for those four specialists.

Strengths and limitations

In a study involving real patients, GPs would be unlikely to consult 24 patients with cancer symptoms warranting urgent referral within the study period. However, a key strength of this trial was that all GPs were exposed to the same clinical challenge. Second, as the cases were all standardised actor–patients, the GPs were not required to obtain informed consent before forwarding the data to the researchers; as such, it was more likely that all referral data were captured.

Despite the value of these research findings, three limitations warrant mention. First, it is possible that the specialists could distinguish RW letters from control group letters. Second, there was no doctor–patient interaction; in practice, this may influence referral decisions. Finally, some patients who were not referred may have been referred later following urgent investigation by a GP; for example, a fine-needle biopsy and mammogram in the case of a patient with a breast lump or a computed tomography scan for a patient with weight loss, haemoptysis, and a suspicious chest X-ray. However, an exploration of this possibility was beyond the scope of this study.

Comparison with existing literature

This study concludes that semi-structured referral letters based on clinical guidelines have limited impact on the scheduling of appointments for most patients referred to the specialties involved. This is in contrast to previous studies that found some referral letters may lead to delayed outpatient appointments.⁷ To the authors’ knowledge, this is the first experimental study to report the impact of referral letters on the scheduling of appointments for the same cases in different formats to different specialists. A previous UK trial similarly found limited clinical value in a referral proforma.¹² An earlier systematic review failed to identify any robust studies that demonstrate an impact from the quality of referral letters on patient outcomes.⁷ The data reported here suggest that semi-structured, guideline informed referral letters are unlikely to influence the scheduling of appointments by specialists. Thus, there remains the potential for delayed diagnosis due to the scheduling of patients with high-risk cancer symptoms with routine appointments.

Implications for research and practice

An earlier systematic review failed to identify any robust studies that demonstrate an impact from the quality of referral letters on patient outcomes.⁷ The data reported here suggest that semi-structured, guideline-informed referral letters are unlikely to influence the scheduling of appointments by specialists. Thus, there remains the potential for delayed diagnosis due to the scheduling of patients with high-risk cancer symptoms with routine appointments.

In many countries, routine appointments may be made significantly later than urgent appointments, and in some cancers, this may lead to poorer outcomes. For example, in a 2010 report from the UK,²⁶ where GPs similarly play a gatekeeper role to specialist services, it was noted that 24% of cancer diagnoses are made from what are classified as routine GP referrals.

It was encouraging from the data in the current study that the quantity of clinically relevant information relayed by the RW GPs increased and that RW participants referred more cases; as such, the scope for relaying more information and more referrals of at-risk patients was enhanced. Despite this encouraging result, the data also suggest there is limited consensus between secondary care clinicians on what constitutes an urgent case. It may also be relevant to the outcome of the study that neither GPs nor specialists were formally inducted to the UK NICE referral guidelines.¹¹

Further research is needed to determine whether doctors in both sectors who are familiar with these (or other guidelines) are more likely to request and be offered timely appointments.

Funding

The Western Australian Cancer and Palliative Care Network funded this study. The study sponsors had no role in study design, data collection, data analysis, data interpretation or writing the report.

Ethical approval

The Curtin University Human Research Ethics Committee (RD-14-11) approved this study. All participants provided informed consent.

Trial registration

Australia and New Zealand Clinical Trials Register (ACTRN12611000760976).

Provenance

Freely submitted; externally peer reviewed.

Competing interests

None declared.

Discuss this article

Contribute and read comments about this article: www.bjgp.org/letters