Introduction of Electronic Referral from Community Associated with More Timely Review by Secondary Services

Abstract

Background

Electronic referral (eReferral) from community into public secondary healthcare services was introduced to 30 referring general medical practices and 28 hospital based services in late 2007.

Objectives

To measure the extent of uptake of eReferral and its association with changes in referral processing.

Methods

Analysis of transactional data from the eReferral message service and the patient information management system of the affected hospital; interview of clinical, operational and management stakeholders.

Results

eReferral use rose steadily to 1000 transactions per month in 2008, thereafter showing moderate growth to 1200 per month in 2010. Rate of eReferral from the community in 2010 is estimated at 56% of total referrals to the hospital from general practice, and as 71% of referrals from those having done at least one referral electronically. Referral latency from letter date to hospital triage improves significantly from 2007 to 2009 (p<0.001), from a paper referral median of 8 days (inter-quartile range, IQR: 4–14) in 2007 to an eReferral median of 5 days (IQR: 2–9) and paper referral median of 6 days (IQR: 2–12) in 2009. Specialists upgrade the referrer-assigned eReferral priority in 19.2% of cases and downgrade it 18.6% of the time. Clinical users appreciate improvement of referral visibility (status and content access); however, both general practitioners and specialists point out system usability issues.

Discussion

With eReferrals, a referral’s status can be checked, and its content read, by any authorized user at any time. The period of eReferral uptake was associated with significant speed-up in referral processing without changes in staffing levels. The eReferral system provides a foundation for further innovation in the community-secondary interface, such as electronic decision support and shared care planning systems.

Conclusions

We observed substantial rapid voluntary uptake of eReferrals associated with faster, more reliable and more transparent referral processing.

1. Background

Electronic referral (eReferral) is one of the underpinning technologies in New Zealand’s national health IT plan [1]. As such, the Ministry of Health commissioned the authors in 2010 to investigate four pioneering implementations of eReferral in New Zealand (NZ). Findings from the best-established and first investigated of those four implementations are reported here.

‘Referral’ is a term that exists in common parlance, and indicates recommendation of a person or organization to another party for a particular purpose, such as to supply a service. Particularly relevant to the present context is the definition in Cole's Medical Practice in New Zealand:

Referral involves transferring all or some of the responsibility for the patient’s care, usually temporarily and for a particular purpose, such as additional investigation, care or treatment, which falls outside your competence or knowledge (p. 130) [2].

Cole’s also sets surrounding expectations: the notion that the referrer, usually a General Practitioner (GP), maintains overall responsibility for the management of the patient; that the referrer is responsible to include all relevant history and information about the current condition when referring; and that the referred specialist will provide back the information required for the continuing management of the patient. It is also notable that there will be a waiting time between the request for shifting of responsibility and the assumption of responsibility at the other end [3].

To describe a system as supporting electronic referrals indicates that the referral process is supported by IT for at least part of the workflow. eReferral systems of various sorts are becoming increasingly commonplace. eReferrals have been piloted in Australia [4]. In the US, the Blue Care Network of Michigan exemplifies a system focused on the insurance and gate keeping aspects of eReferral [5]. eReferral systems for GP-to-specialist referral have been evaluated in the UK, where they were found to improve demographic content of referrals but worsen clinical content [6]. eReferral is commonplace in Denmark, where the technology is accessible to essentially all GPs. Although eReferral is used for only about 40% of referral transactions in Denmark, a study finds that it offers substantial financial benefits through reduced staff time in handling the referral as compared to paper [7]. Common barriers to successful coordination of eReferrals, observed in the context of the US Veterans Administration, include lack of an institutional referral policy, lack of standardization in referral procedures, ambiguity in roles and responsibilities, and inadequate resources to adapt and respond to referral requests effectively [8]. Experience from Norwegian eReferral implementations highlights the importance of collaborating actors – clinicians on both sides of the referral process – to understand each other's needs and work processes [9].

eReferrals offer the potential for communication that is quick, well-documented and reliable as compared to post or fax. There is also the potential to control and structure content as optional and mandatory fields; this may promote completeness, but also may introduce disadvantages through its rigidity. Because NZ enjoys a high-level of sophistication in Practice Management System (PMS) usage in its general practice sector [10], eReferrals also offer the opportunity for data to be automatically supplied from the PMS database into the referral form (auto-population). Through these features, eReferrals offer the potential for transformation in health delivery toward a more coherent interface between services across the community-hospital boundary as compared to freeform posted letters and point-to-point faxes. Moreover, eReferrals can provide a ‘hook’ for electronic decision support and for an IT-mediated social network among the stakeholders in the health of the referred patient – in fact, achieving ‘Share Care Planning’ systems with decision support is a key objective of NZ’s national health IT plan [1].

Herein we present the findings of a retrospective study of the impact of introducing an eReferral system that manages referrals from community into public secondary healthcare services. The eReferral system was introduced in 2007 to a regional NZ healthcare jurisdiction, Hutt Valley District Health Board (HVDHB), which has one principal facility for provision of secondary services, Hutt Hospital. We hypothesize that eReferral, if uptake is substantial and sustained, should result in more efficient (and thus timely), as well as more transparent, processing of referrals from the community.

2. Objectives

To understand the impact of the introduction of eReferral to a region in quantitative and qualitative terms. Impact is assessed by measurements of electronic transactions, allowing assessment of uptake as eReferral volume and trend in volume (the latter testing whether uptake is sustained), and as penetration in terms of total percentage of referral volume that is electronic. Transaction analysis also allows measurement of changes in the time to secondary clinician review (‘triage’) of the referral letter. To ensure that eReferral is associated with a net benefit to system performance we examine both eReferral and total (paper and electronic combined) triage times. Further characterization of eReferral usage patterns is given by analysis of distribution of GP and secondary-assigned referral priority and use of attachments. Qualitative feedback from interviews and focus groups provides insight on benefits and/or liabilities of the solution, including influence on workflow and usability.

3. Methods

3.1 Setting

In NZ, District Health Boards (DHBs) operate public hospitals and purchase community services from Primary Health Organisations (PHOs) for a geographic region. HVDHB covers a population of around 150,000 in the area of the Hutt River valley, including the city of Lower Hutt which is largely a dormitory community for the capital city of Wellington. Currently there are 5 PHOs in the Hutt Valley region with 36 general practices registered to them, where approximately 116 GPs offer their services. Practices are often overbooked, making processing time a critical factor for managing referrals.

HVDHB offers public secondary services through the 260-bed Hutt Hospital in Lower Hutt. As of November 2010 there was no central referral management at Hutt Hospital, however the general Out Patient Department (OPD) with 8 administration staff manages 15 services; the remainder receive and manage their own referrals. In the context of a NZ public hospital, services undertake clinical triage of referrals that assigns priority levels to them, including declining to service some requests (noting that private services are also available). Referral management within the DHB involves two concurrent systems: a Clinical Workstation (integrating data feeds, such as lab results, as well as local data and templates), and a Patient Information Management System (PIMS, providing general inpatient tracking, by a different vendor from the Clinical Workstation).

3.2 eReferral Solution

HVDHB’s eReferral solution was deployed in 2007. The solution is limited to referrals that occur between general practices and HVDHB’s secondary services (essentially Hutt Hospital) – it does not include referrals from other community based providers, to other secondary (or tertiary) service providers, or among secondary services within Hutt Hospital (although it does allow a GP referral to one hospital based service to be transferred to another Hutt Hospital service). By October 2007 twenty-eight services – 16 with service-specific forms and 12 using a generic form – had been deployed across 25 general practices. With the exception of the Emergency Department (excluded from eReferrals for clinical safety reasons) these 28 services constituted the total of Hutt Hospital services at the time the business case for the IT project had been set. Service-specific forms were developed where appropriate clinical staff were available and interested to collaborate in the form development. ►Table 1 provides an overview of the eReferral forms implemented. By end of 2010, thirty general practices were configured to refer into Hutt Hospital with this solution.

Table 1.

An overview of eReferral forms implemented

Form	Focus	Service-specific data items
		Constrained*	Free-text
Adult Mental Health	Possible disorders and identified stressors	23	25
Audiology	Condition summary; speech or cognitive issues	3	5
Child Mental Health	Possible disorders and identified stressors	23	25
Diabetes	Blood and body measurements	10	0
Dietetics	Adiposity, exercise and social data	5	1
General Surgery	Breast, menstrual and colorectal data	10	8
Obstetrics Secondary Care	Pregnancy details and obstetrics history	2	4
Occupational Therapy	Living arrangements, current services and impairments	15	9
Oncology	Patient awareness of problem; previous consultation date and consultant	3	2
Ophthalmology	Visual acuity and optometrist’s report	2	3
Plastic Surgery	Skin lesion history and characteristics	33	14
Podiatry	Ulceration and amputation data; risk factors	7	7
Respiratory	Summary by condition (asthma, bronchial carcinoma, etc.)	17	15
Social Work	Referral reasons, identified stressors and other agencies involved	32	32
Special Rehabilitation Services	Additional contact person, current support services and personal ability	12	21
Speech Language therapy	Adult and child referral reasons; swallowing and voice data	3	8
Generic	Used by cardiology, dermatology, ear nose and throat (ENT), gastroenterology, general medicine, gynaecology, neurology, orthopaedics, paediatric medicine, physiotherapy, renal, and rheumatology services.	0	0

* Pick-list, date or numeric

The GP creates a referral from within their PMS software (with the solution currently limited to the brand with dominant market share) using PMS-based templates developed by HVDHB working with the region’s health messaging service provider. The form is pre-populated with PMS data, which can be edited by the GP prior to submission (which stores a copy to the PMS). The GP referral is messaged as XML via the regional service where it is mapped to HL7 message format (in keeping with an NZ Health Information Standards Organisation, HISO, standard [11]) and sent on to the Integration Engine underlying the HVDHB’s Clinical Workstation. The Integration Engine generates an acknowledgement back to the network confirming receipt of the referral, which is relayed back to the GP PMS.

The Integration Engine maps the HL7 message to XML and the GP referral displays in the Clinical Workstation as a two-part form. The first part is the GP Referral, which is read-only for hospital based users. This may include attachments, particularly reports of diagnostic tests, generally in Rich Text Format (RTF). The second part of the form allows hospital staff to view and process the eReferrals displaying in automated role-based work lists. This section provides drop down selections and comment fields for clinicians to communicate within the service administration to manage clinics, required tests or follow up with patients prior to the appointment. The referral management activity creates an automated sequence of process events through clinician triage (assignment of priority) and, if not declined at triage, regarding creation of a booking for a first specialist appointment (FSA).

The scope of the project as implemented by late 2010 has not included changes to the pre-existing referral handling processes done by staff in the PIMS referral module. As such, manual transfer of data from the eReferral form to the PIMS is done by clerks using dual screens. All referrals (electronic or paper) are logged to the PIMS. The aspects of the referral workflow most relevant to the present study are illustrated in ►Figure 1.

Fig. 1.

Referral workflow

3.3 Data Sources, Protocol and Analytical Methods

Transactional data was collected from the eReferral database as stored with the Clinical Workstation: 33,958 records representing the total operation of the system from October 2007 to end October 2010. Records of all referrals (electronic and paper) from the hospital PIMS were extracted from January 2004 to end October 2010: 108,652 records with ‘source’ code of General Practice (the system also tracks internal referrals and referrals from other community services). These data allow examination of eReferral uptake (eReferral volume over time and proportion of referrals that are electronic); changes in latency from letter date to triage at secondary services; distribution of referral priority as assigned by referring GPs and receiving specialists; and frequency of eReferral attachments of various formats. The extracts, de-identified and using encrypted health identifiers (matchable across data sets, but not re-identifiable by the authors), were made available to the authors by HVDHB.

Because of the loose coupling of the Clinical Workstation and PIMS, estimating proportion of referrals that are electronic requires several considerations. Not all general practice referrals into Hutt Hospital are within the scope of the eReferral solution; notably, Hutt Hospital is a regional referral centre for plastic surgery (including excision of skin lesions, a common procedure for rural populace in ozone-poor southern latitudes). Conversely, not all eReferrals are logged to the PIMS, with allied and mental health services, and the endoscopy component of the gastroenterology service using other solutions. Furthermore, multiple related eReferrals may be combined into one record in the PIMS, and the operator can fail to transcribe the eReferral ID to the PIMS (thus having it appear as a paper referral). Within these limitations, we estimate two relevant proportions: (a) proportion of eReferrals electronic out of the total at Hutt Hospital (representing penetration from the perspective of hospital staff); and (b) proportion electronic from providers registered with HVDHB and having used the solution at least once (indicative of penetration among those with demonstrated access to the solution).

The online eReferral form includes a GP-assigned priority that is stored in the Clinical Workstation data. The specialist assigns the HVDHB’s priority at the time of triage, which is represented in both the Clinical Workstation and PIMS data (GP-assigned priority, however, is not systematically encoded in paper referrals and is not recorded in the PIMS).

Statistical analysis is conducted in terms of linear regression on eReferral volume by month; and Kolmogorov-Smirnov (K-S) test for differences of the cumulative frequency distributions of triage latency between electronic and paper referrals, and for total referrals across years. 2010 data is omitted from statistical analysis of triage latency since that year’s data is incomplete (and longer latencies could span into 2011); also eReferral data in 2007 is limited to a small percentage of transactions in the latter part of the year, and is thus excluded. All analyses were computed in Microsoft Excel.

We also conducted interviews and focus groups/forums. One-on-one interviews included 3 vendors, 6 members of DHB management, and 3 DHB clinical users. Three focus groups/forums were conducted: for GPs (10 GPs attending), DHB clinicians (6 attendees) and Outpatient Services (2 managers [one clinical] and 5 administrators). Interviews had a set agenda of open questions around user acceptance, effect on referral processing and potential impact on quality of care. The interview approach was influenced by: Grounded Theory and Theoretical Sampling [12] – generation of theory during the research process, and extending interviewee lists based on emerging themes; and Action Research [13] – explicitly iterative action interleaved with reflection, and overt interest in helping to make the system work better. After completion of six interviews, audio tapes and the interviewer’s notes were reviewed collectively by the authors to review the interviewing and analysis style and revisit the interview schedule. Prior to the focus groups, a further review was conducted to determine the issues in need of greatest emphasis for clinical input.

4. Results

4.1 Analysis of transactional data

Uptake

►Figure 2 graphs the PIMS volumes for referral records indicating source as General Practice along with the transaction volume for all eReferrals (from the Clinical Workstation database) by year. ►Figure 3 graphs eReferral volume by month to provide a finer-grained view of eReferral use.

Fig. 2.

General practice referral volumes by year (* 2010 data inflated by 6/5ths to estimate full year volume)

Fig. 3.

eReferral volume by month October 2007-October 2010

A boost in total general practice referrals after relative stability in earlier years, tracking with increased eReferrals particularly between 2008 and 2009, indicates interaction of eReferral uptake and increase in total referrals.

►Figure 3 can be interpreted as showing a rapid growth period in the year starting October 2007, and then a slower growth in the remaining three years. Linear regression on monthly volume from October 2007 to September 2008 places growth at 53.5 transactions per month (95% Confidence Interval [CI] 42.7 – 64.2); linear regression on October 2008 to October 2010 yields the average increase at 11.6 transactions per month (95% CI 5.7 – 17.5). Thus, the latter period is still one of growth (p<0.0005), but the variability in monthly volumes places a wide interval on the exact rate. Dips around end-of-year 2008 and 2009 are logical to attribute to the summer-holiday period (Christmas and New Year coming in summer in the Southern Hemisphere).

For 2010, the raw volume of Clinical Workstation eReferral records is 70.1% of the number of PIMS referrals with source from general practices. However, this cannot be viewed as indicating that 70% of GP referrals into Hutt Hospital are electronic. ►Table 2 provides a breakdown of 2010 referral volume (total and proportion electronic from the PIMS database, and eReferral count from the Clinical Workstation database) showing the overall distribution of the services referred to. There are substantial variations in the accounting between the two systems along the expected lines (see Methods). Relevant to estimating eReferral proportion, we find 103 of the 15,805 PIMS GP referral records for 2010 to be concatenated IDs from merger of two or more electronic referrals. While manual analysis and review with HVDHB staff confirmed that eReferral IDs are sometimes omitted in the PIMS, we were unable to estimate a percentage rate. We add 0.5% to estimates of eReferral proportion as a (we believe conservative) adjustment for concatenation and omission of eReferral IDs in the PIMS. As such, we find:

• •55.26% of PIMS general practice referral volume is marked with a non-blank eReferral ID (i.e. explicitly electronic), the adjusted estimate being 56% electronic.
• •Of the 30 practices and 243 referring professionals appearing on the HVDHB list of Hutt Valley based referrers as of November 2010, 22 practices and 116 referring professionals sent at least one eReferral under their name (i.e. under the name of the practice, or under the name of the individual) in 1 January to 31 October 2010. Based on PIMS records, during this period these referrers sent 10,367 referrals of which 7,326 have an eReferral ID, indicating an electronic referral rate of 70.7%; the adjusted estimate thus being 71% as the eReferral rate for those who have used the solution at least once in 2010.

Table 2.

Distribution of referrals by clinical service referred to in 2010

Clinical Service Referred to	GP referrals (paper and electronic) in PIMS	eReferrals in PIMS		eReferrals in Clinical Workstation
	#	#	%	#
Plastic Surgery	2435	778	32.0	847
Orthopaedics	1509	633	41.9	1003
General Surgery	1273	896	70.4	1034
Gynaecology	1182	843	71.3	956
Physiotherapy	1065	770	72.3	794
Ear Nose and Throat	985	651	66.1	714
Paediatric Medical	814	544	66.8	626
Cardiology	780	507	65.0	537
Specialist Rehabilitation	708	368	52.0	313
Respiratory	652	206	31.6	245
Rheumatology	650	154	23.7	230
Gastroenterology	495	371	74.9	937
General Medical	434	316	72.8	486
Audiology	366	227	62.0	255
Diabetes	287	94	32.8	104
Dermatology	279	178	63.8	205
Surgical/Arthritis *	277	199	71.8	0
Ophthalmology	265	191	72.1	205
Dietetics	254	185	72.8	236
Podiatry	229	149	65.1	156
Occupational Therapy	177	107	60.5	158
Endocrinology *	153	83	54.2	0
Obstetrics	149	102	68.5	109
Neurology	144	102	70.8	105
Renal	76	39	51.3	44
Oncology	69	15	21.7	18
Speech Language Therapy	43	26	60.5	38
Child Development Service *	24	0		0
Dental *	11	0		0
Anaesthetic *	10	0		0
Urology *	7	0		0
Vascular *	3	0		0
Mental Health – Adult	0	0		374
Mental Health – Child	0	0		136
Social Work	0	0		221

* Services instituted at Hutt Hospital after the eReferral implementation

It should be noted that not all of the above 116 referring professionals would be GPs (notably, they can be practice nurses) and that mixture of referrals by practice name and by personal name confounds accounting of uptake as a percentage of providers.

Triage latency

►Figure 4 shows the median, first and third quartiles (thus inter-quartile range, IQR) for electronic and paper referrals in 2007 to 2010. ►Figure 5 shows the cumulative frequency distribution of triage latency for paper referral in 2007, 2008 and 2009 and eReferral in 2008 and 2009. ►Figure 6 shows the distribution of whole-of-system triage latency (paper and electronic combined) for 2007, 2008 and 2009.

Fig. 4.

Median, first and third quartile (‘Med’, ‘1stQ’ and ‘3rdQ’ respectively) of letter-to-triage latency for eReferrals and paper Referrals by year.

Fig. 5.

Cumulative frequency distribution of letter-to-triage latency for paper (‘p’) and electronic (‘e’) referrals by year.

Fig. 6.

Cumulative frequency distribution of letter-to-triage latency for whole-of-system (electronic and paper referrals combined) by year.

Due to the tail that includes some very long latencies, median and IQR (rather than mean and standard deviation) are used as the statistics to characterize service pattern. Manual review and discussion with HVDHB IT staff indicates that Triage Date fields can have their initial values overwritten in cases of cancellation while awaiting FSA (with common reasons including patient being unreachable, confirmed deceased or receiving the service privately). In other cases Triage Dates associated with cancelation are indeed the date of initial specialist review, including declines below service threshold and cases resolved by telephone (‘virtual consultation’). Thus, the visual indication that >10% of referrals wait more than a month for triage is at least partially a data artifact, but one that could not be removed without also removing a great many valid cases.

Buoyed by the large sample sizes, the critical value at p<0.001 for the K-S test is reached by any gap in the cumulative frequency distribution larger than 2.7%. Thus the differences in distribution of referral triage latency are significant (p<0.001) for: 2007 paper referrals versus paper or eReferral in 2008 or 2009; eReferral versus paper in 2008 and 2009; and total referrals in 2007 versus 2008 or 2009. 2008 and 2009 total referral latency is superior (quicker) over 2007 at all points on the distribution except for 0 days, where 2007 was 7.8%, 2008 was 6.1% and 2009 was 7.5% (i.e. there was a slightly higher percentage of same-day referral triages in 2007).

Priority

►Table 3 shows the Clinical Workstation based distribution of GP-assigned priority and subsequent priority assessment at the hospital. ►Table 4 shows the PIMS record of HVDHB priority for both paper and electronic referral. It should be noted that ‘Routine’ HVDHB priority assignment includes referrals that are declined as below the service threshold; ‘NULL’ priority is given for cancelled services (e.g. patient seen elsewhere) as well as some services outside the scope of PIMS (e.g. mental health). The profile of NULL shows no bias to Routine over Urgent GP-assigned priority. On this basis we interpret ►Table 3 as indicating that referrer-assigned eReferral priority is upgraded in 19.2% of cases (predominantly from Routine to Semi-Urgent) and downgraded 18.6% of the time. It is notable from ►Table 4 that paper is used more often for referrals that are classed by specialists as Urgent. The higher Not Seen rate in eReferrals may attribute to the use of decline fields to pass a ‘virtual consult’ message back to a GP through the system’s automatic notification channel.

Table 3.

eReferral priority levels in 2010 from Clinical Workstation database: Referrer and HVDHB assignment for eReferrals; priorities assigned for all referrals from GPs

Referrer (GP) Priority	N	%	HVDHB (specialist-assigned)
			Priority	%
Urgent	1,849	16.7	Urgent	32.2
			Semi-Urgent	34.0
			Routine	12.5
			NULL	21.3
Semi–Urgent	3,932	35.5	Urgent	7.7
			Semi-Urgent	47.5
			Routine	30.7
			NULL	14.2
Routine	5,305	47.9	Urgent	3.6
			Semi-Urgent	30.8
			Routine	46.0
			NULL	19.7

Table 4.

Referral priority levels (paper and electronic) in 2010 from PIMS

HVDHB Priority	Paper		Electronic
	N	%	N	%
Urgent	1347	19.0	976	11.2
Semi-Urgent	2337	33.1	3057	35.0
Routine	1856	26.2	2364	27.1
Not seen	1531	21.2	2337	27.1

Attachments

51.7% of eReferrals in 2010 have at least one RTF attachments, with a mean of 6.2 RTF attachments per eReferral with an RTF attachment; these generally represent laboratory test results, although other documents can be attached this way (e.g. letters). JPEG attachments are used a total of 476 times in just 1.8% of 2010 eReferrals. No other attachment types are frequent.

4.2 Thematic analysis from interviews

Several inter-related themes emerge from the interviews and focus groups/forums. These are summarized in ►Table 5 and include: positive messages about the availability and transparency of the referral under electronic management; generally positive transformations and new opportunities in workflow through electronic handling; generally neutral feelings about impact on content; and some complaints about outstanding user interface issues, although also with some GPs seeing the efficiency advantages of auto-population. A number of further themes involve the implementation process, management and leadership issues, as well as deeper consideration of the role of eReferral in transforming the relationship of GPs and specialists. We do not consider these themes in depth in the present paper, although it is worth saying that clinical and administrative leadership was essential in persisting with the solution despite some early technical glitches, and particularly to win over initial resistance of some specialists.

Table 5.

Themes emerging from interviews

Theme	Key points
Availability/ transparency	eReferrals are highly unlikely to be misplaced/lost, or lose attachments, as compared to paper. Receipt is confirmed. Present in online work queues – less likely to overlook. Can be accessed by other departments (e.g. emergency department).
Work transformation	General practice: Typing the letter is a big change for some – some forego use to stay with dictation. Format of letter is changed by auto-population and cut-and-paste – less formal than traditional narrative. eReferral writing mostly done at end of day but may be done in consult: (a) if relatively simple or (b) to show patient that a referral is being made. Staff don’t have to print and post (or feed to fax). Hospital: No change in outpatient administrator staffing levels, despite dual entry into Clinical Workstation and PIMS. eReferral can be reviewed without getting up from desk. Have modified procedures in some services for periodic and consolidated review, for consistency and so that paper referrals get equal attention. Easy for specialists to redirect triage tasks when away (although have to remember to do so). Tedious to work with ‘multi-modal’ referrals that have posted images associated with online letter. Between: GP can send a referral more as a query; immediacy of electronic format makes a decline result more acceptable than with paper. eReferral does not replace the need for specialists and GPs to have other channels of communication / education with respect to each other’s expectations and the information needs in referral letters.
Data quality	Legibility: Better than fax (esp. forwarded fax). [Note that ‘paper’ referrals are still generally typed, but would be free-form letters, possibly augmented by locally-adopted templates, then printed from the PMS] Content: Presentation is not entirely ideal – service (rather than condition or investigation) level forms may not have place for key fields and make it difficult to position explanation. Specialists still reconfirm data with patient. Specialists felt that online format neither prevented nor ensured a good referral. Appropriateness: Cut-and-paste and sometimes out-of-date content from PMS may be a distraction. GP may not know what information specialist thinks are important until (or even after) they get a decline message.
User interface	General practice: Some think auto-population from the PMS speeds process; some see typing as a slow down. Attaching images difficult and tedious. Cannot preview lab results from screen where they are attached. Hospital: Attachments open separately from main letter. Can be slow to open; not problematic for lower volume services (e.g. mental health) but a big problem for high-volume services (e.g. ENT).

5. Discussion

eReferrals have seen rapid, substantial and sustained uptake after their introduction as an alternative to paper referrals from general practice clinics to secondary services of Hutt Hospital in late 2007. Our best estimate is that 71% of referrals are electronic from providers who have sent at least one electronic referral, constituting 56% of non-Emergency general practice referral volume into Hutt Hospital. This is a substantial rate of uptake – noting that general practice referrals often come from outside the DHB, and given that providers are still free to use familiar letter writing tools in their PMS instead of more structured eReferral forms. While electronic Referral is now the dominant modality, it has not replaced paper in this setting. Although the eReferral solution does not include referral to the Emergency Department, eReferral is used for cases that are assessed as urgent by GPs and by specialists; however, paper referrals are more often assessed as urgent by specialists (19% versus 11% electronically). This may indicate a tendency, at least by some GPs, to abandon the novel modality for the most complex and/or demanding cases and revert to traditional letter writing skills.

Referral latency from letter date to hospital triage has improved significantly since calendar year 2007 with no changes in outpatient administration staffing levels. This effect was measured with the remaining paper referrals as well as eReferrals, although eReferral processing remains significantly quicker. There are efficiency downsides for the hospital staff: duplicate data entry to transcribe from the Clinical Workstation (integrated with the eReferral messaging network) into the hospital PIMS (which is not), and latency for triagers waiting for attachments to open. These appear to be more than offset, however, by the visibility of task assignment and ease of access associated with eReferrals.

The measured attributes of eReferrals appear to be in the direction of improving the patient service experience generally, and patient safety in particular. Specialist triage to assign priority is more timely, and includes many upgrades in GP-assigned priority (slightly more than rate of downgrades). This helps patients with greatest urgency to be seen soonest. Moreover, eReferral provides a foundation for controlled management of referral workflow in the hospital – to view worklists, and to forward work to other staff as capacity and availability dictate. In fact, due to the high visibility of eReferrals in need of triage as compared to paper, some of the hospital services have opted for periodic (e.g. weekly) consolidated review of all referrals (electronic and paper), both to improve consistency in triage decisions and so as not to have patients with paper referrals unduly disadvantaged in level of service. eReferrals provide a further safety advantage in that any authorized user can be aware of and access the content without the restriction of access to a specific physical location (such as a desk or filing cabinet at a referred service). This could be an advantage, for instance, if a patient presents to Emergency while awaiting their FSA. In general, concern about a referral being misplaced (in whole or part) is much less with electronic management. Automated updates keep the GP informed on referral status, including receipt, triage, date of FSA and FSA attendance. Potential safety downsides associated with the possibility of auto-populated data being incorrect or irrelevant are mitigated by specialists maintaining traditional practice of reviewing information with the patient at the FSA.

Clinicians point out a number of continuing usability issues. For GPs, the transformation to use of online forms for referral is a mixed experience. Some feel that auto-population leads to a speed up, whereas others, particularly those most committed to dictation, find typing to be a burden, and in some cases a burden that has kept them from transitioning to eReferral. Form designs do not always allow for key information to be readily presented, or for free text explanations to be well-positioned in the resultant letter. Procedure for attachment of images is found to be cumbersome, and is not widely understood or used. The inability to preview test results on the screen for attaching them can lead to addition of irrelevant results. For specialist review the interface can be cumbersome as compared to paper, particularly due to slow system response in opening attachments (although the implementers have identified that this problem can be largely overcome by synthesizing the entire report, including attachments, as a single PDF file).

The solution leverages NZ HISO standards for messaging between hospital and community (11) and for security (14). The former of these provides for structured payloads that could be used in a more granular fashion for decision support or clinical data repository population within the receiving hospital systems. The solution pre-dated the HISO technical specification which provides a recommended architecture for online forms to interoperate with practice management software (15), which should facilitate maintenance of online forms as well as auto-population. The relevant NZ standards suite may benefit from further review and uptake of Integrating the Healthcare Enterprise (IHE) standards, notably the Query for Existing Data Profile (16) within the Patient Care Coordination Framework for more standardized form auto-population methods.

In terms of limitations, the present study did not attempt direct time-and-motion analysis or to produce a cost/benefit assessment based on staff time savings. The findings are consistent, however, with the study of Danish eReferral systems that found significant savings in staff time (7), at least for hospital outpatient administration. Conversely, we find somewhat higher levels of uptake than in Denmark. As with the Danish setting, it is important to note our findings are situated in an environment with a strong role of general medical practice as a secondary referral gateway, and with essentially universal uptake of general practice computing.

The study was limited to a relatively simple regional setting with just one secondary referral hospital. The study is limited to an interrupted time-series without a control other than performance at the implementation site in the period immediately prior to implementation. Moreover, we were unable to go back earlier in assessment of letter-to-triage latency as triage dates were only routinely recorded in the PIMS from the start of 2007.

The present study did not include systematic review of quality of referral content. We do not find, however, that our results are consistent with Shaw and de Berker (6) who found that clinical content quality went down with eReferrals. While clinical interviewees acknowledged that online format transformed the content, specialist opinion was overtly neutral about the impact on quality, pointing out that a GP could write a good, or bad, referral either on paper or online. While there may be some ‘laggard’ effect with respect to the less-than-100% uptake in the HVDHB system, it may be that GPs opt for paper when they feel the online format would impair their ability to write a referral of the required quality (since they turn more to paper for the most urgent cases). It is notable that the HVDHB forms are service-specific, not more fine-grained such as forms targeted to specific conditions or investigations. It has been pointed out that this lack of specificity in the form designs limits the opportunity for the forms to prompt for required clinical content. It remains as an area for future research to determine what impact on uptake, efficiency and content would results if the eReferral solution evolved to one with a much larger number of more prescriptive forms and possibly with active decision support recommendations.

The eReferral solution provides a more rapid, and perhaps informal, feedback mechanism as compared to paper that some interviewees noted as opening the channel for a ‘virtual consult’ where the GP uses the referral mechanism more for information than in expectation of actually transferring care to secondary services. In particular, the decline message can provide rapid feedback from secondary to community on further community-based management of the patient. We observed an increase in overall referral volume in 2008 and 2009 that may relate to this use of eReferral, as well as a generally greater willingness to refer into a faster and more transparent system. The increase in total referral volume concurrent with the uptake of eReferral suggests that the transformation associated with introduction of eReferrals is something deeper than simply automation of the old way of doing things. Our findings are consistent with findings of reduced waiting time measured in association with an email triage system managing referrals to a neurologist in Northern Ireland, where the reduction was observed despite an increase in total referrals (and where replacement of face-to-face consults with virtual consults appears to have been substantial) (17).

The eReferral system provides a foundation for further innovation in the community-secondary interface. Online referral forms, particularly if they are made investigation or condition specific, could be designed to provide electronic decision support about referral appropriateness and management options. While the current eReferral solution automated a process with workflow little-changed from the paper procedure, it could be extended into a shared care planning (18) system, emphasizing the cooperation of a GP and specialist as members of a care team working together rather than transferring the patient between. Finally, it should be noted that the electronically-mediated communication on specific patients may serve to help GPs learn about specialist expectations. Our interviewees pointed out, however, that it does not replace the need for other processes, conventional or virtual, to allow general practice and specialists to communicate with and educate one another on their expectations and requirements around information needs to provide the best care for their patients.

6. Conclusions

We have observed the introduction of eReferrals into secondary services to have substantial rapid voluntary uptake from GPs in an environment where GPs are regular users of electronic medical records for their practice. Although the secondary services still receive many paper referrals, with a majority of referrals managed electronically, overall management of referrals is faster, more reliable and more transparent as compared to the previous paper-only state. Moreover, the online messaging of referral status from secondary to community opens a communication channel for a declined referral to function as a virtual consultation.

Conversion to an online referral form is a change for GPs, leading to more use of auto-population and cut-and-paste from the practice management software as compared to crafting a letter conventionally, and favoring physicians typing over dictation. Speed of opening attachments is a downside as compared to paper for specialists working at high volumes. The remaining referrals in paper format appear to be more heavily weighted toward urgent priority as compared to eReferrals (even though the scope of the solution does not include Emergency Department), which may indicate a tendency to revert to paper for difficult cases.

Clinical Relevance Statement

In an environment with established use of general practice computing, voluntary uptake of electronic referrals can see electronic modality overtake paper.

Introduction of electronic referral management into hospital services is associated with faster, more reliable processing of the referral from authoring of letter to assignment of priority by a specialist

Electronic referrals may promote patient safety through timeliness of assignment of priority, reliability and overall transparency of the referral management process, and wider availability of referral information.

Conflicts of Interest

The authors performed this work under contract to the New Zealand Ministry of Health through Auckland UniServices. The authors have no commercial interest in any of the software products involved in the solution.

Protection of Human Subjects

The research protocol for this study was approved by the Multiregional Ethics Committee of New Zealand as protocol MEC/10/066/EXP and executed with the permission of HVDHB.