Electronic checklists improve referral letters in gastroenterology: a randomized vignette survey

Abstract

Objective

Investigate whether gastroenterologists rate the quality of referral letters higher if electronic dynamic checklist items are added to a standard free-text referral letter. Assess how this affects the gastroenterologists’ assessment of the patient’s need for healthcare and the agreement between their assessments.

Design

Randomized vignette study.

Setting

Norwegian primary gastroenterology services.

Participants

Thirty-two Norwegian gastroenterologists.

Intervention

Between June 2015 and January 2016, participants were recruited through an open invitation to all members of the Norwegian Society of Gastroenterology. They were asked to rate 16 referral letters (vignettes) in a web interface: eight letters in free text following a general template and eight letters based on a general referral template combined with diagnosis-specific checklist items. The study was completed in two subsequent rounds ≥3 months apart.

Main Outcome Measures

Quality of referral letters assessed on a rating scale from 0 to 10. Agreement in the referral assessment and accuracy of the selection of correct preliminary diagnosis and appropriate work-up.

Results

The mean quality assesses on the rating scale was 7.0 (95% confidence interval [CI] 6.8–7.2) for all letters combined (n = 511), 6.5(CI 6.2–6.8) for the free-text referrals (n = 256) and 7.5(CI 7.3–7.7) for the checklist referrals (n = 255) (P < 0.001, paired t-test). No difference was observed in the triage of the patients, but fewer gastroenterologists felt the need to collect additional information about the patients in the checklist group.

Conclusion

Checklist items may ease the assessment of the referrals for gastroenterologists. We were not able to show that checklists significantly influence the management of patients.

Introduction

When a patient is referred by the primary care physician to the secondary healthcare system, the referral letter is often the sole piece of information available regarding the patient’s medical history. This letter is used by the hospital consultant to assess the need (and urgency) for secondary healthcare.

The quality of this important document is often unsatisfactory, and essential information regarding the patients’ symptoms and findings is often missing [1–16]. This makes the referral assessment challenging. Consultant specialists have expressed frustration due to extra workload needed to assess low-quality referral letters and called for interventions to improve its quality [2].

Checklists have improved the patient safety in surgical procedures [17] and clinical handovers [18]. They have also been documented to improve the quality of referral letters, both in studies done in standardized settings using vignettes [19] and in clinical trials [20–22].

However, the clinical impact of these findings is not yet fully studied. Improved patient management as a consequence of improved referral quality is crucial for the patient as well as for the efficiency of the healthcare system.

In a previous study, we have shown that electronic checklists improve the referral letter quality [19], assessed by a specific and objective Thirty Point Score [23]. A paper version of the contents of one of the checklists is shown in Supplementary file 1.

The aim of the current study was to assess how the increased quality score was perceived by gastroenterologists, whether the checklist influenced the gastroenterologists’ triage of the patients, and whether it resulted in increased agreement in their assessments. In detail, we compared (1) the gastroenterologists’ quality rating and proposed waiting time of referrals for the same case vignette generated with and without checklist templates; (2) the perceived need for additional information to assess referrals with and without checklist; (3) suggested diagnosis and work-up for each case vignette for referrals with and without checklist and (4) agreement and accuracy of the selection of correct diagnosis, scheduled work-up, right for specialized healthcare, waiting time and quality based on referral letters generated with and without checklist.

Methods

Study procedure

This is a follow-up of the Interactive Dynamic Referral Interface (IDRI) trial, in which 25 Norwegian general practitioners (GPs) participated in a randomized cross-over vignette trial and generated two referral letters for eight different indications, one with and one without the help of a checklist [19]. The vignettes forming the basis for the referrals were presented with varying degrees of seriousness of symptoms and findings evoking a predefined diagnosis. Most requiring urgent (2 weeks) or semi-urgent appointments (4–6 weeks), while others did not require medical attention before 6 months according to the national guidelines [24].

A random sample of the referral letters generated in the first round of the cross-over trial was retrieved from the database for use in the current study. We used random sampling to select one free-text referral and one checklist referral for each of the eight indications used in the IDRI trial (dyspepsia, dysphagia, diarrhea, change of bowel habit, constipation, rectal bleeding, long-lasting abdominal pain and jaundice/elevated liver enzymes), thus obtaining eight pairs of referral letters used as vignettes in the present study. Referral letters in the checklist group were only eligible if the GP had actually used the checklist to write the referral letter. We transcribed the selected referral letters to blind the observer for whether the referral letter was generated with the support of checklists or not. In an online questionnaire study, we presented the 16 referral letter vignettes in a web interface to gastroenterologists in two rounds. Each round contained eight vignettes; one randomly selected from each indication pair.

In June 2015, we invited all members of the Norwegian Association of Gastroenterology (n = 364) to participate in the first round of the study. A reminder was sent in January 2016. The doctors were asked to provide their e-mail address in the first round for participation in the second round. After a minimum of 3 months from completion of the first questionnaire, all participants who had provided their e-mail addresses were contacted again to complete the second round. Exclusion criteria were responders who were not gastroenterologists/gastroenterologists in training or did not work with assessing referrals in gastroenterology. Gastroenterologists who did not complete both rounds were not included in the analysis.

Variables assessed

The gastroenterologists were asked to assess the quality of the referrals on a fixed 11-point Rating Scale (FPS) ranging from 0 (worst) to 10 (best), and subsequently triage the referral letters by deciding whether the patient had the right for specialized healthcare (yes/no), the waiting time (number of days/weeks), schedule the work-up (gastroscopy, colonoscopy, radiology or consultation) and indicate the suggested preliminary diagnosis (free-text assessment) of the referred patient. The doctors were also asked to report whether they would need to obtain additional information for the patient referral. We also recorded the gastroenterologists’ demographic variables: age, gender, subspecialty, years of experience, primary workplace and whether they were assessing referrals in gastroenterology.

Statistical analyses

Descriptive data are reported as means or proportions with 95% confidence intervals (CI).

Differences in the rating scale quality assessment and proposed waiting times in days, overall and stratified by case type, were assessed by using paired t-test and univariable linear regression models. The quality assessment and waiting time were the dependent variables and having used a checklist or not the independent variable, respectively. For the overall comparison, we adjusted for the medical doctor cluster. Second, we compared the proportion of gastroenterologists who would obtain additional information for the referral letters with and without checklist using chi-square statistics. Third, we described the preliminary diagnosis and suggested work-up for each diagnosis with and without checklist and compared them using chi-square statistics. Fourth, we assessed the inter-rater agreement of the referral assessments for referral letters with and without checklist by calculating the Fleiss’ kappa for categorical variables (diagnoses, work-up and right for specialized healthcare) and Kendall’s W for ordinal variables (waiting time and FPS). The agreement results were interpreted as <0 = poor, 0.01–0.20 = slight, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial, 0.81–1.00 = almost perfect. Finally, we assessed the sensitivity of the referral letters for enabling the gastroenterologists to select the correct management of the patient, stratified by case type for the preliminary diagnoses and scheduled work-up. Since all cases were designed with a given true diagnosis (no false positives), it was not possible to calculate the specificity of the diagnoses and work-up.

P-values <0.05 were considered statistically significant. All statistical analyses are performed using the SPSS 23.0 and STATA 14.1 (StataCorp LP).

Results

Study population

Between June 2015 and January 2016, 54 gastroenterologists (14.8% of the 364 invited) opened the web-based referral assessment tool, 48 started assessing the referrals and 41 completed Round 1 of the study (Fig. 1). Of those, four were excluded, one was not a gastroenterologist/gastroenterologist in training, three were not working with referral assessment and four were lost to follow-up because they did not record their contact information for the second round. Thirty-three gastroenterologists were thus invited to the second round of the study, of which 32 (97.0%) completed it and were included in the analysis. A total of 512 referral assessments were completed by the 32 gastroenterologists on the 16 referrals. The gastroenterologists were on average 50 years old with 13 years of work experience and predominantly male (91%; Table 1). Of the 32 participants, 25 (78%) were trained gastroenterologists and 7 (22%) were still in training.

Figure 1.

Flowchart.

Table 1.

Characteristics of the participating gastroenterologists

	Participating gastroenterologists (N = 32)
Age, mean (range), years	49.8 (33–75)
Gender, n (%)
Male	29 (90.6)
Female	3 (9.4)
Training n (%)
Specialist in gastroenterology	25 (78.1)
Gastroenterologist in training	7 (21.9)
Years of experience as gastroenterologist, mean (range)	13.2 (1–40)
Workplace, n (%)
Public hospital	31 (96.9)
Private clinic	1 (3.1)

Quality rating and waiting time

The average quality of the referral letters assessed was 7.0(95% CI 6.8–7.2) for all letters combined (n = 512), 6.5(95% CI 6.1–6.9) for the free-text referrals (n = 256) and 7.5(95% CI 7.0–7.8; P < 0.001, paired t-test) for the checklist referrals (n = 255) after accounting for the cluster gastroenterologist. The rating of the quality with and without checklist varied between the different indications (Fig. 2). For the indication ‘abdominal pain’, the gastroenterologists rated the quality of the non-checklist referrals significantly higher.

Figure 2.

Gastroenterologists’ rating of the quality of the referral letters with and without checklist on a 0–10 rating scale, stratified by indication. Footnote: P-value is calculated using paired t-test.

For each indication, the gastroenterologists reported the time they would let the patients wait until they had to be seen by a specialist. The suggested mean waiting time for all referrals did not differ whether the referrals were written with or without checklist, with mean waiting time (weeks) of 4.7 (95% CI 4.2–5.1) for the non-checklist referrals and 4.6 (95% CI 4.0–5.2) for the checklist referrals (P = 0.940). The indications with significantly differently scheduled waiting time were dysphagia (P < 0.001), diarrhea (P = 0.033) and constipation (P = 0.025) (Fig. 3). The most pronounced difference in waiting time was observed for dysphagia with an average waiting time of 1.6 weeks (95% CI 1.3–1.9) for checklist referrals and 2.4 (95% CI 2.0–2.8) for non-checklist referrals, respectively.

Figure 3.

Gastroenterologists’ assessment of waiting time, with and without checklists, stratified by indication.

Additional information needed

More gastroenterologists reported that they felt the need to collect additional information about the patient after reading the referrals from the non-checklist group (27.3%) compared with the referrals from the checklist group (14.9%). The odds ratio for needing additional information was 0.47 (95% CI 0.30–0.73, P < 0.001) when comparing the checklist group with the non-checklist group.

Preliminary diagnosis and suggested work-up

When assessing the preliminary working diagnoses and the scheduled work-up selected by the gastroenterologists, we observed only small differences between the two groups. Most of the gastroenterologists suspected the correct underlying diagnosis and suggested a correct work-up independent of whether the referral letter was written with or without checklist (Supplementary file 2).

Inter-rater agreement of the referral assessments with and without checklist

There were no differences in the agreement between the gastroenterologists in the assessment of the referral letters when comparing the checklist referrals with the non-checklist referrals (Table 2). Fleiss’ kappa for the preliminary working diagnoses, work-up and the assessment of whether the patient had a right to specialist healthcare were all similar in the two groups with moderate agreement between gastroenterologists for the diagnoses, almost perfect agreement for the work-up and poor agreement for the right to specialist treatment. Equally, there was no difference in the Kendall’s W for the FPS and the waiting time to see a specialist.

Table 2.

Agreement of the referral assessment between the gastroenterologists

	Diagnosisa	Scheduled work-up/ consultationa	Right for specialist healthcarea	Waiting timeb	FPSb
Checklist	0.537	0.891	0.126	0.570	0.305
No checklist	0.488	0.943	0.127	0.508	0.325

Interpretation of the results (agreement): <0 = poor, 0.01–0.20 = slight, 0.21–0.40 = fair, 0.41–0.60 = moderate, 0.61–0.80 = substantial and 0.81–1.00 = almost perfect.

^aCalculated using Fleiss’ kappa.

^bCalculated using Kendall’s W.

Accuracy of the referrals with and without checklist

The gastroenterologists were able to select the correct working diagnoses and scheduled work-up for the patient cases with an accuracy of 0.77 and 0.86 for the checklist cases and 0.70 and 0.82 for the non-checklist cases.

Discussion

We have evaluated the effect of diagnosis-specific checklists on the quality of referral letters in gastroenterology, both assessed subjectively by gastroenterologists on a 0–10 rating scale, and on the clinical assessment and the triage of the referrals.

We observed that gastroenterologists considered letters written with the support of checklist to be of higher quality than those written without. However, the difference was only 15% for all the referrals together, and for some indications not significant (rectal bleeding and jaundice/elevated liver enzymes) or even with an opposite effect, showing a higher value on the rating scale for the non-checklist referral (abdominal pain). We also found a higher proportion of gastroenterologists wanting additional information regarding the patient in the non-checklist group, which further emphasizes the usefulness of checklists to achieve a higher quality of referral letters. Collection of additional patient-information from either the GP or the patient itself is time-consuming and is reported as a considerable workload on hospital consultants [2]. Thus, improved quality of the referral letters seems to result in a greater consultants’ satisfaction. Whether this is specific for referrals generated with the use of checklists or a mere consequence of improved referral quality for any reason remains unanswered in this trial. However, it is reasonable to assume that the effect would be seen for most interventions generating more clinical information in the referral letter as long as this is done without compromising the structure or conciseness of the letter. The checklist may exert its effect by prompting the GPs to provide information in a more structured and reticent way, combining the need for more information with a short and concise presentation. We found a small, but hardly relevant difference in the assessment of the referrals in terms of tentative diagnosis (Supplementary file 2), prioritization (proportion of patients with the right for prioritized healthcare), work-up (Table 2) or scheduled waiting time (Fig. 3) between the checklist referrals and the free-text referrals. We believe that this may reflect that the impact on patient outcome of the checklist intervention may be better assessed in clinical practice, but it is also possible that increased the quality of referral letters may not influence patient management. In addition, the quality of the referral letters in this trial was generally high. The gastroenterologists scored the referrals significantly higher on the 0–10 rating scale than in other similar trials [13, 23]. It is possible that the non-checklist referrals already had a quality where correct and uniform assessment was fairly easy to achieve for the gastroenterologists, reflected by, e.g. the almost perfect agreement in the scheduled work-up. Further, it is likely that increased the quality of the referral letter makes the letter easier to read and less frustrating to assess even if it makes no difference in the patient outcome. This is a potentially important benefit for the gastroenterologists spending time on performing this task on a regular basis. In our health region, hospital doctors mainly have to rely on the referral letter and only occasionally have access to the patient’s previous laboratory and radiology work-up when these tests are initiated by the GP. Accurate information of this in the referral letters is thus of importance for the referral assessment.

Though the study was conducted in Norway, we consider that the results also apply to other countries with similar referral systems, where specialists make a triage of patients referred to secondary healthcare based on the content of the referral letter.

The present study provides new evidence regarding the value of clinical checklists, and to our knowledge, it is one of few studies assessing the impact of electronic checklists on the clinical assessment of referral letters [21, 25]. Studies evaluating the impact of improved referral quality on patient outcomes are also scarce [25–29].

Thomas et al. [27] achieved shorter waiting times and lower cost per patient with urinary symptoms in a mixed intervention study including a structured paper-referral checklist and guideline distribution due to more appropriate pre-referral investigations. However, no significant changes were detected in patient outcomes. Wahlberg et al. [22] showed improved referral quality after implementing referral templates for dyspepsia, colorectal cancer, chest pain and chronic obstructive pulmonary disease, but were not able to show any impact on prioritization of the patients [29].

Another observational study showed that insufficient referral information can lead to longer waiting times to see a specialist, as well as delayed diagnosis [28].

In the study of Rokstad et al. [21], the impact of electronic checklists for pulmonary diseases was assessed with a focus on improved referral appropriateness, and a significant impact of the checklist was seen on referral quality and the time spent by the consultant assessing the referrals. Whether this also influenced the clinical decisions of the consultant is not clear, but the effect of the improved referral letters on the workload of the consultant is positive.

However, a recent vignette study from Jiwa et al. [25] did not show any impact of more clinical information in the referral letter on the scheduling of the patients. Consequently, the clinical impact of the intervention was questionable. This is in accordance with the findings in the present study, where we did not find any impact of checklist referrals on the waiting times for the outpatient appointment or on the type of scheduled consultation.

The current study has several strengths. Firstly, it has been done in a standardized setting where both the generation of the referral [19] and the assessment of the referral are done on referral letter vignettes. This reduces the bias produced by patient case mix and practice settings of the gastroenterologists. In addition, the gastroenterologists were blinded to whether the referral letter was generated with the use of checklists or not through the transcription of the checklist items into the text in the general referral template. Nevertheless, the study has some limitations that we would like to address. There was a low participation rate in the survey. This was probably because we chose to invite all members of the Norwegian Association of Gastroenterology, without any prior check for the willingness to participate. The invitation e-mail included the link to the survey. We preferred this solution to avoid multiple steps with an increased risk of drop out before participation, but we did not expect a high response rate. With a response rate of 15%, there is a risk of participation bias. However, studies have shown that a low response rate does not necessarily lead to response bias and the bias associated with non-response in population studies may be negligible [30].

We also observed that the majority of participants (90%) were male gastroenterologists. This is representative for the male predominance (84%) in Norwegian gastroenterology [31].

We observed a very high overall mean score for all the referrals, and we think that the transcription of the referrals may have made them so structured and similar to each other that an effect of the checklists was hard to detect using a subjective measurement like the rating scale. In addition, referrals may have appeared to be of a more similar quality on a first glimpse, even if the objective difference in quality was larger. Using an objective score would have highlighted this issue, but in this study, the variable of interest was the gastroenterologists’ opinion, and therefore this subjective scoring was more relevant. Further, we did not provide any definition of the term ‘quality of the referral’ for the gastroenterologists; which may have caused a heterogeneous rating on the visual analogue scale.

Another weakness is the small number of referrals selected for assessment in this study. Increasing the number of referral letters for each case would likely have reduced the effect of the quality of individual referral letters and thus made the results easier to interpret and more valid. However, we chose to use only 16 referral letters and split the study into two rounds to increase the likelihood of participation. Finally, we believe that even if the standardized setting may have been an appropriate way of studying the decision making of the gastroenterologists, the patient outcome cannot be assessed in this study. A clinical study would be more appropriate to explore the impact of increased the quantity of clinical information items in referral letters on patient-related outcomes.

Conclusion

The results of the present study indicate that the use of checklists increases the subjectively assessed quality of referral letters, but the effect is limited. Increased quality may improve the working conditions of the consultant specialists who assess the referrals, and can make the task easier and less time-consuming. However, this study was unable to demonstrate that checklists significantly influence the clinical management of the patients. Whether this also is true in a clinical setting remains unanswered.

Supplementary material

Supplementary material is available at International Journal for Quality in Health Care online.

Funding

PhD student salary of S.L.E. was funded by the South-Eastern Norway Health Authority’s research grant [grant agreement no. 2008040] and the Norwegian Medical Association grant for quality and patient safety [grant agreement no. 14/1689]. C.S.R. has received funding from the European Union Seventh Framework Programme (FP7-PEOPLE-2013-COFUND) [grant agreement no. 609020—Scientia Fellows]. Researchers were independent of the funder; no funders or sponsors were involved in the design and conduct of the trial; collection, management, analysis, and interpretation of the data; preparation, review or approval of the manuscript; or decision to submit the manuscript for publication.

Author’s Contribution

S.L.E., T.dL. and L.A. designed the study. S.L.E. performed the data collection. S.L.E., C.B. and C.S.R. performed the power- and data-analysis. S.L.E. drafted the paper. All authors critically reviewed and improved it. S.L.E. is a guarantor. All authors had access to all the data and take responsibility for the integrity of the data and the analysis.

Ethics approval

The study was reported to and approved by the Data Protection Official for research. The Regional Ethics Committee considered the study outside its mandate and its approval was not required.