Evaluation of risk assessment tools for suspected cancer in general practice: a cohort study

Abstract

Background

Diagnostic delay is deemed to account for an estimated 5000 to 10 000 extra cancer deaths each year in the UK. Many cancer patients do not have symptoms meeting national referral criteria for rapid investigation. Risk assessment tools (RATs) have been developed to assist GPs in selecting patient for cancer investigation.

Aim

To assess the usability and acceptability of lung and colorectal RATs, as well as subsequent resource use and cancer diagnoses.

Design and setting

Cohort study with nested qualitative study with 614 GPs from 165 practices in seven English cancer networks were provided with RATs applicable to patients aged ≥40 years with bowel or respiratory symptoms. In-depth interviews were conducted with 34 individuals (11 project managers and 23 GPs).

Method

The study measured the number of RATs used, and subsequent cancer investigations and diagnoses, over a 6-month period and compared these with the previous 6 months.

Results

A total of 2593 RATs (1160 lung, 1433 colorectal) were completed. Compared with the preceding 6 months, there were 292 more chest X-rays, 104 extra 2-week chest clinic appointments, and 47 additional diagnoses of lung cancer. For suspected colorectal cancer, there were 304 more 2-week referrals, 270 more colonoscopies, and 10 more cancers identified. RATs appeared to help GPs in their selection of patients for cancer investigation. Users reported that RATs helped to confirm a need for investigation as well as allowing reassurance when investigation was not needed.

Conclusion

Use of RATs in primary care was accompanied by increased diagnostic activity and additional cancer diagnoses.

INTRODUCTION

The UK has poor cancer outcomes when compared with most other European countries, with an estimated additional 10 000 deaths annually.^1,2 Much of this is attributed to diagnostic delay, which can reflect later presentations for medical attention by symptomatic patients, or delays after presentation. Nine out of 10 patients with cancer present with symptoms, and most present to primary care.³ Initiatives to expedite diagnosis include urgent referral pathways (established in Spain and Denmark as well as the UK); guidance on selection of patients for urgent referral or investigation (in Denmark and the UK),⁴ and, in the UK, the creation of the National Awareness and Early Diagnosis Initiative (NAEDI) in 2008.⁵ A criticism of urgent referral guidance has been that it describes common high-risk presentations of cancer, neglecting patients who present with ‘low-risk-but-not-no-risk’ symptoms.⁶ Such patients have longer times to diagnosis and worse mortality.⁷

A possible solution is to develop clinical decision support instruments to help GPs identify patients for investigation. One such instrument is the Risk Assessment Tool (RAT), derived from primary care cohort studies and validated on separate patient cohorts.^8,9 These give risk estimates for patients aged >40 years presenting to primary care with symptoms of possible cancer, for single symptoms, pairs of symptoms and repeat attendances with the same symptom. The values are colour-coded to aid interpretation. An example is shown in Figure 1. A small pilot evaluation of a similar scoring system had been encouraging, suggesting that interested GPs would be motivated to use RATs.¹⁰ The RATs were designed to be an adjunct to National Institute for Health and Clinical Excellence (NICE) guidance, which described moderately high-risk symptom profiles warranting urgent investigation.⁴

Figure 1.

Risk Assessment tool (RAT) for colorectal cancer⁶.

The study evaluated the utility and acceptability of RATs for suspected bowel and lung cancer, measuring their use, their incorporation into the clinical encounter, and their effect on resource use and new cancer diagnoses. The study was conducted in two phases: a quantitative and qualitative phase.

METHOD

Cohort study

All 28 English cancer networks were offered participation, from which seven were selected to ensure a broad geographical spread. Each network identified a GP cancer lead and recruited local practices, to which RATs were supplied in two formats; as mousemats and desktop flipcharts. The electronic version of the paper shows all three RATs, along with the instructions on their use, exactly as supplied to the participants. Lung and bowel RATs were selected, as both cancers are common, and each has an excess annual mortality of nearly 1000 deaths.¹¹

How this fits in

The UK has poor cancer outcomes, some of which is blamed on delays in diagnosis. Decision support systems have been developed to help clinicians in primary care select patients for investigation of cancer, but they have not been studied. GPs are willing to use decision support systems for lung and colorectal cancers and usage is accompanied by an increase in diagnostic activity with additional cancers identified.

Data for the evaluation were collected using an A4 pad that contained all three RATs (lung having one for smokers and one for non-smokers) on a single page. No specific recommendations were given about what level of cancer risk should be investigated. This was deliberately left to the individual GP’s discretion. The study defined investigation for cancer as a 2-week referral to the appropriate speciality, or a chest X-ray (CXR) in possible lung cancer. Cancer networks collected resource use and outcome data from practices and local trusts for the two 6-month periods before and after the distribution of RATs.

Data analysis used simple descriptive statistics, with non-parametric analyses where appropriate: The study used Stata, (version 11). The evaluation was designed to provide 1600 completed RATs; this would have 95% power (5% 2-sided alpha) to identify a change in investigative behaviour from 5% to 10% of patients (colorectal) and 89% power for a change from 20% to 30% (lung smokers). The evaluation was not powered to identify change in the secondary outcomes of resource use, cancer diagnoses or staging.

Qualitative interviews

GP cancer leads in the participating networks, together with GP users from a purposeful sample of practices were approached to take part in in-depth telephone interviews to ascertain their perceptions and experiences of using the RATs in practice. The study selected participants to include areas of affluence and poverty, and practices with non-white patient populations ranging from 2–50%. Interviews were transcribed verbatim and a systematic qualitative methodology based on the Framework method of analysis was applied to the data.¹² Close reading of each interview transcript and accompanying contextual and de-briefing notes was used to generate codes and cross-checking of the transcript coding between researchers was conducted.

RESULTS

A total of 614 GPs from 165 practices were recruited. They completed 2720 RATs, of which 127 were subsequently excluded (60 had been used in patients <40 years; 67 had not identified the reported symptoms), leaving 2593 for analysis. Figure 2 shows RAT use over time. RATs were used in more females than males (1413 females (54.5%); 1162 males (44.8%); 18 (0.7%) missing data; 1007 RATs (38.8%) were used in patients aged ≥70 years.

Figure 2.

RAT usage during the evaluation.

The commonest symptoms for which the 1433 colorectal RATs were used were: diarrhoea (588 (41%), of which 422 (72%) had a second documented symptom) and rectal bleeding (462 (32%), of which 275 (60%) of had a second documented symptom). For the 1160 lung RATs, the commonest symptoms were cough (832 (72%), of which 456 (55%) had a second documented symptom, and dyspnoea (280 (24%), of which 266 (95%) had a second documented symptom). For 702 patients in whom a colorectal RAT was used, a 2-Week Wait referral was made, with a further 337,¹¹ being either admitted as an emergency or having an urgent (n = 144) or routine (n = 182) referral to an appropriate speciality. A CXR was recorded for 684 (59%) patients in whom a lung RAT was completed: 10 others were admitted as an emergency, while 149, 32, and 55 patients had a 2-week, urgent, or routine referral to respiratory medicine, respectively. Some of these last three categories may have had a preceding CXR which the GPs omitted from the data collection form. The proportions being investigated greatly exceeded the estimated in the sample size calculations. The median (interquartile range [IQR]) risk estimate, expressed as a percentage risk of an underlying cancer, was 2.4% (IQR 1.5–3.4) in those selected for investigation, and 1.1% (IQR 0.6–1.8) in those not investigated: P<0.001, Mann Whitney test.

GPs gave their own personal estimate of the risk of cancer as higher, lower, or the same as the RAT estimate, with 12% considering the risk to be higher, and 13% lower. If the GP considered the ‘true’ risk to be higher, they were more likely to investigate (odds of investigation 3.5: P<0.001), when compared to the group who estimated the risk to be the same. However, if GPs estimated the ‘true’ risk to be lower, they were no less likely to investigate (odds of investigation 0.93; P = 0.59).

Resource use and cancer diagnoses

These data are summarised in Table 1. One cancer network found it impossible to obtain the data for resource use, and a second network had similar problems for 13 of their practices. New lung cancer diagnoses increased from 127 in the 6 months before the evaluation to 174 during the pilot; a 37% rise. The proportion of stage 1 and stage 2 cancers combined (as a proxy for potential cure) increased by 19% (from 26 to 31). New colorectal cancers increased by 76% (from 134 to 144), a 7% rise. No significant change in staging was seen (data not shown).

Table 1.

Investigations for colorectal and lung cancer before and during the evaluation

		2-week referrals for lung,n		GP requested CXR (for ages <40 years),n		2-week referrals for colorectal,n			Colonoscopies (for ages <40 years),n
Cancer Network practices (n)	Lung RATs	6 months before	During	6 months before	During	Colorectal RATs completed,n	6 months before	During	6 months before	During
NC London (19)	42	24	33	818	1015	114	95	144	246	324
SW London (14)	345	34	30	563	548	244	94	112	178	187
Sussex (42)	206	106	151	1549	1742	333	406	457	646	903
Three Countiesa (25)	40	29	37	1247	1204	175	148	156	233	259
York & Humber (25)	347	69	119	2294	2134	285	231	356	219	112
North Trent (26)	133	70	66	960	1080	248	199	252	240	247
NE Londona (14)	86	–	–	–	–	122	–	–	–	–
Total	1199	332	436	7431	7723	1521	1173	1477	1762	2032
Increase		31%		4%			26%		15%

Only 12 of the 25 practices in the Three Counties Cancer Network returned data on referrals and investigations for analysis in both periods. None was returned from North East London. CXR = chest X-ray. NC = north central. NE = north east. RAT = risk assessment tool. SW = south west.

Qualitative interviews

Interviews were conducted with 23 GP responders who had personal experience of using the tool in practice: the study reports findings on how RATs were used to aid decision making related to referral (Box 1). To varying degrees RATs affected GPs’ referral thresholds and in turn, their decision making; for example, in several instances the tool was perceived to give more credence to a decision to refer that had already been made. For some responders, the tool urged referrals that may not have been made. At other times, the tool was used to confirm decisions not to refer.

Box 1. Illustrative quotations from the interview study

Referral thresholds

‘Mid-project we looked at our numbers and we felt that under the clause, would you have referred this patient if you hadn’t been using the risk assessment tool, there was a significant minority that said, you know, the tool had pushed them to a different decision … it was 10–15% of people that may have waited longer if they hadn’t had the tool.’ (GPL/2)

‘I think our referral thresholds for lower GI have definitely gone down.’ (GPL/3)

‘I’d say particularly it got us thinking about patients with COPD, because, um, there’s a bit in the, ah, in the lung tool which is smokers with COPD saying they automatically should have a referral for a chest X-ray, and that made us think about how frequently we should do chest X-rays in our COPD patients.’ (GPL/8)

Questioning of practice

‘It did make us all [in the practice] think though that we should be a bit more careful with people that we put … where we put rectal bleeding down to anal, um, conditions such as piles and things like that; you know, when you should really say, no, enough is enough and you should refer this patient.’ (GPL/8)

‘It reinforced key ideas in there about combination of symptoms and key symptoms, and it has certainly helped to stimulate debate at the practice about … patients with anaemia, for example … I think as an educational tool for a team, it’s also very useful … when people actually study the detail they do gain knowledge from it and so it does confer a little bit more confidence in what to do in certain situations.’ (GP/6)

Raising awareness of symptoms

‘Although I wasn’t sure it changed much of how I practice, um, it did prompt probably a few more referrals for multiple presentations of the same thing, like abdominal pain … there were one or two new things that, um, most GPs really weren’t that aware of … different combinations of symptoms.’ (GPL/7).

‘It is quite a useful checklist of symptoms, it reminds you to ask the patients the right things that are going to help you to make a better diagnosis or make a better decision about whether to refer them or not.’ (GPL/1)

Decision making

‘If I had a patient with a vague set of symptoms then finding and using the tool showed that it was an amber, for instance, would just perhaps, while not encouraging a referral there and then, I might have followed up the patient in a different way, um, and so made sure that I kept in touch with them … we [as a practice using the tool] were actually more proactive in saying, actually, I’d like to see you again, just to see how these symptoms are, um, rather than leaving it to the patient to contact us.’ (GPL/4).

‘Where someone’s come back a couple of times and you think, oh, I’ve safety-netted there, should I have done? But you go back and reflect on it and then think, actually, I don’t know that I should and, you know, phone the patient back and say, ‘actually, on reflection, I think I’ll send you’. So it’s a minority but I think in some cases where, perhaps you were safety-netting rather than referring, and then you have a little reflect and think, oh no, I will refer. So it’s a minority, but I think it is helpful.’ (GPL/3).

‘A lot of early cancer diagnosis is just, you know … good basic basics … always ask that question in the back of your mind … could this be cancer? and then disprove it in your clinical assessment.’ (GPL/6).

The role of diagnostic aids in consultations

‘I don’t think you can ever protocolise or … make a risk schedule that is better than, than looking at experience.’ (GP/11).

‘I think they all (risk assessment tools) have their place in the total overall assessment of the patient and probably none of the tools are … of course, none of them are infallible, um, but I think it helps inform the decision. It does help.’ (GP/12).

‘This tool could be very useful … [GPs] know the history of a patient, they know that that patient used to smoke, they know that patient is generally unwell … they know the patient perhaps would be comforted by going to have a test or not … not to send a patient because of the tool is a danger … it must be an aid, and not a rooted instruction.’ (PM/4).

‘What I teach my GP trainees is that actually you rely on your gut instinct to refer … and so we do, but that’s where I could find this a being more of a kind of, an algorithm that would help you with … patients that you feel they need the 2-week rule but you haven’t got quite the diagnostic criteria that fits the criteria for the 2-week rule referral.’ (GP/4).

Several interviewees were surprised at the symptoms the RATs listed and some queried why particular symptoms/combinations of symptoms should alert a referral, but the way in which the RATs led GPs to question given ways of practice was, however, repeatedly described as positive, particularly in relation to referral thresholds. The tool thus encouraged a deeper level of thinking about symptom presentations and raised awareness of symptom patterns and combinations, thereby promoting actions such as ensuring patients returned earlier for review.

The qualitative data confirm that the RATs brought about changes to GP practice. That said, although the tools were perceived overall to be a valuable aid to diagnosis, responders’ comments indicated the tools did not override clinical judgement, nor supersede other guidance; rather they were an adjunct to the diagnostic process.

DISCUSSION

This evaluation showed that clinical decision support tools (RATs) for lung and colorectal cancer are used by GPs to help select patients for investigation and/or referral. Use of RATs was accompanied by an increase in cancer investigations and urgent referrals and more cancers were diagnosed.

Strengths and limitations

This evaluation has several limitations, the most important being that it was a before and after design. The timetable for implementation by the National Cancer Action Team (NCAT) of RATs for lung and colorectal cancer was driven by the need to address poor cancer outcomes, the desire to pilot what appeared a promising tool and to align the intervention to the first national public awareness campaign. As a result, the study cannot know to what extent the changes in investigative behaviour are due to RAT usage, though the changes in the outcome measures suggest there has been some effect, and the evidence from the qualitative data support the argument that behaviour did change. Some referrals clearly would have happened anyway, such as emergency admissions (RATs were not designed to facilitate these), and other patients may have been investigated at a later date, and the RAT has simply expedited this. There is an apparent disparity between the internal evaluation figures showing — for example — 702 2-week colorectal referrals, yet the external data identifying an increase of 304 such referrals. This can be explained by several possibilities: the most likely being that many of the RAT-based referrals would have been made anyway, or perhaps made later. Secondly, the external data were collected at a practice level, so included any GPs in participating practices who did not personally use the RATs. Thirdly, seasonal changes may have affected the incidence of bowel and respiratory symptoms, prompting changes in referral numbers. The evaluation took place in the spring and summer of 2011 while the comparison data came from the autumn and winter of 2010/2011. It was also disappointing two networks had difficulties with external data collection, although there is no reason to believe this introduced bias. Finally, the study did not collect data on patients in whom no RAT was performed, so the percentage of total consulting patients who had a RAT performed, or whether this differed between the sexes, cannot be estimated.

Against this, the evaluation was large and well geographically spread, and is likely to be representative of English general practice. The qualitative study helped the study understand what was happening in consultations to produce the extra CXR and 2-week referrals.

Comparison with existing literature

Almost one-half of patients in whom a colorectal RAT was used were subsequently referred under the 2-week system, with another quarter having alternative (but relevant) referrals. Even more lung patients were investigated, with nearly 60% having a CXR. Those not being X-rayed were of lower risk, and presumably too low in the GP’s view to warrant investigation. Interpretation of this depends upon whether GPs were using RATs on every patient aged >40 years with a bowel or respiratory symptoms, or whether they were being selective in using the RATs in patients deemed to be at higher risk of cancer. The study did not collect data on symptomatic patients who had no RAT completed. It is likely there were many of these, as approximately 2.6% of GP consultations have a bowel symptom recorded in the GP records.¹⁰ Furthermore, isolated symptoms documented on the RATs were in the minority. Together this suggests GPs were using RATs in the more complex presentations of symptoms, which is their purpose; data from the interviews support this interpretation.

The increase in use of diagnostic resources was moderate: approximately 270 additional colonoscopies. As there is no simple diagnostic test for colorectal cancer for use in primary care, suspicion of the cancer has either to be followed by review in primary care with referral on persistence or worsening of symptoms (so-called ‘safety netting’), or proceeding without delay to imaging of the bowel, usually by colonoscopy, or perhaps CT colonography.¹³ More is spent in England on diagnosis of colorectal cancer than on treatment: reflecting the number of colonoscopies required to identify one cancer.¹⁴ Ten additional cancers were identified during this evaluation. The study know that 270 more colonoscopies were performed during the study, but do not know if all the ‘extra’ colonoscopies were in patients in whom a RAT was used. If, for simplicity, it can be assumed they were, then the yield of these ‘extra’ colonoscopies was approximately 4%, which is similar to the risk estimates for many of the symptomatic presentations on the RAT. In other words, the figures are about what could be expected. It will require health-economic analyses to determine if such a yield is cost-effective.

In contrast, the lung results are better than would be expected. The investigation of possible lung cancer in primary care is very different from that of possible colorectal cancer. GPs have easy access to CXRs, which have good performance in terms of sensitivity and specificity. A 2-week referral generally follows an abnormal CXR, although as there are a small percentage of false-negatives CXRs in lung cancer,¹⁵ national guidance recommends referral without an abnormal CXR in a few circumstances.⁴ The external data suggest that around 300 additional CXRs had been taken: for these to have identified 47 new cancers is unrealistic. Seasonal issues may be relevant. Respiratory symptoms are much more common in the winter, so the ‘true’ increase in CXR usage may have been higher: certainly the GPs reported taking at least 684 in patients who had entered the study, although some of these may well have been requested even if RATs had not been used. Even so, the considerable increase in new cancer diagnoses (some of which were of an early stage, and potentially curable) is in the right direction, although very unlikely to be wholly a result of introduction of RATs.

Implications for research and practice

There is considerable interest in the use of clinical decision support tools to aid earlier diagnosis of cancer. The study shows that RATs encourage GPs to think about referral thresholds, prompt them to investigate and may lead to earlier diagnosis. Increased resources are being given to English cancer diagnostics in 2012;¹⁶ the results suggest that RATs have a role in ensuring the efficient use of both this resource and specialist referral, thus supporting the quality, innovation, productivity and prevention agenda of the NHS.¹⁷

Use of RATs may bring a mortality benefit, although the size of this is difficult to quantify. Nevertheless, the National Cancer Action Team, through cancer networks, has distributed colorectal and lung RATs to all English GPs, following unpublished interim results from this evaluation. As well as its potential clinical benefits, this dissemination represents an excellent opportunity to study whether diagnostic tools are assisting GPs to investigate for possible cancer, although once again observational methods will have to be used.

Funding

The evaluation was funded by the National Cancer Action Team.

Ethical approval

This was a service development evaluation so ethical approval was not required.

Provenance

Freely submitted; externally peer reviewed.

Competing interests

All authors have declared no competing interests.

Discuss this article

Contribute and read comments about this article on the Discussion Forum: http://www.rcgp.org.uk/bjgp-discuss