The utility of an online diagnostic decision support system (Isabel) in general practice: a process evaluation

Abstract

Objectives

To evaluate the utility of Isabel, an online diagnostic decision support system developed by Isabel Healthcare primarily for secondary medical care, in the general practice setting.

Design

Focus groups were conducted with clinicians to understand why and how they used the system. A modified online post-use survey asked practitioners about its impact on their decision-making. Normalization process theory (NPT) was used as a theoretical framework to determine whether the system could be incorporated into routine clinical practice.

Setting

The system was introduced by NHS County Durham and Darlington in the UK in selected general practices as a three-month pilot.

Participants

General practitioners and nurse practitioners who had access to Isabel as part of the Primary Care Trust's pilot.

Main outcome measures

General practitioners’ views, experiences and usage of the system.

Results

Seven general practices agreed to pilot Isabel. Two practices did not subsequently use it. The remaining five practices conducted searches on 16 patients. Post-use surveys (n = 10) indicated that Isabel had little impact on diagnostic decision-making. Focus group participants stated that, although the diagnoses produced by Isabel in general did not have an impact on their decision-making, they would find the tool useful if it were better tailored to the primary care setting. Our analysis concluded that normalization was not likely to occur in its current form.

Conclusions

Isabel was of limited utility in this short pilot study and may need further modification for use in general practice.

Background

Decision support systems, in the form of software or online programmes, are increasingly being used to assist with problem solving in areas with complex structures and large knowledge bases. Diagnostic decision support systems (DDSSs) in health care are designed to be interactive and aid in decision-making, rather than to provide the ‘right’ answer to a particular problem. Systematic reviews have shown that they can improve practitioner performance, but the evidence for their effectiveness in terms of patient impact and quality of care shows that these effects are limited.^1–4

Isabel is an online DDSS that aids clinicians by providing a list of 10 possible diagnoses based on the clinical features of the case in question. The clinician can access the system at any time, and is prompted to enter details about the patient (age, sex, pregnancy status and geographical region) and a minimum of one clinical feature. It is supported by customized components, such as web resources, protocols and guidance notes (www.isabelhealthcare.com). It incorporates a post-use survey, which asks the clinician about his or her experience of the tool and can be used to generate evidence for the clinician's appraisal portfolio.

The effectiveness of Isabel has been the subject of a number of evaluations. In specialist settings Isabel decreased unfavourable outcomes, decreased diagnostic omission errors and increased the number of ‘correct’ diagnoses.^5–11 Two studies using the post-use survey in primarily specialist settings found a majority of users reported that it was useful and offered diagnoses not considered before use.^12,13 However, a postal survey of primary and secondary care clinicians found that it was not considered useful or convenient by 90% of users.¹⁴

The introduction of the Isabel system within the British National Health Service (NHS) is relatively recent, and its applicability to the primary care setting has yet to be established. Diagnostic decision-making in primary care differs from that in secondary care, in that general practitioners (GPs) develop an initial hypothesis of the diagnosis, which they then refine through further questioning, examination and investigation before arriving at a final diagnosis. This process may also take place over time and through more than one consultation.¹⁵ In 2010, the Isabel system was made available to a limited number of general practices as part of a three-month pilot by NHS County Durham and Darlington in the UK to see whether it might have utility in the general practice setting. We sought to determine its uptake and impact on clinical decision-making and patient management and to elicit users’ views of its utility.

Methods

Pilot design

All 85 practices in NHS County Durham and Darlington were invited to participate in a three-month pilot of the Isabel system. Participating practices were provided with training in its use by Isabel Healthcare and NHS County Durham and Darlington, and were encouraged to run test searches in the system to further their training.

Each practice was encouraged by the Primary Care Trust (PCT) to identify a champion for use of the system during the period of the pilot, however no incentives to use the system were provided by the PCT.

Study design

We obtained anonymized extracts from the clinical records of those patients for whom searches were conducted. We used the patient's unique practice identifier and date of consultation to link data extracted by the Isabel system with practice record data.

We conducted focus groups of participants and non-participants. Focus groups encourage participants to share their experiences with each other, which can stimulate a fuller development of ideas and perspectives. This dialogue can also trigger one's memory, which was particularly important to this evaluation as practitioners were expected to recall past experiences. We offered one-to-one interviews, either over the phone or in person, as an alternative if focus groups could not be arranged with all practice staff involved. Focus groups and interviews were facilitated by EH and audio recorded.

An online post-use questionnaire, designed and pretested by the research team, appeared on the system immediately after a search was conducted in Isabel. It asked four questions on decision-making related to that search (Box 1).

Box 1.

Online post-use survey

1. How did Isabel PRO help with your differential diagnosis?

   1. Confirmed my differential

   2. Broadened my differential

   3. No impact

2. How did Isabel PRO influence your decision to run diagnostic tests (blood tests, scans, xrays)?

   1. Initiated a test

   2. Confirmed a test

   3. Cancelled a test

   4. Changed a test

   5. No impact

3. How did Isabel PRO influence your decision to prescribe medication?

   1. Initiated a prescription

   2. Confirmed a prescription

   3. Cancelled a prescription

   4. Changed a prescription

   5. No impact

4. How did Isabel PRO influence your decision to refer the patient?

   1. Initiated a referral

   2. Confirmed a referral

   3. Cancelled a referral

   4. Changed a referral (e.g. different clinic/consultant/specialty)

   5. No impact

Analysis

We determined Isabel's role in patient management and clinical decision-making by comparing the searches conducted with the associated clinical record. The cluster of clinical features that had been entered for each search was assigned to a clinical category by a practising general practitioner, GR, to identify the types of problem for which searches had been conducted.

A conventional qualitative content analysis was applied to the focus group data.¹⁶ Audio recordings were listened to and field notes were read several times, and then coded into categories. Categories were generated inductively in order to develop a theory from the data.¹⁷ Coding was rechecked for consistency.

Data from the post-use survey were imported into Excel 2007 and frequencies for each response were calculated.

Normalization process theory (NPT) was used as a theoretical framework to assess whether and how well the system had been embedded in everyday practice, based on the evidence available from the focus groups and post-use survey.¹⁸ NPT can be used to understand how clinicians adopt new technologies into their working practice. It proposes that practices become routinely embedded within social contexts as a result of individual and collective work to implement them (May and Finch 2005). Implementation is operationalized through the four generative mechanisms of coherence, cognitive participation, collective action and reflexive monitoring, and factors (immediate and organizing components) promote or inhibit routine embedding or normalization.

Handling of data

Informed written consent was obtained from each practice for participation in the evaluation and from individual clinicians who participated in the focus groups. We obtained the necessary PCT and Isabel Healthcare permissions to download responses to the online post-use questionnaire. All data were treated in accordance with the Data Protection Act (1998).

Results

All 85 practices in NHS County Durham and Darlington were invited to take part in the three-month pilot and, through purposive selection by NHS County Durham and Darlington from those practices willing to participate, Isabel was made available to seven general practices across a range of practice size and geography. A reminder notice was sent to all participating practices, encouraging them to use the system, midway through the pilot period. Figure 1 shows the use of the system over the course of the pilot.

Figure 1.

Use of Isabel over three-month pilot

Practice characteristics and usage are shown in Table 1. All seven practices accessed the system to run test searches (data not reported here); however, data were only eligible for analysis if searches were made for a specific patient (by using the patient practice number). Four practices made 16 patient specific searches. In two cases, a practitioner conducted multiple searches for the same patient (2 in 1 case, 3 in the other), leaving a total of 21 searches conducted. The remaining three practices did not participate, and did not provide reasons for this.

Table 1.

Practice summary

Practice	No. patients for whom advice was sought	No. general practitioners during pilot	No. registered patients during pilot	No. patients seen during pilot
A	0	2	3195	2209
B	9	5	3506	1563
C	0	3	5642	2797
D	0	7	10,223	N/A
E	2	5	7479	4992
F	1	7	10,063	7283
G	5	8	15,952	10,196
Practice average	3	5	8009	4840

N/A data not made available by practice

The most frequent searches were for adult males aged 50 years and over, followed by searches for younger children. On average, each search took three minutes and 55 seconds (SD ± 3:06) to complete (n = 15). Data on duration of searches were not made available by the system in four cases.

Analysis of clinical records

The limited use of Isabel meant that any impact of its use on referral rates and investigations was too small for meaningful quantitative analysis. Instead, we illustrate the diagnostic process by comparing the features entered and corresponding diagnostic output from the system (n = 21) with the features, actions taken and diagnoses noted in the clinical record (Table 2).

Table 2.

Process of diagnostic decision-making in general practice using Isabel

Clinical category	Features entered into clinical record	Features entered into Isabel	Differential diagnosis (Isabel)	Action	Diagnosis (identified by Isabel?)
Abdominal	N/A	Right hypocondrium abdominal pain, tiredness	Ischaemic heart disease, non-Hodgkin's lymphoma, neoplasms of the kidney, CMV colitis, colorectal cancer, Whipple disease, pancreatic neoplasms, adrenal neoplasms, endocarditis, inflammatory bowel disease	N/A	N/A
		Right abdominal pain, tiredness, bursting	Whipple disease, mycotic aneurysm, diphtheria, HIV/AIDS, depression, pemphigus, acute appendicitis, non-Hodgkin's lymphoma, thoracolumbar spinal injuries, multiple sclerosis
		Right hyperconial abdominal pain, tiredness	Ischaemic heart disease, non-Hodgkin's lymphoma, neoplasms of the Kidney, CMV colitis, colorectal cancer, Whipple disease, pancreatic neoplasms, adrenal neoplasms, endocarditis, inflammatory bowel disease
CVS	Swelling of legs, bilateral	Bilateral leg swelling, overweight	Hypothyroidism, sleep apnea, Cushing's syndrome, ischaemic heart disease, pulmonary thromboembolism, SLE, chronic venous insufficiency, deep vein thrombosis, glomerulonephritis, myocarditis	FBC, U&E, LFT, ESR, CRP	Iatrogenic due to calcium channel blocker drugs (No)
		Bilateral leg swelling, overweight, rheumatoid arthritis	Endocarditis, pericarditis, intestinal bypass arthritis, hypothyroidism, Cushing's syndrome, neutrophilic dermatoses, sleep apnea, palindromic rheumatism, SLE, coccidiodomycosis
Endocrine	Tiredness, poor appetite, feels hot and cold	Fatigue, anorexia, weight increase, forgetfulness	Urinary tract infection, bipolar disorders, depression, diabetic nephropathy, glomerulonephritis, hypothyroidism, ovarian neoplasms, paraneoplastic neurological syndrome, renal failure, COPD	TFT, thyroid antibodies LFT, FBC, CRP, blood glucose, antinuclear factor, PHQ9	Hypothyroid (Yes)
Gastro-intestinal	Epigastric pain, dizzy, relief with alginate	Jaw pain, heartburn, epigastric discomfort	Ischaemic heart disease, hypothyroidism, aortic aneurysm/dissection, gastroesophageal reflux, peptic ulcer disease, pancreatic neoplasms, optic neuropathy, acute visual loss, facial injury/fractures, trigeminal nerve disorders	ECG gastroscopy	Oesophagitis (Yes)
Musculoskeletal	Foot pain, bilateral, dorsal	Foot pain, bilateral, dorsal, recent	Charcot's joint, CNS TB & TB meningitis, cellulitis, diabetic lower limb disease, diabetic neuropathy, endocarditis, flatfoot disorders, foot fractures, metatarsal fractures, morton neuroma	Referral	Osteoarthritis (No)
	Fatigue, dizziness, generalized paraesthesia, headache	Numbness, headache, tired	Migraine, transient ischaemic attack, Lyme disease, herpes zoster, megaloblastic anemias, head injury, interstitial nephritis, iron deficiency anaemia, lymphoma, multiple sclerosis	FBC, U&E, LFT, TFT, blood glucose, bone profile, B12, folate	OA spine? Vertical collapse? (No)
	Neck pain, off balance	Neck pain, dizziness	Arterial aneurysms/dissection, atlantoaxial instability, cervical spondylosis, migraine, brain neoplasms, atlantoaxial dislocation, bacterial meningitis, bartonella infection, cerebral sinus venous thrombosis, cervical plexopathy	None	Cervical spondylitis (Yes)
	Leg pain	Leg pain, limping	Megaloblastic anaemias, Factor V Leiden, leukaemia, meningococcal disease, arachnoid cysts, autoimmune hemolytic anemias, femoral fractures, intervertebral disk herniation, malignant bone tumours, Marfan syndrome	Referral	N/S
	n/a	Hip pain, limping	Hip arthritis, pelvic injuries, fractures of the hip, avascular necrosis of the femoral head, coccygeal fractures, herpes zoster, hip dislocation, hip trauma, ilopsoas bursitis, sickle cell disease/crisis	N/A	N/A
Neurological	Headache, worse on coughing	Headache, numbness	Transient ischaemic attack, Lyme disease, migraine, polyneuropathy disorders, arachnoid cysts, megaloblastic anaemias, adrenal neoplasms, head injury, lymphoma, brain neoplasms	Emergency admission	Subarachnoid haemorrhage? (No)
Non-specific	Abnormal weight loss	Weight loss	Neoplasms of the kidney, lung abscess, pancreatic neoplasms, non-Hodgkin's lymphoma, Whipple disease, gastritis, urinary tract infection, Hodgkin's disease, adrenal neoplasms, hyperthyroidism	PSA, FOBT	Anxiety? (No)
	Raised temperature, intermittent	Temperature control, Parkinson's disease, swings	Urinary tract infection, Alzheimer's disease, non-Hodgkin's lymphoma, Parkinson's disease, progressive supranuclear palsy, borderline personality disorder, bacterial meningitis, bipolar disorders, CNS TB & TB meningitis, candidal infection	FBC, CRP, TFT	N/S
	Mass in groin, pain	Swelling groin, painful, groin lump	Genital herpes, genital ulcer syndrome, arterial aneurysms/dissection, Haemophilus ducreyi infection, syphilis, lipoma, lymphoma, proctitis, biliary pain, colorectal cancer	Referral	Abscess (No)
		Lump, painful	Uterine neoplasms, pulmonary thromboembolism, benign bone tumours, neutrophilic dermatoses, psychosomatic illness, soft tissue neoplasms, epidermoid cyst, head and neck neoplasms, oral cancer, peritonsillar abscess/quinsy
		Lump, painful, groin lump	Uterine neoplasms, pulmonary thromboembolism, benign bone tumours, neutrophilic dermatoses, psychosomatic illness, soft tissue neoplasms, epidermoid cyst, head and neck neoplasms, oral cancer, peritonsillar abscess/quinsy
Respiratory	Oesophegeal spasm, cough, throat tightening	Chest discomfort, spasmodic, spasm, globus	Ischaemic heart disease, dystonia disorders, Sturge–Weber disease, lung neoplasms, esophageal neoplasms, progressive supranuclear palsy, pulmonary thromboembolism, supraventricular tachycardia, adult-onset basal ganglia disease, anthrax	C×R	COPD (No)
	Dyspnoea cough	Breathlessness	Iron deficiency anemia, sarcoidosis, ARDS, heart failure/CHF, ischaemic heart disease, asthma, myocarditis, pulmonary AV fistula, aspiration syndromes, pericardial effusion/tamponade	FBC	Asthma (known) (Yes)
Skin	Intermittent, erythema of lower limbs	Raynauds, redness skin, intermittent, erythema	Dermatomyositis/polymyositis, paronychial inflammation of the nail, autoimmune hepatitis, testicular torsion, parvoviral infections, gonococcal infection, Lyme disease, urticaria/angioedema, allergic rhinitis, intestinal fistulas and abscesses	Discussed with immunologist	N/S

FBC, full blood count; LFT, liver function test; ESR, erythrocyte sedimentation rate; CRP, C-reactive protein; ECG, electrocardiogram; SLE, systemic lupus erythematosus; PSA, prostate-specific antigen; COPD, chronic obstructive pulmonary disease; FOBT, fecal occult blood test

N/S Not stated in clinical record; N/A Clinical record not made available

Clinical records data show that the majority of patients had multiple co-morbidities. The most frequent clinical categories were musculoskeletal and non-specific, followed by abdominal and respiratory. In 11 cases, a diagnosis was noted in the clinical record. Based on the features entered into the system, Isabel only identified four of these, and missed seven. There were 13 cases where a referral or investigation was recorded in the clinical notes; however it was not possible to attribute these to the Isabel output or otherwise.

Post-use survey

Practitioners in all of the four actively participating practices completed at least one online post-use survey, for a total of 10 of the 21 searches conducted. The majority of responses indicated that Isabel did not have an impact on any aspects of diagnostic decision-making, including differential diagnosis, diagnostic tests, prescription of medication or referrals. Five users reported that Isabel helped to either broaden a differential (n = 4) or confirm an intention to refer (n = 1).

Focus groups and individual interviews

We conducted two focus groups, one each at Practice B and Practice G. Because only four practices used the system, we could only invite these to the participant focus groups. Practice E could not arrange a meeting for a focus group with its entire staff, and while one-to-one interviews were offered as an alternative, a suitable time to hold the interview could not be found, and it did not take place. We had no reply from the fourth participating practice.

We sought to arrange group or one-to-one interviews with practitioners in the three practices that did not use the system, but were unable to either obtain a response or agree a suitable time.

There were five GPs present for the focus groups at Practice B. Only one, who was the most senior partner of the practice, had accessed the system. Three were new members of staff and had not received the training. The fifth, a practice partner, had attended the training, but reported that she did not find cause to ever use the system. The practitioners stated that they did not ‘remember’ to use the system, and weekly reminders that Isabel was an option for them to use would have helped. At the same time, this practice also reported that they did not appoint a champion for the pilot as advised by the PCT.

For the focus group with Practice G, there were no GPs present, and only one senior nurse practitioner and the practice manager. The nurse practitioner informed us that the GPs agreed to have her trial the programme on their behalf. She remembered to use the system, but reserved it for specific circumstances, for example in cases that were complex or not responding to treatment. She felt it would have been more beneficial as a training tool.

Practitioners from both practices reported that, although the diagnoses produced by Isabel in general did not have an impact on their decision-making, they might find it more useful if it were tailored to the primary care setting and which they could then use if and when necessary.

Assessment of implementation using NPT

Table 3 summarizes the evidence for each of the factors and components that should be considered in applying NPT.¹⁸ Most of the evidence indicates that normalization was unlikely to occur. The exceptions were for the components involving expertise, where practitioners felt competent using the system, and resources, where practitioners had easy access to the system.

Table 3.

Model exploring reasons for non-use using normalization process theory

Endogenous (professional) factors	Exogenous (organizational) factors
Interactional workability	Skill-set workability
1. Congruence: normalization is likely if the actors have a shared belief in the process Lack of use by most practitioners suggests that there was doubt about the value of the tool: ‘I attended the training and thought (Isabel) would be quite useful but not to use on every patient. I didn't really get around to using it because you don't really see many of those patients in that time frame’ (1, Senior female GP) Doctor/patient relationship: ‘I'd be a bit concerned about the patients’ perception of entering information into a system that gives you an answer. You hear patients say, “Oh I went to see my GP and they looked at a book”’ (2, Junior female GP). 2. Disposal: normalization is likely if the actors have a shared belief in the goals Practitioners agreed that a decision support system could be useful, but not Isabel as it is currently configured. ‘[Isabel] comes up with the bizarre…not your common garden stuff you see in general practice’ (3, Senior nurse practitioner)	5. Allocation: normalization is likely if actors’ responsibilities are agreed within contexts Only the practice managers agreed to participate, not clinicians Use of the system was at the discretion of the practitioner Roles were not agreed ‘It would have been good for [the new trainee]….It is more placed as a learning tool…I don't think the GPs would have used it. They would have done it once and then thought…’ (3, Senior nurse practitioner) 6. Performance: normalization is likely if the level to which actors perform is agreed within contexts No agreement at a practice level: At practice G all GPs agreed that the practice nurse should be the sole person to trial the system.
Relational integration 3. Accountability: normalization is likely if actors have the necessary expertise Training provided was enough to use the system 4. Confidence: normalization is likely if actors believe it falls within their remit The system is not more knowledgeable than they are: ‘With Isabel you're putting data in and you're not giving the mind the exercise it needs. What you've been trained in medical school is going up against that.’ (4, Junior male GP)	Contextual integration 7. Execution: normalization is likely if resourcing issues are agreeable between/within contexts All practices had sufficient IT resources Time was not adequately resourced, e.g. no extra time was allotted for consultations 8. Realization: normalization is likely if organizational systems between/within contexts are minimally disrupted Searches took 4 minutes within a 10 minute period for consultations: ‘We have so many things thrown at us…the PCT telling us to do this and that you can get a little overwhelmed.’ (5, Senior male GP)

In respect of professional (endogenous) factors, practitioners shared a belief in the utility and process of DDSS systems generally, but considered that Isabel itself was not configured adequately for primary care. There was also doubt about the tool itself, indicated by a lack of belief that the system fell within the practitioners’ remit.

At the organizational level (exogenous factors), there was allocational conflict, as agreement to participate had been primarily by practice managers, while its use was to be by clinicians. As a result, roles were not agreed. This was linked to a major issue of no time having been identified for clinicians to use the system during a consultation.

Discussion

In this study of a short-term pilot of the Isabel diagnostic decision support software we found it had limited utility in the UK general practice setting. Not all practices participating in the study used the system and, in those that did, very few clinicians accessed it. There was evidence from the clinical records that outputs from Isabel were associated with clinical decisions in some cases. The main finding of user focus groups was that the current format of Isabel would need modification for use in general practice.

This is one of the first studies of Isabel in a primary care setting in the UK. It was a pragmatic study done in the context of routine general practice. Adoption should be accompanied by interventions to increase uptake; in this instance one reminder was given with short-term effect. The three-month pilot was short in comparison to other published studies of decision support systems. These can take some time to reach ‘steady state’ usage, though we did not see evidence of increasing uptake over the study period.

An important weakness of the study is the small amount of data available for analysis. Low participation in evaluations of poorly designed interventions is a common finding, and in the case of Isabel the low participation observed was probably due to both poor implementation of the tool and its limited utility.¹⁹ The limited funding from the PCT for the initiative meant that only seven general practice sites could be recruited. Coupled with their very low usage rates, this meant the effect on important outcomes of direct relevance to diagnostic decision-making, such as specialist referral and use of diagnostics, was so small as to rule out quantitative analysis. It was also difficult to get health-care professionals to participate in the qualitative interviews, and recruiting non-participants into interview studies is a recognized challenge in qualitative research.

Previous experimental studies of Isabel have been undertaken in controlled clinical contexts, and have used outcome measures such as rates of diagnostic omission errors, unfavourable outcomes and ‘correct’ diagnoses, using more senior clinicians to validate decisions and diagnoses.^5–11 It would be methodologically challenging to take the same approach in a pragmatic general practice study and with the case-mix seen by GPs. One observational study in general practice, published in abstract only, reported the usage by 25 GPs over a six-month period. Although 335 queries were entered, only 49 post-use surveys were completed. In 29% of these, Isabel was reported to have modified a referral decision and in 40% it prompted additional test(s).²⁰ In our study we also found a discrepancy between queries entered and surveys completed. This appeared to be explained by problems with the programme itself or with its output for a given query.

In a previous study, users reported that Isabel offers diagnoses not considered beforehand.¹³ However other studies have found that it was not useful or convenient due to the structure of NHS service delivery, specifically the lack of system-wide internet availability and time allocated by practices to clinicians to learn and use the tool.¹⁴ In our study, users had access to the internet, but appeared not to think of using the system. The average search in this study lasted nearly four minutes compared with less than two minutes in other studies.^5,6 The longer time required may reflect the range of symptom complexes being searched on, or a lack of familiarity with the system given the three-month duration of the pilot.

In general practice the concept behind Isabel may also presuppose a level of diagnostic certainty that is unrealistic. In up to 40% of patients it is not possible to apply a diagnostic label.²¹ Howie²² described ‘the relative rarity of fully developed hospital illness’ in general practice, and Jones et al²³ have described the danger of spurious and erroneous diagnostic precision. Furthermore, and in distinction to secondary care, diagnostic decision-making in primary care is a complex process that often extends over time, while the causes of misdiagnosis are not limited to cognitive oversight.^15,23 Nevertheless, Isabel might have greater utility in general practice if it were to be configured to more closely to mirror the ways that problems present in primary care, and our findings indicate that practitioners would be receptive to diagnostic support of this sort.

NPT is a sociological model which focuses on the dynamic processes that lead to innovations becoming embedded in everyday work. We used the construct of collective action from NPT to structure the interpretation of our findings.²⁴ Collective action is the operational work that people do to enact a new practice, such as the use of a new technology. We found that most components of this construct were not fulfilled, making it unlikely that Isabel would be integrated into practice in its current form.

Conclusions

Isabel as it is currently configured is unlikely to be incorporated into clinical practice in the primary care setting. There is a need for rigorously designed trials of Isabel in primary care that are adequately powered to determine impact on decision-making. Outcome measures should include measures of patient safety, diagnostic accuracy and health-care resource utilization. Its introduction into primary clinical care by primary care organizations should await or be in the context of such research.

DECLARATIONS

Competing interests

None declared

Funding

The research was funded through an infrastructure grant from NHS County Durham and Darlington.

Ethical approval

The Sub-Ethics Committee of the School of Medicine and Health at Durham University provided ethical approval. The County Durham NHS ethics committee deemed this study to be an evaluation not requiring NHS ethical approval.

Guarantor

EH

Contributorship

EH and GR designed the study and analysed the results. EH conducted the fieldwork. EH and GR prepared, read and approved the final manuscript

Acknowledgements

We thank Wendy Stevens at NHS County Durham for coordinating with the practices and providing useful feedback. We also thank Natalie Cronje at Isabel Healthcare who provided technical support for the Isabel system. Finally, we are grateful to all practices that participated in the pilot and especially those that participated in our focus groups.

Reviewer

Peter Manser