What's the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patients' notes, and interviews

Abstract

Objectives To assess the extent and pattern of implementation of guidance issued by the National Institute for Clinical Excellence (NICE).

Design Interrupted time series analysis, review of case notes, survey, and interviews.

Setting Acute and primary care trusts in England and Wales.

Participants All primary care prescribing, hospital pharmacies; a random sample of 20 acute trusts, 17 mental health trusts, and 21 primary care trusts; and senior clinicians and managers from five acute trusts.

Main outcome measures Rates of prescribing and use of procedures and medical devices relative to evidence based guidance.

Results 6308 usable patient audit forms were returned. Implementation of NICE guidance varied by trust and by topic. Prescribing of some taxanes for cancer (P < 0.002) and orlistat for obesity (P < 0.001) significantly increased in line with guidance. Prescribing of drugs for Alzheimer's disease and prophylactic extraction of wisdom teeth showed trends consistent with, but not obviously a consequence of, the guidance. Prescribing practice often did not accord with the details of the guidance. No change was apparent in the use of hearing aids, hip prostheses, implantable cardioverter defibrillators, laparoscopic hernia repair, and laparoscopic colorectal cancer surgery after NICE guidance had been issued.

Conclusions Implementation of NICE guidance has been variable. Guidance seems more likely to be adopted when there is strong professional support, a stable and convincing evidence base, and no increased or unfunded costs, in organisations that have established good systems for tracking guidance implementation and where the professionals involved are not isolated. Guidance needs to be clear and reflect the clinical context.

Introduction

The National Institute for Clinical Excellence (NICE), established in 1999 by the Department of Health, aims to improve standards of care for patients and reduce inequalities in access to innovative treatments.¹ NICE's technology appraisals result in guidance on the use of individual health technologies; around 50 for implementation by the NHS in England and Wales since 2000.² Hitherto, guidance on treatments was uncoordinated. This nationally coordinated programme of guidance is probably unique and represents a policy embodiment of evidence based medicine that, it is hoped, will lead to the rapid and systematic uptake of evidence based medicine into routine practice. This paper presents the results of a national evaluation examining the pattern of implementation of NICE guidance by healthcare organisations. More details on NICE and a copy of the full report can be accessed via www.nice.org.uk.

Methods

We assessed the response of the NHS to 12 pieces of “tracer” NICE guidance. We selected guidance for study if at least six months had elapsed since its release. By the time the research began in October 2001, 22 sets of guidance were eligible for inclusion (table 1): four procedures (four selected), five devices (three selected), one diagnostic test, and 11 pharmaceutical drugs (five selected). We audited 11 (50%) of the eligible sets of guidance, chosen to reflect a range of drugs, devices, and procedures; different care settings; and cost consequences. Some included clear stopping messages (wisdom teeth, laparoscopic surgery for colorectal cancer); some fairly clear messages to use a technology (implantable cardioverter defibrillators, hearing aids); and other complex messages regarding appropriate use (hip prostheses, taxanes for breast cancer, orlistat).

Table 1.

NICE guidance sampling for audit

	Title	Completed	Selected
Procedure	Wisdom teeth—removal (No 1)	Apr 2000	Yes
Device	Hips—prostheses for primary total hip replacement (No 2)	Mar 2000	Yes
Pharma	Ovarian cancer—taxanes (No 3)	May 2000	Yes
Device	Drug eluting stents (No 4)—obsolete, replaced by No 71	May 2000	No
Diagnostic	Cervical smear tests—liquid based cytology (No 5)—obsolete, replaced by No 69	Jun 2000	No
Pharma	Breast cancer—taxanes (No 6)	Jun 2000	Yes
Pharma	Dyspepsia—proton pump inhibitors (No 7)	Jul 2000	No
Device	Hearing disability—new advances in hearing aid technology (No 8)—obsolete, withdrawn	Jul 2000	Yes
Pharma	Diabetes (type 2)—rosiglitazone (No 9)—replaced by No 63	Aug 2000	No
Device	Asthma—inhalers for children under five (No 10)	Aug 2000	No
Device	Arrhythmias—implantable cardioverter defibrillators (No 11)	Sep 2000	Yes
Pharma	Glycoprotein IIb/IIIa inhibitor guidance for acute coronary syndromes (No 12)—obsolete, replaced by 47	Sep 2000	No
Pharma	Attention deficit hyperactivity disorder (ADHD)—methylphenidate (No 13)	Oct 2000	No
Pharma	Hepatitis C—interferon alfa and ribavirin (No 14)	Oct 2000	No
Pharma	Flu—zanamivir (Relenza) (No 15)—obsolete, replaced by 58	Nov 2000	Yes
Procedure	Knee joints (defective)—autologous cartilage transplantation (No 16)	Dec 2000	No
Procedure	Colorectal cancer—laparoscopic surgery (No 17)	Dec 2000	Yes
Procedure	Hernia (inguinal)—laparoscopic surgery (No 18)	Jan 2001	Yes
Pharma	Alzheimer's disease—donepezil, rivastigmine, and galantamine (No 19)	Jan 2001	Yes
Pharma	Motor neurone disease—riluzole (No 20)	Jan 2001	No
Pharma	Diabetes (type 2)—pioglitazone (No 21)—replaced by No 63	Mar 2001	No
Pharma	Obesity—orlistat (No 22) (last eligible guidance)	Mar 2001	Yes

The research consisted of three phases, each one using a different method of data collection to answer different but complementary questions.

Phase I

We analysed routine national or regional data and national surveys (centrally collected data; box 1) to assess the extent to which practice changed after publication of the tracer guidance. We used data from the NHS Prescription Pricing Authority, which covers all of England, to explore trends in the primary care prescribing of relevant drugs. We used hospital episode statistics data (covering England and Wales) to assess the trends in use of invasive procedures. We accessed or directly collected a range of other sources of national and regional data where these two sources did not have relevant or adequate data (box 1). Table 2 shows data sources used for each of the 12 sets of guidance.

Table 2.

Overview of the 12 sets of guidance selected as “tracer” guidance, research questions, and data sources

Guidance (date of publication)	Health technology (health sector)	Summary of NICE guidance	Research questions	Centrally collected data	Locally collected data
Removal of wisdom teeth (March 2000)	Procedure (hospitals and general dental practitioners)	The routine practice of prophylactic removal of pathology free, impacted third molars should be discontinued in the NHS The surgical removal of impacted third molars should be limited to patients with evidence of pathology	Has there been a reduction in the number of wisdom teeth removed in situations where there are no apparent complications?	Hospital episode statistics (HES)	Patient audits
				Dental Practice Board and the Scottish	Survey
				Practitioner Services data	Interviews
Prostheses for hip replacement (April 2000)	Procedure (hospitals and NHS Purchasing and Supply Agency)	Surgeons should use prostheses for total hip replacement that either have a demonstrable replacement rate of 10% or less at 10 years, or a minimum of three years provided that their performance is consistent with the 10 year benchmark	Are approved prostheses used in replacement operations?	Hospital episode statistics (HES)	Patient audits
			Are the numbers of approved prostheses rising and of non-approved prostheses falling?	Trent Arthroplasty Audit Group and Welsh	Survey
				Arthroplasty Audit Group Database (TWAAG)	Interviews
Taxanes for treatment of breast cancer (May 2000)	Drugs (hospitals)	Paclitaxel and docetaxel should be used for advanced cancer when previous chemotherapy has failed	Has the use of paclitaxel and docetaxel increased to NICE recommended levels?	Hospital pharmacy survey	Patient audits
					Survey
					Interviews
Taxanes for ovarian cancer (June 2000)	Drugs (hospitals)	Paclitaxel should be used after surgery	Has the use of paclitaxel increased to NICE recommended levels?	Hospital pharmacy survey	Patient audits
					Survey
					Interviews
Hearing aids (July 2000)	Devices (audiology centres and NHS Purchasing and Supply Agency)	The full range of analogue hearing aids in the current NHS range should be available at all NHS audiology centres, including binaural fitting, and reflecting patients' choice	Has there been a change in the availability of analogue hearing aids in NHS audiology centres?	A new survey of hearing aid provision sent to 228 audiology departments in England and Wales.	Patient audits
		The NHS Purchasing and Supply Agency should review the existing NHS range of analogue aids	Has the NHS Purchasing and Supply Agency reviewed the NHS range of analogue hearing aids?		Survey
					Interviews
Implantable cardioverter defibrillators for arrhythmias (September 2000)	Devices (hospitals)	The use of implantable cardioverter defibrillators for patients with specific pathologies should be routinely considered	Has there been an increase in the use of implantable cardioverter defibrillators to NICE recommended levels?	British Pacing and Electro-physiology Group ICD database	None collected
Zanamivir for influenza (November 2000)	Drugs (primary care)	Zanamivir should be prescribed only to adults at risk, who have presented within 36 hours of the onset of influenza-like illness, when influenza is circulating in the community	Has there been in an increase in the prescribing levels of zanamivir concomitant with increases of the presence of influenza-like illness in the community?	Prescribing data from Prescription Pricing Authority	None collected
				Annual flu levels from Public Health Laboratory Service bulletins
Laparoscopic surgery for the treatment of colorectal cancer (December 2000)	Procedure (hospitals)	Open rather than laparoscopic resection should be the preferred procedure for the treatment of colorectal cancer	Is laparoscopic surgery being undertaken on patients with colorectal cancer outside clinical trials?	Hospital episode statistics (HES)	None collected
		Laparoscopic surgery should be undertaken for colorectal cancer only as part of a randomised controlled clinical trial
Laparoscopic surgery for the treatment of inguinal hernia (January 2000)	Procedure (hospitals)	Open (mesh) surgery should be the preferred method of repair for primary inguinal hernia	Has there been an increase in the number of laparoscopic repairs for patients with recurrent and bilateral inguinal hernia, and a reduction for those with primary inguinal hernia?	Hospital episode statistics (HES)	Patient audits
		Laparoscopic surgery should be considered for repair of recurrent and bilateral inguinal hernia			Survey
					Interviews
Donepezil, rivastigmine, and galantamine for Alzheimer's disease (January 2001)	Drugs (hospitals and primary care)	The three drugs should be made available to people with mild and moderate	Has the use of the three drugs increased?	Prescribing analysis and cost (PACT) data	Patient audits
		Alzheimer's diseases, with scores in the mini-mental state examination above 12 points as assessed in specialised clinics		Hospital pharmacy survey	Survey
Orlistat for obesity (March 2001)	Drugs (hospitals and primary care)	Orlistat should be made available only to people who have sustained weight loss before prescription and a body mass index of 30 kg/m2 or more with no comorbidities or 28kg/m2 or more with comorbidities	Has the use of orlistat increased?	Prescribing analysis and cost (PACT) data	Patient audits
				Hospital pharmacy survey
Chemotherapy for non-small cell lung cancer (June 2001)	Drugs (hospitals)	Gemcitabine, paclitaxel, and vinorelbine should each be considered as part of initial (first line) chemotherapy. Docetaxel should be used for locally advanced cancer but only where previous chemotherapy has failed	Has the use of the four drugs increased?	Hospital pharmacy survey	None collected

We used interrupted time series analysis to assess if the pattern of practice had changed after NICE guidance³ and so infer whether the intervention had an impact. We used an autoregressive integrated moving average (ARIMA) model with dummy variables to examine the impact of publication on the growth rate and the average rate of use (constant parameter) of a technology once any growth rate had been removed.⁴

Phase II

Most NICE guidance provides criteria for appropriate use rather than a simple recommendation to use the technology or not. To assess whether the guidance was being implemented properly requires scrutiny of patient records by using methods developed for appropriateness studies.⁵,⁶ We selected a random sample of 20 (out of 221) acute trusts and their associated mental health hospitals and 21 (out of 303) primary care trusts (within which we selected a stratified random sample of five practices), from which we reviewed 50 relevant case notes for each of eight guidance topics (table 2). The resulting trusts reflected a good cross selection of geographical spread and size.

Box 1: Other sources of centrally collected data

• Dental Practice Board and the Scottish Practitioner Services data for national information on extractions of wisdom teeth in the community

• Trent Arthroplasty Audit Group and Welsh Arthroplasty Audit Group Database (TWAAG) for information on prostheses used in hip replacement. This register of knee and hip replacements covers the former NHS Trent Region and the North Wales region contained the records of 7898 patients who had received hip replacements since January 1998 in 22 hospitals (accounting for 40% of the hip replacements undertaken in these hospitals)

• British Pacing and Electro-physiology Group ICD Register for information on implantable cardioverter defibrillators for arrhythmias. The data, submitted to the register by hospital clinicians, covered the period from the first quarter of 1995 (33 centres) to the last quarter of 2001 (58 centres) and represents about 95% of activity

• A new survey of hearing aid provision sent to all 228 audiology departments in England and Wales (50% response rate)

• A new survey of all 331 hospital pharmacies in England and Wales (68% response rate and 60% with usable data)

Local audit staff agreed to extract data from patients' records by using audit proforma. Overall usable audit forms for 6308 patients were returned (table 3). We calculated the proportion of cases conforming to the NICE guidance for each healthcare organisation in the sample at two periods of time and estimated the overall average.

Table 3.

Response from audit of patient case notes with reasons for failure to complete

Guidance	No of trusts participating	No of individual completed audit forms returned	No used	Reasons for non-return
Wisdom teeth	18	892	836	Service provided by primary care trust
Hip replacement	20	990	980	All returned
Breast cancer	17	708	707	Clinicians in two trusts declined to participate. One trust did not complete as not a cancer centre
Ovarian cancer	16	521	520	Clinicians in two trusts declined to participate. One trust did not complete as not a cancer centre
Hearing aid technology	18	875	875	One service provided by primary care trusts; one trust declined to participate as pilot site for digital aids
Inguinal hernia	19	950	938	One trust did not return—no reason given
Drugs for Alzheimer's disease (mental health trusts)	17	703	583	One participating trust did not return—no reason given
Drugs for Alzheimer's disease (primary care trusts)	18	215	180	Three trusts did not return forms—no reason given
Orlistat in primary care	18	689	689	Three trusts did not return—no reason given

Phase III

We surveyed the chief executives, leads of clinical governance, and leads of clinical specialties of the 20 acute trusts that participated in the audit of patients' notes (65% response rate for chief executive officers (13 out of 20) and 57% overall (68 out of 120)) to assess their handling of NICE guidance. We also conducted semistructured interviews to access professional and managerial perspectives on our quantitative findings and, in particular, to explore the response to the NICE guidance.

We purposively selected five acute trusts⁷ that returned positive consent forms, to represent differing degrees of implementation of guidance (see phase II). In each trust we approached the chief executive, medical director, and lead clinicians for the guidance topic. We supplemented a common interview schedule (see appendix on bmj.com) by specific interview questions based on findings from the audit. We offered all interviewees face to face interviews but finally conducted three interviews by telephone. Where possible we recorded and transcribed interviews. Where recording proved impossible we returned notes to the interviewee for checking.

Data analysis was concurrent, and clear thematic categories and subcategories emerged.⁸,⁹ Two researchers working independently developed these categories and compared and reconciled differences by discussion. We used analytic matrices to examine differences between trusts and across sets of guidance.¹⁰

Results

Wisdom teeth extraction

National data indicate a sharp decline in the number of extractions between 1995 and 2001 (fig 1). Although these fell in the year in which the guidance had been published (in March 2000), we found no evidence of a change in the downward trend in extractions (-8.9 extractions per month, 95% confidence interval -57.5 to 39.6). Case note review showed that more than 90% of extractions were compliant with the guidance. Survey respondents indicated that compliance was high because the costs of implementation were low, involved a single specialty service, and had professional support and a strong evidence base.

Fig 1.

Wisdom teeth extraction activity, 1992-2001

Hip prostheses

The guidance recommended the use of prostheses with a demonstrable replacement rate of 10% or less, at 10 years, or a minimum of three years, provided that the performance of the prostheses is consistent with the 10 year benchmark. The guidance did not specify which prostheses met the benchmark and the NHS Purchasing and Supply Agency was slow to issue this information (which was subsequently withdrawn).

Use of more than 50 different prostheses was documented in the Trent and Wales register over the period 1998-2002. Of the single prostheses, 69% (3671) met the 10 year benchmark and 81% (4327) met the three year benchmark, proportions that had been declining over recent years. There is some evidence that the decline in 10 year benchmarked prostheses may have stabilised slightly after guidance (1.7 monthly increase after guidance (95% confidence interval 0.4 to 3.1) compared with a long term monthly decrease of -1.8 prostheses per month (-0.75 to -2.80)). We observed similar results from the audit of patients' notes: 66% (325), 74% (727), and 75% (734) of heads, cups, and stems, respectively, meeting the benchmarks. Interviews and the survey indicated that surgeons thought that the guidance did not acknowledge the complexity of hip surgery, and some surgeons interviewed believed that cementless prostheses would ultimately prove to offer longer service.

Taxanes for breast and ovarian cancer

We obtained usable data on taxanes from 24 hospital pharmacies (including nine cancer centres). We found a significant increase in the use of docetaxel and paclitaxel of 1112 (95% confidence interval 530 to 2222, P < 0.001) patient months and 3.7 (1.1 to 7.8, P < 0.002) patient months, respectively (fig 2), but no evidence of a change in the growth rate in the use of gemcitabine (0.5% per month, -36.2% to 37.2%) or vinorelbine (-1.0 patient months, -3.3 to1.3). This is unlikely to be due to lack of statistical power.

Fig 2.

Hospital use of paclitaxel (top) and docetaxel (bottom)

Case note review showed that, of the 707 patients identified as receiving taxanes for breast cancer, all but one were receiving this appropriately (for more advanced forms of the disease or in the context of randomised controlled trials). Interviewees acknowledged that the NICE guidance had made funding easier to obtain. The picture for ovarian cancer, however, was more varied. Of the 520 women with ovarian cancer for whom we have data, only 33% (166) were recorded as having been prescribed paclitaxel. However, oncologists interviewed believed that the guidance had overstated the effectiveness of taxanes in ovarian cancer. NICE subsequently amended its guidance.

Implantable cardioverter defibrillators

Although the number of implantable cardioverter defibrillators implanted has risen, we found no evidence of a significant change after NICE guidance had been published (fig 3). Given the small data set, the power is low to detect a change as significant, but visual inspection does not indicate any structural break. This may reflect the high costs of implantable cardioverter defibrillators, at around £20 000 ($36 000; €29) per device, competition for resources with other interventional procedures in cardiology, and scarcity of skills in electrophysiology.¹¹

Fig 3.

Total number of implantable cardioverter defibrillators implanted by quarter, 1995-2001

Hearing aids

The NICE guidance seems to have been received enthusiastically by audiology departments; all of those surveyed had undertaken an immediate audit of their service against the guidance requirements. However, review of case notes indicates that the range of analogue hearing aids offered does not seem to have been extended. Funding was described in the interviews as a major impediment to implementation. The guidance was issued at the same time as the Department of Health implemented a series of pilots of digital hearing aids, which cut across the guidance on analogue aids, which was subsequently withdrawn.

Laparoscopic surgery for primary inguinal hernia repair and colorectal cancer

Only 4% of primary inguinal hernia repairs in England and Wales were undertaken laparoscopically (contrary to NICE guidance), and this did not change after the guidance (0.3 monthly increase in hernia repairs, 95% confidence interval -5.48 to 6.08).

Although hospital episode statistics data for the 19 trusts that returned audit forms also showed 96% compliance, our audit of 545 repairs of primary unilateral hernias indicated only 65% compliance, indicating that coding of hospital episode statistics data may be unreliable. However, both national and audit data agreed that most laparoscopic procedures were concentrated in a few trusts, and that did not change over time.

Interviews showed that some local expert surgeons had the support of managers and commissioners to continue the use of laparoscopic surgery for primary repair. It was also claimed that patients often requested laparoscopic procedures.

The percentage of cases of colorectal cancer treated with laparoscopic surgery remained unchanged, at around 0.1% from 1998 to 2001.

Zanamivir for influenza

National prescribing data show little inappropriate prescribing of zanamivir in the absence of high levels of flu; prescriptions remained very low, at 499 in 2001, 190 in 2002, and 124 in 2003.

Orlistat for obesity

We found a significant increase in the average monthly prescribing of orlistat after the guidance had been published 22 per month (0.43, 95% confidence interval 15.9 to 27.8, P < 0.001; fig 4). Health authorities increased their use of orlistat, standardised by age, in the year after NICE guidance had been issued by about eight patient months per 1000 people aged 18-75 years. The variation in use, as measured by the coefficient of variation, fell from 0.4 in 2000-1 to 0.3 in 2001-2.

Fig 4.

Use of orlistat in the community

In the 689 primary care patients for whom we have a record we found evidence that the drug was not being prescribed in accordance with guidance. Only in 12% of cases (n = 83) were there data showing compliance in the three key areas of age, body mass index, and weight loss. However, data recording was poor, and data were missing in one or more fields in 80% (551) of returns, particularly in respect of weight and weight loss. In those 308 cases where patients' weight loss prior to the visit had been recorded fully, the weight loss criterion was not met in 40% (123) of cases. Of patients with recorded weight loss data, only 41% (127) and 25% (40) still being prescribed orlistat at 3 months and 6 months, respectively, had reduced their weight as advised in the NICE guidance. The rise in prescribing of orlistat therefore does not necessarily imply a rise in appropriate prescribing.

Drugs for Alzheimer's disease

Total use grew logarithmically since February 1999 (fig 5). Once this had been taken into account, the growth rate at the time of publication of guidance (-1.4% per month; 95% confidence interval -4% to 1.2%) did not increase significantly. However, the data are also compatible with a view that an increase in trend occurred shortly before the guidance was formally issued. Variability between health authorities decreased; the coefficient of variation fell from 1.1 to 0.97.

Fig 5.

Total use of Alzheimer's drugs in the community

Based on the audit data from 583 usable forms, compliance with the five recommendations in the guidance at first prescription varies between 52% and 85% for mental health organisations and 21% and 46% in primary care. Compliance with the recommendations at follow up is low, mainly because information regarding mini-mental state examination scores, follow ups, or the presence of relatives at assessment was not routinely recorded (a requirement of the guidance).

Discussion

Principal findings

The evidence that NICE guidance has made a difference either to the quality of care or to variations in practice is mixed. Some NICE guidance has been associated in time with changes in prescribing. Use of orlistat and taxanes grew rapidly after NICE guidance had been published, and uptake of drugs to treat Alzheimer's disease also increased, although this slightly preceded the release of that guidance. The guidance for wisdom teeth was published during a long downward trend in the extraction rate and did not have a discernable additional effect. The guidance for hips in general showed no effect. We found no evidence of a change in the number of implantable cardioverter defibrillators used, the numbers of inguinal hernias or colorectal cancers treated laparoscopically, or expansion in the range of hearing aids made available. In some cases audit showed that clinical practice was highly compliant with the indications for treatment laid out in the guidance (for example, wisdom teeth and the use of taxanes for breast cancer), but compliance was low for some (such as orlistat) and more variable for others. Some trusts seemed to exhibit more consistent compliance than others across a range of guidance, and box 2 shows their characteristics, although all identified funding as a major issue, especially where infrastructure costs were high.

Strengths and weaknesses of the study

In a retrospective observational study we cannot fully assess the impact of NICE guidance because of the absence of the counter-factual and because NICE guidance is just one, although potentially important, factor influencing professional practice. We can only observe whether clinical practice is consistent with the guidance. This research was based on a large data gathering exercise, and although data taken from national routine hospital and primary care are complete, those obtained from hospital pharmacies were less so. Where data were recorded, the audits allowed us to assess more reliably whether the details of practice corresponded to the indications of appropriateness in the guidance compared with subjective surveys¹² and analyses based simply on routine data.¹³ It was, however, sometimes difficult to design audits where the original wording of the NICE guidance had been complex or ambiguous. This is also likely to affect the ease with which the guidance is implemented. Our survey also offered a view of the management context into which the guidance was received, and the interviews were useful in highlighting the managerial, financial, and clinical perspectives on implementation.

Meaning of the study

The establishment of NICE as a mechanism for institutionalising evidence based health care was a unique initiative. This institutional response by itself is, however, not sufficient for the rapid and universal implementation of evidence based health care. This is unsurprising; NICE guidance is being issued at a time of great change in the NHS, with new structures, competing high level priorities, funding deficits, and staff shortages. In addition, healthcare organisations are complex, containing strong professional bureaucracies.¹⁴ Change is therefore heavily dependent on the actions of groups of professionals and individual clinicians.¹⁵ The ability to manage change in an organisation is further complicated by the increasing importance of networks that span several organisations.

The diffusion of innovations literature indicates that the adoption of guidance would depend on several other factors (box 3; after Rogers).¹⁶,¹⁷ These also emerged in the thematic analysis of the interview data (table 4). In areas where practitioners and managers see advantages to adoption, where the value is hardly disputed (clear evidence), and where there is professional endorsement (taxanes for breast cancer, wisdom teeth extraction, and orlistat for obesity) practice has changed relatively fast. The marketing activity of the pharmaceutical industry should not be overlooked as a possible explanation for the apparent increased uptake of drugs over devices and changes in surgical behaviour.

Table 4.

Analysis of interviews: factors influencing likelihood of implementation

		Positive tendency towards implementation		Negative tendency towards implementation
Theme	Subtheme	Influencing factors	Illustrative quotes	Influencing factors	Illustrative quotes
Trust culture		Committed to implementing NICE guidance	“Locally NICE advice is seen to be binding....”	Not viewed as priority No implementation pressure from centre	“The clinician in me says that NICE is actually important because it's about the treatment and care the patient gets. The pressures in the service from the centre are about finance and waiting lists; they're not about NICE.”
Locality decisions	Structures	Responsibility for NICE guidance vested in health community	“The management of NICE guidance is not just a trust affair—it is managed within the whole health economy. We have a NICE implementation group, chaired by a primary care trust public health director.”	Left to individual clinicians	“Here, it has traditionally been left to individual clinicians.”
	Priorities	Topic high on priority list	“As an economy we are signed up to NICE, but the politics are of priorities and choice. By and large NICE outweighs other issues.”	Topic low on priority list	“There are lots of such initiatives in the NHS—trust managers were supportive in principle but just couldn't make the money available. They did not refuse—the decision was deferred until the next financial year.”
Systems for managing guidance	Robust	Regular reports on compliance to Clinical Governance Committee. Adequate staffing of clinical governance function	“We have a 10 stage implementation model that seems to work very well.”	No tracking Small trusts with limited resources in clinical governance function	“Our clinical governance function consists of the medical director and one other person plus a couple of part time audit clerks.”
	Proactive	Early identification of topics being considered by NICE Identification of local implications and funding requirements	“We proactively hunt out guidance pertaining to the directorate. Additionally, there is someone responsible for NICE guidance implementation in the clinical effectiveness department who asks each department head about the management of appropriate guidance.”	Guidance dealt with on publication	“When the guidance comes in we send it to the appropriate directorate.”
	Audited	Regular audits of compliance	“We have limited resource to audit, so we've tended to audit where we know we've not been complying.”	No audit of compliance	“We have been asked whether or not we comply, but nobody from the Trust has looked through our records as far as I am aware.”
Funding		Sufficient funding identified for all revenue consequences including infrastructure	“Financially we have taken a hard line—no money no implementation—on drugs at any rate.”	Commissioner argument that money is already in the baseline	“Because money has not been hypothecated in the proper fashion, we end up with arguments between government regions, strategic health authorities, primary care trusts, and ourselves around where the money sits and we end up with this recurrent thing, saying it's in the baseline.”
Consultant buy in	Perceived robustness of evidence	Guidance consistent with other sources of evidence, royal college guidelines, etc	“No different from what we're already doing. We'd already looked at it, and I remember it coming out and discussing it, and people saying it was along the same lines as what we'd already done....”	Local evidence at odds with NICE assessment. Belief that the evidence base does not justify the guidance.	“This area has pioneered laparascopic surgery, and I think that's one of the only NICE guidances where we have varied from them. We actually wrote to NICE at the time, stating our reasons for doing so, on the basis that we had additional expertise and there was a deliberate policy decision taken both within the trust.”
	Consultation	Contributed to guidance or views properly represented	“We, particularly in this department, had a very active role in drawing up the guidelines... way back in the mid to late 1990s we were already disseminating guidelines on wisdom teeth.”	Guidance viewed as biased, or failing to take key factors into account	“By and large the perception is that this particular advice was `London centred,' and getting NICE to change their minds is a difficult task.”
	Clinical freedom	Guidance is mandatory	“Even though we have not agreed with advice from NICE either on laparoscopic hernia repair or laparoscopic colorectal surgery, there has been no suggestion locally that we should `plough our own furrow.' ”	Guidance is guidance only	“I don't use guidance; I use my own clinical experience.”

Box 2: Features of trusts consistent with high compliance

• Commitment to managing process of implementing guidance

• Identification of lead clinician at point of NICE announcement of topic for review

• Proactive assessment of local costs and implications of implementation

• Responsibility for funding and implementation vested in locality-wide group

• Strong clinical governance function appropriately resourced

• Culture of consensus

• Recognition of legitimacy of NICE

• Involvement of clinicians in guideline process

• Financial stability

• Expectation that compliance is mandatory, subject to identification of funding

• Targeted audit of areas of non-compliance

Box 3: Factors influencing adoption of innovation (after Rogers¹⁶)

• Perceived attributes and consequences of adoption (for example, relative advantage, complexity, observability)

• Type of innovation decision (optional, collective authority)

• Communication channels (for example, media, interpersonal, professional)

• Nature of the social system (norms, degree of interconnectedness of networks, concentration of opinion leaders)

• Extent of promotion efforts by agents of change

If the evidence has been disputed, or the costs not covered by increased income, adoption is more variable. Where practice is complex and also depends on an interaction with a practitioner's skills (such as with hip replacements or laparoscopic surgery), more formative research to understand the clinical context of practice may have helped to produce more influential guidance. Adoption may also be influenced by the degree to which decisions rest with an individual or requires team or organisational agreement. Individual surgeons are likely to have a high degree of professional autonomy, and this may partly explain the variability in techniques for hernia repair and hip replacement.

The extent to which trusts are prepared for NICE guidance and have put in place structures and processes to manage their implementation was variable.

The degree of active promotion by NICE is likely to have some impact on adoption, although probably not directly proportional to the effort invested. The greatest effect is likely when opinion leaders including the professional bodies and associations adopt and promote the guidance.

What is already known about this topic

Research on the implementation of guideline implementation has been summarised, but its relevance to this unique national initiative was unknown

The implementation of NICE guidance has not been evaluated overall. Previous work has been limited to single health technologies

What this study adds

Some clinical practice has changed in line with NICE guidance, in particular around prescribing (for example taxanes and orlistat)

Other technologies have been adopted in line with NICE guidance but continued pre-existing practice patterns

There is evidence that NICE guidance has been less influential in surgical procedures and use of medical devices

Routine data are not sufficient to assess compliance with guidance; this needs review of case notes

NICE guidance seems to have had an uneven impact on the uptake of evidence based medicine. This impact is likely to be greater if more effort is devoted to clarity of the guidance and its relevance to practice; adequate funding provision; getting professional support; and encouraging healthcare organisations to set up formal mechanisms for handling guidance

Unanswered questions and future research

Our research covers the early period of NICE guidance; it would be interesting to see if the response to subsequent guidance is different, particularly given the recent attention that NICE is giving to implementation. More research would also be useful to understand the professional and organisational responses to evidence based guidance better and to evaluate the relative contributions of various implementation strategies to practice patterns.¹⁷

Conclusions

NICE guidance has been associated with uptake of some technologies, although this has been variable. Implementation is likely to be improved if the guidance is clear and based on an understanding of clinical practice, if the evidence is strong and relatively stable, if adequate funding is available, and if the guidance is supported and disseminated by professional bodies. Trusts should institute strong supportive internal systems for handling guidance and gathering data on implementation.