Abstract
Objectives
To identify prescribing indicators based on prescribing analysis and cost (PACT) data that have face validity for measuring quality or cost minimisation.
Design
Modified two round Delphi questionnaire requiring quantitative and qualitative answers.
Setting
Health authorities in England.
Participants
All health authority medical and pharmaceutical advisers in the first round and lead prescribing advisers for each health authority in the second round.
Main outcome measures
Face validity (median rating of 7-9 on a nine point scale without disagreement) and reliability (rating 8 or 9) of indicators for assessing quality and cost minimisation.
Results
Completed second round questionnaires were received from 79 respondents out of 99. The median rating was 7 for cost minimisation and 6 for quality, and in all except four cases individual respondents rated indicators significantly higher for cost than for quality. Of the 41 indicators tested, only seven were rated valid and reliable for cost minimisation and five for quality.
Conclusion
The 12 indicators rated as valid by leading prescribing advisers had a narrow focus and would allow only a limited examination of prescribing at a general practice, primary care group, or health authority level.
Introduction
Quality of care within the NHS is a seminal focus of government policy. This focus on quality has driven the development of new organisational structures such as the National Institute for Clinical Excellence and the national performance framework to measure progress in six areas of health care.1,2 In addition, clinical governance structures are being put in place to provide “a framework through which primary care groups will be accountable for continuously improving the quality of their services.”1
Prescribing indicators for general practice have been used in the NHS for over two decades3 and are likely to have a central role in the clinical governance activities of many primary care groups. This is because prescribing continues to grow at about 9% a year4 and two thirds of all general practice consultations generate a prescription.5 The national performance framework described cost effective prescribing as an important element of the “effective delivery of appropriate healthcare.”2
Prescribing is a controversial area of quality assessment.6 Previous research has highlighted the importance of critical approaches to prescribing,6,7 defining and measuring the appropriateness of prescribing,8–10 variations in prescribing across general practices,11 adherence to standards,12 and the role of prescribing analysis and cost (PACT) data in general practice.13 Few validated quality indicators exist for prescribing in the public domain.6,7 Avery et al concluded that “further research is needed into the development and use of indicators based on PACT.”11 The Prescribing Support Unit has developed a set of indicators based on PACT data. It advocates their use as a starting point when comparing the performance of health authorities or primary care groups with that of other authorities or groups or when comparing prescribing among general practices to identify outliers or those which are more likely to benefit from interventions to modify behaviour. PACT data are comprehensive, universal, and timely but they are not combined with diagnoses, data on specific patients, or any outcome measure. Projects are underway that aim to link prescribing and clinical data in order to produce quality indicators.
Anecdotal evidence suggests that prescribing indicators are more appropriately related to cost than quality, particularly at the practice level (the unit of analysis for most prescribing indicators). We report the findings of a two round Delphi consultation14,15 that sought to identify which of the most commonly used prescribing indicators in the United Kingdom are face valid and reliable indicators of quality or cost minimisation. The questionnaire included only indicators of drug use that could be derived from PACT data and basic demographic features of practice populations.
Participants and methods
A list of 31 prescribing indicators was generated from on two main sources: prescribing indicators with evidence of face validity in a previous Delphi consultation7 and, most importantly, prescribing indicators used at the time of the survey by the Prescribing Support Unit.3
In May 1999 we sent the first questionnaire of a modified two round Delphi consultation to every pharmaceutical and medical adviser in England (n=305). Respondents were asked to rate each indicator against two continuous 1 to 9 integer scales: “Is this indicator a useful measure of cost minimisation?” and “Is this indicator a useful measure of quality?” Definitions of these two constructs were provided in the covering letter and questionnaire sent to respondents. Cost minimisation was defined as “using the lowest cost preparation with no adverse effect on benefit” and quality as “optimisation of health/well-being for the whole practice population.” Respondents were also asked to state whether they currently used each indicator. The questionnaire invited respondents to comment on each of the 31 indicators.
No indicators were discarded between rounds, but 10 indicators were added; two indicators related to statins and eight were minor variations on indicators used in the first round and were based on comments received in that round. The second round questionnaire therefore contained 82 ratings (41 each for cost minimisation and quality).
Participants who were sent the second round questionnaire were given three types of feedback from the first round for each indicator included in both rounds: a frequency distribution of scores (on scales of 1 to 9), a median (face validity) score for both scales, and qualitative comments (figure). Qualitative comments made during round one were transcribed and summarised. We included comments that illustrated the negative and positive attitudes expressed in the first round to provide contextual information on which respondents could base their ratings. We did not feed back to respondents their previous score.
After obtaining comments from a wide range of medical and pharmaceutical advisers (n=154) in the first round, we used the second round to achieve consensus among respondents at the health authority level. Second round questionnaires were sent in July 1999 to the lead prescribing adviser at each health authority in England (n=99) in order to obtain one single health authority return. Respondents were asked to rate each indicator using the same method as in the first round. Non-responders received one follow up mailing.
The validity data presented in this paper are based on second round median ratings only.16 We used a rating scale based on the RAND appropriateness method.16 Indicators with an overall median rating of 7, 8, or 9 without disagreement were rated face valid; indicators rated with an overall median of 1-3 and 4-6 were rated as invalid and equivocal respectively. Disagreement was defined as 30% or more scores in both the bottom (1-3) and top (6-9) tertile.17 Previous research has found that indicators rated 8 or 9 without disagreement are also reliable because higher rated indicators are more likely to be reproduced by a different panel of the same stakeholders rating the same set of indicators.18 Indicators rated with an overall median of 8 and 9 were therefore considered face valid and reliable.
Scores were analysed by using SPSS with non-parametric tests (Wilcoxon's z test) to examine whether indicators were significantly more likely to be rated valid for cost or quality.
Results
Completed second round questionnaires were received from 79 respondents out of 99 (response rate of 79%). Overall median ratings for the complete set of 41 ratings in the second round were calculated for cost minimisation and quality. These were calculated from individual medians not from the raw scores. The median rating was 7 for cost minimisation and 6 for quality. Table 1 shows that 17 indicators were rated higher for cost, 16 higher for quality, and eight were rated identically. Overall, there was no significant difference in ratings for cost or quality (Wilcoxon's z =−0.76, P=0.45). However, in all except four cases individual respondents rated indicators significantly higher for cost than for quality (by Wilcoxon's z test). The four exceptions were ratio of compound diuretics items to all diuretic items; ratio of antibiotic items for co-amoxiclav or 4-quinolones to the number of items for all antibiotics; percentage of total net ingredient cost on drugs of limited clinical value; and percentage of non-steroidal anti-inflammatory drug items from ibuprofen, diclofenac, and naproxen.
Table 1.
Second round validity ratings for cost minimisation and quality
| Cost ratings indicator (n=79) | Cost validity score | Quality ratings indicator (n=79) | Quality validity score |
|---|---|---|---|
| Generic prescribing rate (%) | 8 | % of antibiotic items contained in predefined list (health authority, primary care group, or practice formulary) | 8 |
| Potential generic savings as % of total drug expenditure | 8 | DDDs benzodiazepines/benzodiazepine STAR-PU (including zopiclone and zolpidem) | 8 |
| Antibiotic generic prescribing rate (%) | 8 | Ratio of co-trimoxazole items to trimethoprim items | 8 |
| β blocker generic prescribing rate (%) | 8 | Items/STAR-PU for antibiotics | 8 |
| % of total NIC of modified release NSAID preparations* | 8 | Ratio of bendrofluazide 2.5 mg items to all bendrofluazide items | 8 |
| NIC/DDD for ulcer healing drugs | 8 | % of NSAID items from ibuprofen, diclofenac, and naproxen | 7 |
| Overall prescribing cost/ASTRO-PU (excluding high cost and specialist drugs) | 8 | Ratio of compound diuretics items to all diuretic items | 7 |
| Cost/DDD of inhaled corticosteroids | 7 | % of antibiotic items in the practice's top 10 antibiotic items | 7 |
| Ratio of compound diuretics items to all diuretic items | 7 | No of items for appetite suppressants/patient | 7 |
| % of antibiotic items contained in predefined list (health authority, primary care group, or practice formulary) | 7 | No of items for cough suppressants or nasal decongestants/patient | 7 |
| Ratio of No of items for co-amoxiclav or 4-quinolones to No of items for all antibiotics | 7 | Ratio of No of items for co-amoxiclav or 4-quinolones to No of items for all antibiotics | 7 |
| Ratio of No of items for 4-quinolones to No of items for all antibiotics | 7 | Ratio of No of items for 4-quinolones to No of items for all antibiotics | 7 |
| NIC/item for antibiotics | 7 | No of items for peripheral and cerebral vasodilators/patient | 7 |
| % of total NIC on drugs of limited clinical value† | 7 | % of total NIC on drugs of limited clinical value† | 7 |
| % of total NIC on modified release preparations* | 7 | % of NSAID items from ibuprofen, indomethacin, diclofenac, and naproxen | 7 |
| % of total NIC on brand named combination products‡ | 7 | DDDs/STAR-PU for ulcer healing drugs | 7 |
| % of total NIC on combination products‡ | 7 | DDDs/STAR-PU for oral NSAIDs | 7 |
| % of total NIC on compound analgesics‡ | 7 | % of total NIC on compound analgesics‡ | 7 |
| % of NSAID items from ibuprofen, indomethacin, diclofenac, and naproxen | 7 | % of total NIC on combination products‡ | 6 |
| % of NSAID items from ibuprofen, diclofenac, and naproxen | 7 | % of total NIC on modified release preparations* | 6 |
| NIC/DDD for oral NSAIDs | 7 | % of total NIC of modified release NSAID preparations* | 6 |
| DDDs/STAR-PU for ulcer healing drugs | 7 | % of total NIC on brand named combination products‡ | 6 |
| % of ulcer healing DDDs from proton pump inhibitors | 7 | DDDs inhaled corticosteroids/inhaled corticosteroid STAR-PU | 6 |
| NIC/month of hormone replacement therapy | 7 | No (range) of antidepressants prescribed which comprise 80% of all antidepressant prescribing | 6 |
| Overall prescribing cost/ASTRO-PU | 7 | NIC/DDD for ulcer healing drugs | 6 |
| DDDs/STAR-PU for oral NSAIDs | 6 | % of ulcer healing DDDs from proton pump inhibitors | 6 |
| Items/STAR-PU for antibiotics | 6 | No of months of hormone replacement therapy/woman aged 45-64 years | 6 |
| % of antibiotic items in practice's top 10 antibiotic items | 5 | Items of lipid lowering drugs/patient aged 45-75 | 6 |
| No of items for peripheral and cerebral vasodilators/patient | 5 | Antibiotic generic prescribing rate (%) | 6 |
| % of total NIC on SSRIs | 5 | Generic prescribing rate (%) | 5 |
| No (range) of antidepressants which comprise 80% of all antidepressant prescribing | 5 | Potential generic savings as % of total drug expenditure | 5 |
| No of items for cough suppressants or nasal decongestants/patient | 4 | β blocker generic prescribing rate (%) | 5 |
| DDDs benzodiazepines/benzodiazepine STAR-PU (including zopiclone and zolpidem) | 3 | Cost/DDD of inhaled corticosteroids | 5 |
| Items of lipid lowering drugs/patient aged 45-75 years | 3 | NIC/item for antibiotics | 5 |
| No of items for statins/1000 patients | 3 | NIC/DDD for oral NSAIDs | 5 |
| No of months of hormone replacement therapy/woman aged 45-64 years | 3 | Overall prescribing cost/ASTRO-PU | 5 |
| DDDs inhaled corticosteroids/inhaled corticosteroid STAR-PU | 3 | Overall prescribing cost/ASTRO-PU (excluding high cost and specialist drugs) | 5 |
| Ratio of bendrofluazide 2.5 mg items to all bendrofluazide items | 2 | No of items for statins/1000 patients | 5 |
| No of items for appetite suppressants/patient | 2 | Ratio of benzodiazepines to antidepressants | 4 |
| Ratio of benzodiazepines to antidepressants | 2 | % of total NIC on SSRIs | 4 |
| Ratio of co-trimoxazole items to trimethoprim items | 1 | NIC/month of hormone replacement therapy | 4 |
ASTRO-PU=age sex temporary resident originated prescribing unit, DDD=defined daily dose, NIC=net ingredient cost, NSAID=non-steroidal anti-inflammatory drug, SSRI=selective serotonin reuptake inhibitor, STAR-PU=specific therapeutic groups age sex related prescribing unit.
Modified release preparations: ibuprofen, diclofenac, indomethacin, etodolac, flurbiprofen, ketoprofen, naproxen, tiaprofenic acid, propranolol, verapamil, isosorbide dinitrate, isosorbide mononitrate, salbutamol tablets.
Drugs of limited clinical value (British National Formulary code): antidiarrhoeals (code 1.4), peripheral vasodilators (excluding thymoxamine) and cerebral vasodilators (2.6.4), cough preparations (excluding methadone and diamorphine) (3.9), systemic and topical nasal decongestants (3.10 and 12.2.2), appetite suppressants (4.5), bitters and tonics (9.7), topical antirheumatics (10.3.2), anti-infective preparations (excluding mupirocin and chlorhexidine/neomycin) (12.2.3), antiseptic lozenges and sprays (12.3.3), topical circulatory preparations (13.14).
Branded name combination products of following generic combination products: co-amilofruse, co-flumactone, furosemide (frusemide) and potassium chloride, bumetanide and potassium chloride, co-amilozide, triamterene and chlorthalidone, triamterene and benzthiazide, triamterene and furosemide (frusemide), triamterene and hydrochlorthiazide, bendrofluazide and potassium chloride, co-codamol 30/500 (codeine phosphate 30 mg and paracetamol 500 mg), paracetamol 500 mg and dihydrocodeine 20 mg.
No indicators were rated with an overall median of nine. Twenty five indicators were rated face valid for cost minimisation and 18 for quality. Of these, nine were rated valid for both (table 1). Although the remaining indicators were all rated as equivocal quality indicators, nine were rated as invalid for cost minimisation. No indicators were rated invalid for quality. Twelve indicators were rated reliable, seven for cost minimisation and five for quality.
Only two of the indicators rated valid and reliable for cost or quality in this study were currently being used by over 50% of the sample (table 2). These were generic prescribing rate and ratio of bendrofluazide 2.5 mg items to all bendrofluazide items.
Table 2.
Use of indicators rated valid and reliable to assess performance
| % use* | |
|---|---|
| Cost | |
| Generic prescribing rate (%) | 97% |
| Potential generic savings as % of total drug expenditure | NA |
| Antibiotic generic prescribing rate (%) | 37% |
| β blocker generic prescribing rate (%) | 17% |
| % of total NIC of modified release NSAID preparations | NA |
| NIC/DDD for ulcer healing drugs | 23% |
| Overall prescribing cost/ASTRO-PU (excluding high cost and specialist drugs) | NA |
| Quality | |
| Ratio of bendrofluazide 2.5 mg items to all bendrofluazide items | 55% |
| % of antibiotic items contained in predefined list (health authority, primary care group, or practice formulary) | NA |
| Items/STAR-PU for antibiotics | 32% |
| DDDs benzodiazepines/benzodiazepine STAR-PU (including zopiclone and zolpidem) | 32% |
| Ratio of co-trimoxazole items to trimethoprim items | 13% |
Use calculated from data obtained in first round Delphi consultation. ASTRO-PU=age sex temporary resident originated prescribing unit, DDD=defined daily dose, NIC=net ingredient cost, NSAID=non-steroidal anti-inflammatory drug, STAR-PU=specific therapeutic groups age sex related prescribing unit, NA=not available (indicator not rated in first round).
Discussion
Clinical governance is designed to be a means by which health organisations at different service levels can maintain and improve quality of care. Prescribing will constitute a key clinical governance objective of many primary care groups. This study aimed to discover how many of the most frequently used prescribing indicators are rated valid by advisers responsible for managing prescribing in health authorities in England. Our findings suggest that advisers believe that prescribing indicators based on PACT at the population level are less valid for quality than for cost minimisation.
Thirty three of the 41 indicators rated in the second round were found to be face valid for either cost (n=25) or quality (n=18) or both (n=9). However, only 12 indicators were also rated reliable—seven for cost and five for quality. These 12 indicators have a narrow focus and will allow only a restricted assessment of prescribing—for example, four of the seven indicators for cost minimisation relate to generic prescribing. Hence, the results obtained with these indicators need to be interpreted carefully and their limitations explicitly acknowledged.
PACT data make some, but by no means all, aspects of prescribing measurable. Three important decisions have to be made when collecting data on prescribing indicators. Firstly, what is the intended unit of analysis (for example, practice population, all individuals with a given condition, an individual)? Secondly, who is going to collect the data (health authorities, primary care groups, individual practices)? Thirdly, what are the resources required for data collection (patients' medical records or PACT)? Prescribing indicators can be used for various purposes, and it is vital for quality assessment that this purpose is made explicit.19 The validity of any type of indicator is related to its intended purpose. Additional resources are needed to produce and collect data for indicators relating to individual patients rather than populations and for indicators requiring examination of individual patients' records rather than PACT data.9,20
What is already known on this topic
Indicators based on PACT data have been developed to allow comparison of prescribing behaviour between health authorities, primary care groups, and general practices
Little is known about the way that PACT based indicators are used in practice
What this study adds
Some PACT based indicators are currently viewed as measures of quality
Consensus about the validity of PACT based indicators was low: five of 41 were judged to be valid for quality and seven for cost minimisation
These indicators have a narrow focus and allow only limited examination of prescribing
Our finding that five of the indicators were rated face valid for measuring quality does not fit within the model of indicators proposed by Queenborough and Roberts.20 They advocated that only when prescribing and clinical data are linked can quality be measured, usually with reference to individual patient data. This view was shared by many respondents in the survey when answering an open ended question about how the quality of prescribing can or should be measured. Respondents felt that PACT indicators are process measures that need to be linked to clinical audit or patient outcome measures. This view also reflects the opinion that definitions and measurements of quality of care are most meaningful when applied to individual patients.21
Use of indicators
Our findings suggest three further caveats for people engaged in quality assessment or improvement, including primary care groups in the United Kingdom. Firstly, indicators are not measures of poor performance. Rather, they identify potential problems that may require investigation by other methods, usually audit. Secondly, it is important to be clear about what the indicators are intended to measure and what conclusions can be claimed from their use. Thirdly, for indicators to be useful for quality assessment or improvement, consistent and comparable data must be available across the relevant healthcare organisations.
Only two of the 12 indicators rated valid and reliable in this study were being used by over 50% of prescribing advisers in England. This has important implications for proponents of national sets of prescribing indicators. Prescribing indicators at the population level, such as those examined here, can never be robust enough to give any more than an absolute rather than relative measure of performance. However, our findings provide a starting point for developing a common set of prescribing indicators.
Figure.
Example of feedback on indicators included in second round questionnaire
Acknowledgments
We thank Clive Jackson, director of the National Prescribing Centre, for help with sending out the questionnaire and the respondents who took part in the Delphi consultation.
Footnotes
Funding: This project was funded out of National Primary Care Research and Development Centre core funding from the Department of Health. The Prescribing Support Unit is funded by the Department of Health.
Competing interests. None declared.
References
- 1.Department of Health. Quality in the new NHS: a first class service. London: DoH; 1998. [Google Scholar]
- 2.Secretary of State for Health. The new NHS. London: Stationery Office; 1998. . (Cm 3807.) [Google Scholar]
- 3.Prescribing Support Unit. Prescribing measures and the application. An explanation. Leeds: PSU; 1998. [Google Scholar]
- 4.National Prescribing Centre and NHS Executive. GP Prescribing Support Unit—a resource document for the new NHS. London: NPC, NHSE; 1998. [Google Scholar]
- 5.Audit Commission. A prescription for improvement: towards more rational prescribing in general practice. London: HMSO; 1994. [Google Scholar]
- 6.Roland M, Holden J, Campbell S. Quality assessment for general practice: supporting clinical governance in primary care groups. Manchester: National Primary Care Research and Development Centre; 1999. [Google Scholar]
- 7.Campbell SM, Roland MO, Quayle JA, Buetow SA, Shekelle PG. Quality indicators for general practice: which ones can general practitioners and health authority managers agree are important and how useful are they? J Public Health Med. 1998;20:414–421. doi: 10.1093/oxfordjournals.pubmed.a024796. [DOI] [PubMed] [Google Scholar]
- 8.Buetow SA, Sibbald B, Cantrill JA, Halliwell S. Appropriateness in health care: application to prescribing. Soc Sci Med. 1997;45:261–271. doi: 10.1016/s0277-9536(96)00342-5. [DOI] [PubMed] [Google Scholar]
- 9.Cantrill JA, Sibbald B, Buetow S. Indicators of the appropriateness of long term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability. Qual Health Care. 1998;7:130–135. doi: 10.1136/qshc.7.3.130. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Blades S, Eccles M, McColl E, Campbell M. Understanding the appropriateness of prescribing in primary care. Eur J Gen Pract. 1998;4:60–64. [Google Scholar]
- 11.Avery AJ, Heron T, Lloyd D, Harris CM, Roberts D. Investigating relationships between a range of potential indicators of general practice prescribing: an observational study. J Clin Pharm Ther. 1998;23:441–450. doi: 10.1046/j.1365-2710.1998.00187.x. [DOI] [PubMed] [Google Scholar]
- 12.Bateman DN, Eccles M, Campbell M, Soutter J, Roberts SJ, Smith JM. Setting standards of prescribing performance in primary care: use of a consensus group of general practitioners and application of standards to practices in the north of England. Br J Gen Pract. 1996;46:20–25. [PMC free article] [PubMed] [Google Scholar]
- 13.Majeed A, Evans N, Head P. What can PACT tell us about prescribing in general practice? BMJ. 1997;315:1515–1519. doi: 10.1136/bmj.315.7121.1515. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Cantrill JA, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract. 1996;1:67–71. [Google Scholar]
- 15.Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1996;311:376–380. doi: 10.1136/bmj.311.7001.376. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Brook RH. The RAND/UCLA appropriateness method. Santa Monica: RAND; 1995. [Google Scholar]
- 17.Brook RH, Chassin MR, Fink A, Solomon DH, Kosekoff J, Park RE. A method for the detailed assessment of the appropriateness of medical technologies. Int J Tech Ass Health Care. 1986;2:53–63. doi: 10.1017/s0266462300002774. [DOI] [PubMed] [Google Scholar]
- 18.Shekelle PG, Kahan JP, Park RE, Bernstein SJ, Leape LL, Kamberg CA, et al. Assessing appropriateness by expert panels: how reliable? J Gen Intern Med. 1995;10(suppl):81. [Google Scholar]
- 19.Cantrill J, Devlin M, Jackson C, Queenborough R. Improving quality in primary care: supporting pharmacists in primary care groups and trusts. Manchester: National Primary Care Research and Development Centre; 1999. [Google Scholar]
- 20.Queenborough R, Roberts D. The relationship between the quality of prescribing indicators and their availability: a model for primary care groups. Prescriber. 1999;10:47–51. [Google Scholar]
- 21.Campbell SM, Roland MO, Buetow SA. Defining quality of care. Soc Sci Med (in press). [DOI] [PubMed]