Abstract
Since guidelines on antibiotic drug treatment often focus on appropriate first choice drugs, assessment of guideline adherence should only concentrate on the first drug prescribed, and not on subsequent antibiotics prescribed after failure of the first one.
Purpose
To determine a valid cut-off point for a definition of “first” or “new” prescription in indicators for the assessment of the quality of antibiotic drug treatment on the basis of pharmaceutical data.
Methods
Three possible definitions for the term “new prescription” were compared, based on three different periods of time, viz. more than 35, 28, or 21 days after starting a previous antibiotic. In an observational study, 1,225 antimicrobial prescriptions from the medical files of five family practices were audited (“clinical classification”) and compared with a classification based on the three definitions (“technical classification”). Agreement between these clinical and technical classifications was determined by calculating Cohen’s kappa. The technical classification was analyzed as a diagnostic test, using the clinical classification as the gold standard, and sensitivity, specificity, likelihood ratios, and post-test probabilities were calculated.
Results
Defining “new prescription” as “more than 35 days after a previous prescription was issued” resulted in a Cohen’s kappa of 0.93 (95% CI 0.92–0.98). The diagnostic value of this definition was extremely high, with a sensitivity of 0.976, specificity of 0.987, positive likelihood ratio of 77.7, and negative likelihood ratio of 0.02.
Conclusion
We recommend using a cut-off value of 35 days since the last antimicrobial prescription as the definition of a “new prescription” when no diagnostic information is available, i.e., when using pharmaceutical data to assess the quality of antibiotic prescribing behavior.
Keywords: Quality indicators, health care; Primary health care; Prescriptions; Guideline adherence
Introduction
Resistance of microorganisms to antimicrobials is an increasing problem worldwide, posing a potential threat to public health [1]. The European Surveillance of Antimicrobial Consumption (ESAC; www.ua.ac.be/ESAC) project, funded by the European Commission, has collected data on the use of antibiotics in Europe since 2001. Several of their studies found that higher consumption of antibiotics correlates strongly with high rates of resistance [2]. General practitioners play a key role in fighting antimicrobial resistance, since they provide 80% of all prescriptions of antibiotics in human medicine [3]. In addition, veterinarian use of antibiotics in Western Europe is also accountable for a major part of the total volume of antibiotics used [4], and it is especially the nontherapeutic use of antibiotics as growth promoters which may have played an important part in the development of antimicrobial resistance, although some authors doubt this [5]. The use of antimicrobial growth promoters was prohibited in the beginning of 2006, but this issue is beyond the scope of the present article.
To minimize antimicrobial resistance, general practitioners are recommended to adhere to the clinical practice guidelines on antibiotics. Guidelines on prescribing antimicrobial drugs focus on the type of antibiotics that should be used as a treatment of first choice and the types that should be avoided as first choice treatment. Only a few guidelines also mention what choice should be made after failure of the initial treatment.
In situations where one evaluates individual prescriptions and with the availability of a diagnosis it might be possible to detect whether a prescription is the first one or a subsequent after failure of the first, and so whether the choices made are appropriate. However, with the use of a pharmaceutical database that lacks information about diagnosis this is not possible. When studying all prescriptions in a given period, however, one could detect what proportion of the first prescriptions are in line with guidelines. This means that when assessing the guideline adherence in antibiotic drug treatment in a pharmaceutical database, only those prescriptions that represent the first antibiotic treatment for a disease/diagnosis (“first” or “new” prescriptions) should be taken into account.
In the context of a regional quality improvement project, we developed a set of quality indicators that measure guideline adherence regarding the prescription of antimicrobial drugs by general practitioners. Our indicators are derived from 12 guidelines by the Dutch College of General Practitioners and have been validated by a panel of experts using the RAND-UCLA method [7], covering face and content validity.
One option for assessing guideline adherence is to use pharmaceutical databases [6]. Since these databases lack diagnostic information, however, we used a predefined minimum number of days between the current prescription and an earlier antibiotic prescription as a proxy measure to determine whether the current prescription should be regarded as first or new and should be taken into account when assessing the quality of prescribing.
The prescribing quality indicators are applied to a large pharmaceutical database containing PACT (prescribing analyses and cost) data. Pharmacists in the Netherlands charge all delivered drugs to health insurance companies. Their accounts include information about the patient (age and sex), the prescriber, and the drug [8, 9]. General practitioners do not, however, provide information on indications/diagnoses to pharmacists, due to autonomy considerations. We collected data from two major health insurance companies in the southern part of the Netherlands, which meant that huge quantities of information on all drugs supplied in 1 year were available for analysis.
In four of our indicators, we encountered problems relating to the definition of a “new prescription” (Table 1). If the period of time since the previous prescription that is used in our indicator is too short, many prescriptions that are in fact second choice prescriptions for the same indication/diagnosis, after failure of the first treatment, would incorrectly be labeled as new prescriptions by our indicator. If the period of time is too long, many new prescriptions will be missed in the calculation of the indicator score. In this study we therefore tried to determine after what period of time (21, 28, or 35 days) it is safe to assume that a prescription of antibiotics is new, i.e., it is the first prescription for a new indication. We investigated this by comparing the pharmaceutical data with information from the medical files of patients kept by five family practices.
Table 1.
Four indicators containing the term “new prescription”
| No. | Description | Numerator (N) | Denominator (D) | Indicator |
|---|---|---|---|---|
| 1 | Minimize prescribing of amoxicillin in combination with clavulanic acid as an antibiotic of first choice | All new prescriptions of amoxicillin in combination with clavulanic acid | All new prescriptions of antimicrobial drugs | 1 − (N/D) |
| 2 | Minimize prescribing of macrolides as an antibiotic of first choice | All new prescriptions of macrolides | All new prescriptions of antimicrobial drugs | 1 − (N/D) |
| 3 | Minimize prescribing of quinolones as an antibiotic of first choice | All new prescriptions of quinolones | All new prescriptions of antimicrobial drugs | 1 − (N/D) |
| 4 | Preferably prescribe amoxicillin, doxycyclin, and nitrofurantoin as antibiotics of first choice | All new prescriptions of amoxicillin, doxycyclin, and nitrofurantoin | All new prescriptions of antimicrobial drugs | N/D |
Methods
Study design
In a cross-sectional observational study, a random sample of prescriptions of antimicrobial drugs was taken from patient files for 2008 kept by family physicians. We used these prescriptions to calculate how the four indicators containing the phrase “new prescription” (Table 1) would classify each prescription, using different definitions of the period since the previous prescription (35, 28, 21 days).
Population
A total of 1,225 antibiotics prescriptions were retrieved from five family practices in the south of the Netherlands. All practices are participating in a registration network for general practitioners (RNH) managed by the University of Maastricht [10]. Four of them were group practices including at least three doctors, both men and women, with different years of clinical experience. One practice was run by a single female general practitioner.
Variables
The indications for the prescription of all examined antibiotics prescriptions were determined from the medical files. We also recorded whether the prescription was the first antibiotic treatment for the diagnosis (“clinical classification”). We then assessed how the three different definitions of “new prescription” would have classified the prescription at stake (“technical classification”).
Data collection
A senior medical student with sufficient medical knowledge visited the five practices. The electronic medical records were used to select antimicrobial drug prescriptions from 2008, taking a random sample from these lists of prescriptions, to limit the workload. For each prescription of an antimicrobial drug, the corresponding part of the medical record was extracted and made anonymous. The following items were recorded:
Is the prescription the first antimicrobial prescription for this diagnosis according to the medical judgment of the investigator?
Would the indicator definitions have classified this antimicrobial prescription as new or not?
The first practice visited was used as a trial for the procedure, to see whether any problems would arise with the interpretation of the medical files. This meant that more prescriptions (n = 505, about 1/3 of all 1,792 antibiotic prescriptions for 2008) were audited in this first practice than in the other practices. All records were also assessed by the first author, a general practitioner with more than 20 years of experience. Since no unforeseen problems were encountered, the results of the first practice were included in the analysis.
A fixed total number of 180 prescriptions were audited in each of the remaining three practices. These included 90 prescriptions for indicators 1–3 (30 prescriptions for each indicator) and 90 for indicator 4. Since indicator 4 includes three different types of antibiotics, and we wanted to ensure sufficient numbers per subgroup (not analyzed in this paper), 30 files per antibiotic type were audited for this indicator, so 90 in all.
Analysis
The results of the scoring forms for all practices were pooled for each indicator and for each definition, and entered into a 2 × 2 table. Cohen’s kappa and the corresponding 95% confidence interval were then determined to estimate the agreement between the “technical classification” (i.e., the calculation of an indicator score based on a pharmaceutical database) and the “clinical classification,” based on knowledge of the medical file, as to whether a prescription should be seen as “new.”
We also analyzed the data by considering the technical classification as a diagnostic test, using the clinical classification as the gold standard, and calculated the sensitivity, specificity, likelihood ratios (LR), and post-test probabilities of this technical classification.
Results
The audit of the first practice related to 505 prescriptions, those of the remaining practices to 720 prescriptions. For each of these 1,225 prescriptions, we determined whether the indicator would classify them as “new” (first) or “not new” (not first). According to the assessment of the medical files by the investigators, there were 962 new antimicrobial prescriptions for which the period of time after the previous prescription of an antibiotic drug was more than 35 days. There were only two prescriptions that had been issued more than 35 days after an earlier prescription but which were in fact not new (i.e., related to the same diagnosis as the previous prescription). These prescriptions were incorrectly classified as “new” by the indicators, regardless of which of the three definitions was used. Table 2 shows how these numbers changed when different periods of time were used in the definition of a “new” prescription. Extending the chosen period to 42 or even 49 days only added two new prescriptions that were missed otherwise (data not shown).
Table 2.
2 × 2 tables of all prescriptions audited. All prescriptions in the selected patient files were compared with our indicators using three different definitions of “new prescription” (>35, >28, and >21 days after the previous antimicrobial prescription)
| Medical file | Total | |||
|---|---|---|---|---|
| New | Not new | |||
| Classified by indicator using 35 days definition | New | 962 | 3 | 965 |
| Not new | 24 | 236 | 260 | |
| Total | 986 | 239 | 1,225 | |
| Classified by indicator using 28 days definition | New | 966 | 23 | 989 |
| Not new | 20 | 216 | 236 | |
| Total | 986 | 239 | 1,225 | |
| Classified by indicator using 21 days definition | New | 974 | 51 | 1,025 |
| Not new | 12 | 188 | 200 | |
| Total | 986 | 239 | 1,225 | |
The results of all practices were pooled for each indicator and for each period of time used in the definition, and a Cohen’s kappa and 95% confidence interval could be determined (Table 3). Results for each indicator showed that the classification of new prescriptions agreed best with the classification based on the medical data if the definition involved a minimum of 35 days after a previous antimicrobial drug prescription. Of the 986 new prescriptions, 962 (97.6%) were correctly identified by the indicators as new, while 24 (2.4%) were incorrectly identified as not new. Of the 239 not-new prescriptions, 236 (98.7%) were correctly identified by the indicators as not new, and 2 (1.3%) were incorrectly identified as new.
Table 3.
Cohen’s kappa with confidence interval for each indicator, defining a “new prescription” as one issued 21, 28, and 35 days after a previous one (n = number of prescriptions analyzed)
| Indicator | Type of antibiotic | 21 days | 28 days | 35 days | |||
|---|---|---|---|---|---|---|---|
| Cohen’s kappa | 95% CI | Cohen’s kappa | 95% CI | Cohen’s kappa | 95% CI | ||
| 1 | Amoxicillin with clavulanic acid (n = 197) | 0.90 | 0.83–0.97 | 0.93 | 0.87–0.99 | 0.99 | 0.96–1.00 |
| 2 | Macrolides (n = 171) | 0.76 | 0.63–0.89 | 0.87 | 0.78–0.96 | 0.93 | 0.82–1.00 |
| 3 | Quinolones (n = 176) | 0.79 | 0.69–0.88 | 0.89 | 0.82–0.96 | 0.93 | 0.87–0.98 |
| 4 | Amoxicillin, doxycillin, nitrofurantoin (n = 681) | 0.81 | 0.78–0.87 | 0.86 | 0.85–0.92 | 0.92 | 0.90–0.98 |
| 1–4 | Pooled antibiotics of indicators 1–4 (n = 1,225) | 0.83 | 0.77–0.88 | 0.89 | 0.86–0.94 | 0.93 | 0.92–0.98 |
We also analyzed the data by considering the technical classification as a diagnostic test, using the clinical classification as a gold standard, and calculating sensitivity, specificity, and likelihood ratios for the three different operationalizations (Table 4). The 21-day definition had a sensitivity of 0.988, a specificity of 0.788, an LR+ of 4.6, and an LR− of 0.015. The 28-day definition had a sensitivity of 0.980, a specificity of 0.904, an LR+ of 10.2, and an LR− of 0.022. The 35-day definition had a sensitivity of 0.976, a specificity of 0.987, an LR+ of 77.7, and an LR− of 0.025.
Table 4.
Sensitivity, specificity, likelihood ratios for positive and negative results, and pre-test and post-test probabilities for positive and negative results, defining a “new prescription” as one issued 21, 28, and 35 days after a previous one
| 21 days | 28 days | 35 days | |
|---|---|---|---|
| Sensitivity | 0.987 | 0.980 | 0.976 |
| Specificity | 0.787 | 0.903 | 0.987 |
| Likelihood ratio + | 4.63 | 10.18 | 77.73 |
| Likelihood ratio − | 0.02 | 0.02 | 0.02 |
| Pre-test probability + | 0.805 | 0.805 | 0.805 |
| Post-test probability + | 0.948 | 0.976 | 0.997 |
| Pre-test probability − | 0.195 | 0.195 | 0.195 |
| Post-test probability − | 0.003 | 0.005 | 0.006 |
The likelihood ratio incorporates both the sensitivity and specificity of the test and provides a direct estimate of how much a test result will change the odds of having a certain property. The likelihood ratio for a positive result (LR+) tells you how much the odds of the property increase when a test is positive. The likelihood ratio for a negative result (LR−) tells you how much the odds of the property decrease when a test is negative. The pre-test probability is the proportion of prescriptions that do or don’t have the target characteristic (“first” or “not first” antibiotic). The post-test probability is the proportion of prescriptions with that particular test result that have the target characteristic
Given a pre-test probability of an antibiotic prescription being new of 80.5%, the post-test probabilities for positive test results were 94.9, 97.6, and 99.7% for the 21-, 28-, and 35-day definitions, respectively, and the post-test probabilities for negative test results were 0.38, 0.55, and 0.61%, respectively.
Discussion
We developed a set of indicators of the quality of doctors’ prescribing behavior in the context of a regional project to measure guideline adherence in general practice, using a pharmaceutical database. The validity and accuracy of such quality indicators must be guaranteed to enable the quality of health care to be assessed. In addition to aspects such as face and content validity, problems arose with four of the indicators concerning the definition of the term “new prescription” in relation to antimicrobial drugs, in the sense that a “new” antimicrobial drug has to be the first treatment for a given diagnosis. Practically speaking, this means that there should be a certain minimum period of time between the “new” prescription and the time when the previous antimicrobial prescription was issued. We examined the effect of three different periods of time (35, 28, and 21 days) for this definition by auditing medical files in five family practices in the south of the Netherlands and calculating which period would best allow our indicators to classify antimicrobial prescribing behavior correctly. We found that our four indicators showed the best fit with reality if we defined a “new prescription” as being issued at least 35 days after the previous antimicrobial prescription was issued (Cohen’s κ ≥ 0.90). This “clinical” classification (based on patient files) was compared with a “technical” classification, based on our indicators (which were to be applied to a pharmaceutical database). When analyzing the technical classification as a diagnostic test, the post-test probability also showed very high values. This high score legitimizes the use of 35 days as the cut-off value, with only a low rate of misclassification remaining. In fact, most misclassifications involved new prescriptions being classified as not new, which would mean they are not taken into account when calculating the score for a particular quality indicator.
Our study was subject to certain limitations. Firstly, the results might not be representative of all family practices in the Netherlands, since our study took place in the southern part of the country only, and only in five family practices. However, the population of practices included in the RNH registration network are generally well comparable to the Dutch population as a whole [10]. Secondly, the auditing of the medical files was done by a student and not by an experienced doctor. However, this was a senior medical student, who immediately after completing this research project received his medical degree and started vocational training in general practice. Furthermore, the student discussed all problematic classifications (3%) with the first author. The third limitation was that we only compared results for three periods of time: 35, 28, and 21 days, and not for periods longer than 35 days. Extending this period beyond 35 days would definitely improve the accuracy of the concept of “new prescription,” but would incorrectly exclude from the numerator and denominator too many prescriptions that were in fact new, thus affecting the validity as well as the robustness of the indicator.
The strengths of our study were that the five practices we visited differed in terms of location, population, and degree of urbanization, and that the medical files in these practices were audited by an independent senior medical student with no conflict of interest.
Unfortunately, we found no other studies on this subject at the time we performed our study to compare our results with.
To put these findings into perspective, it is essential to distinguish between the external and internal use of quality indicators. Indicators for external use are intended to be used by governments, patient organizations, or health insurance companies, whereas internal indicators are mainly used by the health care providers themselves [6, 11]. They can use the information generated by quality indicators to monitor the health care provided by themselves or the organization they belong to, and can investigate the potential and current problems by reflecting on these data [6]. In the case of disappointing quality, the scores should be carefully analyzed to identify explanatory factors, and if the data are correct and concern modifiable factors, they can be used to improve the outcome. Given the different purposes of internal and external indicators, they need to meet different criteria [6, 11]. External indicators need to have the highest validity, as they are used to judge the work of medical professionals. Internal indicators also need to be accurate, but since they are mainly used for self-reflection and their results have no major consequences, certain errors of measurement are acceptable. Our indicators have a high validity but are not infallible, and we recommend cautious application, merely for internal use.
In conclusion we can state that our indicators of the quality of prescribing behavior show good validity and accuracy for internal use if we define the term “new prescription” as a prescription issued 35 days (5 weeks) or more after the previous antimicrobial prescription was issued.
Acknowledgements
We would like to thank the following general practitioners for allowing their medical files to be audited: Isabel Buijs, Stefan van der Eerden, Henk Jochems, Joep van Proosdij, Caroline Robertson.
Conflict of Interest Statement
The authors declare that they have no conflict of interest.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
References
- 1.European Antimicrobial Resistance Surveillance System (EARSS) (2007) Annual report 2007. EARSS, Bilthoven
- 2.Goossens H, Ferech M, Vander Stichele R, Elseviers M. Outpatient antibiotic use in Europe and association with resistance: a cross-national database study. Lancet. 2005;365(9459):579–587. doi: 10.1016/S0140-6736(05)17907-0. [DOI] [PubMed] [Google Scholar]
- 3.Braspenning J, Schellevis F (eds) (2004) Tweede nationale studie naar ziekten en verrichtingen in de huisartspraktijk. Kwaliteit huisartsenzorg belicht. NIVEL/WOK, Utrecht
- 4.MARAN (2008) Monitoring of antimicrobial resistance and antibiotic usage in animals in the Netherlands in 2008. www.maran2008.wur.nl
- 5.Philips I, Casewell M, Cox T, et al. Does the use of antibiotics in food animals pose a risk to human health? A critical review of published data. J Antimicrob Chemother. 2004;53:28–52. doi: 10.1093/jac/dkg483. [DOI] [PubMed] [Google Scholar]
- 6.Andersen M. Is it possible to measure prescribing quality using only prescription data? Basic Clin Pharmacol Toxicol. 2006;98(3):314–319. doi: 10.1111/j.1742-7843.2006.pto_411.x. [DOI] [PubMed] [Google Scholar]
- 7.Fitch K, Bernstein SJ, Aguilar MS, et al (2001) RAND/UCLA appropriateness method user's manual. http://www.rand.org/pubs/monograph_reports/MR1269/
- 8.Campbell SM, Cantrill JA, Roberts D. Prescribing indicators for UK general practice: Delphi consultation study. Br Med J. 2000;321(7258):425–428. doi: 10.1136/bmj.321.7258.425. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Muijrers PE, Janknegt R, Sijbrandij J, Grol RP, Knottnerus JA. Prescribing indicators. Development and validation of guideline-based prescribing indicators as an instrument to measure the variation in the prescribing behaviour of general practitioners. Eur J Clin Pharmacol. 2004;60(10):739–746. doi: 10.1007/s00228-004-0821-5. [DOI] [PubMed] [Google Scholar]
- 10.Metsemakers JF, Hoppener P, Knottnerus JA, Kocken RJ, Limonard CB. Computerized health information in the Netherlands: a registration network of family practices. Br J Gen Pract. 1992;42(356):102–106. [PMC free article] [PubMed] [Google Scholar]
- 11.Berg M, Meijeerink Y, Gras M, Goossensen A, Schellekens W, Haeck J, et al. Feasibility first: developing public performance indicators on patient safety and clinical effectiveness for Dutch hospitals. Health Policy. 2005;75(1):59–73. doi: 10.1016/j.healthpol.2005.02.007. [DOI] [PubMed] [Google Scholar]