Abstract
Adverse reactions to medications to which the patient was known to be intolerant are common. Electronic decision support can prevent them but only if history of adverse reactions to medications is recorded in structured format.
We have conducted a retrospective study of 31,531 patients with adverse reactions to statins documented in the notes, as identified with natural language processing. The software identified statin adverse reactions with sensitivity of 86.5% and precision of 91.9%. Only 9020 of these patients had an adverse reaction to a statin recorded in structured format. In multivariable analysis the strongest predictor of structured documentation was utilization of EMR functionality that integrated the medication list with the structured medication adverse reaction repository (odds ratio 48.6, p < 0.0001). Integration of information flow between EMR modules can help improve documentation and potentially prevent adverse drug events.
Background
Electronic medical records (EMRs) are widely promoted because of their potential benefits in patient care1,2. In particular, a number of studies have demonstrated that EMRs can decrease the incidence of medication errors and potentially adverse drug events3–7. However, these benefits are only realized if medication data in the EMR are stored in structured and coded format, enabling decision support such as renal dose adjustment, drug-drug interaction checking, drug-laboratory test checking, and others. Nevertheless, a large fraction of medication information in EMR only exists in unstructured format, such as narrative provider notes8.
Adverse reactions to medications are an area where EMRs can have a particularly strong impact. Studies have shown that up to 13% of adverse drug events in both hospital and ambulatory settings are due to patients receiving a medication to which they were previously known to be allergic or intolerant9–11. Decision support that includes drug-allergy checking can reduce the number of these errors by over 50%3. Drug-allergy checking requires that allergies and adverse reactions to medications be recorded in a coded fashion so that they could be compared to the medication orders. However, it is not known how frequently adverse reactions to drugs are actually recorded that way.
If a substantial fraction of adverse reactions to medications are not documented in structured format, it will be necessary to achieve a better understanding of the factors that influence coded documentation of medication side effects and allergies. We therefore conducted this study to determine the prevalence of known adverse reactions to medications not entered into a structured EMR repository and to identify patient, provider and system characteristics that are associated with their structured documentation. We have chosen to focus our initial investigation on 3-hydroxy-3-methyl-glutaryl-CoA reductase inhibitors (a.k.a. statins) as a widely used class of medications that have both well-established benefits and a significant rate of adverse reactions12–14.
Methods
Design
We designed and validated natural language processing software that identifies statin side effects documented in narrative provider notes. We subsequently utilized the software to conduct a retrospective cohort study to determine the factors associated with documentation of statin side effects in a structured EMR repository. We evaluated the relationship between a) patient characteristics; b) provider characteristics and c) event characteristics related to EMR design as predictor variables and probability of statin side effect documentation in structured format (primary outcome variable).
Study Environment
The study was conducted at Partners HealthCare System - an integrated healthcare delivery network in eastern Massachusetts that includes founding members Brigham and Women’s Hospital (BWH) and Massachusetts General Hospital (MGH) as well as several community hospitals and affiliated private practices. Partners HealthCare has an enterprise-wide database of medication allergies and side effects – Partners Enterprise Allergy Repository (PEAR)15. Most out- and inpatient EMR systems used at Partners deposit allergy and side effect information in PEAR.
This study analyzed the data in the most commonly used outpatient EMR at Partners – Longitudinal Medical Record (LMR). LMR is an internally developed CCHIT-certified EMR that allows structured entry of multiple data elements as well as narrative notes and provides extensive clinical decision support. LMR permits data entry into PEAR by two mechanisms (Figure 1). In the first pathway the user clicks on the “Add New” button in the “Allergies” section of the patient’s record. In the second pathway the user is transferred to the side effect documentation interface after selecting “Adverse Reaction” as the reason for discontinuing a medication. PEAR allows both coded and free-text documentation of both medications and their side effects.
Figure 1.
Pathways of Entry of Medication Side Effects into the Structured Repository
Study Population
The study population included all patients from BWH and MGH who have ever been recorded as taking a statin in the LMR between 2000 and 2010.
Algorithm Design for Identification of Adverse Reactions to Statins
A custom natural-language processing (NLP) algorithm was developed for this study. TextMiner is multi-threaded, Perl-based analysis software that mines data from a set of provider notes, based on a custom language model consisting of sets of word classes (WC) and sets of phrase structure rules (PS). WCs are equivalence classes of semio-syntactic function words, written as regular expressions. They are similar to parts of speech, but they contain semantic information as well. Some examples include:
>MORBIDITY - cholesterol, chol, (hyper-?)?(lipid|cholesterol|triglycerida?)emias?, ldls?, hdls?, diseases?, tgs?
>DISCUSSVERB - discuss(e[sd])?(ing)?, talks?(ed)?(ing)?, reviews?(ed)?(ing)?, counsell?s?(ed)?(ing)?
>POSSPRONOUN - his, her, him, my, your, me
PSs are parsing rules that operate on the parse tree for a sentence-like analysis unit to generate larger phrase structures with meaning. While the software can handle context-dependent grammar rules, most of the rules are written as context-free. When the parsing process arrives at a goal-state PS, an output is generated that uniquely identifies the note, the token itself, and (optionally) the specific statin generic name and the concept unique identifier (CUI) for the adverse reaction phrase from the MedDRA dictionary. This optional process is known as “mapping” a statin adverse reaction to a CUI.
The statin side effect language model was based on a set of 3175 narrative provider notes manually reviewed by trained pharmacy students. In order to enrich the dataset for documentation of adverse reactions to statins, the notes were randomly selected from a set of all notes written on the day a statin was discontinued in the EMR. One reviewer marked up each note set with instances of adverse reactions to statins, and we built the word classes and grammar rules to maximize the accuracy of the resulting output.
In order to process a very large set of notes at acceptable speeds, we implemented a multi-threaded software architecture for parallel note processing and a hierarchical scanning mechanism that scans only notes and sentences that could have documentation of a statin side effect, namely those that contain a statin name. Notes are bundled into packages by a separate note packager thread, and are then placed on a queue for the next available analyzer thread.
Figure 2A shows how a phrase like “myalgias - lipitor” or “developed a rash with zocor” would parse to the <STATINSIDEEFFECT goal state. Figure 2B illustrates the rejection of a phrase like “The statins are not causing her joint pain,” as this instead parses to a negated statin side effect phrase structure (<NEGSSE). The green boxes are the original word arrays, yellow boxes are the word class assignments, and the blue boxes are the resulting phrase structures.
Figure 2A.
Example of a Parsing Path into a Statin Side Effect Goal State
Figure 2B.
Example of a Parsing Path into a Negated Statin Side Effect Goal State
An analyzer thread performs a cursory scan of the notes in the note package for any statin name (including some common misspellings) and only processes those notes that pass the scan, to ensure computational resources are not wasted on parsing irrelevant text. Notes that contain a specific statin (or generic statin) name are pre-processed for formatting and then split into sentences. A similar scanning process then happens at the sentence-level. The software includes a setting to limit the maximum number of analyzer threads in order to prevent it from completely taking over all available CPUs.
Once the note text is split into sentence-like units of analysis, each sentence that passes its scan is split into a word array, and the individual words are assigned to word classes. Since it is possible for a single word to belong to multiple word classes, a multiple resolver file contains sets of word classes and the target word class to which the word should be assigned. For example, “statin-related” meets the criteria for both the >SPECIFIC and >ETIOLOGY word classes, but it is more important that it be seen as a specific medication name to combine with any subsequent side effect phrase. Therefore, the multiple resolver casts this as a >SPECIFIC. This process is not automatic, and is part of the language model specified for this task. Once the word classes are assigned, the phrase structure rules are applied to generate the parse tree from the bottom up. Multiple sets of PS rules can be specified, and each is applied to the parse tree until no further changes can be made in one full pass across the array. Once the first rule set has run to exhaustion, the second rule set is applied, and so on. If at any point the specified goal state is reached, output is generated that identifies the note, the sentence containing the statin side effect, and the subtexts that are later mapped. The subtexts generated for this project were <SIDEEFFECT and <SPECIFIC, so the example in Figure 2A would return “myalgias” and “lipitor”, respectively. In Figure 2B, if the etiology phrase were not negated, the subtexts generated would then be “her joint pain” and “statins”.
Algorithm Evaluation
The accuracy of the software was measured against a “gold standard” set of 242 randomly selected provider notes independently reviewed by two trained pharmacy students. Discrepancies between the two reviewers’ ratings were reconciled by a joint review. At the note level, sensitivity (recall), positive predictive value (PPV or precision), and specificity were calculated. Specificity has no meaning at the token-level because of the arbitrary nature of the tokenization process (i.e. token boundaries in free text are implementation-specific), so only sensitivity and PPV were calculated.
Factors Affecting Statin Side Effect Documentation in Structured Data
In order to determine the factors that affect documentation of statin side effects as structured data we utilized the algorithm described above to identify all patients who ever had a statin side effect documented in narrative provider notes. An individual patient served as the unit of analysis. The presence of a record in PEAR of any side effect to any statin served as the binary outcome variable. We included three groups of predictor variables in the analysis:
patient characteristics: age, sex, race, health insurance, median income by zip code, diagnosis of coronary artery disease, diabetes or depression, primary language, the number of notes that contain documentation of a statin side effect, the number of unique side effects to any statin, and whether the patient ever had a statin discontinued;
provider characteristics: age and experience using PEAR (calculated as the natural logarithm of the number of previous unique PEAR entries); and
- event characteristics related to EMR design. The last category included two variables:
- whether the statin side effect was an allergic reaction and
- whether the patient ever had a statin discontinued with the recorded reason of “adverse reaction”.
The “responsible” provider whose characteristics were included in the analysis was defined as the author of the first note that documented a statin side effect for each individual patient. The analysis was corrected for repeated measures within the same provider.
Statistical Analysis
Summary statistics were constructed by using frequencies and proportions for categorical variables and means and standard deviations for numerical variables. Normal approximation was used to calculate confidence intervals for sensitivity, specificity and positive predictive value. Chi-square test was used for comparison of proportions. Repeated measures logistic regression model was constructed to test the effect of predictor variables on probability of structured statin side effect documentation. Significance thresholds were adjusted for multiple hypothesis testing using Simes-Hochberg correction16,17. All analyses were performed with SAS statistical software (version 9.2, SAS Institute, Cary, NC).
IRB
The study protocol was reviewed and approved by the Partners Human Research Committee.
Results
Algorithm Performance
Accuracy: Of the 242 notes in the double-reviewed validation set, 161 contained statin side effect documentation according to the software. At the note level the algorithm achieved sensitivity of 86.5% and PPV of 91.9% (Table 1).
Table 1.
Algorithm Accuracy (%, with 95% confidence intervals in parentheses)
| Sensitivity | Specificity | PPV | ||||
|---|---|---|---|---|---|---|
| Original | Corrected | Original | Corrected | Original | Corrected | |
| Note-level | 86.5 (82.0–91.0) | 87.4 (83.0–91.8) | 81.7 (76.6–86.8) | 98.3 (96.5–100) | 91.9 (88.3–95.5) | 99.4 (98.2–100) |
| Sentence-level | 77.7 (72.2–82.2) | 80.6 (75.4–84.9) | N/A | N/A | 82.6 (77.6–86.7) | 98.6 (96.9–100.0) |
The corrected values are based on post hoc review of the tokens that the software identified as instances of a documented statin side effect. Those instances that were missed by the manual reviewers (sentence-level: 46, note-level: 12) are counted as false positives in the original accuracy results, but as true positives in the corrected accuracy results.
Speed: The software was run on a Dell Server T610, running Windows Server© Standard Edition (Service Pack 2). Two Intel® Xeon® quad-core processors (2.27GHz) with 32 GB of RAM processed 4,706,545 total notes (with mapping) at an average rate of 39.78 notes/second. These numbers reflect the speed of processing a non-enriched data set, where the actual prevalence of documented statin side effects in clinical notes was 3.54%, as detected by the software.
Study Cohort
We identified 224,421 patients as having been on a statin during the study period. Of these, 39053 had ever had a statin side effect documented in a provider note. We excluded 7522 patients due to missing date of birth information for the provider who wrote that patient’s first note recording a statin side effect, missing zip code information for the patient, or missing median income information for the patient’s zip code.
Among 31,531 patients with documented statin side effects that were included in the study (Table 2), only 9020 (28.6%) had a statin side effect recorded in structured format in PEAR. Although three quarters of the patients had a statin discontinued at least once during the study period, only a quarter of these patients showed an adverse reaction as the reason for discontinuation. The mean age of providers who documented the initial statin side effect was 48.6 years. On average, they had recorded 129.0 adverse medication reactions in PEAR prior to authoring the note documenting the statin side effect.
Table 2.
Study Patients Characteristics
| Variable | Value |
|---|---|
| Study patients, n | 31531 |
| Demographic data: | |
| Patient age, years (±SD) | 63.0 (± 12.6) |
| Male, n (%) | 14,441 (45.8%) |
| White, n (%) | 23,661 (75.0%) |
| Primary language is English, n (%) | 29,197 (83.1%) |
| Private insurance, n (%) | 16,095 (51.0%) |
| Median income by zip code, $104 (±SD) | 5.90 (± 2.25) |
| Comorbidities: | |
| Coronary artery disease, n (%) | 6,334 (20.1%) |
| Diabetes mellitus, n (%) | 8,954 (28.4%) |
| Depression, n (%) | 4,299 (13.6%) |
| Statin adverse reactions: | |
| Unique statin adverse reactions1, mean (±SD) | 1.55 (± 1.25) |
| Number of notes documenting a statin side effect, mean (±SD) | 4.20 (± 6.06) |
| Statin discontinued for any reason, n (%) | 23,606 (75.0%) |
| Statin discontinued because of an adverse reaction, n (%) | 6,123 (19.4%) |
| Statin allergic reaction2, n (%) | 1,423 (4.5%) |
Unique preferred CUIs to which statin side effects documented in the notes were mapped
Patients who had at least one true allergic reaction (e.g. hives, rash) to a statin documented
Factors Associated with Entering a Statin Adverse Reaction into PEAR
In multivariable analysis (Figure 5) the strongest predictor of whether a patient had a statin side effect recorded in structured format in PEAR was recording of an adverse reaction as the reason for statin discontinuation, associated with the 48.6 odds of a PEAR record (p < 0.0001). This was followed by having a true allergic reaction to a statin, associated with the 1.8 odds of a PEAR record (p < 0.0001). A greater number of unique statin side effects and a greater number of notes documenting statin side effects were also associated with higher odds of PEAR documentation (p < 0.0001 for both), as was provider’s experience using PEAR (p < 0.0001). A patient’s demographic characteristics and comorbidites did not have an effect on PEAR documentation, with the exception of the patient’s age and health insurance (p < 0.0001 for both). Having a statin discontinued for any reason was also significantly associated with a PEAR entry (p < 0.005).
Figure 5.
Odds ratios of entry of a statin adverse reaction in PEAR at any time, for a given patient
Discussion
In this large retrospective cohort study that utilized a high-validity algorithm to identify adverse reactions to statins documented in narrative notes, we have shown that elements of EMR design have a strong influence on documentation of adverse reactions to medications in structured format. It is well established that tight integration of EMR components is necessary to reap full benefits of computerization of healthcare18, and user surveys show that this is one of the most commonly requested enhancements19. Consistent with these reports, we found that integration of medication order entry and adverse reaction repository had by far the strongest effect on structured documentation of statin side effects.
User interface is another critical component of EMR design, and researchers have reported cases where suboptimally designed user interface contributed to medication errors5. Anecdotally at our institution some clinicians refrain from documenting statin adverse reactions in the “Allergy” section of the EMR because they are not true allergies. Consistent with these reports, in our analysis we have found that true allergies were nearly twice as likely to be entered into PEAR, implying that even a simple change to ensure concordance between a module’s name and its intended scope can have a strong effect.
Several researchers have reported that younger physicians are more likely to use EMRs20,21. However, in our study we did not find a significant relationship between provider age and probability of side effect entry into a structured record. Instead providers who had more experience with the PEAR module were also more likely to use it in the future. This finding points to the importance of user training in ensuring effective utilization of EMR functionalities.
Entry of adverse reaction information into structured format is important for several aspects of patient care. Most obviously, it enables point-of-care drug allergy checking against medication orders to ensure that patients are not given medications to which they are known to have an adverse reaction. Just as importantly, structured data entry facilitates quality of care monitoring and clinical research. While initial clinical trials that include several thousand patients can demonstrate common side effects, only large scale post-marketing surveillance of medication side effects can uncover dangerous but subtle or rare adverse reactions, as has been amply demonstrated in recent years for several blockbuster drugs22,23. Pharmacovigilance analyses of this scale would benefit greatly from structured documentation of medication side effects.
While the algorithm developed for this study was effective at identifying documentation of statin side effects in unstructured format, the language model developed is generalizable to identification of side effects to any medication class. The algorithm is easily extensible to identification of side effects to any set of medications, because the algorithm already identifies side effects to a general list of 4,517 medication names when they are found within sentences related to statins. It is not uncommon for side effects to different classes of medications to be documented within the same sentence even though the software was configured to only produce output for statin side effect phrases. Future work should include studies of unstructured documentation of side effects to other medication classes as well.
Our study had several limitations. Our study population was limited to statin-therapy patients from two primarily academic hospitals in eastern Massachusetts, so the results may not be generalizable to patients on other classes of medications or those at other medical institutions. Also, many of the excluded patients had a responsible provider who was not a physician (making provider demographic information unavailable to us) which may have also biased our results. The frequent lack of temporal information in narrative provider notes further limited our ability to study temporal relationships in structured and unstructured documentation of adverse reactions. Accuracy of the natural language processing algorithm was not perfect; if the errors of the algorithm were distributed unequally with respect to the predictor or outcome variables, it could have biased the results of our analysis. Also, the threshold at which a provider decides that a statin side effect has actually occurred may not be the same as the threshold at which the software determines that a side effect has occurred. The software always sees the phrase “LFT elevation” as a side effect, but a provider may see this as an acceptably small deviation in liver function tests during the course of statin therapy, depending on the magnitude of the deviation. Fine-grained severity information is not always explicitly stated in a note. While the “corrected” accuracy ratings that included rectification of reviewer errors discordant with the software were significantly higher, they did not correct possible reviewer errors concordant with the software and should be interpreted accordingly. Our analysis was retrospective in nature and therefore could only establish associations rather than causal relationships. For example, while higher odds of structured documentation of adverse reactions by physicians who had greater experience using PEAR could be due to their familiarity with the functionality (which could be imparted by training), this association could also be explained by a general tendency by specific providers to use the functionality, independent of their EMR proficiency.
Conclusions
Most adverse reactions to statins documented in the notes were not recorded in structured data. Integration of the adverse reaction repository with medication order entry and optimization of the user interface are likely to increase documentation rate. Further studies are needed to confirm these findings in a prospective fashion and extend them to other medication classes.
Figure 3.
TextMiner main program flow diagram
Figure 4.
Application of parsing rules to a sentence token
Acknowledgments
We are grateful to Ana Delgado, Elizabeth Haftel and Dayton Yuen for their assistance with record review, which allowed for development and validation of the statin side effect identification language model. We would like to thank Perry Mar for his help with mapping the MedDRA terminology hierarchy. This work was supported in part by the grant from the National Library of Medicine (1RC1LM010460).
References
- 1.IOM . Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001. [PubMed] [Google Scholar]
- 2.Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med. 2010 Aug 5;363(6):501–504. doi: 10.1056/NEJMp1006114. [DOI] [PubMed] [Google Scholar]
- 3.Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. Jama. 1998 Oct 21;280(15):1311–1316. doi: 10.1001/jama.280.15.1311. [DOI] [PubMed] [Google Scholar]
- 4.Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med. 1998 Jan 22;338(4):232–238. doi: 10.1056/NEJM199801223380406. [DOI] [PubMed] [Google Scholar]
- 5.Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999 Jul-Aug;6(4):313–321. doi: 10.1136/jamia.1999.00660313. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Mullett CJ, Evans RS, Christenson JC, Dean JM. Development and impact of a computerized pediatric antiinfective decision support program. Pediatrics. 2001 Oct;108(4):E75. doi: 10.1542/peds.108.4.e75. [DOI] [PubMed] [Google Scholar]
- 7.Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003 Jun 23;163(12):1409–1416. doi: 10.1001/archinte.163.12.1409. [DOI] [PubMed] [Google Scholar]
- 8.Turchin A, Shubina M, Breydo E, Pendergrass ML, Einbinder JS. Comparison of information content of structured and narrative text data sources on the example of medication intensification. J Am Med Inform Assoc. 2009 May-Jun;16(3):362–370. doi: 10.1197/jamia.M2777. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Bates DW, Cullen DJ, Laird N, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. Jama. 1995 Jul 5;274(1):29–34. [PubMed] [Google Scholar]
- 10.Gandhi TK, Burstin HR, Cook EF, et al. Drug complications in outpatients. J Gen Intern Med. 2000 Mar;15(3):149–154. doi: 10.1046/j.1525-1497.2000.04199.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Gandhi TK, Weingart SN, Borus J, et al. Adverse drug events in ambulatory care. N Engl J Med. 2003 Apr 17;348(16):1556–1564. doi: 10.1056/NEJMsa020703. [DOI] [PubMed] [Google Scholar]
- 12.Law MR, Wald NJ, Rudnicka AR. Quantifying effect of statins on low density lipoprotein cholesterol, ischaemic heart disease, and stroke: systematic review and meta-analysis. BMJ. 2003 Jun 28;326(7404):1423. doi: 10.1136/bmj.326.7404.1423. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Scott RS, Lintott CJ, Wilson MJ. Simvastatin and side effects. N Z Med J. 1991 Nov 27;104(924):493–495. [PubMed] [Google Scholar]
- 14.Wierzbicki AS, Lumb PJ, Semra Y, Chik G, Christ ER, Crook MA. Atorvastatin compared with simvastatin-based therapies in the management of severe familial hyperlipidaemias. Qjm. 1999 Jul;92(7):387–394. doi: 10.1093/qjmed/92.7.387. [DOI] [PubMed] [Google Scholar]
- 15.Kuperman GJ, Gandhi TK, Bates DW. Effective drug-allergy checking: methodological and operational issues. J Biomed Inform. 2003 Feb-Apr;36(1–2):70–79. doi: 10.1016/s1532-0464(03)00063-7. [DOI] [PubMed] [Google Scholar]
- 16.Simes RJ. An improved Bonferroni procedure for multiple tests of significance. Biometrika. 1986;73(3):751–754. [Google Scholar]
- 17.Hochberg Y. A sharper Bonferroni procedure for multiple tests of significance. Vol 75: Biometrika Trust. 1988. pp. 800–802.
- 18.Hillestad R, Bigelow J, Bower A, et al. Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Aff (Millwood) 2005 Sep-Oct;24(5):1103–1117. doi: 10.1377/hlthaff.24.5.1103. [DOI] [PubMed] [Google Scholar]
- 19.Turchin A, Hamann C, Schnipper JL, et al. Evaluation of an inpatient computerized medication reconciliation system. J Am Med Inform Assoc. 2008 Jul-Aug;15(4):449–452. doi: 10.1197/jamia.M2561. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.DesRoches CM, Campbell EG, Rao SR, et al. Electronic health records in ambulatory care--a national survey of physicians. N Engl J Med. 2008 Jul 3;359(1):50–60. doi: 10.1056/NEJMsa0802005. [DOI] [PubMed] [Google Scholar]
- 21.Burt CW, Sisk JE. Which physicians and practices are using electronic medical records? Health Aff (Millwood) 2005 Sep-Oct;24(5):1334–1343. doi: 10.1377/hlthaff.24.5.1334. [DOI] [PubMed] [Google Scholar]
- 22.Juni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M. Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004 Dec 4–10;364(9450):2021–2029. doi: 10.1016/S0140-6736(04)17514-4. [DOI] [PubMed] [Google Scholar]
- 23.Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. N Engl J Med. 2007 Jun 14;356(24):2457–2471. doi: 10.1056/NEJMoa072761. [DOI] [PubMed] [Google Scholar]