Abstract
An understanding of the strengths and limitations of automated data is valuable when using administrative or clinical databases to monitor and improve the quality of health care. This study discusses the feasibility and validity of using data electronically extracted from the Veterans Health Administration (VHA) computer database (VistA) to monitor guideline performance for inpatient and outpatient treatment of schizophrenia. The authors also discuss preliminary results and their experience in applying these methods to monitor antipsychotic prescribing using the South Central VA Healthcare Network (SCVAHCN) Data Warehouse as a tool for quality improvement.
To improve the quality and outcomes of care for veterans, the Veterans Health Administration (VHA) has emphasized guideline development and implementation as well as the systematic assessment of clinical performance using electronic medical record (EMR) data to measure quality of care.1 One mechanism for achieving these goals was the development, in 1998, of the VHA Health Services Research and Development Service's Quality Enhancement Research Initiative (QUERI), whose overall mission is to facilitate and support implementation of evidence-based practices to promote ongoing improvement in outcomes and clinical care delivery.2,3 As one of eight QUERI groups, the Mental Health QUERI (MH QUERI) has a primary mission to implement research findings and innovations related to major depressive disorder or schizophrenia to promote better patient care and systems improvements.4 This paper presents and discusses findings from two MH QUERI studies related to treatment of schizophrenia.
Background
Schizophrenia is widely considered the most debilitating and costly of mental illnesses, and many large health care organizations such as the VHA care for disproportionate numbers of persons with schizophrenia. The VHA treats approximately 100,000 patients with schizophrenia each year, who comprise 3.4% of all veterans who receive care annually at VHA facilities and clinics.5,6 These individuals' use of all health services (including mental and physical health care) accounts for 11.7% of total VHA health care costs.6
Decades of research have shown that four types of interventions can result in improved clinical outcomes for persons with schizophrenia: antipsychotic medication management, intensive case management, family psychoeducation, and psychosocial rehabilitation.7,8,9 High-quality medication management is a critical component of care for schizophrenia. This paper focuses on the use of the EMR to assess one aspect of prescribing, use of moderate antipsychotic doses, because of evidence that many patients do not receive guideline-recommended doses in routine care settings,5,10,11,12,13,14,15,16 and that use of recommended doses is associated with better symptom outcomes.8,17,18 Moreover, this prescribing practice is measurable (using standard “chart review” or electronic extraction of computerized data)19 and potentially modifiable.20,21 Thus, quality improvement efforts for patients with schizophrenia should assess and aim to improve antipsychotic dosing. It is important to note that there are other critical aspects of medication management for schizophrenia, such as prescribing clozapine for treatment of refractory cases and side effect monitoring and management,7 that should be included in quality assessment and improvement.
This paper describes two studies that examined the use of VHA's automated information system, the Veterans Health Information Systems and Technology Architecture (VistA),22,23,24 to assess the extent that antipsychotic prescribing adhered to recommendations of evidence-based guidelines for schizophrenia. VistA is the integrated electronic record system for the VHA that contains, among other information, records of inpatient stays and outpatient visits, diagnostic codes, Current Procedural Terminology codes,25 pharmacy records, and laboratory test results. In addition, the EMR system includes order entry functions for tests and prescriptions and image storage and display functions. VistA also stores electronically entered progress notes for all patients. All this information, except the content of narrative progress notes, can readily be tabulated by means of automated queries.
The use of automated data to assess aspects of health care quality has a number of clear advantages over more traditional medical record review approaches, but it is also important to understand the strengths and limitations of using administrative or clinical databases to monitor and improve the quality of health care.26 Such issues have been examined for conditions other than schizophrenia including depression27 and diabetes.28 The first study presented in this paper addresses the feasibility and validity of using data electronically extracted from VistA to monitor adherence to evidence-based recommendations for antipsychotic dosing for inpatient and outpatient treatment of schizophrenia. The second study describes the application of such methods in a MH QUERI project that tested an intervention to improve antipsychotic prescribing using data obtained from the South Central VA Healthcare Network (SCVAHCN) Data Warehouse, a repository of data derived in part from the VistA system.
Defining Health Care Performance Measures for Schizophrenia
Our previous work in this area17 has described the development and initial application of a clinical performance measure that assesses providers' adherence to the evidence-based antipsychotic dose range of 300–1,000 mg chlorpromazine equivalents (CPZE)7 for the treatment of acute schizophrenia, showing that such an approach is feasible.19 A survey of VHA mental health experts also found that measurement of antipsychotic dosing within guideline-recommended ranges, which some guidelines provide for individual antipsychotic agents while others convert doses for drugs to the CPZE for comparison, is considered very meaningful and valid.29 We have established adequate sensitivity and specificity (concurrent validity) for the dosing criterion using structured implicit review methods as the gold standard.30 Our findings suggested that the measure of adherence to guideline-recommended dose ranges could best be employed as a screen for potentially inappropriate dosing rather than to compare performance of providers, facilities, and health care systems.30 We have also demonstrated that the 300–1,000 CPZE dosing criterion has adequate predictive validity when applied to patients treated with oral antipsychotics, in that patients meeting this criterion had better 6-month symptom outcomes.17 For patients on any antipsychotic medication, including long-acting injectable antipsychotic medication, the relationship between dose range and outcome was not significant, although mean symptom severity ratings were still lower for those patients prescribed total daily doses within the 300–1,000 CPZE range.17 Thus, further investigation is needed to determine the utility of the dosing criterion in assessing long-acting injectable (depot) antipsychotic medication prescribing.17 In addition, development of performance measures for other aspects of medication management and for nonmedication therapies is needed as well as systems to monitor clinical outcomes with validated instruments.31
Measuring Guideline Performance for Inpatient and Outpatient Treatment: Study 1
We recently conducted a study to determine the feasibility and validity of using data electronically extracted from VistA to monitor guideline performance for inpatient and outpatient treatment of schizophrenia. We compared such data to data abstracted by trained raters' manual review of the EMR including the narrative of progress notes. The next section of this paper provides an overview of this multisite study and presents the methods and findings.
Study Overview
Three VA medical centers (VAMCs) that provide mental health specialty care participated in the study: the Central Arkansas Veterans Healthcare System, the VA Salt Lake City Health Care System, and the VA Puget Sound Health Care System. The study was approved by each site's Institutional Review Board. VistA data were electronically extracted and measures of medication treatment were compared with analogous data manually abstracted from review of written medical records for a random sample of 261 patients (113 outpatients, 148 inpatients). Data elements and performance measures compared between the two methods included: (1) whether the patient was prescribed antipsychotic medication, (2) whether the patient received depot antipsychotic medication injections, and (3) whether the antipsychotic dose prescribed was concordant with treatment guidelines for schizophrenia (e.g., above, below, or within the recommended range of 300–600 CPZE for outpatients and 300–1,000 CPZE for inpatients).7
Patient Selection and Automated Extraction of VistA Data
Data for this study including dates, diagnoses, and Global Assessment of Functioning (GAF) scores for outpatient encounters and inpatient stays were extracted from VistA files. Data were also extracted from VistA pharmacy files, which contain information on both inpatient medications and outpatient prescriptions dispensed to patients by the VAMC pharmacy. Prescription data are entered by pharmacy personnel primarily to fill a prescription. Drug data are stored according to the drug's generic name, trade name, and drug classification. Prescription data include medication generic and trade names and strength and quantity dispensed as well as days supply and number of refills available for that prescription.
We queried the VistA database using M routine, formerly the Massachusetts General Hospital Utility Multiprogramming System (MUMPS), to identify all patients with a diagnosis of schizophrenia (ICD-9 codes 295.1–295.3, 295.6, 295.8, or 295.9) who received inpatient or outpatient treatment at one of the three sites between September 30, 1998 and August 1, 1999 inclusive. The automated data were then consolidated and grouped by patient to ensure that each patient was included only once in the dataset and to account for multiple care episodes per patient. From this dataset, 150 inpatient cases were randomly selected for review (Central Arkansas, 50 cases; Puget Sound, 48 cases; Salt Lake City, 50 cases; two patients had incomplete data for analysis). To select the outpatient cases, we narrowed the above criteria by requiring that eligible patients have two GAF scores at two different visits logged in VistA during the study interval, permitting assessment of adherence to another performance measure (whether medication was changed if GAF score was low or decreased). A total of 113 outpatients were then randomly selected from this pool for inclusion in the study (25 from Central Arkansas, 63 from Puget Sound, 25 from Salt Lake City). The sample size of outpatients at Puget Sound was larger because the initial sample included some patients who did not have GAF scores recorded at two consecutive visits. Therefore, additional subjects were randomly selected until 25 were identified who met this inclusion criterion for a separate analysis. Initial sample sizes were selected based on a priori power calculations. The actual sample sizes for inpatients and outpatients (148 and 113, respectively) allowed for approximately 80% power to detect a kappa of 0.30 and 0.35, respectively at 0.05 level of significance. For the selected patient sample, a separate M program routine extracted pharmacy data for the study period.
The index antipsychotic prescription at hospital discharge (for inpatients) was defined as the outpatient antipsychotic prescription filled closest to the discharge date and within the window from 7 days before to 30 days after discharge. For outpatients, the index antipsychotic prescription was defined as the prescription filled either on the day of the first visit with a GAF score or as the most recent prescription within the preceding 30 days. If any other antipsychotic prescriptions were filled within 7 days of the index date, these were also included in the daily dose calculation for both inpatient and outpatient groups. Long-acting injectable antipsychotic medications were noted but were not included in the CPZE calculations because insufficient information was available to determine dose and frequency of administration.
Total daily antipsychotic dose was calculated from VistA data using the drug strength in milligrams and duration of supply in days. We estimated daily dose by multiplying drug strength by quantity supplied and dividing by duration of supply. Daily antipsychotic doses were converted to CPZE for comparison.17 For each patient, we determined whether total daily antipsychotic dose was below, within, or above the Patient Outcomes Research Team (PORT) guideline-recommended range of 300–1,000 CPZE per day for inpatients and 300–600 CPZE for outpatients.7
Medical Record Abstraction
For each of the cases selected for this study, three trained research assistants (one at each VAMC) used standardized chart abstraction forms (for both inpatients and outpatients) developed by the research team to manually review and abstract medical record data from EMRs, using the CPRS interface. The CPRS is a user-friendly graphical interface program tied to the VistA database system. The data abstracted from narrative progress notes that are displayed in the CPRS cannot be readily extracted by means of an automated query. Chart abstraction forms are available from the first author (RRO).
The research assistants reviewed CPRS files including progress note sections of the medical record (narrative text) and medication orders to abstract data to compare with analogous data extracted from computable data fields in VistA. The research assistants were blind to the automated data electronically extracted from VistA. Prior to collecting medical record data, adequate interrater reliability among the three trained research assistants was established for the key chart abstraction data elements using 10 practice cases.32 Hard-copy charts were examined for a subsample of patients. Although these charts sometimes contained additional narrative material, we found sufficient information to complete the chart abstractions in CPRS, and the hard-copy chart material did not reveal new information.
Sample
Of the 113 outpatients, 98% (111) were male , 78% (89) white, 10% (11) African American, 3% (three) Asian, 1% (one) Hispanic, and ethnicity was missing in the dataset for 8% (nine). The average age for outpatients was 51.9 years (SD=10.4). Of 148 inpatients, 98% (145) were male, 76% (112) white, 17% (25) African American, 1% (two) Asian, 2% (three) Hispanic, and ethnicity was missing in the dataset for 4% (six). The average age for inpatients was 52.3 years (SD=12.2).
Results
Presented in ▶ are the results for agreement between data electronically derived from VistA using M programming language and analogous data manually derived from chart review for guideline-concordant antipsychotic dosing. According to both medical record review and VistA data, approximately one-fourth of both inpatients and outpatients were receiving doses of antipsychotic medication above the guideline-recommended range. As in previous studies,15,17 quality of care assessments using both automated and manual chart review data for veterans with schizophrenia revealed that nearly half of patients were prescribed antipsychotic doses outside the guideline-recommended dose range.
Table 1.
Categorization of Patients with Respect to the Guideline-Recommended Antipsychotic Dose Range: Agreement between Automated VistA and Medical Record Data Sources
| Measure | VistA n (%) | MRR n (%) | VistA/MRR Agree n (%) | Kappa (Weighted) |
|---|---|---|---|---|
| Inpatient (n=88) | ||||
| < 300 CPZE | 19 (22) | 15 (17) | 10 (11) | — |
| 300–1,000 CPZE | 50 (57) | 43 (49) | 36 (41) | — |
| >1,000 CPZE | 19 (22) | 30 (34) | 18 (20) | — |
| Inpatient total agree | — | — | 64 (73) | 0.55 |
| Outpatient (n=43) | ||||
| < 300 CPZE | 12 (28) | 10 (23) | 8 (19) | — |
| 300–600 CPZE | 10 (49) | 12 (28) | 9 (21) | — |
| >600 CPZE | 21 (23) | 21 (49) | 17 (40) | — |
| Outpatient total agree | — | — | 36 (84) | 0.63 |
MRR=medical record review; CPZE=chlorpromazine equivalents.
▶ presents results for agreement in documentation of the use of long-acting injectable (depot) antipsychotic medications between the VistA and chart review data sources. The written medical record review indicated that 30 inpatients and 15 outpatients were prescribed depot antipsychotics, but only five of the inpatients and one outpatient had a corresponding prescription for depot antipsychotic recorded in VistA pharmacy files.
Table 2.
Agreement between Automated VistA and MRR Methods for Antipsychotic Medication Prescriptions and Use of Depot Medications
| Measure | VistA n (%) | MRR n (%) | VistA & MRR Agree n (%) | MRR+VistA − n (%) | MRR − VistA+n (%) | Kappa (Not Weighted) |
|---|---|---|---|---|---|---|
| Antipsychotic medication | ||||||
| Inpatient total | 95 (64) | 127 (86) | 104 (70) | 38 (26) | 6 (4) | 0.25* |
| Outpatient total | 59 (52) | 79 (70) | 67 (59) | 33 (29) | 13 (12) | 0.17† |
| Depot medication | ||||||
| Inpatient total | 5 (3) | 30 (20) | 121 (82) | 26 (18) | 1 (1) | ‡ |
| Outpatient total | 1 (1) | 15 (13) | 99 (88) | 14 (12) | 0 (0) | ‡ |
MRR = medical record review.
There were 148 inpatients and 113 outpatients.
McNemar's test = 23.3, df = 1, p < 0.0001.
McNemar's test=8.7, df=1, p < 0.0032.
Not computable.
Consideration of Data Quality Issues
In this first study, measures of guideline-concordant antipsychotic dosing derived from data electronically extracted from pharmacy files in VistA showed moderate agreement for inpatients (kappa=0.55) and substantial agreement for outpatients (kappa=0.63) with measures obtained through abstraction of the written portion of the EMR.33 This study found that the major deficiency of the extracted VistA pharmacy data for assessing guideline-concordant antipsychotic dose resulted from the underrecording of long-acting injectable antipsychotic prescriptions in the pharmacy database. This finding was somewhat expected because these medications are primarily administered from clinic stock and are only occasionally dispensed directly by the pharmacy to the patient. Thus, patients prescribed a depot antipsychotic alone are likely to be excluded from the automated antipsychotic dose performance measure calculation. For those prescribed both oral and depot antipsychotic medications, the electronic data extraction and analysis will underestimate actual antipsychotic dose, resulting in an underestimate of the performance measure that examines the proportion of patients with schizophrenia who are prescribed above the guideline-recommended antipsychotic dose range.
Application of Methods in a Quality Improvement Project: Study 2
Overview of Mental Health QUERI Translation Project
The findings described above informed the methods of a QUERI-funded quality improvement demonstration project. The project, using a multicomponent intervention, aimed to implement evidence-based recommendations for antipsychotic treatment of schizophrenia into clinical practice. Briefly, the multicomponent intervention consisted of providing a package of educational and clinical tools to all sites (pocket cards, guidelines fact sheets, and patient activation materials) as well as a program of evidence-based behavioral change strategies implemented only at the translation sites including recruitment and training of opinion leaders, external facilitation, reminder messages about recommended dose that appear in the computerized pharmacy ordering screen, and monthly feedback of performance measures and related data derived from computerized pharmacy and service utilization files. A quasi-experimental design was used to evaluate the impact of the intervention on (1) the proportion of patients receiving antipsychotic doses in excess of guideline-recommended doses and (2) the proportion of inpatients admitted on conventional antipsychotics who were discharged on newer agents. To illustrate the use of computerized data for quality improvement in this project, the current paper focuses on the methods and baseline results for the first measure (proportion of patients receiving antipsychotic doses above the upper limit of the guideline-recommended range). Eight VA facilities were matched as pairs according to their baseline antipsychotic-prescribing patterns, number of patients with schizophrenia, and number of mental health providers. One randomly selected site in each pair received the multicomponent intervention.
Data Collection and Analysis
MH QUERI used data electronically extracted from the SCVAHCN Data Warehouse to calculate performance measures at baseline and for feedback to providers in monthly reports. The SCVAHCN Data Warehouse is a data repository that consolidates clinical and administrative data extracted monthly from the VistA systems at each of the ten VA medical centers and associated outpatient clinics in SCVAHCN. The Data Warehouse, containing information on 1.7 million patients treated over six fiscal years, includes data elements related to inpatient and outpatient services, radiographic and surgical procedures, outpatient prescriptions dispensed, laboratory results, and patient demographics.
Patients were included in the baseline sample if they had received ICD-9 diagnoses of schizophrenia (295.x) at least twice in general psychiatry outpatient mental health clinics (with diagnoses recorded in VistA's outpatient encounters file) or at least once on discharge from psychiatry inpatient bed sections from June 1, 2000 to May 31, 2001 (with diagnosis recorded in VistA's Patient Treatment File) and if they had a prescription for an antipsychotic medication between March 1, 2001 and May 31, 2001. The most recent antipsychotic prescription in this three-month window was classified as the index prescription. Any prescriptions for other antipsychotic medications within seven days prior to the index prescription date were included in the analysis according to the methods of Leslie and Rosenheck.5 Total daily doses for each antipsychotic were estimated by multiplying milligram strength by quantity supplied and dividing by the number of days supplied as indicated in the prescription data. These results were compared with dose ranges for each antipsychotic that are recommended in the Schizophrenia Treatment Algorithm of the VHA's Clinical Guidelines for Management of Persons with Psychoses.34 Patients were classified as receiving doses above the recommended range if they were prescribed daily doses of one or more antipsychotic agents that were above the upper limit of the recommended range for that drug. VHA guidelines provide specific dose ranges in milligrams for moderate doses that are most efficacious,34 which are analogous to the recommended ranges in the Schizophrenia PORT Treatment Recommendations7 used in study 1. The proportion of patients at each site who were receiving antipsychotic doses above the guideline-recommended range was included in monthly performance reports, created electronically by MH QUERI, and then fed back to opinion leaders at each translation site to use as part of the intervention.
Because the results of study 1 indicated that depot antipsychotic prescriptions were often not recorded in VistA and thus would not be available from the SCVAHCN Data Warehouse, the MH QUERI team conducted a manual chart review of CPRS progress notes to obtain this information. Each site provided a list of patients receiving depot antipsychotic medication at baseline, and chart abstractions were completed to determine antipsychotic doses of both oral and depot medications for these patients. Chart review was conducted using the same methods described in study 1 above. Daily antipsychotic doses were converted to CPZE as described in Owen et al17 so that doses for patients receiving both oral and depot medications could be combined and compared with the recommended dose range for chlorpromazine (300–1,000 mg/day).7 To determine the extent to which depot underrecording in VistA would affect the high-dose measure result derived from the Data Warehouse, we compared the measure results from the dataset, which combined chart review documentation of depot antipsychotic use and Data Warehouse prescription data for oral antipsychotics to the results obtained using the Data Warehouse data source alone.
Results
The total sample for this analysis consisted of 4,424 patients who met the baseline diagnostic criteria (mentioned above) at the eight participating facilities. Of these 4,424 patients, 11.9% (528 patients) were prescribed depot antipsychotic medication (either alone or with oral antipsychotics) at baseline according to combined data from the SCVAHCN Data Warehouse and chart review. Using the Data Warehouse dataset only (4,266 patients), 2.6% (113) of the total sample was identified as receiving depot medication. The extent of electronic recording of depot prescriptions varied considerably at the eight sites (range, 1%–45%).
Estimation of the number of patients prescribed high antipsychotic doses using the combination of both data sources showed that 15.1% (668 patients) were prescribed antipsychotic doses above the recommended range. In contrast, analysis of data available in the Data Warehouse showed that of 4,266 patients, 10.7% (455) were prescribed above-recommended doses. The prevalence of high dosing across the eight sites ranged from 4.8% to 21.1% when both chart review and Data Warehouse sources of information were combined and from 4.0% to 14.8% when only the Data Warehouse was used to estimate high dosing. The site's ranks on the high-dose measure were almost identical whether the Data Warehouse or combination method was used. The only discrepancy in ranks between the two approaches (combined data versus Data Warehouse only) was caused by the site that had the most dramatic underrecording of depot prescriptions, which was ranked one position lower with respect to extent of high-dose prescribing when Data Warehouse data were used to calculate the measure compared with its rank when both data sources were combined. These results suggest that, while the data derived from VistA and stored in the Data Warehouse can provide an approximate indication of guideline-concordant antipsychotic dosing and could be used to identify potential problems, the underrecording of depot antipsychotic use represents a substantial limitation at the present time.
Discussion and Implications
Our first study found that, for patients prescribed oral antipsychotics alone, the measure of the proportion of patients prescribed antipsychotic doses above the guideline-recommended range that was derived from electronically extracted VistA data agreed at an acceptable level with the same measure determined by manual review of progress notes and orders in the written EMR. However, substantial underrecording of prescriptions for depot antipsychotic agents limited the accuracy of VistA data–derived estimates of the proportion of patients prescribed any antipsychotic agent, the proportion of patients prescribed a depot agent, and the proportion of patients prescribed antipsychotic doses above the guideline-recommended range.
Our second study, which tested a quality improvement intervention to reduce antipsychotic doses that were above the guideline-recommended range, confirmed and extended these findings. By systematically identifying patients who were prescribed depot medication and abstracting data from their progress notes, we determined that the extent of underrecording of depot prescriptions in the VistA pharmacy files varied dramatically among the medical centers in our study. Moreover, this study determined that this limitation affects estimates of the proportion of patients receiving depot agents (2.6% compared with 11.9% observed using review of written EMR). Because patients receiving depot antipsychotics are more likely to be prescribed high antipsychotic doses,35 this limitation also results in an underestimate of the performance measure screening approach for above-range antipsychotic doses (10.7% versus 15.1%). Thus, the automated data would not identify a substantial proportion of these patients. Finally, depot underrecording itself varied among the eight study sites, so that the above-range dose measure must be interpreted with caution when comparing results across sites. Naturally, the impact of this limitation on performance measurement and quality improvement is proportional to the prevalence of depot antipsychotic use in the population of interest as well as the extent of underrecording in a given facility or clinic.
Other possible explanations for disagreement between EMR and chart measures might be due to patients who received prescriptions from non-VA sources, inaccurate recordings of current medications in the narrative text, or changes in providers instructions without corresponding change in electronically entered prescriptions. In a VA study of patients with depression,27 few subjects in the sample were found to be receiving prescriptions elsewhere. Other studies have also shown out-of-system use to be infrequent among VA psychiatric patients.36
One implication of these studies is that electronic recording of depot antipsychotic prescriptions should be improved. For example, the provider order entry software available in the VA's EMR can be used to enter depot medication orders. If these orders are then recorded in the pharmacy software, as they were for some patients in our second study, they can be extracted from VistA or from the SCVAHCN Data Warehouse. Improving provider awareness of the importance of appropriate documentation for using this performance measure may help improve data quality as well. This improvement in data quality would not only benefit clinical performance monitoring and quality improvement efforts but also the ability of mental health providers, pharmacists, and clinical managers to track the use of long-acting injectable agents. While improved data recording is needed for accurate measurement of antipsychotic dosing, it also is important because specific data elements in the EMR can trigger reminders to the clinician at the point of care to comply with evidence-based practice guidelines. Such clinical reminder systems have been shown to be effective methods of increasing guideline compliance (see Cannon and Allen37 for a recent example from a VA mental health clinic). Electronic recording of all medication doses, including those administered in the clinic setting, would have implications for treatment settings other than mental health clinics as well. For example, recording clinic administration of insulin, antibiotics, and other medications could be useful to providers, clinic managers, and researchers.
The use of data electronically extracted from the EMR to monitor and improve guideline implementation has several advantages. It has the potential to replace review of the written medical record, electronic or otherwise, for performance measurement and quality improvement projects because it is more efficient. The ability to electronically extract and analyze EMR data makes it possible to collect a 100% sample of the cases that meet the performance measure inclusion criteria, which increases the reliability of performance measurement compared with review of a relatively small sample of written medical records. While our research showed that electronically extracted data derived from VistA was incomplete, our second study demonstrated that these data can be combined with a targeted and brief chart review to quantify depot antipsychotic doses as part of a quality improvement effort.
While the studies presented herein suggest that data stored in the VHA's extensive administrative and clinical databases can be used to assess whether antipsychotic doses prescribed to patients with schizophrenia conform to evidence-based guidelines, the findings must be understood in the context of the limitations of this measure. Most importantly, prescription of moderate antipsychotic doses represents only one treatment recommendation of many for this complicated illness. Improving antipsychotic dosing alone, while it may benefit some patients, will not obviate the need for improvement in other areas of medication management as well as in psychosocial rehabilitation and other aspects of management.
Neither of these studies collected clinical data to confirm that within-range dosing relates to better outcomes. Further research will be needed to demonstrate that VistA-derived measures of adherence to the guideline are associated with outcomes and that increasing the use of within-range doses will result in improved patient outcomes. A limitation of this approach is that, while aggregate outcomes ratings have been shown to be lower for patients prescribed doses within the guideline-recommended range,17 some patients may have adequate responses to doses outside the guideline-recommended range.30 In a previous study, we found that 43% of patients prescribed doses above the moderate dose range were rated by expert clinicians' implicit reviews as receiving appropriate antipsychotic dose management. The quality indicator for antipsychotic dose within the guideline-recommended range had 85% sensitivity for detecting inappropriate care as rated by the reviewers, but its specificity was only 72%. We suggested that a guideline-recommended dose indicator could serve as a screening tool for potentially inappropriate medication management, with further clinical review of cases who screened positive.30 Because antipsychotic dosing practices vary significantly among VHA facilities and other settings,5,35,38 this screening tool could identify facilities or clinics whose practices would benefit from further review. The power of this approach would be greatly enhanced if it could be combined with valid and reliable data on clinical and functional outcomes.
Conclusions
Taken together, the findings of our two studies suggest that data electronically extracted from VistA files can be used to estimate specific measures of quality of antipsychotic treatment, although caution must be exercised when interpreting the results. The findings also indicate the feasibility of combining data from electronic databases with targeted review of narrative-text notes in the EMR to monitor and perhaps improve the quality of one aspect of antipsychotic treatment. Our observations—that the majority of patients on depot agents receive doses above those recommended by evidence-based guidelines and that electronic recording of depot prescriptions was possible but usually incomplete—suggest that providers and facilities should improve recording so that automated data could be used to more accurately monitor and improve this important aspect of medication management for schizophrenia.
Supported by grants from the Department of Veterans Affairs HSR&D Service (Mental Health QUERI Coordinating Center, MNY 99-243, and MNT 01-033), and with funding from the VA South Central Mental Illness Research, Education and Clinical Center.
The authors thank Lisa Geisselbrecht, Paula Stock, Sandra Pineros, Anthony Diblasi, Keith Williams, Marty Curtis, Betsy Lancaster, and other members of the South Central VA Health Care Network (VISN 16) Information Technology Group for their assistance with this work.
The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
References
- 1.Feussner JR, Kizer KW, Demakis JG. The Quality Enhancement Research Initiative: from evidence to action. Med Care. 2000;38(suppl I):I-1–6. [DOI] [PubMed] [Google Scholar]
- 2.Demakis JG, McQueen L, Kizer KW, Feussner JR. Quality Enhancement Research Initiative (QUERI): a collaboration between research and clinical practice. Med Care. 2000;38(suppl I):I-17–25. [PubMed] [Google Scholar]
- 3.McQueen L, Mittman BS, Demakis JG. Overview of the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI). J Am Med Inform Assoc. 2004;11:339–43. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Fischer EP, Marder SR, Smith GR, et al. Quality Enhancement Research Initiative in mental health. Med Care. 2000;38(suppl I):I-70–81. [DOI] [PubMed] [Google Scholar]
- 5.Leslie DL, Rosenheck RA. Use of pharmacy data to assess quality of pharmacotherapy for schizophrenia in a national health care system: individual and facility predictors. Med Care. 2001;39:923–33. [DOI] [PubMed] [Google Scholar]
- 6.Wei Y, Ravelo A, Wagner TH, et al. Prevalence and costs of chronic conditions in the VA health care system. Med Care Res Rev. 2003;60:146S–147S. [DOI] [PubMed] [Google Scholar]
- 7.Lehman AF, Steinwachs DM. Translating research into practice: the Schizophrenia Patient Outcomes Research Team (PORT) treatment recommendations. Schizophr Bull. 1998;24:1–10. [DOI] [PubMed] [Google Scholar]
- 8.Baldessarini RJ, Cohen BM, Teicher MH. Significance of neuroleptic dose and plasma level in the pharmacological treatment of psychoses. Arch Gen Psychiatry. 1988;45:79–91. [DOI] [PubMed] [Google Scholar]
- 9.Dixon L. Providing services to families of persons with schizophrenia: present and future. J Ment Health Policy Econ. 1999;2:3–8. [DOI] [PubMed] [Google Scholar]
- 10.Rosenheck RA, Desai R, Steinwachs D, Lehman A. Benchmarking treatment of schizophrenia: a comparison of service delivery by the national government and by state and local providers. J Nerv Ment Dis. 2000;188(4):209–16. [DOI] [PubMed] [Google Scholar]
- 11.Segal SP, Bola JR, Watson MA. Race, quality of care, and antipsychotic prescribing practices in psychiatric emergency services. Psychiatr Serv. 1996;47:282–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Walkup JT, McAlpine DD, Olfson M, Labay LE, Boyer C, Hansell S. Patients with schizophrenia at risk for excessive antipsychotic dosing. J Clin Psychiatry. 2000;61:344–8. [DOI] [PubMed] [Google Scholar]
- 13.Young AS, Sullivan G, Burnam MA, Brook RH. Measuring the quality of outpatient treatment for schizophrenia. Arch Gen Psychiatry. 1998;55:611–7. [DOI] [PubMed] [Google Scholar]
- 14.Chen RS, Nadkarni PM, Levin FL, Miller PL, Erdos J, Rosenheck RA. Using a computer database to monitor compliance with pharmacotherapeutic guidelines for schizophrenia. Psychiatr Serv. 2000;51:791–4. [DOI] [PubMed] [Google Scholar]
- 15.Lehman AF, Steinwachs DM. Patterns of usual care for schizophrenia: initial results from the Schizophrenia Patient Outcomes Research Team (PORT) client survey. Schizophr Bull. 1998;24:11–20. [DOI] [PubMed] [Google Scholar]
- 16.Galletly CA. Antipsychotic drug doses in a schizophrenia inpatient unit. Aust N Z J Psychiatry. 1992;26:574–6. [DOI] [PubMed] [Google Scholar]
- 17.Owen RR, Thrush CR, Kirchner JE, Fischer EP, Booth BM. Performance measurement for schizophrenia: adherence to guidelines for antipsychotic dose. Int J Qual Health Care. 2000;12:475–82. [DOI] [PubMed] [Google Scholar]
- 18.Sohler NL, Walkup J, McAlpine D, Boyer C, Olfson M. Antipsychotic dosage at hospital discharge and outcomes among persons with schizophrenia. Psychiatr Serv. 2003;54:1258–63. [DOI] [PubMed] [Google Scholar]
- 19.Hudson TJ, Owen RR, Lancaster AE, Mason L. The feasibility of using automated data to assess guideline-concordant care for schizophrenia. J Med Syst. 1999;23:299–307. [DOI] [PubMed] [Google Scholar]
- 20.Lantz MS, Giambanco V, Buchalter EN. A ten-year review of the effect of OBRA-87 on psychotropic prescribing practices in an academic nursing home. Psychiatr Serv. 1996;47:951–5. [DOI] [PubMed] [Google Scholar]
- 21.Ray WA, Taylor JA, Meador KG, et al. Reducing antipsychotic drug use in nursing homes a controlled trial of provider education. Arch Intern Med. 1993;153:713–21. [PubMed] [Google Scholar]
- 22.Hynes DM, Joseph G, Pfiel C. Veterans Health Information System and Technology Architecture (VistA) as a Research Tool, Vol. 3, No. 1. Hines, IL: VIReC, 2002.
- 23.Department of Veterans Affairs, Office of Information. VistA Monograph, 2003-2004. Department of Veterans Affairs, Office of Information, September, 2002. Available at http://vista.med.va.gov/monograph/2002/12_02_03-04.pdf. Accessed September 16, 2003.
- 24.Kolodner RM. Computerizing Large Integrated Health Networks: The VA Success. New York, NY: Springer-Verlag, 1997.
- 25.American Medical Association. Physicians' Current Procedural Terminology, Standard Edition. Chicago, IL: American Medical Association, 1998.
- 26.Iezzoni L. Assessing quality using administrative data. Ann Intern Med. 1997;127(8 pt 2):666–74. [DOI] [PubMed] [Google Scholar]
- 27.Kramer TL, Owen RR, Cannon D, Sloan KL, Thrush CR, Williams DK, Austen MA. How well do automated performance measures assess guideline implementation for new-onset depression in the Veterans Health Administration? Jt Comm J Qual Saf. 2003;29:479–89. [DOI] [PubMed] [Google Scholar]
- 28.Jones D, Hendricks A, Comstock C, et al. Eye examinations for VA patients with diabetes: standardizing performance measures. Int J Qual Health Care. 2000;12:97–104. [DOI] [PubMed] [Google Scholar]
- 29.Owen RR, Cannon D, Thrush CR. Mental Health QUERI Initiative: expert ratings of criteria to assess performance for major depressive disorder and schizophrenia. Am J Med Qual. 2003;18:15–20. [DOI] [PubMed] [Google Scholar]
- 30.Owen RR, Thrush CR, Hudson TJ, et al. Using explicit guideline-based criteria and implicit review to assess antipsychotic dosing performance for schizophrenia. Int J Qual Health Care. 2002;14:199–206. [DOI] [PubMed] [Google Scholar]
- 31.Cuffel BJ, Fischer EP, Owen RR. Jr, Smith GR Jr. An instrument for measurement of outcomes of care for schizophrenia: issues in development and implementation. Eval Health Professions 1997;20:96–108. [DOI] [PubMed] [Google Scholar]
- 32.Owen RR, Cannon D. Documentation of major depressive disorder- and schizophrenia-related performance criteria. Final report for project MNY 99-243, submitted to VA Health Services Research and Development Service, October 2002.
- 33.Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. [PubMed] [Google Scholar]
- 34.Mental Health Strategic Health Care Group, The Psychosis Working Group, Office of Performance and Quality. Veterans Health Administration clinical guideline for management of persons with psychoses. Washington, DC: Department of Veterans Affairs, 1997.
- 35.Owen RR, Fischer EP, Kirchner JE, et al. Clinical practice variations in prescribing antipsychotics for patients with schizophrenia. Am J Med Qual. 2003;18:140–6. [DOI] [PubMed] [Google Scholar]
- 36.Desai RA, Rosenheck RA, Rothbard A. Cross-system service use among VA mental health patients living in Philadelphia. Adm Policy Ment Health. 2001;28:299–309. [DOI] [PubMed] [Google Scholar]
- 37.Cannon DS, Allen SN. A comparison of the effect of computer and manual reminders on compliance with a mental health clinical practice guideline. J Am Med Inform Assoc. 2000;7:196–203. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Owen RR, Feng W, Thrush CR, Hudson TJ, Austen MA. Variations in prescribing practices for novel antipsychotic medications among Veterans Affairs hospitals. Psychiatr Serv. 2001;52(11):1523–5. [DOI] [PubMed] [Google Scholar]