Electronic medical record system at an opioid agonist treatment programme: study design, pre-implementation results and post-implementation trends

Abstract

Rationale

Electronic medical record (EMR) systems are commonly included in health care reform discussions. However, their embrace by the health care community has been slow.

Methods

At Addiction Research and Treatment Corporation, an outpatient opioid agonist treatment programme that also provides primary medical care, HIV medical care and case management, substance abuse counselling and vocational services, we studied the implementation of an EMR in the domains of quality, productivity, satisfaction, risk management and financial performance utilizing a prospective pre- and post-implementation study design.

Results

This report details the research approach, pre-implementation findings for all five domains, analysis of the pre-implementation findings and some preliminary post-implementation results in the domains of quality and risk management. For quality, there was a highly statistically significant improvement in timely performance of annual medical assessments (P < 0.001) and annual multidiscipline assessments (P < 0.0001). For risk management, the number of events was not sufficient to perform valid statistical analysis.

Conclusions

The preliminary findings in the domain of quality are very promising. Should the findings in the other domains prove to be positive, then the impetus to implement EMR in similar health care facilities will be advanced.

Introduction

Despite decades of predictions that the electronic medical record (EMR) revolution is coming, most health care organizations still use paper charts and manual processes. The transformation to an electronic platform has been promoted to reduce costs, provide better patient care and services and dramatically improve outcomes. A recent survey of EMR implementation published in The New England Journal of Medicine indicated that only 4% of doctors have a fully functional system, with 13% having a basic system [1]. This means a majority of health care transactions are still on paper, which has been a practice since the 1950s. For the component of the US health system delivering substance abuse services, the picture is even worse.

Because published evaluations of the implementation of integrated EMRs in substance abuse treatment programmes are virtually non-existent, we report in this paper the design, some baseline assessments and some preliminary post-implementation findings. Such information has the potential to inform decision making for both providers and policy makers.

Materials and methods

Setting

The Addiction Research and Treatment Corporation (ARTC) is a community-based not-for-profit 501(c)(3) corporation treating substance abusers in Brooklyn and Manhattan since 1969. ARTC is one of the largest minority-operated substance abuse treatment organizations in the nation, and the largest non-hospital-based opioid treatment programme (OTP) in New York State, serving more than 3000 patients annually. Since inception, ARTC has provided a wide range of comprehensive health care including primary medical care, HIV/AIDS care and substance abuse treatment services to over 30 000 patients throughout Brooklyn and Manhattan.

Addiction Research and Treatment Corporation’s seven OTP clinics are CARF-accredited, and ARTC is dually licensed by the New York State Office of Alcoholism and Substance Abuse Services for substance abuse treatment and the New York State Department of Health under Article 28 regulations for primary medical services, including HIV/AIDS care and case management.

Despite this history and current capacities, considerable challenges remain. These include a largely paper-based information system without any integration of clinical, fiscal and administrative data. The only major components of ARTC’s operations where information is stored in an electronic database are selected counselling and medical services, methadone administration/dispensing data and billing. Even these areas are not thoroughly integrated, and any assessment of the quality or integrity of these sub-systems of information is limited.

Study design

Parallel with the decision by the agency to implement an EMR, the agency received a grant from the National Institute on Drug Abuse to conduct this prospective, comparative study utilizing a pre- and post-implementation design to determine whether there were improvements post implementation, based on needs assessment meetings described later. The pre-implementation period was defined as from 1 July 2006 to 30 June 2007, and was chosen to have a sufficient amount of patients enrolled just prior to the commencement of plans to implement the EMR. The post-implementation period was defined as from 1 November 2009 to 31 October 2010 and reflects the 12-month period following installation and training of all staff at ARTC’s clinical sites and administrative headquarters.

The authors conducted needs assessment meetings with agency stakeholders (patients, direct-care providers and supervisors/managers) to choose the specific aims of: (1) quality; (2) productivity; (3) satisfaction; (4) risk management; and (5) financial performance. Every effort was made to control for extraneous variables, such as staff turnover and patient demographic changes between the pre- and post-implementation periods that might confound the analysis. For each of the specific aims, there were at least two hypotheses. As shown in Table 1, for quality we proposed five quality-related hypotheses that post implementation there would be improvements in the timeliness of completion of patient (1) annual medical assessments; (2) 30-day multidiscipline assessments; (3) 90-day multidiscipline assessments; (4) annual multi-discipline assessments; and (5) assessments for hepatitis C (HCV) viral load in those patients with a positive HCV antibody test.

Table 1.

Specific aims, measures, data sources and desired outcomes

Specific aim	Measure	Data source	Desired outcome
Quality	hepatitis C viral load	Patient chart	Increased # (%) clinically indicated tests
	Annual medical assessments	Patient chart	Completion ± 30 days of 1-year anniversary
	30-day multidiscipline assessments	Patient chart	Completion ≤ 30 days of admission
	90-day multidiscipline assessments	Patient chart	Completion ≤ 90 days of admission
	Annual multidiscipline assessments	Patent chart	Completion ≤ 365 days of anniversary
Productivity	Counselling visits per clinician	Clinician logs	Increased average per clinician
	Primary care visits per clinician	Clinician logs	Increased average per clinician
	HIV case management visits per clinician	Clinician logs	Increased average per clinician
Satisfaction	Satisfaction of patients	Survey	Increased overall satisfaction score
	Satisfaction of clinicians/managers	Survey	Increased overall satisfaction score
Risks	Complaints, incidents, medication errors	Surveillance system	Decreased frequency
Financial performance	Revenue per capita staff	Billing data	Long-term increase
	Cost per visit	Billing data	Long-term decrease

For the productivity specific aim, three hypotheses were advanced. We proposed that post implementation, the annual number of visits per clinician would increase for (1) individual addiction counselling; (2) primary medical care; and (3) HIV-related case management.

For the satisfaction-related hypotheses, we proposed that post implementation overall satisfaction would increase for (1) patients and (2) clinical and management staff. For risk management, the hypothesis was that following implementation, there would be a decrease in the annual combined rate of patient complaints, incidents and medication errors. For the financial performance specific aim, we hypothesized that post implementation (1) revenue per capita staff per annum would increase and (2) costs per visit per annum would decrease.

Study population

For the satisfaction-related hypotheses, patients were recruited proportional to the census at each of the seven clinics, using a convenience sampling technique and resulting in 1000 participants of the nearly 2800 patients. Patients received a $4 MetroCard (for public transportation) for their time and inconvenience. All patient-related information was eligible for inclusion in assessing risk management, financial and productivity-related specific aims. However, for the quality-specific aims, patients were eligible only if admitted during either the pre-implementation period or the post-implementation period. For the study, 148 staff members (direct care and supervisors/managers) from the seven clinical and central administrative sites in New York City were eligible to participate to investigate the satisfaction-related hypotheses. Of these, 99 (66.9%) participated.

Data sources and data collection

The investigators developed case report forms to collect data to test the quality-related hypotheses. As shown in Table 1, paper patient charts provided the pre-implementation information and electronic patient charts provided the post-implementation information.

For three specific aims (productivity, risk management and financial performance) various clinical logs and spreadsheets in mixed paper and electronic formats provided the pre-implementation data, while this same information was provided in only electronic formats post implementation. Patient, clinician and management stakeholders participated in completing an anonymous written survey for the satisfaction specific aim. For patients, the survey instrument consisted of six questions using a 1–5 Likert scale (not satisfied, slightly satisfied, somewhat satisfied, satisfied and very satisfied). For staff, the survey instrument consisted of 17 questions using the same 1–5 Likert scale as used with the patient survey. The domains of the patient and staff satisfaction survey are displayed in Table 2. Figure 1 provides a flowchart detailing the data collection process.

Table 2.

Patient and staff satisfaction survey items

Survey items	Study population
	Patients	Staff
Overall patient record organization		✓
Ability to access patient information system		✓
Reliability, integrity and efficiency of information flow		✓
Ability to communicate information internally and externally	✓	✓
User friendliness of the system		✓
Quality of care provided	✓	✓
Productivity		✓
Wait time for service	✓
Direct contact time	✓	✓
Overall satisfaction	✓	✓

Figure 1.

Pre- and post-implementation data collection flowcharts. HCV, hepatitis C virus; LOS, length of stay; CRF, case report form.

Statistical analysis

For continuous outcomes, the anticipated sample sizes were sufficiently large enough for a minimal effect size detected with 80% power at two-sided alpha = 0.05 and 0.01. For binary outcomes, the anticipated sample sizes were sufficiently large enough for a minimal difference detected with 80% power at two-sided alpha = 0.05 and 0.01. Thus, the study is well powered to observe even small differences when comparing pre- and post-intervention data.

The proportion of patients receiving annual medical assessments within 30 days of the anniversary of their admission will be compared pre- and post-intervention using chi-squared exact tests with P-value < 0.05 to determine statistically significant differences between the two time periods. The proportion of patients receiving their 30-day, 90-day and annual multidiscipline assessments on or prior to the due date will be compared pre- and post-intervention using chi-squared exact tests with P-value < 0.05 to determine statistically significant differences between the two time periods. The rate for obtaining HCV viral load will be measured quantitatively in percentage. The proportion of HCV antibody positive patients receiving HCV viral load testing will be compared pre- and post-intervention using chi-squared exact tests with P-value < 0.05 to determine statistically significant differences between the two time periods. For productivity, the mean number of addiction counselling, primary care and HIV case management visits per clinician will be compared pre-and post-intervention using t-tests for continuous variables with P-value < 0.05 to determine statistically significant differences between the two time intervals.

In addition to questions capturing patient satisfaction, demographic variables will also be collected to account for potential confounders. Other possible confounders to be assessed will include satisfaction in accessing care in other treatment settings and duration of substance abuse treatment. Each of these outcomes is binary; therefore, multiple logistic regression analysis will be used. With the use of a 1–5 Likert scale, the survey findings will be reported in ‘collapsed’ format as a percentage for satisfied and very satisfied, and as a continuous variable.

Ease of access and use will be two of the items to be included in the clinician/manager survey, adapted from a version previously published [2]. For categorical outcomes, chi-squared tests with P-value < 0.05 will be used to determine the statistically significant differences between the two time intervals. For binary outcomes, McNemar’s discordant pairs (matching before/after for each clinician), conditional logistic regression and other approaches for binary outcomes will be used for analysis.

The analysis for risk management will include examining the combined total of patient complaints, incidents and medication errors for appropriate 12-month periods using t-tests with P-value < 0.05 to determine statistically significant differences between time intervals. Cochran-Armitage test for trend will be used to determine significant decreases in the combined total of patient complaints, incidents and medication errors over time.

Revenue per capita and the cost per visit will be compared pre- and post-implementation of the EMR using t-tests with P-value < 0.05 to determine statistically significant differences between the two time intervals. In addition, significant increasing and/or decreasing trends over time for each measure will be conducted using Cochran-Armitage test for trend to determine statistically significant increases (or decreases) over time. Should costs be highly skewed, transformations (such as taking powers, logarithms or Winsorization of high values) will be considered.

As collection of the mixed paper and electronic data and survey material did not involve clinical interventions or gathering of protected health information, requirements for Health Insurance Portability and Accountability Act authorization were precluded. The ARTC Institutional Review Board approved this study protocol, surveys (including payment for patient participants) and case report forms prior to the study via expedited review and waiver of informed consent. This project was also exempt from the regulatory requirements for human subjects research under 45 CFR 46.101(b)(2).

The evaluation reported in this paper was in response to a National Institute on Drug Abuse Request for Applications (Enhancing Practice Improvement in Community-Based Care for Prevention and Treatment of Drug Abuse or Co-occurring Drug Abuse and Mental Disorders: RFA-DA-06-001).

Results

Pre-implementation (baseline) findings

During this interval, 772 patients were assessed for HCV infection with 670 patients testing positive for the HCV antibody (87% HCV antibody rate). HCV viral load testing was not offered to 27 patients and 4 patients refused HCV viral load testing.

Annual medical assessments were due for 420 patients. Of these, 348 (83%) were completed within 30 days of their anniversary. Of the remainder, 22 were not completed by the end of the pre-implementation period. There were 339 of 420 (81%) 30-day multidiscipline assessments, 192 of 420 (46%) 90-day multidiscipline assessments, and 294 of 420 (70%) annual multidiscipline assessments completed by their respective due dates. By the end of the pre-implementation period, three 30-day, 17 90-day and 38 annual multidiscipline assessments were not completed.

During the 12-month pre-implementation interval, there were 64 345 addiction-related individual visits or an average of 92.8 visits per month per counsellor, 5221 primary medical care visits or an average of 43.9 visits per month per medical staff and 2680 HIV-related case management visits or an average of 51.9 visits per month per case manager.

Table 3 displays many of the patient and staff responses to the satisfaction survey during the pre-implementation period. For the question ‘How satisfied are you with the overall quality of care you receive’, 74% of patients answered ‘satisfied’ or ‘very satisfied’, and the mean score was 3.86 (range 1–5; SD 0.99). For the question ‘How satisfied are you with the system overall’, 33% of staff answered ‘satisfied’ or ‘very satisfied’, and the mean score was 3.14 (range 1–5; SD 0.90).

Table 3.

Patient and staff pre-implementation satisfaction rates

Study population	Survey question	Number (%) satisfied/very satisfied
Patients (n = 1000)	Wait time to see doctor or doctor assistant	550 (55)
	Wait time to see counsellor	760 (76)
	Overall satisfaction	740 (74)
Staff (n = 99)	Quality of records and reports	50 (50)
	Work experience	48 (48)
	Overall satisfaction	33 (33)

Finally, during the pre-implementation period there were 64 patient complaints, 15 patient-related incident reports and 8 reports of medication errors during administration/dispensing of 584 126 medication doses. During this same period, financial performance revealed that the revenue per capita staff was $75 814 in 2006 and $66 900 in 2007 while the cost per patient visit was $31.45 in 2006 and $31.34 in 2007.

Selected post-implementation quality findings

Because the post-implementation interval had not expired as of submission of this manuscript, we performed a preliminary assessment of a few quality measures, comparing the first 6 months of the pre-implementation period (1/7/2006–31/12/2006) to the first 6 months of the post-implementation period (1/11/2009–30/4/2010). As shown in Table 4, there were statistically significant differences in the timely completion of annual medical assessments, demonstrating a rate of 82% of 194 eligible patients in pre-implementation period compared to a rate of 92% of the 143 patients in the post-implementation period. Similarly, the rates of timely completion of the annual multidiscipline assessments differed significantly between the pre-implementation and post-implementation intervals, 72% of 194 eligible patients in pre-implementation period and 93% of the 143 patients in the post-implementation period.

Table 4.

Pre- and post-implementation quality assessment rates

Measure	Study period	Total number of patients	Number (%) on-time completion	P-value*
Annual medical assessment†	Pre-implementation 1/7/2006–31/12/2006	194	159 (81.9)
	Post-implementation 1/11/2009–30/4/2010	143	132 (92.3)	<0.001
Annual multidiscipline assessment‡	Pre-implementation 1/7/2006–31/12/2006	194	140 (72.2)
	Post-implementation 1/11/2009–30/4/2010	143	133 (93)	<0.0001

^*Comparing pre- and post-implementation periods.

^†± 30 days of 1-year anniversary.

^‡≤365 days after admission.

Discussion

In order to better conceptualize the various clinical and management issues involved in implementing an EMR, a hierarchy of corporate objectives was devised, consisting of (from most to least importance): compliance with regulations, financial performance, quality of care, patient satisfaction and staff satisfaction. It is noteworthy that each of the five specific aims of this research was related to at least one of these objectives, indicating that the stakeholders, who had chosen these domains many months before this hierarchy was formally articulated, had an intuitive sense of the critical elements of the programme that needed to be addressed in developing the EMR.

In light of this hierarchy and years of quality assurance monitoring, the findings from the pre-implementation data collection yielded expected and unexpected information. Among the expected findings were (1) the relatively high timely completion rates of the annual medical and multidiscipline assessments; (2) the reasonably high rate of offering HCV viral load testing; (3) that patients were more satisfied with their care than staff were with the system in place for providing that care; and (4) that the risk management events were relatively small. Among the unexpected findings were (1) a higher number of missed medical and multidiscipline assessments than expected; (2) a relatively low timely completion rate of the 90-day multidiscipline assessments; and (3) the productivity of the clinicians, although this finding does not include the time spent by clinicians in group counselling or in interdisciplinary meetings among all clinicians or the addiction medicine-related visits provided by medical staff. Preliminarily, we report the results of some select findings comparing the first 6 months of the pre-implementation and the first 6 months of the post-implementation results, indicating improvements in the timely conduct of the annual medical and annual multidiscipline assessments.

Admittedly, this study design neither reflects a randomized clinical trial nor is there a control condition. However, such an approach has major disadvantages in many settings where clinical care is provided, including, but not limited to, ethical concerns about the withholding of potential benefits (such as more evidence-based practices) and the potential for patient and staff dissatisfaction about their exclusion from technological advances. The pre- and post-implementation design represents a reasonable approach in these circumstances.

User (patient or staff) receptivity and computer-related skills and experience may create difficulties and influence the findings of this study. Review of the literature of the impact of EMR indicates that these barriers are universal [2–6]. The substantial efforts by managers to promote buy-in by stakeholders, along with the research activities that were part of this undertaking ameliorated this issue. Computer-related skills and experience were taken into account as part of the pre-implementation satisfaction survey, and after the post-implementation surveys are carried out, the findings will be used to determine whether or not these factors should be included in the evaluation of quality and productivity post implementation.

We acknowledge that there is a potential for bias (unreported or under-reported events) in many data sources, especially when information is extracted from source documents (like clinical charts) to complete case report forms. We believe the inclusion of the perspectives of clinical staff in the planning and execution of the study served to limit the impact of these concerns.

Some may argue with the specific measures we chose to assess quality, productivity, satisfaction, risk management and financial performance. We believe that our choices were substantiated by the support these measures received by the stakeholders, their endorsement by New York State regulations and/or their use in many published studies [2–14]. The number of measures selected in this study was large compared to other studies cited, but the data to conduct the analyses were readily accessible.

The study intervention, implementation of an EMR, was not defined as specifically or rigorously as is the case in most clinical intervention studies, but the literature does not validate any particular type of EMR to be used across the varied medical and subspecialty, inpatient and outpatient settings, not to mention mental health and substance abuse treatment settings. At the time of this study, most EMR vendors chose not to focus on both medical and behavioural settings of care. It is hoped that this study will provide findings to not only support those ultimately chosen by ARTC, but may stimulate other investigators to replicate the study in other combined medical and behavioural (addiction and mental health) settings.

Another potential confounder that can occur, particularly in the risk management measure, is that post implementation, easier reporting and capture of patient complaints, incidents and medication errors might result in a greater number of reports. Unmeasured or unknown confounders, as well as changes in the field pre-and post-implementation of the EMR, might also affect the results. The study design, to the degree possible, sought to minimize these issues. As it turned out, however, the number of events was too small, and new processes for handling these events had not yet been incorporated into the EMR, necessitating that this measure be dropped from the pre–post statistical analysis.

It is possible that the results are not generalizable to other substance abuse treatment settings. ARTC is somewhat unique as an OTP providing onsite primary medical care and HIV-related services to a largely disenfranchised population that experiences significant disparities in access and quality of health care services. Therefore, some findings may be unique to the EMR ultimately chosen. However, these limitations were viewed as a potential strength in that this study assesses benefits that can accrue to a population that generally has not been included as participants in research. To minimize the impact of these issues, the current design included measures of interest to most modalities of addiction treatment. Additionally, the results may fuel other studies to assess the value of these findings of interest to other substance abuse treatment settings.

Finally, cost considerations and training logistics added to the amount of time needed to implement the EMR. This enhances the potential for the introduction of additional confounders, but all health care institutions must grapple with the allocation of scarce financial and human resources, so that the burden on ARTC was not any greater than for other agencies.

Conclusion

Based on preliminary findings and trends, we believe that implementation of an EMR at ARTC will prove to be successful for all stakeholders, and will serve as a template for other agencies providing similar services to underserved populations. Our patients and all patients receiving addiction-related and other medical services deserve no less.