Implementation of the Department of Veterans Affairs' first point-of-care clinical trial

Abstract

Objectives

The Massachusetts Veterans Epidemiology Research and Information Center in collaboration with the Stanford Center for Innovative Study Design set out to test the feasibility of a new method of evidence generation. The first pilot of a point-of-care clinical trial (POCCT), adding randomization and other study processes to an electronic medical record (EMR) system, was launched to compare the effectiveness of two insulin regimens.

Materials and Methods

Existing functionalities of the Veterans Affairs (VA) computerized patient record system (CPRS)/veterans health information systems and technology architecture (VISTA) were modified to support the activities of a randomized controlled trial including enrolment, randomization, and longitudinal data collection.

Results

The VA's CPRS/VISTA was successfully adapted to support the processes of a clinical trial and longitudinal study data are being collected from the medical record automatically. As of 30 June 2011, 55 of the 67 eligible patients approached received a randomized intervention.

Discussion

The design of CPRS/VISTA made integration of study workflows and data collection possible. Institutions and investigators considering similar designs must carefully map clinical workflows and clinical trial workflows to EMR capabilities. POCCT study teams are necessarily interdisciplinary and interdepartmental. As a result, executive sponsorship is critical.

Conclusion

POCCT represent a promising new method for conducting clinical science. Much work is needed to understand better the optimal uses and designs for this new approach. Next steps include focus groups to measure patient and clinician perceptions, multisite deployment of the current pilot, and implementation of additional studies.

The Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC) and the Stanford Center for Innovative Study Design have developed a new method for the implementation of experimental clinical research. The point-of-care clinical trial (POCCT) is designed to be embedded directly into the clinical care setting thereby addressing the issues of cost and translation, and creating an integrated environment of research-based care. The POCCT capitalizes on the Veterans Affairs (VA) electronic medical record (EMR) system to perform study activities traditionally conducted by a study team such as enrolment, randomization, and longitudinal data collection. In addition, as evidence accumulates in favor of a specific intervention, it can be transitioned to decision support in the same EMR. As a result, POCCT integrate clinical research and clinical care providing a valuable tool for achieving the vision of a ‘learning healthcare system’.

In this article, the implementation of the VA's first POCCT is described, citing experience to date with an ongoing comparative effectiveness study comparing two common regimens of administering insulin. The informatics-related challenges and strategies used to overcome them are the primary focus of this article, with additional consideration of the potential of POCCT to diffuse within and beyond the VA healthcare system.

Background and significance

Reports from the Institute of Medicine, the Federal Coordinating Council for Comparative Effectiveness Research, and the Congressional Budget Office^1–4 cite the lack of evidence that can be used to support a given course of treatment as a significant obstacle to improving the quality and lowering the cost of healthcare. Also recognized is the inability of current models to meet this need fully. Currently used methods of scientific evidence generation may not be enough to meet the growing demand for relevant evidence. Randomized controlled trials (RCT) are considered to be the gold standard in clinical research. However, the apparatus (ie, the infrastructure) needed to conduct these clinical studies is often cost prohibitive. A large proportion of the cost to conduct RCT derives from support of the personnel needed to conduct recruitment activities, to collect and analyze data, and to perform surveillance for safety events. Furthermore, the generalizability of the results generated by RCT to a broad patient population is often limited due to the narrowly defined inclusion criteria and the intensive study protocol. Observational studies exist as alternative study designs to the RCT and offer a more feasible and cost-effective method to provide clinical evidence. Study-defined procedures for observational studies are often less intensive than those found in RCT and the generalizability of the results is not as limited. Observational studies, though, may be inadequate to provide evidence in support of medical decision-making due to inherent issues of bias and confounding by indication.

It is in light of the widening evidence gap and need for alternative scientific models that MAVERIC and the Stanford Center for Innovative Study Design sought to design and implement a methodology that combines the scientific rigor of randomization with treatment delivered at the clinical point of care. POCCT are designed to be an intermediate strategy for experimental comparative effectiveness research that retains the benefits of both types of study design. Randomization is maintained from RCT in order to overcome the issues of confounding that plague observational studies, and an observational style of follow-up is used to improve feasibility, cost, and generalizability. Moreover, the POCCT study is intended to be implemented at the bedside while the patient is receiving medical care from their provider, therefore eliminating the need for a large-scale infrastructure that is not re-usable. In essence, POCCT is a randomized observational study that can be easily conducted within the context of medical care and deployed for minimal cost.⁵

Aspects of POCCT have been proposed and in some cases implemented by others.^6–8 Vickers and Scardino⁹ discussed the idea of implementing pragmatic clinical trials in some detail. This is, to the best of our knowledge, the first implementation of a clinical trial using the EMR to randomize interventions and then collect all study variables. Implementation of the mechanisms required to facilitate enrolment, randomization, and longitudinal collection of patient data is made possible by the flexible design of the VA's computerized patient record system (CPRS), the clinical care component of the veterans health information systems and technology architecture (VISTA). For convenience and due to the interrelated nature of the two products, they are referred to here as CPRS/VISTA. CPRS/VISTA is available as open source software and was developed in collaborative, open source fashion by clinicians and information technology professionals within and outside of the VA healthcare system over the course of more than three decades. As a result, it has several functionalities that have proved valuable in their ability to support patient care management.¹⁰

Rather than offer only applications designed to perform specific clinical tasks such as decision support or drug–drug interaction monitoring, CPRS/VISTA capitalizes on customizable data objects and workflows. Clinical application coordinators employed by each VA hospital use these functionalities to assemble customized clinical improvement programs in CPRS/VISTA such as quality measurement, decision support, and clinical reminders. A partial list of existing CPRS/VISTA functionalities and their common uses is provided in table 1.

Table 1.

A partial list of existing functionalities in CPRS/VISTA

CPRS/VISTA functionality	Description
Consults	Used by a clinician to notify other clinicians or individuals that their services are needed
Orders	Any type of order can be entered from customizable order menus. Orders can also be placed via reminder dialogs, allowing orders to be automatically entered based on values specified as ‘finding items’. Orders are released with electronic signatures
Order set	Order sets are a group of any type of orders setup to be entered by clicking on a single entry
Progress note template	Local clinical application coordinators create progress note templates with custom titles and form fields to document an event or service delivered
Reminder dialog template	A special type of template designed to allow a clinician to process a clinical reminder that is due (eg, a flu shot, beta-blocker after a heart attack, annual diabetic foot examination, etc). Reminders can be configured to enter orders and can also be associated with progress note titles, as is being done with this POCCT
Finding item	Structured data can be flagged as a finding item allowing workflows and logic to be keyed off of them (eg, enter a specific order when finding item value = x). Finding items are often nested within reminder dialogs, alerting clinicians of specific patient conditions and requisite actions
Health factor	Data object named locally and attached to elements of reminder dialogs as findings. Health factors can be tracked and associated with visits, making it possible to capture the arm each POCCT subject is randomly assigned to
Computed finding	Accessible through the reminder dialogs, computed findings are a way to invoke a MUMPS programme or routine. We use a computed finding to call the MUMPS randomization function
CPRS alert	Alerts generally relate to ordered items, like a consult, or to bring attention to an event (eg, consult has been answered). Recipients can be set at the system, division, team level or with user preference. An orderable item can be flagged to send an alert when ordered (as we do with orders of sliding scale and weight-based insulin). Local sites can create alerts or choose from any of any number of alerts available nationally

CPRS, computerized patient record system; POCCT, point-of-care clinical trial; VISTA, veterans health information systems and technology architecture.

Materials and methods

Description of the pilot study

The POCCT insulin regimen pilot is an open-label, randomized trial comparing sliding scale versus weight-based insulin therapies for all non-intensive care unit inpatients with diabetes. Consented patients are randomly assigned to treatment arms using a Bayesian adaptive randomization method. Adaptive randomization methods adapt over time to favor the ‘winning’ intervention. This approach is more pragmatic in nature, allowing evidence to be used more quickly to inform better care. The details of this randomization design are described elsewhere.⁵ The primary endpoint in the pilot study is length of stay. Secondary endpoints include glycemic control and readmission for glycemic control. The VA Boston Healthcare System (VABHS) is the first hospital enrolling for this study with enrolment scheduled to extend to other New England VA hospitals within the year. The protocol was approved by the VABHS Institutional Review Board including a Health Insurance Portability and Accountability Act (HIPAA) authorization waiver for access to protected health information in CPRS/VISTA.

Adaptation of CPRS/VISTA to support POCCT

The first POCCT pilot was launched primarily to evaluate the feasibility of performing a POCCT in an inpatient setting that offered a controlled environment and easy identification of a specific patient population. We are interested in both the challenges to adapting the EMR to support POCCT as well as house staff physicians and patient willingness to participate.

The implementation of a POCCT is dependent on bringing three vectors into alignment: (1) the processes of an RCT; (2) the processes of clinical care; (3) and the functionality of the EMR. The closer these three vectors can be aligned, the less friction there is in combining clinical science and clinical care using a POCCT. In attempting to reach this alignment, we avoided the development of new functionality within CPRS/VISTA to increase the likelihood of wider deployment of POCCT. Table 2 shows the intersection of these three vectors as implemented for this POCCT. The design is described in additional detail in the following sections.

Table 2.

Intersection of study processes, clinical processes, and CPRS/VISTA functionality as implemented for this POCCT

RCT process	Clinical process	CPRS/VISTA functionality
Identify eligible subjects	Clinician begins to order insulin regimen for patients with diabetes	Customizable order menu displays randomization option (see figure 1)
Educating interested clinicians/patients about study	Clinician reads CPRS/VISTA study option on order menu and discusses with patient and both agree to consider enrolment	Consult sent to study nurse. Health factor created to track clinician and patient consideration
Documentation of consent/non-consent	Study nurse reviews informed consent using official, paper-based HIPAA authorization and informed consent forms	Health factors capture agreement to consent/non-consent or record review-only consent allowing electronic tracking. A progress note is added to the medical record for consented patients
Randomization	N/A	A computed finding calls the MUMPS randomization routine
Intervention	Clinician receives an alert to sign unsigned order	An order set is automatically created based on the results of the computed finding. A CPRS alert is created and sent to the clinician
Data collection	N/A	Periodic pulls from the CPRS/VISTA databases provide longitudinal data collection

CPRS, computerized patient record system; HIPAA, Health Insurance Portability and Accountability Act; POCCT, point-of-care clinical trial; VISTA, veterans health information systems and technology architecture.

Initial order process

The POCCT workflow begins when any clinical provider attempts to place an order for sliding scale or weight-based insulin regimens from the VABHS existing endocrine order menu. As shown in figure 1, the VABHS order entry screen was changed by a local clinical application coordinator to include a third option entitled, ‘VA clinical trial. Randomize to sliding scale or weight based insulin study. Choose this option if there is no preference for insulin protocol.’ Clinicians who choose this third option are shown an informational screen that describes the study and provides order options indicating whether they are interested in proceeding with enrolment. By selecting, ‘No. The patient may not be approached. Proceed with usual care’, clinicians are returned to the previous order screen. A health factor is automatically created to allow the study team to track the number of refusals generated at this stage in the process.

Figure 1.

The endocrine medication menu currently in use in the Veterans Affairs Boston healthcare system with option 1 set to enrol and randomly assign patients into a point-of-care clinical trial.

Alternatively, the clinician may select ‘Yes. The research team may approach this patient for consideration of enrolment.’ CPRS/VISTA features ‘consults’ that can be generated automatically from placed orders. Selecting that the patient may be approached at the order screen automatically pre-populates and sends a consult to the study nurse. The clinician is then directed back to the order entry menu to place an order for either weight based or sliding scale until the patient can be consented and randomly assigned. This ‘holding order’ also ensures that care is not disrupted in the event that the study nurse is unavailable (eg, after hours, on weekends, etc).

Response to consult

On receiving the ‘POC research insulin dosing request’ consult, the study nurse explains the study to the patient and obtains informed consent. If the patient is randomly assigned, the pre-randomization insulin order is discontinued by the study nurse. If the patient declines to participate in the study, a pre-populated progress note is automatically entered into the EMR, which is forwarded to the ordering clinician for review and signature. In this first pilot the study nurse also notifies the clinician directly to ensure proper communication. Refusal or acceptance of random assignment and/or chart review is tracked using health factors, making it possible to track patient decisions. Patients not interested in consenting for random assignment are invited to consent for chart review. The chart review option is incorporated in the study to enable comparisons of patients accepting versus refusing random assignment. If the research nurse is not available to consent the patient, an alternative member of the research team who is designated to receive POCCT consult notifications selects the option ‘patient cannot be enrolled for other reasons’ on the consult reminder dialog. The clinician is alerted through the automatic generation of a pre-populated progress note that is forwarded to the ordering clinician.

Randomization

An enrolment progress note is created if the clinician and patient agree to random assignment. The progress note capitalizes on a CPRS/VISTA feature called a computed finding. A computed finding allows structured data to be passed to underlying methods to derive weights, averages, comparisons, etc. For POCCT a computed finding is employed that calls a random number generator ($RANDOM) native to the MUMPS programming language that underpins VISTA. The maximum allowable number of 1000 is passed and $RANDOM returns a random number between 0 and 999. The return of a number between 0 and 999, as opposed to a binary result for intervention assignment (eg, 0 or 1), is necessary to support the study's Bayesian adaptive randomization design. As the trial ‘learns’ which intervention is more beneficial, the returned integer allows the team to set up a moving threshold for that assignment (eg, 60/40, 70/30, etc).

The returned value is used by the computed finding to create an insulin order, in effect assigning the subject to the appropriate intervention arm. Progress notes for both patients accepting and declining participation are automatically created for and forwarded to the ordering clinician. Medication orders are pre-populated according to treatment assignment and must be signed by clinicians. A CPRS/VISTA alert is therefore used to prompt the clinician to sign and complete the randomized order. The study nurse also contacts the clinician directly and later verifies that the order has been ‘released’ by the clinician. Finally, a health factor is created that documents which of the two arms the patient was randomly assigned to. This allows the study team to identify subjects and their interventions quickly in the CPRS/VISTA database. The ‘response to consult’ and ‘randomization’ processes and the CPRS/VISTA mechanisms used to facilitate them are shown in figure 2.

Figure 2.

Point-of-care clinical trial insulin regimen pilot study workflow and computerized patient record system functionalities used to support it.

Data collection

Although this first pilot POCCT was launched in Boston, the POCCT programme is planned for national expansion. Towards the goal of national expansion, the pilot study was used to test the feasibility of collecting national clinical data from CPRS/VISTA. CPRS/VISTA is a distributed but integrated system with over 100 instances at VA medical centers, each containing its own database. It relies on calls from one system to another to create a complete picture of a given patient's history. At the time of the launch of our first POCCT, there was no single source of all national clinical data. The Veteran's Information and Computing Infrastructure (VINCI), a collaborative effort between the VA Office of Information Technology and the VA Office of Research and Development and the Office of Information Technology's corporate data warehouse are making progress towards providing such a resource. Both VINCI and the corporate data warehouse provided a foundation for much of the data needed for our study. Additional data elements not yet included in VINCI were obtained using the medical domain web services, which package CPRS/VISTA remote procedure calls as web services.

At present, all data extraction processes are launched manually on a weekly basis. Extract transfer load routines are in development that will automatically extract data from VINCI and a batch processes will call the appropriate web services, both on a nightly basis.

Outreach and education

At the start of the project a grand rounds presentation was given about the POCCT programme and more specifically about the VA Boston healthcare system's role as the first pilot site. Other study promotion activities have included informational sessions at weekly conferences, posting study flyers in house staff work stations and informal meetings with house staff physicians and nurse practitioners assigned to each of the ward teams. The chief resident has been enlisted as a clinical champion of the programme and has played an important role in supporting our efforts to inform new interns and residents about the study. The nurse coordinator approaches interns and residents with eligible patients for whom random assignment was not considered. Nurse coordinator follow-up is intended to increase awareness and guide house staff through the process as well as to understand reasons why providers may not elect to assign their patients randomly.

Results

The first patient was enrolled into the insulin regimen POCCT on 12 October 2010. After several rounds of system testing and validation of both workflow and the accuracy of the data collected, all previously described CPRS/VISTA functionalities are working well and data are being collected periodically from the CPRS/VISTA databases. Based on user feedback, minor changes to verbiage on the insulin order menu screens have been made to make it easier for clinicians to recognize the randomization option. The total amount of time to set up all necessary customizations for a site, including validation/quality assurance, amounts to approximately 1 week of one full-time employee's time.

As of 30 June 2011, 105 patients were eligible for enrolment. There were 18 cases in which clinicians declined enrolment because of a preference for one of the insulin regimens. Another 17 eligible patients were not considered for the study because house staff did not initiate a consult (10 patients) or respond to the nurse coordinator's enquiries (seven patients). Of the 67 patients invited to participate, 55 were randomly assigned, three declined participation, and eight agreed to chart review only. Clinicians initiated the point-of-care consults unprovoked by the study coordinator in 27.14% of opportunities (table 3).

Table 3.

Enrolment into the insulin regimen POCCT pilot as of 30 June 2011

Enrolment
Eligible patients	105
Patients enrolled	64 (60.95%)
Of those enrolled
Patients randomly assigned	55
Patients in chart review	8
Patient withdrawn	1
Clinician initiated point-of-care consults	19/67 (28.36%)
Of those declined
Clinician refusal	18
Patients who declined participation	3
Not enrolled
Patients not enrolled (no consult received or no response from physician)	17
Patients not enrolled for other reasons (administrative issues)	3

POCCT, point-of-care clinical trial.

Discussion

Although we are still at the early stages of understanding the implications of and optimal designs for POCCT, our experience in designing and deploying this first POCCT has led to several lessons learned from which others considering similar efforts may benefit.

Patient and clinician acceptance

We are encouraged by a positive enrolment rate of approximately 61% of all eligible patients. Our preliminary results indicate strong patient support for the idea of a POCCT, based on the assignment of randomized interventions of 82% of patients approached. Most patients were agreeable to random assignment, and anecdotal comments from them suggest that they were supportive of the study question and perceived their participation as minimal risk.

Our experience with engaging house staff physicians in recruiting patients at the point of care has not been as successful. House staff physicians initiated the randomization option at the time of ordering insulin in only 28.35% of eligible patients, although they did agree to random assignment and entered a point-of-care consult when approached by the study coordinator in 80% of requests. While in most academic medical centers residents actively manage the inpatient services, our institution may present specific challenges leading to low participation. The medical ward teams in the VA Boston healthcare system are composed of 16 interns and residents from three residency training programs with rotating schedules every 3 weeks. The high rate of turnover and relatively small population of eligible patients (four to five a week) lessens the opportunity for interns and residents to incorporate this novel mechanism into their practice.

To address the low rate of participation among residents and interns the initial order screen in CPRS/VISTA was recently revised to force an opt-in or opt-out of randomization before proceeding to the standard weight-based and sliding scale order menu. This requires a purposeful decision to accept or reject randomization and allows more granular tracking of reasons for refusal. A study is currently underway that will conduct patient and clinician focus groups to understand more fully these stakeholders' perspectives.

In retrospect, we believe the ideal setting for implementing the first use case might have been in a clinical area where a limited and more stable number of providers exists. Clinicians who completed the POCCT order set in response to the study nurse's request commented that the process was easy and quick to complete. Unfortunately, only a few have had the chance to repeat the process during their rotation. A more stable group of providers would provide a better opportunity to assess clinician behavior and evaluate how well POCCT might be adopted into practice.

Using the EMR to support POCCT

We have thus far been impressed with the ability of existing CPRS/VISTA capabilities to support the functionalities required to conduct a POCCT. Most beneficial from a software development standpoint is the modular and generalizable design of CPRS/VISTA. Underlying any custom quality measurement or specific clinical reminder application within CPRS/VISTA are objects and workflows that can be assembled and then customized to meet any number of clinical information initiatives. In addition, access to national longitudinal clinical data, although still from multiple sources, has proved feasible.

An important next step for the POCCT programme is deployment to additional sites. While some of the customizations we have made to CPRS/VISTA can be packaged and exported (eg, clinical reminders, alerts, health factors) the architecture of CPRS/VISTA prevents the export of order menus. As a result, any site wishing to implement this insulin-based POCCT must create custom order menus. Detailed step-by-step instructions were made to support clinical application coordinators responsible for installing POCCT-related deployable packages and menu customizations.

Our experiences in adapting CPRS/VISTA may hold lessons for those considering the design and adoption of other EMR systems. In table 2 we outline the necessary functionalities for conducting a POCCT. The two ways for EMR systems to achieve such functionalities are to create specific clinical trial modules or to design their systems to be modular and customizable such as the workflows and data objects of CPRS/VISTA. Current requirements for reimbursement under Health and Human Services' ‘meaningful use’ and EMR vendor certification policies are based more on the implementation of specific functionalities (eg, implement drug–drug interaction checks) and digitizing data rather than supporting customizable workflows, standard data formats, and unfettered access to well-defined and documented EMR databases. Policies that ensure the ability of owners of EMR systems to access all data and develop new workflows will be necessary to foster future innovations such as POCCT.¹¹

Limitations

While integration with the EMR introduces a range of previously unavailable advantages, there are limitations introduced by the dependency of POCCT on the EMR that must be considered. The questions POCCT can be used to answer are limited by the data elements collected in the EMR. In addition, the quality of data elements available must be carefully considered during the design of a POCCT. Healthcare institutions interested in implementing POCCT must also assess the ability of their EMR systems to support the functionalities required to identify, enrol, randomly assign, and track the data elements of individual subjects.

The ability of EMR systems to support POCCT may also be dependent on the specifics of a proposed POCCT. For example, our ability to identify patients in the insulin pilot is based on the use of an endocrine order menu. Studies of a non-pharmaceutical intervention (eg, delivery of a mental health therapy) may require alternative mappings of existing clinical workflows. Finally, the ability of local clinical application coordinators to customize CPRS/VISTA that made our pilot possible may present challenges to national deployment efforts. In researching the next sites to deploy the insulin regimen POCCT we have encountered sites with endocrine ordering menus different from the one employed in Boston. These lessons learned have led the POCCT team and VA Office of Research and Development to create a new process for assessing the appropriateness of proposed studies for the POCCT mechanism. This new process will combine existing deliberations (eg, scientific validity, study design, etc) with POCCT-specific considerations.

Sociocultural considerations

The greatest obstacles to widespread adoption of POCCT are likely to be imposed by policy and cultural considerations. POCCT blurs the line between the two often distinct paradigms of clinical care and clinical research. Patients, clinicians, and hospital administrators must consider the effect on the clinician–patient relationship introduced by the admission of equipoise and the assignment of care by randomization. The introduction of POCCT also challenge institutional review boards to consider carefully the definitions of ‘engaged in research’ and the requirements related to informed consent. A system designed to gather evidence in support of one treatment versus another at the point of care that can be transitioned to clinical decision support may force reconsideration of what is research versus operational improvement.

Another important consideration is the interdisciplinary nature of the design and implementation of POCCT and the level of commitment required from several organizations within the healthcare system. The core team involved in the VA's first pilot study using a POCCT has required an expert in diabetes care, experts in clinical trial design and execution, biostatisticians, an epidemiologist, a project manager, an ethicist, informatics expertise in database design, CPRS/VISTA and medical domain web services, and a dedicated study nurse. The team's modest success is contingent on support received from our local institutional review board, hospital administrators, and house staff. In addition, modifications to CPRS/VISTA were approved and facilitated by leadership at the network level (the New England Veterans Integrated Service Network) and access to longitudinal clinical data was supported by several teams within the Office of Information Technology. As is always the case for interdepartmental system development, executive sponsorship at the highest levels of the organization has been critical for POCCT.

Conclusion

The first implementation of a POCCT in the VA has demonstrated the feasibility of this new method of evidence production. Existing functionalities within the VA's EMR system are currently employed to identify eligible patients, facilitate enrolment, perform randomization, and collect longitudinal data. Early results show both patient and clinician acceptance of the integration of a clinical trial into routine clinical care, although more work needs to be done to understand stakeholders' perspectives. Executive sponsorship and interdisciplinary collaboration have been critical to our success to date, and a national programme for designing and deploying POCCT is underway within the VA's Office of Research and Development. As evidence accumulates in this first trial, we look forward to converting it to actionable decision support at the point of care using existing CPRS/VISTA decision support functionality. The next step in our assessment of the feasibility of POCCT is expansion of the current pilot study to VA sites throughout the New England region. In the meantime, new studies are in consideration and alternative models of obtaining informed consent are being explored as next steps in the development of the Office of Research and Development's point-of-care research programme.