Abstract
Creating shareable decision support services is a complex task requiring effort from multiple interdisciplinary role players with a wide variety of experience and expertise. The CDS Consortium research project has developed such a service, defining a multi-layer representation of knowledge and building upon an architectural service design created at Partners Health Care, and is demonstrating its use in both a local and an external institutional setting. The process was iterative, and we encountered unexpected requirements based on decisions made at various points. We report in this paper on challenges we faced while pursuing this research: knowledge representation and modeling, data interchange and standards adoption, the process of getting agreement on content, logistics of integrating into a system that already has multiple CDS interventions, legal issues around privacy and access, inter-team communication and organization.
Introduction
Providing clinical decision support (CDS) has been part of clinical systems design at Partners Health Care since the inception of computerized provider order entry (CPOE) at Brigham and Women’s Hospital (BWH) in 19921,2. Concurrent with the evolution of our internally developed CPOE and electronic medical record (EMR) systems, we have conducted and reported on research into evidence-based practice that makes use of these systems3, assessed how well decision support interventions are received and acted upon4–6, and continue to study, document, and report on the importance of providing CDS within clinical systems and EMRs7–9.
Despite the proliferation of CDS interventions that we have piloted, we have not been able to extend them to clinical systems at all our member institutions. Differences in terminologies and technologies, combined with the proprietary nature of the vendor systems have made it nearly impossible to share or exchange data, resulting in CDS rules and interventions that are unknown to each other, unavailable for editing or review, hard-wired into application code, and inconsistent as to the advice presented, making them difficult if not impossible to maintain, and expensive in terms of development and testing time to improve. As patients move from one institution to another, these inconsistencies test our ability to be a true enterprise, and more importantly, provide the best patient care.
To address this problem, we designed the Enterprise Clinical Rules Service (ECRS), a set of components that accepts input from a subscribed consumer to run rules using a commercial rules engine, returning results and recommendations to the caller10,11. In late 2007 an opportunity to extend our efforts beyond Partners became available through the Clinical Decision Support Consortium (CDSC)12, in which we participate as founding members. Moving ECRS forward as part of the CDSC created additional interdisciplinary challenges, some unanticipated. This paper describes how we met challenges around:
▪ data interchange and standards
▪ adoption knowledge representation and modeling
▪ content governance
▪ integration logistics
▪ legal issues around privacy and access
▪ inter-team communication and organization
Background
CDSC is funded by the Agency for Healthcare Research and Quality (AHRQ) and has as its goal “to assess, define, demonstrate, and evaluate best practices for knowledge management and clinical decision support in healthcare information technology at scale – across multiple ambulatory care settings and EHR technology platforms.”13 Ten teams were set up by CDSC to accomplish its research goal. We consider the experience of three: Knowledge Representation and Specification (KTS), CDS Services, and CDS Demonstrations (Demo) teams, addressed these CDSC research questions:
▪ How do we optimally represent knowledge and data required to make actionable clinical decision support content in both human and machine readable form?
▪ How do we demonstrate broad adoption of evidence-based clinical decision support at scale in a wide array of healthcare information technology products used in disparate ambulatory care delivery settings?13
KTS concerned itself with defining how best to represent clinical knowledge and data, specifying formats for guideline content in human readable, semi-structured, and structured forms14, and populating a patient clinical information model with guidelines selected by the CDSC. CDS Services team provided specifications for the fourth, machine executable, layer, translated the guidelines from the structured layer to it, and developed the service components specified by ECRS architecture. The Demo team comprises site implementation teams; an overall team lead provides direction to each site and a connection to the rest of the CDSC. Two demonstrations were planned; one in PHS’ EMR, the Longitudinal Medical Record (LMR), and the other at Regenstrief Institute (RI) in Indianapolis, Indiana.
Methods
Research of this kind emphasizes discovery of what is possible and defining good practice out of that learning. By its very nature, therefore, it provides challenges not always found in “operational” software projects, including underspecified requirements that are dependent on the discovery process, and solutions that are determined only through cycles of iteration, trial, and evaluation.
As each CDSC team began to analyze what it needed to do to accomplish its goal, some overlaps in tasks were found. We therefore set up the Joint Information Modeling (JIM) group, composed of members from each team, to research and recommend standards for terminology and modeling for the CDSC and create the necessary information models for the project.
The guidelines to be demonstrated were defined at the outset by the CDSC and fell into two areas: preventive care guidelines as defined by the United States Preventive Services Task Force (USPSTF), and chronic care management rules for two conditions: diabetes and coronary artery disease (CAD). Although there are complex guidelines available for the treatment of chronic diseases such as these we chose PHS-developed rules for the screening and management of diabetes and the USPSTF guideline for the use of aspirin in CAD patients. Prior to selecting the specific guidelines to use, we reviewed existing guidelines in use as reminders or Smart Form17 interventions in the LMR, studied a variety of USPSTF guidelines, and reviewed published literature to find those that would both be achievable within the project and provide useful decision support in the ambulatory setting.
Results
The JIM group reviewed several models to find one on which to base its patient information model. The strongest candidates were the Virtual Medical Record15 (vMR) and HL7 Clinical Statements in the Continuity of Care Document (CCD). Both are based on the HL7 v3 RIM and provide a rich, structured data model. However, vMR was not yet a draft standard, and we found no commercial EMR with plans to implement it, and decided to base our model on HL7 Clinical Statements and recommend that an HL7 CCD be specified for input to the service. We expect this approach to be more broadly acceptable because (1) the CCD is an approved standard and recommended by HITSP16, (2) CCHIT requires CCD support in ambulatory EHRs for certification, increasing the likelihood of vendor support, (3) a CCD provides a practical transport mechanism for the clinical data when calling the service, and (4) the service can translate between an input CCD and the information model without complex transformations.
The JIM group also defined terminology standards for use by the service, selecting code sets specified in CCD and HITSP C-32 implementation guides. These include SNOMED-CT for problems, procedures, and social history, LOINC for laboratory tests and vital signs, RxNorm for drugs and allergies, and NDF-RT for drug and allergy classes. JIM recommendations were adopted by the CDSC Steering Committee.
Data and the adoption of standards
The earliest challenges presented themselves as a result of the selected standards. First, CCDs needed to be created where none had been done before; second, patient data had to be coded with the required terminologies.
The use of CCDs had not been previously considered by ECRS, nor did LMR have experience with creating one. Review of LMR’s available resources, and consideration of future ECRS consumers’ need to create a CCD resulted in a decision to create a team to build CCDs for PHS patients, and to make this “CCD Factory” service available within Partners’ systems. This has proved to be a positive decision; other consumers of CCDs have requested its use.
Mitigating the challenge of coding patient data required the addition of two service-providing teams to the project. PHS has developed local dictionaries that are used throughout its systems, some of which have been mapped to standard vocabularies, but the process is incomplete. Services were available to translate our problem codes to SNOMED-CT, but not for local drug or allergy codes to RxNorm, nor for drug or allergy classes to NDF-RT. Also, some data for which we had no formal code system defined has had to be mapped, such as vital signs to LOINC.
Efficiency of rule design and flexibility of rule use by differing institutions or applications indicated the need for data to be grouped into classes or subsets, where all of the individual codes included would be applicable for the given condition. This has required the addition of classification along with translation services. Each guideline generates a recommendation based on a set of conditions, one of which is that the patient has a specified condition, e.g., diabetes. A variety of similar SNOMED-CT concepts may be acceptable as indications for a given condition, plus clinicians may choose different concepts to represent a patient’s problem. We designed the rules to use classes rather than listing all possible options in a long, complex, and hard to maintain form. Problems are classified using subsets created and maintained by PHS; they are based on SNOMED-CT hierarchies but may be limited or constrained for specific use. For example, a diabetes subset may exclude pregnancy-related diabetes. Initial versions of these new services is moving us closer to being truly interoperable.
Representing and modeling knowledge
The decision support rules needed for this project were simple and did not present a challenge from a knowledge representation perspective. However, the lack of a standard information model to enable access to patient data was a major hurdle. Analysis of available models confirmed the need to develop a flexible patient information model focused on specific decision support needs. The implemented model addressed these needs and confirmed that it is possible to create a model for simple decision rules without requiring an extensive effort. The model has been periodically revised as new needs arise, with limited impact on existing rules and services.
We found no directly usable existing model for returning recommendations produced by rules execution in a standard fashion, and therefore developed one that is consistent with the input in the terminologies and classes used, and that provides the caller of ECRS with data either to create order or observation objects in the local application or simply to display the recommendations as text.
Integration Logistics
Several technical challenges arose when incorporating the externalized rule set into the LMR. First, the structure for triggering the rules differed between the existing LMR reminder system and the ECRS. The existing CDS system pre-generated the rules and re-generated the output of the rules when there was a change in the patient’s relevant clinical information. The CDSC rules were real time rules, designed to be initiated when applicable in the user’s workflow. This required coordinating two separate triggering mechanisms for the research practices: one that pre-generated the output from the existing rule set and the other that initiated a real time transaction with the CDSC rules service. A second, resulting, challenge due to the two separate rules outputs involved consolidating the reminders at the time of presentation to the clinician.
Third, the new rules were “stateless,” meaning actions taken on the rules were not known to the rules service and not part of the CCD input into the rules service. In the existing system, to avoid over alerting, certain user actions on the reminders turned the rules off for a period of time. For the reminder that a diabetic patient is overdue for an HbA1C, selecting “Patient declined” sets a “snooze period” to 3 months. Indicating “done” defers the reminder for 6 months, and choosing “Appointment scheduled” holds it for just 1 month. Maintaining this functionality required capturing and storing the user’s actions in the LMR for the external CDSC rules and mapping the CDSC rules to the logic that determined whether or not to present the output from the rules.
As of July 2010, ECRS is running the initial set of guidelines in the LMR, and the RI integration team expects to complete their implementation by October.
Discussion
Following a research-oriented “discovery-iteration-trial-evaluation” model as we have in this project guaranteed that we would encounter challenges. Those we faced fell into two general types: objective ones whose solutions present themselves in a straightforward and readily agreed-upon way, such as those described in the previous section, and others that are more subjective, whose solutions are not readily apparent, require careful consideration, and may only be mitigated following consensus by multiple stakeholders.
External integration
Commencing the second integration with the RI team is providing additional challenges. One of these relates to the mappings between the CCD data and the local patient information model. Each data element and value was individually mapped; we encountered the major issues when we started analyzing CCD documents provided by the RI team. We learned that the CCD standard is not sufficiently defined to eliminate all ambiguities in data representation. This issue is being solved at the CCD document creation level, where the document creator, RI or PHS, makes sure that the required data elements are present and that the same terminology values are used.
Content governance
Issues related to content that emerged early and unfortunately were not entirely anticipated at the start of the project fell into two types, intra-institutional and inter-institutional. Examples of the former included making decisions about which area in the LMR application would host the demonstration rules and how they would integrate with current decision support interventions. The LMR has an existing rich set of clinical decision support rules supported by a well-defined governance process. The CDSC guidelines could be integrated into either LMR’s reminder or Smart Forms modules, and considerable thought and discussion was given to determining the optimal place for this trial. Factors considered included the nature of the software underpinnings of each area, suitability of the specified guidelines to the module, expected effort that would be needed, and available time given both contract milestones and LMR release schedules. The guidelines are integrated in LMR’s reminder system.
Since the CDSC rules are focused on specific clinical conditions and the existing rule set is much broader, LMR needs to maintain coexisting CDS systems. For the research practices, users view the output of the CDSC rules along with the existing rule sets; all other practices only view the output of existing rule sets. It became necessary to provide suppression capabilities on the existing rule set in order to avoid duplicate alerting on the same or similar clinical scenarios.
Another intra-institutional issue is governance around the new rules. The IRB had traditionally approved protocols to test additions of rules, never to replace currently active rules. Furthermore, wary of the increasing problem of alert fatigue, LMR’s oversight committee for production level decision support increasingly enforced a policy of only approving new reminders that were “easy to heed but hard to remember”. After a draining period of shuttle diplomacy among the stakeholders, a policy was approved which clarified the respective roles of the research team, IRB, and LMR oversight committee.
Inter-institutional governance issues included the identification of local authority groups, not always affiliated with the research team, in order to enlist their commitment to implement the demonstration rules at each demonstration site; and the development of a mutually agreed upon set of policies, procedures, roles and responsibilities amongst the consortium members for co-authoring, sharing, modifying and implementing each other’s content. The major decisions here were that these activities were all voluntary, that institutions retained ownership rights of the content they submitted, that consortium members retain the right to modify and adapt content for their own use with proper attribution to the original source, that the member institutions indemnify each other from any harms that came from implementing shared rules, and that no consortium members could sell any content for monetary gain.
Legal Issues
Protecting privacy of patients in the context of shareable CDS services presents a practical challenge that must be addressed before the RI integration can be accomplished. Driven by HIPAA regulations and interpreted by Health Information Services and legal teams at both sites, this is proceeding slowly and carefully. Appropriate documents must be drawn up and approved by parties who are not part of the research team, but who are responsible for protecting patient data from misuse.
Use of the service means allowing an external entity access to send and receive data inside our firewall. Resolving this issue requires policy decisions and input from technical teams at PHS who have not previously needed to address the question of allowing access to our systems for an organization with no ties to Partners institutions.
Shareable clinical decision support delivered via a service-oriented architecture raises complex questions about accountability and liability when an organization “outsources” this function to potentially imperfect external agencies. Nadkarni and Miller have asserted that CDS services are not “licensed practitioners”, thus the clinician is still legally responsible for overriding any erroneous interpretation or guidance offered by a SOA service18. Although CDS software can augment a clinician’s data interpretation and decision making, such guidance is only as good as the CDS content embedded in the software and the patient-data richness that is delivered to it19. The implementation of CDS services raises further questions about distribution of accountability across multiple service providers and consumers. Other challenges include ensuring that the patient data transferred to the service is complete and accurately translated, verifying that the services are implemented correctly as specified, and that clear procedures are in place in the event of issues such as unanticipated down-time of any components. The CDSC team, in consultation with their legal advisers, is currently in the process of developing policies, procedures, and legal agreements that address the above considerations.
Inter-team communication and organization
The CDSC organizational structure itself presented some challenges. The project manager (PM) for the contract was responsible for outward facing communication to AHRQ and for the contract and research management, and each team had its own PM, but no one was responsible for organizing the work where multiple teams were involved. In additional challenge, due to the discovery-iteration-trial-evaluation cycle, was the volume of new requirements that resulted from discovery of new work needing to be done.
To manage the changes, the Services Team PM set up weekly meetings (dubbed the “Vortex”) with resources from all overlapping teams, including service providers and RI integration. These meetings centralize discussions and decision making, improve inter-team communication and have helped us meet the contract deadline for integration with the LMR. By providing an open and flexible communication venue, the Vortex has become central to our success.
Conclusion
In summary, through the CDSC project we have learned that producing shareable CDS services requires a large number of teams to come together, that there are myriad issues that increase in complexity when multiple institutions become involved, and that content that appears to be simple or obvious may not in fact be so. Nevertheless, we have succeeded in creating such a service, are testing it using three shared guidelines in two locations, and look forward to new challenges as we move forward.
Acknowledgments
This work was supported under contract #HHSA-290-2008-10010 from the Agency for Healthcare Research and Quality. The opinions expressed in this document are those of the authors and do not reflect the official position of AHRQ or the U.S. Department of Health and Human Services.
References
- 1.Teich JM, Hurley JF, Beckley RF, et al. Design of an easy-to-use physician order entry system with support for nursing and ancillary departments. Proc Annu Symp Comput Appl Med Care. 1992:99–103. [PMC free article] [PubMed] [Google Scholar]
- 2.Glaser JP, Teich J, Kuperman G. The future of clinical information systems: one hospital’s perspective. Top Health Inf Manage. 1993 Aug;14(1):12–24. [PubMed] [Google Scholar]
- 3.Bates DW, O’Neil AC, Boyle D, et al. Potential identifiability and preventability of adverse events using information systems. J Am Med Inform Assoc. 1994 Sep–Oct;1(5):404–11. doi: 10.1136/jamia.1994.95153428. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kuperman GJ, Teich JM, Gandhi TK, et al. Patient safety and computerized medication ordering at Brigham and Women’s Hospital. Jt Comm J Qual Improv. 2001 Oct;27(10):509–21. doi: 10.1016/s1070-3241(01)27045-x. [DOI] [PubMed] [Google Scholar]
- 5.Paterno MD, Maviglia SM, Gorman PN, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc. 2009 Jan–Feb;16(1):40–6. doi: 10.1197/jamia.M2808. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Piazza G, Goldhaber SZ. Physician alerts to prevent venous thromboembolism. J Thromb Thrombolysis. 2009 Nov 4; doi: 10.1007/s11239-009-0404-5. [DOI] [PubMed] [Google Scholar]
- 7.Gross PA, Bates DW. A pragmatic approach to implementing best practices for clinical decision support systems in computerized provider order entry systems. J Am Med Inform Assoc. 2007 Jan–Feb;14(1):25–8. doi: 10.1197/jamia.M2173. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Glaser J. Clinical decision support: the power behind the electronic health record. Healthc Financ Manage. 2008 Jul;62(7):46–8. 50–1. [PubMed] [Google Scholar]
- 9.Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003 Nov–Dec;10(6):523–30. doi: 10.1197/jamia.M1370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Goldberg HS, Vashevko M, Pastilnik A, et al. Evaluation of a commercial rule engine as a basis for a clinical decision support service. AMIA Annu Symp Proc. 2006:294–8. [PMC free article] [PubMed] [Google Scholar]
- 11.Paterno MD, Schaeffer M, Van Putten C, et al. Challenges in creating an enterprise clinical rules service. AMIA Annu Symp Proc. 2008:1086. [PubMed] [Google Scholar]
- 12.Middleton B. The clinical decision support consortium. Stud Health Technol Inform. 2009;150:26–30. [PubMed] [Google Scholar]
- 13.The Clinical Decision Support Consortium website. 2008. Available from: http://www.partners.org/cird/cdsc/. [PubMed]
- 14.Boxwala A, Rocha B, Maviglia S, et al. Multilayered knowledge representation as a means to disseminating knowledge for use in clinical decision-support systems. AMIA Spring Cong. 2010 [Google Scholar]
- 15.Cheng, et al. Implementation of virtual medical record object model for a standards-based clinical decision support rule engine. AMIA Annu Symp Proc. 2006:958. [PMC free article] [PubMed] [Google Scholar]
- 16.HITSP Summary Documents Using HL7 Continuity of Care Document (CCD) Component, version 2. Aug 3, 2008.
- 17.Schnipper JL, Linder JA, Palchuk MB, et al. “Smart Forms” in an Electronic Medical Record: documentation-based clinical decision support to improve disease management. J Am Med Inform Assoc. 2008 Jul–Aug;15(4):513–23. doi: 10.1197/jamia.M2501. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Nadkarni PM, Miller RA. Service-oriented architecture in medical software: promises and perils. J Am Med Inform Assoc. 2007 Mar–Apr;14(2):244–6. doi: 10.1197/jamia.M2349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Waitman LR, Miller RA. Pragmatics of implementing guidelines on the front lines. J Am Med Inform Assoc. 2004 Sep–Oct;11(5):436–8. doi: 10.1197/jamia.M1621. [DOI] [PMC free article] [PubMed] [Google Scholar]