Model-driven approach to data collection and reporting for quality improvement

Graphical abstract

Highlights

• •Addressing the challenge of the second translational gap is key to improving healthcare processes.
• •Data-driven methodologies improve likelihood of success.
• •We propose the Improvement Data Model (IDM) for data collection and reporting for local improvement.
• •WISH, a prototype software tool based on IDM is used by over 600 users in 50+ improvement projects.

Abstract

Continuous data collection and analysis have been shown essential to achieving improvement in healthcare. However, the data required for local improvement initiatives are often not readily available from hospital Electronic Health Record (EHR) systems or not routinely collected. Furthermore, improvement teams are often restricted in time and funding thus requiring inexpensive and rapid tools to support their work. Hence, the informatics challenge in healthcare local improvement initiatives consists of providing a mechanism for rapid modelling of the local domain by non-informatics experts, including performance metric definitions, and grounded in established improvement techniques. We investigate the feasibility of a model-driven software approach to address this challenge, whereby an improvement model designed by a team is used to automatically generate required electronic data collection instruments and reporting tools. To that goal, we have designed a generic Improvement Data Model (IDM) to capture the data items and quality measures relevant to the project, and constructed Web Improvement Support in Healthcare (WISH), a prototype tool that takes user-generated IDM models and creates a data schema, data collection web interfaces, and a set of live reports, based on Statistical Process Control (SPC) for use by improvement teams. The software has been successfully used in over 50 improvement projects, with more than 700 users. We present in detail the experiences of one of those initiatives, Chronic Obstructive Pulmonary Disease project in Northwest London hospitals. The specific challenges of improvement in healthcare are analysed and the benefits and limitations of the approach are discussed.

1. Introduction

Rising demand for efficiency and effectiveness in health services with increasingly limited resources puts health systems world-wide under pressure to continuously deliver high quality care [1], [2], [3], [4]. More rapid implementation of research into practice has the potential to improve outcomes and value, however it is recognised that implementation in health systems is slow, incomplete, and often not sustainable, with variation in compliance to best practice [5]. This challenge of implementing new approaches into practice has been identified as the second translational gap [6] and research bodies such as Institute for Healthcare Improvement (IHI) in the US, UK’s Institute for Innovation and Improvement, and Australian Institute for Health and Welfare, have been set up to investigate the mechanisms behind effective and sustainable improvement initiatives.

Local improvement initiatives represent an important part of achieving change in a healthcare system, complementing the top-down institutional and national initiatives. Local improvement is also well-suited to implementing clinical research into practice. The proximity of the implementation teams, consisting of, among others, doctors, nurses, administrative staff, pharmacists, patients and members of the public, to front line care delivery ensures relevance and focus of such improvement projects. However such projects face their own set of challenges, including the complex nature of the internal organisational processes, lack of capability and capacity for improvement work in busy staff with diverse responsibilities, and low visibility of the changes being effected.

Data-driven methods are recognised as essential to achieving improvement in healthcare. Boaden’s report on quality improvement in healthcare [7], established that appropriate and rigorous use of data, both quantitative and qualitative, is essential to test out interventions during an improvement initiative. Such reliance on data improves the chances for success, and for a higher-level evaluation to determine whether the initiative has a significant impact upon quality for patients and carers. Meyer et al. [8] also stresses the need for appropriate selection of metrics to be used in measurement to fit the needs of both end-users and service providers. However, improvement data required by local improvement initiatives are often not available in hospital EHR systems, not at the required level of granularity, not appropriately reported, or not easily accessible to the implementation team members. While national and regional improvement programmes can invest resources in overcoming these problems through adapting their software and information systems, such facilities are rarely available to smaller teams.

To address this need for targeted data to drive local improvement initiatives, we developed an approach based on Improvement Data Model (IDM), a novel computational model of improvement, and a prototype software Web Improvement Support for Healthcare (WISH) to demonstrate its use. Local teams in medical organisations use IDM to specify the metrics to track their performance during the intervention, together with the data points necessary to calculate these metrics. Based on the team’s IDM specification, the WISH software automatically generates appropriate data collection pages, and a set of live reports containing the required metrics for team members to access as the improvement project progresses. The standardised reports use Statistical Process Control (SPC) [18], [19] as the statistical tool for visual analysis.

The IDM/WISH model-based approach started as the core part of UK’s NIHR-funded CLAHRC NW London initiative, where it was used and refined on over 50 projects in Northwest London hospitals between 2007 and 2013. WISH supports multiple concurrent improvement projects inside a single enterprise, with users being parts of several initiatives, potentially with differing administrative privileges, e.g. improvement team members, managers, or administrators.

In order to demonstrate the feasibility of a model-driven approach to improvement data collection and reporting, we describe the usage of IDM and WISH in one CLAHRC NWL improvement project focused on improving care for patients with Chronic Obstructive Pulmonary Disease at discharge from a hospital. The aim of the project was to improve outcomes by ensuring a set of care elements (COPD bundle) is offered to all patients admitted to hospital with a diagnosis of acute exacerbation of COPD. The exemplar IDM model is provided for the project, including the data items captured and metrics defined. The WISH data capture and reporting components generated from the model are presented, together with the user experiences.

2. Background and significance

Performance metrics are increasingly derived from routinely collected data [9], [10] to achieve efficiency and effectiveness in health systems. Availability of up-to-date simple measures of performance linked to the improvement aim has been shown to help the understanding of the relationship between actions and outcomes, inform decision making, and drive success [11], [12], [13]. For example, if a local improvement team is implementing a change in a particular patient pathway, such as community acquired pneumonia treatment, hospital EHR systems can be used to measure high-level outcome measures (e.g. mortality rates) whilst a change is being implemented, but are poorly suited to providing more finely grained data specific to the local task (e.g. number of anti-smoking leaflets handed out in the ward each week).

Effective data feedback for quality improvement has a number of characteristics: timeliness, specificity to local context, credibility, and sustainability over time [14]. However, routinely collected data often fails to meet these criteria. Measurement should always reflect the current state, requiring data collection in good time and efficient and fast sharing of results within the team. In a large institution, this is typically a lengthy process since the routinely collected data need to be extracted, curated, and analysed before it can be used for performance analysis and the results sent to the improvement implementation team. Furthermore, the specificity to local context is missing, since the local improvements may require bespoke data that do not reside in an existing EHR system. In order to avoid compromising on measure definitions, by restricting them to data available in the EHR system, such data is often collected in spreadsheets or other local data collection instruments, with no central oversight and no visibility to teams. While potentially useful in the short-term, without a common information model and shared quality standards, this approach fails to produce comparable results that can be used for learning best practices. Finally, analysis techniques used should reflect the needs and skills of the improvement team to achieve credibility and sustainability. Popular Electronic Data Collection (EDC) tools for clinical trials, such as REDCap [25], lack the integrated user-defined analytics that is required for providing understandable and current improvement information to the users. For example, REDCap provides inline frequency counts for individual questions, but any further data analysis is left to specialist researchers using external analytical tools [40].

Quality improvement has been extensively studied in business and manufacturing domains. Continuous data collection, analysis, and feedback form the core of Langley’s influential Model for Improvement [15], [16], [17]. The Model for Improvement uses Statistical Process Control (SPC) [18], [19] as the preferred statistical approach for establishing significant changes in time-series data. Quantitative data in Model for Improvement is enriched by the qualitative Plan-Do-Study-Act (PDSA) cycles [20], [21], which provides qualitative description of a change that should result in the improvement, encompassing hypothesis, testing, analysis, and refinement.

The informatics challenge in healthcare local improvement initiatives consists of providing a mechanism for rapid modelling of the local domain by non-informatics experts, including performance metric definitions, and grounded in established improvement techniques. The Improvement Data Model (IDM) provides a flexible information model to local improvement initiatives, similarly to CDISC’s ODM standard that provides structure for trial data collection [22]. Such model-based approach encourages sharing of information and data between different departments and/or organisations and overcoming the culture of internal silos to share best practices, compare results, and preserve generated knowledge. Our work goes further than just being based on a model, and is fully model-driven in that the system responds dynamically to changes in the model, such as addition of new improvement projects.

3. Materials and methods

A key challenge in model-driven healthcare software [24], [25], [26] is the simplification of the model development and the workflow deploying the model to its translation into practice. To address this challenge when using IDM, we have developed the WISH software platform and an associated methodology for developing the IDM models. WISH is a collaborative framework for local improvement teams to specify the quality metrics for their improvement projects, and rapidly deploy the data collection web interfaces for the required data. The basic steps for using WISH are:

• 1.Define the metrics that will be used to measure the effect of an improvement initiative.
• 2.Specify the data items needed to calculate those metrics and formally express the metrics using those items.
• 3.Encode the data specification and metrics into the IDM model, together with question labels, data types, and reporting parameters.
• 4.Load the IDM model into the WISH tool. At this point, data collection and reporting become available to users.

The initial modelling work was performed through a series of meetings and workshops where the teams would establish ideas for improvement aims and interventions and capture them in an Action-Effect Diagram, a type of cause-effect chart derived from Driver Diagrams [15] which expresses how the interventions are intended to achieve the aims and defines the key actions that need to be performed. Users also perform process mapping to produce a flow diagram in order to understand and document how the actions fit into the healthcare workflow that they are trying to improve. Following this, a system of measures is developed to track the translation of actions into concrete changes in delivery of patient care. The WISH team would assist in the development of these measures from the conceptual stage through to detailed operational definitions to ensure consistency of data collection and interpretation. The data collection is then piloted, to collect baseline data prior to the intervention, and initial data from early tests of change, initial experiences of which may necessitate a redesign of some or all of the measures. This iterative approach ensures the quality of the final IDM model. The full process is shown in Fig. 2.

Fig. 2.

Improvement projects using IDM/WISH start with a process mapping stage, in which the IDM model for the project is generated. The model is then translated into a new table in the database schema, web-based data collection pages, and a set of associated reports based on data measures. During the course of the project, the users regularly enter their improvement data into the tool, and track their progress through live reports. The solid lines denote sequential ordering between steps, while dotted lines represent documents that are inputs and outputs to steps.

3.1. Improvement Data Model (IDM)

IDM represents the full data and metric specification of an improvement initiative, as shown in Fig. 3. The core model concept is that of an improvement process, which may contain several data collections. These collections represent logical groupings of improvement data, related to a particular set of improvement metrics. Each data collection consists of a set of typed data elements, with associated textual labels for data entry and display, default values and graphical data entry widgets (e.g. drop-down lists, checkboxes, radio buttons, text boxes). Basic data types that are supported are text, integers, floating point values, and dates, together with sets of predefined categorical values. Simple verification logic is present to ensure individual values are within defined ranges. In addition to user-defined data items, a data collection has to contain a nominated date attribute that is used for time-based aggregations in reports.

Fig. 3.

Improvement Data Model contains the description of the data to be collected for the improvement project and the definition of the measures used to track its progress.

Named aggregated variables are defined over the basic data measures in IDM, using aggregation operators (count, sum, cumulative sum, maximum, minimum, average, median). These variables are used in metrics that are associated with each data collection. The metrics are defined by a JavaScript expression using aggregated variables, and standard JavaScript functions. Each metric also contains a title and scale information. The metrics are combined into reports for each data collection, including parameters and filters to be applied, e.g. retrieving results only for the specified wards inside a hospital. An example of a concrete IDM model is shown in Fig. 6, together with generated data collection and reporting artifacts.

Fig. 6.

The fragment of the IDM model for the COPD project. The top bold section describes the data item to be collected, while the bottom section specifies the measure to be represented in the SPC report. The WISH renderings of the generated data collection and reporting interfaces are shown on the right.

IDM maintains separate definitions of data elements for each model instance, so as to support finely-grained customisations that are frequently required. Therefore, the sharing of knowledge happens on the level of model instances, which are analysed and adapted by the local teams to best suit their purposes. For example, a project looking to implement its own Chronic Obstructive Pulmonary Disease (COPD) improvement project may look at a similar COPD projects in another setting, directly copy several data elements and measure algorithms, modify labels and numerators in some other measures, and add several new ones to construct the IDM instance that best addresses their local problem. This is in contrast to the data collection models used in clinical trials, which are commonly formally validated, strictly regulated, and reused verbatim.

3.2. WISH architecture

The WISH software is an Electronic Data Collection and Reporting framework based on the IDM model. The tool extends several open source technologies, and is hosted inside a JBoss application server, making it compatible with all major operating systems. A deployment tool converts the XML files containing the IDM models into objects in the database, and generates the data collection pages and web reports as shown in Fig. 4.

Fig. 4.

WISH system consists of an application server with an Electronic Data Collection module that is used for entering improvement data, and a live reporting framework that visualises the data using Statistical Process Control methods. The data collections and reporting measures are based on IDM model XML files which are deployed into WISH. Collected data and user information is stored in the underlying database.

All collected data is stored inside a MySQL database. The relational schema, shown in Appendix A contains a project registry, with user authentication and role authorisation tables that are used upon login to present each user only with the projects they are participating in, and in accordance with the roles they have had granted. Each project model is mapped onto one table, named after that projects acronym. Project tables store the user identifier and a time stamp for each data entry, while qualitative data (e.g. PDSA cycles and text comments) are stored in separate tables with similar user attribution information.

The data collection tool is implemented using the JBoss SEAM technology, with user interfaces for data viewing and editing dynamically generated from the underlying database objects that are mapped through Hibernate technology. This fits in well with the overall model-based approach and ensures both the logical integrity of the system and the ability to rapidly generate new interfaces once a new IDM model has been added.

The reporting framework is based on the Eclipse BIRT technology, which was extended with custom code libraries for Statistical Process Control (SPC) functions, and layering of qualitative and quantitative metrics within a single chart. The qualitative and management reports are shared between all projects, and parameterised with individual project identifiers passed on from the invoking instance of the tool. The quantitative reports are constructed from the model at project creation time. The deployment tool extracts the data elements and the improvement metrics from the XML model, and creates individual tables and graph elements in the chart, together with all the user parameters and default values, and uploads it to the server.

3.2.1. Data collection

The data collection pages are dynamically constructed from the data elements defined in the IDM, with the input data persisted into the relational database. Forms consist of graphical widgets (text fields, drop-down boxes, calendars, etc.) associated with the data elements that the users fill in, with the field error checking provided at entry. The hidden elements that are automatically captured by the tool include the unique identifier for the entry made, exact date and time the entry was made, and the identity of the user making the entry.

In addition to the IDM-generated pages that are specific to a particular project, a number of standard data collection forms are available in all projects. These include forms for collecting Sustainability Model data [27], PDSA cycles [20], [21], qualitative data on the level of patient and public involvement in the initiative, and free-text comments on progress of the initiative. These additional data provide a qualitative view of the improvement process that adds a new dimension to the quantitative IDM data expressed through project metrics.

3.2.2. Reporting

IDM reports are based on the Statistical Process Control (SPC) approach [17], [16] which is specifically intended to provide insight into the nature of the variation exhibited in a measure, and is often applied to healthcare [28]. The SPC XmR charting in particular is a highly flexible and robust method of analysis, that is suitable for time series analysis in improvement [29].

The XmR charts in WISH plot monthly- or weekly-aggregated data against time. SPC limits can be switched on or off by the user, together with the additional parallel moving range plot used for detecting large jumps in the data; this is shown in Fig. 1. The user also has the option to overlay the run chart with annotations derived from qualitative data. Resulting graphs show PDSA cycles and free-text comments. Clicking on any of the annotations opens a new window with the dedicated report for that PDSA cycle or comment.

Fig. 1.

An example of one data measure being reported, taken from the COPD project. WISH improvement report is generated from the IDM model elements and uses XmR charts with Statistical Process Control (SPC) limits, and an additional moving range chart below. The graph is overlaid with qualitative Plan–Do–Study–Act annotations and textual comments that user can click on to get full details.

Dedicated qualitative improvement reports display the different types of qualitative data that are collected in all WISH projects. The Sustainability Model report uses custom bar charts that compare the weighted sustainability results in several categories with the maximum possible result, as specified in the sustainability metric definition document [27]. PDSA cycles, Patient-Public involvement and free-text comments, are all available as tabular reports to facilitate export into other document types.

The reporting component allows the user to export either the entire report in a graphical format, with PDF, Excel, and Powerpoint supported, or only export the core data into a delimited text file. In the case of the latter, a subset of metrics can be specified so that they can easily be imported into an external statistics tool for further analysis.

3.2.3. Enterprise features

To support deployments in large environments with users participating in multiple improvement projects, several enterprise features were implemented in the WISH software.

The WISH authentication and authorisation is based on a hierarchical role model in which users can have varying access permissions on different project, from just viewing entered data to adding new users. Typical user can enter new data, view and edit the data they entered, and run project reports. Lower permission levels are meant for external auditors and project observers who do not have edit rights and can only run reports. Higher permission levels are intended for project managers and administrators who can see and/or edit all data entered, run cross-project reports, and manage user permissions.

To enhance security and ensure auditability of collected data, whenever a user logs into the system, the details of the session are stored in the database, together with the mode in which the session ended – whether the user logged out manually or the session timed out.

Project managers have access to activity reports, such as the one shown in Fig. 5. They depict the integrated view of activity across multiple projects, e.g. within the CLAHRC NWL programme there is a need to target support to projects who need it within a particular funding round. The key generic indicators include the number of entries, the number of distinct active users entering data, and days since last data entry. A traffic-light colouring system is employed to quickly identify problem areas, and key figures in the table are hyper-linked to the detailed data report about the measure displayed.

Fig. 5.

Activity report for a set of projects is used to provide a common view on the activities of an entire programme of work, with semaphore schema highlights marking out projects that need additional support. Each table entry is linked to a further project-specific improvement report.

Finally, to facilitate communication between team members, a messaging system has been implemented that allows intra-project, inter-project and direct messages to be exchanged, with the group messages effectively acting as a news feed for project members.

4. Results

The IDM approach has been implemented in the WISH software and piloted within the CLAHRC NWL programme, funded by UK’s National Institute for Health Research. CLAHRC NWL ran between 2008 and 2013 as a five-year programme to tackle the problem of the second translational gap [6], with three aims: to improve care for patients in Northwest London, to develop a systematic approach to implementation of research findings into practice, and to build capacity and capability for healthcare research and improvement in northwest London. To achieve these aims the programme funded four overlapping rounds of 18-month improvement projects set in hospitals, health centres, mental health trusts and other environments. The programme has been funded to continue for another five years until 2018.

The IDM model-driven approach has been used in over 50 of these improvement projects and by more than 700 named users. It has also been adopted by three external medical organisations for their projects that are not associated with the programme. While the formal evaluation of the software effectiveness is not the subject of this paper, we shall demonstrate the usage of IDM and the WISH software environment in one of the CLAHRC NWL projects – COPD care bundle project at Northwest London sector hospitals.

4.1. COPD care bundle project

The Chronic Obstructive Pulmonary Disease (COPD) Discharge Care Bundle project was developed as part of an early CLAHRC NWL improvement initiative at Chelsea and Westminster Hospital, London, funded between 01/04/2009 and 01/10/2010 [30]. Care bundles are sequences of evidence-based interventions that a certain group of patients should receive [31]. The COPD bundle is targeted at all patients who are admitted to hospital with a diagnosis of acute exacerbation of COPD, and the goal of the project was to improve outcomes for these patients by ensuring that every patient received the care bundle. The elements of the care bundle, which must be in place at discharge from acute care, are:

• 1.Referral to smoking cessation service.
• 2.Referral to pulmonary rehabilitation service.
• 3.Patient is able to demonstrate and understand correct inhaler technique.
• 4.Patient has been given written information about their condition, including self-management advice.
• 5.Follow-up appointment is in place and given to the patient in writing.

The improvement measures, defined by the team to track performance, comprise percentage compliance with each of the elements of the bundle. Patients’ care is deemed compliant against an element if they received that element of care prior to discharge. Overall compliance is defined as the percentage of eligible patients discharged having received all 5 elements of the bundle. This initiative was subsequently rolled out to other interested trusts in northwest London, including The Northwest London Hospitals NHS Trust between 01/04/2010 and 01/10/2011.

Since COPD IDM model used for data collection did not specify any de-identified patient data, there were no ethical restrictions on the location of the WISH server. In projects where de-identified patient data is used, the server is placed inside the NHS N3 secure network [32], a Wide-Area Network (WAN), with 1.3 million NHS end users and over 40,000 connections in hospitals and other health organisations.

4.2. COPD improvement data model

The Improvement Data Model for the COPD initiative consists of a single data collection, with data attributes reflecting the elements in the bundle. thus each entry represents a patient admitted to hospital with COPD, with a data point for every bundle element. Fig. 6 shows the fragment of the COPD model, and how the data descriptor and the metric definition map to the generated data collection web page and the report. Smoking_Cessation attribute is a categorical text value (Yes/No/Declined) with null values allowed and maximum length of 45 with a defined label. The Smoking Compliance metric is defined as a percentage of patients who were referred to the smoking cessation program, and defined in terms of two aggregated values – number of patients in the time period (week/month) who were referred and total number of patients.

The full list of data elements in the model reflects the bundle elements:

• •Identified_Smoker. Was the patient identified as a smoker at admission clerking?
• •Smoking_Cessation. If yes, was the patient referred to smoking cessation service?
• •Suitable_For_Rehab. Is the patient suitable for pulmonary rehab?
• •Referred_For_Rehab. If yes, was the patient referred for pulmonary rehab?
• •Info_To_Patient. Was the patient (or carer) given the COPD Patient Information Pack?
• •Inhaler_Demonstrate. Satisfactory use of inhalers demonstrated and understood?
• •Followup_Appointment_Made. Has the followup appointment been made with the patient?

Similarly, the improvement metrics in the IDM data model consist of the percentage of patients that satisfy all compliance criteria, and compliance percentages for each individual element of the bundle.

4.3. COPD data collection interface

When the COPD project user logs in to the WISH user interface (Fig. 7 top), they are presented with an overview of all the projects that they are part of in any role (researcher, auditor, administrator, etc.). Clicking on the COPD project will take them to the COPD project page, where they can enter data and access reports, as specified in the COPD IDM model (Fig. 7 bottom). Present here is also the news feed for the project and the messaging interface for communicating with other project participants.

Fig. 7.

WISH tool welcome page (top) contains project selection panel and most commonly accessed functions, together with the messaging panel. Individual project data entry pages (bottom) are generated from IDM model.

4.4. COPD reports

The COPD report consist of a series of compliance measures for various parts of the COPD bundle, with an additional overall compliance measure. Fig. 1 shows an example control chart with SPC limits and additional moving range plot of weekly overall compliance with the care bundle for this initiative. The quantitative data is overlaid with qualitative PDSA annotations which the user can click to get further information. Moving average, qualitative annotations and SPC limits are all optional elements which can be switched on or off by the user.

4.5. Usage

Fig. 1 provides an example of successful improvement in a healthcare process, facilitated by the WISH tool. The team used WISH to capture, store, analyse and present data on compliance with the care bundle, showing clear improvement, and the organisation has subsequently decided to use the WISH tool to support a number of other improvement initiatives.

Compliance data has been collected and entered onto the WISH tool for this project weekly in real time since August 2010, beyond the end of the funded period (30th September 2011) and shows a sustained improvement in quality of care against this measure. At the time of writing, improvement data have been entered against 106 weeks (89% of the 119 weeks elapsed) for a total of 518 patients with documented care bundles (mean 4.4 care bundle entries per week) by 2 users. The team has also made 55 sustainability model entries, 23 comments and 20 PDSA cycles. The team continued to use the system even after the project ended, and the graph in Fig. 1 shows the sustained improvement.

The compliance of care increased during the project from an average of 34% in first 6 months (29/08/10 – 13/2/11), to an average of more than 80% for the next 14 months (13/02/11 – 08/04/12). This represented the culmination of a complex intervention to the delivery of healthcare, using IDM and WISH in conjunction with several other quality improvement techniques. While this makes it difficult to isolate the singular effect of WISH, it is encouraging that compliance increased as did the use of the tool, with largely positive feedback from the users.

4.5.1. User experience

Throughout the development of the application, user feedback has been incorporated into the testing and development via solicited and unsolicited user feedback. WISH includes an online feature request and bug management system, implemented in the Bugzilla package, so that any user of the system can submit a bug report or a feature request to the system. The feature requests submitted by users from COPD Bundle and other projects are then assessed by the WISH Team and if found to benefit to a wide number of users, are scheduled for implementation.

In addition to this continuous feedback, the team occasionally undertakes one hour sessions with individual users, working through a new version of the system to elicit opinion of different types of users. One such set of sessions took place in August 2012 with users from Chelsea and Westminster Hospital in London, who were asked to move through the application and were encouraged to express their experiences whilst finding and using features such as data entry, reporting, user management, and messaging. These thoughts were written down by the researcher verbatim and live, allowing the users to clarify any points. Observations regarding ease of use or bugs encountered were also noted. Comments collected were largely positive and constructive, and transcripts of the user testing were then discussed by the development team, and used to form the final task list for the upgrade, including features to be brought online post-rollout.

Another method of capturing user experience is through round table conversations between the WISH team and selection of various project team members, managers, and the public. These typically coincide with larger user gatherings, such as CLAHRC NWL Collaborative Learning Events, where the broader user group was present, but only those who actively wished to participate sat at the table, and were able to leave at any point of their choosing. When asked about the key features that made the system sustainable and usable, the users pointed out the high-level of flexibility and customisation, enabling them to fit the data capture and reporting to their projects. They also felt that such an electronic data capture and reporting system is essential to performing improvement work. In terms of future feature requests, most popular were the ability to link collected improvement data to other data sources, including staff and patient satisfaction, and the model creation tool to be made available to project leads.

At the time of writing, we are yet to conduct the full evaluation of the projects’ experiences in using WISH, since the feedback data collected during the course of the project was intended mainly for feature requests and is interlinked with non-informatics aspects of the improvement projects conducted. The analysis of the user acceptance of the full methodology will be published in a subsequent paper. For now, we rely on user adoption and overall project success rates as indirect measures of the effectiveness of the technology.

4.5.2. Reuse of COPD IDM model

COPD project also illustrates how the IDM model and WISH software can be used to disseminate successful initiatives. With minor modifications, an IDM descriptor of an intervention can be adapted to meet the needs of the local context, whilst maintaining the core aspects of the intervention as defined by research evidence. Correspondingly, the data models from successful projects are made available to subsequent project sites, as was the case with COPD, which was deployed in four further locations, each modified to meet the needs of the local context, e.g. by adding an optional compliance element of contacting the patient by phone, or by introducing a spirometry measurement. In this way, knowledge is re-used in new implementations of the intervention, conserving resources and establishing data-driven best practice.

4.6. Data governance

Collection and reuse of patient data in United Kingdom is regulated mainly by the Data Protection Act of 1998 and NHS Act of 2006. The aim of these acts is to regulate data flows in the health settings, protect a patient’s right to privacy, enforce duties of confidentiality and protect public interest in the benefits of research. The conservative interpretation of this framework is one of “consent or anonymise”, requiring explicit consent of any identifiable subject in a data set or full anonymisation [23] before data being used for research. However, section 251 of the NHS Act 2006, allows for identifiable information to be used when it is necessary and not practicable to seek consent or perform anonymisation.

The improvement data currently collected as part of COPD and other CLAHRC Northwest London projects is fully anonymised, and no legal restrictions apply to its usage and distribution. In improvement projects that do require de-identified data, WISH software can be deployed within NHS’s secure N3 network, which satisfies the security requirements of most NHS organisations. Still, usage of this type of data must be locally approved by the users through their internal procedures prior to software deployment. Future WISH projects are likely to include integration with existing Electronic Health Record systems, and we are currently establishing links with selected Government-approved safe haven providers to support that linkage.

5. Discussion

The gap between healthcare analytics and live clinical processes and, in particular, performance metrics and tasks being evaluated, has been noted in literature [9], [33]. We postulate that a large class of tasks, particularly local ones aiming to improve an existing process, cannot be adequately measured by exclusively using routinely collected data residing in hospital’s EHRs. While routinely collected data provide a valuable evidence base for large initiatives, local improvements often require specific additional data points and metrics that are not present, or easily extracted, from existing data sources. A model-driven approach allows for such data to be specified and collected and analysed by the local teams with little impact on the larger institutional processes.

The mismatch between the data that is required for the improvement task, and that is available in the local EHR system, mirrors a fundamental data issue in a number of domains, most notably in randomised clinical trials. Even though a certain percentage of data required from the patients in clinical trials does get routinely recorded in EHR systems, trials still use electronic Case Report Forms (eCRFs) to collect the exact data needed to achieve completeness and accuracy. There are research projects and commercial initiatives that investigate the integration of eCRFs with the EHR systems (TRANSFoRm [34], EHR4CR [35], Cerner [36], Trialviz [37]), but the standard practice is still to use dedicated data collection mechanisms that do not directly rely on existing data.

With that in mind, it is not surprising that the eCRF software market is a prime example of a model-driven approach to data collection. Software such as REDCap [25], Catalyst Web Tools [38] and OpenClinica [39] rely on trial data collections described using a dedicated model, and deploying the associated data collection interfaces to users in an automated and consistent fashion, typically via a Web tool. This is the same approach that IDM/WISH takes to improvement data. OpenClinica in particular uses CDISC’s Organizational Data Model (ODM) which has conceptual similarities with the data collection segment of IDM, while lacking the quality metrics that are present in IDM.

While REDCap in particular has been used for improvement data capture [40], both the tool and the underlying model lack the capabilities for supporting user-defined analytics, requiring researchers to download the data and import it into external tools such as Excel, STATA, or R. This is in contrast with the IDM, which incorporates the improvement metrics in the model, and the WISH tool which can show SPC charts and other reports at the click of the button, requiring minimal training. This reflects the wider target user group for the WISH software, which includes not only researchers, but clinical and administrative staff in the organisation that perform the improvement data collection and want to track their progress in real time. Another difference in the REDCap model is the explicit support for collection time points, which are essential for clinical trial data, but were found to add needless complexity when working with improvement data.

Statistical Process Control that is employed in WISH reporting has been used for analysing quality metrics in a number of health areas [28] where it was found to contribute to users distinguishing special from common cause variation. Its other benefits included ease-of-use, enabling valuable prediction of future process performance, helping describe and quantify process variability, and increasing process transparency. Medical domains where SPC was used include laboratory turn-around time, surgical site infections, and appointment access satisfaction, among others [18]. All of these examples were found to be similar to WISH improvement projects and could be implemented within the system.

SPC also forms part of Six Sigma [41], a popular approach to quality improvement developed by Motorola in the 1980s, which focuses on the reduction of defects or errors in a process to extremely low levels. Six Sigma draws on statistical tools from SPC, thus software providing SPC analysis is well-suited to organisations applying Six Sigma. In addition, Six Sigma contains Define-Measure-Analyze-Improve-Control (DMAIC) cycles, which are conceptually similar to the Plan-Do-Study-Act cycles used in WISH, and can be visualised in the same way.

The model-based approach implemented through a software infrastructure is in contrast with the current efforts by the Institute for Health Improvement, which opted for a collection of separate tools, including an Improvement Tracker, PDSA worksheet, and Run Chart tools, inputs and outputs from which have to be managed by the users. While, once configured and deployed, these tools do provide some similar features, they still do not support data warehousing, enterprise features, or automated reporting, relying instead on the users putting these together manually. No other single collaborative software infrastructure comprising similar features is known to the authors.

The design of IDM and WISH is agnostic of the regulatory environment in which they are to operate. In United Kingdom, use of WISH for collecting and analysing non-patient-identifiable quality improvement data requires only internal institutional approvals, while future integration with Electronic Health Record systems will introduce the need for deployment on the N3 secure network and performing secure data linkage in safe havens. Deploying WISH in the United States organisations would proceed likewise for quality improvement tasks that are not considered research activities, however integration with the EHR systems would require a full IRB research approval, since WISH model requires dates of health care services to be present in the data, and this prevents the dataset from being considered “limited” under HIPAA. Given the specific nature of quality improvement data, projects in other legal jurisdictions would require similar considerations to be studied.

Improvement Data Model (IDM) and Web Improvement Support in Healthcare (WISH) were developed to provide full software support for improvement teams wanting to implement evidence-based interventions in their environments. It is a collaborative, enterprise tool that supports multiple concurrent improvement initiatives, with users who can be parts of different initiatives in various roles. The data collected and analysis provided in each initiative consists of quantitative measures, bespoke to each project, and a set of generic qualitative data measures, including PDSA cycles, improvement sustainability data and others. The collected data is immediately accessible to users as a set of live reports, and the software runs inside a standard web browser. The tool and the associated method are now moving beyond the initial user base of CLAHRC NWL projects and towards wider adoption, with three external UK organisations running pilots in 2013, and others being negotiated. WISH software is available under the Apache Foundation license version 2.0 at http://www.wish-tool.org.

6. Conclusion

Motivated by the challenge of the second translational gap, and using local improvement projects to rapidly translate research into practice, we proposed a model-driven approach for facilitating electronic data collection and reporting in health settings and demonstrated it on a Chronic Obstructive Pulmonary Disease improvement project run in Northwest London hospitals. Through doing so, we discovered conceptual overlap between various data-driven medical domains that face the challenge of collecting and managing data from diverse research and clinical contexts. A key factor that sets improvement apart from other domains, e.g. clinical trials, is the need for immediate feedback implemented through live reports from collected data, which is well served by encapsulating both data collection and report definitions inside a single model.

Whilst improvement data collection and reporting could be developed and embedded within the context of existing EHR and hospital systems, the bespoke nature of the data, together with the need for fast deployment, and a number of highly improvement-specific tools, make such solutions costly, complex and inefficient. IDM model-driven approach ensures that custom data collections can be efficiently integrated into the system and deployed to the users. Combined with an enterprise infrastructure, such as WISH, it can be used to support any number of improvement projects within a health organisation, with minimum resource overheads, thus leading to a culture of data-driven improvement.

Our future work shall focus on fully integrating IDM within the larger family of data collection standards, primarily CDISC’s ODM, and extending them with reporting facilities that we have introduced. Also, through introduction of flexible common data elements, we shall extend the model with integration points to existing EHR systems to allow parts of it to be reused in model-defined data collections and reporting, while preserving the customisation flexibility required by the improvement projects. Achieving such standardisation across different medical domains will significantly contribute to integrating research outputs into clinical practice.

7. Disclaimer

This article presents independent research commissioned by the National Institute for Health Research (NIHR) under the Collaborations for Leadership in Applied Health Research and Care (CLAHRC) programme for North West London. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Appendix A. WISH Database schema

Fig. A.8.

Fig. A.8.

WISH relational database schema. Each improvement project data is stored in a separate table that is generated from IDM. In the diagram, only one such table is shown – copd_icht.