Evidence-based Public Health Policy Models Development and Evaluation using Big Data Analytics and Web Technologies

Abstract

Introduction.

According to WHO, “health policy refers to decisions, plans, and actions that are undertaken to achieve specific health care goals within a society”. Although policymaking is important to be based on scientific evidence, in many countries, evidence-informed decision-making remains the exception rather than the rule.

Aim:

This work presents a cloud-based Decision Support System for public health decision-making.

Methods:

In CrowdHEALTH, the concept of a Public Health Policy (PHP) is directly connected with one or more Key Performance Indexes (KPIs). The design and technical details of the system implementations are reported, along with use case scenarios.

Results:

The Policy Development Toolkit presents a unique interface and point of reference for policymakers, allowing them to create policy models and obtain analytical results for evidence-based decisions and evaluations.

Conclusions:

The hierarchical structure of the Public Health Policy Model offers versatility in the creation and handling of the policies, resulting in Health Analytics Tools Results Objects which offer quantitative policy support and provide the basis for meta-analytic operations.

1. INTRODUCTION

According to a definition given by the World Health Organization (WHO), “health policy refers to decisions, plans, and actions that are undertaken to achieve specific health care goals within a society” (1). Although policymaking must be based on scientific evidence, in many countries, particularly low- and middle-income countries, evidence-informed decision-making remains the exception rather than the rule (2). Even into high-income countries internal data, reports and the opinions of internal staff members are the kinds of information used most frequently instead of research evidence (3-5). In this direction, evidence-based policies comprise one of the European Union’s (EU) priorities in the public health area for 2016-2020 (6). The five-fold increase in annual publications in the field of health policy and systems research over the period 1990-2015 and analogous funding, demonstrates the worldwide growth of interest for evidence-based health and policies (7). However, evidence-based policies require the existence of large-scale comparable data for obtaining secure forecasts in potential interventions. On the one hand, there are cases where there is a lack of appropriate data. Two such issues are mentioned in the current strategic plan of the EU Directorate-General Health and Food Safety where evidence is hard to produce and so not sufficiently conclusive for health policy regulation: innovative topics and controversial results (8). On the other hand, there is such an abundance of data that we enter the domain of Big Data Analytics (BDA) in Healthcare (9). To tackle both of these directions, scarcity or abundance of data, EU responds -among others- with the Health Programme (10) and Horizon 2020 (11) (providing funding on health projects) and the strengthening of partnership within EU agencies that produce scientific evidence. CrowdHEALTH (12, 13) is an international research project co-funded by the European Commission that integrates high volumes of health-related heterogeneous data from multiple sources intending to support policymaking decisions. The front-end of the platform is the Policy Development Toolkit (PDT), a health policy creation and evaluation environment, which provides advanced decision support, through data-driven analytic tools, both in aggregate as well as in personalized fashion. The modularity of the architecture and the secure big data processing workflow is presented in (14, 15). In this work, we focus on the structure and implementation of a Public Health Policy Model (PHPM) and the environment of the PDT by which policymakers can implement such models. The rest of this section describes related work. Section 2 provides the structure of a PHPM and the interactions with PDT. Section 3 presents the PHPM modeling process in practice within the PDT environment following a real use case scenario and Section 4 presents concluding remarks and directions for future work.

2. RELATED WORK

Agent-based dynamic simulation platforms to identify beneficial policies and interventions have been recently reported for cases such as childhood obesity control (16), the impact of sugar-sweetened beverage warning labels (17), the relation of urban crime with obesity (18), and reducing alcohol-related harms (19). The platforms take residential and sociodemographic data and, through experimental scenarios, estimate the probability and the evolution of various factors. The participation of stakeholders in running simulations and different scenarios builds an understanding of the modeling process, and thus trust in the model and its outputs as a decision-support tool (20). Recently, many projects have been spawned in the direction of evidence-based policymaking via the effective use of big data analytics. We can divide them into two categories. Into the first category, heterogenous big-data datasets are collected, even real-time, to produce quantitative evidence supported by what-if scenarios. Such projects are BigO (Big data against childhood obesity) (21), BD2DECIDE (Decision support for cancers of the head and neck region) (22), iASiS (Big data for precision medicine) (23) and MIDAS (Meaningful Integration of Data, Analytics, and Services) (24). In the second category, there is an additional layer, where the scientific evidence is framed in a way to support the formulation of public health policy models and their management. The EVOTION Project (Big data for hearing loss interventions) (25), to the best of our knowledge, is the only other attempt, with specific outcomes, to formulate evidence-based policies. PHP decision making (PHPDM) models are structures having the following set of building elements: Goals, Objectives, Decision Criteria, Data, Factors, Types of Analysis and Policy Actions (26). The ontology instance of the PHPDM is compiled through a reasoner, producing the corresponding Big Data Analytics (BDAs) components for the delivery of quantitative results (27).

From the reported early attempts to develop platforms assisting policymakers to benchmark, simulate and forecast outcomes of healthcare policy decisions, we can discern unmet challenges towards many directions, some of which are listed here (28):

• Representing a health policy with measurable and quantitative variables.

• Finding, collecting, converting and handling big data sources at spanning time scales.

• Covering the sensitivity of personal data, security, and trustworthiness.

• Distributed reusable Big Data Analytics independent of cloud vendors, architectures or analytics frameworks.

• Full tracking and versioning of developed PHPMs along with supporting evidence and confidence intervals/error metrics.

• Hiding technical complexity, providing easy interaction of the policymaker with the platform. Next, we describe the implementation of the system to address these challenges.

3. AIM

This study sought to describe the structure of a Public Health Policy Model as defined in the CrowdHEALTH Project which sets the baseline for the web-based policy creation environment, the Policy Development Toolkit - PDT. Use Cases presents aspects of the PDT in the process of evidence-based public health decision-making.

4. METHODS

In CrowdHEALTH, the concept of a Public Health Policy (PHP) is directly connected with one or more Key Performance Indexes (KPIs). A KPI is a countable and measurable indicator linked to the related PHP, and it is expressed utilizing a formula. This way, and by the notion of evidence-based decisions and policies, we minimize ambiguity when referring to the meaning of a Health Policy (14).

Figure 1 shows the hierarchical structure of a Public Health Policy Model (PHPM) along with the main properties of each component:

Figure 1. Hierarchical Structure of a PHPM.

• PHP: The Public Health Policy, with its description, the owner id, and the list of KPIs components that support this policy

• KPI: The Key Performance Index, with its description, the purpose of the index (including stakeholders), actors, Datasets, a formula for its calculation, and the list of Analytics Tools supporting/producing the Index.

Currently, the Analytics is included in one of the following categories (for a detailed explanation please refer to (14)):

• - Risk Stratification

• - Multimodal forecasting

• - Causal analysis

• - Clinical pathway mining

• HAT: Health Analytics Tool, with its description, dataset(s), and the provided service Analysis. The latter describes the RESTful endpoint, the type of visualizations it supports, and the parameter types with their constraints the Tool is expecting as input. HATs are created by data analysis and computer experts. They may reside in different platforms utilizing different frameworks and big data analytics. However, the communication with the PDT and the Datastore Backend is made through a common protocol followed by all HATs. The results from HATs executions are following, again, a common JSON (JavaScript Object Notation) format, with directives for the proper type of visualizations to be shown in the PDT. The three components: PHP, KPI, and HAT (Health Analytics Tool) exhibit a many-to-many relation in between. For instance, a KPI can contain many HATs that calculate, predict or analyze the index, and a HAT can be linked to support many indexes. In the same manner, a Policy model may depend on many KPIs, and a performance indicator can be part of many Policies. For persistence, the whole PHP model can be exported as a simple JSON file. The key-value data structure and the ability for nested objects and nested arrays can help to capture the whole PHPM hierarchy.

The analytics results are also in JSON format, following a common grammar for all HAT outputs.

Figure 2 shows the three main components of the CrowdHEALTH platform, along with the information context of their interactions. PDT is the Frontend of the system, hiding the technical complexity of the overall distributed architecture and the message exchange between the components.

Figure 2. System Components: Policy Development Toolkit, Backend/Datastore and HATs.

Figure 3 shows the main interactions of the policymaker (as a primary actor) with the system. Roles for the users determine access to relevant actions, and users with a PolicyAdmin role can create-modify-delete their PHPs. These actions appear in the PDT as buttons, depending on the current context. The HAT Developer registers his or her tool in the Backend, after which it is accessible from the PDT. Next, policies can be built with KPIs that are linked to the HAT, in one of the available analytics categories. Policymakers can evaluate the KPI of a PHP by submitting specific parameter values to the HAT and getting back the outcome of the analytics. The Transactions History component collects all the policymaker’s evaluation requests and facilitates the policy formulation and tuning process. Numbered use cases in Figure 3 are described in detail in the following section.

Figure 3. Use case diagram for the main interactions of the Policymaker.

5. RESULTS

The Policy Development Toolkit - PDT is the web application frontend which facilitates all the interactions with the policymakers in the construction and the handling of PHPMs. The User Interface intentionally hides the complexity of the system to smooth the process of making policy models. The design is based on a Service Oriented Architecture where communication with HATs and the Datastore is facilitated through RESTful web services. PDT is built as a Single Page Application developed using the Angular framework and integrates Angular Material (29) User Interface components. The open-sourced full-fledged Angular framework offers multiplatform targeting without particular hardware or other software requirements.

As an example, in the following Use Cases, we take a PHP aimed at providing support to school physicians and pediatricians to help in the early detection of children with increased health risks linked to poor physical fitness and obesity. The PHP is described in the Slovenian National Program on Nutrition and Physical Activity for Health 2015-2025 (30). One of the proposed KPIs is the health-related fitness index (31), which is an overall evaluation of physical effectiveness according to age and gender using the SLOfit Data (32).

5.1. Creation of a PHPM (Use Case 1-2)

PDT offers two ways of constructing a new PHPM: the bottom-up approach by starting from scratch using an empty JSON template, and top-down by cloning an existing PHPM. In the second case, the created model changes ownership to the current policymaker initiating the action. Figure 4a shows the blank policy creation option at the end of the existing policies list, which may be either system policies or belonging to a specific policy owner ID.

Figure 4. PHPM Creation process: (a) From an empty template, (b) By cloning an existing policy model.

In Figure 4b, the user has chosen the ‘Obesity Prevention Policy in Schools’ PHPM, which is a system policy, and clones it to a new PHPM to take ownership and start modifications in a new PHPM. The cloning of a PHPM copies the whole structure of the policy model, including the supporting KPIs and the included HATs.

5.2. Addition of KPIs to the PHPM (Use Case 3)

Once we have ready a PHPM, the next step is to add/ remove or modify existing KPIs that are going to support a policy. The user selects existing KPIs which are close to the context of their policy and opts for a simple copy or a cloning action. The two options differ in the KPI’s behavior to future changes and modifications:

• The copy KPI action, adds the KPI to the PHPD as a reference to the original KPI. So, if the user modifies the KPI, the changes will propagate to all the PHPMs which are using this KPI as a reference as well. The same happens, if another policy expert or the initial owner of the KPI modifies it.

• The clone KPI action returns a deep copy of the KPI, making it an independent copy of the original. In this way, all future modifications, either to the original KPI or to the cloned one, will not propagate to other PHPMs. Figure 5 shows the KPI selection process through the PDT, where a list of existing KPIs is presented, along with the option of copying or cloning the selected KPI(s). Hovering over a KPI displays related info, as to help the policymaker in the assembling of KPIs into a policy model.

Figure 5. KPIs selection step.

5.3. Addition of HATs to the KPI (Use Case 4)

Each KPI should be supported/evaluated with at least one HAT producing the relevant index. In the same way as before, the user can select existing HAT(s) for inclusion into the current KPI. Figure 6 shows the list with available HATs. Hovering over a HAT displays related info, which here is the HAT with the name “Forecasting Analytical Tool” (ID 1).

Figure 6. HAT selection.

5.4. Evaluation/Analytics results (Use Case 5-6)

With the addition of HAT(s) to the KPI(s) of a PHPM, the creation of the new policy model is complete. The PHPM has a unique ID and it is under the ownership of the policymaker who created it. Other policymakers can see the model, run its Analytics Tools and create a copy of the policy if they wish to further modify or adjust it to their purpose. Now that the first version of the PHPM is ready, the policymaker can start the evaluation of the KPI(s) through the HAT(s), which is the essential step for the creation of quantitative evidence regarding a policy. Multiple submissions can be sent, covering various sets of parameter values, to explore the range of the outputs and study what-if scenarios. The PDT component responsible for the collection of research data-based evidence is called “My Transactions History”. It hosts all the user’s submissions to the HATs along with the results of the executions. Figure 7 shows the start of a transaction list, in reverse chronological order. The timestamp, user description of the parameter-value set during submission, the status of the execution and the ID of the job, label the card for each transaction. The titles of three transactions are shown regarding the HAT “Forecasting Analytical Tool” which is the HAT selected in Figure 6. The transactions execute the forecasting of the related KPI for three Slovenian regions (6, 10, 12). Figure 8 shows the content of one card for the transaction related to the region no. 6, which is a binding of the submitted values set along with the visualization of the results. This is a Health Analytics Results Object (HATRO). Each HAT can present its result using different types of graphs and/or with a combination of visualizations. The policymaker, by selecting groups of their transactions to display, can collect evidence for different parameter-value sets related to one or more KPIs. The assembling of a portfolio of related HATROs accumulates the research data for the support of the policy in focus.

Figure 7. User transactions with Analytics Tools results.

Figure 8. Parameter values set along with the visualized results.

6. DISCUSSION

The instantiation of policy models in the form of JSON files, as reported in Section II, provides flexibility regarding the properties of and the relations between the health policy objects. These policy objects may derive from corresponding ontology models to diminish ambiguity in their terms. Also, JSON files can easily be produced and consumed by the distributed RESTful endpoints through deployment in a cloud environment. Additionally, the hierarchical structure of the PHPM presented in this work offers a unique versatility in the creation and handling of health policies. Each layer is decoupled from the others, allowing the synthesis of models and the collection of research data in the complex social domain of healthcare. The calculation of a KPI can be carried out by a collection of HATs, each of those targeting different aspects of an Index, such as simulation, forecasting, causal analysis, semantic and linked data, etc. all of which are earmarked for policy-making purposes (33). Each HAT developer can define the statistical confidence/error metrics that their tool provides, depending on the analytics method and the dataset context. These metrics can be visualized either by utilizing the graph capabilities of the PDT or by graphs produced directly by the HAT. The modularity in the creation of Health Analytics Tools Results Objects (HATROs) is also one unique feature of the CrowdHEALTH architecture, from the perspective of the ability to create Meta-HATs. Meta-HATs are Analytics Tools that accept as input other HATROs, along with user-selected meta-processing options, to process them accordingly. In this way, chains of data-processing workflows can be assembled, based on a wide span of datasets, algorithms and population selection criteria.

However, more effort should be given in the versioning and traceability of all the HATRO properties. Each analytics result should be tagged with the identification & version of the used Datasource, along with the version of the HAT and visualization library. Furthermore, each HAT developer may employ different BDA frameworks using various deployment configuration mechanisms. The CrowdHEALTH architecture provides fertile ground for the growth of a HATs ecosystem (34), where Data Experts develop specialized HATs who register them in the CrowdHEALTH platform by following a common communication protocol. In consequence, HATs can operate and provide their services under different financial procedures, including commercialized options. As seen from the results of similar efforts mentioned in Section II, a need emerges towards the creation and adoption of a common standard for the structure of the PHPMs and their results, the HATROs. Such a standard would considerably lower the barrier of communication between policy supporting frameworks and provide the grounds for a global paradigm shift into big data public health policymaking.

CrowdHEALTH’s decoupling architecture can support and pilot such an effort. The healthcare industry has lagged other industries in the use of big data, primarily due to privacy concerns and fragmented clinical technologies (35). Nowadays, proper algorithms and protocols for protecting patient privacy have been put in place, and Electronic Health Records (EHRs) facilitate the massive collection of clinical data. Big Data Analytics can start consuming datasets and produce fine-tuned analytical and predictive models. Early detection of treatment protocols outcome and respective occurring costs are such cases (34-41).

Current HATs registered in the CrowdHEALTH platform and accessible through the PDT are spanning from clinical pathway mining to forecasting of clinical effectiveness to risk stratification regarding fraud detection.

7. CONCLUSION

The recent WHO’s Methods Guide stresses the value of evidence synthesis as a critical resource for health policymaking and health systems strengthening. The research evidence though should be the outcome of transparent and reproducible methods before their collection and appraisal. The CrowdHEALTH’s Health Policy Model is the core of the systems architecture which enables the collection of Health Analytics Results Objects for preparing and using policy briefs to support evidence-informed policymaking.

Authors

Each author gave substantial contribution to the conception or design of the work and in the acquisition, analysis and interpretation of data for the work. Each author had role in drafting the work and revising it critically for important intellectual content. Each authorgave final approval of the version to be published and they agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflicts of interest

The authors declare that there was no conflict of interest.

Financial support and sponsorship

Nil.