A strategy for measuring health outcomes and evaluating impacts of interventions on asthma and COPD—common chronic respiratory diseases in Global Alliance against Chronic Respiratory Diseases (GARD) countries

Introduction

The use of standardized, evidence-based indicators is essential for the assessment of risk factors for chronic respiratory diseases (CRDs) and their overall burden, as well as the evaluation of clinical and public health interventions across countries (1). The collection and reporting of accurate and timely health information, including disease-specific measures of morbidity and mortality, helps to direct the allocation of health care resources and optimizes decision making in health policy. International comparisons of CRD burden are primarily drawn from existing population-based studies. For example, the International Study of Asthma and Allergy in Childhood (ISAAC) has provided information on the prevalence of asthma symptoms in children using samples from 56 countries (2). The European Community Respiratory Health Survey is another example of the standardized collection of information from multiple countries, providing abundant information on morbidity, risk factors and the cost of CRDs (3). There are also the PLATINO studies (4) and the BOLD studies (5), which have examined the burden of chronic obstructive pulmonary disease (COPD) across Latin America and other continents, respectively. Existing CRD work by the World Health Organization (WHO) includes a report of findings from the World Health Survey (6), iterations of the Global Burden of Diseases reports (7), and the GARD Basket (8).

The use of common measures for the assessment of modifiable risk factors and health outcomes in research and practice allows for the comparison of the effectiveness of different interventions. Unfortunately, the availability, applicability, interpretation and usefulness of some indicators have not been widely assessed or demonstrated, suggesting an urgent need for a common/uniform strategy to assess the impact of health initiatives and to measure the overall global burden of CRDs.

Therefore, the objective of this paper is to recommend a strategy to support GARD countries to conduct impact assessments and estimate the burden of CRDs using evidenced-based, validated CRD indicators and a stream-lined process to statistical reporting. Our recommended strategy will form a supporting assessment tool to GARD countries that are implementing individual and community-based interventions that support the Action Plan of GARD (9) and the 2013−2020 Action Plan of The Global Strategy for Prevention and Control of Non-communicable Diseases (10). It should be noted that, whereas CRDs are a broad category encompassing many conditions, recommending a strategy that incorporates all CRDs is beyond the scope of this paper. Thus, from this point forward we focus more specifically on asthma and COPD, common respiratory conditions for which there are existing evaluation frameworks in support of this work. The proposed strategy presented herein is meant to illustrate the evaluation process and may serve as a template that can be modified, as appropriate, for the evaluation of other CRDs in the future.

CRD surveillance and evaluation strategy

The Global Alliance against Chronic Respiratory Diseases (GARD) is a WHO-affiliated alliance of national and international organizations, institutions, and agencies committed to the common goal of reducing the global burden of respiratory diseases. GARD was established in 2006 with the following aims: (I) to develop a standard way of obtaining relevant data on CRDs and risk factors; (II) to encourage countries to implement health promotion and CRD prevention policies; and (III) to recommend strategies for CRD management (11-13). Currently, there are over 50 countries and 50 organizations affiliated with GARD (14,15). Although GARD countries share a common vision and mission, there is diversity amongst countries that is important to recognize when considering a common strategy because “one size may not fit all”. Health priorities and geographic variability in the risk factors and epidemiology of different CRDs require that any recommendations be adaptable to ensure their appropriateness in the community where they will be applied (16). The diversity of health care systems across countries and jurisdictions, and variations in the availability and affordability of treatments is another relevant issue demanding tailored assessment (16-18). It is crucial that the strategy be flexible to accommodate local adaptation, while still maintaining the integrity of the system.

The following sections outline the proposed step-by-step approach for a CRD surveillance and evaluation strategy, giving examples of a resulting statistical report and highlighting opportunities and challenges for implementation.

Step 1: adopting evidence-based indicators to assess quality of care

CRDs, chronic diseases of the airways and the other structures of the lungs, represent a wide array of serious diseases (12). Major preventable CRDs include asthma, rhinitis, COPD, occupational lung diseases, sleep apnea syndrome and pulmonary hypertension (12). Worldwide, hundreds of millions of people of all ages are affected by these preventable CRDs. Globally, asthma and COPD are two common causes of chronic disease burden as measure by disability-adjusted life years (19). However, chronic rhinitis, either allergic or non-allergic, is even more prevalent and associated with temporary disability (20), but has not yet been assessed by the studies of the Global Burden of Diseases.

Despite the existence of well-established disease management guidelines, variations in the quality of asthma and COPD care are common in primary care settings and there is a large gap between evidence-based recommendations and real world practice. As a result, these conditions are associated with high morbidity and health-care use, and narrowing these gaps may significantly reduce morbidity associated with CRDs. Community-based performance (or quality of care) indicators can help identify barriers to, and enablers of, the uptake of clinical guidelines for CRD management. Currently, there are no universally recommended or standard CRD (asthma- or COPD-specific) quality of care indicators that are being used to measure process and health outcomes as part of a broad, comprehensive CRD evaluation, surveillance or monitoring system. Guideline development alone is insufficient; policymakers are shifting from a “science of recommendation” to a “science of implementation”, emphasizing the need for a better understanding of how performance measures can lead to improved care.

In recent years, the European Union funded IMCA (Indicators for Monitoring COPD and Asthma) project (1), Canada (21,22), the United Kingdom (23,24), Australia (25), the Global Initiative for Asthma (26) and a joint task force of the American Thoracic Society and the European Respiratory Society (27) have all suggested different sets of indicators for assessing the burden of CRDs. The Global Initiative for Chronic Obstructive Lung Diseases and other organizations have proposed indicators for COPD (5,28,29), which are applicable for large population-based studies (30). In general, indicators were selected based on the best available expert opinions and evidence. The data sources for the indicators in these four domains may be different, ranging from national population health surveys to independent research studies.

Previously, the Ontario Asthma Surveillance Information System (OASIS, http://lab.research.sickkids.ca/oasis/) in Canada has developed 15 evidence-based primary care asthma performance indicators (PC-APIs) (Figure 1) to evaluate the quality of asthma care in the primary care setting (31). These indicators covered nine quality of care domains: (I) diagnosis [use of pulmonary function test (PFT)]; (II) medication use; (III) asthma control; (IV) exacerbations; (V) health care use; (VI) the use of action plan; (VII) asthma education; (VIII) smoking cessation; and (VIIII) measure of quality of life. These indicators were generated based on 10 years of published literature and a critical assessment from 17 expert panelists using a modified Delphi approach. These indicators were pilot tested and demonstrated feasibility in Ontario in the primary care setting (32). Common indicators such as smoking cessation, use of spirometry, monitoring symptoms and use of action plan can be applied in asthma, COPD or other CRDs. Depending on the specific needs and priorities of GARD countries, these indicators can be used as they are (i.e., all 15 indicators) or selected based on priorities. For example, in Ontario, Canada, based on the gaps of both asthma and COPD care, it has been recommended to consider 5 of the 15 indicators for implementation: use of spirometry for diagnosis, assessment of control, provisions of controller medication, patients received a written action plan and smoker patients received smoking cessation advice (see details in Figure 2).

Figure 1.

Asthma performance indicators.

Figure 2.

An Example of Selecting and Using some of the Primary Care Asthma Performance Indictors (PC-API) to Measure Both Asthma and COPD Quality of Care

Additional indicators may be used to supplement the PC-API. Further to modifying the suggested indicators based on country-level needs and priorities, we also encourage the consideration of any other measures that may be more appropriate to the country or intervention being evaluated. These may be drawn from the WHO’s 2018 Global Reference List of 100 Core Health Indicators and Sustainable Development Goals (SDGs) (33). An adapted sub-set of core health indicators that are relevant to CRDs for consideration is outlined in Figure S1. In addition to measures of disease burden and health service utilization similar to the PC-API, this set of indicators also recognizes the potential importance of other population-level measures; for example, measures of disease-related productivity loss, the overall prevalence of tobacco use, and exposure to clean fuels and environments. The WHO list may be of particular use to low- or middle-income countries, where the PC-APIs may be less appropriate. Where suitable, patient reported outcomes (PROs) such as asthma control can be collected using the Asthma Control Test (ACT) or CAT (COPD Assessment Test) for COPD.

Figure S1.

Proposed core health indicators and health-related sustainable development goals relevant to CRDs. Adapted from the 2018 Global Reference List of 100 Core Health Indicators (plus health-related SDGs) (33). *, more information on the measurement of economic productivity losses can be found in the WHO Guide to Identifying the Economic Consequences of Disease and Injury (37). CRD, chronic respiratory disease; SDG, Sustainable Development Goal; WHO, World Health Organization.

Step 2: identifying reliable data sources for populating the CRD indicators

It is essential to identify the origin of CRD indicators in order to have some insight on the quality of the data and their levels of evidence. Often the data sources are at a national or regional level, which may come from national population health surveys, independent published research studies or census data. The European Union funded IMCA project (1) suggested that the sources of data for CRD indicators can be classified into the following groups:

• ❖ Available from routine data and no modifications required;

• ❖ Available from routine data but methodological changes required;

• ❖ Available from national surveys (less than 10 years);

• ❖ Available from national surveys (more than 10 years);

• ❖ Available from specific research surveys (less than 10 years);

• ❖ Available from specific research surveys (more than 10 years);

• ❖ Not available and, in the future, data should be developed from routine data;

• ❖ Not available and, in the future, data should be developed from public health surveys;

• ❖ Not available and, in the future, data should be developed from specific national or international surveys.

GARD countries will need to conduct their own “environmental scan” to identify the data sources for the CRD indicators for their countries. In addition, documentations on the definitions used in those data sources will be necessary to ensure that comparison of indicators across countries would be suitable. Data not meeting these quality standards may not be included or used for reporting.

Step 3: uploading data for analysis

Once the data sources are identified and deemed reliable, the data collected by the GARD country will be ready to be transmitted and processed centrally. The data collected may be at different levels; for example, national or regional grouped level data or data at the individual patient level. Either way would be accommodated by the PC-API. When available, individual level data will be paramount to monitoring patient-specific outcomes and disease progression. Individual level data will be “rolled up” to a group level (e.g., regional or national) for the evaluation of programs and interventions, as well as for the overall assessment of CRD burden.

To support the use of these evidence-based PC-APIs to measure quality of care, impact on process and health outcomes and for surveillance, the GREAT (Global Research And Training; http://lab.research.sickkids.ca/oasis/greatnetwork-pc-api/) Data Centre of the OASIS built an electronic and online tool using RedCap^© (Research Electronic Data Capture; https://redcap.vanderbilt.edu/). Participating GARD countries can use the system with internet access; no specific software or hardware installation is required. Data will be securely transmitted and stored in the Research Institute of the Hospital for Sick Children, Ontario, Canada—the home of the GREAT Data Centre. Alternatively, aggregated data could be sent to the GREAT Data Centre by email, regular mail or fax using a fillable PDF or hard copies of completed indicator forms (such as Figures 1 and 2).

Step 4: generating common statistics and metrics for comparisons

If individual patient level data are submitted on an ongoing basis, a quarterly report will be generated by the GREAT Data Centre and sent to participating countries and sites for their information. The quarterly report is based on aggregating the patient data, so no individual patient level data will be released, thereby safeguarding patient privacy and protecting data confidentiality. Each participating GARD country and site will see their site-specific statistics. Annually, based on data submitted by all participating GARD countries, an overall GARD CRD Indicator report will be generated showing the distribution of CRD burden by indicator and by country. A sample report is shown in Figure S2. This report will highlight areas for GARD countries to target for intervention and disease management in order to reduce the CRD burden. These reports will be presented annually during the GARD General Meeting, given to the WHO, and may result in a peer-reviewed publication.

Figure S2.

Example of statistical report on indicators.

The GREAT Data Centre has the mission of (I) establishing and promoting a methodological strategy for outcome evaluation; (II) supporting uniform data collection and statistical analyses at country and regional levels; (III) encouraging cross-country and global comparisons; and (IV) assessing the impact of activities undertaken in support of GARD objectives. As well, since 2010, the GREAT Data Center has been providing students, trainees, clinicians, researchers and policymakers with a platform for knowledge transfer and exchange. The Center will offer ongoing support to GARD activities through hosting a platform for learning (methods), sharing and exchanging (experiences and challenges), generation of knowledge through health policy investigative research.

The data gathered across countries and over time will ultimately contribute to the establishment of a global CRD surveillance system, yielding information on CRD morbidity, mortality and the need for primary health care services (34-36).

Discussion

In this paper, we proposed a strategy for CRD surveillance and the evaluation of CRD interventions in GARD countries. This strategy is implemented through four steps: (I) adopting evidence-based indicators to assess CRD quality of care; (II) identifying reliable data sources for populating CRD indicators; (III) uploading data for analysis; and (IV) generating common statistics and metrics for comparisons. We also recommended the use of a simple standardized and platform-free tool (PC-API) to measure and compare asthma quality of care. The GREAT Data Centre will provide ongoing support to centralize collection and reporting of data, and will also facilitate cross-country comparisons of CRD burden over time.

For every country interested in developing a GARD action plan, these indicator data should be collected and reported on an ongoing basis for the population-level surveillance of CRDs, as well as prior to and following the implementation of interventions. However, the availability of these data may vary across countries, and the proposed strategy will need to be modified so that it can be adapted to local circumstances. Regional differences in the delivery of public health and health care services may also affect the interpretation of these indicators. For instance, whereas a low rate of CRD-related ED visits in a developed country might reflect a lower relative disease burden, the same finding elsewhere may be related to inadequate access to hospitals or lack of health care in general rather than actual disease burden. Considering this, we recommend that additional core health indicators relevant to CRD evaluation (see Figure S1) be used to supplement the PC-API in the case that these measures are not relevant to the GARD country or specific intervention being evaluated.

The proposed CRD health indicators will help address fundamental questions related to disease burden, management and outcomes. Our proposed surveillance and evaluation strategy is intended for use by health professionals, researchers, managers and policymakers on decision-making, with special attention to primary health care. The recommended uniform approach to data collection and analysis will further enhance global comparisons and the evaluation of the impact of the GARD approach in different countries.

Management thinker Peter Drucker is often quoted as saying “ … you can’t manage what you can’t measure and if you can't measure it, you can't improve it…”. We would not know whether or not our implemented programs or interventions are successful unless success is defined and tracked. With clearly established metrics, we can monitor but also quantify progress and adjust our processes to produce the desired outcome. Our proposed CRD surveillance and evaluation strategy will allow physicians, researchers, hospitals and health care organizations to identify and prioritize gaps in care, design strategies to address them, and track improvement or lack of improvement. The ultimate goal of this strategy of evaluating impacts and measuring performance is to improve health outcomes, to increase the efficiency of investments in prevention and control of CRD, and to provide people living with CRDs care that is safe, effective, patient-centered, timely and equitable.

Disclaimer: Some authors are staff members of governmental and nongovernmental organizations. The authors alone are responsible for the views expressed in this publication and they do not necessarily represent the decisions, policy or views of their organizations.