Field-testing of primary health-care indicators, India

Abstract

Objective

To develop a primary health-care monitoring framework and health outcome indicator list, and field-test and triangulate indicators designed to assess health reforms in Kerala, India, 2018–2019.

Methods

We used a modified Delphi technique to develop a 23-item indicator list to monitor primary health care. We used a multistage cluster random sampling technique to select one district from each of four district clusters, and then select both a family and a primary health centre from each of the four districts. We field-tested and triangulated the indicators using facility data and a population-based household survey.

Findings

Our data revealed similarities between facility and survey data for some indicators (e.g. low birth weight and pre-check services), but differences for others (e.g. acute diarrhoeal diseases in children younger than 5 years and blood pressure screening). We made four critical observations: (i) data are available at the facility level but in varying formats; (ii) established global indicators may not always be useful in local monitoring; (iii) operational definitions must be refined; and (iv) triangulation and feedback from the field is vital.

Conclusion

We observe that, while data can be used to develop indices of progress, interpretation of these indicators requires great care. In the attainment of universal health coverage, we consider that our observations of the utility of certain health indicators will provide valuable insights for practitioners and supervisors in the development of a primary health-care monitoring mechanism.

Introduction

Under the thirteenth general programme of work and the triple billion targets,¹ the World Health Organization (WHO) aims to increase the number of people benefitting from universal health coverage (UHC) by one billion between 2019 and 2023. Central to this effort is the expansion and improvement of primary health-care services.^2,3 Progress in achieving UHC can be analysed using the WHO and World Bank’s UHC monitoring framework,^4,5 but this requires adaptation to local contexts to ensure health reforms keep pace with targets. Health programmes in India,⁶ as well as the national health policy⁷ and flagship Ayushman Bharat scheme,⁸ are being evaluated in relation to the aims of UHC; various efforts are currently underway at both a national⁹ and state level, notably in Haryana¹⁰ and Tamil Nadu.¹¹

According to National Sample Survey estimates from 2017–2018, morbidity levels in the southern state of Kerala are reportedly four times the national average with disparities by sex and place of residence.¹² Although the state has made gains in maternal and child health,¹³ it must sustain these gains while addressing the substantial and growing burden of hypertension, diabetes¹⁴ and cancer;¹⁵ vaccine-preventable diseases;^16,17 and emerging viral infections such as Nipah virus¹⁸ and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).^19–21 Kerala has been subject to unregulated privatization and cost escalation,²² resulting in persistent inequalities in service access and health attainment between population subgroups.¹³ In 2016, the Government of Kerala announced Aardram, a programme of transformation of existing primary health centres to family health centres;²³ with increased staffing, these family health centres provide access to a greater number of services over longer opening hours compared with the original primary health centres.

Apart from the WHO’s monitoring framework,²⁴ many countries have done UHC and primary health centre monitoring exercises^25,26 alongside independent exercises such as the Primary Health Care Performance Initiative.²⁷ However, most of these frameworks are intended for global comparison or decision-making at national levels. The argument for tracking health reforms is clear, but such a monitoring process must be specific to Kerala and local decision-making, while also complying with national and global reporting requirements. Periodic household surveys offer population-level data, but are not frequent enough to inform ongoing implementation decisions. Routinely collected and disaggregated health system data are vital,²⁸ but are often marred by quality issues as well as technological and operational constraints.²⁹

We began a 5-year implementation research study assessing equity in UHC reforms in January 2018. In our first two phases we aimed to develop a conceptual framework and a health outcome indicator shortlist, followed by validation of these indicators using data from both health facilities and a population-based household survey. We report on the field-testing and triangulation components of this implementation research project, which took place during 2018 and 2019.³⁰ We reflect on early lessons from the field-testing and triangulation and, drawing broadly from Ostrom’s institutional analysis and development framework,³¹ we emphasize how monitoring can support learning health systems.^32,33 We also discuss how the monitoring of UHC progress requires a flexible approach that is tailored to the local political economy.^34–36

Methods

Study design

We began with a policy scoping exercise for the state of Kerala in 2018. We then created an 812-indicator longlist from existing primary health-care monitoring inventories,^9,37–40 and undertook an extensive data source and mapping exercise, adapting a process previously conducted in the region.⁴¹ We applied a modified Delphi process in two rounds, consulting key health system stakeholders of the state (frontline health workers, primary care doctors, public health experts and policymakers), and obtained a shortlist of 23 indicators (available in the data repository).⁴² We then field-tested and triangulated some of the indicators using facility-based data (phases 1 and 2) and a population-based household survey (phase 2).

Phase 1: facility data collection

In phase 1 (December 2018) we selected three family health centres in coastal, hilly and tribal districts (Trivandrum, Idukki and Wayanad, respectively) of the state. We communicated the definitions and logic of the indicators to facility staff, and studied their data-recording methods to synergize our processes with theirs. From these initial steps, we prepared a structured data collection template (available in the data repository)⁴³ that we provided to the three family health centres.

Phase 2: facility data collection

Based on inputs from phase 1 and a second round of consultations with state-level programme officers, we refined the indicator list. In phase 2 (June–October 2019), we used a multistage random cluster sampling technique to generate data related to the indicators at the population and facility level. We applied principal component analysis using Stata version 12 (StataCorp, College Station, United States of America) to data from the latest National Family Health Survey (2015–2016)⁴⁴ to categorize districts into one of four clusters according to health burden and systems performance. Using an open-source list randomizer from random.org, we randomly selected one district from each of the four clusters, and then randomly selected both a primary and a family health centre from each of the four selected districts. The people served by these eight health facilities were the population of interest in our study.

We held on-site meetings with the staff of the eight health facilities and provided them with Excel-based templates (Microsoft Corporation, Redmond, United States of America) to input data for the financial year March 2018–April 2019 (data repository).⁴³ Data were sourced from manual registers maintained at facilities. In addition to off-site coordination, we also provided data-entry on-site support to the health staff, visiting each facility at least four times between May and December 2019. We compiled data from the facilities to obtain annual estimates for all health outcome indicators using Excel.

Phase 2: household survey

Our sample size estimation was based on the proportion of men and women eligible for blood pressure screening under the national primary care noncommunicable disease programme, that is, those aged 30 years or older. We estimated a sample size using routine data reported by the noncommunicable disease division of the Kerala Health and Family Welfare Department (2018–2019), aiming at a precision of 8% at a 95% confidence interval (CI), with a conservative design effect of 2 (i.e. a doubling of the sample). Health facility catchment areas were grouped by wards, also referred to as primary sampling units. Eligible households within a primary sampling unit had at least one member aged 30 years or older. Individual written informed consent was sought from each participant before administration of the survey. We employed and trained staff to collect data using hand-held electronic tablets with a bilingual (English and Malayalam) survey application. The survey, conducted during June–October 2019, included questions on sociodemographic parameters, health outcome indicators (e.g. noncommunicable disease risk behaviours and screening; awareness of components of Aardram and family health centre reform) and financial risk protection (e.g. out-of-pocket expenditure). National Family Health Survey (Round IV) state level weights were applied during analysis.⁴⁴

Triangulation of phase 2 data

We compared data on selected indicators using Stata and Excel. Since our focus was on how indicators were being understood and reported across facilities, we did not expect indicators to directly correspond between facilities and households, but only to approximate each other.

Ethics

All components of the study were approved by the Institutional Ethics Committee of the George Institute for Global Health (project numbers 08/2017 and 05/2019).

Results

We obtained data from 11 health facilities in total (seven family health centres and four primary health centres) during phases 1 and 2. During phase 2, we acquired facility data on indicators from eight health facilities (four family, four primary) jointly serving a population of 273 002 (Table 1). The household survey was undertaken in the catchment areas of these facilities, and we acquired data from a representative sample of 13 064 individuals in 3234 households (Table 1).

Table 1. Demographic characteristics of sampled population in field-testing of health-care indicators, Kerala, India, 2018–2019.

Characteristics	Family health centresa	Primary health centresa	Total
Population served by facility	161 317	111 685	273 002
No. male (%)	79 841 (49.5)	54 190 (48.5)	134 031 (49.1)
No. female (%)	81 476 (50.5)	57 495 (51.5)	138 971 (50.9)
Population of households surveyed in catchment area of facilities	5022	8042	13 064
No. households surveyed	1631	1603	3234
No. of people aged ≥ 30 years (%)	3076 (61.2)	4760 (59.2)	7836 (60.0)
No. male (%)	2435 (48.5)	3756 (46.7)	6191 (47.4)
No. female (%)	2582 (51.4)	4279 (53.2)	6861 (52.5)
No. other (%)	5 (0.1)	7 (0.1)	13 (0.1)

Note: Inconsistencies may arise in some values due to rounding.

^a Data combined from four different family health centres and four different primary health centres during phase 2 of the implementation study.

We observed both variations between and uniformity in the indicators from health facilities and the household survey (Table 2). In studying these patterns, we made four key observations (Box 1).

Table 2. Triangulation of health-care indicators between populations served by facilities and from household survey, Kerala, India, 2018–2019.

Health-care indicator	No. (%) served by facilitya	No. (%; 95% CI)b from household surveyc
Pregnant mothers who received all recommended types of antenatal care for most recent live birth in the past year (facility: n = 2255; household: n = 94)	2479 (109.9)d	85 (90.9; 84.7–97.1)
Children < 5 years with diarrhoea in the past year (facility: n = 13 552; household: n = 900)	912 (6.7)	195 (21.6; 18.1–25.2)
Low birth weight among newborns in the past year (facility: n = 2803; household: n = 157)	177 (6.3)	17 (11.0; 2.9–19.1)
Pre-check servicee by staff nurse, where available (facility: n = 146 643; household: n = 1152)	98 139 (66.9)	801 (69.5; 59.1–79.9)
Blood pressure screening in the past year (facility: n = 273 002; household: n = 6367)	39 081 (14.3)f	5467 (85.9; 84.5–87.2)g
Blood glucose screening in the past year (facility: n = 273 002; household: n = 6367)	33 965 (12.4)f	5254 (82.5; 81.0–84.0)g

CI: confidence interval.

Note: Inconsistencies may arise in some values due to rounding.

^a The reference period for the facility survey, for which only crude totals reported in registers were available, was April 2018–March 2019.

^b Note that numbers used to calculate percentages were provided by Stata to 2 decimal places; we have reported these as whole numbers.

^c Weighted using the Fourth National Family Health Survey, India.⁴⁴

^d Antenatal care data from one family health centre were considered invalid and excluded from analysis.

^e Includes vital sign assessment and case history recording of patients by a staff nurse in a designated area, which is entered into the e-health portal to be available in real time to the consulting physician. Pre-check coverage is now reported as a daily outpatient tally, and is adjusted for returning patients.

^f Calculated as number of individuals screened as a proportion of total population served by the facility; age-disaggregated eligible population (≥ 30 years) data were not available.

^g Data only obtained from individuals aged ≥ 30 years.

Box 1. Four observations from field-testing and triangulating health-care indicators, Kerala, India, 2018–2019.

Observation 1: Data are available at the facility level, but in varying formats and platforms meant for different purposes; digitization may improve this situation.

Observation 2: Established global indicators may not be useful or interpreted as intended in a local context, and may need to be adapted.

Observation 3: Operational definitions, thresholds for interpretation and processes of routine data collection must be refined for older indicators and developed for newly introduced indicators.

Observation 4: Triangulation and feedback from the field level, with qualitative input from local actors, remains vital, particularly for chronic diseases.

First, the method of reporting our indicators varied between facilities, even although all raw data required to calculate selected indicators were present in manual registers. In the case of indicators related to national programmes (e.g. reproductive, child health and tuberculosis-related indicators), data were uploaded directly to national digital portals without any analysis at the facility level; officers responsible for data compilation and analysis exist only at the district level. Feedback from facility staff included requests for adequate training on new or revised reporting systems, and clarification of their role. This situation may improve with the complete digitization of health records under Kerala’s e-health programme.

Our second observation is that there exist two problems with the globally recommended indicators: (i) manual routine data reporting at the facility level may be inadequate to construct the global indicator precisely; and (ii) globally relevant data may not be considered relevant to the periodicity (monthly) or level (facility) of review. From the facility-level data, the coverage of antenatal care reported by family health centres was 109.9% (2479/2255); in household surveys, full coverage of antenatal care was observed for 90.9% (85/94) of eligible women (Table 2). Here, antenatal care refers to women aged 15–49 years having a live birth in the past year and receiving four or more antenatal check-ups, at least one tetanus toxoid injection, and iron and folic acid tablets or syrup for at least 100 days as numerator. The coverage rate is calculated from a denominator of the total number of women aged 15–49 years who had a live birth in the past year, which requires retrospective verification of antenatal coverage. However, in some facilities, the antenatal care coverage indicator was calculated using the previous year’s number of deliveries plus 10% as the denominator, and the number of pregnant women who had received antenatal care as the numerator. It was therefore not always clear that the data from any particular individual were included in both the numerator and denominator and, with a target as the denominator, coverage could surpass 100%. Practitioners noted the disconnect between monthly target-based reporting and annual retrospective measurement.

Our third observation is that definitions and reporting that reflect actual health-provision patterns require to be standardized; otherwise, discrepancies will be observed between data sets. For example, the indicator for acute diarrhoeal diseases among children younger than 5 years was 6.7% (912/13 552) according to facility records; however, a prevalence of more than 3 times this percentage (21.6%; 195/900; 95% CI: 18.1–25.2) was reported in the household survey (Table 2). Several chronic care indicators, newly introduced as part of the introduction of family health centres, also showed discrepancies. For instance, the percentage of people screened for blood pressure and blood glucose was 85.9% (5467/6367; 95% CI: 84.5–87.2) and 82.5% (5254/6367; 95% CI: 81.0–84.0), respectively, according to the household survey; however, facility data yielded a screening coverage of 14.3% (39 081/273 002) and 12.4% (33 965/273 002), respectively. These observed discrepancies could be the result of: (i) a large proportion of the population being screened for certain conditions and risk factors in the private health sector; (ii) the fact that age-disaggregated data are not available at the facility level; and/or (iii) while the national guidelines recommend population-based screening of all adults aged 30 years and older, Kerala has introduced opportunistic screening for blood pressure among all aged 18 years and older. Discrepancies were not observed for all indicators, however; results for low birth weight and pre-check services were similar between the two data sets.

Our fourth observation is that such triangulation exercises, as well as obtaining feedback from health workers, programme managers and administrators, are vital for accurate assessment of UHC coverage.⁴⁵ A major problem reported by staff and officials is that health facility data are usually just a tally of patient visits, which is simple to produce, as opposed to the actual number of (potentially repeat) patients receiving care or services. State officials have been encouraging a move towards electronic health records to generate more precise indicators, but adoption and integration of these will only be possible when the technology itself is better aligned to facility-level process flows, requiring user inputs, investment and time. Other issues raised include: the need for appropriate staff (including temporary contractual staff) training in programme guidelines and reporting requirements; the need for clarity in definitions of treatment (e.g. chronic disease patients may be advised to modify lifestyle factors, which would be missed if treatment monitoring included only those prescribed medication); and the availability of free or subsidised tests relevant to disease control that are reflected in monitoring indicators, particularly for chronic care (e.g. glycated haemoglobin tests for diabetes care⁴⁶) at the primary health centre level.

Discussion

As already observed in India and other low- and middle-income countries,²⁹ our results indicate that any approach to improving or monitoring the quality of health-care must be adaptable to local methods of data production and reporting, while ensuring that emerging concerns of local staff are considered. Although validity checks are a staple of epidemiological and public health research, such triangulation processes in health systems are infrequent. The Every Newborn-BIRTH study was a triangulation of maternal and newborn health-care data in low- and middle-income countries,⁴⁷ and some smaller-scale primary-care indicator triangulation exercises have been undertaken by India’s National Health Systems Resource Centre.^48,49 While there exists a variety of approaches to monitoring primary health-care reforms,³⁰ we consider the most appropriate to be the generation (and modification, if necessary) of indicators from routine data, and their triangulation with household survey data.⁴⁹

Increasingly, routine data are being digitized to improve accessibility and interpretation, as is the case in Kerala. Useful considerations when introducing digital health interventions in low- and middle-income countries are intrinsic programme characteristics, human factors, technical factors, the health-care ecosystem and the broader extrinsic ecosystem.⁵⁰ Our observations demonstrate the continuous and complex interplay between these characteristics; the real value of selected indicators may also be determined by how staff understand and interpret them.

Our study had several limitations. Our indicator selection using the Delphi method could have undergone additional rounds, but we considered it more important to get the monitoring process underway and reduce the burden on health workers. Some facility-based information could not be acquired due to the additional health department burden of flood relief and Nipah outbreak management in the state. Our household survey sample was the population aged 30 years and older, resulting in undersampling for other indicators being field-tested (e.g. newborn low birth weight). An increase in sample size could allow a more precise estimation of all indicators. Finally, the reference periods for the facility data and the household survey did not directly overlap; a timed sampling should be undertaken in the future to improve the precision of triangulation.

Observing the utility of indicators in practice is a key first step in the move towards UHC, requiring investment and commitment. Using indicators, standards and other forms of technology, which are easy to adopt, can be problematic because we amplify certain aspects of the world while reducing others.⁵¹ Our examination of family health centre reforms cautions that, while data can be used to develop indices of progress, interpretation of these indicators requires great care precisely because of the way they are related to powerful decisions around what constitutes success or failure, who will receive recognition or admonition and, ultimately, the legacy of Aardram reforms. We anticipate that our observations will contribute to health-care reforms in low- and middle-income countries, such as the use of field triangulation to enhance the accountability and relevance of global health metrics.³⁶ If such activities are carried out in constructive partnerships with state stakeholders and do not introduce unfeasible costs to the system, they may contribute to a sustained and reflexive monitoring process along the path to UHC.