SUMMARY
India’s Revised National Tuberculosis Control Programme (RNTCP) used the international benchmarks of 70% case detection rate and 85% treatment success rate among new smear-positive tuberculosis (TB) cases for assessing programme performance. This approach overemphasises outcomes and focuses on quantitative benchmarks without sufficient regard to developing systems to monitor appropriate programme practice to achieve a minimum standard of TB care services. The RNTCP has developed a novel composite indicator tool based on a logical framework pathway to move beyond narrow-focused outcome indicators such as case detection to encourage a broad-based analysis of programme implementation. The constituent indicators are from routinely monitored information, spanning input, process, output and outcome indicators across various thematic categories of the RNTCP.
Keywords: monitoring and evaluation, tuberculosis, programme implementation, India
Monitoring and evaluation are essential components of national tuberculosis (TB) control programmes (NTPs).1 Measurements of both implementation outcome and impact are required for budgeting and resource allocation, long-term planning and policy development.2,3
Like most NTPs, India’s Revised National Tuberculosis Programme (RNTCP) used the monitoring and evaluation strategy for assessing programme performance against the international benchmarks of a 70% case TB detection rate and an 85% treatment success rate among new smear-positive TB cases4 at district, state and national levels. This approach emphasises outcomes, focusing on quantitative benchmarks without giving sufficient regard to developing systems that monitor appropriate programme practice to achieve a minimum standard of TB care services.
Case detection is widely recognised as a problematic indicator, mainly due to uncertainties about TB incidence.5 In India, the national TB incidence rate is based on mathematical models, which may not provide an accurate estimate. Applying a national case detection to monitor state and local programme performance may thus be inappropriate, as it ignores the underlying heterogeneous distribution of TB disease among local populations. Furthermore, administrators and programme managers from districts and states that achieve the 70% case detection benchmark become complacent, despite experiencing significant underlying technical and operational issues that impede programme performance.
In an attempt to improve the assessment of programme performance, more rapidly identify programmatic areas for improvement and ultimately enhance the quality of service provided in India, the RNTCP developed a novel composite indicator tool based on a logical framework pathway. The structure was designed to move beyond narrow-focused outcome indicators such as case detection, and to encourage a more broad-based analysis of programme implementation.
Constituent indicators of the composite score were derived from routinely monitored information, spanning input, process, output and outcome indicators across various programmatic thematic categories of the RNTCP. Specific indicators focused on human resource, financial management, epidemiology (e.g., case-finding efforts), quality of services, drug procurement and distribution and programme logistics (Table 1). These thematic categories were assigned weights relevant to programme implementation based on the opinion and experience of a panel of experts. A standard grading scale was developed taking into account both current performance and trends in previous reporting periods, to develop a composite score. An automated composite score at the district, state and national levels can be generated via the RNTCP’s reporting system, EpiCentre, using routinely collected aggregated quarterly reports. EpiCentre also produces district-level unweighted categorical scores and an overall composite score on command (Table 2). Achieving 80% of the maximum possible score in a thematic section is considered a qualitative indicator of success (labelled as good). This design not only focuses on the outcome indicators, it also integrates the overall scores and encourages a more broad-based analysis of programme implementation.
Table 1.
Composite indicator: sub-component scores and number of indicators in each thematic area
| Section | Sub-component | Scores* n (%) | Indicators | Indicators n |
|---|---|---|---|---|
| 1 | Human resource management | 65 (26) | Vacancies of certain key positions and period of vacancy; training status and competency of staff | 2 |
| 2 | Financial management | 20 (8) | Payments to staff (e.g., DOT provider honoraria) | 1 |
| 3 | Case-finding efforts | 30 (12) | Trends in presumptive TB examination rate; trends in case notification rates; cases with presumed MDR-TB referred for culture and drug susceptibility testing | 3 |
| 4 | Quality of services | 115 (46) | Default rates; TB-HIV collaborative activities such as percentage of TB patients tested for HIV; participation of NGO/private providers; community involvement | 8 |
| 5 | Drugs and logistics | 20 (8) | Availability of drug stocks; storage and maintenance; transport | 1 |
| Total | 250 (100) | 15 |
Scores of >80% are graded as good performance, those <50% as under-performing and those between 50% and 80% as average performance.
DOT = directly observed treatment; MDR-TB = multidrug-resistant TB; TB = tuberculosis; HIV = human immunodeficiency virus; NGO = non-governmental organisation.
Table 2.
Sample composite score report District A: January–March 2012
| Section | Thematic areas | Scores, n/N (%) | Remarks |
|---|---|---|---|
| 1 | Human resource management | 52/65 (80) | Good |
| 2 | Financial management | 20/20 (100) | Good |
| 3 | Case-finding efforts | 19/30 (63) | Requires improvement |
| 4 | Quality of services | 104/115(90) | Good |
| 5 | Drug supplies and logistics | 8/20 (40) | Requires improvement |
| Composite score | 203/250 (81) | Good |
Constituent indicators are readily measured and have a firm denominator against which they can be scored. This design enables measurement not only of current performance but also of past performance. For example, failing to hire key staff and leaving positions vacant for long periods is scored more severely than simply having a short-term vacancy. A district management unit can see each of the categorical thematic scores and an overall composite score (Table 2). Districts can explore why a particular category score is low, perform a more comprehensive review of this programme area and implement specific corrective action. For example, if a district scores low in human resource management, it is possible to review whether any key positions were vacant, if staff need training, if salary payments and other benefits have lapsed or whether attrition was low. The district can thus identify specific weaknesses and take targeted, corrective action.
These composite performance indicators do have a few limitations. First, although based on routinely collected and reported data, the data are aggregated at the district level; it is thus difficult to assess quality. Second, indicator selection and assignment of weights were based on expert opinion and experience and are yet to be validated against qualitative programme performance measures or impact. Third, it remains unclear if this tool can be translated into public health action to reduce the burden of TB, as some managers may continue to focus on the score and not the meaning, and may seek to selectively improve only those aspects informing the indicators. The indicator weighting and assignment may accordingly evolve over time, and calculations may be withheld from line workers in the future. Continuous training and support for programme managers will be essential to minimise the effects of these limitations.
Since March 2012, composite indicators have been available at the district, state and national levels to monitor programme performance, improve universal access and provide quality diagnostic and treatment care services. Given this implementation experience, and to provide a more evidence-based process for continual programme improvement, the RNTCP plans to validate the composite score against a representative sample of district-level internal evaluations. These evaluations measure the qualitative aspects of programme implementation and reflect current practice.
Prima facie, the composite indicators seem to be an effective programme management and monitoring tool for gauging performance. The holistic design brings an incisive insight into programme management through the identification of specific programmatic constraints leading to targeted interventions for improved performance.
Footnotes
Conflict of interest: none declared.
References
- 1.Centers for Disease Control and Prevention. Essential components of a tuberculosis prevention and control program. Recommendations of the Advisory Council for the Elimination of Tuberculosis. MMWR Recomm Rep. 1995;44(RR-11):1–16. [PubMed] [Google Scholar]
- 2.Lee T, Price M. Indicators and research methods for rapid assessment of a tuberculosis control programme: case study of a rural area in South Africa. Tubercle Lung Dis. 1995;76:441–449. doi: 10.1016/0962-8479(95)90012-8. [DOI] [PubMed] [Google Scholar]
- 3.Phillips VL, Teweldemedhin B, Ahmedov S, Cobb J, McNabb SJ. Evaluation of program performance and expenditures in a report of performance measures (RPM) via a case study of two Florida county tuberculosis programs. Eval Program Plann. 2010;33:373–378. doi: 10.1016/j.evalprogplan.2010.01.001. [DOI] [PubMed] [Google Scholar]
- 4.World Health Organization. Global tuberculosis control, epidemiology, strategy and financing. WHO/HTM/TB/2009.411. Geneva, Switzerland: WHO; 2009. [Accessed April 2014]. http://www.who.int/tb/publications/global_report/2009/en/index.html. [Google Scholar]
- 5.Borgdorff MW. New measurable indicator for tuberculosis case detection. Emerg Infect Dis. 2004;10:1523–1528. doi: 10.3201/eid1009.040349. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ministry of Health and Family Welfare Government of India. Revised National Tuberculosis Control Programme – India. Strategy document for the supervision and monitoring of Revised National Tuberculosis Control Programme. New Delhi, India: Directorate General of Health Services, Ministry of Health and Family Welfare Government of India; 2012. [Google Scholar]
- 7.Ministry of Health and Family Welfare Government of India. Tuberculosis India, 2012: annual report of the Revised National Tuberculosis Control Programme. New Delhi, India: Directorate General of Health Services, Ministry of Health and Family Welfare Government of India; 2012. Revised National Tuberculosis Control Programme – India. [Google Scholar]