Abstract
Purpose
Understanding the resilience of tuberculosis healthcare services (TB-HSs) is crucial for the targeted reinforcement of weak links and mitigation of the impact of public health emergencies (PHEs). However, assessment systems in this domain are lacking. This study aims to construct a resilience assessment index system (RAIS) for TB-HSs in China.
Methods
The following four-step design process was conducted: 1) establishing the RAIS index pool based on a literature review, 2) designing an initial RAIS for TB-HSs following Donabedian’s “structure-process-outcome” framework through expert meetings; 3) employing a two-round Delphi survey to obtain a consensus on the RAIS; and 4) using the analytic hierarchy process to quantify the weight of each indicator included in the final RAIS.
Results
The expert panel (n=26) had 100.0% and 96.2% response rates in the first and second Delphi rounds, respectively. The expert authority coefficients for the two rounds were 0.908 and 0.906. Both rounds showed high levels of expert coordination (P<0.001). The final RAIS comprising three first-grade, nine second-grade, and 39 third-grade indicators. Our findings reveal that tuberculosis service provision/utilization tops the first-grade indicators, weighing 49.05%. Among the second-grade indicators, service outcomes (20.79%) and patient treatment (20.67%) were the top-weighted. Of the third-grade indicators, the treatment discontinuation proportion (10.29%) and the treatment coverage rate in confirmed TB patients (8.90%), were critical components in evaluating the resilience of TB-HSs.
Conclusion
This study developed a unified hierarchical resilience assessment index system for TB-HSs in China. Further research is required to validate and improve our results. The study’s findings could help develop strategies that benefit public health.
Keywords: tuberculosis, health system, resilience, evaluation, delphi method
Introduction
Resilience, originating in physics, materials science, and engineering, emphasizes the ability of physical materials to return to their original state after disturbances.1 The concept was introduced to the health system following the 2014 Ebola outbreak.2 The COVID-19 pandemic extensively disrupted essential health services in numerous countries and regions, causing substantial collateral damage and loss of life and livelihoods.3 Consequently, health system resilience (HSR) has garnered significant attention in the international community since 2020, particularly with regard to improving HSR and mitigating the damage and disruptions to essential health services caused by the COVID-19 pandemic and similar public health emergencies (PHEs).4,5 In October 2021, the World Health Organization (WHO) developed a position paper, “Building health systems resilience for UHC and health security during the COVID-19 pandemic and beyond”,6 which systematically presents the rationale and seven policy recommendations for building resilient health system based on primary health care. In March 2023, China’s document “Opinions on Further Improving the Medical and Health Service System” clearly put forward the goal of “forming an integrated medical and health service system compatible with the basic realization of socialist modernization by 2035”.7 Notably, resilience was first proposed by the Chinese government in designing the health system, highlighting its importance.
However, building a resilient health system faces three challenges: defining a resilient health system, evaluating health system resilience, and improving it.8 Although no international consensus exists on the definition of HSR,9,10 a widely recognized characteristic of resilient health system is the ability to maintain service continuity when facing external shocks.2 Therefore, the focus lies on how to improve the resilience of the health system, with comprehending its resilience being the prerequisite.11,12 However, consensus on how to assess the HSR is absent, and no tailored theoretical framework exists for the evaluating the HSR.8 Some studies have suggested that a resilient health system should be a capability rather than an outcome,13 and pointed out that positive effects are achieved by sufficient capability and effective strategies.13,14 Other scholars considered the assessment of health system performance during PHEs as an important aspect in measuring its resilience.15 In February 2024, WHO released a package of indicators for measuring and monitoring the resilience of a national health system,16 including 64 core indicators divided into three dimensions: input/structure, process, and output, covering seven domains, such as service delivery, health financing, workforce, for evaluators to choose reasonably. This indicator system measures health system resilience but requires tailored, clear setlist indicators for sub-health fields.
Tuberculosis (TB) is the leading killer among communicable diseases.17 The respiratory infectious diseases TB and COVID-19, as well as other similar PHEs, usually share health resources.18,19 Therefore, TB healthcare services (TB-HSs) are among the domains most susceptible to the shock of PHEs.20,21 Hence, it is significant and pressing to establish resilient TB-HSs. Nevertheless, owing to the scarcity of evaluation tools, it is difficult to assess the resilience of TB-HSs and direct their construction. China ranked third in the number of new patients with TB worldwide,17 and its TB-HSs have been affected by the COVID-19 pandemic with extensive, long-duration, and multiple waves during 2020–2022.19,22 Abundant experiences and lessons have been accumulated in carrying out TB-HSs during the COVID-19 pandemic.
In light of the above context, this study aims to construct a resilience assessment index system (RAIS) for TB-HSs in China, addressing a critical gap in the current literature and practice.
Methods
A modified Delphi study was conducted by consensus among experts and developed through a two-round Delphi survey. We used an analytic hierarchy process (AHP) to weigh the indices. The study lasted seven months in 2024. The selection of indices and initial RAIS designing began in April, the Delphi surveys were completed between July and September, and the final RAIS was completed in October. The RAIS constructing flowchart is shown in Figure 1.
Figure 1.
The flowchart of constructing a resilience assessment index system for tuberculosis healthcare services.
Abbreviations: TB-HSs, tuberculosis healthcare services; RAIS, resilience assessment index system.
Designing the Initial RAIS
We established a research team that included five experts specializing in TB control and responsible for defining resilient TB-HSs and formulating an index pool of the RAIS, which was completed by systematically reviewing the content of TB-HSs and analyzing the relevant literature on the resilience of health system and TB-HSs in the context of PHEs. We then introduced an expert meeting method to design the initial RAIS by referencing the index pool but not limited to this pool. The services of seven TB control experts, four of whom had experience handling the COVID-19 pandemic, were engaged. After thorough discussions at two expert meetings, the initial RAIS was established following Donabedian’s “structure-process-outcome” (SPO) framework.23
Expert Selection
Delphi experts were purposefully selected based on recommendations from expert meetings. The inclusion criteria required experts to be professionals in tuberculosis (TB)-related fields, with practice settings covering all seven geographic regions of China to ensure national representation. Their affiliations spanned four administrative levels: national, provincial, prefectural, and county. Additionally, no less than 50% of the participants were required to be from cities that implemented lockdowns lasting at least one month during 2020–2022 due to the COVID-19 pandemic. Furthermore, the experts had to meet the following criteria: 1) a minimum of five years of TB-related work experience, 2) a bachelor’s degree or higher, 3) an intermediate or higher technical title or a position at or above the section level, and 4) voluntary participation.
Delphi Process
The Delphi survey was conducted following the guidelines of the Conducting and Reporting Delphi Studies,24 in which experts were requested to fill in and return electronic questionnaires within 14 days of receipt. If they failed to do so within ten days, reminder phone calls were made. The returned questionnaires were summarized, and the indicator rating results and expert comments were analyzed. In the second round, a revised Delphi questionnaire and feedback on the results of the first round were simultaneously sent to experts, who were requested to re-rate the indicators with reference to the feedback.
The experts were asked to rate the four dimensions (necessity, feasibility, stability, and sensitivity) of each indicator using a 9-point Likert scale, with the level of agreement ranging from “1” (strongly disagree) to “9” (strongly agree). The experts were allowed to provide revision comments on the indicators. The experts’ judgment (Ca) and degree of familiarity (Cs) with each indicator were simultaneously rated based on self-evaluation. The basis for making the judgment comes from four aspects: practical experiences, theoretical analysis, counterparts, and intuitive feeling; each aspect was classified into three levels (large, medium, and mild) according to the experts’ degree of influence,25 (Table 1). The degree of familiarity also used the 9-point Likert scale, where “1” referred to “very unfamiliar”, and “9” referred to “very familiar”. In Round 1, experts were required to weigh the necessity, feasibility, stability, and sensitivity of the indicators, with the sum of these weights equal to 100.00%.
Table 1.
Expert Judgment Basis and Corresponding Scores
| Basis | Degree and Scores of Influences on Expert Judgment | ||
|---|---|---|---|
| Large | Medium | Mild | |
| Practice experiences | 0.5 | 0.4 | 0.3 |
| Theoretical analysis | 0.3 | 0.2 | 0.1 |
| From counterparts | 0.1 | 0.1 | 0.1 |
| Intuitive feeling | 0.1 | 0.1 | 0.1 |
Criteria for Indicator Revision
Based on the literature,26 the following criteria were applied for indicator revision:
1. An indicator 
with an importance score (
) of ≥6.5 and a coefficient of variation (
) of <0.25 was retained.
2. If the importance score (
) was between 3.5 and 6.5, or if 
was ≥6.5 but 
≥0.25, the indicator was required to enter the next round of Delphi survey.
3. An indicator with importance score (
) of <3.5 was removed.
The importance score (
) of an indicator was calculated as the arithmetic mean of the importance scores assigned by all participating experts. For each expert, the importance score (
) of an indicator was derived by summing the products of the weights and ratings for each of its four dimensions, as follows:
 |
where: 
=1−4 represents the four dimensions of the indicator (necessity, feasibility, stability, and sensitivity), 
denotes the rating result of indicator
in dimension 
(assigned by the expert), and 
represents the weight of the 
-th dimension, obtained by calculating the arithmetic mean of the expert assessments. Additionally, the 
is calculated as the standard deviation of all 
values divided by, 
reflecting the degree of dispersion among the importance scores assigned by the experts.
Regarding indicators suggested by two or more experts for amendment, addition, or removal, the research team decided through a thorough discussion.
Determining the Weight of Indicators
The differences between the importance scores of indicators within the same classes, when subtracted pairwise, were compared with a set value based on the Saaty 1–9 scale,27 and importance values were assigned accordingly (Table 2). Subsequently, judgment matrices were established, and the weights of each indicator were determined using AHP, provided that the judgment matrices passed the consistency test,28 which assessed the consistency of the judgment matrices. A consistency ratio (CR) of ≤0.10 indicated that the judgment matrix had acceptable consistency.
Table 2.
The Saaty1-9 Scale Assignment for Pairwise Comparison of Indicator Importance
| Value | Saaty Marked | Interpretation |
|---|---|---|
| Im–In=0.00 | 1 | m and n with equal importance |
| 0.25<Im–In≤0.50 | 3 | m is slightly more important than n |
| 0.75<Im–In≤1.00 | 5 | m is more important than n |
| 1.25<Im–In≤1.50 | 7 | m is obviously more important than n |
| Im–In>1.75 | 9 | m is absolutely more important than n |
| Other* | 2,4,6,8 | Adopted for compromising |
Notes: *Values of two adjacent level. Im and In represent the importance scores corresponding to indicators m and n in same class, respectively.
Statistical Analysis
Data were analyzed using SPSS software (version 22.0). Categorical data were described as frequencies and percentages, and the ratings of indicators were expressed as means and CVs. Independent Samples t-tests (Student’s t-test or Welch’s t-test, selected based on the results of Levene’s test for equality of variances) were performed to compare the mean importance scores of indicators between male and female experts. Differences were considered statistically significant at P<0.05. The online Scientific Platform Serving for Statistics Professional (www.spsspro.com) was used to calculate the weights of the indicators.
The following three parameters were calculated to evaluate the reliability of the Delphi survey results: 1) the expert positive coefficient, 2) the expert authority coefficient (Cr), and 3) expert coordination. The expert positive coefficient was reflected by the questionnaire return rate before the deadline, with a rate of >70% indicating high expert motivation.29 The formula for Cr calculation is Cr= (Ca + Cs)/2. The Ca score was computed based on judgment and degree (Table 1). The Cs value was derived by dividing the 9-point familiarity rating result by 9; Cr of >0.7 indicated high expert authority.30 Expert coordination was judged by Kendall’s coefficient of concordance (W). W ranges from 0 to 1, with higher values denoting greater coordination.31 The significance of W was determined using a chi-squared test.
Results
Characteristics of Delphi Experts
Among the 26 experts consulted, 12 were males (46.2%), mostly within the 40–49 age group. Twenty-one experts (81.8%) held doctoral or master’s degrees, and 20 experts (76.9%) held senior technical titles. Furthermore, 23 experts (88.5%) played administrative roles. Thirteen (53.8%) experts had provincial affiliations. All experts had over five years of professional experience in the TB field. Half of the participants were engaged in TB control (Table 3). Additionally, 23 participants (88.5%) were from cities that experienced COVID-19 lockdowns lasting over one month.
Table 3.
Characteristics of Experts Took Part in Delphi Survey
| Characteristics | n | % | |
|---|---|---|---|
| Gender | Male | 12 | 46.2 |
| Female | 14 | 53.8 | |
| Age(year) | 30–39 | 2 | 7.7 |
| 40–49 | 15 | 57.7 | |
| 50–60 | 9 | 34.6 | |
| Education level | Doctoral | 6 | 23.1 |
| Master | 15 | 57.7 | |
| Bachelor | 5 | 19.2 | |
| Technical title | Chief senior | 20 | 76.9 |
| Vice senior | 4 | 15.4 | |
| None | 2 | 7.7 | |
| Administrative position | Director | 9 | 34.6 |
| Department director | 14 | 53.9 | |
| None | 3 | 11.5 | |
| Affiliation level | National | 6 | 23.1 |
| Provincial | 14 | 53.8 | |
| Prefectural/county | 6 | 23.1 | |
| Professional orientation | TB control | 13 | 50.0 |
| TB clinical | 5 | 19.2 | |
| Management of TB diagnosis and treatment | 5 | 19.2 | |
| Disease control administration | 3 | 11.5 | |
| Professional years | ≥25 | 5 | 19.2 |
| 15–24 | 13 | 50.0 | |
| 5–14 | 8 | 30.8 | |
Abbreviation: TB, tuberculosis.
Reliability of the Delphi Results
Twenty-six questionnaires were distributed in both rounds of the Delphi survey, and the number (rate) of returned questionnaires was 26 (100.0%) and 25 (96.2%), respectively. This demonstrates a strong willingness to participate in the study. In Round 1, Cr was 0.908 (Ca = 0.886, Cs = 0.930), and in Round 2, Cr was 0.906 (Ca =0.879, Cs = 0.933), indicating high expert authority. In the first round, W was 0.130 (χ2=151.538, P<0.001), and in the second round, it was 0.315 (χ2=393.815, P<0.001), revealing good coordination among expert opinions in both rounds, with the second round showing even better coordination.
Establishment of the RAIS
Based on a literature review, we defined resilient TB-HSs in China as the continuous provision and utilization of TB case finding, diagnosis, treatment, and management services during PHEs. Accordingly, a pool of index systems comprising of 55 items was formulated (Table S1). Subsequently, the initial RAIS, including three first-grade indicators, nine second-grade indicators, and 34 third-grade indicators, was developed using the expert meeting method (Table S2). The weights for the four indicator dimensions were as follows: necessity (30.22%), feasibility (28.73%), stability (18.82%), and sensitivity (22.23%). Based on this, importance scores for the indicators were calculated using the following formula: necessity × 0.3022 + feasibility × 0.2873 + stability × 0.1882 + sensitivity × 0.2223. During the first round of the Delphi process, importance scores for all indicators ranged from 6.88 to 8.51, with CVs falling between 0.07 and 0.23. Statistical analysis revealed no significant differences in the mean importance scores of all indicators between male and female experts (P>0.05) (Table S3). Nineteen experts provided 59 suggestions for indicator revision. After synthesizing their opinions, the research team removed one third-grade indicator, amended six third-grade indicators, and added six third-grade indicators according to predefined criteria. During the second round of the Delphi survey, the mean importance scores for all indicators ranged from 6.92 to 8.49, with CVs falling between 0.04 and 0.13, and no statistically significant differences were observed in the mean importance scores of all indicators between male and female experts (P>0.05), with the exception of one tertiary indicator, which exhibited a borderline difference (P=0.049) (Table S4). Only one expert recommended deleting one indicator and modifying another. Ultimately, a consensus was reached, leading to the establishment of an RAIS for TB-HSs comprising three first-grade, nine second-grade, and 39 third-grade indicators (Table 4).
Table 4.
Final Resilience Assessment Index System for Tuberculosis Healthcare Services in China
| First-Grade Indicators (Code) | Second-Grade Indicators (Code) | Third-Grade INDICATORS (Code) |
|---|---|---|
| (F1) Service guarantee | (S1) Response preparation | (T1) Emergency plan formulation |
| (T2) Stock of diagnostic reagents for TB | ||
| (T3) Stock of anti-TB drugs | ||
| (T4) Stock of medical protective masks in TB-designated hospital | ||
| (T5) Amount of emergency reserve funds | ||
| (T6) Number of institutions with the capacity and authority to treat tuberculosis | ||
| (S2) Financial support | (T7) Actual per capita funding for TB services providing | |
| (T8) Actual per capita funding for TB patients | ||
| (T9) The actual additional emergency funds implemented for TB care services | ||
| (S3) Human resources | (T10) Number of TB control staff per 100,000 serviced population | |
| (T11) The ratio of TB control staff on duty during PHEs to those on duty before PHEs | ||
| (T12) Number of TB doctors per 100,000 serviced population | ||
| (T13) The ratio of TB doctors on duty during PHEs to those on duty before PHEs | ||
| (T14) Number of TB laboratory personnel per 100,000 serviced population | ||
| (F2) Service provision and utilization | (S4) Patients diagnosis | (T15) Timeliness of health-seeking for TB patients |
| (T16) The referral proportion of TB patients or suspected TB patients reported by non-TB-designated hospital | ||
| (T17) The proportion of patients who reached TB-designated hospital among those referred by non-TB-designated hospital | ||
| (T18) The proportion of patients who reached TB-designated hospitals among those who have been traced | ||
| (T19) The proportion of patients who undergo molecular biological testing upon their first visit to TB-designated hospitals | ||
| (T20) Timeliness of diagnosis for TB patients | ||
| (T21) The proportion of drug resistance screening among bacteriologically positive TB patients | ||
| (S5) Patients treatment | (T22) The proportion of confirmed TB patients who are enrolled in anti-TB treatment | |
| (T23) Timeliness of treatment for confirmed TB patients | ||
| (T24) The proportion of TB patients who undergo reexamination among those who have completed two months of treatment | ||
| (T25) The proportion of TB patients who undergo reexamination within one month prior to the end of treatment course | ||
| (T26) The proportion of TB patients who undergo examination at the end of treatment course | ||
| (S6) Patients management | (T27) The proportion of TB patients under management in primary health care institutions | |
| (T28) The proportion of TB patient under standard management | ||
| (T29) The proportion of TB patients managed by intelligent tools | ||
| (S7) Patients care | (T30) The proportion of TB patients who receiving nutritional/transportation support | |
| (T31) The proportion of TB patients who received at least one psychological support | ||
| (F3) Service outcomes and effects | (S8) Service outcomes | (T32) The proportion of patients contacted TB-designated hospital for TB appraisal to the serviced population |
| (T33) The proportion of TB patients with positive bacteriological result | ||
| (T34) The proportion of TB patients who discontinued treatment | ||
| (T35) The proportion of TB patients taking drugs regularly during treatment | ||
| (S9) Service effects | (T36) Reported incidence of TB | |
| (T37) The proportion of TB patients with successful treatment | ||
| (T38) Mortality of TB patients | ||
| (T39) Satisfaction of TB patients |
Abbreviations: TB, tuberculosis; PHEs, public health emergencies.
Weight Determination of Indicators
The importance scores for each indicator were calculated based on the results of the second round of the Delphi survey. Using the Saaty 1–9 scale method and AHP, the weights of all indicators included in the final RAIS were determined. Notably, all indicators had weights greater than 0.5% except T2, T4, T5, and T6. (Figure 2). During the calculation process, all the judgment matrices passed the consistency test (CRs<0.1).
Figure 2.
Diagram of weight distribution indicators in the resilience assessment index system for tuberculosis healthcare services. The four indicators with weights below 0.5%, namely T2 (0.46%), T4 (0.29%), T5 (0.20%), and T6 (0.34%), were not displayed.
Discussion
Accelerated climate change, expanding human activities, and cross-species pathogen transmission fuel the persistent and intensifying threat of emerging/re-emerging infectious diseases.32 The emergence of “X diseases” such as COVID-19 is inevitable in the future.33 In China, a considerable proportion of TB institutions also treat emerging infectious diseases, which will strain TB healthcare resources during outbreaks.19 Thus, assessing and bolstering TB-HS resilience is imperative.
The current study, rooted in the TB prevention and control landscape of China, developed a comprehensive index system to assess the resilience of TB-HSs. This index system comprises three primary, nine secondary, and 39 tertiary indicators, which are rigorously constructed to provide practical insights into enhancing the resilience of TB-HSs. RAIS is based on robust methodologies. The Delphi method facilitates consensus across domains,34 while the AHP provides a logical and reliable means of quantifying indicator weights through mathematical processing.35 The indicators are structured around the classic SPO framework, which logically posits that “Service guarantee” underpins “Service provision and utilization”, which subsequently leads to “Service outcomes and effects”.36 Our panel of experts represented all seven major regions of China, ensuring broad geographical representation. With over five years of experience in TB-related work and recent experience during the COVID-19 pandemic, they possessed a deeper understanding of TB-HSs’ resilience. Finally, we employed an innovative approach to measure the indicators’ importance by considering four dimensions: necessity, feasibility, stability, and sensitivity. Expert judgment determined the weights assigned to these dimensions, ensuring a scientific basis for assessing the indicator weights.
In the final RAIS, F2, “Services provision and utilization” had the highest weight (49.05%) among the three primary indicators. This differs from some Chinese medical quality assessment frameworks based on the Donabedian model,37,38 potentially because of the clearer definition and easier accessibility of “Structure” and “Outcome” indicators in such systems. Another possible explanation is the emphasis on maintaining service processes during PHEs, a key attribute of HSR: maintaining service continuity during crises.2 Regarding structural aspects, while some developed nations have ample, high-quality healthcare facilities under normal circumstances, they demonstrated low resilience during the COVID-19 pandemic,12 highlighting the importance of rapid response and efficient organization and coordination during PHEs.
Among the nine secondary indicators, S8 accounted for 20.79%, highlighting the significance of service outcomes in assessing the resilience of TB-HSs. These outcomes reflect short-term service provision and utilization and directly gauge resilience levels. This underscores the need to focus on both TB-HSs’ abilities and processes as well as results-oriented resilience building. Additionally, S5 and S4 shared 20.67% and 14.68% of the weight, respectively, emphasizing the maintenance of patient diagnosis and treatment services as core tasks and resilience manifestations during PHEs.
Among the 39 tertiary indicators, T34, the proportion of patients with TB who discontinued treatment stood alone with a weight exceeding 10%. Typically, such treatment interruptions are linked to various factors.39 However, during PHEs, they are closely tied to the operational status of TB-designated facilities and anti-TB drug supply. Similarly, T22 weighed 8.90%, highlighting the importance of ensuring that diagnosed patients receive treatment during crises. Access to treatment hinges on the availability of services and drugs. During the COVID-19 pandemic, the facilities and human resources responsible for TB-HSs were redirected, potentially leaving patients diagnosed at non-designated facilities without treatment. These two indicators also correlate with upstream preparedness, downstream treatment outcomes, and patient satisfaction and can sensitively reflect the resilience of TB-HSs. For future crises, strengthening patient treatment safeguards is crucial to enhance TB-HSs’ resilience.
Unlike other evaluation index systems used under normal conditions,40 the RAIS comprehensively captures the dynamic capabilities of the resilient TB-HSs. For instance, indicator T13, which quantifies the ratio of TB doctors on duty during and before PHEs, exemplifies the dynamic resilience of TB-HSs by illustrating how TB doctors are deployed during such crises. A higher proportion of TB doctors on duty during emergency situation suggests that the region can effectively mobilize and maintain TB diagnosis and treatment services. The RAIS, which evaluates specific time points within the same region, facilitates an understanding of the evolution of TB-HSs over time. Furthermore, the RAIS encompasses various domains, allowing for the identification of vulnerabilities when applied regionally and enabling targeted enhancements and improvements.
In contrast to the traditional Delphi method, which relies on open-ended questionnaires to gather research questions in the first round,41 the initial RAIS for Round 1 was designed based on a literature review and expert meetings. This alternative approach was necessitated by the nascent application of the resilience theory to healthcare services, particularly in the realm of TB prevention and control. Given that experts have limited familiarity with the resilience theory, it would have been difficult to achieve consensus among freely generated opinions. Furthermore, the experts involved in the research team and panel discussions had extensive experience in TB control and frontline management during the COVID-19 pandemic, ensuring that the initial RAIS was precisely targeted and problem-oriented.
However, this study has several limitations. First, the predetermined approach, deviating from the classic Delphi method, may introduce bias in the selection of indicators. Second, as all Delphi experts were TB healthcare service providers, the perspectives of TB service utilizers, particularly regarding the resilience of TB healthcare services during the COVID-19 pandemic, were not captured, which could affect the generalizability of the findings. Finally, the application of resilience theory in healthcare services, especially in TB-HSs, is still in its early stages, with limited research literature available for reference. Despite these limitations, this study provides a foundational framework for future research to validate and refine the index system.
Conclusion
A novel hierarchical indexing system was established to assess the resilience of TB-HSs in China during PHEs, using a modified Delphi process with AHP. We expect this tool to aid in the construction of resilient TB-HSs. However, further refinement and enhancement are required for practical applications.
Acknowledgments
The authors gratefully acknowledge to the 26 experts who provided their expertise and experience in the Delphi process. We would like to thank Editage (www.editage.cn) for English language editing.
Funding Statement
This study was supported by The National Thirteenth Five-year Mega-Scientific Projects of infectious diseases of China (grant No. 2017ZX10201302) and the Research Project of Chinese Center for Disease Control and Prevention (grant No. JY22-3-13).
Abbreviations
AHP, analytic hierarchy process; HSR, health system resilience; PHEs, public health emergencies; RAIS, resilience assessment index system; SPO, structure-process-outcome; TB, tuberculosis; TB-HSs, tuberculosis healthcare services; WHO, World Health Organization.
Ethics Approval and Informed Consent
This study was approved by the Ethics Committee Review Board of the Chinese Center for Disease Control and Prevention (No. 202230). Informed consent was obtained from all participants included in this study.
Disclosure
The authors report no conflicts of interest in this work.
References
- 1.Biddle L, Wahedi K, Bozorgmehr K. Health system resilience: a literature review of empirical research. Health Policy Plan. 2020;35(8):1084–1109. doi: 10.1093/heapol/czaa032 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kruk ME, Myers M, Varpilah ST, Dahn BT. What is a resilient health system? Lessons from Ebola. Lancet. 2015;385(9980):1910–1912. doi: 10.1016/S0140-6736(15)60755-3 [DOI] [PubMed] [Google Scholar]
- 3.Msemburi W, Karlinsky A, Knutson V, Aleshin-Guendel S, Chatterji S, Wakefield J. The WHO estimates of excess mortality associated with the COVID-19 pandemic. Nature. 2023;613(7942):130–137. doi: 10.1038/s41586-022-05522-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ellis LA, Saba M, Long JC, et al. The rise of resilient healthcare research during COVID-19: scoping review of empirical research. BMC Health Serv Res. 2023;23(1):833. doi: 10.1186/s12913-023-09839-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Saulnier DD, Duchenko A, Ottilie-Kovelman S, Tediosi F, Blanchet K. Re-evaluating our knowledge of health system resilience during COVID-19: lessons from the first two years of the pandemic. Int J Health Policy Manag. 2023;12:6659. doi: 10.34172/ijhpm.2022.6659 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.World Health Organization. Building health systems resilience for universal health coverage and health security during the COVID-19 pandemic and beyond: a brief on the WHO position. 2021. Available from: https://www.who.int/publications/i/item/WHO-UHL-PHC-SP-2021.02.2021. Accessed 20, April, 2024.
- 7.General Office of the CPC Central Committee, General Office of the State Council. Opinions on further improving the medical and health service system. 2023. Available from: https://www.gov.cn/gongbao/content/2023/content_5750620.htm. Accessed 15, April, 2024.
- 8.Fleming P, O’Donoghue C, Almirall-Sanchez A, et al. Metrics and indicators used to assess health system resilience in response to shocks to health systems in high income countries-A systematic review. Health Policy. 2022;126(12):1195–1205. doi: 10.1016/j.healthpol.2022.10.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Odhiambo J, Jeffery C, Lako R, Devkota B, Valadez JJ. Measuring health system resilience in a highly fragile nation during protracted conflict: South Sudan 2011-15. Health Policy Plan. 2020;35(3):313–322. doi: 10.1093/heapol/czz160 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Turenne CP, Gautier L, Degroote S, Guillard E, Chabrol F, Ridde V. Conceptual analysis of health systems resilience: a scoping review. Soc Sci Med. 2019;232:168–180. doi: 10.1016/j.socscimed.2019.04.020 [DOI] [PubMed] [Google Scholar]
- 11.Bhandari S, Alonge O. Measuring the resilience of health systems in low- and middle-income countries: a focus on community resilience. Health Res Policy Syst. 2020;18(1):81. doi: 10.1186/s12961-020-00594-w [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Sagan A, Thomas S, Webb E, McKee M. Assessing resilience of a health system is difficult but necessary to prepare for the next crisis. BMJ. 2023;382:e073721. doi: 10.1136/bmj-2022-073721 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Topp SM. Power and politics: the case for linking resilience to health system governance. BMJ Glob Health. 2020;5(6):e002891. doi: 10.1136/bmjgh-2020-002891 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Haldane V, Ong SE, Chuah FL, Legido-Quigley H. Health systems resilience: meaningful construct or catchphrase? Lancet. 2017;389(10078):1513. doi: 10.1016/S0140-6736(17)30946-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Poroes C, Seematter-Bagnoud L, Wyss K, Peytremann-Bridevaux I. Health system performance and resilience in times of crisis: an adapted conceptual framework. Int J Environ Res Public Health. 2023;20(17). doi: 10.3390/ijerph20176666 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.World Health Organization. Health system resilience indicators: an intergrated package for measuring and monitoring health system resilience in countries. 2024. Available from: https://www.who.int/publications/i/item/9789240088986. Accessed 31, March, 2024.
- 17.World Health Organization. Global tuberculosis report 2024. 2024. Available from: https://iris.who.int/bitstream/handle/10665/379339/9789240101531-eng.pdf?sequence=1. Accessed 31, October, 2024.
- 18.Ansumana R, Keitell S, Roberts GM, et al. Impact of infectious disease epidemics on tuberculosis diagnostic, management, and prevention services: experiences and lessons from the 2014-2015 Ebola virus disease outbreak in West Africa. Int J Infect Dis. 2017;56:101–104. doi: 10.1016/j.ijid.2016.10.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Fei H, Yinyin X, Hui C, et al. The impact of the COVID-19 epidemic on tuberculosis control in China. Lancet Reg Health West Pac. 2020;3:100032. doi: 10.1016/j.lanwpc.2020.100032 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Dheda K, Perumal T, Moultrie H, et al. The intersecting pandemics of tuberculosis and COVID-19: population-level and patient-level impact, clinical presentation, and corrective interventions. Lancet Respir Med. 2022;10(6):603–622. doi: 10.1016/S2213-2600(22)00092-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Migliori GB, Thong PM, Akkerman O, et al. Worldwide effects of coronavirus disease pandemic on tuberculosis services, January-April 2020. Emerg Infect Dis. 2020;26(11):2709–2712. doi: 10.3201/eid2611.203163 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Li K, Rui J, Song W, et al. Temporal shifts in 24 notifiable infectious diseases in China before and during the COVID-19 pandemic. Nat Commun. 2024;15(1):3891. doi: 10.1038/s41467-024-48201-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Donabedian A. The quality of care. How can it be assessed? JAMA. 1988;260(12):1743–1748. doi: 10.1001/jama.260.12.1743 [DOI] [PubMed] [Google Scholar]
- 24.Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on conducting and reporting delphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31(8):684–706. doi: 10.1177/0269216317690685 [DOI] [PubMed] [Google Scholar]
- 25.Du W, Fu J, Su C, Zhang Q, Zhai F, Zhang B. Application of Delphi method on screening indexes of nutrition education comprehensive evaluation questionnaire. Chin J Health Educ. 2010;26(4):250–253. [Google Scholar]
- 26.Fitch K, Bernstein SJ, Aguilar MD, et al. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica, CA: RAND Corporation; 2001. [Google Scholar]
- 27.Liu XW, Lu XL, Zhong XY, et al. Development of a breast cancer screening capacity assessment system for primary care institutions. Chin General Pract. 2024;27(19):2415–2420. doi: 10.12114/j.issn.1007-9572.2023.0057 [DOI] [Google Scholar]
- 28.Saaty T. Highlights and critical points in the theory and application of the analytic hierarchy process. Eur J Op Res. 1994;74(3):426–447. doi: 10.1016/0377-2217(94)90222-4 [DOI] [Google Scholar]
- 29.Zhu AA, Cao CJ, Zhang L, Li JT, Fu H, Zhu KQB. Application of Delphi method and analytic hierarchy process to construct the evaluation index system of health enterprises. Chin J Hyg Occup Dis. 2024;42(2):112–117. doi: 10.3760/cma.j.cn121094-20221201-00571 [DOI] [PubMed] [Google Scholar]
- 30.Guang Z, Hui L. Method and Applications for Modern Epidemiology. Beijing: United Peking Medical University and Peking Union Medical College Press; 1996. [Google Scholar]
- 31.Graham B, Regehr G, Wright JG. Delphi as a method to establish consensus for diagnostic criteria. J Clin Epidemiol. 2003;56(12):1150–1156. doi: 10.1016/s0895-4356(03)00211-7 [DOI] [PubMed] [Google Scholar]
- 32.Baker RE, Mahmud AS, Miller IF, et al. Infectious disease in an era of global change. Nat Rev Microbiol. 2022;20(4):193–205. doi: 10.1038/s41579-021-00639-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Iserson KV. The next pandemic: prepare for “Disease X”. West J Emerg Med. 2020;21(4):756–758. doi: 10.5811/westjem.2020.5.48215 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Broder MS, Gibbs SN, Yermilov I. An adaptation of the RAND/UCLA modified delphi panel method in the time of COVID-19. J Healthc Leadersh. 2022;14:63–70. doi: 10.2147/JHL.S352500 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Schmidt K, Aumann I, Hollander I, Damm K, von der Schulenburg JMG. Applying the analytic hierarchy process in healthcare research: a systematic literature review and evaluation of reporting. BMC Med Inform Decis Mak. 2015;15(1). doi: 10.1186/s12911-015-0234-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Hammermeister KE, Shroyer AL, Sethi GK, Grover FL. Why it is important to demonstrate linkages between outcomes of care and processes and structures of care. Med Care. 1995;33(10 Suppl):OS5–OS16. doi: 10.1097/00005650-199510001-00002 [DOI] [PubMed] [Google Scholar]
- 37.Cao J, Wen M, Peng Y, Lin Y, Bao M, Zeng H. Construction of evaluation index system of nursing quality in pulmonary tuberculosis specialty. Shenzhen J Integr Traditional Chin West Med. 2023;43:14–20. doi: 10.16458/j.cnki.1007-0893.2023.04.004 [DOI] [Google Scholar]
- 38.Liu J, Xu JM, Wang XY, Yu XY, Huang XG. Construction of a comprehensive medical service capability evaluation indicator system for grade III traditional Chinese medicine hospitals. Chin Hospitals. 2024;28:48–52. doi: 10.19660/j.issn.1671-0592.2024.6.12 [DOI] [Google Scholar]
- 39.Zegeye A, Dessie G, Wagnew F, et al. Prevalence and determinants of anti-tuberculosis treatment non-adherence in Ethiopia: a systematic review and meta-analysis. PLoS One. 2019;14(1):e0210422. doi: 10.1371/journal.pone.0210422 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Chen H, Zhang Y, Wang L. A study on the quality evaluation index system of smart home care for older adults in the community --based on Delphi and AHP. BMC Public Health. 2023;23(1):411. doi: 10.1186/s12889-023-15262-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Gibson JM. Using the Delphi technique to identify the content and context of nurses’ continuing professional development needs. J Clin Nurs. 1998;7(5):451–459. doi: 10.1046/j.1365-2702.1998.00175.x [DOI] [PubMed] [Google Scholar]