Indicators to assess the functionality of clubfoot clinics in low-resource settings: a Delphi consensus approach and pilot study

Abstract

Background

This study aims to determine the indicators for assessing the functionality of clubfoot clinics in a low-resource setting.

Methods

The Delphi method was employed with experienced clubfoot practitioners in Africa to rate the importance of indicators of a good clubfoot clinic. The consistency among the participants was determined with the intraclass correlation coefficient. Indicators that achieved strong agreement (mean≥9 [SD <1.5]) were included in the final consensus definition. Based on the final consensus definition, a set of questions was developed to form the Functionality Assessment Clubfoot Clinic Tool (FACT). The FACT was used between February and July 2017 to assess the functionality of clinics in the Zimbabwe clubfoot programme.

Results

A set of 10 indicators that includes components of five of the six building blocks of a health system—leadership, human resources, essential medical equipment, health information systems and service delivery—was produced. The most common needs identified in Zimbabwe clubfoot clinics were a standard treatment protocol, a process for surgical referrals and a process to monitor dropout of patients.

Conclusions

Practitioners had good consistency in rating indicators. The consensus definition includes components of the World Health Organization building blocks of health systems. Useful information was obtained on how to improve the services in the Zimbabwe clubfoot programme.

Introduction

Measurement of the functionality (capability and suitability) of health facilities is central to the concept of quality improvement. It allows identification of opportunities to strengthen and improve health services and quality of care. Measurements of quality are difficult to compare and interpret¹ due to the complex nature of health facilities. Internationally comparable measures are scarce and there is a lack of data on facility quality assessments in low-resource settings.²

Clubfoot, or congenital talipes equinovarus (CTEV), is a rigid deformity of the foot and one of the most common congenital musculoskeletal malformations that affects mobility.³ Incidence is estimated as 1.2 per 1000 live births.⁴ Clubfoot correction of a high quality continues to be a key requirement for reducing disability and improving function related to the congenital foot deformity. The minimally invasive Ponseti technique⁵ is now the most common method of treatment, and involves manipulation and casting and a regime of bracing. In high-income settings, this technique radically reduces the need for extensive corrective surgery and the corrective phase is reported to have a success rate of 98%.⁶ Although the definition and measurement of success of treatment are being explored in resource-constrained settings,^7–9 the provision of appropriate care is difficult to define, measure and evaluate. It is increasingly evident that patient outcomes are not solely a function of efficacious clinical interventions and practices but are also affected by the quality of non-clinical services. Evidence and consensus on what clinic qualities to measure are lacking.

A structured consensus technique that may be used to reach agreement about health care quality indicators¹⁰ is the Delphi method. The anonymous opinions of participants are sought through a sequential process¹¹ and this allows equal weight to be given to all participants.^12,13 The collated group responses are fed back to participants after the completion of each round of questionnaires. Establishing consensus does not ensure validity; however, agreement provides a basis for establishing criteria that are likely to have clinical sensibility.¹⁴

Strategies to improve services for children with clubfoot require consideration of the clubfoot clinic within the health system and the paradigm shift from condition-specific interventions to interventions that strengthen health systems. The interdependence between caregiver demand for clubfoot services, the identification and referral of clubfoot within the context of primary health care, the service provision of the clubfoot clinic and the enabling environment of the broader health facility is required for good clubfoot treatment. Within this health system–oriented approach, appropriate measures are needed to assess the functionality of clubfoot clinics.¹⁵ There are few formal monitoring systems in place to inform this planning.¹⁶ This study aims to determine the indicators for assessing the functionality of clubfoot clinics in low-resource settings by establishing a consensus among expert Ponseti trainers in the Africa region and then to use those indicators to assess the Zimbabwe clubfoot programme.

Materials and methods

Study design

This study was designed and reported according to the recommended guidelines for selection of health care quality indicators.¹⁰ A Delphi method was employed with experienced clubfoot practitioners (Ponseti technique) throughout Africa to rate the importance of indicators of a ‘good Ponseti clubfoot clinic’. Based on the final consensus definition, a set of questions was developed to form the Functionality Assessment Clubfoot Clinic Tool (FACT). The FACT was used between February and July 2017 to assess the functionality of 12 clinics of the Zimbabwe national clubfoot programme.

Delphi method to develop a consensus definition

In the first Delphi exercise conducted in January 2016, 18 experienced Ponseti method clubfoot practitioners who are trainers in 10 national clubfoot programmes in Africa attended a workshop. They included orthopaedic surgeons, physiotherapists and orthopaedic technicians. The participants were chosen based on their knowledge and experience of the topic and willingness to participate.¹⁷ Facilitated discussion in the workshop identified criteria (indicators) likely to demonstrate a well-functioning Ponseti clubfoot clinic. A questionnaire was then developed consisting of 28 potentially relevant indicators. It was pilot tested for understanding. The 18 trainers were invited to participate and rate each of the 28 indicators for their relative importance. A 10 cm visual analogue scale (VAS) with the anchors ‘completely unimportant’ and ‘extremely important’ was used. The paper questionnaires were completed by hand. The second round occurred 2 days later. The trainers were given the mean score and standard deviation (SD) for each indicator from the first round and asked to repeat the rating for the second round. No indicator was excluded and no discussion was allowed among the participants.

The consistency among the 18 trainers was determined with the intraclass correlation coefficient (ICC). The ICC is interpreted as follows: ≤0.40, poor consistency or large variation in opinion; 0.41–0.74, acceptable consistency; and ≥0.75, good consistency.¹⁸

In July 2016, 6 months after the first workshop, a second workshop was held with 17 different trainers of the Ponseti technique (from 10 countries). The experts were not considered to differ from the first group and their attendance in July was based on convenience of the workshop schedule. Indicators generated in January 2016 that had a mean VAS >7 (on the 10-point scale) with an SD <2 were included (21 indicators) in the questionnaire of the second Delphi exercise.

The two-round Delphi exercise, with an identical process to that undertaken in January 2016, was used to reach a consensus on indicators to assess the functionality of clubfoot clinics in low-resource settings and therefore define a well-functioning Ponseti clubfoot clinic. Of the 17 indicators rated, those that scored ≥9 with an SD <1.5 were considered to have high agreement and form the consensus definition. As there is variability in the measurement of distribution of scores in studies that use the Delphi method,¹⁹ the thresholds for the VAS mean and SD were decided a priori and the cut-offs were selected to include indicators with high agreement.

The study methodology and course of action for the management of responses are outlined in Figure 1. The questionnaires answered in the second rounds of Delphi exercises 1 and 2 are in Supplementary files Appendix 1 and 2, respectively.

Figure 1.

Flow chart of criterion selection.

Assessment of clubfoot clinics in Zimbabwe using the consensus definition

A questionnaire based on the 10 indicators that met the criteria for the consensus definition was developed and piloted. The questionnaire was reviewed by the Zimbabwe national clubfoot clinical supervisor and the researchers. Each question has four response categories, scored from 0 to 3, where 3 is the ideal result. The maximum clinic score is 30. A response matrix was developed to analyse the results (Table 1).

Table 1.

The FACT

Domain	Criteria	Rating				Score
		0	1	2	3
Leadership	There is an identifiable person in charge of the clinic	In the past 8 weeks:
		There was no identifiable person in charge of the clinic	There was an identifiable person in charge of five or fewer clinics	There was an identifiable person in charge of six to seven clinics	There was an identifiable person in charge of every clinic
Human resources	There are regularly two or more Ponseti-trained health care workers (HCWs) available at each clinic	In the past 8 weeks:
		There were fewer than two trained HCWs available at every clinic	There were two or more trained HCWs available in five or fewer clinics	There were two of more trained HCWs available in six to seven clinics	There were two or more trained HCWs available at every clinic
Essential medical equipment	There is always plaster of Paris available	In the past 8 weeks:
		Plaster of Paris was not available for three or more clinics	Plaster of Paris was not available for two clinics	Plaster of Paris was not available for one clinic	There was always plaster of Paris available
	There is always equipment (such as bucket and scissors/blade, tenotomy set, correct size braces) available	In the past 8 weeks:
		A bucket, scissors/blade, tenotomy set or correct size braces were not available for three or more clinics	A bucket, scissors/blade, tenotomy set or correct size braces were not available for two clinics	A bucket, scissors/blade, tenotomy set or correct size braces were not available for one clinic	There was always a bucket and scissors/blade, tenotomy set, correct size braces available
Service delivery	There is a standard treatment protocol	There is no standard protocol for treatment of idiopathic clubfoot in children <2 y of age, the older child with clubfoot or non-idiopathic clubfoot	There is a standard verbal treatment protocol but nothing in writing for treatment of idiopathic clubfoot in children <2 y of age, the older child with clubfoot and non-idiopathic clubfoot	There is a written protocol for treatment of idiopathic clubfoot in children <2 y of age, the older child with clubfoot and non-idiopathic clubfoot, but it is not consistently used	There is a standard written protocol for treatment of idiopathic clubfoot in children <2 y of age, the older child with clubfoot and non-idiopathic clubfoot and it is followed consistently
	There is a functioning referral system for tenotomy	In the two most recent tenotomy cases:
		There was no clinician to whom children requiring a tenotomy could be referred	The children were referred for tenotomy and there is no record of the outcome	The children were referred but the tenotomy was not completed as anticipated	The children were referred for tenotomy and it was completed as anticipated
	There is a process for surgical referrals	In the two most recent cases who required surgery:
		There was no clinician to whom children requiring a surgical review could be referred	The children were referred and there is no record of the outcome	The children were referred and there is a record of the outcome without a plan for follow-up	The children were referred for surgery and there is a record of the outcome with a plan for follow-up
	There is a process to monitor dropout of patients	There is no process to monitor dropout of patients	There is a verbal process to monitor dropout of patients but nothing in writing	There is a written protocol to monitor dropout of patients but it is not consistently followed	There is a written protocol to monitor dropout of patients and it is consistently followed
	The clinic occurs on a specific day, at minimum weekly	In the past 8 weeks:
		The clinic did not have an identified day for clubfoot treatment	The clinic has been held on the identified day(s) for less than six of the weeks	The clinic has been held on the identified day(s) for six to seven of the weeks	The clinic has been held on the identified day(s) every week
Health information system	There is a completed clinic record for each patient visit	In the last 10 clinic records:
		≤5 of the clinic records were filled in completely	6–7 of the records were filled in completely	8–9 of the clinic records were filled in completely	All of the clinic records were filled in completely
Total score

Data collection

Data were collected between February and July 2017 during routine clubfoot clinic mentoring visits by the clinical supervisor of the Zimbabwe Sustainable Clubfoot Programme. The questionnaire was paper based and completed by hand. Routine discussion with clinic staff, observation and review of equipment and clinic records provided data for completion of the questionnaire. The clubfoot clinic staff were informed of the outcome and subsequent recommendations discussed with the team. Data were collected from all 12 national clubfoot clinics.

Data management and analysis

All data were entered into an Excel 2000 (Microsoft, Redmond, WA, USA) spreadsheet. All data were managed and analysed using Stata 14.2 (StataCorp, College Station, TX, USA). A descriptive analysis compared the characteristics of the clinics. Proportions were calculated and a comparative analysis of criteria was explored.

Ethics

Ethical approval was granted by the Medical Research Council of Zimbabwe and the London School of Hygiene & Tropical Medicine.

Results

Delphi method to develop a consensus definition

In Delphi exercise 1 (January 2016) the response rate of trainers to the questionnaires was 94.4% (17/18) and in exercise 2 (July 2016) the rate was 94.1% (16/17).

The consistency of Ponseti method trainers in Africa in rating indicators to assess the functionality of Ponseti clubfoot clinics was good. The first Delphi ICC had an external consistency of 0.96 (95% [CI 0.94–0.98]) and the second Delphi ICC had an external consistency of 0.84 (95% CI 0.70–0.93).

From the initial 28 indicators, 17 met the inclusion criteria for Delphi exercise 2. The results of the rating of each indicator by trainers in Africa are shown in Table 2 for Delphi exercise 1 (January 2016) and Table 3 for Delphi exercise 2 (July 2016).

Table 2.

Indicators for a functioning clubfoot clinic ranked by mean score and SD from the second round: Delphi exercise 1, January 2016

	Indicator	Mean	SD
1	100% foot abduction brace uptake post-tenotomy	9.68	0.42
2	Functioning referral for tenotomy	9.49	0.67
3	Completed record for each visit	9.37	0.49
4	Someone in charge (for continuity)	9.32	0.63
5	Two or more trained therapists regularly at the clinic	9.28	0.76
6	Standardized treatment protocol	9.20	0.88
7	Plaster of Paris available	9.19	0.85
8	Clinic occurs on a specific day	9.13	1.12
9	Equipment is available	8.94	1.17
10	Counsellor contact with 90% of families at each clinic	8.89	0.78
11	There is a process for surgical referrals	8.61	1.38
12	Clinicians refer if there is no change in Pirani score for four visits	8.61	1.59
13	The nearest maternity ward refers all cases	8.51	1.96
14	Clinicians refer if >10 casts	8.41	1.24
15	The family is followed up if an appointment is missed	8.32	1.96
16	There is a clubfoot champion in charge of the clinic	7.96	2.64
17	90% of patients wear a foot abduction brace at 4 months	7.92	2.10
18	Children are <6 months of age at the first treatment	7.52	1.69
19	Braces are received within 2 weeks of ordering	7.52	2.53
20	Tenotomy coverage >90%	7.42	2.35
21	Relapse rate <10%	7.25	2.00
22	Tenotomy coverage >70%	7.10	2.00
23	Relapse rate <20%	5.99	2.05
24	Only 10% drop out after 6 months of bracing	5.95	1.82
25	Relapse rate not >30%	5.88	1.98
26	Only 25% drop out after 6 months	4.25	1.86
27	<70% tenotomy coverage	3.95	2.05
28	50% drop out after 6 months of bracing	2.83	2.19

Table 3.

Indicators for a functioning clubfoot clinic ranked by mean score and SD from the second round: Delphi exercise 2, July 2016

	Indicators	Mean	SD
1	Plaster of Paris available	9.69	0.55
2	Equipment available	9.66	0.70
3	Completed record for each visit	9.65	0.58
4	Functional referral for tenotomy	9.59	0.81
5	Clinic occurs on a specific day	9.47	0.72
6	Standardized treatment protocol	9.46	0.53
7	Someone in charge (for continuity)	9.39	0.88
8	There is a process for surgical referrals	9.35	1.04
9	There is a process to monitor dropouts	9.09	0.78
10	Two or more trained therapists regularly at the clinic	9.00	1.33
11	The nearest maternity ward refers all cases	8.99	0.84
12	Clinicians refer if no change in Pirani score for four visits	8.96	1.04
13	Tenotomy coverage >70%	8.76	0.89
14	The family is followed up if an appointment is missed	8.71	1.06
15	Counsellor contact with 90% of families at each clinic	8.68	1.18
16	Clinicians refer if >10 casts	8.53	2.31
17	Children are <6 months of age at the first treatment	7.30	2.19

Ten indicators in the second Delphi exercise met the a priori definition (mean score ≥9 [SD <1.5]) for inclusion in the consensus definition: plaster of Paris available, equipment is available, completed record for each visit, functional referral for tenotomy, clinic occurs on a specific day, standard treatment protocol available, someone in charge (for continuity), there is a process for surgical referrals, there is a process to monitor dropouts and two or more trained therapists are regularly at the clinic.

Assessment of clubfoot clinics in Zimbabwe using the consensus definition

The average clubfoot clinic score was 21 (range 16–26) (Table 4). All clubfoot clinics had an identifiable person in charge, two or more Ponseti clubfoot therapists that were regularly available and the clinic occurred on a specific day, at a minimum weekly (Figure 2). Indicators of a standard treatment protocol, a process for surgical referral and a process to monitor dropout of patients were overall the lowest scoring indicators of the clinics.

Table 4.

Individual clinic scores

Health system building block	Leadership	Human resources	Medical equipment		Service delivery					Health information system
Indicator	1	2	3	4	5	6	7	8	9	10	Total
Clinic ID	Identifiable person in charge	Two trained therapists	Plaster of Paris	Bucket/scissors/blade/tenotomy set/braces	Standard treatment protocol	Tenotomy referral	Process for surgical referrals	Process to monitor dropouts	Clinic occurs on a specific day	Completed clinic record	Clinic score out of 30
1	3	3	3	2	0	2	1	1	3	3	21
2	3	3	3	2	0	3	0	1	3	3	21
3	3	3	1	3	1	2	0	1	3	2	19
4	3	3	1	3	1	3	2	1	3	3	23
5	3	3	1	3	1	0	0	1	3	1	16
6	3	3	3	3	1	3	1	1	3	3	24
7	3	3	3	3	1	3	0	1	3	3	23
8	3	3	1	3	1	2	0	1	3	0	17
9	3	3	3	3	1	3	0	1	3	3	23
10	3	3	3	3	1	3	0	1	3	2	22
11	3	3	3	3	1	3	3	1	3	3	26
12	3	3	3	3	1	3	0	1	3	1	21
Mean score	3.00	3.00	2.33	2.83	0.83	2.50	0.58	1.00	3.00	2.25

Figure 2.

Mean scores for the 12 clubfoot clinics.

Legend:

1. There is an identifiable person in charge of the clinic.
2. There are regularly two or more Ponseti-trained health care workers available at each clinic.
3. There is always plaster of Paris available.
4. There is always equipment (e.g. bucket and scissors/blade, tenotomy set, correct size braces) available.
5. There is a standard treatment protocol.
6. There is a functioning referral system for tenotomy.
7. There is a process for surgical referrals.
8. There is a process to monitor dropout of patients.
9. The clinic occurs on a specific day, at minimum weekly.
10. There is a completed clinic record for each patient visit.

Regarding the health system building blocks, the indicators for leadership and human resources achieved full scores, while service delivery had the lowest score in all clinics (Table 4).

Discussion

This study determined the opinions of experts from 11 countries in Africa about the indicators for a successful Ponseti clubfoot clinic within health facilities. The aim of the Delphi method was to define criteria to evaluate the functionality of clubfoot clinics in low-resource settings, therefore regional trainers of the Ponseti method were deemed the most appropriate experts to participate in this context. A questionnaire was developed, based on the consensus indicators, to evaluate the functionality of the Zimbabwe clubfoot clinics.

Delphi exercise

The trainers had good consistency in rating indicators to assess the functionality of a clubfoot clinic. The consensus indicators include components of five of the six World Health Organization (WHO) building blocks of health system strengthening,²⁰ namely leadership, human resources, essential medical equipment, health information systems and service delivery.

Assessment of 12 clinics

The FACT took 15 min to complete and was undertaken as part of routine supervision visits. Of the 10 indicators piloted, leadership and human resources were found to score the highest in the Zimbabwe national clubfoot programme, with service delivery demonstrating the greatest need for improvement.

Comparison to other literature/previous studies

The indicators developed in this study reflect the WHO building blocks for health systems.²⁰ To our knowledge there are no quality indicators for clubfoot clinics published and evaluated in the literature; however, individual non-governmental organizations regularly use checklists and reporting templates for accountability and quality improvement purposes.

Strengths and limitations

This study has used many experts, in the context of Africa, to develop and rate criteria (indicators) of clubfoot clinic qualities that are viewed to be the most important. The response rate of the survey was high (94%). There are also study limitations. The panel in this study was selected for their expertise but may not be representative of all Ponseti treatment practitioners. Previous research has shown that panel composition influences ratings.²¹ In addition, indicators will never completely capture the richness and complexity of a health system. Their design must be understood in context and there may be some indicators that we have not considered that may be important. For example, there was no consensus on the number of patients and one finding has been that hospitals caring for greater numbers of patients with similar conditions tend to have better outcomes for surgical procedures.^22,23 Consequently, while the questionnaire can provide valuable information, other data may be relevant to have a more complete understanding of the clubfoot clinic.

Implications

The data from this study provide useful information to assist in monitoring and improving services for children with clubfoot in low-resource settings. For example, clubfoot service provision in Zimbabwe utilizes a task-shifting approach, where trained ‘clubfoot therapists’ undertake manipulation and casting and brace reviews, and medical officers complete tenotomies. Tenotomies are undertaken in the outpatient clinic under local anaesthetic, and the decision of when to verbally refer for tenotomy rests with the clubfoot therapist. Surgical procedures that extend beyond a tenotomy require written referral to a specialist. This study highlights the need for defined care pathways to monitor standard procedures and contribute to a robust referral system.

Indicators are summary measures and no matter how valid they are, they will rarely by themselves motivate people to change. Using the questionnaire, clubfoot programme staff determined if all necessary elements were in place for a successful clinic that can deliver clubfoot care. Based on the results of the assessment, the need for standard treatment protocols, a process for surgical referrals and a process to monitor dropouts were identified in every clinic in Zimbabwe. These processes were discussed and a plan to address them in every clinic was created. The usefulness of this tool in clubfoot clinics is therefore twofold: (1) to assess the design and implementation of effective clubfoot clinic programmes and (2) to monitor services and highlight the improvements needed as the programme develops.

Future research

It is important that these indicators are tested in other situations to see if they are valid. The definition of indicators in greater detail (e.g. the development of a specific equipment list that includes braces and essential equipment for tenotomy) will identify gaps in resources and allow for clubfoot clinics to be appropriately equipped. Also, research questions such as how provider performance can be improved, responsiveness to change and why some clinics perform better than others may provide further insights into quality improvement.

Approaches to encourage completion of both the treatment and bracing phases include designing education resources with parents, who have unique insights about their challenges and situations but are often excluded from the design process about issues that directly affect their lives. An understanding of how caregivers can best be supported, when and where counselling is needed and mechanisms for delivery (e.g. peer group, one to one) warrant further investigation.

A well-equipped and managed clubfoot clinic may still provide poor care. A child’s experience of care and parent-reported outcomes provide valuable insights on quality of care but are rarely measured. Agreement on how to measure these outcomes using metrics that are robust, comparable and financially efficient is required.

Conclusion

Appropriate measures are required to determine clubfoot clinic qualities and to compare different clinics. Using the Delphi method among experts from across Africa, we found a consensus for the most important indicators of a good clubfoot (Ponseti method) clinic. The consensus definition includes 10 indicators covering five of the six WHO building blocks of health system strengthening: leadership, human resources, essential medical equipment, service delivery and health information systems. The FACT indicators recommended are accompanied by descriptions of their technical properties, and methodological guidance is provided for their assessment.

In the Zimbabwe clubfoot clinics, the area of leadership scored the highest and the area of service provision had the greatest room for improvement. This article contributes to the data on clinic indicators globally. The data from this study provide useful information on planning services and may direct health care planning towards the areas of need.

Supplementary data

Supplementary data are available at International Health online (http://inthealth.oxfordjournals.org).