Evaluating a traditional medicine policy in South Africa: phase 1 development of a policy assessment tool

Abstract

Background

Policies that empower individuals and communities may be appropriate for public health, and more broadly. Simple, transparent and acceptable tools are therefore required to evaluate policies from an empowerment perspective. In 2008, the South African Department of Health (DOHSA) drafted a policy to endorse the integration of African Traditional Medicine (ATM) into the public health sector, following the World Health Organization's (WHO) long-standing directives.

Objective

The purpose of this study is to critically analyze this policy using a novel evaluation tool.

Design

A 12-point ‘Policy Empowerment Index’ (PEI) is introduced, and used to classify and score the policy according to five theoretical policy types. The evaluation was based on a stepwise review and associated publications: policy drafts, policy statements and news announcements.

Results

According to the assessment tool, the ATM policy was marginally ‘supportive’ of constituent empowerment, although several ‘directive’ features were also observed. The importance of ATM to SA's communities and the promotion of education, employment, entrepreneurship and peripheral resource mobilization were the main empowering elements. Centralised conception, planning and implementation, the absence of provisions for local adaptations and the authoritative legislation context were sub-optimal features.

Conclusions

South Africa's ATM legislation may need to further involve communities in policy design and implementation to capitalise upon the broader benefits of community empowerment. However, the iterative nature of method and evaluation is important. Indeed, they are proposed as points to initiate participatory development, and improve policy evaluation. Such instruments can empower constituents in the political process.

Empowerment is the process through which individuals, communities and organizations increase their self-determination, mainly by augmenting the ability of the powerless to make choices that improve their lives (1–3).

Although alleged economic empowerment and decentralization may reinforce inequalities, if the power balances are not controlled (4), and participation claims can be used to mislead, control and profit from the powerless (5), increasing evidence indicates that empowering policies can bring broad benefits (6–9). Community empowerment's importance for health is underscored by the World Health Organization's (WHO) persistent focus on community participation, capacity and rights (10, 11). Furthermore, empowerment, seen from a human rights’ perspective (3, 6, 12), increases self-determination, an essential component of freedom. Increased resources, agency and achievements are essential and interactive prerequisites for empowerment (3, 13) and programs that aim to enhance empowerment focus on improving individual awareness and capacity, community resources, social capital and networking and organizational orientation and governance (3, 8).

Policies can have major impacts on empowerment, especially those that relate to the financial and health sectors (8) and empowerment has become central in the political and academic debate, increasingly recognized across the political spectra (6, 14).

Although attempts have been made to evaluate empowerment as an end (15), and despite the fact that the policy factors that affect empowerment are becoming increasingly clear (8), the process of empowerment has been harder to assess (3). A synoptic evaluation of how policy papers apply empowering factors could be a useful tool in policy impact evidence collection.

For this study, a ‘Policy Empowerment Index’ (PEI) was developed, based on evidence on how policy factors affect population empowerment (4, 5, 7), systematically reviewed by Kar et al. (3), Pratchett et al. (8) and on WHO-Euro (11) and World Bank (16) recommendations. The index was subsequently applied to a South African (SA) policy draft on African Traditional Medicine (ATM). This policy was chosen because ATM is an important but controversial asset of African communities, which calls for a richer participation and discourse of related policies in order to improve their planning, implementation and impact.

The South African traditional medicine policy

Despite the controversy that surrounds it, traditional medicine (TM) carries recognized ancient wisdom. The WHO (10) officially declared its importance for global health during the Alma Ata declaration, and has since passed several resolutions and declarations advocating national policies and regulations promoting the role of TM in health systems (11, 17, 18). Several countries worldwide have integrated TM practices into their healthcare systems (such as China, India, UK, Ghana and Mali), while others are making efforts in that direction or tolerate TM to varying extents (19). The African Union, at the 2001 Assembly of Heads of State and Government, adopted an action plan for health system (NHS) integration of ATM by 2010 (20).

Eighty percent of SA's population is using ATM for preventive, curative and palliative, purposes, choosing indigenous medicine for its holistic approach – spiritual and physical – and better accessibility (21, 22) and due to shortfalls of modern healthcare provision (23). The South African government established a ‘Directorate of Traditional Medicine’ in 2006 within the Department of Health and a ‘Traditional Practitioners Counsel’ in 2007. In 2008, a ‘Presidential Task Team’ proposed a national ATM policy, published in the Government Gazette, and the public was invited to ‘submit any substantiated comments’ (19).

The purpose of this study was to design and apply a concise tool to critically analyse SA's ATM policy draft per empowerment.

Design

A method for evaluation of policy empowerment properties was developed and applied to SA's ATM policy draft.

Policy Empowerment Index (PEI)

In Pratchett et al.‘s systematic review (8), the policy and program factors that mainly influence empowerment were fit in the following categories: (1) asset transfer (such as community-run enterprises), (2) citizen governance (participation in decision and policy making), (3) electronic participation, (4) participatory budgeting, (5) petitions and (6) redress (a feedback mechanism between citizens and the polity). The effects of these factors from several policies were examined for their effects on empowerment of individuals and communities to influence policies that concern them. Each mechanism was found to have an impact, though varied, depending on elements such as policy design, implementation (flexibility, openness, supporting previsions and connectedness to formal political mechanisms) and context (8).

Kar et al. (3) reviewed policy and health promotion factors affecting women's empowerment, and found skill, leadership, knowledge and asset development, media support, social services (medical, legal, financial) and human right's promotion to be most paramount and commonly used.

Laverack and Labonte (24), proposed a planning framework for mainstreaming of empowerment in health promotion project planning cycles, based on community participation, control and capacity augmentation.

WHO-Euro's Health Evidence Network (11) evaluated evidence on the empowering characteristics and developmental and health outcomes of various strategies and interventions, concluding on the critical importance of citizen participation, facilitated, among other factors, by ‘governments that sponsor or mandate mass mobilizations’, with expert non-dominant support. The evidence supported the notion that participatory empowerment increases efficiency, sustainability, and equity and overall project impact, as well as social capital and overall community development. Favourable effects on child and adult, communicable and non-communicable disease outcomes were detected. In addition, support networks and education/knowledge interventions increased individual decision-making efficacy, efficient use of health services and social action/participation in policy changes. Entrepreneurship, through micro-financing and income generation, improved women's health, community and political participation even faster than education, but only if combined with increased, legislatively supported, autonomy and power. Integrated economic, educational and political strategies had the best outcomes (11).

The World Bank (16) emphasizes access to information and decision making, local organizational capacity and public accountability as essential elements for participation's role in empowerment.

In Africa, several attempts are underway to increase women's capacity through budget initiatives and gender mainstreaming in economic policies. Although impact evaluations are still unavailable, such empowering policies have been well received and spreading in other sectors such as health and education (25).

For this study, a 12-point questions’ ‘Policy Empowerment Index’ (PEI) was developed stepwise, questioning policy planning on the following primary indicators of empowerment, that were selected according to the evidence and logical reasoning: (1) participation (q1–4), (2) capacity building (q5–9), (3) evaluation/adaptability (q10–11) and the (4) Related Legislative Framework (q12). The number of questions intuitively corresponds to the perceived relative importance of each factor, although not enough evidence exists for a more precise question number and weight assignment, and can thus be subject to future modification according to quantitative and qualitative evidence. The justification of the indicator selection is as follows:

Participation

Participation is the most commonly cited, defining factor of individual and community empowerment (3, 6, 8, 11–13, 26). An eight-step ladder of citizen participation has been proposed to indicate the different grades and uses of the term, as well as its potential abuse to mislead and control the powerless (5). Non-participation often results in the loss of wider empowerment, as vested interests are uncontrolled to increase their power. Therefore, wide and essential citizen participation in decision making is necessary to maintain majority empowerment (3, 6).

Capacity and opportunities’ expansion

Individual and community capacity development are essential parts of empowerment. Efficacious knowledge/skills, employment/entrepreneurship, financing, budgeting and resource mobilization promote community ownership, experience and asset development, if adequate and centrally supported, while horizontal (as between individuals or communities) and vertical (as between civil groups and the government) networks/links are essential for empowerment's sustainability (3, 8, 11, 16, 24, 25).

Evaluation and adaptation

Empowering policies should be easy to evaluate and to modify at all levels. Rigidity and bureaucracy disempower citizens and communities from active participation in the political process (6, 11).

Legislative context

The overall political and legislative context relating to a policy was seen as another determining factor of their empowering potential. A consistently empowering policy framework is necessary for the positive effects of policies on empowerment to take place (8).

The 12 questions of the PEI are intended to describe whether and how policies empower their constituents. Distinct policy elements are graded from 0 to 5 (maximum score 60) and the sum score is used to classify the policy in one of five, index categories.

The index questions cover policy empowerment elements as follows:

1. Constituent concern with the policy issue and participation in agenda setting (q1, 2), policy planning (q3) and implementation (q4).

2. Building constituent capacities and opportunities through education/training (q5), employment/entrepreneurship (q6), network formation (q7), addressing power inequalities (q8) and resource mobilization (q9).

3. Modes of policy evaluation (q10) and adaptation (q11).

4. The empowerment features of related policies (q12).

Due to the lack of precise quantitative evidence on the relative empowerment potential of these factors (8), the questions were, arbitrarily, weighted equally, subject to future adaptation. Moreover, the scoring standards within each question were also iteratively set, based on speculative decisions on, for example, what constitutes a small, significant or vast constituent minority or majority (with limits set at 10, 50, 75 and 90% of constituents), the relative importance of lay versus expert and central versus peripheral control or participation, explicit versus implicit and direct versus indirect policy objectives and expected outcomes.

Accounting for predominant values, indirectness and uncertainty is not new in evaluations. In the GRADE methodology for guideline development, the value of wide stakeholder consensus and of procedural transparency in addressing the inherent imperfections and diversity of evidence is emphasized (27). Intuitive recognition and overall heuristic relevance are also accepted by methodologists as necessary in the systematic appraisal of qualitative studies, if combined with rigorous descriptions and documentation (28, 29). The PEI index can be accordingly fine-tuned in forum discussions, tested and validated, to increase the objectivity and applicability of its measurements.

The PEI evaluation questions, standards and scores are as follows:

Q1. How many policy-affected constituents are informed and concerned with the problem being addressed?

Standard (score): <10% (1), 10–50% (2), 50–75% (3), 75–90% (4), 90–100% (5) of constituents.

Q2. How was the policy agenda set?

Standard (score): it was set by experts: centrally (1), multi-level or peripherally (2), or mixed (experts and lay representatives): centrally (3), multi-level (4), peripherally (5).

Q3. How was the policy planned?

Standard (score): it was designed by experts: centrally (1), multi-level or peripherally (2), or mixed (experts and lay representatives): centrally (3), multi-level (4), peripherally (5).

Q4. What proportion of the policy measures are delegated peripherally for implementation?

Standard (score): <10% (1), 10–50% (2), 50–75% (3), 75–90% (4), 90–100% (5).

Q5. Does the policy call for education/training of constituents affected?

Standard (score): <10% (1), 10–50% (2), 50–75% (3), 75–90% (4), 90–100% (5) of constituents.

Q6. Are peripheral employment opportunities and entrepreneurship being enhanced?

Standard (score): employment/entrepreneurship for <1% (1), limited employment for 1–10% (2), wide employment for >10% (3), some entrepreneurship and self-employment for 1–10% (4), broad entrepreneurship and self-employment for >10% (5) of constituents.

Q7. Does the policy promote constituent participation in horizontal and vertical networks?

Standard (score): any for <1% (1), indirectly for 1–10% (2) or for >10% (3), explicitly/directly for 1–10% (4), or for >10% (5) of constituents.

Q8. Are hard to reach, vulnerable or disadvantaged populations being considered and affirmatively protected and empowered (including vulnerable gender and age groups, socially/physically/economically disadvantaged individuals, groups and communities)?

Standard (score): yes, any for <1% (1), indirectly for 1–10% (2) or for >10% (3), explicitly/directly for 1–10% (4), or for >10% (5) of vulnerable constituents.

Q9. Does the policy provide for adequate financial, human and other resources?

Standard (score): inadequate: central (1), mixed or peripheral (2) or adequate: central (3), mixed (4), peripheral (5).

Q10. Will the policy be evaluated formatively?

Standard (score): no (1) or evaluated by: central: quantitative methods (2), quantitative/qualitative (3) or peripheral: quantitative (4), quantitative/qualitative/participatory (5)

Q11. Is the policy adaptable?

Standard (score): inflexibly: centrally (1), mixed or peripherally (2), or flexibly: centrally (3), peripherally (4), mixed (5).

Q12. How empowering are associated policies?

Standard (score): the policy is mainly related with dictative (1), directive (2), supportive (3), enabling (4), empowering (5) policies.

Where the above criteria are non-applicable, for example when decisions are made non-transparently, in non-democratic ways, and are not benefiting the constituents or have a negative impact, the score of 0 should be applied. The final score is presented as a proportion of the maximum (60) to increase intuitiveness.

According to the index a policy can be classified as:

Dictative. In this policy type, the policy makers are addressing a problem that a minority of their constituents know or are concerned about. Mainly expert opinion and central interests guide the policy agenda setting and planning. The plans call for top-down implementations, without significant adaptation to peripheral particularities, needs and participation. The policy does little to build community capacity through education, training or employment opportunities or to protect vulnerable populations. Human, financial and other resources are centrally managed and inadequate. No formative or summative impact evaluations are considered necessary and central, inflexible mechanisms are or are put in place for policy modifications. The policy is interdependent with other dictative policies. PEI: 0–20%

Directive. A specific, detailed policy is designed, based on expert opinion and firm strategic goals, after considering public opinion and possible opposition, but mainly based on partisan views (e.g. the ruling party political ideology). Central and peripheral, expert panel committees may contribute to the policy plan. The policy experts provide a detailed guidance to reach the policy objectives, allowing little deviation in planning and implementation, however taking into consideration local particularities, input and voices of objection, also allowing for some peripheral participation, variation and adaptation to an extent such that the end outputs remain within the prescribed margins. Contrary to the previous level, peripheral authorities are respected although relatively passive partners in the policy planning and implementation, under the firm guidance of the higher legislative framework. The policy builds peripheral capacity to some extent through education, training and provisions for limited employment opportunities and participation in peripheral networks. It protects and empowers affirmatively a minority of vulnerable constituents. Adequate, mainly central, resources are planned. The policy is evaluated centrally for predefined outputs and outcomes, using quantitative measurements and can be adapted by central or peripheral inputs, albeit rather inflexibly. It relates and interacts with directive policies. PEI: 21–40%

Supportive. The policy addresses a problem that the majority of its constituents know and are concerned about. Lay representatives together with experts contribute in setting the policy agenda and in policy planning, albeit centrally. A significant part of the implementation depends on peripheral mechanisms, and peripheral capacity building is provided. Also, the policy is facilitating wider employment and peripheral network formation and action through legislational provisions and affirmatively supports the majority of vulnerable constituents. The central authority is providing adequate funds, advice and actively supports peripheral implementation, for example by providing expert advisers, information and a permissive national policy framework. It evaluates the policy implementation and impact formatively using both quantitative and qualitative methods and flexible, central mechanisms for adaptation exist or are set. The policy relates mainly with other supportive policies. PEI: 41–60%

Enabling. An enabling policy frequently addresses a problem that the vast majority of those affected know and are concerned with. The agenda is set by an interaction of central and peripheral actors, and the policy planning occurs at close collaboration between the centre and the periphery. Implementation is mainly delegated to the periphery. The policy also encourages peripheral sustained participation in assessment of needs and relevant action through sustained training and education of constituents, and through promotion of peripheral entrepreneurship, networking and cooperation at and between different levels. Adequate financial, informational and legislative support are combined with local resources and used as catalysts for peripheral involvement, participation and leadership, thereby promoting peripheral capacity development, proactively supporting and empowering the vast majority of vulnerable populations. Peripheral formative evaluation occurs, mainly through quantitative data collection and the policy can be adapted at least at the peripheral level, flexibly. The policy is part of a wider enabling policy matrix. PEI: 61–80%

Empowering. The policy is inspired as well as planned and implemented by the constituents under an empowering central and peripheral legislative framework, in a representative, consensual, polyphonic, equitably participatory manner, addressing issues that nearly all the affected constituents are informed and concerned with. It promotes further peripheral achievement and a sustainable, social, democratic development through essential, broad capacity development, through knowledge generation and communication, innovative and expansive peripheral entrepreneurship and asset generation, and through broad, inclusive and active, vertical and horizontal networks/links. Most vulnerable groups are affirmatively empowered. Peripheral participatory needs, formative and summative evaluations are conducted in a holistic manner (quantitative and qualitative, participatory action research) and a continuous feedback drives adaptations flexibly at the peripheral and central-framework level. Related policies follow the same principles. PEI: 81–100%

The policy types that form the basis of the PEI index are ideal, theoretical models, forming a value ladder reminiscent of a previous ladder of citizen participation (5). Real policies may contain elements of all the five policy types, and the average score illustrates which ideal policy type more closely relates to the evaluated policy. The PEI provides an indicative classification, enabling political discourse.

Data collection and assessment

25 related journal articles and essays, 6 policy documents and 1 statement, 6 relevant publications and 5 news announcements were retrieved through PubMed, ISI, Google Scholar and the Lund University Library (LibHub) using the search queries ‘Empowerment’, ‘Empowerment AND Policy’, ‘Community Empowerment’, ‘[(“South Africa”) AND (“traditional” OR “indigenous” OR “herbal”) AND (“healers” OR “practitioners” OR “medicin*”)]’. The most relevant documents were selected, examined and used for substantiation of the PEI and in the policy evaluation.

Results

PEI analysis of ATM

The evaluation scores and the source references are summarized in Table 1. The evidence and judgments behind the scores are as follows:

Table 1.

PEI evaluation of SA's ATM policy

	Questions	Standards (scores)	ATM PEI score (range)	Comments/ References
1	How many policy-affected constituents are informed and concerned with the problem being addressed?	<10% (1), 10–50% (2), 50–75% (3), 75–90% (4), 90–100% (5) of constituents	4 (3–5)	Over 80% of S. Africans use ATM (18, 19, 21, 22, 30, 40–42)
2	How was the policy agenda set?	It was set by: experts: centrally (1), multi-level or peripherally (2), or mixed (experts and lay representatives): centrally (3), multi-level (4), peripherally (5)	1	International agenda setting (10, 17, 20)
3	How was the policy planned?	It was designed by: experts: centrally (1), multi-level or peripherally (2), or mixed (experts and lay representatives): centrally (3), multi-level (4), peripherally (5)	1	Expert committee (31–33)
4	What proportion of the policy measures are delegated peripherally for implementation?	<10% (1), 10–50% (2), 50–75% (3), 75–90% (4), 90–100% (5)	1	Central implementation (19)
5	Does the policy call for education/training of constituents affected?	<10% (1), 10–50% (2), 50–75% (3), 75–90% (4), 90–100% (5) of constituents	2 (1–3)	For a significant minority (19)
6	Are peripheral employment opportunities and entrepreneurship being enhanced?	Employment/entrepreneurship for <1% (1), limited employment for 1–10% (2), wide employment for >10% (3), some entrepreneurship and self-employment for 1–10% (4), broad entrepreneurship and self-employment for >10% (5) of constituents	4 (3–5)	ATM employs locally (21, 22, 34)
7	Does the policy promote constituent participation in networks?	Any for <1% (1), indirectly for 1–10% (2) or for >10% (3), explicitly/directly for 1–10% (4), or for >10% (5) of constituents	2 (1–3)	Indirectly for THP, other health workers, patients? (32, 33). (See footnote 1).
8	Are hard to reach, vulnerable or disadvantaged populations being considered and affirmatively protected and empowered (including vulnerable gender and age groups, socially/physically/economically disadvantaged individuals, groups and communities)?	Yes, any for <1% (1), indirectly for 1–10% (2) or for >10% (3), explicitly/directly for 1–10% (4), or for >10% (5) of vulnerable constituents (including women, the elderly, children, people with disabilities, displaced/immigrants)	2 (1–3)	ATM enhances access to health (21, 22, 10, 17, 18)
9	Does the policy provide for adequate financial, human and other resources?	Inadequate: central (1), mixed or peripheral (2) or adequate: central (3), mixed (4), peripheral (5)	2 (1–3)	Central financing, ATM is a peripheral resource (18, 22)
10	Will the policy be evaluated formatively?	No (1) or evaluated by: central: quantitative methods (2), quantitative/qualitative (3) or peripheral: quantitative (4), quantitative/qualitative/participatory (5)	2 (1–3)	Through academic, quantitative research (19)
11	Is the policy adaptable?	Inflexibly: centrally (1), mixed or peripherally (2), or flexibly: centrally (3), peripherally (4), mixed (5)	2 (1–3)	Not predicted, dependable. (32, 33, 35)
12	How empowering are associated policies?	The policy is mainly related with dictative (1), directive (2), supportive (3), enabling (4), empowering (5) policies	2 (1–3)	Directive policy environment. (19, 36–39). (See footnote 2)
	Result	Non-applicable or negative	0	Supportive
		PEI score	42%	(directive-supportive)
		Range	27–57

Notes: Policy types according to PEI: 0–20%: ‘dictative’, 21–40%: ‘directive’, 41–60%: ‘supportive’, 61–80%: ‘enabling’, 81–100%: ‘empowering’.

Q1. It is a well-known fact that ATM is of universal use in the African Continent (18, 21), with more than 80% of South Africans using it (19, 22). Moreover, many among the non-users of ATM (e.g. part of the white population) are also expected to be concerned with ATM, such as with the health, economic and social consequences of its uncontrolled status. A large and increasing number of complaints for medical malpractice are recorded in SA, showing the broad concern with health issues (30). It can therefore be safe to assume that between 50 and 90% of the constituency is informed and concerned with the problem at hand. PEI-score (range): 4(3–5)

Q2. The policy agenda was set at international (WHO and AU) and country level (10, 17, 20). Although the ATM supporting civil society is growing, mainly through THP and support group associations, these groups and the average user of ATM have evidently had little to do directly with the policy agenda-setting. Many might prefer no or custom, regional or communal regulations for reasons of vested interests (22), uncertainty and lack of trust in central initiatives. Judging from the available documentation, no such dissident voices were taken into consideration. Instead, policy experts and government representatives identified the problem and set the agenda for ATM regulation. PEI-score: 1

Q3. The current policy was drafted by a central, expert panel, the ‘Presidential Task Team’ for ATM after consultation with ‘some stakeholders’ (19). The team was appointed by the former minister of health and included medical professors, advocates, pharmacists and professors of botany – some of whom were criticized for having vested, economic interests in TMs – (31–33), but no lay representatives or community members have been reportedly involved. PEI-score: 1

Q4. A top-down implementation was chosen for the main policy parts. All the organ and legislative adaptations and developments proposed concern the central government, and a central implementing institute, the National Institute of ATM (NIATM) will be overseeing practice (in a central hospital and pharmacy), as well as research and educational activities. The development of educational curricula and research projects in peripheral universities and the potential development of peripheral ATM clinics will be centrally planned, controlled and coordinated and cannot be considered as significantly devolved implementation. No decentralized implementation authority is being envisaged by the policy. PEI-score: 1

Q5. This policy draft puts considerable weight on education and training. ATM education is envisaged for THP and allopathic healthcare workers, health science students and in public schools, while information distribution to the public through publications, journals and other means of health promotion are proposed to expand the ATM knowledge and acceptability. However, no extensive, specific patient targeted education on health issues is planned, such that would empower them. The patients constitute the majority of the policy constituents. PEI-score: 2(1–3)

Q6. ATM employs a great number of health professionals, as described above. The current policy creates new career opportunities for THP and for others interested in ATM, in education, research, plant cultivation and ATMs production. Although some professions may be at risk (e.g. the traditional plant harvesters) by the proposals for organized cultivation and industrialized production, the institutionalization of ATM can be expected to broaden the scope and reach of ATM and to have an overall positive effect on local employment and entrepreneurship (21, 22, 34). A close follow-up of these effects is warranted. PEI-score: 4(3–5) (open for follow-up adaptation)

Q7. No specific reference is made for support to THP and other health professional associations, patient support groups and other constituents, although indirect positive effects can be expected from ATM officialisation. THP associations may be empowered and broadened, with participation of new scholars and supportive allopathic healthcare workers, and their links with other groups and government may be strengthened. The divide between indigenous and allopathic associations could be narrowed (32–34). The effect on patient's networks is harder to predict, although likely to be positive, as the political dimensions of ATM will be recognized and debated. PEI-score: 2(1–3)

Q8. No explicit consideration of the effects of the policy on vulnerable groups is noted, for example on how the policy will affect women's subsistence. Overall, vulnerable populations’ access to health will most likely be expanded by the institutionalization of ATM (i.e. by insurance coverage and free provision in public healthcare facilities) (10, 17, 18). Some groups may be adversely affected, such as the elderly THP and harvesters, less prone to adapt to the new situation (21, 22). PEI-score: 2(1–3)

Q9. Only a vague mention of central ‘resource mobilization’ is made as regards to financial resources. Since no estimation of the costs and no provision for funds are made, government funds are most likely to be inadequate in SA (23, 35). Ideally, ATM should be sustainably self-financed by the communities, as it currently is, but avoiding excessive out of pocket payments and generating local income from drug and knowledge development. Local human resources should be mobilized to sustain ATM and empower the communities; however, more emphasis is given to central expertise (i.e. the NIATM and universities) and infrastructure (a central hospital and botanical garden, pharmaceutical industries) than in local business and resource development. However, ATM being a community resource, its institutionalization constitutes to some extent a de facto peripheral resource mobilization (18, 21, 22). PEI-score: 2(1–3)

Q10. Although some form of monitoring of the policy implementation should be expected, no specific provision for formative policy evaluation is made. The measures to evaluate ATM safety and effectiveness, through scientific research, constitute a type of, central/quantitative evaluation of the policy results. No mention of qualitative methods is made, but a ‘holistic’ approach to that research is called for. PEI-score: 2(1–3)

Q11. No mechanism of adaptation of the policy to central or local needs is predicted beyond the initial call for ‘substantiated comments’ on the policy draft (19), although, given the contradictory nature of this policy in SA, especially in the area of HIV/AIDS (32, 33), some adaptability may prove necessary. This may now depend more on the composition of the supervisory central bodies (i.e. the personalities involved) than on established mechanisms for adaptation. Flexibility is unlikely given SA's challenged relationships between local authorities and the central government (35). PEI-score: 2(1–3)

Q12. The Biodiversity and Patent Acts, the medicine registration and such legislations relating to the current draft, although difficult to evaluate independently herein, mostly appear to be of a dictative or directive nature (19, 36–39). No sufficient peripheral support and empowerment can be expected from legislations that establish central bureaucratic hurdles. PEI-score: 2(1–3) (provisional)

A total PEI score (range) of 25 (16–34) or 42% (27–57%) of the maximum defines the policy as supportive and ranging between the directive and supportive policy types.

Discussion

A new policy evaluation tool has been proposed and applied in the evaluation, from an empowerment perspective, of SA's policy draft for ATM (19). The PEI index was based mainly on evidence reviews and recommendations on the empowering factors of policies by the UK government, WHO and the World Bank (3–5, 8, 11, 16), and it assessed the policy effects on citizen and community participation, capacity and social capital development, as well as its adaptability to local contexts. It was put forward as an iterative tool for policy evaluation, constructive critique and discussion, open to further development.

The proposed SA ATM policy was scored, albeit marginally, as a ‘supportive’ policy according to the PEI definition. It is a policy that addresses a problem that the majority of its constituents are concerned with. Albeit centrally conceived and planned, significant parts of supportive policies’ implementation depend on peripheral mechanisms and capacity building and are adapted comprehensively.

However, the confidence range places the policy also close to the ‘directive’ policy type. ‘Directive’ policies are based mainly on expert opinion and seek to build peripheral capacity through education, training, employment opportunities and participation in peripheral networks, although mainly central resources are provisioned, and central, quantitative monitoring makes adaptations difficult.

Although neither description fits entirely to the evaluated policy draft, the overall classification of the SA ATM policy between these two policy types appears plausible and could motivate a policy improvement discussion.

The following factors of the SA ATM policy draft were evaluated:

Participation. In order for a policy to empower its constituents, the voice of the affected population must be heard, through active and effective participation; implementation can be devolved to the peripheral level, promoting community ownership and involvement (8, 16). Although the ATM policy agenda was first set by the experts of WHO, and the policy was planned by local politicians and expert advisers (10, 17, 20), with a limited role of the populace, an important factor for the index scoring of this policy as a supportive policy has been addressing an issue of overarching community concern and empowering potential (18, 21, 22). An empowering ATM policy could be inspired, developed and executed through the grassroots community movements, where the local needs are better felt and a multitude of ways to address them can be put forward. A local adoption may promote a more efficient integration to the NHS.

Capacity development. Empowering policies should promote equitable constituent education/training and (self-) employment, participation in budgeting/financing as well as access to health (3, 4, 7, 8, 9, 11, 12, 16). ATM is practised closer to the local communities than modern medicine, by respected community members, elders and women (18, 21); therefore the regulation promotes an expansion, diversification and evolution of these activities, assisting wider employment and entrepreneurship, calling for community intellectual property rights’ protection and allowing for local investments (versus large-scale manufacturing and commercialization). Optimally, local communities should be made capable to mobilize and use their own mechanisms in support of the ATM expansion and institutionalization, also through improved networks with other organizations and the government. The result could be closer to their needs and easier to sustain.

Evaluation and adaptation. Participatory evaluations and feedback mechanisms are essential for empowering policies (2, 4, 8, 12). There is no provision for central or peripheral evaluations in the current policy draft. Given the policy environment in SA (35), an inflexible mechanism for policy adaptation can be expected if no exceptional provisions are made, although a lot will depend on the personalities involved. Although flexible adaptations are not unlikely, given the strengthening of the ATM civil society1 and the changing global political and technological environment, participatory research would strengthen both the quality of the feedback and the communities’ power in the process. Follow-up and PEI re-evaluation may be necessary for such policy elements, uncertainty and fluidity being part of the political reality.

Legislative context. An empowering overall legislative context is essential for the empowering effects of policies to materialize, since few policies are not dependent on previous policies and regulations. From all the available information on policies relating to the SA ATM draft (18, 36–39),2 a directive overall tone was asserted, albeit, with a high degree of uncertainty, due to practical difficulties in applying the PEI to the other policies. More systematic investigations are necessary for such judgments. The result can also be interpreted as the policy moving a step forward from its legislative background towards constituent empowerment.

The present endeavour to evaluate a policy with a custom-made evaluation tool has several limitations. The objectivity of the evaluator, being also the developer of the method can be questioned. The tool and evaluation result are the work of modern, non-South African physicians, and therefore their beliefs and values are undoubtedly behind the index and its application to the policy. Furthermore, the PEI is based on some sweeping generalizations of the value of evidence on empowerment and the impact of empowering policies.

Also, due to practical difficulties in collecting and assessing all the required information, and the necessary use of ‘grey’ literature, a degree of uncertainty in the evaluation could not be avoided, and thus the iterative nature of the evaluation is emphasized. The results need to be confirmed by lay and expert stakeholders, at different time points during the policy implementation. The field of politics is volatile, and unpredictable cause and effect relationships are common. Further triangulation and validating application of the index in other policies would also better establish the confirmability of these results.

Therefore, both the PEI and the evaluation should be regarded as a work in progress, to be further established, discussed, triangulated, validated, developed and fine-tuned in pluralistic dialogue. Despite its limitations, the PEI can be useful in better understanding the impact of empowerment and empowering policies, if used to collect, summarize and assess policy-related information. It could also be useful as a basis and part of broader and participatory evaluation components integrated in ATM and other policies. The main purpose of the PEI is to serve as a discourse instrument. The evaluation was mainly used as a formative tool in the development of the PEI.

It is possible that a more community-empowering approach could be more effective to overcome the obstacles towards institutionalization of ATM in SA. The media can be used to pluralize the debate over controversial issues such as the management of HIV/AIDS (30–33). The national and international civil society should be invited to the discourse, the THP associations should be strengthened and healers and patients should be empowered to claim their rights in the medical and political, as in the social environment. A bottom-up implementation approach, allowing for flexibility and adaptation to context, may consequently be more effective in accounting for the contextual variation and complexities.

More research is needed in order to develop and improve policy evaluation tools, such as the PEI. Quantitative and qualitative assessments and measurements of the multiple dimensions of empowerment and their benefits on health (e.g. on morbidity, mortality, access to health and health perceptions), education (gender-specific access and perceptions, literacy levels), capacity (such as poverty and unemployment rates, perceived autonomy and power to influence one's environment), and social capital (perceived and measured equity, level of trust). Analysis of the empowerment features of diverse policies, and assessment of their implementation and results can support and fine-tune the PEI.

Conclusions

As the factors affecting empowerment are increasingly being recognized, valid methods for the appraisal of these factors in policies need to be developed. The hereby proposed index has been developed based on a comprehensive review of the current evidence and guidance, and has been applied on SA's 2008 policy draft on ATM. The policy was evaluated as marginally supportive of population empowerment, mainly due to ATM's closeness and importance, as a resource, for the communities. However, many disempowering (‘directive’) elements were identified, and the involvement of the constituency in further policy planning and implementation may be salient.

This evaluation method and the results need to be reviewed, discussed and adjusted polyphonically, their primary purpose being motivating political awareness and dialogue on empowerment of policy constituents. As an evaluation tool, the PEI can also be used, in combination with empowerment measurement methods, in political research and development. Health policies, where a significant part of the evidence of the mechanisms and benefits of empowerment can be found, may be the first to benefit.

Key messages.

An evaluation tool of the impact of policies on empowerment is missing and necessary, in order to facilitate relevant, systematic information and opinion collection and assessment. Such an instrument is hereby developed and proposed.

South Africa's Traditional Medicine policy proposal appears to possess several community-empowering features, but can benefit from more participation as well as from formative evaluation and impact assessments in its design.

Conflict of interest and funding

No conflicts of interest are reported. No funding or other type of financing was received for this work. The authors have no political or other conflicting affiliations.