Improving global health: measuring the success of capacity building outreach programs: a view from the International Society of Nephrology

Abstract

Capacity building is key to the advance of health care in the developing world, but capacity building can take several forms, and it is not yet clear which forms and funding models are most effective. The International Society of Nephrology (ISN) has developed and sustained a portfolio of global outreach programs that make a unique contribution at low cost to the appropriate training and continuing support of nephrologists throughout the developing world. We describe the programs and present encouraging findings of their outcomes. Finally, we reflect on how the ISN interventions and evaluations measure up against recommended approaches in the literature and consider lessons for ISN and other organizations involved in planning, evaluating, and benchmarking similar programs.

Capacity building is a term with many definitions and at least 4 distinct forms.¹ In this paper, we use the term to mean the ongoing and autonomous ability at personal, institutional, and societal levels to respond better to local need, arising from interventions that identify and overcome barriers to development and that develop stakeholders’ required skills and abilities.

Enhancement of renal health care facilities is necessary to address patients’ needs in low-resource settings, but this alone will not overcome the problem. There remains a great shortage of appropriately trained clinical staff who can deliver and develop enhanced treatments and services, within the medical, socioeconomic, and political environment in which they work. For example, there are 20 nephrologists in Tanzania to serve a population of 45 million.² It is increasingly acknowledged that capacity-building programs must deliver training and offer opportunities to develop individuals and teams in ways that minimize the “brain drain” that continues to threaten professional services and civic society in low- and middle-income countries (LMICs). Educational programs are therefore required to train and retain a health-care workforce with competencies relevant to the medical and socioeconomic environment in which they work.³

Over the past 30 years, the International Society of Nephrology (ISN) has expanded its efforts to support improvements in care for people with kidney disease in LMICs through sustained investment in a portfolio of complementary education and training programs (www.theisn.org/programs).

The ISN Programs

The ISN has 5 programs that focus effort exclusively on LMICs. The global distribution of ISN programs in 2016 is shown in Figure 1, with more details in the interactive online repository at http://www.theisn.org/programs/isn-programs.

Figure 1.

Distribution of International Society of Nephrology programs in 2016. For additional details, go to http://www.theisn.org/programs/isn-programs. CRP, Clinical Research Program; EAP, Educational Ambassador Program; ISN, International Nephrology Society; SRC, Sister Renal Center; STC, Sister Transplant Center.

In brief, the goals, duration, and achievements of these programs are as follows (and summarized in Table 1).

Table 1.

Summary of the ISN programs

Program	Year established	Program structure	Number of beneficiaries
Fellowships	1985	2–12-month training at expert center (in same or different region). Fellow must return to home center	>600 fellows trained since 1985
CME	1993	Speakers sponsored to attend local organized CME 40–50 per year ISN speakers also give teaching rounds and meet local health care leaders	CMEs attended by ∼20,000 health professionals per year
Sister Renal Centers	2003	Supporting center in developed world linked to emerging center in developing world Center “graduates” after a 6- to 8-year program supported with increasing budget if there is continuing success	24 sister centers “graduated” since 2006 40 pairs supported each year since 2009
Clinical research	2005	Seed funding for small clinical research projects Support for kidney disease surveillance programs; helping LMICs to generate their own epidemiologic data Each project assigned a mentor Support to attend a scientific writing course	51 clinical projects supported since 2005 Average of 8 projects per year since 2014
Educational Ambassadors	2006	Experienced nephrologists go to an LMIC nephrology center for 1–4 weeks Help to establish new programs that will benefit local patient care through skills training and advice on organizing clinical services	83 visits since 2010

CME, continuing medical education; ISN, International Society of Nephrology; LIMC, low- to middle-income country.

These programs are exclusively in low- and middle-income countries.

Fellowships

ISN’s Fellowship Program is the largest and oldest international subspecialty program in internal medicine dedicated to training physicians from developing countries. Since 1985, it has supported >600 nephrologists from >80 LMICs to undertake appropriate training at expert centers in fellowships of short (2–6 months) or longer (up to 12 months) duration. Fellows are expected to have had at least a year’s experience in nephrology and are required to sign an agreement stipulating return afterward to their home center. Preference is given to hands-on clinical training and, in recent years, to training within the fellow’s own region of the world to increase the relevance and utility of the training to the fellow’s home institution to reduce costs and to lessen the risk of “brain drain.” Fellows have come from all developing regions of the world, and, historically, 80% trained in Europe or North America, although training within the same geographical region as the fellow's home center currently stands at 35%.

Continuing medical education

ISN has supported continuing medical education (CME) programs since 1993. Currently, it supports ∼40 to 45 CME nephrology programs per year, organized locally and equitably distributed across LMICs. ISN strengthens each program by supporting as many as 3 international experts who supplement local speakers. We estimate that some 12,000 health-care providers in the developing world participate in these educational programs each year. As well as teaching in the CME event itself, ISN-supported teachers will often give nephrology teaching rounds at the local hospital. In addition, ISN leaders at CMEs usually meet with local physicians, leaders of hospitals and medical schools, and sometimes with health ministers and other government officials to assess the current needs and future opportunities for nephrology and determine how ISN can contribute.

Sister renal centers

A unique feature of the ISN programs is the focus on sustainable development of nephrology centers in LMICs through a “sister” relationship with an established center in the developed world. The ISN Sister Renal Center (SRC) program was originally started in 1997, but it has had its current effective structure since 2003. Centers typically “graduate” after 6 to 8 years, having received increasing ISN support as they demonstrate success. After graduating, the center in turn supports another emerging center in its own country or region with continuing input from its developed world sister center (known as a “trio” arrangement). Education, training, and the development of new treatment options are key features of the sister center relationship—a 2-way process with as much to learn for faculty, fellows, and students from the supporting center as for those in the emerging center.

Twenty-four SRC partnerships have graduated since 2009. The SRC program has been supporting ∼40 centers each year since 2006.

Clinical research

Since 2005, the ISN Clinical Research program has aimed to build capacity for research and chronic kidney disease surveillance in LMICs in 3 ways. The first is supporting research through seed funding for small clinical research projects. The second is enhancing chronic kidney disease surveillance programs, helping LMICs to generate their own epidemiologic data, which is critical to influencing health policy and directing further research. The third is assigning an experienced ISN member as a research mentor to recipients of an ISN Clinical Research grant and providing funding to attend a structured course in scientific writing, aiming to increase the likelihood that the work will be published in a peer-reviewed journal.

Since 2005, 51 clinical research projects have been supported, with an average of 8 projects per year since 2014, and 29 have been completed to date.

Educational Ambassadors

Educational Ambassadors (experienced nephrologists who are skilled practitioners and teachers) go with ISN support to an LMIC nephrology center for periods of 1 to 4 weeks to help establish new programs that will benefit local patient care. The key features of this program include training for health-care personnel and improved organization of clinical services. Although the Educational Ambassador program was only established in 2010, the ISN already has a volunteer database of 130 international experts, covering every aspect of clinical nephrology, who have offered to travel and teach anywhere in the developing world. Between 2010 and 2014, 83 training visits by ISN Educational Ambassadors were carried out, 33% in Africa and 23% in Latin America, with smaller numbers in other developing regions of the world. Training has been provided in clinical nephrology, pediatric nephrology, acute kidney injury, prevention of kidney disease, dialysis, interventional nephrology, renal pathology, and transplantation.

Integration of Programs

Although described for convenience as 5 separate programs, synergies ensure maximum effectiveness and efficient use of limited budgets. For example, a young nephrologist in an SRC will be prioritized for Fellowship support; an SRC may be a preferred site for a Clinical Research Project or an Educational Ambassador visit or a preferred site for an ISN-supported CME.

Partnerships in ISN Programs

The ISN values partnership working in all these programs. These partnerships reflect natural synergies with professional societies, including regional and national nephrology societies that share our aims and also a range of individuals, foundations, and corporate partners who provide financial support that helps the ISN maximize the reach of the programs.

Do These Programs Meet the Needs of the Countries They Purport to Assist?

These 5 programs represent a sustained commitment to enhancing health care in the developing world through the training of local physicians and other health-care workers and through improvements to clinical services, and as far as we know, these developments are together more extensive and broader in reach than those offered by any other specialist medical society.

There seems little doubt that such education and training programs will result in improved expertise of health professionals in the care of those with kidney disease. Anecdotal responses from those who benefit from such training are helpful, but alone are not sufficient evidence of success, and the ISN has sought to provide more robust evidence of the utility of the programs and the possibilities for their development.

The eventual goal of education and training in this arena is to improve patient care, and Kirkpatrick⁴ describes this as the “Results” in his 4-level model of evaluation. He also includes 3 intermediary levels that might be usefully evaluated. The 4 levels of the model include reaction or feedback from the participants on their learning experiences, learning achieved as demonstrated in assessment results, behavior changes such as prescribing patterns, and results such as changes in the well-being of patients or the health-care system arising from the training intervention.

Impact Assessment

A variety of impact assessments have been carried out for the 5 programs over a number of years, and in the following we report the results of evaluation across Kirkpatrick’s 4 levels where possible (examples are also shown in Table 2).

Table 2.

Impact of ISN programs

	Reaction (Feedback on the learning experience)	Learning (Assessment results)	Behavior change (Alterations in individual practice)	Results (Changes in patient care and health care systems)
Fellowships	Fellows and home mentors give good or high rating to relevance of training for: •home region (85%) •home country (88%) •home institution (92%) •individual fellow (95%)	Some fellows obtain postgraduate qualifications during training	Past fellows become senior leaders in nephrology and medicine	Returning fellows report innovation including: •new educational programs (73%) •new renal units (38%) •increased patient numbers (60%) •new techniques (43%) •improved quality of renal care services (61%)
Sister Renal Centers	Positive impact at emerging center reported on clinical care, clinical training	Publication of >50 peer-reviewed articles	Introduction of standardized clinical protocols and working practices	Improvements in kidney transplant rates (Belarus, Gaza) New dialysis programs (Ghana, Nepal) Expanded dialysis programs (Indonesia, Argentina, India)
Educational Ambassadors	High response rate, 82% of visits rated highly successful	Improvement in skills (e.g., renal pathology, vascular access surgery, acute PD)	Improved renal biopsy skills Improved vascular access surgery skills	50% increase in renal biopsies (Vietnam) Higher quality vascular access (Vietnam) Improvement in quality of dialysis provision Those trained become resource persons for their region
Clinical research	52% indicated that the research grant had helped them to achieve their current position	Individual research skills enhanced through mentorship and scientific writing academy	Grant recipients •published ≥1 peer-reviewed articles (59%) •presented findings at national or international congresses (56%) •received national or international award based on their ISN-funded research (20%) Project in Zhuhai, China led to large EU-funded project, 17 papers, training for 16 PhD degree students, 12 Master’s degree students	80% indicated that research results had a positive impact on their home institution or country Provincial laboratory of metabolic diseases founded after impact of research (China)
CME	Low response rates Education highly valued, increased knowledge Preference for didactic teaching	Lecture materials available after CME through ISN online education	Encouragement of scientific publications and development of fellowship programs Discussions during CME led to start of peritoneal dialysis/AKI programs in DRC and Cambodia	Publications from Sudan Successful implementation of PD/AKI programs Physicians chose to train in nephrology after stimulus of CME

AKI, acute kidney injury; CME, continuing medical education; DRC, Democratic Republic of the Congo; ISN, International Society of Nephrology; PD, peritoneal dialysis.

Impacts are described according to the criteria of Kirkpatrick.⁴

Fellowships

Methods

Since 2008, the Fellowship Program has conducted a biennial survey of past fellows. A Web-based survey is sent to all fellows who have recently completed their training and have returned to their home center. The mixed-methods survey includes specific questions with a Likert scale and free text. It collects subjective information about the quality of the training received, as perceived by the fellow, and evaluates the impact of the fellow’s training through information about new clinical, educational, or research programs that have been established in the home center. Where possible, the responses were corroborated by their home mentors who were asked similar questions.

In an effort to increase the relevance of the data collected, the survey was modified in 2014 and sent to all 130 fellows who had undertaken their training between 2005 and 2013 and their mentors. Periodic reminders are sent to maximize response rate.

Results

The results of the 2010 survey showed in summary a very positive impact on the individual trainees and their home institutions.⁵ The 2014 response rate for fellows was 61% and 23% for mentors. Eighty-eight percent of responding fellows were re-employed by their original home institution. The impact and the relevance of the training were rated as high or good by the large majority of fellows and home mentors with regard to the needs of the home region (85%), home country (88%), home institution (92%), and the individual fellow (95%). As a result of the training, new educational programs were developed in the home institution by 73% of returning fellows; 38% of respondents set up new renal units and 60% reported an increase in patient treatments after their fellowships. In 61% of responses, the quality of renal care services was thought to have improved as a result of the training, and 43% of respondents reported introducing new techniques. Examples of successful ISN Fellowships are given in Table 3.

Table 3.

Examples of the impact of successful ISN fellowships

In Africa

Razeen Davids was an International Society of Nephology (ISN) Fellow in Toronto, Ontario, Canada in 2000–2001, with Professor Mitch Halperin. His training was focused on electrolyte and acid-base disorders. He also was a recipient of a clinical research program grant that helped to complete a chronic kidney disease prevalence study in Cape Town, South Africa.

He is now Head of the Division of Nephrology at Stellenbosch University and Tygerberg Hospital. He is also Secretary of the South African Renal Society, Co-chairperson of the South Africa Renal Registry, and Chair of the Registry Committee of the African Association of Nephrology (AFRAN).

The ISN Fellowship was a strong force that helped shape his career. He continued to work with Professor Halperin, under whose supervision he recently completed his PhD. He is now, in return, serving as a host mentor to several ISN Fellows from other African countries and from Nepal and serves as an ISN Educational Ambassador, teaching hands-on skills in Nepal and Ghana.

In Central and Eastern Europe:

Rumeyza Kazancioglu learned about ISN when she participated in a continuing medical education course in Istanbul in 1997. In 2000, she was awarded an ISN Fellowship to spend 6 months in the Department of Nephrology and Transplantation in Vanderbilt University Medical Center, Nashville, Tennessee with Dr. Simin Goral and Dr. J Harold Helderman. When she returned home, she continued to collaborate with colleagues at Vanderbilt University and published a series of papers. Rumeyza Kazancioglu is now Professor of Nephrology at Bezmialem Vakif University Faculty of Medicine in Istanbul. She founded the Nephrology Clinic at Haseki Training and Research Hospital, Istanbul, Turkey, which she directed from 2001 to 2010. She became an Associate Professor of Nephrology in 2003 and Professor in 2010. She is now the President of Bezmialem Vakif University, Councilor of ISN and chair of the ISN Central and Eastern Europe Regional Board.

Continuing medical education

Methods

At each ISN-sponsored CME, an evaluation form was provided for each delegate until 2014. The form sought comments on the overall educational value and relevance of the meeting program as well as evaluation of individual speakers.

Results

Response rates were very low (only 500 responses were received between July 2012 and April 2014 out of an estimated 20,000 individuals who attended these events during that time). Efforts to improve response rates by follow-up e-mails were ineffective. The great majority of health-care professionals who attend these programs are not ISN members and have no continuing interaction with the ISN. Use of the delegate evaluation form was therefore discontinued in 2014.

Nevertheless, the 500 responses received came from CME meetings in 29 countries: mainly India (16%), China (10%), Nigeria (9%), and Russia (8%). Of the respondents, 94% agreed that topics were well selected, and 88% indicated that the course had enhanced their knowledge of the topics. Eighty-one percent indicated this new knowledge would have a direct impact on patient care, 47% said that didactic lectures were the most effective and valuable part of the program, 22% preferred case discussions, 31% did not state a preference, and 69% agreed there was sufficient time for discussion and interaction.

Reports from conference organizers after each CME are always very positive about the impact of ISN’s support. Important outcomes, which are less easily quantified, are that outside speakers supported by the ISN, attract attendees to meetings, stimulate ideas in attendees, and create relationships that fertilize plans giving rise to new collaborations both within and beyond the ISN programs. For example, after a recent CME in Tanzania, ISN leaders began to work with attendees from the Democratic Republic of the Congo and established a peritoneal dialysis program for acute kidney injury in Goma in the Democratic Republic of the Congo. A CME in Sudan started discussions that led to a series of publications on the Sudan experience with peritoneal dialysis for end-stage renal disease and acute kidney injury. At a CME in Ethiopia, a discussion with local leaders supported the development of a national nephrology fellowship program. A CME in Cambodia resulted in the successful establishment of a peritoneal dialysis program for acute kidney injury.

Sister Renal Centers

Methods

The first impact survey of the SRC program was undertaken in 2013. Both emerging and supporting centers in active sister center partnerships receive Web-based surveys designed to gather evidence about the impact of the sister center on the development of clinical services as well as education and training opportunities. The mixed-methods survey include specific questions with a Likert scale and free text. Periodic reminders are sent to maximize response rate. In addition, an analysis of the SRC graduating between 2011 and 2014 was done from routine annual reports backed up by targeted follow-up questionnaires.

Results

Twenty-seven of 34 (79%) active SRC partnerships with emerging centers in 19 different countries responded to the 2013 impact survey. Developing clinical services was the main focus of most links (81%), with general nephrology (48%), hemodialysis (44%), and transplantation (33%) judged to be the most improved across the centers. The establishment of new services (41%), direct interactions and advice between staff members (41%), and training exchanges between centers (41%) were thought to be important accomplishments. The SRC program was reported to have had a positive impact on clinical care (median impact rating, 8/10), clinical training (9/10), and research (7/10) within the emerging center. Supporting centers noted increased awareness and interest in global health among their faculty.

Based on the routine annual reports and targeted follow-up questionnaires, there was notable success in developing transplantation services.⁶ Elsewhere, access to renal replacement therapy improved with new dialysis programs established in Nepal and Ghana and expanded peritoneal dialysis programs in Jakarta, Indonesia (3-fold increase), and Cordoba, Argentina (7-fold increase). In Bangalore, India, pediatric hemodialysis services increased 5-fold. SRC support has allowed training of specialist nurses, dietitians, pharmacists, pathologists, radiologists, physicians, and surgeons, giving a new multidisciplinary team focus to many clinical services. Finally, these SRC partnerships undertook research, resulting in >50 joint publications in peer-reviewed journals.

Examples of successful ISN SRCs are given in Table 4.

Table 4.

Examples of the impact of successful International Society of Nephrology (ISN) Sister Renal Centers

Bangalore/Montreal/Ethiopia

In 2007, a Sister Renal Center (SRC) was established between the Children’s Kidney Care Center at St. John’s Medical College Hospital, Bangalore, India, and Montreal Children’s Hospital, Montreal, Quebec, Canada. The Indian center faced the challenges of providing cost-effective clinical services, of suboptimal multidisciplinary expertise, and of inadequate community-oriented services in addition to its limited academic resources.

Over the years, the SRC supported faculty development and capacity building in areas of clinical services and research. An active academic exchange program between faculty and trainees resulted in telemedicine sessions, courses, and regional continuing medical education covering a broad spectrum of topics. The SRCs launched a “Manual in Pediatric Nephrology” for pediatricians, intensivists, and pediatric nephrology trainees.

The Center in Bangalore is now recognized as a training center by the International Paediatric Nephrology Association and provides India’s most respected postdoctoral course in pediatric nephrology.

Research opportunities included the opening of a state-of-the-art renal genetic laboratory. Community awareness programs on common renal disorders in children, deceased organ donation, and World Kidney Day themes were successful annual events in this SRC journey.

In 2015, the SRC started another sister relationship within the ISN TTS Sister Transplant Center Program with Bambino Gesu Children’s Hospital, Rome, Italy. They have also established an SRC trio, linking with Addis Ababa University in Ethiopia.

Liverpool/UK

An SRC partnership was established in 2013 between the Royal Liverpool University Hospital in the United Kingdom and Al Shifa Hospital in Gaza, Palestinian Authority. This is part of the Sister Transplant Center program, a partnership between the ISN and the Transplantation Society.

At the time that it was established, there were 550 patients on long-term dialysis in Gaza, but all who required kidney transplantation had to travel abroad.

In 2013 to 2015, the Liverpool team visited Gaza 5 times and performed 15 kidney transplantations, 6 in children. All these transplantations used living donors because deceased organ donation was illegal in Gaza. In 2014, a team of doctors, nurses, and technicians from Gaza were hosted in Liverpool and trained in all aspects of kidney transplantation. In 2015, a surgeon from Gaza started a 3-year training in Liverpool.

Other infrastructure support through the Sister Transplant Center includes installation of a clinical information system to support renal care including collecting data on all patients receiving dialysis as the basis for a Renal Registry and a Transplant waiting list. There are plans to establish a tissue-typing laboratory.

With the support of the Liverpool team, Gaza's legislative council has now established a legal framework allowing deceased organ donation, as well creating an independent transplant authority to regulate and monitor transplant activity in Gaza and promote organ donation.

The Liverpool team will continue to support Gaza until they are confident to deliver an independent high-quality renal transplant program.

TTS, The Transplantation Society.

Clinical research

Methods

A Web-based impact survey was sent annually to all recipients of Clinical Research Awards in the past 5 years. The survey focuses on the success of the funding, including presentations, publications, and awards that are judged to be a direct consequence of the ISN funding. The survey also asks information about the impact of the Clinical Research Awards on the home institution and country. Periodic reminders are sent to maximize response rate.

Results

The biennial impact assessment survey was most recently distributed in 2015. Of the 27 (59%) previous grant recipients who responded to the survey, 89% reported satisfaction with the funding program. Overall, 56% reported that they had presented the findings in lectures at national or international congresses, and nearly 20% had received a national or international award based on their ISN-funded research. Approximately 80% of respondents indicated that the results of their research had had a positive impact on their home institution or country, and 52% indicated that the research grant had helped them to achieve their current role or position. Overall, 59% of grant recipients had published their findings in 1 or more peer-reviewed journal articles. An example of a successful Clinical Research Program is given in Table 5.

Table 5.

Example of the impact of an ISN clinical research program in China

In 2004, a pilot study of early detection and management of Chronic Kidney Disease, Hypertension, Diabetes and Cardiovascular Diseases (KHDC) in developing countries started in Zhuhai, Southern China. This successful effort led to further International Society of Nephrology (ISN) funding in 2007 for a larger protocol of the KHDC Program in the southern provinces of China. The study was led by Professor Hequn Zou, Director, Institute of Urology and Nephrology, Southern Medical University and former ISN fellow. Building on the award from the ISN Clinical Research Program, the project obtained further political and financial support from China, leading to a grant from the European Union Seventh Framework Program.

More than 15,000 community residents between the ages of 18 and 75 years were involved in an epidemiologic study across southern China. Seventeen original articles (including 10 Science Citation Index papers) and 2 editorials have been published based on the study. Sixteen PhD students, 12 Master’s degree students, and >100 medical staff received training in the KHDC project, and so far, 5 students have finished their PhD thesis and 3 have finished their Master’s degree thesis from the KHDC project.

Due to the great social and economic contribution of the project to public health, a Provincial Key Laboratory of Metabolic Diseases was founded in Southern Medical University Affiliated 181 Hospital, Zhuhai.

Educational Ambassadors

Methods

Two impact assessments are carried out: at 6 months after each visit and at longer term at 2 to 5 years. The web-based survey focuses on short-term and sustained improvement in clinical care in the emerging center. Information is sought about new clinical care programs that have developed as a result of the Educational Ambassador’s teaching and training and identifies any continuing interactions between the Educational Ambassador and the emerging center. Periodic reminders are sent to maximize response rate.

Results

The response rate for the 6-month survey was 93% and 54% for the later survey. Analysis of the results shows that 82% of visits were highly successful, and 18% reported success. Emerging centers and the ISN Educational Ambassadors keep in contact over the subsequent years via e-mail, teleconferences, video conferences, and meetings. Educational Ambassadors provide ongoing support through advice on clinical cases and research and assistance in the implementation of new projects and initiatives in the emerging centers. Several ISN Clinical Research grants have subsequently been awarded to the emerging centers. Fellows from the host centers are trained at the Educational Ambassadors’ institutions, and SRCs have developed. Most importantly, quality of care (diagnostic and therapeutic) has improved; for example, more kidney biopsies are performed and vascular access is created on-site instead of patients having to travel to other centers or other countries. An example is the 1-year training in renal pathology at the Royal Melbourne Hospital of a pathologist from the Cho Ray Hospital in Ho Chi Minh City, Vietnam, after the visit of the Renal Pathology Educational Ambassador from Melbourne. There was a >50% increase in renal biopsies in 2014 (from 101 to 159) during the year after the training visit, with expansion in the indications for renal biopsies to include all patients with lupus nephritis, rapidly progressive renal failure, and unknown kidney failure.

An example of a successful Educational Ambassador visit is shown Table 6.

Table 6.

Example of the impact of an ISN Educational Ambassador program in Southeast Asia

In 2015, John Swinnen from Westmead Hospital in Sydney, Australia, made an Educational Ambassador visit to Cho Ray Hospital, Ho Chi Minh City, Vietnam. This hospital already had an established Sister Renal Center partnership with Westmead and Royal Prince Alfred Hospitals in Sydney, which was supporting clinical progress, particularly in kidney transplantation. But the visit of Swinnen, a vascular surgeon, enabled a step change to be made in the provision of high-quality vascular access for hemodialysis, which is being sustained by the staff whom he trained.

Discussion

The importance of capacity building for health professionals in LMICs is self-evident. An informed and educated workforce should help to drive improvements in direct patient care and will also provide the leadership necessary to have an impact on health policy and investment at institutional, regional, and national levels. It would be easy for the ISN to make the assumption that its 5 LMIC programs described here are “doing good work.” But the challenge is to measure that success, and use information about variations in success to drive continuing improvement in the structure and delivery of the programs.

The survey methods that we have used are mostly questionnaires that focus on individual assessment of the value of specific educational elements, as well as broader questions about the improvements in practice that are thought attributable, at least in part, to the ISN program being evaluated. The evidence that we have obtained gives a positive picture of the sustained impact of the ISN’s 5 programs. There are, however, further reflections and lessons to be drawn from each program.

As well as the encouraging information from the Fellowship surveys about both short- and long-term benefits of training,⁵ many former fellows are in senior leadership positions in nephrology and more broadly in medicine in their own countries. But because the fellowship program sets out to identify the best young physicians, it could be argued that they would have achieved such goals anyway. However, the international experience and training afforded them through the fellowship are likely to have increased their profile and prominence locally. It may not be surprising that the fellows report a strong satisfaction with their training, not infrequently describing it as a pivotal period of their professional life, but other outcome measures were corroborated by the fellows’ home mentors.

Evaluation of SRCs could provide the best opportunity directly to quantify impact because the partnerships between the centers sustained over several years are susceptible to serial assessment of progress in the quality of clinical care as well as the direct education and training experience. Here again available survey evidence is encouraging. However, when an SRC is evaluated over a period of years, it is not easy to separate the influence and benefits of the ISN’s investment from other changes and developments that are occurring in health care. One such example is one of the most successful ISN SRCs, a partnership between Minsk and Oxford that focused on establishing a kidney transplant program in Belarus.⁶ A rapid and sustained growth in the transplant program has been achieved. The necessary government investment was forthcoming; without it, little would have been achieved. But that funding is unlikely to have been secured without the local leaders whose clinical and leadership skills were honed through the SRC. Nevertheless, in this example and others, there is a risk of overstating the direct impact of ISN when other partners are actively contributing.

Each SRC partnership identifies its own focus to maximize its chance to flourish; current SRCs are asked to undertake audits as part of their activities to help demonstrate the programs’ achievements. Therefore, although a more general evaluation of the SRC program may be measuring outcomes that have not been set by the local centers, the ISN will still be able to benchmark the type and amount of change at each center.

The Educational Ambassador program is the newest of ISN’s approaches to capacity building for renal health care and has had focused goals for each Educational Ambassador visit from the outset. This makes it easier to evaluate specific outcomes, and the focus of ISN’s evaluation of this program has been on exploring the range of the resulting health care developments. Although the specificity of the goals has made it more difficult to determine the overall success of the program, measures such as the excellent response rate and the very high rate of visits reported as “highly successful” are encouraging when taken in association with the descriptions of the clinical developments.

The Commission on Health Research for Development (1990)⁷ argues that research within local contexts is essential to develop health care appropriate to the needs of those in developing countries and that developing research capacity and international partnerships is key to this process. There is no definitive way to evaluate the effectiveness of developing research capacity,⁸ but Nchinda⁹ (2002) has drawn on several reviews of research capacity building in the 1980s and 1990s to provide guidelines on the criteria most associated with successful outcomes. These criteria include well-selected projects directed at national health priorities, carefully selected researchers, access to high-quality research training, and continuity of support by an experienced mentor, all components of the ISN Clinical Research program. Satisfaction with the Clinical Research program is generally high, and the program has had success, with 59% of the responding grant recipients publishing peer-reviewed papers. This is a particular success because it has been noted¹⁰ that, although interventions to develop research capacity may improve research proposals, this success may not translate into well-implemented projects or published results. It is, therefore, noteworthy that the ISN clinical research program has also included funding for a structured course in scientific writing. Supporting local research may help to promote status and reward those who stay in country, thus countering the “brain drain.” However, whether this translates into productive research careers for the recipients over the long term remains to be shown.

Least satisfactory are the available data assessing of the impact of the ISN through support of locally developed CME meetings. At almost all these meetings, ISN-supported speakers are only a minority of the faculty, and educational contact is brief, making it particularly challenging to evaluate the impact of ISN’s contribution. In addition, only a minority of delegates at each CME complete evaluation forms.

These limited survey responses report high satisfaction with the CME meetings and give a positive evaluation of didactic lectures, which typically dominate these CME programs. It would be a widely held educational view that active learning is more effective than traditional lectures. Therefore, case discussions and small group seminars are likely to add to the educational value. However, further developments must take account of the respondents’ reluctance to move away from didactic approaches; this may require exploration of the reasons for this reluctance and ways in which the ISN and other educating bodies can encourage new approaches to learning. Appreciation that ISN speakers have willingly come a long way to contribute a CME may make delegates less likely to provide critical feedback, which in some cultures may be seen as disrespectful. Furthermore, a measurable immediate assessment of the perceived educational benefit of a CME is less important than the real goals of improved patient care and sustained locally delivered educational programs.

The available evaluation also fails to capture fully the benefits of ISN support for a CME event. ISN-supported speakers often give added value through teaching rounds at local hospitals. They also take opportunities to meet local health care leaders and emphasize the opportunities to build capacity and improve care for individuals with kidney disease.

Evaluating the impact of the ISN’s 5 educational programs is important to direct the ISN’s sustained work in this field, to develop the processes within each program, and to influence and justify funding decisions. However, it is not straightforward to evaluate such complex interventions, and in hindsight, several weaknesses have come to light. Although we have stressed the desire to minimize the “brain drain” among those we support, we have not systematically collected data to judge our performance in this area. We must also acknowledge the limitations of our conclusions from the moderate response rates achieved in some programs. To identify and address the weaknesses in the program, we need to seek out the nonresponders or find alternative sources of information such as local and training center mentors. Finally, although the ISN seeks to deliver effective training cost-efficiently, these evaluations have not included a financial analysis. The challenge is therefore to create appropriate evaluation systems that are not so onerous that response rates decrease.

Evaluation systems for educational interventions in the developing world described in the literature focus on the clinical impact of “short-term medical missions” in the developing world, with a minor focus on educational impact¹¹ or on the impact of a faculty development program for teachers¹² and capacity building within medical education.¹³

Although educational programs can improve health care, educational research is also needed to make a proper determination of which educational methods being used are “efficiently effective.” Conventional measures of education quality from resource-rich countries are not extensively evidence-based and so should be regarded with skepticism when seeking to judge the quality, effectiveness, and efficiency of medical education in LMICs.

We used the 4 levels of output proposed by Kirkpatrick⁴—reaction, learning, behavior, and results—to evaluate these programs.

Reaction focuses on the learners’ satisfaction with the experience, including their critique of the resources and teaching provided and their perception of their own changed attitudes and improved learning. Learning achieved is demonstrated in assessment results, although demonstrating differential learning gain is complex. The evaluation of the ISN programs also includes data on Kirkpatrick’s level 3 and 4 outcomes, those related to changed behavior and the results (i.e., changes in health care delivery and outcomes).

An alternative strategy might be to use the CIPP framework,¹⁴ which focuses on the context as well as the input, process, and product. The emphasis on the differing contexts in LMICs¹⁴ is apposite and would have steered us to evaluate and log the needs, problems, and opportunities at the outset, using these to define suitable outcomes such as the location of fellows in the short- and longer term after the fellowship. Evaluation of the input should lead to an options appraisal before launching the favored intervention. The product evaluation can include any of the goals delineated by Kirkpatrick’s framework, although a key feature of CIPP’s framework is that the delineated products or outcomes must be based on the contextual evaluation. In other words, the outcomes should be prospectively agreed on based on the identified needs and goals. Finally, the inclusion of process evaluation is a reminder of the need to explore how the intervention has created change as well as explaining any unintended consequences.

The UK Medical Research Council guidance on developing and evaluating complex interventions¹⁵ has much in common with the CIPP framework, although the iterative steps in the development and evaluation of the intervention are termed development, feasibility and piloting, evaluation (process, product, cost), and longer term implementation and dissemination. The Medical Research Council guidance emphasizes the strength of experimental designs but recognizes that this is not always possible, especially when randomization is not feasible or acceptable. Focusing on a few predetermined outcomes is recommended for ease of analysis, and although it is recognized that process evaluation can shed light on how an intervention may be exerting its effect, this is no substitute for evaluation of the outcomes; encouraging process findings may not be reflected in the achievement of the desired outcomes.¹⁶ Craig et al.¹⁵ acknowledge the complexity of many health care and clinical education interventions and, accepting that they cannot always be completely standardized, they encourage researchers to describe their interventions and variations as fully as possible to permit others the opportunity to make judgments about the transferability of the findings and to emulate their activity.

Conclusion

We remain confident that the ISN programs are beneficial and regard the evidence that we have accumulated, despite its limitations, as supporting that confidence. Nevertheless, we do not rest on our laurels. Continuing work is required to design evaluation of the early stages of our capacity-building programs in LMICs to ensure that we can direct funding to the most effective programs and/or make improvements to others.

To do this well may require additional funding, and we must seek a balance between poorly evidenced educational programs and poorly funded educational programs impoverished by complex evaluative procedures.

Disclosure

DH has received a lecture fee from Roche Myanmar. MT received grant support from Abbvie and the Canadian Institutes of Health Research. All the other authors declared no competing interests.