Does research through Structured Operational Research and Training (SORT IT) courses impact policy and practice?

Abstract

Setting: Structured Operational Research and Training Initiative (SORT IT) courses are well known for their output, with nearly 90% of participants completing the course and publishing in scientific journals.

Objective: We assessed the impact of research papers on policy and practice that resulted from six SORT IT courses initiated between July 2012 and March 2013.

Design: This was a cross-sectional study involving e-mail-based, self-administered questionnaires and telephone/skype/in-person responses from first and/or senior co-authors of course papers. A descriptive content analysis of the responses was performed and categorised into themes.

Results: Of 72 participants, 63 (88%) completed the course. Course output included 81 submitted papers, of which 76 (94%) were published. Of the 81 papers assessed, 45 (55%) contributed to a change in policy and/or practice: 29 contributed to government policy/practice change (20 at national, 4 at subnational and 5 at hospital level), 11 to non-government organisational policy change and 5 to reinforcing existing policy. The changes ranged from modifications of monitoring and evaluation tools, to redrafting of national guidelines, to scaling up existing policies.

Conclusion: More than half of the SORT IT course papers contributed to a change in policy and/or practice. Future assessments should include more robust and independent verification of the reported change(s) with all stakeholders.

Avital element in assessing the success of operational research (OR) is to measure its impact on policy and/or practice.¹ The International Union Against Tuberculosis and Lung Disease (The Union) and Médecins Sans Frontières (MSF) have developed an OR training model that is well known for its output, with nearly 90% of participants successfully completing the courses and publishing in peer-reviewed scientific journals.² Both The Union and MSF are members of the Structured Operational Research and Training Initiative (SORT IT), a global partnership led by the Special Programme for Research and Training in Tropical Diseases (TDR) at the World Health Organization (WHO). The Union/MSF courses are accredited as SORT IT courses.

Two follow-up studies published in 2014 assessed the impact of the SORT IT courses: 74% of course papers had contributed to changes in policy and practice, and participants from the first eight SORT IT courses had continued to conduct and publish OR after the course.^3,4 A limitation of the studies was that the follow-up period was variable, ranging from 9 months to 3 years after course completion.³ Since then, we have been more systematic, and we now contact participants 18 months after course completion to assess the impact of their OR on policy and/or practice.

In this paper, we describe the self-reported impact of SORT IT course papers on policy and practice.

METHODS

This was a cross-sectional mixed-methods study with both quantitative and qualitative components. This involved e-mail-based, self-administered questionnaires and telephone/Skype/in-person responses from first and/or senior co-authors. The study population included all participants of six SORT IT courses initiated between July 2012 and March 2013 and first authors and/or senior co-authors of viewpoints that were written in conjunction with the courses. The courses were run in Chennai, India (n = 1); Paris, France (n = 1); Luxembourg (n = 1); Addis Ababa, Ethiopia (n = 1); Nadi, Fiji (n = 1); and Kathmandu, Nepal (n = 1).

The questionnaire was sent to course participants as part of routine follow-up about 18 months after course completion between December 2014 and March 2015; the respondents were given 3 months to respond, as previously described.³ One question asked whether their course-related research contributed to changes in policy and/or practice. If the answer was ‘Yes’, participants were asked to provide details; if the answer was ‘No’, participants were asked to provide reasons. Where responses were unclear, the first author and/or one of the senior co-authors of the paper (who were also mentors in the course) were contacted via e-mail, telephone and/or skype to obtain clarification. The verbal responses over Skype/telephone/in-person were transcribed in an e-mail and validated for their accuracy by the respondents. This constituted validation of responses by the respondents.

In case of viewpoints led by course faculty, either the first author or one of the senior co-authors was contacted to find out if the work (usually shared between faculty and participants) contributed to policy change. For every viewpoint, three of the authors (AK, RZ and ADH) debated whether it had impacted policy and arrived at a consensus.

Claims of policy change were verified by referring to policy documents or minutes of meetings available in the public domain or by contacting the first authors of the paper or participants where possible.

Data entry and analysis

The data were entered into an MS Excel file (Microsoft, Redmond, WA, USA) and a descriptive content analysis of responses was performed. The e-mail responses satisfied the criterion of ‘low inference descriptors’, as participants did their own transcribing and the responses were used directly for content analysis.⁵ Because of the small data set, content was analysed manually by three authors trained in qualitative research methods. Responses were classified into the following broad themes: changes in government policy and/or practice (subdivided into national, subnational and hospital level, depending on the level of impact) and changes in non-government organisation (NGO) policy and/or practice (when decisions were made by NGOs such as the MSF or The Union).⁴ Responses were categorised into themes by three authors (AK, HS, JP) together to enhance interpretive credibility. Any disagreement between the three authors was resolved by consensus after consultation with two senior authors (ADH and RZ). A conservative stance was adopted in assessments: when in doubt, the study was deemed not to have impacted policy/practice.

The Ethics Advisory Group of The Union, Paris, France, determined that neither ethics clearance nor participant informed consent were required for this type of study.

RESULTS AND DISCUSSION

Of 72 (31 female) participants enrolled in the six courses, 63 (88%) successfully completed the course and submitted one or more papers to a scientific journal. The participants came from 36 countries (34 in Asia [20 from India], 24 in Africa, 12 in the South Pacific and one each from Latin America and Europe). The majority of the participants were medical doctors (n = 48), followed by research officers (n = 6), nurses (n = 3) and other paramedical staff, including Monitoring & Evaluation Officers (n = 6). Most were working for governments (n = 28), followed by NGOs (n = 24), academia (n = 10) and donors (n = 1).

Output: publications and impact on policy/practice

A total of 81 papers (including nine viewpoints) were submitted for publication, of which 76 (94%) were published by 30 June 2015. The topics covered included tuberculosis (TB) (47%), human immunodeficiency virus (15%), non-communicable diseases (10%), OR (10%), maternal and child health (7%) and others (tobacco control, neglected tropical diseases, health system issues). All of the 72 research papers were led by course participants, of which 67 (93%) were published. This is in line with previously published reports of high publication output and might be related to strict selection criteria, adherence to timelines and milestones, coupled with strong hands-on mentorship offered by the programme faculty.^3,4,6 Of the 9 viewpoints, 2 were led by course participants, 1 by a former SORT IT participant and current faculty, 1 by the course administrator, and the rest were led by the course faculty; all were published. Very few capacity-building programmes track for output, although this is fast changing,⁷ and SORT IT creates a benchmark in this area.⁸

All study participants responded to the questionnaire survey. Of the 81 papers assessed, 45 (55%) were reported to have contributed to a change in policy and/or practice (Figure). This is encouraging, and can be attributed to several factors, including the policy relevance of the research question, which was often based on constraints faced by the programmes; the involvement of policy makers as co-investigators; ownership of the results, especially when programme managers were principal investigators of the research project; and other windows of opportunity available to the individual researchers by virtue of their being in national/state-level committees.

FIGURE.

Output from six completed SORT IT operational research courses run in Europe, Africa, Asia and the South Pacific during 2012–2013.* * The six courses were run in Chennai, India (n = 1); Paris, France (n = 1); Luxembourg (n = 1); Addis Ababa, Ethiopia (n = 1); Nadi, Fiji (n = 1) and Kathmandu, Nepal (n = 1). Output was assessed as of 30 June 2015. SORT IT = Structured Operational Research and Training Initiative; NGO = non-governmental organisation.

Nevertheless, output was lower than in the previous study, which reported that nearly three quarters of the studies had impacted policy.⁴ We have identified three possible reasons for this: 1) the increased rigour and conservative stance in our assessments; 2) a change in denominator—we used ‘all submitted papers’ as the denominator in this study, as compared to ‘published studies’ in the previous assessment (this amendment was made to reflect SORT IT course targets and our belief that policy change can happen before publication, and even without publication); and 3) the varied profile of course participants: a smaller proportion of course participants had senior posts, e.g., decision makers and national programme managers, in the current cohort than in previous cohorts, and they were therefore less likely/able to act on research findings.

Of the 45 papers that had an impact, 29 contributed to a change in government policy or practice—20 at the national level, 4 at the subnational level and 5 at the hospital level, 11 contributed to a change in NGO policy or practice; and 5 added additional evidence to existing policies and helped in making national scale-up decisions. The changes included taking new policy decisions, redrafting national technical and operational guidelines, and scaling up existing policies to changes in recording and reporting, leading to improved programme monitoring. Some projects had an impact on policy even before publication, perhaps indicating the ownership of results by policy makers participating in the study.

Changes in government policy/practice

Changes in national policy/practice

Three papers from the India SORT IT course generated evidence to support the shift from two sputum samples to one sputum sample during follow-up sputum microscopy among drug-susceptible TB patients.^9–11

Yes, there has been a national policy change to switch to one specimen from two specimens during follow-up sputum microscopy. (Participant from India)

This was verified by referring to the policy documents and government orders issued by the Revised National TB Control Programme in India. Two participants from India mentioned changes in national guidelines for screening malnourished children for TB:^12,13

Yes, the new strategy document being published for the elimination of TB 2020 has taken into account the screening of all undernourished children in NRCs (nutritional rehabilitation centres).

A participant from Mongolia reported that her research had led to the redrafting of national guidelines for multidrug-resistant TB (MDR-TB).¹⁴

Changes in subnational policy/practice change

Some studies contributed to improvements in record keeping, leading to improved programme implementation at the district level. A participant from Gazeera State, Sudan, reported that her study had led to the modification of monitoring and evaluation tools, including the introduction of some new variables in the asthma card and register, and regular monitoring for completeness of record keeping.¹⁵ A study from Sri Lanka had led to the creation of a mid-level programme officer post at subdistrict level.¹⁶

Change in hospital-level policy/practice

There were several instances of change in policy at the level of the hospital or institution. A study from China had led to the prolongation of the treatment of TB patients with co-existing diabetes mellitus in their hospital.¹⁷ In a tertiary hospital in Kenya, management decided to improve TB care among health workers by providing a dedicated space for treatment, access to new diagnostics and more budgetary allocation for care.¹⁸ In another experience from Rwanda, the participant reported the following:¹⁹

A few things that we learned from this evaluation have impacted our newest offering [of research workshops]: 1) clinical staff were less likely to complete, we believe due to difficulties leaving clinical duties if in the middle of consultations/procedures. Solution: we developed online modules to complement the course. Individuals who missed could make up sessions (up to two out of the 10) using these online modules. 2) Few women enrolled, and were less likely to complete. Solution: we allocated 50% of training slots for women, and online modules added flexibility for their competing demands. 3) Assessment targets were not set. The course was changed to require that participants had an average of >80% on all assessments to graduate.

Change in non-government organisational policy/practice

A study from Somalia by MSF on telemedicine led to changes in organisational policy for its wider application in MSF-assisted health care facilities in rural Kenya and Syria.²⁰ A participant from Ukraine referring to her study reported the following:²¹

This research has shown relatively low levels of HIV testing and re-testing among most at-risk populations. A new testing mechanism called ‘self-assisted testing’ was developed and introduced in 2015 by HIV/AIDS Alliance Ukraine.

Viewpoints were also reported to have led to policy change, particularly at the organisational level. A paper on open access publications led MSF to take an institutional stance.²² As quoted by the lead author of the author in an e-mail:

MSF has taken an institutional position to support Open Access. Funds have been made available as part of routine country level budgets to cover open access costs. This applies for all SORT IT course publications and also those from routine operational projects.

Another viewpoint helped to formalise changes to the structure and content of SORT IT courses run by The Union and MSF.²³

Lack of policy change and reasons

Most of the participants (n = 22) who reported ‘no policy change’ did not elaborate on the reasons. Those mentioned by a few participants (n = 14) could be classified into the following themes: lack of priority by the national programmes and institutions, need for more time and more evidence for policy change, turnover of decision makers in the ministry and ineffective dissemination. One participant from the South Pacific region mentioned that his research was not policy-relevant and he did not expect any policy change to occur as a result. This underlines the importance of choosing the right research question from a policy perspective. According to a participant from China:²⁴

No, one reason is that maybe the voice did not reach out to those policy makers enough, another reason is that our policy makers have been focusing on setting up a new standard treatment procedure in the health insurance system which will regulate clinical practice and thus solve the problem brought out by my paper.

As mentioned previously, some course participants were conservative in their assessment and mentioned a ‘no’ despite some evidence of impact.

No, not sure about the direct effect on policy and/or practice… But being one of the first large cohort studies among the injection drug users who attended methadone maintenance programme (MMP) in China, our research added strong and important evidence to the effectiveness of health education in the MMP.²⁵ (Participant from China)

One participant from India attributed the lack of policy change to his inability to actively follow up with the national programme:²⁶

No. Possible reasons: the findings of my research work were not perceived as a priority issue under the national programme. Another reason, I didn't pursue it further as I am not working directly with the programme.

One of the viewpoints called for uniformity in referencing styles and in particular criticised the PLOS journals and the Journal of Tropical Medicine and International Health for their challenging reference styles in the paper.²⁷ Although these journals have now changed their referencing styles to align them with the generic Vancouver style, the authors of the viewpoint were not sure if it was due to their paper. They thus took a conservative stance and decided that their viewpoint did not contribute to policy change.

Strengths and limitations

This study is one of the few attempts to systematically track the effects of OR on policy and practice.⁴ We believe this type of follow-up should be routine, to minimise creation of ‘research waste’ (i.e., research that stops at publication and does not benefit the populations).^28,29 We also call upon journals and researchers to operationalise this process and track their publications for policy impact.³⁰ We had a 100% response rate, and clarification was sought whenever responses were ambiguous, thus achieving participant validation. Three authors analysed the content, and disagreements were resolved in consultation with two senior authors. This helped to minimise subjectivity in assessments and enhance interpretive credibility. An important limitation is the self-reporting of impact by the authors of the papers. This is especially important in the case of viewpoints led by course faculty who are also involved in assessing the policy impact in the current paper. While we have tried to mitigate this limitation by verifying official policy documents wherever possible and by adopting a conservative stance in our assessments, responder bias cannot be completely ruled out. It is also important to note that these assessments were for changes in policy and practice, and the further step of determining whether the changes improved the health of the population was not part of this study. Finally, we highlight two important challenges below which may be of use in future assessments.

Challenges in evidence-policy linkage

While many of the SORT IT studies and viewpoints appear to have contributed to policy change, it was challenging to find documentary evidence linking the evidence to policy decision. This is one of the limitations of self-reported data. A participant from Swaziland mentioned that the decision on policy would have been made based on analysis of routine data, irrespective of its publication in a journal, although publication increased the credibility of the findings:³¹

This is a really an issue of how literally you expect respondents to take the question. If I answer the question as it was phrased, then the answer is no—it was not the study per se that led to policy change, but rather the use of routine data to feed the advocacy efforts of MSF and Clinton Health Access Initiative (Kigali, Rwanda)—this would have happened regardless of the study. However, it could be argued that the fact that we intended to write up the routine data analysis for publication meant that we paid increased attention to data quality, which made our analysis more reliable, thus improving our advocacy work. In this way, the study could be argued to have had an impact on policy. I guess you know what the question was trying to get at—if you feel that my answer constitutes a ‘yes’, then that is fine by me.

In line with our conservative approach to assessment, we took this as a ‘no’. The response from a participant from Nepal also highlighted the difficulty in attributing a link between evidence and challenge of evidence of policy change:³²

Yes, practice changed to screen all retreatment cases and expand Xpert… I am not sure this was because of the study and the paper published, but there is improvement in the situation, as this was recommended in the paper. The policy has been changed to screen all retreatment TB patients for MDR-TB as much as possible and the Xpert service has been expanded to more places.

Challenges related to varied interpretations

Different participants interpreted the question of impact differently, leading to varied interpretations of the effects of their research on policy. Five participants responded that their studies provided additional evidence for existing policies, which led to national scale-up decisions. One example was from India, where the study provided additional evidence to scale up existing policy: the use of light emitting diode fluorescence microscopy (LED-FM) in high-workload settings in India.³³ According to another participant from Bangladesh:³⁴

Yes. It had a great effect on policy and practice. The Bangladesh NTP [National Tuberculosis Programme] gave permission to extend the programme on a larger scale.

In contrast, some participants felt that their study did not impact change in policy/practice, although it did support an existing change. For example, a participant from Zimbabwe interpreted the same situation as not having impacted policy change:³⁵

No, it provided local evidence to scale up integration of antiretroviral therapy in antenatal clinics, which was already taking place in Zimbabwe.

In another case, a participant from India mentioned that his study had impacted policy change, although the study simply reinforced existing policy and did not change practice.^36,37 On further clarification and discussion by e-mail and telephone, he gave the following explanation:

Our study findings called for a need to revisit the WHO recommendation of switching to same-day diagnosis [in TB]. With the dissemination of this study result, the National TB Control Programme in India took a policy decision not to change its current policy as per WHO recommendation.

We agreed with this reasoning and considered the study to have had an impact on policy.

In conclusion, over half of course output resulting from recent SORT IT OR courses has contributed to a change in policy and/or practice. Given the complexities in interpreting policy impact assessment, future assessments should focus beyond self-reporting of data and use more robust and independent verification of the reported change(s) with all concerned stakeholders to enhance the richness and rigour of the assessments.