Skip to main content
PMC home page
Implementation Science : IS logoLink to Implementation Science : IS
. 2011 May 27;6:53. doi: 10.1186/1748-5908-6-53

Still too little qualitative research to shed light on results from reviews of effectiveness trials: A case study of a Cochrane review on the use of lay health workers

Claire Glenton 1,✉, Simon Lewin 2, Inger B Scheel 1
PMCID: PMC3117743  PMID: 21619645

Abstract

Background

Qualitative research is used increasingly alongside trials of complex interventions to explore processes, contextual factors, or intervention characteristics that may have influenced trial outcomes. Qualitative research conducted alongside trials can also be used to shed light on the results of systematic reviews of effectiveness by looking for factors that can help explain heterogeneous results across trials. In a Cochrane review on the effects of using lay health workers on maternal and child health and infectious disease control, we identified 82 trials. These trials showed promising benefits but results were heterogeneous.

Objective

To use qualitative studies conducted alongside these trials to explore factors and processes that might have influenced intervention outcomes.

Methods

We attempted to identify qualitative research carried out alongside the trials by contacting trial authors, checking papers for references to qualitative research, searching Pubmed for related studies, and carrying out citation searches. For those qualitative studies that we included, we extracted information regarding study objective, data collection and analysis methods, and key themes and categories.

Results

For 52 (63%) of the trials, we found no qualitative research that had been conducted alongside the trials. For 16 (20%) trials, some form of qualitative data collection had been done but was unavailable or had been done before the trial. For 14 (17%) trials, qualitative research had been done during or shortly after the trial, although descriptions of qualitative methods and results were often sparse. Most of these 14 studies aimed to elicit trial participants' perspectives and experiences of the intervention. A common theme was participants' appreciation of the lay health workers' shared circumstances, for instance with regard to social background or experience of the health condition. In six studies, researchers explored the experiences of the lay health workers themselves. Issues included the importance of regular supervision and health professionals' support or lack of support.

Conclusions

Qualitative studies carried out alongside trials of complex interventions could offer opportunities to authors of systematic reviews of effectiveness wishing to understand the heterogeneity of trial results. For interventions of lay health worker programmes at least, too few such studies exist at present for these opportunities to be realised.

Background

Interventions that aim to improve the organisation and delivery of healthcare often involve complex socio-behavioural processes, and are frequently 'made up of various interconnecting parts' [1] that act both 'independently and inter-dependently' [2,3], and that may be highly context-dependent [4]. There is growing acknowledgement of the contribution that qualitative research can make to both the development and evaluation of these complex interventions, and randomised trials of such interventions are increasingly including qualitative components [5,6]. Qualitative research can be used prior to a trial of a health system intervention to increase the quality and relevance of the intervention and to help select relevant outcomes, but can also be used during or after a trial to explore processes, contextual factors or intervention characteristics that may have influenced the trial results. In principle, qualitative research can also be used to shed light on the findings of systematic reviews of the effectiveness of health system interventions by looking for processes and other factors that could help explain homogeneous or heterogeneous results across trials or that could suggest new sub-group analyses for reviews.

In a recent Cochrane review on the effects of using lay health workers for maternal and child health and infectious disease control [7], we identified 82 randomised trials. These trials showed promising benefits in a number of areas, including in the use of lay health worker programmes to increase breastfeeding and childhood immunization. However, the results within these subgroups were heterogeneous. We wished to explore this heterogeneity by reviewing whether qualitative research conducted alongside these trials could increase our understanding of the processes that took place in these trials as well as contextual factors potentially influencing the outcomes of the intervention.

Objective

Our objective was to use qualitative studies conducted alongside randomised trials of lay health worker programmes included in a Cochrane review to explore the factors and processes that might have influenced the outcomes of these programmes.

Methods

We attempted to identify published and unpublished qualitative research carried out alongside the trials included in the Cochrane review. We defined a qualitative study as any study that used qualitative methods for data collection and analysis. We contacted the authors of the 82 trials, asking if any such research had taken place. For the 26 trials where no response was forthcoming, one researcher (CG) checked the main text and the reference list of each trial for descriptions of, or references to, related qualitative research; located each trial in Pubmed and searched for related studies and for other studies published by the same authors; and located each trial in the Science and Social Science Citation Index and checked the list of studies that had cited this paper. The same researcher then assessed full versions of potential papers to determine whether they were related to the trial and whether they had used qualitative research methods. For those studies that were included, we extracted information regarding the objective of the qualitative study, the methods of data collection and analysis used, and the key themes and categories identified.

Results

Fifty-two (63%) of the 82 trials had no qualitative research linked to them. For ten (12%) of the trials, some form of qualitative data collection was referred to briefly in the paper or in emails from authors, but was unavailable. At least half of this research appeared to have been done before the trial in order to develop the intervention. For a further six (7%) of the trials, qualitative research had been carried out before the trial and was available as either published or unpublished reports. The aim of these studies was to help develop the intervention by exploring the study population's health knowledge and behaviour, factors that influenced this behaviour, experiences of illness and healthcare, or healthcare needs. While these studies may have been important to the development of the trialed intervention, they did not allow us to explore directly the processes or other factors that may have influenced the outcomes of the trials and were therefore not explored further (See also Figure 1).

Figure 1.

Figure 1

Open in a new tab

Flow chart.

For 14 (17%) of the 82 trials [8-21], qualitative data collection had been carried out during or shortly after the trial, or, in one case, after the pilot study for the trial (See Figure 2 and Figure 3 for examples). For four trials [10,12,16,19], these data were presented in the same paper as the trial, while for one trial, these data were presented both in the same paper and in a separate paper [10,22]. For the remaining ten trials [8,9,11,13-15,17,18,20,21], qualitative data were presented separately, and in most cases published [23-32] and also cross-referenced with the trial publications. Descriptions of qualitative methods and results were often sparse, particularly for six of the studies [12,16,23-25,31] where authors offered little or no information about data collection methods and/or data analysis. In at least four of these six cases, the qualitative data were not the only focus of the paper.

Figure 2.

Figure 2

Open in a new tab

Example of a qualitative study carried out alongside a randomised trial: lay health workers for families of children with cerebral palsy (Adapted from Weindling 2007 [19]).

Figure 3.

Figure 3

Open in a new tab

Example of a qualitative study carried out alongside a randomised trial: lay health workers for people with tuberculosis (Adapted from Clarke et al 2005 [17]and Daniels et al 2005 [29]).

In these 14 trials, lay health worker programmes had been used to support women with poor pregnancy outcomes or families with sick children, to promote breastfeeding, to improve tuberculosis-related outcomes, to reduce child mortality and morbidity, and to prevent child injuries in the home. The trials were conducted in the USA (five studies), UK (three studies), South Africa (two studies), Bangladesh (two studies), Ghana and Nepal, and generally made use of lay health workers who were local to the setting and who had been selected on the basis of their similarity to the trial participants, for instance with regard to illness experiences.

The qualitative studies either looked at the perspectives of trial participants (eight studies); lay health workers (one study); or both (five studies). A common theme among trial participants was their appreciation of the similarities between them and the lay health workers, for example with regard to social background or because of first-hand experience of the health behaviour in question (breastfeeding) or the health condition (children with a particular illness). These similarities represented to participants an opportunity for emotional support as lay health workers similar to them were seen as being more accepting of participants' thoughts and actions [22]. These similarities were also seen as a source of practical support as these lay health workers would 'know all the pitfalls' [19]. One of the studies describes how participants who did not find the lay health worker programme helpful often pointed to factors associated with a lack of 'perceived sameness,' for example because of differences between lay health workers and study participants regarding illness experiences or preferences and values [22]. Participants across studies also described a number of other characteristics they regarded as important for a lay health worker, including patience and persistence, compassion and tolerance, accessibility, knowledge and common sense.

The shared experiences of the lay health workers and the trial participants were also valued by lay health workers in these studies. In addition, the lay health workers highlighted other issues including the importance of regular supervision and their experiences of support, or lack of support, from health professionals and the community in which they were based. One study of South African farm dwellers' experiences of becoming lay health workers illustrates how the transition from peer to lay health worker, and the new relationships this created with project staff, farm owners, and health professionals, led to mistrust and criticism from their family and the community [29].

Discussion

Randomised trials are considered the most rigorous design for evaluating whether an intervention is effective. However, trials generally yield limited insights into intervention mechanisms [33], and other approaches are therefore needed to understand how the intervention was delivered and why it achieved the outcomes that it did, and indeed to assess whether the outcomes measured were the most appropriate ones [33]. These types of questions are particularly pertinent for interventions intending to change the organisation or delivery of healthcare, where a broader understanding of process is necessary if we are to understand the intervention's success or failure. For lay health worker programmes, the wider inclusion of qualitative research alongside the trials would have allowed us to explore a number of factors that may have influenced programme outcomes. These include factors associated with the programme itself, such as how the lay health workers were selected and trained and their relationship with communities and with professional health workers; but also the broader context of the programme, such as political, social or cultural conditions.

Qualitative studies of lay health worker programmes can also be carried out independently of trials of interventions. Such studies have described a range of issues that may influence programme sustainability and success, including factors that affect lay health worker motivation and retention (for instance [34-37]). But our goal was to expand our understanding of the trial interventions included in the Cochrane review, to see if certain patterns would emerge that could help us to understand the heterogeneity of the review results. However, only 14 of the trials had carried out some form of qualitative data collection during or after the intervention. These data suggest that perceived similarities between trial participants and lay health workers are seen as important by these groups. The identification of factors such as these may offer a basis for subgroup analyses in the Cochrane review, and may help explain heterogeneity in trial results. In general, however, the data we identified was sparse, and methods and results were often poorly described, making our study aim difficult to achieve.

This work reflects findings from an earlier study, where we examined the use of qualitative approaches alongside randomized trials of complex health service interventions [38]. In a sample of 100 trials, only 30 had associated qualitative work, around one-half of which had been carried out before the trial. Factors that may influence whether qualitative studies are done alongside trials include the attitudes of funding bodies and the attitudes and skills of the research community [39]. When mixed methods are used, lack of time or experience as well as journal formats may prevent findings from qualitative studies and trials or reviews of effectiveness from being integrated or presented together [39]. The revision of formats for trial and review reporting is one way forward, and electronic publication now creates opportunities for publication of supplementary materials providing further detail regarding qualitative and other studies conducted alongside trials. Journals encouraging mixed methods will also, however, need to ensure that these papers receive appropriate peer reviewing. In addition, qualitative studies and trials that are reported separately need to be more clearly linked to one another to facilitate retrieval. All trials now require a universal trial reference number, and qualitative studies carried out alongside trials should utilize this number to facilitate linkage. Electronic publication databases could also utilise these reference number to show linked groups of studies when any one of the studies are retrieved.

Conclusion

Qualitative studies carried out alongside trials of complex health system interventions could offer insights into intervention mechanisms, and give authors of systematic reviews of effectiveness an opportunity to explore the reasons for heterogeneity among trial results [38,40]. For interventions involving lay health workers at least, too few such studies exist at present for these opportunities to be realised. Those conducting trials of lay health worker programmes should incorporate in-depth process evaluation, including qualitative analysis to explore the reasons for the outcomes of these complex interventions. Methodological and practical guidance may be needed for trial teams who plan to use qualitative approaches for this purpose.

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

CG, SL and IBS conceived of and designed the study. CG searched for and assessed the studies and drafted the manuscript. SL assessed papers where there was doubt regarding inclusion or allocation. CG drafted the paper and the other authors then contributed to this. All authors read and approved the final manuscript.

Contributor Information

Claire Glenton, Email: claire.glenton@nokc.no.

Simon Lewin, Email: simon.lewin@nokc.no.

Inger B Scheel, Email: inger.b.scheel@nokc.no.

Acknowledgements and Funding

This research was funded by the Norwegian Research Council.

References

  1. Campbell M, Fitzpatrick R, Haines A, Kinmonth A-L, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321:694–696. doi: 10.1136/bmj.321.7262.694. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Medical Research Council. A framework for development and evaluation of RCTs for complex interventions to improve health. London: MRC; 2000. [Google Scholar]
  3. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. doi: 10.1136/bmj.a1655. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Rychetnik L, Frommer M, Hawe P, Shiell A. Criteria for evaluating evidence on public health interventions. J Epidemiol Community Health. 2002;56:119–127. doi: 10.1136/jech.56.2.119. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Hawe P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. J Epidemiol Community Health. 2004;58:788–793. doi: 10.1136/jech.2003.014415. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Team RS. Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006;332:413–416. doi: 10.1136/bmj.332.7538.413. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Lewin S, Munabi-Babigumira S, Glenton C, Daniels K, Bosch-Capblanch X, van Wyk BE, Odgaard-Jensen J, Johansen M, Aja GN, Zwarenstein M, Scheel IB. Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. Cochrane Database Syst Rev. 2010;17:3CD004015. doi: 10.1002/14651858.CD004015.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Spencer B, Thomas H, Morris J. A randomized controlled trial of the provision of a social support service during pregnancy: the South Manchester Family Worker Project. BJOG. 1989;96(3):281–8. doi: 10.1111/j.1471-0528.1989.tb02387.x. [DOI] [PubMed] [Google Scholar]
  9. Ernst CC, Grant TM, Streissguth AP, Sampson PD. Intervention with high-risk alcohol and drug-abusing mothers: II. Three-year findings from the Seattle model of paraprofessional advocacy. J Community Psychol. 1999;27(1):19–38. doi: 10.1002/(SICI)1520-6629(199901)27:1<19::AID-JCOP2>3.0.CO;2-K. [DOI] [Google Scholar]
  10. Singer GHS, Marquis J, Powers LK, Blanchard L, Divenere N, Santelli B. et al. A multi-site evaluation of parent to parent programs for parents of children with disabilities. J Early Interv. 1999;22(3):217–29. doi: 10.1177/105381519902200305. [DOI] [Google Scholar]
  11. Haider R, Ashworth A, Kabir I, Huttly SR. Effect of community-based peer counsellors on exclusive breastfeeding practices in Dhaka, Bangladesh: a randomised controlled trial. Lancet. 2000;356(9242):1643–7. doi: 10.1016/S0140-6736(00)03159-7. [DOI] [PubMed] [Google Scholar]
  12. Malotte CK, Hollingshead JR, Larro M. Incentives vs outreach workers for latent tuberculosis treatment in drug users. Am J Prev Med. 2001;20(2):103–7. doi: 10.1016/S0749-3797(00)00283-X. [DOI] [PubMed] [Google Scholar]
  13. Kartin D, Grant TM, Streissguth AP, Sampson PD, Ernst CC. Three-year developmental outcomes in children with prenatal alcohol and drug exposure. Pediatric Physical Therapy. 2002;14(3):145–53. doi: 10.1097/00001577-200214030-00004. [DOI] [PubMed] [Google Scholar]
  14. Graffy J, Taylor J, Williams A, Eldridge S. Randomised controlled trial of support from volunteer counsellors for mothers considering breast feeding. BMJ. 2004;328(7430):26. doi: 10.1136/bmj.328.7430.26. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Manandhar DS, Osrin D, Shrestha BP, Mesko N, Morrison J, Tumbahangphe KM. et al. Effect of a participatory intervention with women's groups on birth outcomes in Nepal: cluster-randomised controlled trial. Lancet. 2004;364:970–9. doi: 10.1016/S0140-6736(04)17021-9. [DOI] [PubMed] [Google Scholar]
  16. Sullivan-Bolyai S, Grey M, Deatrick J, Gruppuso P, Giraitis P, Tamborlane W. Helping other mothers effectively work at raising young children with type 1 diabetes. Diabetes Educ. 2004;30(3):476–84. doi: 10.1177/014572170403000319. [DOI] [PubMed] [Google Scholar]
  17. Clarke M, Dick J, Zwarenstein M, Lombard CJ, Diwan VK. Lay health worker intervention with choice of DOT superior to standard TB care for farm dwellers in South Africa: a cluster randomised control trial. Int J Tuberc Lung Dis. 2005;9(6):673–9. [PubMed] [Google Scholar]
  18. Pence BW, Nyarko P, Phillips JF, Debpuur C. The effects of community nurses and health volunteers on child mortality: The Navrongo Community Health Family Planning Project. Pop Council. 2005;200:1–27. doi: 10.1080/14034940701349225. [DOI] [PubMed] [Google Scholar]
  19. Weindling AM, Cunningham CC, Glenn SM, Edwards RT, Reeves DJ. Additional therapy for young children with spastic cerebral palsy: a randomised controlled trial. Health Technol Assess. 2007;11(16):iii–iv. doi: 10.3310/hta11160. ix-x, 1-71. [DOI] [PubMed] [Google Scholar]
  20. Sloan NL, Ahmed S, Mitra SN, Choudhury N, Chowdhury M, Rob U. et al. Community-based kangaroo mother care to prevent neonatal and infant mortality: a randomized, controlled cluster trial. Pediatrics. 2008;121(5):e1047–59. doi: 10.1542/peds.2007-0076. [DOI] [PubMed] [Google Scholar]
  21. Swart L, van Niekerk A, Seedat M, Jordaan E. Paraprofessional home visitation program to prevent childhood unintentional injuries in low-income communities: a cluster randomized controlled trial. Inj Prev. 2008;14(3):164–9. doi: 10.1136/ip.2007.016832. [DOI] [PubMed] [Google Scholar]
  22. Ainbinder JG, Blanchard LW, Singer GH, Sullivan ME, Powers LK, Marquis JG, Santelli B. A qualitative study of Parent to Parent support for parents of children with special needs. J Pediatr Psychol. 1998;23(2):99–109. doi: 10.1093/jpepsy/23.2.99. [DOI] [PubMed] [Google Scholar]
  23. Spencer B, Morris J, Thomas H. The South Manchester family worker scheme. Health Promot. 1987;2(1):29–38. doi: 10.1093/heapro/2.1.29. [DOI] [Google Scholar]
  24. Grant T, Streissguth A, Ernst C. Benefits and challenges of paraprofessional advocacy. Zero to Three. 2002. pp. 14–20.
  25. Haider R, Kabir I, Huttly SR, Ashworth A. Training peer counsellors to promote and support exclusive breastfeeding in Bangladesh. J Hum Lact. 2002;18(1):7–12. doi: 10.1177/089033440201800102. [DOI] [PubMed] [Google Scholar]
  26. Graffy J, Taylor J. What information, advice, and support do women want with breastfeeding? 2005. pp. 179–86. [DOI] [PubMed]
  27. Mesko N, Osrin D, Tamang S, Shrestha BP, Manandhar DS, Manandhar M, Standing H, Costello AM. Care for perinatal illness in rural Nepal: a descriptive study with cross-sectional and qualitative components. BMC Int Health Hum Rights. 2003;3:3. doi: 10.1186/1472-698X-3-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Morrison J, Tamang S, Mesko N, Osrin D, Shrestha B, Manandhar M, Manadhar D, Standing H, Costello A. Women's health groups to improve perinatal care in rural Nepal. BMC Pregnancy Childbirth. 2005;5:6. doi: 10.1186/1471-2393-5-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Daniels K, Van Zyl HH, Clarke M, Dick J, Johansson E. Ear to the ground: listening to farm dwellers talk about the experience of becoming lay health workers. Health Policy. 2005;73(1):92–103. doi: 10.1016/j.healthpol.2004.10.006. [DOI] [PubMed] [Google Scholar]
  30. Clarke M, Dick J, van Zyl H, Johansson E. Farmers' perceptions of the lay health worker on farms in the Western Cape, South Africa. Agrekon. 2004;43(4):465–483. doi: 10.1080/03031853.2004.9523661. [DOI] [Google Scholar]
  31. Quasem I, Sloan NL, Chowdhury A, Ahmed S, Winikoff B, Chowdhury AMR. Adaptation of Kangaroo Mother Care for Community-Based Application. J Perinatol. 2003;23:646–651. doi: 10.1038/sj.jp.7210999. [DOI] [PubMed] [Google Scholar]
  32. Odendaal WA, Marais S, Munro S, van Niekerk A. When the trivial becomes meaningful: Reflections on a process evaluation of a home visitation programme in South Africa. Eval Program Plann. 2008;31:209–216. doi: 10.1016/j.evalprogplan.2008.02.006. [DOI] [PubMed] [Google Scholar]
  33. Kane SS, Gerretsen B, Scherpbier R, Dal Poz M, Dieleman M. A realist synthesis of randomised control trials involving use of community health workers for delivering child health interventions in low and middle income countries. BMC Health Serv Res. 2010;10(1):286. doi: 10.1186/1472-6963-10-286. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Pope C, van Royen P, Baker R. Qualitative methods in research on healthcare quality. Qual Saf Health Care. 2002;11:148–152. doi: 10.1136/qhc.11.2.148. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Khan SH, Chowdury AM, Karim F, Barua MK. Training and retaining Shasthyo Shebika: reasons for turnover of community health workers in Bangladesh. Health Care Superv. 1998;17(1):37–47. [PubMed] [Google Scholar]
  36. Landon B, Loudon J, Selle M, Doucette S. Factors influencing the retention and attrition of community health aides/practitioners in Alaska. J Rural Health. 2004;20(3):221–30. doi: 10.1111/j.1748-0361.2004.tb00032.x. [DOI] [PubMed] [Google Scholar]
  37. Excott S, Walley J. Listening to those on the frontline: lessons for community-based tuberculosis programmes from a qualitative study in Swaziland. Soc Sci Med. 2005;61(8):1701–10. doi: 10.1016/j.socscimed.2005.03.040. [DOI] [PubMed] [Google Scholar]
  38. Thomas C, Newell JN, Baral SC, Byanjankar L. The contribution of volunteers to a successful community-orientated tuberculosis treatment centre in an urban setting in Nepal: a qualitative assessment of volunteers' roles and motivations. J Health Organ Manag. 2007;21(6):554–72. doi: 10.1108/14777260710834346. [DOI] [PubMed] [Google Scholar]
  39. Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496. doi: 10.1136/bmj.b3496. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. O'Cathain A, Nicholl J, Murphy E. Structural issues affecting mixed methods studies in health research: a qualitative study. BMC Med Res Methodol. 2009;9(9):82. doi: 10.1186/1471-2288-9-82. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. May CR, Mair FS, Dowrick CF, Finch TL. Process evaluation for complex interventions in primary care: understanding trials using the normalization process model. BMC Fam Pract. 2007;8:42. doi: 10.1186/1471-2296-8-42. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Implementation Science : IS are provided here courtesy of BMC

RESOURCES