Synthesising quantitative and qualitative research in evidence‐based patient information

Abstract

Background

Systematic reviews have, in the past, focused on quantitative studies and clinical effectiveness, while excluding qualitative evidence. Qualitative research can inform evidence‐based practice independently of other research methodologies but methods for the synthesis of such data are currently evolving. Synthesising quantitative and qualitative research in a single review is an important methodological challenge.

Aims

This paper describes the review methods developed and the difficulties encountered during the process of updating a systematic review of evidence to inform guidelines for the content of patient information related to cervical screening.

Methods

Systematic searches of 12 electronic databases (January 1996 to July 2004) were conducted. Studies that evaluated the content of information provided to women about cervical screening or that addressed women's information needs were assessed for inclusion. A data extraction form and quality assessment criteria were developed from published resources. A non‐quantitative synthesis was conducted and a tabular evidence profile for each important outcome (eg “explain what the test involves”) was prepared. The overall quality of evidence for each outcome was then assessed using an approach published by the GRADE working group, which was adapted to suit the review questions and modified to include qualitative research evidence. Quantitative and qualitative studies were considered separately for every outcome.

Results

32 papers were included in the systematic review following data extraction and assessment of methodological quality. The review questions were best answered by evidence from a range of data sources. The inclusion of qualitative research, which was often highly relevant and specific to many components of the screening information materials, enabled the production of a set of recommendations that will directly affect policy within the NHS Cervical Screening Programme.

Conclusions

A practical example is provided of how quantitative and qualitative data sources might successfully be brought together and considered in one review.

The National Health Service Cervical Screening Programme (NHSCSP) first published evidence‐based guidelines addressing the content of screening letters and leaflets in 1997.¹The NHS Cancer Plan (September 2000) called for honest, comprehensive and understandable screening materials² and a White Paper (November 2004) emphasised the need for up‐to‐date and factual health information.³ An important NHSCSP priority is the continual improvement of the quality of written information sent to women about cervical screening. As such, a review was commissioned in early 2004 to develop updated NHSCSP patient‐information guidelines. The focus of the review was directed and informed by two main questions:

1. What is the existing research evidence base regarding the content of the written information sent to women at all stages of the cervical screening process?

2. What are the information needs of women during cervical screening?

The main purpose of the review was to identify important information for inclusion in the screening programme letters and leaflets (introductory, abnormal result and colposcopy). Important information included facts (length of the screening interval), concepts (purpose of the screening test) and general points (suitable attire). For each key information point, a tabular summary was created of the relevant evidence. The full updated guidelines are described elsewhere.⁴ A selection of the review recommendations is presented in fig 1. The reviewers were experienced in conducting quantitative and qualitative research related to cancer screening and patient and health professional information.

Figure 1 The review process (including a selection of the review recommendations).

Systematic reviews

Conventional systematic reviews aim to bring together and summarise the available research addressing a question of interest so that inferences about findings may be guided by the most appropriate forms of evidence.⁵,⁶ Evidence synthesis systems have, in the past, given preference to quantitative studies (mainly randomised trials) and clinical effectiveness. Accordingly, explicit methods have been established for the systematic review of trial‐based evidence including comprehensive literature searching, detailed quality appraisal procedures and standard synthesis techniques.⁷,⁸,⁹,¹⁰ Interest is growing in the development of diverse systematic review methods to incorporate different types of evidence including other quantitative study designs as well as qualitative research.¹¹ Methods for the synthesis of qualitative research are still in the early stages of proposal and evaluation,¹²,¹³ and the suitability of fitting various forms of qualitative research within the framework of conventional review methodology is an active research topic.¹¹ There is, in fact, still debate about whether the synthesis of qualitative evidence is appropriate and whether it is acceptable to combine qualitative studies derived from different traditions.¹³,¹⁴,¹⁵

Limitations of traditional hierarchies of evidence

Traditional evidence hierarchies were developed specifically to address questions of efficacy and effectiveness⁶ and involved assessing research according to study design with the randomised trial as the premier form of evidence.⁷,¹⁶ This becomes problematic in areas of research dominated by non‐trial quantitative evidence owing to the lower quality scores subsequently assigned to these studies.¹⁷ Also, evidence syntheses concentrating on wider issues such as “How does it work?” or “What are patients' experiences?” require a different approach because these types of research questions are best answered by evidence from a variety of sources.¹⁵,¹⁸,¹⁹,²⁰

Qualitative research

Qualitative evidence has a role to play in answering questions not easily evaluated by experimental studies.¹⁸,²⁰,²¹ Although qualitative studies do not establish probability estimates or effect sizes, they can provide important support for quantitative outcomes and identify patient priorities and concerns.¹² However, the usefulness of qualitative research is not limited to the explanation of process measures. In fact, qualitative research may address many healthcare questions directly and inform evidence‐based practice independently of other research methodologies.¹²,²¹,²²

Importance of a range of evidence

Policy‐makers and practitioners assessing complicated healthcare questions draw on diverse sources of evidence during the decision‐making process.¹³,¹⁴,²³ The development of methods for the synthesis of different types of research in a single review is an important methodological challenge. A number of approaches (narrative summary, thematic analysis, meta‐ethnography and bayesian methods) could be applied to the synthesis of both qualitative and quantitative research.¹²,¹³ However, most have evolved from techniques used for primary data analysis and were initially developed for either qualitative or quantitative data—not both.¹³ Several groups are currently working on methods for the incorporation of qualitative evidence into systematic reviews including the international Cochrane Qualitative Methods Group, the international Campbell Collaboration, the Evidence for Policy and Practice Information and Coordinating Centre in the UK and the Joanna Briggs Institute in Australia.²³ The UK Economic and Social Research Council has also funded a programme of research in this area.²³,²⁴

Box 1 First example of studies included in an updated review of evidence‐based guidelines for the content of NHSCSP letters and leaflets (the studies described were focused on colposcopy information)

Colposcopy information leaflets: what women want to know and when they want to receive this information²⁸

• Study design: The study was divided into two parts (A, B)

  • -Qualitative component (Part A) and
  • -Non‐comparative descriptive component (Part B)

• Study aims:

  • -To determine what information women want to receive about colposcopy and when they want to receive it
  • -To evaluate how this relates to the NHSCSP guidelines

• Population:

  • -Part A: A total of 42 women with abnormal Pap smear results attending a pre‐colposcopy counselling session at a UK cancer centre colposcopy clinic
  • -Part B: 100 consecutive women with abnormal Pap smear results newly referred to a UK cancer centre colposcopy clinic

• Data collection:

  • -Part A: Observation and documentation
  • -Part B: Self‐report via questionnaire

• Extracted information:

  • -Part A: A list of 38 questions asked by 50% or more of the women at the pre‐colposcopy counselling session (eg, what is an abnormal smear? or what is colposcopy?)
  • -Part B: Preferred timing of information delivery and information needs identified by women

• Notes:

  • -The third component of this study involved an assessment of information leaflets from 128 colposcopy clinics and is not described in this text box

Although methods of synthesising quantitative and qualitative research are beginning to be developed, few studies have been published that address the problem from a practical point of view.⁹,¹⁵,²⁵,²⁶ The current lack of specific guidance and accumulated experience in the synthesis of a range of research evidence leaves would‐be reviewers in a difficult position. The aim of this paper is to contribute to the debate by describing the review methods developed and difficulties encountered during the process of updating evidence‐based guidelines for the content of NHSCSP letters and leaflets.

Box 2 Second example of studies included in an updated review of evidence‐based guidelines for the content of NHSCSP letters and leaflets (the studies described were focused on colposcopy information)

An observational study of precolposcopy education sessions: what do women want to know?²⁹

• Study design:

  • -Qualitative study

• Study aims:

  • -To observe group counselling educational sessions for women about colposcopy held before the procedure
  • -To identify specific concerns about cervical cancer, the procedure of colposcopy and any longer term effects of the procedure

• Population:

  • -A total of 47 women with abnormal Pap smear results attending 1 of 5 precolposcopy group counselling educational sessions run by two specialist hospital colposcopy clinic nurses

• Data collection:

  • -Observation—participants' questions, comments and non‐verbal communication (eg, laughter or anxiety) were recorded verbatim

• Extracted information:

  • -Information needs identified by women

What is the evidence base?

The literature describing screening information content and women's information needs was dispersed among a variety of disciplines, and numerous research designs potentially addressed the review questions. Consequently, the review evidence base was complex and ill defined. This is not a unique problem, many healthcare questions are not easily evaluated solely by experimental methods (ie randomised trials) or other forms of quantitative investigation.¹²,¹⁵,¹⁷,²⁷ Boxes 1–4 provide examples of the range of data sources included in the review (the four studies described were focused on colposcopy information).

Methods

Literature searching

Systematic searches of 12 electronic databases (January 1996 to July 2004 inclusive) were conducted. During the search development, a “test” subset of relevant studies was used to assess whether thesaurus or free‐text terms could be included in isolation. A combination of both was required to identify all the “test” studies. As discussed by Shaw et al,³² the cost of designing a comprehensive search strategy is the lack of precision. Study design filters may help improve precision, however none were incorporated in this work because of concerns about poor indexing in electronic databases—particularly for qualitative studies.⁹,¹⁸,²⁰,³²,³³,³⁴ Instead, a core set of subject specific terms (cervical smears, colposcopy, screening and neoplasms) was used in combination with thesaurus and general free‐text terms such as “leaflet*”, “knowledge” or “understand*” to identify relevant studies.

Box 3 Third example of studies included in an updated review of evidence‐based guidelines for the content of NHSCSP letters and leaflets (the studies described were focused on colposcopy information)

Patient‐based evaluation of a colposcopy information leaflet.³⁰

• Study design:

  • -Non‐comparative descriptive study

• Study aims:

  • -To evaluate an information leaflet, “Prevention of cervical cancer,” routinely sent to women referred for colposcopy
  • -To use the findings of the evaluation to propose changes to the leaflet

• Population:

  • -A total of 137 women with abnormal Pap smear results newly referred to the Ninewells Hospital colposcopy clinic in Dundee, Scotland

• Data collection:

  • -Self‐report via questionnaire

• Extracted information:

  • -Information leaflet text (terms and language used)
  • -Information needs identified by women
  • -Satisfaction with information provided

• Notes:

  • -Text of information leaflet included with the study report

Additional references were taken from the table of contents of selected journals, reference sections of relevant studies, and the NHSCSP research literature online database (http://www.nhs.thescienceregistry.com/main.asp). Grey literature was sought from internet resources and contact with subject area specialists. In total, 1063 citations were identified (both published and unpublished, with no language restrictions).

Initial citation and full study report assessments

Titles and abstracts of citations were independently prescreened by two reviewers according to review study selection criteria (box 5). Others have noted that when present, the abstracts of qualitative studies vary in content (ie, study design details may not be reported).³²,³⁴ The inclusion or exclusion criteria were liberally applied and if doubt existed, individual studies were provisionally included for consideration on the basis of full text reports. Only about 3% of the screened citations lacked an abstract.

A total of 233 studies were retrieved for further evaluation. Two reviewers independently assessed the full study reports; any uncertainty was resolved by discussion. Most of the the exclusions fell into three categories: studies reporting interventions aimed at increasing screening uptake (35), non‐UK based studies investigating non‐generalisable cervical screening issues (41), and studies focused on aspects of cervical screening other than written information materials (51). The remaining 27 excluded studies belonged to several different categories listed in box 5. An important consideration for prospective reviewers is the time required to complete the search strategy development as well as the initial citation and full study report assessments. In this review, around 4 months of full‐time work was dedicated to these preliminary steps.

Box 4 Fourth example of studies included in an updated review of evidence‐based guidelines for the content of NHSCSP letters and leaflets (the studies described were focused on colposcopy information)

Is the provision of information leaflets before colposcopy beneficial? A prospective randomised study³¹

• Study design:

  • -Randomised controlled trial

• Study aims:

  • -To assess the usefulness of a leaflet sent to women before colposcopy in reducing anxiety

• Population:

  • -A total of 210 women diagnosed with moderate dyskaryosis or less newly referred to the North Staffordshire colposcopy clinic

• Intervention:

  • -Intervention group: information leaflet sent with clinic appointment letter
  • -Control group: clinic appointment letter only

• Data collection:

  • -Self‐report via questionnaire

• Extracted information:

  • -Information leaflet text (terms and language used)
  • -Information leaflet assessment (intervention group only)

• Notes:

  • -Text of information leaflet included with the study report

Data extraction

Data extraction was conducted for 79 studies (52 quantitative and 27 qualitative). A data extraction form was used to record full study details and guide decisions about the relevance of individual studies to the review questions. The form was developed using guidelines produced by the UK National Health Service Centre for Reviews and Dissemination (CRD)³ and other publications.³⁵,³⁶,³⁷ Similar information was extracted for all studies and included: aims, setting, population, research design, methods, interventions (if appropriate), results and conclusions. Data were extracted from relevant studies by one reviewer and checked by a second reviewer.

Identification of key information points

A core set of key information points for the screening programme letters and leaflets was developed from the 1997 guidelines¹ and the extracted data. All the information points from the original guidelines as well as new points identified from the findings of the data extracted studies were included in the core set. In total, 32 studies reported relevant results (the range of study designs is described in box 6). Although a large number of studies underwent data extraction and ultimately many did not report relevant findings, it was essential to consider a variety of potentially applicable studies to obtain a comprehensive set of key information points. The excluded studies fell into the following categories: studies reporting interventions aimed at increasing screening uptake (n = 3), non‐UK based studies investigating non‐generalisable cervical screening issues (n = 3), studies focused on aspects of cervical screening other than written information materials (n = 14) and studies containing information‐related findings not directly relevant to the agreed key information points (n = 20). A further seven studies (five quantitative and two qualitative) contained insufficient information about study design, methods of analysis and findings for a full assessment.

Box 5 Study selection criteria

Inclusion criteria

• Population:

  • -Women, 20–64 years

• Setting:

  • -Organised, systematic cervical screening programme (screening delivered in GP surgeries, community or hospital clinics)

• Research Questions:

  • -Assessment of the content of written information materials provided to women about cervical screening at all stages of the cervical screening process
  • -Investigation of the information needs of women at all stages of the cervical screening process

• Study designs:

  • -All (except for opinion pieces)

Exclusion criteria

1. Interventions focused on medical professional education, general practice performance and systems, cervical screening technology, protocols and technical aspects of treatment for cervical intra‐epithelial neoplasia and cervical cancer

2. Interventions that aimed to increase cervical screening uptake (except where the content of participant information materials was evaluated and/or included with the study report)

3. Studies reporting non‐information based predictors of screening uptake and risk factors for cervical cancer

4. Studies reporting knowledge, attitudes, health beliefs or barriers towards cervical screening without reference to written information materials or information needs

5. Non‐UK based studies investigating cervical screening issues not generalisable to the UK screening population and/or setting

6. Studies reporting insufficient information about study design, methods of analysis and findings for a full assessment

7. Studies reporting information‐related findings not directly relevant to the core set of key information points

Study methodology—quality scoring

Study quality appraisal is undertaken in quantitative systematic reviews to assess bias—for example, appropriate randomisation for trials—and to identify other study design specific defects.¹¹,³⁸ Many quality appraisal checklists are available for different types of quantitative evidence however, for study designs other than randomised trials, the key elements of quality are not as well agreed.³⁸ The issue of how or whether to assess qualitative studies is a matter of some debate.¹³,¹⁴,³³,³⁹,⁴⁰ The UK NHS CRD guidance favours structured appraisal of qualitative research but recognises that consensus is lacking on appropriate criteria.⁷

Although the exact function of quality appraisal in reviews of quantitative and qualitative evidence is controversial, it is recognised that reviewers should highlight evidence quality issues—in conduct, reporting, or both for review users.¹³ As such, the quality of each individual study included in the review was assessed by two reviewers using established checklists. Separate checklists were used for different quantitative study designs whereas a single checklist was developed for all qualitative studies (box 6).⁷,⁴¹,⁴²,⁴³,⁴⁴ All checklists incorporated a coded comments system that allowed reviewers to record an assessment of each checklist component. After consideration of all items on a given checklist, the methodological quality of each study was rated as: ++ (all or most of the criteria have been fulfilled), + (some of the criteria have been fulfilled) or – (few or no criteria have been fulfilled).

The quality appraisal provided an indication of the strengths and weaknesses of each study. Although study‐design flaws (eg, no intention‐to‐treat analysis conducted for a randomised trial) could prompt the exclusion of quantitative evidence, in practice, none was excluded at this stage. No exclusion rules were applied to the qualitative studies as guidance is lacking on how to exclude “weak” qualitative findings. The quality scores assigned to individual studies were taken into account during the research synthesis.

Research synthesis

The first step in the synthesis process was to construct a tabular summary of all studies related to each key piece of information identified as important for inclusion in the NHSCSP letters and leaflets. An example, the evidence profile created for the colposcopy leaflet information point “Indicate that treatment can occur at the first colposcopy clinic visit”, is presented in table 1 (a summary of the study findings is also included).

Box 6 Included studies and appraisal checklists.

Study design	Appraisal checklists
Randomised controlled trial (n = 3)	SIGN and CASP
Cluster randomised controlled trial (n = 1)	SIGN and CASP
Quasi‐randomised controlled trial (n = 3)	SIGN and CASP
Retrospective case control study (n = 1)	SIGN and CASP
Cross‐sectional study (n = 3)	SIGN and NZGG
Non‐comparative descriptive study (n = 7)	Reviewer checklist
Non‐comparative time series study (n = 1)	Reviewer checklist
Qualitative study (n = 13)	CASP and UKGCSRO

CASP, Critical appraisal skills programme⁴¹; SIGN, Scottish intercollegiate guidelines network⁴²; NZGG, New Zealand guidelines group⁴³; UKGCSRO, UK Government chief social researcher's office.⁴⁴

Reviewer checklist included: purpose, population, response rate, outcome definition and assessment.

Many different systems exist for grading quantitative evidence and recommendation strength,⁴⁵ however, there is no agreed hierarchy of evidence in qualitative research¹⁸ or across all research methods.¹³ Recently, a new system of grading quantitative evidence has been proposed by an international group of experts in the field of systematic reviews (the GRADE working group).⁴⁶ For the purposes of this review, a modified GRADE approach was adopted for the quantitative evidence synthesis and a similar but separate system was developed for the qualitative research.

Synthesis of quantitative evidence

The basic principles of the GRADE approach were applied to the synthesis of the quantitative evidence. The quantitative studies were tabulated for each important information point and analysed together in an evidence profile (table 1). An initial level of evidence (high, low or very low) was assigned to the group of quantitative studies reporting findings related to each information point. The initial level of evidence was then increased or decreased after consideration of the following factors: study design, quality, consistency and directness (box 7).⁴⁶,⁴⁹ Data on the size or magnitude of effects or associations were taken into account whenever possible—for example, if a good quality randomised trial of written information provision showed a beneficial effect on outcomes such as knowledge, understanding, acceptability or anxiety, then the kinds of information included in the trial materials were recommended. Data related to population baseline risks and resource utilisation were not available and were omitted from the review evidence profiles.

Table 1 Evidence profile for important information point in national colposcopy leaflet: “indicate that treatment can occur at the first colposcopy clinic visit”.

Studies	Assessment					Summary of findings
	Design	Quality	Consistency * across studies	Directness †	Other factors‡	Overall assessment	Overall recommendation
Gath, 199547	Non‐comparative descriptive	++	No important inconsistency	DirectDirect	None	Very low
Olamijulo, 199730	Non‐comparative descriptive	+
Howells, 199931	Randomised trial	++		Uncertain
							Definite
Kuehner, 200148	Qualitative	++	No important inconsistency	Direct	Close conformity based on direct evidence	High
Byrom, 200328	Qualitative	++		Direct
Neale, 200329	Qualitative	++		Direct

• The quantitative studies indicated that the provision of a colposcopy leaflet or sheet containing information about the possibility of treatment at the first colposcopy clinic visit was acceptable to women. A need for clearer information about the possibility of receiving treatment at the initial visit was identified.

• The qualitative studies reported that women have unanswered questions about whether treatment will be received on the day of the colposcopy appointment.

*Consistency among quantitative studies refers to the similarity of estimates of effect or observations across studies.⁴⁶ Consistency among qualitative studies refers to similarities in developed themes and participant experiences across studies.

†Directness refers to the extent to which people, interventions and findings are similar to the NHSCSP population.

‡Other factors include imprecise or sparse data, strong or very strong association, high risk of reporting bias, evidence of a dose‐response gradient, effect of plausible residual confounding and close conformity of findings based on direct evidence (see box 7).

Box 7 Quantitative and qualitative research evidence synthesis—assessment criteria for the level of evidence.

1 An initial level of evidence was assigned to a group of studies that addressed a particular information point as shown
Quantitative studies 46	Qualitative studies
High = randomised trial	High = checklist quality score “++”
Low = observational study	Low = checklist quality score “+”
Very low = any other evidence	Very low = checklist quality score “−“
The initial level of evidence assigned was based on the lowest hierachical type of evidence (ie, study design) in the group of studies.	The initial level of evidence assigned was based on the lowest checklist quality score of any study in the group.*
*The use of the lowest checklist quality score enabled any uncertainty in the quality of the available evidence (in conduct, reporting or both) to be incorporated in the initial level of evidence assigned.
2 The initial level of evidence was modified into one of four levels (high, medium, low and very low) according to several additional considerations
Quantitative studies46	Qualitative studies
Decrease level of evidence if:	Decrease level of evidence if:
Serious (−1) or very serious (−2) limitation to study quality
Important inconsistency (−1)	Important inconsistency (−1)
Some (−1) or major (−2) uncertainty about directness	Some (−1) or major (−2) uncertainty about directness
Imprecise or sparse data (−1)
High probability of reporting bias (−1)
Increase level of evidence if:	Increase level of evidence if:
Strong evidence of association (+1)	Close conformity of findings based on two or more studies rated as ++, directly applicable to the target population and with no major threats to validity (+1)
Very strong evidence of association (+2)
Evidence of a dose response gradient (+1)
All plausible confounders would have reduced the effect (+1)

Synthesis of qualitative evidence

Qualitative studies addressing one particular information point were collated and assigned an initial level of evidence (high ++, low + or very low −) on the basis of the lowest checklist quality score obtained for any study in the group (table 1, box 7). The initial level of evidence was then modified according to the consistency and directness of the evidence across the group of studies (these additional considerations acted cumulatively). As in the GRADE system, four overall qualitative levels of evidence (high, moderate, low and very low) were used during this phase of the evaluation process. Consistency referred to similarities in developed themes and participant experiences across studies whereas directness referred to the similarity between people, interventions and findings compared with the NHSCSP population. It is important to note that individual studies were not directly penalised for reporting contradictory or less direct evidence. Judgements about consistency and directness were made on the basis of the evidence provided by all of the studies, which in turn influenced the guideline recommendation for that specific information point. The existence of contradictory findings across studies was not taken to be evidence of poor quality research instead; uncertainty was reflected in the strength of the given recommendation.

For example, the qualitative studies in table 1 all indicated that women have unanswered questions about the possibility of treatment at the initial colposcopy appointment and that the studies were conducted in directly applicable populations. Therefore the initial level of evidence “high” remained unchanged. If, however, one of the three qualitative studies had received a checklist quality score of “+” instead of “++” then the initial level of evidence assigned to the group of studies would have been “low” instead of “high”. In this situation, the initial level of evidence would have been promoted from “low” to “moderate” owing to the direct and consistent findings reported by the two remaining “++” rated studies. In theory, one study could have found that women prefer to receive treatment information in‐person at the colposcopy clinic. At this point, a further decision regarding the importance of the inconsistency between studies would have been necessary. This type of grading system provides increased flexibility for reviewers when making decisions about evidence.

Recommendation system and review findings

Both overall levels of evidence (quantitative and qualitative) were used to determine which information points should be included in the screening letters and leaflets. Two categories of recommendation “definite” and “suggestive” were developed (fig 1). Items for which the quantitative and/or qualitative evidence was graded as “high” and/or “moderate” were given a “definite” recommendation for inclusion in the screening information. Items where quantitative and qualitative evidence were graded as “low” and/or “very low” were designated as “suggestive”. In this way, a final recommendation for each information point was included in the review without obscuring the contribution of each type of evidence or complicating the grading system.

Although different levels of evidence were often assigned to the quantitative and qualitative research, contradictory findings were not reported for any of the information points considered. If the qualitative research had been omitted from the review, many recommendations would not have been incorporated or achieved a “definite” status. This applied to the recommendations presented in fig 1, table 1 and the following:

• The use of statements intended to reassure such as “not to worry” or “no big deal” should be avoided.

• The term “wart virus” should be not be included.

• The staff members participating in colposcopy and treatment appointments should be identified.

Discussion

The development of methods for the synthesis of diverse data sources in a single review is a complex challenge.¹⁴,¹⁵,³⁴ It has been said that, “ultimately a subjective and pragmatic judgement must made about methodological issues”.²³ Despite the current lack of guidance, we decided that it was still a valid exercise to attempt to pull together updated evidence‐based guidelines for the production of NHSCSP information using both quantitative and qualitative studies. The purpose of this article is to share our experiences with other researchers and contribute to the ongoing discussion about the best way forward.

The GRADE methodology has been developed⁴⁶ and refined by its application to existing systematic reviews composed mainly of randomised trials.⁴⁹ The reliability and sensibility of the approach is currently being assessed.⁴⁹ The attraction of the GRADE system was the ability to modify the level of evidence assigned to a group of studies through consideration of factors other than study design alone. However, even with the improved flexibility of the GRADE system, study design considerations affected the level of evidence assigned to the quantitative research (it was often rated as “low” or “very low”). On the other hand, the qualitative research often obtained evidence level ratings of “high” or “moderate”. This may partly be explained by the checklist quality scores assigned to the qualitative studies and the increase in evidence level allowed by the close conformity of findings between two or more studies. A sensitivity analysis investigating the effect of the grading system on the results of the review was not attempted–further work is required on this subject.

Study quality appraisal is an important unresolved issue in the synthesis of quantitative and qualitative research.¹³ Many different quality criteria for the assessment of qualitative studies have been proposed but no common standards have been agreed.¹³,⁵⁰ Controversy also exists about the defining characteristics of good quality qualitative research and the prescription of structured quality criteria.⁹,¹³,³³,⁵⁰ In contrast, quality checklists and formal hierarchies of evidence are included in procedural manuals for quantitative reviews of effectiveness studies.⁷,⁸

A pragmatic decision was taken to use a quality checklist modified from published sources⁴¹,⁴⁴ to assess the various qualitative studies included in the review. The checklist quality score was used primarily as a means for highlighting the strengths and weaknesses of each study.⁹ Although checklists are convenient tools, it was often difficult to determine whether the quality of reporting or the design and execution of the study was being assessed. For some checklist criteria, the information required to make a decision about whether an item was “well covered”, “adequately addressed” or “poorly addressed” was not always available. Also, separate reviewers applied and interpreted checklist criteria in different ways (even with a coded comments system), although agreement was always reached by consensus. The quality of each study was often better captured by the detailed study assessment notes compiled during the appraisal process.

The review questions were best answered by evidence from a range of data sources. The inclusion of the qualitative research, which was often highly relevant and specific to many issues covered by the screening information, enabled the production of a set of guidelines that will directly affect policy in the NHSCSP. The updated guidelines for the content of the NHSCSP leaflets and letters appear to be sensible standards that confirm and strengthen the previous recommendations.¹,²⁸

The systematic review and synthesis of diverse research evidence, though rewarding, is challenging and time‐consuming. Several aspects of the review process that involve considerable time investment on the part of researchers include searching, study appraisal and analysis. As current methods are further developed the process should become increasingly streamlined, transparent and explicit.¹³,²⁶,⁴⁶ In presenting this research we hope to provide a practical example of how numerous data sources might be brought together to inform guidelines related to the contents of patient information materials.

Abbreviations

NHSCSP - National Health Service Cervical Screening Programme