Text-in-Context: A Method for Extracting Findings in Mixed-Methods Mixed Research Synthesis Studies

Abstract

Aim

Our purpose in this paper is to propose a new method for extracting findings from research reports included in mixed-methods mixed research synthesis studies.

Background

International initiatives in the domains of systematic review and evidence synthesis have been focused on broadening the conceptualization of evidence, increased methodological inclusiveness and the production of evidence syntheses that will be accessible to and usable by a wider range of consumers. Initiatives in the general mixed-methods research field have been focused on developing truly integrative approaches to data analysis and interpretation.

Data source

The data extraction challenges described here were encountered and the method proposed for addressing these challenges was developed, in the first year of the ongoing (2011–2016) study: Mixed-Methods Synthesis of Research on Childhood Chronic Conditions and Family.

Discussion

To preserve the text-in-context of findings in research reports, we describe a method whereby findings are transformed into portable statements that anchor results to relevant information about sample, source of information, time, comparative reference point, magnitude and significance and study-specific conceptions of phenomena.

Implications for nursing

The data extraction method featured here was developed specifically to accommodate mixed-methods mixed research synthesis studies conducted in nursing and other health sciences, but reviewers might find it useful in other kinds of research synthesis studies.

Conclusion

This data extraction method itself constitutes a type of integration to preserve the methodological context of findings when statements are read individually and in comparison to each other.

INTRODUCTION

Data extraction is a process foundational to all kinds of research synthesis studies and therefore a prominent feature in instructional texts (e.g. Pope et al. 2007, Sandelowski & Barroso 2007, Cooper 2010, Higgins & Green 2011, Hannes & Lockwood 2012). The word ‘extraction’ connotes removal of one clearly defined entity from another larger entity (as in the extraction of teeth from the gums); ‘data extraction’ might therefore convey that information can simply be read off the pages of a report and easily separated from it. Glossed is the fact that the extraction process itself entails a series of judgments concerning what information to extract and the form this information should take to make it workable for synthesis (Sandelowski 2008). The interpretive work of data extraction is especially evident in the extraction of findings from reports of methodologically diverse studies. Our purpose in this paper is to propose a method for extracting findings whereby they are formatted to preserve their methodological context and thereby to render them useful for the synthesis of qualitative and quantitative findings.

BACKGROUND

This purpose is especially relevant to international initiatives in the domains of systematic review and evidence synthesis from such agencies as the Cochrane and Campbell Collaborations and the Joanna Briggs Institute to broaden the conceptualization of evidence, be more methodologically inclusive and produce syntheses of evidence that will be accessible to and usable by a wider range of consumers. This purpose is also in line with initiatives in the general mixed-methods research field to develop truly integrative approaches to data analysis and interpretation (e.g. Bryman 2007).

DATA SOURCE

We developed this method in the course of an ongoing (2011–2016) study entitled ‘Mixed-Methods Synthesis of Research on Childhood Chronic Conditions and Family’ (hereafter referred to as the Family Study). The overall aim of the Family Study is to synthesize findings from English-language reports of primary qualitative, quantitative and mixed-methods studies on the intersection between family life and childhood chronic physical conditions to (a) explain how aspects of family roles, relationships and functioning contribute to varying child, family member and family system outcomes and (b) examine the effects of family-directed interventions on child and family outcomes and the factors mediating, moderating, or otherwise influencing the intervention effects. The Family Study is broad in its topical scope as it is directed toward integrating findings within and across chronic conditions and family configurations to address recommendations calling for reviewers to draw from a broader base of evidence to further delineate how aspects of family structure and process influence child health (Barlow & Ellard 2006, Carr & Springer 2010). A panel of family researchers and practitioners will assist us to specify the research questions addressing the two broad aims of the study. We hope thereby to counter the criticism that systematic reviews exclude too many relevant reports for questionable reasons thereby resulting in empty reviews (MacLure, 2005, Lang et al. 2007) and that they often do not yield actionable results relevant for practice (Pawson, 2006, Egan et al. 2009).

The Family Study is broad in its methodological scope by including empirical reports of primary qualitative, quantitative and mixed-methods studies instead of limiting reviews only to certain methodological types of studies and in the use of research synthesis approaches with the distinctive capacity to accommodate the challenges of integrating findings produced in diverse methodological contexts. We chose Bayesian meta-analysis (Crandell et al. 2012) and realist synthesis (Pawson 2006) methods to achieve the aims of our study because of their inclusiveness and flexibility in managing methodologically diverse and incomparable data sets and sample sizes and because they allow qualitative and quantitative findings to play equally important roles in all phases of a research synthesis study. Bayesian methods are directed largely toward the synthesis of findings by aggregation thereby yielding summative statements on relationships among variables; realist methods are directed largely toward the synthesis of findings by configuration thereby yielding explanations for how they are related (Sandelowski et al. 2011).

The broad topical and methodological scope of our study has important implications for the extraction of research findings as they must be formatted in ways amenable to these broad research aims and to the content and nature of the findings themselves. As we propose in this article, the extraction of findings in mixed-methods mixed research synthesis studies is a highly interpretive process where reviewers must define what information in a report will constitute a finding and resolve issues related to content (what a finding says), context (what information in a report is most relevant to understanding a finding) and form (how a finding-in-context is to be expressed).

DISCUSSION

Defining the Extractable Qualitative and Quantitative Finding

A sine qua non of any research synthesis study is that reviewers have a clear definition of what will constitute a finding relevant to their research aims and questions. For example, in the Family Study we defined ‘child,’ ‘chronic physical condition’ and ‘family’ not only to delimit the parameters for our search and to select the reports of studies retrieved from that search to be included in our dataset, but also to guide the extraction of findings in those reports relevant to our review. We are here drawing key distinctions between (a) studies and reports of studies as any one study might yield more than one report with findings drawn from the same sample and therefore related to each other and between (b) reports of studies and the findings in those reports because a report selected as relevant may contain findings all or only some of which are relevant to the research aims and questions guiding the research synthesis study and therefore subject to extraction. Reports from the same parent study must be marked as related for later stages of analysis.

Foundational to the extraction of findings relevant to particular research aims and questions is the specification of what information in reports will be viewed as constituting findings. This is much more demanding than it might seem because findings are conceived and therefore presented and located differently in reports of qualitative and quantitative studies.

In quantitative reports, findings (hereafter referred to as quantitative findings) are typically conceived as the data-based outcomes of the various descriptive and inferential statistical tests applied in answer to research questions and/or to support or refute hypotheses specified in advance of data collection and analysis. Quantitative findings typically offer the size and statistical significance of the relationships between and among sets of pre-selected variables and/or explicit comparisons between groups of participants on these relationships. Quantitative findings are typically linked to numbers of participants because statistical inference depends on the prevalence or frequency of observations. Quantitative findings are most commonly presented in text, tables and figures in the forms of group means and standard deviations, frequency tables, odds ratios and p-values, correlation coefficients or regression coefficients and standard errors. Occasionally, they will be described only non-numerically (e.g. a significant positive relationship was found between X and Y). Quantitative findings typically appear only in the section of the report labeled ‘results,’ with researchers’ interpretations of these results—for example, their speculations about why those results were obtained and what they might mean, their comparison of their own results to those of other investigators—confined to the section of the report labeled ‘discussion.’ The typical presentation and location of quantitative findings reflect what is commonly conceived as a neo-positivist or objectivist orientation to inquiry whereby results are seen to be derived solely from the unbiased and consistent application of techniques and procedures such that anyone applying them would obtain the same results (Sandelowski & Barroso, 2007).

In contrast, in qualitative reports, findings (hereafter referred to as qualitative findings) are themselves typically conceived as researchers’ interpretations of the data generated in a study, or researchers’ configurations of segments of data assembled into novel wholes to address research aims and questions that were not fully specified until after analysis was underway. The interpretations constituting the data-based qualitative finding derive not only from those data researchers generate in interaction with participants (e.g. from interviews and field observations) but also in interaction with themselves in their written reflexive encounters with those data (e.g. in process and analytic notes). Moreover, qualitative findings are frequently not linked to numbers of participants as it is typically deemed less informative to present counts of the number of persons expressing a theme than to interpret the thematic lines themselves and to ensure that they are adequately distinguished from each other (Chang et al. 2009). A N=1 finding might therefore merit inclusion in a qualitative report while in a quantitative report it will likely have been discarded as an outlier too small to deal with statistically. Qualitative findings are most commonly presented in narrative form organized thematically or temporally, but also in tables organized by themes or in figures representing theoretical or other abstract renderings of the experiences and events studied. Qualitative findings are not necessarily confined to sections of the report labeled ‘results’ nor are references to other studies necessarily confined to sections labeled ‘discussion.’ Indeed, because qualitative findings may appear anywhere in a report or not be labeled as findings, reviewers must be able to distinguish the findings produced from the study featured from researchers’/authors’ descriptions of how data were analyzed (e.g. arranged, coded), presentations of theory and from the findings from other studies to which researchers/authors refer to place their own findings in context. The varied presentations and locations of qualitative findings reflect what is commonly conceived as a constructivist orientation to inquiry whereby results are seen to be derived from unique and situational encounters between researchers and the persons and events studied and within researchers in reflexive encounters with data (Sandelowski & Barroso, 2007).

Moreover, data extraction in any research synthesis study always involves extracting much more from reports than simply the findings as these findings will be difficult to interpret without knowing the methodological context in which the findings were produced (e.g. composition and size of the sample, the nature of data collection). A quantitative finding placed in the results section of a data extraction sheet that there was a significant Group X Time effect will be uninterpretable without reference to information located in other sections of that sheet about the groups and times studied. A qualitative finding placed in the results section of a data extraction sheet that there were three themes including shock, coping and surviving will be uninterpretable without reference to information located elsewhere on that sheet about the persons and events studied.

In most respects, the multi-page extraction sheets used in the Family Study (and available on request from the first author) reflect the usual practice whereby reviewers extract information about such things as the purpose, sample and data collection and analysis and place this information in sections of their data extraction sheets that align with the sections of the conventional research report. For example, information extracted from the sample section of a report is located in the sample section of a data extraction sheet and information extracted from the data collection section of a report is located in the data collection section of a data extraction sheet.

Although this practice allows reviewers to ensure that all relevant information in a report has been extracted, it also separates the results (typically placed in the results section of data extraction sheets) from the information in other sections of that sheet that makes them meaningful. This format accommodates well research synthesis studies guided by highly specific research questions to which reviewers commit prior to beginning data extraction, but it is less useful in more expansive studies such as the Family Study where the specification of research questions are themselves data-derived.

Transforming Findings to Preserve the Context of their Production

Findings might be more usefully presented in the results section of data extraction sheets in ways that preserve the context of their production. In the Family Study that means creating statements of findings that capture those aspects of methodological context captured in sections of the extraction sheet other than the findings section deemed most relevant to understanding these findings. These statements of findings reassemble data disassembled throughout the data extraction sheets and are understandable on their own apart from the data extraction sheets and the reports from which they were derived. The extraction of findings from primary study reports thus entails data transformation whereby portable declarative sentences are constructed that integrate findings with information about the study deemed most relevant to characterizing those findings.

The work of transforming findings as presented in reports into portable declarative sentences includes first removing method talk that might interfere with reader comprehension such as talk about themes and sub-themes emerging, hypotheses having been supported or not supported and specific main and mediating/ moderating effects having been found. Instead of the statement that CES-D (Center for Epidemiologic Studies Depression) scores were significantly negatively correlated with FACES (Family Adaptability and Cohesion Evaluation Scale) scores, a re-statement might read: Among mothers of children with cystic fibrosis, there was a significant negative correlation between maternal depression and family functioning such that the more depressed the mother the less adaptable and cohesive the family. Instead of the statement that there were three themes that emerged in the data, a re-statement might read: Mothers’ initial reactions to their child’s diagnosis of cystic fibrosis included shock followed closely by fear and/or relief.

After subtracting all unnecessary method talk, the work of transforming findings into declarative stand-alone sentences includes adding the most salient information extracted from other sections of the report. Because it would be impossible to create comprehensible but concise statements that include all of the demographic, clinical, data collection and other information related to findings extracted from reports, information must be chosen most relevant to contextualizing findings. Drawing from discourse theory to show how qualitative text analysis might be made more transparent, Stenvoll and Svensson (2011) described context as those things that have to be known to understand properly an action or event and literal contextualization as the anchoring of statements in a text regarding those actions and events to what is explicitly stated in the text. Accordingly, our goal was to create statements that were themselves data-based syntheses of text (relevant findings)-in-context (other relevant information).

Findings formatted this way not only make them more understandable and portable, but also preserve the context of their production and enable reviewers to ascertain the varied and dynamic relationships among findings foundational to integrating them. Such sentences might be especially amenable to Banning’s (2003, July) ‘ecological sentence synthesis,’ a mode of integrating the findings across a set of reports that entails finding sentence patterns or structures that can encompass findings to allow comparisons among them. Such sentences begin to address Pawson’s (2006, p. 91) criticism of standardized data extraction processes whereby reviewers ‘load uniform bits of information onto the review conveyor belt,’ and his recommendation that data extraction be both customized and integrated.

Our sentence construction approach constitutes a data-near integration of text and context beginning in the data extraction phase of a study. Our approach has the potential to offset the criticism of realist synthesis that it is too idiosyncratic (Sheldon, 2005) because we have based it on decision rules of thumb for the anchoring of context to text that are transparent and can be applied consistently. We intend the rules summarized in Table 1 and detailed below to be used flexibly, that is, tailored to the needs of individual synthesis studies.

Table 1.

Rules of Thumb for Anchoring Text to Context in Restatements of Findings

Findings anchored to relevant information about:	Rules of thumb
Sample	Specify sample according to extent of variation and/or whether these variations are addressed in the findings
Source of information	Insert source of information whenever there are multiple sources of information about an index person or event
Time	Insert time phrase in longitudinal and intervention studies and if time is featured as a variation in the target phenomena in the findings
Comparative reference points	Insert between-group and between-theme comparisons
Magnitude & significance	Insert frequency, percent participants associated with a finding in survey findings and statistical significance
Study-specific conceptions	Specify target phenomena as operationalized in study

Anchoring findings to sample information

All restatements of findings in the Family Study begin with a designation of the participants linked to findings. The specificity of these designations is dependent on the variation in the sample and on whether these variations were addressed in the findings. For example, restatements of findings concerning the child or parent participants in a study of families with children with cystic fibrosis whose ages varied from infancy to adolescence and where the findings did not differentiate these children or parents on any parameter (e.g. age, sex, severity of condition) might simply begin with the phrase: Children with cystic fibrosis … or Parents of children with cystic fibrosis …

Whenever variations in the sample are the targets of analysis or the sample is largely homogeneous on one or more parameters, restatements are anchored to more sample detail, as in: Low-income mothers of adolescents with cystic fibrosis… Sometimes a key characteristic of the sample will not have been featured (or perhaps not noticed) by the authors of reports. Here we anchor findings to those details deemed most important to the interpretation of individual study findings and comparisons of findings across studies. In a study focused on adherence, for example, knowing that the findings were derived from a mostly adherent child and/or parent sample will be highly relevant, as indicated in the phrase: In families with children with cystic fibrosis mostly adherent to their prescribed regimen…

Anchoring findings to source of information

The Family Study targets reports of studies involving minor (< 18 years old) children. Parents will therefore often have been the major or only source of information about their child’s condition, behavior, or health status. In addition to parents, study samples will sometimes have included siblings and healthcare providers as sources of information about themselves and/or each other. For example, the Swift et al. (2003) study involved children following traumatic brain and orthopedic injuries, their siblings and their parents. In this study, both the injured children and their siblings were asked to respond to questions about their relationships with each other and parents were asked to respond to questions about the behaviors of both the injured child and their sibling.

The findings of studies where persons other than or in addition to the index participants were sources of information about the index participants are therefore anchored to their sources. In any study of ‘family,’ a construct that implies two or more persons or systems in relation, anchoring findings to their source is especially important to enable examination of the link between source of information and the information itself and therefore to enable comparison of vantage points of family members within and across studies. An example of a restatement of findings from the Swift et al. (2003) report that captures the different vantage points targeted is: In families with children with traumatic brain injury, more parent-reported child behavior problems were significantly associated with more sibling-reported conflict and rivalry with their injured siblings and more parent-reported sibling behavior problems.

Anchoring findings to information about time

Statements of findings in the Family Study include the anchor of time whenever any factor of time related to the research design itself or to the persons, conditions, or events studied was a key element in a study, as, for example, in longitudinal studies, baseline and follow-up data collection in intervention studies, or such factors as time since diagnosis or in treatment and time in caregiving. Examples including the anchor of time are: In children with traumatic brain injury an average of four years after injury … (Swift et al. 2009) and in families of children with cystic fibrosis receiving behavior + nutrition education or nutrition education alone, less child mealtime problem behaviors prior to receiving these interventions predicted better weight gain immediately after receiving these interventions (Opipari-Arrigan et al. 2010).

Anchoring findings to comparative reference points

Statements of findings are anchored to the in-relation-to-who or –what of findings. By far the most prevalent comparison, especially in quantitative research reports, is between groups of participants. An example of such a statement derived from the Ward et al. (2009) report is: Children younger than 6 years old with cystic fibrosis whose primary caregivers (mostly mothers) reported harsh parenting were almost four times more likely to display internalizing problem behaviors such as anxiety, depression and withdrawal than children whose caregivers did not report harsh parenting. Anchoring information might overlap, as in the case of intervention studies involving group comparisons (i.e. one or more intervention and control groups) and time (i.e. one or more points before and after intervention).

In qualitative studies, the comparative reference point will often be between groups differentiated by theme or between themes derived from participants as a group but not necessarily linked to particular participants. Here, key elements of the way authors structured their findings constitute the comparative anchors. For example, Nicholas (1999) created a data-derived typology of mothers of children with end-stage renal disease composed of the defining attributes of each of three types of mothers that were compared with each other. The restatement that captures this typology is: Mothers of children with end-stage renal disease constructed themselves as trapped, adaptive, or embedded caregivers. Examples of restatements that capture the defining attributes of each of these three types of mothers are:

Trapped mothers of children with end-stage renal disease resented the demands and restrictions of caregiving, perceiving that their ambitions and independence were sacrificed in meeting their ill child’s needs.

Adaptive mothers of children with end-stage renal disease focused on making the ill child’s care fit into their lives and their lives fit with the demands of their child’s care.

Embedded mothers of children with end-stage renal disease viewed their identity and role in life to be consistent with providing care to their ill child and quietly accepted any difficulties they encountered.

An example of a restatement that captures a contrast among these three types of mothers is: Embedded mothers of children with end-stage renal disease—unlike trapped and adaptive mothers—viewed the provision of demanding care to be normal because they strongly identified and defined themselves as caregivers. Anchoring findings to the thematic structure constituting the findings in qualitative studies where findings are composed of abstract renderings of data allows reviewers to evaluate these structures for their potential utility for synthesizing related findings from other qualitative and quantitative reports (Sandelowski et al. 2011).

Anchoring findings to information about magnitude and significance

The meaning of research findings, especially quantitative findings, is derived in large part from their magnitude. Quantitative findings are linked to numbers of participants, with statistical significance dependent on frequency counts. The themes constituting qualitative findings imply number because to say that something was thematic implies a discrete and therefore countable entity (Sandelowski et al. 2009).

In the case of qualitative and quantitative surveys, which are both characterized by frequency counts of the number and/or percent of participants linked to an item response or expression of a theme, statements are anchored to indicators of magnitude sufficient to make findings meaningful. An example is the following restatement derived from the Wong and Heriot (2008) report: Most (70%) parents of children with cystic fibrosis most frequently used as coping strategies acceptance, active coping, planning and emotional support and least frequently used substances, behavioral disengagement, denial and religion. Whenever possible, words such as ‘few,’ ‘most,’ or ‘some’ are replaced with actual numbers available in the narrative text or tables in the report (or, if deemed essential, from authors). Yet, because this kind of verbal counting is characteristic of many qualitative reports of basic descriptive studies, these verbal counts are retained in the absence of actual numbers as they offer some information on magnitude that might be useful in later efforts to pool such findings (Chang et al. 2009).

In all cases of findings addressing statistical significance, statements of findings include all relationships relevant to research synthesis aims found also to be not statistically significant as in this restatement from the Swift et al. (2003) report: Siblings of children with severe and moderate traumatic brain injury reported significantly more conflict and rivalry with siblings of the opposite sex than siblings of children with orthopedic injury an average of four years after the injury, but there were no significant differences in conflict and rivalry with siblings of the same sex, or in closeness to and respect for siblings of the same and opposite sex. This way of formatting findings to capture magnitude and significance is in addition to traditional ways of formatting findings for meta-analysis, which involve the calculation of effect sizes from the numerical data in reports (or available from authors) supporting the finding.

Including the anchor of significant difference/not significant difference to statements of quantitative findings is especially important in a mixed research synthesis study. Individual quantitative findings not reaching statistical significance might reach statistical significance when pooled with other comparable findings, or might be judged as clinically or thematically significant when pooled or configured with other qualitative and/or quantitative findings addressing the same relationship.

Anchoring findings to study-specific conceptions of phenomena

A common challenge in all research synthesis studies is that no two studies—even those focused on ostensibly the same relationships—will address them in the same way. One construct might be measured via two or more different instruments. For example, both the Family APGAR (Smilkstein 1978, Austin & Huberty 1989) and the Family Hardiness Index (McCubbin & McCubbin 1996) are used to measure family functioning, but the former targets satisfaction with various aspects of family life while the latter targets the internal strengths and durability of the family. One instrument might be used to measure two or more different constructs. For example, the Perceived Competence Scale for Children (Harter 1982) was used in the Riker et al. (1998) study to measure child self-perception across four domains (cognitive, social, physical, overall competence) and in the Thompson et al. (1999) study to measure child self-efficacy.

Accordingly, statements of findings in the Family Study are anchored to information indicating as precisely but as concisely as possible the phenomena actually studied to assess the comparability of findings. For example, DeLambo et al. (2004) assessed family functioning using the Iowa Family Interaction Rating Scale (Melby 2001) and White et al. (2009) assessed family functioning using the Family Adaptability and Cohesion Evaluation Scale (Olson et al. 1983). Our restatement of a finding from the DeLambo et al. (2004) report is: In families of older children and adolescents with cystic fibrosis, the better the family problem solving the better was mother-reported adherence to the prescribed airway clearance/aerosol regimen. Our restatement of a finding from the White et al. (2009) report is: Families with children with cystic fibrosis demonstrating more balance between cohesion and flexibility (neither overly rigid nor flexible) had higher rates of child- and parent-reported adherence than families demonstrating less balance between cohesion and flexibility (overly rigid or overly flexible).

As our analysis proceeds, these restatements will allow us to compare results from other studies addressing family functioning in general and family problem-solving and family rigidity/flexibility in particular. With guidance from our clinical and research advisory panel, this comparison will allow us to determine the level of abstraction most relevant to practice and future research: that is, whether combining all findings about family functioning regardless of operationalization or combining findings targeting specific types of family functioning (e.g. problem solving, adaptability) is warranted. In any research synthesis study, such decisions derive in part from the number of comparable findings available to synthesize; for example, in the Family Study, there might not be enough findings targeting specific types of family functioning to synthesize. Ultimately, reviewers determine the level of abstraction that will make the most sense to the users of the synthesis.

Other considerations

Data extraction is a laborious and time-consuming process that can contribute to delays in producing and disseminating the results of systematic reviews (Sampson et al. 2008). The process we describe here might therefore be seen as making data extraction even more laborious and time-consuming. Although it certainly took us time to arrive at our rules of thumb, we believe that our efforts in the data extraction phase of our study will save us time later and contribute to the overall quality of analyses focused on grouping and comparing findings in preparation for synthesizing them. Given the broad topical scope of the Family Study that will likely include approximately 700 reports, we anticipate our results will continue to have relevance for a longer period of time than, for example, reviews with narrower scopes in domains of rapid medical advancement.

We have chosen the collaborative approach to optimizing inter-rater consistency that Belgrave and Smith (1995) referred to as ‘negotiated validity.’ This designates a social process whereby research team members work to achieve consensus. Favoring a negotiated consensus over a traditional inter-rater reliability approach to optimizing consistency in the Family Study is the highly interpretive nature of the findings extraction process. At least two members of our research team extract information from each report, with a third and even fourth reviewer solicited for reports raising questions. (Effect sizes for all relevant quantitative findings are calculated by yet another reviewer whose calculations are also checked.) All reports calling into question our rules of thumb are discussed also in our weekly research team meetings. The second reviewer is always a senior member of the team (i.e. one of the principal or co-investigators, not a research assistant) whose job it is to check the accuracy of every item on the data extraction sheet completed by the first reviewer and to ensure that the restatements of findings include the relevant anchors. So far, data extraction of approximately 110 reports has taken 2–6 hours per study for the primary reviewer and 1–4 hours for the second reviewer.

Implications for nursing

The data extraction approach featured here was developed to accommodate the distinctive challenges of research synthesis studies that entail findings generated in highly diverse methodological contexts. Reviewers conducting any kind of research synthesis study, however, might find this approach useful in producing accessible and usable integrations of findings.

CONCLUSION

Our process for extracting findings from research reports itself constitutes a type of integration aimed at preserving relevant features of the methodological context in which these findings were produced when read individually and in comparison to other findings statements. We see our approach, not as a replacement of existing methods of data extraction, but rather as a potentially useful additional method.

Summary Statement.

What is already known about this topic

• Data extraction is foundational to research synthesis studies and therefore a prominent feature of instructional texts on qualitative, quantitative and mixed research synthesis.

• The usual practice in data extraction is to extract information about such things as the sample, data collection and analysis and results and to place this information in sections of data extraction sheets that align with the sections of the conventional research report.

What this paper adds

• Because the typical data extraction process separates findings from other information in reports that make those findings meaningful, a method is proposed whereby the text-in-context of findings is preserved.

• The extraction of findings is a highly interpretive process whereby judgments are made about what information in a report will constitute a finding and about content (what a finding says), context (what information in a report is most relevant to understanding a finding) and form (how a finding-in-context is to be expressed).

• The interpretive nature of findings extraction is especially evident in research synthesis studies involving mixed-methods approaches to the synthesis of qualitative and quantitative findings produced in diverse methodological contexts.

Implications for practice and/or policy

• The findings extraction method described here is potentially useful as an additional mode of extracting findings in research synthesis studies.

• This method will make findings more workable for research synthesis.