Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Jun 9.
Published in final edited form as: J Ind Ecol. 2012 Apr 4;16(Suppl 1):S12–S21. doi: 10.1111/j.1530-9290.2012.00476.x

Systematic Review Checklist: A Standardized Technique for Assessing and Reporting Reviews of Life Cycle Assessment Data

Jennifer M Zumsteg 1, Joyce S Cooper 2, Michael S Noon 3
PMCID: PMC4461004  NIHMSID: NIHMS448905  PMID: 26069437

Summary

Systematic review, including meta-analysis, is increasingly utilized in life cycle assessment (LCA). There are currently no widely recognized guidelines for designing, conducting, or reporting systematic reviews in LCA. Other disciplines such as medicine, ecology, and software engineering have both recognized the utility of systematic reviews and created standardized protocols for conducting and reporting systematic reviews. Based largely on the 2009 Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement, which updated the preferred format for reporting of such reviews in biomedical research, we provide an introduction to the topic and a checklist to guide the reporting of future LCA reviews in a standardized format. The standardized technique for assessing and reporting reviews of LCA (STARR-LCA) checklist is a starting point for improving the utility of systematic reviews in LCA.

Keywords: academic publishing, health care, reporting standards, bias, industrial ecology, meta-analysis

Introduction

Approaches that set out to understand systems and processes in a comprehensive way and can account for a range of impacts, such as life cycle assessment (LCA), are of value in better informing the factual grounds of decisions in areas like business and policy. Individuals engaged in related decision making require reliable information on which to base decisions. Some of this information may be about natural resources or manufacturing systems, such as understanding the contribution to climate change from the production of a kilogram of corn grain. These inquiries are not necessarily easy to answer and often the most correct answer will be, “it depends.”

Opportunities in life cycle assessment

Advancements in LCA methodology, such as approaches in consequential LCA (Earles and Halog 2011), that better reconcile overlapping, incomplete, or conflicting data are important in ongoing process improvement and related policy recommendations (Hunkeler and Rebitzer 2005). Real-life applications for deriving a so-called best answer from LCA data have arrived. For example, the U.S. Energy Independence and Security Act of 2007 calls for the ability to directly compare life cycle greenhouse gas emissions for different fuel sources (United States 2007). Efforts to reach conclusions about whether fuel source X or Y has fewer greenhouse gas emissions can be approached in many ways and such summary information or opinions may carry the title of “review.”

Characteristics of reviews

The scope and structure of a review in any field will vary based on its purpose. The goal of a review may be to provide a survey of previous publications, critique current work, or make summary projections about future discoveries. A review may discuss data, opinions, or practices from a focused geographical region; it may explore one specific technology or system, or aim to provide comprehensive coverage about a topic.

The structure of the review may be different if the focus is on summary information, providing an expert opinion, critiquing available literature, or re-evaluating existing data. When a comprehensive literature search is conducted in a transparent and unbiased way, it can be considered a “systematic review” (Neely et al. 2009). Such systematic reviews may also be meta-analyses when one of the aims is combining or re-analyzing data from previous studies to answer new questions, improve accuracy, or identify sources of variation.

The purpose of this article is to describe key factors for conducting and reporting systematic reviews, including meta-analyses, in LCA, based on established recommendations from other fields of study. We propose a checklist for a standardized technique for assessing and reporting reviews of LCA data (STARR-LCA) to assist in structuring the process of conducting and reporting systematic reviews. While we anticipate that the STARR-LCA checklist may be practically useful in a range of review applications, our current focus in systematic reviews, and so the terminology in the checklist, has been shortened to just “review.”

To ease the burden on the reader, we simplify the terminology in the remainder of the current report to allow the concept of “previous studies” to include not only full LCA reports but also data obtained from other sources such as life cycle inventory data from public or private databases. Similarly, in the current discussion we consider meta-analysis a subset of systematic reviews, as data for meta-analyses are often obtained through a prospectively defined literature search. We acknowledge that meta-analyses can occur outside the scope of a systematic review of the literature. Information about key terms for this article is provided here:

  • Systematic review: A structured evaluation of the literature with the goal of answering a specific research or application question with a synthesis of the best available evidence. Generally published to share these results with a wide audience for consideration and implementation.

  • Meta-analysis: A melding of data from multiple studies, usually involving additional mathematical analyses, with the goal of utilizing this synergy of information and data size to answer questions that cannot be answered by existing individual studies or to improve the certainty or impact of known findings by increasing the sample size. Meta-analyses are often performed as part of a systematic review.

  • Life cycle assessment (LCA): Also known as life cycle analysis, LCA is a standardized framework, (ISO 2006a, 2006b) that can improve our understanding of the impacts of a system or product through the stages of its manufacturing, utilization, and disposal. LCA studies can take many forms, but typically include a clear goal and definition of the scope; quantification of what natural resources are utilized during the life cycle of a product, and what is released to the environment in that same cycle; and an assessment of the impacts (usually focused on harm) of the releases to the environment. The process is expected to be iterative and has many steps of interpretation along the way. The assessment is often limited to environmental impacts, such as global warming potential; however, the LCA framework allows for consideration of financial costs and impacts related to social metrics such as job creation.

  • Inclusion and exclusion criteria: Parameters set intentionally at the start of a review process to inform which articles, data, and opinions will be included and/or excluded in the summary analysis of the review. Example criteria include a range of years, geographical region, technology type, or functional unit definition. Criteria may be described to either include previous studies (such as including studies published between 1960 and 1980) or to exclude previous studies that are known to be out of the scope of the review (such as in conducting a review of data related to technology A and therefore planning to exclude data related to technology B that is discovered in the review process).

A schematic for our view of how systematic reviews and meta-analyses fit into the overall organization of knowledge in the scientific community is provided in figure 1.

Figure 1.

Figure 1

Open in a new tab

A schematic of systematic reviews and meta-analyses within the wider context of knowledge within the scientific community.

The evolving state of reviews in LCA

Variety of review types

Reviews have been published in many areas related to LCA; table 1 provides selected examples that illustrate a range of topics and review purposes.

Table 1. Examples of reviews in LCA.

Topic Area References
Bioenergy Rowe et al. 2008; Whitaker et al. 2010; Larson 2006; von Blottnitz and Curran 2007; Cherubini et al. 2009; Farrell et al. 2006
Building materials Werner and Richter 2007
Buildings Sartori and Hestnes 2007
Construction industry Ortiz et al. 2009
Ecosystem services in LCA Zhang et al. 2010
Electricity generation Bhat and Prakash 2008
Farming Mondelaers et al. 2009
Livestock products de Vries and de Boer 2010
Management options for paper waste Villanueva and Wenzel 2007
Nuclear power Sovacool 2008
Packaging options von Falkenstein et al. 2010
Pavements Santero et al. 2011
Remediation technologies Lemming et al. 2010
Wind power Lenzen and Munksgaard 2002
Woody biomass-to-liquid production Sunde et al. 2011
Open in a new tab

Interest in systematic reviews of the literature, with or without meta-analysis, is increasing in many disciplines including LCA. In the medical literature the frequent publication of systematic reviews is reflected in the manuscript title, “Seventy-five trials and eleven systematic reviews a day: How will we keep up?”(Bastian et al. 2010); and there are recent systematic reviews on other topics such as education (Salleh et al. 2011), employee retention (Ghapanchi and Aurum 2011), ecology (Kettenring and Adams 2011), and engineering (Khaddaj-Mallat et al. 2011).

Systematic reviews

Systematic reviews vary in scope, quality, and relevance and may seek to expand on existing data by pooling results in a way that provides a higher degree of certainty, less error, or answers an entirely new question using existing data. The strategy of meta-analysis evolved out of educational research in the 1970s; the term was coined by Gene Glass to describe the process of utilizing statistics to merge the results of multiple studies (Hedges 1986). In LCA, conducting meta-analyses can be challenging as the individual studies of interest often have understandably different parameters. One approach to resolving this conflict is where “harmonization” of data from relevant studies is performed to allow for further insights from previously heterogeneous data (Arvizu et al. 2011). Locating previous data appropriate for harmonization may require a systematic review of the literature (Moomaw et al. 2011), yet there is currently no consensus regarding how such reviews should be approached.

In addition to the systematic review approach utilized for data harmonization, other LCA reviews provide a starting point for considering appropriate protocols. In a review of existing LCAs Villanueva and Wenzel (2007) followed a four-phase protocol that included (i) an inventory of existing LCA studies, (ii) definition of criteria for the selection of LCA studies, (iii) selection of LCA studies, and (iv) identification of key issues. Whitaker and colleagues (2010) conducted a systematic review where the protocol was not only defined ahead of time, but was published as a reviewed manuscript (Rowe et al. 2008).

Standards provide guidance about best practices for conducting and reporting LCA studies while providing flexibility in how reviews of any type should be conducted (ISO 2006a, 2006b). A lack of consensus regarding best practices in systematic reviews, specifically, has many disadvantages, such as less efficient peer review, suboptimal electronic search capabilities for locating systematic reviews once they are published, and inconsistencies in terminology. Fortunately, knowledge from other disciplines can be leveraged to improve systematic reviews in LCA.

Systematic review formats utilized in other disciplines

Guidelines for performing and reporting the results of systematic reviews are utilized in areas of research including software engineering (Staples and Niazi 2007), ecology (Pullin and Stewart 2006), and medicine (e.g., van Tulder et al. 2003; Centre for Reviews and Dissemination 2009). Health care has strongly advocated for utilizing the best available evidence for decision-making (e.g., prescribing patient treatment on the basis of drug trials) and the framework of evidence-based medicine (Sackett et al. 1996) is designed to complement the expertise of individual practitioners.

The systematic review framework from evidence-based medicine has been successfully utilized in other disciplines, such as ecology (Gates 2002) and software engineering (Baldassarre et al. 2008). Similarly, we propose a checklist of essential topics to assist in designing, conducting, and reporting systematic reviews in LCA. The checklist content is intended to facilitate planning successful reviews; improve the localization of review publications in literature searches; ease the ability to update content in future reviews; and allow more transparency of methods to increase the ease of peer review and appropriate generalization of findings. The topics and commentary in our STARR-LCA checklist are guided largely by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement (Moher et al. 2009) and elaboration document (Liberati et al. 2009), which are widely used general guidelines for systematic reviews in medicine.

A standardized technique for assessing and reporting reviews of LCA data (STARR-LCA)

A summary of STARR-LCA checklist items is provided here:

  1. Review title, keywords and abstract

  2. Rationale for the review

  3. Review question and objectives

  4. Description of review protocol

  5. Findings and features of the individual studies in the review

  6. Assessment of bias

  7. Synthesis methods (qualitative and quantitative)

  8. Limitations of the review

  9. Summary of findings and conclusions

An explanation of each category and discussion of related issues is expanded below. A sample worksheet that can be used to support study planning, manuscript preparation, and peer review is included in the supporting information available on the Journal's Web site.

Review title, key words and abstract

An informative title and appropriate key words can greatly assist in the application of scientific knowledge by improving the ability of others to identify appropriate reports (Montori et al. 2005; Dickersin et al. 1990). The title should identify the manuscript as a systematic review, meta-analysis, or both. We recommend that journal keyword databases include the terms “meta-analysis” and “systematic review” in order to support appropriate classification and maximize search functions.

In an age of information overload (Anex and Focht 2008; Fraser and Dunstan 2010) a structured summary of an article, such as a standardized abstract format, is essential in outlining for the reader the central question of the report, the general methodology, a summary of the results, and an estimate of the strength of that evidence, as well as highlights of potential relevance to the reader (Lebrun 2007). We suggest abstracts for systematic reviews related to LCA include at least the following nine components: (i) background, (ii) objectives, (iii) data sources, (iv) study eligibility criteria, (x) scope, system boundaries and functional unit, (vi) study appraisal and synthesis methods (including any original data adjustments, such as harmonization), (vii) results, (viii) limitations and (ix) conclusions with implications of key findings.

Rationale for the review

In the context of what is already known, describe why the systematic review, with or without meta-analysis, was a worthwhile effort. In the context of current knowledge explain how the review may add to the knowledge base of readers from a variety of backgrounds or disciplines.

Review question and objectives

A valuable systematic review will be structured around a focused, answerable question that is worth answering and can be clearly summarized (Counsell 1997). There are established strategies for defining such answerable search questions. In health care, the acronym PICO describes one common model for question structure where the key components are the Population of study, the Intervention, a Comparison group, and the Outcome of interest (Liberati et al. 2009; Booth and Fry-Smith, 2003). A systematic review question in the health sciences using the PICO structure could be phrased as: “For men and women over the age of 50, does the management of a heart attack with treatment X or treatment Y result in a shorter average hospital stay?” If the question had been defined simply as, “what is the best treatment for a heart attack?” we would be troubled with still defining what “best” really means and which treatment options we were considering.

Other question design methods that outline the key components of interest or that perform a limited literature review to better inform a planned systematic review can also be valuable, especially if the amount of literature available on a topic or the appropriate scope for a review question is not initially clear (Booth and Fry-Smith 2003; Torgerson 2003). A structured question format, such as PICO, has a range of advantages including helping to facilitate optimal applied research questions, streamlining peer review of research, and even assisting students in defining appropriate questions to support evidence-based practice (Holloway et al. 2004).

PIFT question format

Based on PICO, we introduce a recommended question format, “PIFT”, related to life cycle assessment that includes: definition of the Product or process category being assessed, the Impacts of interest, the Flows or economic sectors contributing to the impact, and the Type(s) of life cycle assessment of interest. We recommend this structured question format for systematic reviews; however it can also guide other review types or studies.

LCA studies are typically categorized as one of three types: process-based, input-output (IO)-based, or a combination of these methods (hybrid). Process-based LCA, in which data are obtained based on parameters of known, physically utilized components, can be further divided into attributional and consequential types (Finnveden et al. 2009). In contrast, input-output models are based on economic inputs to a system, such as the EIOLCA model created by the Green Design Institute at Carnegie Mellon University (2008).

The sample review question, “Utilizing process-based LCA methods, what is the contribution to climate change from the production of a kilogram of corn grain?” would fit the PIFT approach. This question format can be further defined by adding relevant parameters such as geographic region, technology type, or years of interest. Additional examples of structured questions from LCA studies and details regarding each question element using the PIFT mnemonic are provided in table 2.

Table 2. Examples of LCA questions structured using PIFT.
P:
Product or Process		 I:
Impact(s) of Interest		 F:
Flows or Economic Sectors Included		 T:
Type(s) of Life Cycle Assessment		 Sample Question:
U.S. health care sector in 2007 Climate change North American Industrial Classification System (NAICS) sectors Economic input-output What does EIOLCA using NAICS sectors tell us about the climate change impact of the U.S. Health Care sector in 2007?
(Chung and Meltzer 2009)
The four major classes of U.S. mail in 2006 Total energy consumption, climate change, solid waste disposal Paper, paper board, plastic, fuels Process based attributional What does attributional LCA tell us about the total energy consumption, climate change and solid waste disposal related to service for the four major classes of U.S. mail?
(SLS Consulting 2008)
Open in a new tab

Description of systematic review protocol

Generally, a team of individuals participates in a systematic review and a prospectively designed protocol is helpful for consistency, accuracy, and overall organization. Some disciplines have published detailed guidelines governing the methods of review studies (e.g. Higgins and Green 2011). In health care research, some investigators advocate for the additional rigor of prospectively registered review protocols available for peer review and a database of review protocols (Booth et al. 2011). The review's protocol and methods provide a cohesive understanding about how the review study will be planned, conducted and reported. Details of the protocol may also describe the tasks for different investigators, redundancy in tasks to reduce bias and increase accuracy, and planned methods for resolving disagreements about interpretation of the study data (Neely et al. 2009).

A clear outline of the steps undertaken in a review can provide the transparency needed for replication, as well as appropriate updates when additional data are generated in future studies. We recommend that, at a minimum, a systematic review protocol include five key components: (i) a general description of how studies or data will be discovered for consideration of further review (e.g. literature or database search, manual review of the bibliographies of key publications, previous studies performed by the author's laboratory, solicitation of unpublished data), (ii) enough information about any electronic search strategies that the search can be replicated, (iii) clear parameters describing how the decision will be made to include or exclude individual studies and data for further analysis in the systematic review and/or meta-analysis, (iv) a clear plan for recording and summarizing data from the individual studies, including what parameters or data points are of interest, and (v) what, if any, meta-analyses are planned.

Many discussions of systematic reviews acknowledge the iterative nature of such undertakings (e.g. Liberati et al. 2009; Tranfield et al. 2003; Pino et al. 2008; Pawson et al. 2005) and we advise that protocols for systematic reviews in LCA and any associated meta-analysis methods be trialed for usability by the study group, reviewed, discussed, and revised as indicated during the course of the review. Any revisions should be outlined and commented upon in subsequent publications and the protocol described should be transparent and complete enough to allow for replication.

Findings and features of the individual studies in the review

In order to allow critical interpretation and an understanding of the basis of the review study, key information about the individual studies or sources of data included in the review should be provided. The quality of any review or meta-analysis is dependent on the individual studies upon which it is built. Assisting the reader in understanding the important data, flaws, and strengths of the individual studies is key to a robust and useful review and provides context for the reader that facilitates appropriate application of the review study results. Assembling key elements of the individual studies can be complex and presenting the characteristics or results in a visual format (e.g. table format or plots) can be helpful (Liberati et al. 2009). Summarizing reviewer insights regarding strengths and weakness of individual studies, including elements such as confidence intervals, distributions, and individual study limitations, also assists the reader in evaluating and understanding the steps from individual studies to systematic review conclusions.

Assessment of bias

Merriam Webster's Collegiate Dictionary (2012) defines the concept of bias in several ways including, “bent, tendency,” “an inclination of temperament or outlook especially a personal and sometimes unreasoned judgment,” and “systematic error introduced into sampling or testing by selecting or encouraging one outcome or answer over others.” As human beings we naturally have affiliations and views that unconsciously shape our interpretation of data and the conclusions we draw. Understanding the presence and extent of bias in research is a difficult but important task. When performing a systematic review there are at least two levels of bias that need to be assessed: bias within each original study included in the review and bias occurring across the summarized studies.

Bias within studies

Bias at the level of individual studies often relates to selection of study methods and data interpretation. In LCA, sources of bias include data sources (Mattila et al. 2010), categorization, normalization, or weighting (Reap et al. 2008), as well as variability in the accuracy of models used in the assessment (Finnveden et al. 2009). This form of bias is important to recognize as it can impact the interpretation of a study, though the presence of bias does not necessarily invalidate a study.

Bias across studies

Publication bias is often the main type of bias across studies; it can be a concern when published studies do not accurately reflect the actual range of studies that have been performed (Ioannidis 2008). For example, studies with results that are deemed statistically significant may be more likely to be submitted and accepted for publication over findings that are not statistically significant, and there may be political obstacles to submitting articles that conflict with current practices (Johnson and Dickersin 2007; Taubes 1998). However, assessing variation in LCA data sources, such as LCA databases, can carry additional challenges given the frequent lack of uncertainty estimates. Published LCAs that are included in a systematic review may sometimes be different at face value, but not necessarily significantly different statistically.

Additional sources of bias across studies, such as failing to locate appropriate reports to include and the effects of financial conflicts of interest, should be evaluated as part of the review process; this category of bias can often be minimized by utilizing a well-designed review protocol (Miller 2000). The funding source for the review should be disclosed in publication to allow the reader to evaluate potential bias from conflict of interest. Our understanding of sources and effects of bias continues to evolve in LCA. Additional work to better conceptualize bias specific to LCA would be useful.

Synthesis methods (qualitative and quantitative)

Reviews may utilize a wide range of methods to synthesize the best available evidence in answering a focused question. Approaches range from qualitative work such as grouping and summarizing of expert opinions, to quantitative synthesis, in which the published data are adjusted to a common value or other statistical methods are utilized as part of a meta-analysis. There is also value in understanding what adjustments are utilized in a meta-analysis to “even the playing field” and compare a set of otherwise different studies using a new common set of parameters. The authors may comment on the generalization of the parameters utilized in harmonization or a meta-analysis, as well as how the adjusted parameters themselves influence or provide insight into study variability. Whatever synthesis methods are undertaken in a review, they should be clearly described and justified based on the study protocol and the available data included in the review.

Limitations of the review

In order to assist the reader in appropriate interpretation of the review results it is important to highlight and discuss limitations in any review publication. Reviews in LCA are likely to have parameters defined by factors including time frame, geography, and technology type, and so it is helpful for the authors to discuss where their findings likely can and cannot be applied based on the scope of the review. Limitations are also generally an excellent opportunity to highlight important questions for further research.

Summary of findings and conclusions

The crux of a review is the summary of evidence that was found to provide the best possible answer to the PIFT-formatted question posed at the start of the systematic review. The ultimate goal is to provide a general interpretation of the results in the context of other evidence. Specific points relative to practical problem solving or setting policy may be especially useful to highlight.

Summary of recommendations for the structure of reporting systematic reviews in LCA

Guidelines and checklists are needed

The amount of data generated from life cycle assessment methodology is steadily increasing; reconciling conflicts between data sets and leveraging existing data to answer new questions are important tasks for advancing decision making capacity related to LCA. Systematic reviews of the literature, with or without meta-analysis, can help summarize the existing data in meaningful ways to better answer existing questions or answer altogether new questions.

While the International Organization for Standardization provides a framework for conducting LCA studies in general, there is currently no direct guidance about best practices in conducting systematic reviews in LCA. While there are a variety of reviews that have been published in LCA, there is great variability in methodology and reporting, which makes use of the information difficult. Other disciplines have successfully applied the systematic review structure common in biomedical research; we introduce the STARR-LCA checklist as a starting point for standardizing systematic reviews in LCA.

STARR-LCA checklist

The nine items on the STARR-LCA checklist provide many tangible recommendations for improving the state of systematic reviews in LCA. Article titles should clearly identify the manuscript as a review with or without meta-analysis, and keyword listings for both individual reports and database structure should include the terms meta-analysis and systematic review. A structured summary or abstract should be provided at the beginning of a review; discussions regarding an editorial standard for structured summaries in LCA reviews would be worthwhile. The reason for the systematic review should be clear in the context of what is already known.

An answerable question should be the foundation for a review – and while question articulation can be an iterative process, a focused question should be drafted at the start of the systematic review process. We recommend using our PIFT question format. Additional question parameters such as the technology type, geographic region, or time may be relevant depending on the research question. A systematic review protocol should be prospectively written to guide researchers and, ideally, made available for peer review with key areas as noted above.

We also recommend an assessment of bias both for each study included in the reviewand across the entire review. Potential sources of bias include data, modeling parameters, valuation decisions, publication bias, and incomplete review of the relevant literature, as well as direct conflicts of interest. Bias based on methodology does not necessarily invalidate a study, but it is important to report and understand in order to optimize appropriate interpretation of the study. Detection and minimization of bias in LCA is a new concept worthy of ongoing discussion.

We acknowledge the wide range of both qualitative and quantitative synthesis methods that may be used. We recommend that, in the interest of transparency, methods be clearly described and any new data presented as part of publication. We also recommend that authors specifically highlight the limitations of the systematic review as a way to assist the reader in result interpretation and highlight important questions for future research. Finally, the conclusions of the review should be clearly stated.

Our STARR-LCA checklist provides a framework to consider the elements of a well-designed review. The intent of the STARR-LCA checklist is to facilitate advancements in designing, conducting and reporting systematic reviews, including meta-analyses, in LCA. We look forward to further discussion of the STARR-LCA checklist items through input from the LCA community, including advancements in LCA methodology.

Supplementary Material

S1

Supporting Information: Additional supporting information may be found in the online version of this article:

Supporting Information S1: This supporting information provides a sample STARR-LCA worksheet for use in study planning, manuscript preparation, and peer review.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

Acknowledgments

Support for this work was provided by the Rehabilitation Medicine Scientist Training Program through the National Center for Medical Rehabilitation Research, grant 7K12HD001097-13.

Footnotes

://

Supporting information is available on the JIE Web site.

Contributor Information

Jennifer M. Zumsteg, University of Washington, Department of Rehabilitation Medicine in Seattle, WA, USA.

Joyce S. Cooper, University of Washington, Department of Mechanical Engineering.

Michael S. Noon, University of Washington, Department of Civil and Environmental Engineering.

References

  1. Anex RP, Focht W. Public participation in life cycle assessment and risk assessment: A shared need. Risk Analysis. 2008;22(5):861–877. doi: 10.1111/1539-6924.00257. [DOI] [PubMed] [Google Scholar]
  2. Arvizu D, Bruckner T, Chum H, Edenhofer O, Estefen S, Faaij A, Fischedick M, et al. Technical summary. In: Edenhofen O, Pichs-Madruga R, Sokona Y, Seyboth K, Matschoss P, Kadner S, Zwickel T, Eickemeier P, Hansen G, Schlömer S, von Stechow C, editors. IPCC special report on renewable energy sources and climate change mitigation. Cambridge University Press; Cambridge, United Kingdom and New York, NY, USA: 2001. [Google Scholar]
  3. Baldassarre MT, Boffoli N, Caivano D, Visaggio G. A hands-on approach for teaching systematic review. Computer Science. 2008;5089:415–426. [Google Scholar]
  4. Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: How will we ever keep up? PLoS Medicine. 2010;7(9):e1000326. doi: 10.1371/journal.pmed.1000326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Bhat V, Prakash R. LCA of renewable energy for electricity generation systems - A review. Renewable and Sustainable Energy Reviews. 2008;13(5):1067–1073. [Google Scholar]
  6. Booth A, Fry-Smith A. Developing the research question. In: Topfer L, Auston I, editors. Etext on Health Technology Assessment (HTA) Information Resources. National Information Center on Health Services and Health Care Technology; Bethesda, Maryland: 2003. [Accessed November 2011]. www.nlm.nih.gov/nichsr/ehta. [Google Scholar]
  7. Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. The Lancet. 2011;377(9760):108–109. doi: 10.1016/S0140-6736(10)60903-8. [DOI] [PubMed] [Google Scholar]
  8. Carnegie Mellon University Green Design Institute. Economic input-output life cycle assessment (EIOLCA), US 1997 industry benchmark model. [Accessed November 2011];2008 www.iolca.net.
  9. Centre for Reviews and Dissemination. Systematic reviews: CRD's guidance for undertaking reviews in health care. Centre for Reviews and Dissemination; York, U.K.: 2009. [Google Scholar]
  10. Cherubini F, Bird N, Jungmeier G, Schlamadinger B, Woess-Gallasch S. Energy- and greenhouse gas-based LCA of biofuel and bioenergy systems: Key issues, ranges and recommendations. Resources, Conservation and Recycling. 2009;53(8):434–447. [Google Scholar]
  11. Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Annals of Internal Medicine. 1997;127:380–387. doi: 10.7326/0003-4819-127-5-199709010-00008. [DOI] [PubMed] [Google Scholar]
  12. de Vries M, de Boer I. Comparing environmental impacts for livestock products: a review of life cycle assessments. Livestock Science. 2010;128(1-3):1–11. [Google Scholar]
  13. Dickersin K, Higgins K, Meinert CL. Identification of meta-analyses. The need for standard terminology. Controlled Clinical Trials. 1990;11(1):52–66. doi: 10.1016/0197-2456(90)90032-w. [DOI] [PubMed] [Google Scholar]
  14. Earles JM, Halog A. Consequential life cycle assessment: a review. The International Journal of Life Cycle Assessment. 2011;16:445–453. [Google Scholar]
  15. Farrell AE, Plevin RJ, Turner BT, Jones AD, O'Hare M, Kammen DM. Ethanol can contribute to energy and environmental goals. Science. 2006;311(5760):506–508. doi: 10.1126/science.1121416. [DOI] [PubMed] [Google Scholar]
  16. Finnveden G, Hauschild M, Ekvall T, Guinee J, Heijungs R, Hellweg S, Koehler A, Pennington D, Suh S. Recent developments in life cycle assessment. Journal of Environmental Management. 2009;91(1):1–21. doi: 10.1016/j.jenvman.2009.06.018. [DOI] [PubMed] [Google Scholar]
  17. Fraser AG, Dunstan FD. On the impossibility of being expert. BMJ. 2010;341:c6815. doi: 10.1136/bmj.c6815. [DOI] [PubMed] [Google Scholar]
  18. Gates S. Review of methodology of quantitative reviews using meta-analysis in ecology. Journal of Animal Ecology. 2002;71:547–557. [Google Scholar]
  19. Ghapanchi AH, Aurum A. Antecedents to IT personnel's intentions to leave: A systematic literature review. Journal of Systems and Software. 2011;84(2):238–249. [Google Scholar]
  20. Hedges LV. Review of research in education. American Educational Research Association; Washington, D.C.: 1986. Issues in Meta-Analysis. [Google Scholar]
  21. Higgins J, Green S, editors. The Cochrane Collaboration; 2011. [Accessed November 2011]. Cochrane handbook for systematic reviews of interventions. Version 5.1.0.2011. www.cochrane-handbook.org. [Google Scholar]
  22. Holloway R, Nesbit K, Bordley D, Noyes K. Teaching and evaluating first and second year medical students' practice of evidence-based medicine. Medical Education. 2004;38(8):868–878. doi: 10.1111/j.1365-2929.2004.01817.x. [DOI] [PubMed] [Google Scholar]
  23. Hunkeler D, Rebitzer G. The future of life cycle assessment. The International Journal of Life Cycle Assessment. 2005;10(5):305–308. [Google Scholar]
  24. Ioannidis J. Interpretation of tests of heterogeneity and bias in meta-analysis. Journal of Evaluation in Clinical Practice. 2008;14(5):951–957. doi: 10.1111/j.1365-2753.2008.00986.x. [DOI] [PubMed] [Google Scholar]
  25. ISO (International Organization for Standardization) Environmental management – life cycle assessment – principles and framework. Geneva: ISO; 2006. ISO 14040:2006. [Google Scholar]
  26. ISO. Environmental management – life cycle assessment – requirements and guidelines. Geneva: ISO; 2006. ISO 14044:2006. [Google Scholar]
  27. Johnson RT, Dickersin K. Publication bias against negative results from clinical trials: three of the seven deadly sins. Nature Clinical Practice Neurology. 2007;3(11):590–591. doi: 10.1038/ncpneuro0618. [DOI] [PubMed] [Google Scholar]
  28. Kettenring KM, Adams CR. Lessons learned from invasive plant control experiments: a systematic review and meta-analysis. Journal of Applied Ecology. 2011;48(4):970–979. [Google Scholar]
  29. Khaddaj-Mallat C, Rousset JM, Ferrant P. The transient and progressive flooding stages of damaged ro-ro vessels: A systematic review of entailed factors. Journal of Offshore Mechanics and Arctic Engineering. 2011;133(3) [Google Scholar]
  30. Kubiszewski I, Cleveland CJ, Endres PK. Meta-analysis of net energy return for wind power systems. Renewable Energy. 2010;35(1):218–225. [Google Scholar]
  31. Larson E. A review of life-cycle analysis studies on liquid biofuel systems for the transport sector. Energy for Sustainable Development. 2006;X(2):109–126. [Google Scholar]
  32. Lebrun JL. Scientific writing: a reader and writer's guide. Singapore: World Scientific Publishing Co, Pte. Ltd.; 2007. The heart of your paper. Abstract. [Google Scholar]
  33. Lemming G, Hauschild M, Bjerg P. Life cycle assessment of soil and groundwater remediation technologies: literature review. The International Journal of Life Cycle Assessment. 2010;15:115–127. [Google Scholar]
  34. Lenzen M, Munksgaard J. Energy and CO2 life-cycle analyses of wind turbines - review and applications. Renewable Energy. 2002;26(3):339–362. [Google Scholar]
  35. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009a;339:b2700. doi: 10.1136/bmj.b2700. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Mattila T, Pakarinen S, Sokka L. Quantifying the total environmental impacts of an industrial symbiosis - a comparison of process-, hybrid and input-output life cycle assessment. Environmental Science & Technology. 2010;44(11):4309–4314. doi: 10.1021/es902673m. [DOI] [PubMed] [Google Scholar]
  37. Merriam Webster's Collegiate Dictionary. 11th edition. Online version. [Accessed January 2012]; www.merriam-webster.com.
  38. Miller J. Applying meta-analytical procedures to software engineering experiments. Journal of Systems and Software. 2000;54(1):29–39. [Google Scholar]
  39. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. doi: 10.1136/bmj.b2535. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Mondelaers K, Aertsens J, Van Huylenbroeck G. A meta-analysis of the differences in environmental impacts between organic and conventional farming. British Food Journal. 2009;111(10):1098–1119. [Google Scholar]
  41. Montori VM, Wilczynski NL, Morgan D, Haynes RB. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ. 2005;330(7482):68–73. doi: 10.1136/bmj.38336.804167.47. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Moomaw W, Burgherr P, Heath G, Lenzen M, Nyboer J, Verbruggen A. Annex II: Methodology. In: Edenhofer O, Pichs-Madruga R, Sokona Y, Seyboth K, Matschoss P, Kadner S, Zwickel T, Eickemeier P, Hansen G, Schlömer S, von Stechow C, editors. IPCC special report on renewable energy sources and climate change mitigation. Cambridge University Press; Cambridge, United Kingdom and New York, NY, USA: 2011. [Google Scholar]
  43. Neely GJ, Magit A, Rich J, Voelker C, Wang E, Paniello R, Nussenbaum B, Bradley J. A practical guide to understanding systematic reviews and meta-analyses. Otolarygology – Head and Neck Surgery. 2009;142(1):6–14. doi: 10.1016/j.otohns.2009.09.005. [DOI] [PubMed] [Google Scholar]
  44. Ortiz O, Castells F, Sonnemann G. Sustainability in the construction industry: A review of recent developments based on LCA. Construction and Building Materials. 2009;23(1):28–39. [Google Scholar]
  45. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review - a new method of systematic review designed for complex policy interventions. Journal of Health Services Research & Policy. 2005;10(Suppl 1):21–34. doi: 10.1258/1355819054308530. [DOI] [PubMed] [Google Scholar]
  46. Pino FJ, Garcia F, Piattini M. Software process improvement in small and medium software enterprises: a systematic review. Software Quality Journal. 2008;16:237–261. [Google Scholar]
  47. Pullin AS, Stewart GB. Guidelines for systematic review in conservation and environmental management. Conservation Biology. 2006;20(6):1647–1656. doi: 10.1111/j.1523-1739.2006.00485.x. [DOI] [PubMed] [Google Scholar]
  48. Reap J, Roman F, Duncan S, Bras B. A survey of unresolved problems in life cycle assessment. The International Journal of Life Cycle Assessment. 2008;13:374–388. [Google Scholar]
  49. Rowe R, Whitaker J, Chapman J, Howard D, Taylor G. Prepared for UKERC and TSEC-BIOSYS. [Accessed November 2011];Life cycle assessment in the bioenergy sector: developing a systematic review. 2008 www.nora.nerc.ac.uk/5099/
  50. Sackett DL, Rosenbery WM, Gray MJ, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996;312:71–72. doi: 10.1136/bmj.312.7023.71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. Salleh N, Mendes E, Grndy J. Empirical studies of pair programming for CS/SE teaching in higher education: A systematic literature review. IEEE Transactions on Software Engineering. 2011;37(4):509–525. [Google Scholar]
  52. Santero N, Masanet E, Horvath A. Life-cycle assessment of pavements. Part I: Critical review. Resources, Conservation and Recycling. 2011;55(9-10):801–809. [Google Scholar]
  53. Sartori I, Hestnes A. Energy use in the life cycle of conventional and low-energy buildings: A review article. Energy and Buildings. 2007;39(3):249–257. [Google Scholar]
  54. Sovacool B. Valuing the greenhouse gas emissions from nuclear power: A critical survey. Energy Policy. 2008;36(8):2950–2963. [Google Scholar]
  55. Staples M, Niazi M. Experiences using systematic review guidelines. Journal of Systems and Software. 2007;80(9):1425–1437. [Google Scholar]
  56. Sunde K, Brekke A, Solberg B. Environmental impacts and costs of woody Biomass-to-Liquid (BTL) production and use - A review. Forest Policy and Economics. 2011;13(8):591–602. [Google Scholar]
  57. Taubes G. The (political) science of salt. Science. 1998;281(5379):898–907. doi: 10.1126/science.281.5379.898. [DOI] [PubMed] [Google Scholar]
  58. Torgerson C. Systematic reviews. London: Continuum International Publishing Group; 2003. The stages of a systematic review. [Google Scholar]
  59. Tranfield D, Denyer D, Smart P. Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management. 2003;14:207–222. [Google Scholar]
  60. United States. Energy independece and security act of 2007. Washington, D.C.; 2007. U.S. G.P.O. AE 2.110:110-140. [Google Scholar]
  61. van Tulder M, Furlan A, Bombardier C, Bouter L. Updated method guidelines for systematic reviews in the Cochrane collaboration back review group. Spine. 2003;28(12):1290–1299. doi: 10.1097/01.BRS.0000065484.95996.AF. [DOI] [PubMed] [Google Scholar]
  62. Villanueva A, Wenzel H. Paper waste - Recycling, incineration or landfilling? A review of existing life cycle assessments. Waste Management. 2007;27(8):S29–S46. doi: 10.1016/j.wasman.2007.02.019. [DOI] [PubMed] [Google Scholar]
  63. von Blottnitz H, Curran MA. A review of assessments conducted on bio-ethanol as a transportation fuel from a net energy, greenhouse gas, and environmental life cycle perspective. Journal of Cleaner Production. 2007;15(7):607–619. [Google Scholar]
  64. von Falkenstein E, Wellenreuther F, Detzel A. LCA studies comparing beverage cartons and alternative packaging: can overall conclusions be drawn? The International Journal of Life Cycle Assessment. 2010;2010(15):938–945. [Google Scholar]
  65. Werner F, Richter K. Wooden building products in comparative LCA: A literature review. The International Journal of Life Cycle Assessment. 2007;12(7):470–479. [Google Scholar]
  66. Whitaker J, Ludley KE, Rowe R, Taylor G, Howard DC. Sources of variability in greenhouse gas and energy balances for biofuel production: a systematic review. GCB Bioenergy. 2010;2010(2):99–112. [Google Scholar]
  67. Zhang Y, Singh S, Bakshi B. Accounting for ecosystem services in life cycle assessment, Part I: A critical review. Environmental Science & Technology. 2010;44(7):2232–2242. doi: 10.1021/es9021156. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1

Supporting Information: Additional supporting information may be found in the online version of this article:

Supporting Information S1: This supporting information provides a sample STARR-LCA worksheet for use in study planning, manuscript preparation, and peer review.

Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

NIHMS448905-supplement-S1.pdf (131KB, pdf)

RESOURCES