Guidelines for performing systematic reviews in sports science

Abstract

Most of the reviews carried out in sports science have used the general items suggested by Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA). Due to the specific requirements of each knowledge area, several modifications of the PRISMA are necessary to optimize the process of the systematic reviews and, in consequence, the quality of the conclusions provided in this type of study. Therefore, this work aimed to adapt PRISMA to provide specific guidelines to carry out systematic reviews in sports science. The methodology criteria (search strategy, databases, and eligibility) and the results section (flow diagrams and study contents) were adapted based on previous studies, and several new considerations were added to design the new guidelines. We compiled 28 items suggested by sports science researchers and included two new items: (i) population/problem (i.e., age, level, and country) and (ii) the entire training process, which is monitored and compared between groups (e.g., total training load). To maximize the benefit of this document, we encourage people to read it in conjunction with the PRISMA statement. The main differences between PRISMA and the PRISMA adapted to sports science were related to registration, search strategy, flow diagrams, and results. Application of the new guidelines could improve the information provided to readers and make it easier to generalize and compare the results in sports science.

INTRODUCTION

Mixed methods research is defined as research in which the investigator collects and analyses data, integrates the findings, and draws inferences using both qualitative and quantitative approaches or methods in a single study or programme of inquiry [1]. An example of this type of research is the systematic review. Systematic reviews arise from a field of the literature that contains hundreds of thousands of studies for people involved in providing care to identify and consider when making decisions [2]. In addition, the conclusions of a systematic review could suppose the justification for further research [3]. So, the collection, analysis, integration, and concluding remarks will be of interest to all researchers and others involved in this field.

The value of a systematic review depends on what was done, what was found, and the clarity of reporting [3]. Thus, although the reporting quality of the review paper depends on the authors’ ability to conduct the review, reporting items for systematic reviews have to warrant that the methodology is suitable and, subsequently, that the data are reliable. To date, Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) (formerly, QUOROM) [3–6], with the terminology used by the Cochrane Collaboration guidelines [7], is a widely used set of guidelines for performing systematic reviews. Realizing these issues, an international group that included 29 experienced authors, methodologists, clinicians, medical editors, and consumers developed PRISMA as an evolution of the original QUOROM guidelines for systematic reviews and meta-analyses of evaluations of health care interventions [8].

Due to the specific requirements of each knowledge area, several modifications (e.g., the number of groups in the search strategy or the quality checklist) were made to meet the requirements of each research field. Even though the researchers of sports performance areas did not follow the PRISMA guidelines [9, 10] or any section of it [11], most of the reviews have used the general items suggested by PRISMA [3]. So, it seems pertinent that specific guidelines for sports science adapt and summarize several aspects, such as a quality assessment checklist or flowchart standards on different systematic reviews. This could help sports science researchers to optimize the process of systematic reviews and, in consequence, improve the quality of conclusions provided in this type of study [2]. This work aimed to adapt PRISMA to provide specific guidelines for carrying out systematic reviews in sports science.

GUIDELINES FOR SYSTEMATIC REVIEWS IN SPORTS SCIENCE

Methodology

Identification

The importance of reporting a clear objective for conducting the project is crucial due to its likely influence on the methodology. Once this is done, locating and retrieving the relevant literature is challenging yet crucial to the success of a systematic review [2]. In this regard, the material sources or databases provide the articles and information from which evidence, conclusions, and future considerations are drawn [2, 8]. However, the literature may appear overwhelming [3], given the vast amount of material to study. Therefore, a systematic strategy is necessary before starting the search process. This strategy should address the problem that the authors want to summarize. In a systematic review of individual studies, the database selection and search strategy should be as varied as possible to maximize the likelihood of capturing all relevant data and minimizing the effects of reporting biases [2].

According to recently updated guidance, a search of a minimum number (not just at least one database) of electronic databases is recommended [6]. However, they should be chosen according to the topic of interest. In sports science, a wide range of databases has been used (Table 1). Although it does not seem that each author uses the same databases, PubMed and Web of Sciences (WoS) appear to be the most commonly used in sports settings (Table 1). Beyond this selection, each database has several criteria to limit the search. In this sense, applying period or language restrictions is not recommended, although if there is a lack of access to translation services, the research is usually limited to English articles [2]. This fact has been named LOTE (not including languages other than English), and although it often seems the most feasible option, it introduces the risk of ignoring key data, introducing bias (referred to as language bias), as well as missing important cultural contexts, which may limit the review’s findings and usefulness [12]. So, the period restriction could be recommended only for some specific cases, while the language restriction could be limited more often to English because most of the high impact journals publish their articles in this language, although it should be used only when language resources are unavailable (e.g. professional translators). When it is applied, it should be appropriately justified with consideration of the potential consequences of language restriction in the discussion, which aligns with the PRISMA Statement (Item 6: Eligibility criteria, and Item 25: Limitations of the review process) [12].

TABLE 1.

Methodology specification in sport systematic reviews.

Ref.	Quality Checklist	Guideline	Search Strategy	PubMed	WoK	MEDLINE	Scopus	SPORTDiscus	Dialnet	CINAHL	ProQuest	EMBASE	Cochrane	Google Scholar	Scielo	Ebsco	PsycINFO	Teseo	OneSearch	ScienceDirect	AMED	Informit
Low et al., [13]	Sarmento et al. [14]	PRISMA (with modification: 2 groups)	Population (sport); outcomes	X	X
Rico-González et al., [15]	Sarmento et al. [14]	Yes (with modification: PIO)	Population (sport); technology; outcomes	X	X			X			X
Hader et al., [16]	Ad hoc	PRISMA	population; intervention; comparators; outcome variables; study design.	X	X			X
Agras et al., [17]	-	-	Population (sport); outcomes	X	X	X	X	X	X	X
Palucci Vieira et al., [18]	Castellano et al., [19]	PRISMA	Running performance variables; movement category (intensity)	X	X			X							X
Preciado et al. [20]	GREOM [21]	PRISMA	Groups not specified		X		X		X		X			X	X
Silva et al., [22]	Ad hoc	PRISMA	Population; interventions; comparators; outcomes and study designs	X				X
Sarmento et al., [23]	-	PRISMA	Population (sport); analyses		X
Ramirez-Campillo et al., [24]	PEDro	Cochrane collaboration guidelines and findings with PRISMA	Population (sport); training; intervention (trials)	X	X	X	X
Oglesby et al., [25]	CASP	PRISMA (only for Flowchart)	Population; outcomes					X											X
Vachon et al. [26]	PEDro	QUORUM [4]and PRISMA	Intervention; outcomes		X		X	X
Petway et al., [27]	PEDro	PRISMA	Population (sport); outcomes	X	X			X					X
Grgic et al., [28]	PEDro	PRISMA	Nutrients; intervention	X	X	X	X	X
Natera et al. [9]	PEDro	-	Type of training; intervention; outcomes	X				X		X		X
Blazevich et al., [11]	Cochrane risk of bias [29]	PRISMA for meta analysis	PICOS Problem/intervention; control/comparison; outcome, study design	X	X								X
Naughton et al., [10]	Any criteria from Downs and Black [30]		Population; intervention; outcomes	X	X		X	X
Colomer et al., [31]	No applicable (Quality study)	PRISMA	Population (sport); outcomes					X
Zouhal et al., [32]	PEDro	PRISMA (use PICOS for inclusion criteria)	Population (sport); intervention; outcomes	X	X			X
Nygaard Falch et al., [33]	-	-	Several key words with non-defined mixed method	X				X					X
Androulakis-Korakakis et al. [34]	PEDro	PRISMA	Problem; intervention (2 groups)	X				X						X
Silva et al., [35]	PEDro (Rehabilitation)	PRISMA	Intervention; outcomes	X	X		X
Jansson et al., [36]	Nutrition and Dietetics Quality Criteria Checklist: Primary Research Tool [37]				X	X	X	X								X	X					X
Griffin et al. [38]	Newcastle–Ottawa Scale (NOS) [39]	PRISMA	Problem; intervention			X	X	X		X		X									X
Ehlert & Wilson [40]	PEDro	PRISMA	Population; intervention; outcomes	X	X			X
King et al., [41]	STROBE [42] AND PRISMA	PRISMA AND STROBE	Population; outcomes	X	X		X			X	X					X				X
Applied technology
Rico-González et al., [43]	-	PRISMA (with modification PIO)	3 groups of research: (population (sport); technology; outcomes)	X	X			X			X
Cummins et al., [44]	Ad hoc: Adapted from Downs and Black	-	Population; activity		X	X	X	X		X		X	X	X
Migueles et al. [45]	Ad hoc	PRISMA (with modification)	ActiGraph GT3X; model	X	X
Rago et al., [46]	Adapted from [19] and [47]	PRISMA	Study design; participants; interventions; outcomes; timing; setting	X	X		X									X
Sarmento et al., [14]	Sarmento et al., [23]	PRISMA	Populations (sport); analyses		X
Castellano et al., [19]	Ad hoc	-				X		X			X						X	X
Altmann et al., [48]	Ad hoc. Modified of critical appraisal tool [49]	PRISMA	Population; intervention (test); outcomes (validity; reliability)	X	X

Note = WoK = Web of Science or Web of Knowledge. *The study selection was made randomized between recently published articles in high impact sport journals.

In Table 1, thirty systematic review examples were randomly chosen as proof of concept, highlighting the considered checklist, guidelines, search strategy, and databases.

Step number four of PRISMA’s checklist explains that the authors should provide an explicit statement of questions being addressed concerning participants, interventions, comparisons, outcomes, and study design (PICOS). Issues relating to PICOS impact several PRISMA items (i.e., items 6, 8, 9, 10, 11, and 18) [8]. This approach suggests a division of the search strategy in different groups linked with a Boolean search (e.g., “AND”, “OR”, “NOT”), which has become an essential and suitable method. “AND” has commonly been used to link the different groups, and “OR” to link a word into each word cluster; both have become the most popular Boolean words. Researchers should also consider NOT excluding some specific words from the search. However, the use of more than three groups is not widespread in sports science (Table 1), although in many studies, PICOS has been used to compute five inclusion/exclusion criteria [32]. In sports science, the use of Population (means = population, sport, or a problem) is recommended as a pivotal research word group. This group may be combined with: (1) the Intervention group (if the research question highlights the consequences of this programme); (2) with Outcomes arising (if the research question enhances the Intervention programme used for specific outcomes); or, (3) with both groups (if the systematic review aims to summarize a topic about, specifically, Population, Intervention programme, and Outcomes). As in other knowledge areas, to minimize the risk of missing relevant articles, checking a reference list of articles captured by the initial search to ensure that keywords of the search strategy are commonly used in the titles and abstracts is recommended.

A review team should be established before commencing the screening phase in which individual analysis will be made. The review team should include at least one person with methodological expertise in conducting systematic reviews and at least one person with expertise on the topic under review [2]. This group should define databases, search strategies, and inclusion/exclusion criteria.

Screening

When the referred authors complete the search, and avoiding manual search processes in order to encourage researchers to perform a more efficient process, they automatically download the main data from the articles (title, authors, date, and database) to an Excel spreadsheet (Microsoft Excel, Microsoft, Redmond, USA). For example, the steps to automatically download data in some databases in sports science are as follows:

PubMed: This includes the following steps: (1) send to; (2) file; (3) format: CSV; (4) create file; (5) open an Excel document; (6) select “import”; (7) “delimited by”: select the characters that separate title, authors, journal name and year of publication, and the information will be provided in columns (in PubMed: comma).

Web of Sciences: (1) export; (2) other file format; (3) select the number of files that can be downloaded (clusters of 500 documents if the search is up to 500 files); (4) tab delimited format; (5) send; (6) open an Excel document; (7) select “import”; (8) “delimited by”: select the characters that separate title, authors, journal name and year of publication, and the information will be provided in columns (in WoS: tab).

SPORTDiscus: (1) select for each page 100 files; (2) results (1–100); (3) folder view; (4) export; (5) CSV. In SPORTDiscus, the authors should download characteristics of papers in clusters of 100 articles; (6) open an Excel document; (7) select “import”; (8) “delimited by”: select the characters that separate title, authors, journal name and year of publication, and the information will be provided in columns.

Cochrane Library: (1) select all; (2) export references; (3) download; (6) open an Excel document; (7) select “import”; (8) “delimited by”: select the characters that separate title, authors, journal name and year of publication, and the information will be provided in columns (in Cochrane: comma-separated).

Eligibility

When the articles are downloaded, all of them should appear in an Excel page in five columns: (1) database; (2) authors; (3) title; (4) source; and (5) year. Selecting the first file (with these five nouns), the authors should choose “filter”. Then, two authors should order the columns of titles in alphabetic order and remove the duplicate records, using the first white column with a numeric code (e.g., 1 for original articles; 2 for duplicates). In the second column, each author should attribute to each paper a number code according to the hierarchical inclusion/exclusion criteria (e.g., 1 = included; 2 = other population; 3 = other intervention programmes). When both authors finish the selection phase, they then compare their results to ensure that the same number of articles was found. Any disagreements regarding the final inclusion/exclusion status should be resolved through a discussion between the reference authors.

Quality assessment

A PRISMA checklist is not a quality assessment instrument intended to gauge the worth of a systematic review [3]. In the articles, an additional assessment checklist is presented. Due to the qualitative and quantitative studies that can be used, the quality assessment is not mandatory for all articles. Specifically, studies based on a recognized classification method as the nature of the search are descriptive [15, 31, 43]. In this regard, Colomer et al. [31] suggested a table for suitability and evaluation by a panel of authors before inclusion. All studies had to meet every item on the criteria list to be included in the analysis [31].

However, to report a quantitative result and conclusion, the individual assessment of the studies is a mandatory step in conducting systematic reviews. In this regard, several checklists have been proposed for health care (i.e., the Downs and Black scale [30], PEDro scale [50], Newcastle-Ottawa (NOS) [39], a Cochrane Collaboration tool (RoB 2) [29, 51], and CASP [52]), for nutrition and dietetics [36], as well as for physiotherapy/rehabilitation [35]. In addition, two protocols (STROBE [42]) and GREOM [21]) have been proposed for conducting analyses with an observational methodology. In the sports science area, several authors have chosen and adapted these scales to their articles [14, 23, 44]. For example, Sarmento et al. [14, 23] and Cummins et al. [44] modified the Downs and Black (1998) proposal. Other studies propose the use of ad hoc tools to assess the studies included in the systematic review [16, 19, 46].

The development of the technology applied in sports science has contributed to the exponential growth of published papers. Today, technology is used in most quantitative articles. However, none of the aforementioned checklists assessed the use of such technology. Altmann et al. [48] published a quality checklist about the use of technology, and Adessida et al. [53] proposed a checklist for microelectromechanical systems (MEMS). Recently, Rico-González et al. [54] published an article on specific instruments to assess the use of radio-frequency technology and MEMS and evaluated the quality of the studies individually. In addition to the previous guidelines, we have used these technology surveys and suggested new items to summarize a checklist to assess the quality of the research in sports science (Table 2).

TABLE 2.

A standardization proposal of a quality assessment checklist in sports science.

Journal section	Where is the item from?	Item	Assessment score
Specific criteria
Journal	Original studies [16, 19, 46]	The study was published in an indexed, peer-reviewed journal.
Aim	Original studies [14, 19, 46], CASP, NOS AND PROPERO	The study objective(s) was stated clearly, including any prespecified hypotheses.
Title/Abstract	STROBE	Indicate the study’s design with a commonly used term in the title or the abstract.
	STROBE	Provide in the abstract an informative and balanced summary of what was done and what was found.
Introduction	Original study [14]	The importance of the problem is reported.
	Original study [14], CASP and STROBE	The relevant background literature was reviewed.
Method	NEW ITEM	The population/problem was well defined (i.e., age, level, and country).
	Original study [14]	The design was appropriate for the research question.
	Original studies [19, 46], NOS and PROSPERO	The duration of data collection, number of sessions/matches, and number of individual recordings are specified (representativeness of the cases).
	Original study [19]	The duration of the recordings is clearly detailed.
	[14]	Was informed consent obtained? (If not described, then assume no.)
	PROSPERO	The variables are justified, and they have been clearly defined.
	PROSPERO	The validity and reliability of the evaluation/assessment tool were provided ( if applicable ).
	Original studies [19, 46]	Certain contextual variables (e.g., match status, match location, type of competition, or quality of the opponent) are considered.
	Original study [14] and PROSPERO	Were any drop-outs reported (if applicable)?
Technology used	[54]	The technology guideline (i.e., GPS/LPS) was used (% of data used reported) (if applicable).
	Original study [48]	What is the % of data reported based on the use of VID (if applicable)?
Results	Original study [14] and CASP	The analysis method was appropriate, and the results were reported in terms of statistical significance (how precise are the results?). In other words, the specific value is reported, not only p < or > than 0.001, 0.01, or 0.05.
	Original study [14] and CASP	The results are reported in terms of practical differences, and the specific value is reported (not only interpretation intervals).
	Original study [14] and CASP	The numeric results are carried out (not only using figures).
Conclusions	Original study [14] and CASP	Were all important conclusions considered?
Practical application	Original study [14]	Are there any implications for practice given the results of the study?
Study limits	Original study [14] and PROSPERO	Give a cautious overall interpretation of results. (Were generalizability ( external validity) and/or study limitations considered? )
Specific criteria for cross-sectional studies
Results for cross-sectional studies	PEDro	Sample inclusion/exclusion criteria were specified.
	PROSPERO	Describe the comparability of assessment methods.
	PEDro and RoB 2	Groups’ reasons were reported (random, based on player level, playing position, etc.).
	PEDro and RoB 2	The similarity of the groups at the baseline or initial phase was discussed.
	NEW ITEM	The entire training process is monitored and compared between groups (e.g., accumulated total training load and tactical behavior response during the intervention).
	PEDro and NOS	Reports of measures of variability and estimations of parameters concerning at least one primary variable (comparability).
	NOS	Was the same rate for both groups or non-respondents described?

NOS = Newcastle–Ottawa Scale [39]; GPS = Global Positioning Systems; LPS = Local Positioning Systems; VID = semi-automatic multi-camera based systems

RESULTS

As in other fields of knowledge, in sports science, report summary tables and figures help in presenting results in a structured and clear format. In this sense, the results section should include the conclusions about the articles found using a flow diagram (Figure 1), the results of the quality assessment, and a second section in which the study content is explained.

FIG. 1.

Flow diagram of the studies.

Flow diagram

A PRISMA statement consists of a 27-item checklist and a four-phase flow diagram (i.e., identification, screening, eligibility, and inclusion) [3]. Unlike PRISMA, studies using additional sources should place them after the inclusion/exclusion section while using defined criteria for two main reasons: (1) the included articles are not duplicated and (2) it is common that these articles have been mainly identified through bibliography checking. Therefore, these articles should be included after duplicate identification and after paper checking in the inclusion/exclusion section.

Study content

When the results of a systematic review are presented, they should give the reader the significant conclusions of the review through the provision of answers to the research question, as well as the evidence on which these conclusions are based and an assessment of the quality of the evidence supporting each conclusion [2]. Presenting the results in this way reduces the workload by limiting data extraction to only relevant information. As is common in sports science, the results should be presented in a structured table. However, it cannot provide a standard due to the many research methods in sport settings. The results of the studies may be clustered in groups meeting the research objective/problem. In any study, this brief abstract is added after the “included articles” section in the flow chart [27].

CONCLUSIONS

Recently, sports science has had an exponential growth of systematic reviews due to the increase of papers published. In order to optimize the assessment of sports research, this work aimed to adapt PRISMA [3] to provide specific guidelines for systematic reviews in this field of study. In addition to the suggestions of previous studies, two new items were added: (i) population/problem (i.e., age, level, and country); and (ii) the entire training process, which is monitored and compared between groups (e.g., total training load). These two new items are specific to sports science and help readers to fully understand the context of data collection and make better comparisons between studies. The current purpose is to establish support for sport scientists to improve the information provided to readers and make it easier to generalize and compare results.

Practical application

To maximize the benefit of this document, we encourage people to read it in conjunction with the PRISMA statement [3], PRISMA’s latest extension [5], and update [6]. The main differences between PRISMA and the PRISMA adapted to sports science can be found in Table 3.

TABLE 3.

Differing items between PRISMA and PRISMA applied in sports science.

Nº of item in the PRISMA guidelin	Comments
Item 5	Any systematic sport review may register their project in PROSPERO. However, it is not common. Therefore, although it may be suitable, it is not mandatory.
Item 6	PICO(S) search strategy is not used in sports science. Although it is common to use some of the groups from PICO(S), two or three groups are often classified, whereas PICO and PICOS are not commonly used.
Item 17	Flow diagrams may assume any changes in sports science. Therefore, although we have based them on the PRISMA guideline, changes have been computed (Figure 1).
Item 18	In general, tables are more detailed. Therefore, tables assume more columns than the groups used for a search strategy.

Despite the similarities between health care and sports science, some specificities must be considered, specifically regarding some methodological options. In sports science, the range of experiments is extremely large in that some studies are experimental (e.g., randomized controlled trials, parallel group, or crossover), and many of them are observational analyses (cohorts, cross-sectional and, case-control). This is one of the questions that must be considered during the selection of a risk of bias instrument or protocol register. The register of systematic reviews or meta-analysis in sports science cannot always be accomplished due to the nature of the research question and paper organization. Possibly, studies conducted on non-related health topics may be dismissed from the protocol register in PROSPERO.

In sports science, considering the specificity of study characteristics, it is usually important to use standard procedures to determine the cases of experimental studies (e.g., training interventions) or to better describe the nature of the study (e.g., in the case of crosssectional studies as in match analysis). We recommend in experimental studies that the number of weeks, sessions/week, work volume per session, intensity, work-to-rest ratio, exercises, repetitions, sets, rest between repetitions, and rest between sets be described appropriately. In addition, we suggest the quantification of the external and/or internal training load (TL) of the entire training session to compare with the total TL accumulated during the intervention between intervention groups. In the case of the nature of the study, it is important to describe the context of data collection, the type of samples, a brief characterization of the competitive level of the athletes (if applicable), the type of outcomes collected and their validity and reliability, and the tests used to determine the outcome.

Conflict of interest statement

The authors declare that they have no competing financial interests.

Funding

The authors received no potential support for the research, authorship and/or publication of this article.