Abstract
Background:
While studying the etiology of multiple sclerosis (MS) in children has several methodological advantages over studying etiology in adults, studies are limited by small sample sizes.
Objective:
Using a rigorous methodological process, we developed the Pediatric MS Tool-Kit, a measurement framework that includes a minimal set of core variables to assess etiological risk factors.
Methods:
We solicited input from the International Pediatric MS Study Group to select three risk factors: environmental tobacco smoke (ETS) exposure, sun exposure, and vitamin D intake. To develop the Tool-Kit, we used a Delphi study involving a working group of epidemiologists, neurologists, and content experts from North America and Europe.
Results:
The Tool-Kit includes six core variables to measure ETS, six to measure sun exposure, and six to measure vitamin D intake. The Tool-Kit can be accessed online (www.maelstrom-research.org/mica/network/tool-kit).
Conclusion:
The goals of the Tool-Kit are to enhance exposure measurement in newly designed pediatric MS studies and comparability of results across studies, and in the longer term to facilitate harmonization of studies, a methodological approach that can be used to circumvent issues of small sample sizes. We believe the Tool-Kit will prove to be a valuable resource to guide pediatric MS researchers in developing study-specific questionnaire
Keywords: Multiple sclerosis, pediatrics, etiology, risk factors, questionnaires, sunlight, vitamin D, tobacco smoke pollution
Introduction
Multiple sclerosis (MS) is primarily a disease of adulthood, with peak incidence between 30 and 50 years of age,1–3 although MS in children has recently become increasingly studied. Pediatric-onset multiple sclerosis (PoMS) cases are a small proportion (3%–10%) of all those diagnosed with MS.4 While etiological risk factors identified in studies of adult-onset MS5,6 have also been shown to be associated with PoMS,7,8 more research is needed to inform on the etiology of MS in pediatric populations. PoMS provides a unique opportunity to study MS etiology, relative to adults, because cases are younger, and thus, (1) key exposures occur closer in time to MS onset and (2) the time period in which to search for risk factors is shorter. In addition, studies suggest that MS risk is determined in childhood and early adolescence.9–15
A common methodological problem faced in PoMS epidemiological studies is small sample sizes. Because PoMS is rare, the number of cases that can be obtained in individual studies is low and requires long periods of time to accrue to have sufficient statistical power to precisely estimate main effects or to explore interaction between risk factors. Harmonization, a methodology used to combine data collected in multiple studies, provides a potential solution to small sample sizes. The methodology used for harmonization focuses on the use of a common set of core variables, which serve as a framework to conduct pooled analyses.16
We developed the Pediatric MS Tool-Kit (Tool-Kit) for pediatric MS researchers to design study-specific questionnaires based on variables which (1) have been selected using a rigorous methodological process, (2) enhance comparability of results across studies, and (3) are amenable to future harmonized analyses. This paper describes the methodology that was used to develop the Tool-Kit and provides an overview of how the Tool-Kit can be used in pediatric MS research.
Methods
We solicited input from the International Pediatric MS Study Group (IPMSSG)17 using an online survey to select three risk factors. We searched PubMed in March 2013 (search date range: 2000–2013) for epidemiological studies that examined the etiology of MS to generate a list of etiological factors, which was then filtered using the following predefined criteria: (1) an association was found with the risk of MS, in at least one high-quality study; (2) the timing of exposure is relevant to PoMS; and (3) the risk factor can be measured using a self-report questionnaire. The survey was distributed in May 2014 to 138 IPMSSG members. Respondents were asked to report whether, in their view, each risk factor was (1) a priority, (2) important, but not a priority, (3) not important for future research, or (4) I don’t have an opinion.
A systematic review of measurement property studies was conducted to summarize the available evidence on the validity and/or reliability of relevant questions/questionnaires/scales. Standardized methodology18 was used to perform a review of each risk factor. We searched three electronic databases (PubMed, EMBASE, and CINAHL) in 2014–2015 using a validated measurement properties search strategy.19 Quality assessment was completed using the COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist.20,21
A Delphi study22,23 was then conducted to select and define a set of core variables for each risk factor. A working group (WG) of 11 researchers from Canada, the United States, the United Kingdom, the Netherlands, and Italy was assembled, including epidemiologists, pediatric MS neurologists, adult MS neurologists, and content experts. The WG included nine researchers actively involved in MS research who are also IPMSSG members. We also invited three content experts, selected from among authors identified through the measurement property reviews. The WG was divided into three sub-groups, one for each risk factor, with some overlapping membership and six members in each group. Prior to the start of the Delphi study, the WG met for a 2-day face-to-face meeting during which background knowledge to facilitate participation was provided.
The Delphi study had four rounds that were developed with knowledge gained from the measurement property systematic reviews. Delphi rounds 1, 2, and 4 were completed anonymously online and round 3 was face-to-face. In round 1, a research question for each risk factor was defined and a set of criteria were selected to guide selection of core variables. In round 2, the WG was presented with a list of broad areas that were relevant for each risk factor, for which various variables could be defined. For example, for summer sun exposure, potential variables include duration of sun exposure, frequency of sun exposure, sun exposure during certain periods of the day, and so on.
We initially planned for all rounds to be online; however, because selection of core variables proved to be a complex process, in round 3 the WG actively engaged in guided discussions during a 2-day face-to-face meeting. Given the goal was to limit the number of core variables, the WG also defined a set of ancillary variables—variables that provide important supplementary information about exposure, but were not deemed core. In round 4, the WG gave approval for the proposed Tool-Kit variables. Each Tool-Kit variable includes a variable description, harmonizable response options, and data coding.
We also evaluated the content validity of the core variables using the COSMIN checklist.20,21 Each WG member independently rated (highly relevant, somewhat relevant, not relevant) whether the core variables (1) refer to relevant aspects of the construct being measured, (2) are relevant for the target study population (e.g. age, sex, disease characteristics, country, setting), (3) are relevant for the purpose of the measurement instrument (e.g. predicting exposure), and (4) together comprehensively reflect the construct being measured. Variables that were rated as relevant (highly or somewhat) remained as core variables, and those that were rated as “not relevant” were subsequently classified as ancillary variables.
Age epochs were also defined. Differences in the potential for exposure based on the child’s main activities and changes in activities that represent potential changes in exposure were considered; but we also wanted to select a small number of age epochs to ensure questionnaires were not too long.
Results
Forty-two risk factors previously implicated in MS risk were identified, among 88 publications (of 1400 records), subsequently reduced to 12 risk factors that are relevant for pediatric populations: body size or body mass index, environmental tobacco smoke (ETS), head injury or traumatic brain injury, history of infectious mononucleosis, penicillin use, physical activity, prenatal and perinatal factors, sibling exposure and attending daycare, stressful life events, sun exposure, vaccinations, and vitamin D intake. Forty-eight IPMSSG members completed the survey (35% response). When the responses “a priority” and “important” were combined, sun exposure (96%), vitamin D intake (94%), and ETS (93%) were most highly endorsed (Figure 1).
Figure 1.
Results of the Risk Factor Survey, indicating the percentage of respondents who rated each risk factor as (1) a priority for future research and either (2) a priority for future research or important, but not a priority for future research.
Among the reviews of measurement property studies, we identified 152 publications on ETS questionnaires, 35 on sun exposure, and 13 on vitamin D intake. For vitamin D intake, we also searched country-specific food composition databases and government reports to identify country-specific food sources of vitamin D. Much of the extant measurement property literature focused on questionnaires that measure current or recent exposure; however, we did not find validated questionnaires to assess long-term exposure histories, as required in case-control studies. This is a major gap we identified in the measurement properties literature.
A research question for each risk factor is presented in Table 1 and the eight variable selection criteria in Table 2. In the final Tool-Kit, there are six core and three ancillary variables to measure ETS, six core and six ancillary variables to measure sun exposure, and six core and five ancillary variables to measure vitamin D intake (Table 3). An example of a sun exposure core variable is provided in Table 4. The Tool-Kit variables can be accessed online (www.maelstrom-research.org/mica/network/tool-kit). Our preliminary assessment suggests that the Tool-Kit core variables have good content validity.
Table 1.
Research questions for the three risk factors that are included in the Tool-Kit.
| Risk factor | Research question |
|---|---|
| Environmental tobacco smoke | Everything else being equal, are children who have been exposed to higher levels of environmental tobacco smoke at increased risk of MS compared with children who have been exposed to lower levels of environmental tobacco smoke? |
| Sun exposure | Everything else being equal, are children who have been exposed to lower levels of sun at increased risk of MS compared with children who have been exposed to higher levels of sun? |
| Vitamin D intake | Everything else being equal, are children with lower intake of vitamin D (through supplementation) at increased risk of MS compared with children with higher intake of vitamin D (through supplementation)? |
MS: multiple sclerosis.
Table 2.
Eight criteria for selecting the core variables to be included in the Tool-Kit.
| Selection criteria |
|---|
| The variable is necessary to answer the research question |
| The variable helps to better interpret or understand the level of exposure to the risk factor |
| The variable is a potential confounder or effect modifier |
| The variable can be collected using proxy-report (i.e. parent/guardian) via self-administered and/or interview-administered questionnaire |
| The variable can be collected in a valid and reliable way, given the required retrospective nature of the data collection |
| The level of detail that is asked to recall is reasonable given the retrospective nature, time, and resources available |
| The variable is of high enough prevalence in the source population to ensure sufficient statistical power |
| The variables and response options should be selected to enhance cross-cultural validity |
Table 3.
Tool-Kit core and ancillary variables for the three risk factors.
| Core variables | Ancillary variables |
|---|---|
| Environmental tobacco smoke (ETS) exposure | |
| 1. Home ETS exposure laddera
2. Childcare ETS exposure laddera 3. Frequency of smoking by the child’s mother 4. Frequency of smoking by the child’s father 5. Frequency of smoking by others who lived with the child 6. Residential history |
1. Evidence that previous smoker(s) lived in child’s home 2. Smoking status of close family members and/or friends 3. Type of tobacco products consumed by individuals who lived with the child |
| Sun exposure | |
| 1. Residential history 2. Frequency of daily outdoor activities during daylight Hours 3. Duration of time outdoors on weekends during summer 4. Duration of time outdoors on weekdays during Summer 5. Duration of time outdoors on weekdays during winter 6. Duration of time outdoors on weekends and school holidays during winter |
1. Frequency of travel to sunny destinations during winter 2. Skin color 3. Sun sensitivity 4. Frequency of sun protection use: sunscreen 5. Frequency of sun protection use: wearing a shirt with sleeves 6. Frequency of sun protection use: staying in the shade or under an umbrella |
| Vitamin D intake | |
| 1. Child’s use of dietary supplements 2. Frequency that the child used dietary supplements 3. Duration of time that the child used dietary supplements 4. Child’s use of dietary supplements that contain vitamin D 5. Frequency that the child used dietary supplements that contain vitamin D 6. Duration of time that the child used dietary supplements that contain vitamin D |
1. Brands of dietary supplements that were commonly used by the child 2. Use of dietary supplement was recommended by a health care professional 3. Child’s use of cod liver oil 4. Frequency that the child’s used cod liver oil 5. Duration of time that the child used cod liver oil |
The ETS exposure ladders incorporate sources and locations of exposure, as well as smoking rules in the home.
Table 4.
Example of a core variable included in the Tool-Kit.
| Table | Sun exposure |
| Variable name | Frequency of daily outdoor activities during daylight hours |
| Label | Frequency of outdoors activities |
| Description | ● Classifies the frequency of the child’s usual daily outdoor activities during daylight hours ● Self-report by parent(s) or both child and parent(s) ● Ask for all relevant age epochs |
| Value type | Text |
| Missing | 9999 |
| Unit | Not applicable |
| Category codes and labels | 3: Almost always outdoors 2: More often outdoors 1: More often indoors 0: Almost always indoors 9999: Don’t know/can’t recall |
The following age epochs were selected: (1) baby (birth–1 year), (2) toddler/preschool (2–4 years), (3) child/primary or elementary school age (5–12 years), and (4) teenager/high school age (13–18 years). In utero was also identified as an important epoch; however, given the need to modify variable definitions to focus on mother’s activities/behaviors, it was not included at this time.
Discussion
We developed the Pediatric MS Tool-Kit, which is a novel contribution to the field of pediatric MS etiological research. The Tool-Kit aims to enhance the methodological rigor of newly developed PoMS etiologic studies by proposing a measurement framework to facilitate the design of study-specific questionnaires. The short-term goals of the Tool-Kit are to enhance exposure measurement in individual PoMS studies, by proposing a set of rigorously selected and defined variables that measure priority risk factors, and to enhance comparability of study results, by proposing the use of a common measurement framework. The long-term goal of the Tool-Kit is to enhance the potential for collaboration through data sharing and consequently larger sample sizes in harmonized analyses. The Tool-Kit provides a set of core and ancillary variables that are intended to be used to measure children’s ETS exposure, sun exposure, and intake of vitamin D. The core variables are those that were selected for harmonized analyses. As the WG proposed a number of important variables to comprehensively measure exposure, these additional variables are provided as ancillary variables. The Tool-Kit will reduce the time and resources required to design study-specific questionnaires.
Researchers can use the information in the Tool-Kit to create a questionnaire that is specific to their target population. The exact questions, however, are not provided in the Tool-Kit. We chose to provide information about the variable to develop the questions, rather than exact wording of individual questions. This approach is referred to as flexible prospective harmonization.24 While the use of flexible prospective harmonization may hamper comparability among studies, it may be more appropriate for etiological research given study investigators are most knowledgeable about their study context and target population. In addition, flexible prospective harmonization will enable relevant data collected in existing PoMS studies to be harmonized with data collected in new studies, which would not be possible had we employed a more stringent harmonization methodology.
The need for, and value of, prospective harmonization has been recognized by the research community and is demonstrated by several examples of large prospective harmonization initiatives such as the National Institute of Neurological Disorders and Stroke (NINDS) Common Data Elements (CDE),25,26 European Prospective Investigation into Cancer and Nutrition (EPIC) study,27 the PhenX Toolkit,28 and Canadian Partnership for Tomorrow project.29 The NINDS has recently developed CDE for MS that include variables focused on demographics, clinical assessments, imaging, and neuropsychology/cognition.25,26 The NINDS also used a WG model to select and define the MS CDE. The MS CDE also include a core set of variables, which they define as essential information applicable to any study, and have also defined three sets of CDE which they classify as supplemental–highly recommended, supplemental, or exploratory.
The Tool-Kit variables can be accessed online at www.maelstrom-research.org/mica/network/tool-kit. The information presented online includes the proposed Tool-Kit variables (i.e. name, type, and description), response options, and data coding. To be harmonizable, we recommend that the information online be used to develop individual questions to include in study-specific questionnaires. Ideally, at least one question should be developed for each variable, and the question should be worded so that it links to the variable description. If multiple questions are used, it will be important to ensure the Tool-Kit variables can be derived from the questions used.
The proposed response options for each variable are meant to be used as is, although modification is possible; however, for data to be harmonizable, researchers should ensure the response options in the Tool-Kit can be generated from modified response options. For example, if deemed more appropriate by the study investigators, finer response options may be used, but the new response options should be collapsible into those provided in the Tool-Kit. We do not recommend that investigators exclude any of the response options that are proposed in the Tool-Kit, as they form the basis for harmonization and were developed through a rigorous methodological process.
In addition to retaining the response options as proposed in the Tool-Kit, the age epochs should also be used as is in order for data to be harmonizable. However, unlike the response options, collapsing age epochs is much more methodologically difficult and may render the data non-harmonized. The key is to ensure the Tool-Kit variables and response options can be generated from the data collected in a study.
Once the questionnaire is developed, we recommend that the actual questions and their response options be compared to the variable descriptions and response options in the Tool-Kit by an individual who was not involved in developing the study questionnaire, and if the questionnaire is used in a language other than English, this individual should be fluent in English and in the language used in the questionnaire.
The Tool-Kit is a methodological resource for questionnaire design and is not a repository of data. While we will maintain an inventory of studies that have used the Tool-Kit variables, the decision to share data, at the time of a proposed harmonized analysis, is left to the discretion of the individual study investigators. As the long-term goal of the Tool-Kit is to provide the opportunity for collaboration through harmonization, researchers using the Tool-Kit variables will be asked to provide us with some basic information about their study (e.g. study name, sample size, variables used) to be displayed on the Tool-Kit webpage. To facilitate this process, we ask when using the Tool-Kit to develop a study-specific questionnaire that this publication is cited appropriately.
The main limitation, in light of the rigorous process used to select the core variables, is the possibility that the variables selected have poor measurement properties. We were unable to identify a validated questionnaire to use in a case–control study, and thus we used an expert consensus-seeking approach to select the Tool-Kit variables. The Tool-Kit has not yet been tested in a “real-life” research setting. While we show that the core variables have good content validity, continued evaluation of the measurement properties of the Tool-Kit variables will be imperative to its utility and success. We are open to collaborating with researchers wanting to use and assess the measurement properties of the core variables in their specific research settings.
We believe the Tool-Kit will prove to be a valuable resource to guide pediatric MS researchers in developing study-specific questionnaires. Rigorous epidemiological and expert consensus methods were utilized to develop the Tool-Kit variables and we engaged the pediatric MS research community to ensure that what we developed is relevant. We invite content area experts to take the opportunity to expand the Tool-Kit to develop additional core variables for the other priority MS risk factors.
Acknowledgments
The authors acknowledge Karen Zabowski for providing administrative support and Genevieve Gore for her help in designing the search strategies for the literature reviews of measurement property studies.
Footnotes
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by an operating grant from the US National MS Society (2013–2016, Grant No. 5986412) and an International Meeting Grant from the Multiple Sclerosis International Foundation (November 2014). S.M. received doctoral studentship funding from McGill University, Canadian Institutes of Health Research, and the MS Society of Canada.
Contributor Information
Sandra Magalhaes, Royal Victoria Hospital, Allan Memorial Institute and Neuroepidemiology Research Unit, Research Institute of the McGill University Health Centre, Montreal, QC, Canada/Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, QC, Canada.
Brenda Banwell, Division of Neurology, Children’s Hospital of Philadelphia, Philadelphia, PA, USA.
Amit Bar-Or, Center for Neuroinflammation and Experimental Therapeutics and Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
Isabel Fortier, Research Institute of the McGill University Health Centre, Montreal, QC, Canada.
Heather E Hanwell, Dalla Lana School of Public Health, The University of Toronto, Toronto, ON, Canada.
Ming Lim, Children’s Neurosciences, Evelina London Children’s Hospital, Guy’s and St Thomas’ NHS Foundations Trust, King’s Health Partners Academic Health Sciences Centre, London, UK/ Faculty of Life Sciences & Medicine, King’s College London, London, UK.
Georg E Matt, Department of Psychology, San Diego State University, San Diego, CA, USA.
Rinze F Neuteboom, Department of Pediatric Neurology, Erasmus MC-Sophia Children’s Hospital, Rotterdam, The Netherlands.
David L O’Riordan, School of Medicine, University of California, San Francisco, San Francisco, CA, USA.
Paul K Schneider, Royal Victoria Hospital, Allan Memorial Institute and Neuroepidemiology Research Unit, Research Institute of the McGill University Health Centre, Montreal, QC, Canada.
Maura Pugliatti, Department of Medicine, McGill University, Montreal, QC, Canada/ Unit of Clinical Neurology, Department of Biomedical and Surgical Sciences, University of Ferrara, Ferrara, Italy.
Bryna Shatenstein, Département de nutrition, Université de Montréal, Montreal, QC, Canada/ Centre de recherche, Institut universitaire de gériatrie de Montréal, CIUSSS du Centre-Sud-de-l’Île-de-Montréal, Montreal, QC, Canada.
Catherine M Tansey, School of Physical and Occupational Therapy, McGill University, Montreal, QC, Canada.
Evangeline Wassmer, Department of Neurology, Birmingham Children’s Hospital, Birmingham, UK.
Christina Wolfson, Royal Victoria Hospital, Allan Memorial Institute and Neuroepidemiology Research Unit, Research Institute of the McGill University Health Centre, Montreal, QC, Canada/ Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Montreal, QC, Canada/ Department of Medicine, McGill University, Montreal, QC, Canada.
References
- 1. Kingwell E, Zhu F, Marrie RA, et al. High incidence and increasing prevalence of multiple sclerosis in British Columbia, Canada: Findings from over two decades (1991-2010). J Neurol 2015; 262(10): 2352–2363. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Mackenzie IS, Morant SV, Bloomfield GA, et al. Incidence and prevalence of multiple sclerosis in the UK 1990-2010: A descriptive study in the General Practice Research Database. J Neurol Neurosurg Psychiatry 2014; 85(1): 76–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Marrie RA, Yu N, Blanchard J, et al. The rising prevalence and changing age distribution of multiple sclerosis in Manitoba. Neurology 2010; 74(6): 465–471. [DOI] [PubMed] [Google Scholar]
- 4. Ghezzi A, Baroncini D, Zaffaroni M, et al. Pediatric versus adult MS: Similar or different? Mult Scler Demyelinat Dis 2017; 2(1): 5. [Google Scholar]
- 5. Hedstrom AK, Baarnhielm M, Olsson T, et al. Exposure to environmental tobacco smoke is associated with increased risk for multiple sclerosis. Mult Scler 2011; 17(7): 788–793. [DOI] [PubMed] [Google Scholar]
- 6. Munger KL, Chitnis T, Ascherio A. Body size and risk of MS in two cohorts of US women. Neurology 2009; 73(19): 1543–1550. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Langer-Gould A, Brara SM, Beaber BE, et al. Childhood obesity and risk of pediatric multiple sclerosis and clinically isolated syndrome. Neurology 2013; 80(6): 548–552. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Mikaeloff Y, Caridade G, Tardieu M, et al. Parental smoking at home and the risk of childhood-onset multiple sclerosis in children. Brain 2007; 130(Pt. 10): 2589–2595. [DOI] [PubMed] [Google Scholar]
- 9. Alter M, Leibowitz U, Speer J. Risk of multiple sclerosis related to age at immigration to Israel. Arch Neurol 1966; 15(3): 234–237. [DOI] [PubMed] [Google Scholar]
- 10. Dean G, Kurtzke JF. On the risk of multiple sclerosis according to age at immigration to South Africa. BMJ 1971; 3(5777): 725–729. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Pugliatti M, Solinas G, Sotgiu S, et al. Multiple sclerosis distribution in northern Sardinia: Spatial cluster analysis of prevalence. Neurology 2002; 58(2): 277–282. [DOI] [PubMed] [Google Scholar]
- 12. Pugliatti M, Riise T, Sotgiu MA, et al. Evidence of early childhood as the susceptibility period in multiple sclerosis: Space-time cluster analysis in a Sardinian population. Am J Epidemiol 2006; 164(4): 326–333. [DOI] [PubMed] [Google Scholar]
- 13. Wolfson C, Wolfson DB, Veri JP, et al. Multiple sclerosis: A comparison of the latent periods of different populations. Neuroepidemiology 1993; 12(5): 300–306. [DOI] [PubMed] [Google Scholar]
- 14. Wolfson C, Wolfson DB, Zielinski JM. On the estimation of the distribution of the latent period of multiple sclerosis. Neuroepidemiology 1989; 8(5): 239–248. [DOI] [PubMed] [Google Scholar]
- 15. Pugliatti M, Cossu P, Sotgiu S, et al. Clustering of multiple sclerosis, age of onset and gender in Sardinia. J Neurol Sci 2009; 286(1–2): 6–13. [DOI] [PubMed] [Google Scholar]
- 16. Fortier I, Burton PR, Robson PJ, et al. Quality, quantity and harmony: The DataSHaPER approach to integrating data across bioclinical studies. Int J Epidemiol 2010; 39(5): 1383–1393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. International Pediatric MS Study Group (IPMSSG), http://www.ipmssg.org/
- 18. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 2009; 6(7): e1000097. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Terwee CB, Jansma EP, Riphagen II, et al. Development of a methodological PubMed search filter for finding studies on measurement properties of measurement instruments. Qual Life Res 2009; 18(8): 1115–1123. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Mokkink LB, Terwee CB, Knol DL, et al. The COSMIN checklist for evaluating the methodological quality of studies on measurement properties: A clarification of its content. BMC Med Res Methodol 2010; 10: 22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: An international Delphi study. Qual Life Res 2010; 19(4): 539–549. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Glaser EM. Using behavioral science strategies for defining the state-of-the-art. J Appl Behav Sci 1980; 16(1): 79–92. [Google Scholar]
- 23. Dalkey N, Helmer O. An experimental application of the DELPHI method to the use of experts. Manage Sci 1963; 9(3): 458–467. [Google Scholar]
- 24. Fortier I, Doiron D, Burton P, et al. Invited commentary: Consolidating data harmonization–How to obtain quality and applicability? Am J Epidemiol 2011; 174(3): 261–264. [DOI] [PubMed] [Google Scholar]
- 25. Grinnon ST, Miller K, Marler JR, et al. National institute of neurological disorders and stroke common data element project—Approach and methods. Clin Trials 2012; 9(3): 322–329. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. National Institute of Neurological Disorders and Stroke (NINDS), 2017, http://www.commondataelements.ninds.nih.gov/default.aspx
- 27. Riboli E, Hunt KJ, Slimani N, et al. European prospective investigation into cancer and nutrition (EPIC): Study populations and data collection. Public Health Nutr 2002; 5(6B): 1113–1124. [DOI] [PubMed] [Google Scholar]
- 28. Hendershot T, Pan H, Haines J, et al. Using the PhenX toolkit to add standard measures to a study. Curr Protoc Hum Genet 2015; 86: 1211–1217. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Borugian MJ, Robson P, Fortier I, et al. The Canadian partnership for tomorrow project: Building a pan-Canadian research platform for disease prevention. CMAJ 2010; 182(11): 1197–1201. [DOI] [PMC free article] [PubMed] [Google Scholar]