Common Measures in Mental Health: A Joint Initiative by Funders and Journals

Introduction

Improving rigor and reproducibility of scientific research is a goal for everyone involved in the research enterprise [1]. It is essential to ensuring that health research is valued and trusted, informs clinical practice, and ultimately improves people’s health and care. The inability to harmonize clinical and phenotypic data collected from human research participants in mental health related studies is a significant barrier to reproducing experimental results. The number of different phenotypic or psychometric scales used by researchers depositing their data in the National Institute of Mental Health (NIMH) data archive has surpassed 4200, as recorded by the NIMH Office of Technology Development and Coordination [2]. More scales are defined each week. This lack of standardized measurement makes it difficult or impossible to combine data and undermines the core purpose of data sharing. It limits the ability of researchers to aggregate effect sizes in meta-analyses, sacrificing the potential for more robust and clinically meaningful findings [3]. Most critically, it holds back the mental health field by generating an inconsistent and under-utilized body of knowledge [4]. This data heterogeneity is likely one of the important reasons why the global burden of mental health is not decreasing [5].

This paper describes, in detail, a new initiative of mental health research funders and publishers to define common measures for mental health. An overview of this effort has been described elsewhere [6].

The Common Measures in Mental Health Science (CMMHS) Initiative is an unprecedented collaboration between research funders and medical journals, determined to address the fragmentation of mental health data. Building on the work of existing standard-setting organizations, and with the assistance of the International Alliance of Mental Health Research Funders (IAMHRF)^*, the CMMHS Initiative aims to establish a set of common measures that will be widely used in mental health research, whilst still encouraging the use of additional related measures [7].

The standardization of data formats and measurement tools represents a powerful mechanism to allow data measured in different settings to be aggregated and compared [8, 9, 10]. Despite this, standard measures are often plagued by low uptake among researchers unless expectations are imposed by journals, funders, or regulators. The uptake of standardized measures in clinical research has been examined as part of the Core Outcome Measures in Effectiveness Trials (COMET) initiative [11]. In general, the uptake of most core outcome sets (COS) has been shown to be very low, with the vast majority of published clinical trials not using them even when they are available [12]. A notable exception is in research relating to rheumatoid arthritis, where COS are consistently included in 60–82% of clinical trials [13]. This difference has been linked to the inclusion of the rheumatoid arthritis COS in the guidance of regulatory bodies such as the Food and Drug Administration and European Medicines Agency, which creates a strong incentive for uptake in industry-driven clinical trials [14]. Research funders have tried to create their own incentives to increase COS uptake. In 2012, the National Institute for Health Research (NIHR) revised the guidance notes for its Health Technology Assessment program to add a strong recommendation that researchers include COS in their applications [15].

Similar recommendations have been made by the Health Research Board (HRB), Patient-Centered Outcomes Research Institute (PCORI) and the German Research Foundation (DFG) [16]. Academic journals have also endorsed the use of standard measures, for example, through the CoRe Outcomes in Women’s and Newborn health (CROWN) initiative, a collective of 80 journals in the field of women’s and newborn health [17]. Although formal endorsements and recommendations play an important role in increasing the uptake of standard measures, uptake remains notoriously low which suggests that requirements imposed by funders and/or journals are necessary.

One example of success in mandating measures comes from the protein crystallography field where major funders and journals required data to be deposited into a single data archive, the Protein Data Bank [18, 19]. More recent and more familiar examples of reporting guidelines such as the CONSORT checklist describing randomized trial design [20] or the SAGER guidelines [21] for reporting sex and gender information have followed the earlier work in the Protein Data Bank. A goal of the CMMHS Initiative is to establish common measures for mental health that can be used by research communities globally. We ultimately want to achieve the uptake levels seen in protein crystallography or rheumatoid arthritis. Following the Protein Data Bank example, the CMMHS Initiative was initiated jointly by funders and publishers. The initiative is committed to supporting standardization efforts over the long-term, to provide resources to the movement as needed, to emphasize inclusive and transparent processes, and to engage with a wide range of stakeholders to advance harmonization efforts steadily and reliably.

Selection of Measures

In 2019, two efforts to address fragmentation across mental health measurement emerged from a group of research funders during the annual meeting of the IAMHRF. First, 12 funders (Canadian Institutes of Health Research (CIHR), European Commission, Graham Boeckh Foundation, Grand Challenges Canada, Johnson & Johnson, UKRI - Medical Research Council, MQ Mental Health Research, National Institute of Mental Health (NIMH), Science Philanthropy Alliance, Tata Trusts, Wellcome Trust, ZonMw) formed the Common Data Elements Working Group (CDE WG) to discuss demographic measures. Second, the Wellcome Trust and NIMH initiated discussions about a first set of standardized measures related to mental health. These efforts eventually merged to form the CMMHS Initiative. Since then, the participants in the CMMHS Initiative have been discussing a wide range of issues related to improving the state of measurement in the mental health field.

In the discussions of the demographic measures, the CDE WG met monthly and discussed a wide range of potential CDEs related to basic demographic information with the intention to agree on data to be collected from all funded projects. Once the CDE WG had reached consensus on a first set of demographic measures, and a selected group of researchers had provided initial input on those measures, a survey was distributed to global research communities to evaluate the suggested demographic measures for their utility and cultural appropriateness. The survey yielded responses from 136 researchers collecting demographic data across Europe, North- and Latin America, Africa, Asia and Australasia. Details about all responses can be found in the Supplementary Information. 79% (n=136) and 76% (n=131) of respondents deemed that questions on age and sex at birth respectively should be required rather than optional, with over 90% (n=130, 135) of respondents deeming each question appropriate for use in multiple countries. Between 75% and 60% (n=127 to 131) indicated that information on work status, educational attainment and living arrangements should be collected, and less than 60% (n=127 to 131) deemed information related to marital status, gender identity and income appropriate to collect. Alternative approaches to data collection related to gender identify, living arrangements, marital status, educational level, work status, income and race/ethnicity were proposed, but even though most respondents deemed information on these demographic elements useful, views remained divergent on the appropriate approach.

After the completion of the demographics survey, a subset of IAMHRF funders, together with the journals Lancet Psychiatry and JAMA Psychiatry, formed the Common Measures in Mental Health Science Governance Board (CMB). In parallel, NIMH and Wellcome Trust had selected a first set of five outcome measures and announced their adoption across future funding calls in June 2020. All five outcome measures align with recommendations from international consultations and are contained in International Consortium for Health Outcomes Measurement’s (ICHOM) standard sets for depression and anxiety in adults, children and young people [22]. The strong consensus reached on age and sex at birth in the demographics survey led to their inclusion in the initial set of common measures endorsed by the CMB (see Table 1).

Table 1:

Data collection instruments endorsed by the CMMHS Initiative. Exact text for all questions as well as information about translations are available at https://nda.nih.gov/nda/nimh-common-data-elements.html .

Name of instrument	Abbreviated name	Topic
Patient Health Questionnaire-9	PHQ-9	Depression and anxiety (adults)
General Anxiety Disorder-7	GAD-7	Depression and anxiety (adults)
Revised Children’s Anxiety and Depression Scale-25	RCADS-25	Depression and anxiety (children and adolescents)
World Health Organization Disability Assessment Schedule 2.0	WHODAS 2.0	Impairment (adults)
Diagnostic and Statistical Manual of Mental Disorders-5 crosscutting assessment	DSM-5 crosscutting	Crosscutting assessment
IAMHRF Age	Age	Demographic
IAMHRF Sex at Birth	Sex at birth	Demographic

Several prerequisite criteria guided the selection of this first set of common measures. All measures are freely available. Affordability is a crucial aspect of any potential standard measure as commercial tools may act as a barrier to equitable access, particularly in low- and middle-income countries. It is also important that common measures do not place unnecessary burden on participants or researchers by entailing overly time-consuming administration or interpretation. Therefore, preference was given to shorter scales that can be administered and interpreted relatively quickly. There was also a requirement for measures to be commonly used by researchers and have a demonstrated capability to measure constructs that are relevant to and meaningful for mental health in a variety of contexts.

Governance

Broad adoption of common measures in mental health science requires coordination across the field. Research funders play a key role in driving adoption of standardized measurement tools through funding calls and the targeted disbursement of measurement-related funds. The IAMHRF has been serving as the secretariat for the CMMHS Initiative since its inception in 2019. The initiative is led by the CMB, which has been meeting since October 2020. CMB membership is open to academic publishers that cover mental health research and mental health research funders that are members of the IAMHRF. The CMB is bound by Terms of Reference that require each funder to endorse at least four of the seven suggested common measures (see Table 1) and to commit to their integration in relevant funding calls. The terms engage medical journals to complete integration efforts of the common measures into the publishing process within a five-year time frame beginning in February 2022. The CMB has been co-chaired by the NIMH and Wellcome Trust. The work undertaken by the CMB includes:

1. review measurement tools with advice from an outside group of experts.

2. identify a set of common measures that will be adopted by awardees and/or those who submit manuscripts to participating journals.

The overall goal of the initiative is to make it easier to harmonize clinical and phenotypic data collected in a wide array of different cultural contexts. Over time, as the measures are improved, the CMMHS Initiative is expected to produce standardized measurement tools that can be used for meaningful clinical improvements. Founding CMB members include seven research funding organizations and two medical journals (see Table 2). The Fondation Botnar and journal Nature Mental Health subsequently joined the CMB.

Table 2:

Founding member organizations of the CMB.

Organization	Location/scope	Type
Grand Challenges Canada	CA/international	Public/private funder
International Alliance of Mental Health Research Funders	CA/international	Funder network
Journal of the American Medical Association Psychiatry	US/international	Medical journal
Movember	AU/international	Charitable funder
MQ Mental Health Research	UK/international	Charitable funder
National Institute of Mental Health	US/international	Public funder
One Mind	US/US	Charitable funder
The Lancet Psychiatry	UK/international	Medical journal
UKRI - Medical Research Council	UK/international	Public funder
Wellcome Trust	UK/international	Philanthropic funders

For standardization efforts in the mental health sector to be transformative, they need to build on the collective experience and expertise of stakeholders using measurement tools over time and in a variety of settings. Thus, the success of the CMMHS Initiative relies on the input and advice from measurement experts and people with lived experience of mental health challenges from multiple cultural contexts. To get this advice, the CMB has been working closely with a Common Measures in Mental Health Science Advisory Group (CMA) composed of a group of experts in measurement related to mental health. The primary role of the CMA is to provide information about existing measures regarding their universal validity, usefulness and socio-cultural appropriateness across many settings and to help suggest ways to improve those measures. More details about the work of the CMA are below.

Operationalization

The first year of the CMMHS Initiative consisted of two key actions:

1. research funders required the use of the common measures listed in Table 1. The funders adopted approaches that made sense to them, but each funder requires the use of at least four of the common measures as part of their funding calls.

2. in-depth consultation with the CMA on the strengths and limitations of the newly introduced measurement tools.

NIMH published expectations related to the use of common measures in a Guide Notice that went into effect for applications submitted after October 1, 2020 [23]. In early implementation, NIMH discovered that all of the common measures were not appropriate for every research study. Some of these decisions were driven by whether the researcher had the ability to readily refer research participants for care since some of the common measures have questions that could require rapid intervention by a mental health care professional. Other funders took similar actions [24].

All of the funders encourage researchers to use other measures that might overlap the common measures to some degree. Comparison of those overlapping data should be very helpful in improving the common measures. In parallel, some funders are supporting studies aimed at evaluating current and identifying additional measures for key areas of interest (e.g., psychosis, [24]). Other CMB members have launched early proof-of-concept approaches to endorse the common measures, including Grand Challenges Canada and UKRI-Medical Research Council [25, 26]. IAMHRF members that are currently not part of the CMB are also moving towards adopting the common measures. CIHR launched a funding call as part of a pilot study facilitating the harmonization of mental health data collection by requiring the use of the common measures [27]. Planned follow-up interviews with grant recipients will be conducted to assess the impact of introducing common measures on the research process. Medical journals alike have initiated first steps to move towards data harmonization through common measures. In spring 2021, the Lancet Psychiatry introduced a requirement for researchers to justify their choice of measure by providing information on how accessible, widely used, burdensome and clinically meaningful the measure is [28].

Overall, responses from research communities have been mostly positive. In the initial three NIMH funding rounds that have occurred since the CDE policy went into effect, approximately 550 awards have been made for a total of $314 million. The data from those awardees is currently being deposited into the NIMH Data Archive [29] and will be made available as papers are published or at the end of the grant award. Since NIMH grants are often 4–5 years long, it will be some time before the research community has access to the CDE data from a large number of researchers. During these three funding cycles, few researchers have complained about the expectation to collect the CDEs. For researchers who have expressed concerns, once they understood the purpose of the initiative, they were willing to collect the data, even if their agreement was not enthusiastic in all cases.

With a current focus on clinical diagnostic categories, the Australian charity Movember faced difficulties in requiring the common measures within their priority areas of prevention research, emphasizing mental wellbeing concepts such as social connectedness, relationship quality and life satisfaction. While it is not yet possible to estimate general uptake of the common measures across researchers, it is likely that the NIMH’s Data Archive will be a useful tool to monitor adherence. Grand Challenges Canada observed that many of its funded innovators were already frequently choosing the common measures, eliminating a key barrier to requiring them.

In-depth consultation and evaluation of the first selected set of common measures was initiated through the recruitment of a diverse group of distinct measurement experts to the CMA. CMA members were recruited through an open application process by the CMB, with support from the IAMHRF, that yielded over 150 applications from around the world. Final selections of 13 CMA members was based on several factors including applicants’ measurement expertise, their geographical representation, and experience, with the objective to balance seasoned experts with early career researchers, as well as their lived experience of mental health challenges. Since June 2021, monthly meetings of the CMA along with less frequent meetings of the CMB have focused on assessing the merits and flaws of the original set of measures in different contexts, at both a scale and individual item level. Capitalizing on the strong international interest in the CMMHS Initiative and the large number of applicants to the CMA, in November 2021 a broader consultation of measurement experts was conducted, with a focus on global experiences using the PHQ-9 and GAD-7. Observations from the larger community of practice of measurement experts echoed assessments by the CMA regarding the benefits and disadvantages of these two popular measurement scales.

While the observations by the CMA are yet to be published from a user-perspective, several broad themes have emerged from the consultations from the perspective of the CMB. Firstly, it became clear that there exists no ‘gold standard’ for measures and that existing measurement scales must be improved over time, particularly with respect to cultural transferability. References to culturally relative concepts such as poor appetite or overeating are an obvious source of concern when administering scales in some contexts and suggest that astute adaptation is needed to capture the same underlying concept in those cultural contexts.

Discussion

Data from mental health research cannot afford to remain fragmented if we want to improve our understanding of mental illness and improve treatments and care. The CMMHS Initiative is committed to reforming measurement in mental health by advancing data harmonization on multiple fronts: by funding measurement research, leveraging uptake of data standards and supporting the infrastructure needed to move the field into the 21^st century. Crucially, this initiative simply sets out minimum requirements that should be measured with the hope that these common measures will allow research conducted in different settings to be integrated. It strongly encourages researchers to complement these core measures with other scales tailored to their particular research needs and to explore and develop new measures.

Standardization of mental health measurement should be embarked upon with great care. One concern is that it will create arbitrary standards that are difficult to replace, entrenching a two-tiered system that stifles the development of improved measures [30]. In the absence of a ‘gold standard’ measures that are based on underlying biology and are directly related to a diagnostic state or other construct in mental health, common measures should be viewed as the starting point of a multi-step process. The first set of common measures were not selected to confer a higher status to individual scales, but rather to ensure that researchers are at least asking some of the same questions.

The original set of common measures will be under review and subject to change, and there is commitment from several funders to fund research that evaluates or improves the existing measures or develops new measures. In May 2022, the Wellcome Trust launched a commission to evaluate functional outcome metrics for people experiencing or at risk of developing psychosis. In April 2022, NIMH released a funding call to provide support to maintain the RCADS-25, one of the adopted common measures. The adoption of a wider set of measures will be important in broadening the scope of standardization to encompass different aspects and models of mental health.

Both funders and journals hold broad perspectives on the field and its overall evidence base, which have shaped a shared concern for the current state of data fragmentation. Organizations that are part of the CMMHS Initiative are in a privileged position to leverage positive changes in the way that data is collected and shared. This initiative goes further than previous data standardization efforts in expecting, rather than simply recommending, the use of standard tools. This decision was not taken lightly and is part of a learning process on optimal strategies to achieve data harmonization. Full, transparent and, most importantly, two-way communication between organizations requiring the use and research communities implementing common measures via their research projects is paramount. This approach allows funders to learn as they adopt new standards and act flexibly in recognizing situations where exemptions should be granted. Upcoming consultations on the first round of research funded under these new requirements should expose any issues faced by researchers and consumers alike, informing future funding calls thus improving funding practices over time.

While the adoption of common measures in mental health is intended as a compromise in the short- to medium-term, it is essential that measurement approaches in mental health research improve over time. Consultations with the CMA reinforced that the existing measures have flaws that must be addressed. In particular, cultural transferability of measurement tools remains a key challenge in capturing core constructs across a variety of socio-cultural-economic settings. In the case of mental health, most scales were developed in clinical settings in Western, high-income countries, and there is a dearth of evidence of their validity in other contexts. In mid-2022, the CMMHS Initiative set out to increase the diversity of the CMA by launching a targeted recruitment of experts from Africa, Asia, and Latin America to advise on the validity and common adaptations of the original set of measures in non-Western contexts.

Another approach is to foster collective improvement efforts by funding research that aims to adapt and improve existing measures. The proprietary nature of measurement tools often would require negotiations with the copyright holder before modifications can be made. Copyrights for most of the IAMHRF common measures are held by a variety of organizations or individuals.. Proprietary tools with restrictive licenses cannot be easily modified which likely contributes to the proliferation of additional scales. Furthermore, copyright holders may in the future decide to charge for use of their instruments, particularly as the costs of providing maintenance and support rises due to increased uptake of a tool. Members of the CMB are still discussing potential strategies to deal with these issues.

Various pathways to a more adaptive and sustainable measurement landscape have been proposed, including transitioning to an item-bank model, or to an open-access measurement hub model [31]. The item-bank model offers an attractive solution to the lack of cultural transferability and is based on an item-response theory, whereby different questions are shown to the respondent, depending on the individual’s context and previous responses [32]. However, there is a significant concern that many questions would be interpreted differently when orphaned from their parent scales, presenting a barrier to adopting an item-bank approach.

In the hub model, copyright could be centralized at a non-profit organization responsible for hosting and maintaining the measures. Such a model would not only ensure the long-term accessibility of standardized measures but could also be combined with collaborative code approaches and version control to democratize the process of adapting and updating measures. A centralized hub for standards also represents an appealing approach for standardizing translations and adaptations of measures for each context. The CMMHS Initiative is actively exploring these possibilities with the ultimate goal to develop a long-term solution for effective standardization of measurement efforts in the mental health field.

The CMMHS Initiative is an ambitious effort by a collaborative group of research funders, journals, and measurement experts to build a shared framework for measurement in mental health research. The first phase of the initiative has proven that it is possible to require the use of standardized measures in funded research projects, even if adjustments needed to be made from the initial plans of the funders. With hundreds of projects already funded, there will be the potential for useful secondary analyses to be conducted once the data are released. Hopefully, these studies will show whether the first round of common measures are useful in harmonizing data measured in different settings. Those studies that employed common measures alongside other tools are likely to hold insights into ways to improve the common measures. Such tangible demonstrations of the possibilities that arise from adopting a common language in research will be critical in planning subsequent phases of the initiative.

The CMMHS Initiative plans to continue to engage a broader group of stakeholders, including global research communities and funders, additional journals, professional societies, clinicians, practitioners and people with lived experience of mental health challenges as the initiative moves forward. We look forward to receiving input on those next steps from the community starting now.