The Utility of Self-Perceived Health Ratings in Screening Volunteers for Mental Health Research

Abstract

We report on the utility of online self-perceived medical and mental health ratings (SPH) when screening healthy volunteers for mental health research. These one-item ratings were correlated with eligibility decisions and longer clinical surveys. We found correlations between SPH ratings and blinded clinician ratings of volunteer medical and mental health after in-person evaluation, although additional analysis revealed poor reliability between clinician and volunteer ratings. SPH ratings are a useful addition to screening methods for research studies.

Introduction

Healthy volunteers are often recruited for mental health research either as a comparison group for studies of psychiatric disorders or as the primary study population. However, there is limited guidance on how to best screen and characterize the health status of research volunteers. As a result, research studies may differ in how they define and ascertain health. This exposes a need for a robust literature on best practices for screening and determining eligibility for participation in mental health research.

One variable frequently used to assess health in the general population is self-perceived health (SPH). For instance, the National Institutes of Health (NIH) All of Us Research Program, which has recruited a diverse cohort of Americans for their health database, asks participants to rate their medical and mental health as part of a core set of surveys (NIH, 2020). There is ample evidence that SPH is associated with lifestyle factors and objective health measures (Yamada et al., 2012; Wu et al., 2013). After controlling for these factors, SPH predicts all-cause mortality up to 15 years before death and is correlated with physician ratings of health (LaRue et al., 1979; Stenholm et al., 2016). However, little is known about the utility of SPH when screening for research.

Given its strong association with longer measures of health status, SPH may serve as a useful tool for research screening purposes. The primary aim of this report is to evaluate the utility of single-item SPH medical and mental health ratings when recruiting healthy volunteers for mental health research. This is a secondary analysis of data from a protocol conducted at the National Institute of Mental Health Intramural Research Program (NIMH IRP) with the aim of recruiting and characterizing healthy research volunteers for referral to other NIMH studies (NCT03304665). We compared volunteer SPH ratings to study eligibility decisions, longer surveys, and clinician ratings of volunteer medical and mental health following an in-person evaluation.

Methods

The protocol was approved by the NIH Institutional Review Board. Participants were recruited via postcards, flyers, listservs, and social media. This secondary analysis utilized a convenience sample of volunteers who completed online measures from November 2017-November 2019.

Measures.

Participants used a secure website to consent and complete several surveys, including SPH ratings, a modified version of the DSM-5 Level 1 Cross-Cutting Symptom Measure (DSM-XC) (APA, 2019), the World Health Organization Disability Assessment Schedule 2.0 (WHODAS) (Ustun et al., 2010).

SPH Ratings:

Two items asking, “How would you rate your medical (mental) health?” a continuous sliding scale from 0 (Extremely poor) to 100 (Extremely good).

DSM-XC:

A transdiagnostic mental health symptom measure; our version included 10 cross-diagnostic domains (psychopathology: depression, anger, mania, anxiety, somatic symptoms, psychosis, sleep problems, memory, repetitive thoughts/behaviors, and dissociation).

WHODAS 2.0:

A 15-item measure of functional impairment.

Procedures.

Online Survey Review:

Responses to screening measures were reviewed and volunteers were sorted into three categories: likely ineligible, flagged for review, or likely eligible. Volunteers who reported a history of significant medical or mental health conditions or psychotropic medication use were ineligible. Volunteers who were flagged for review based on equivocal or unclear responses were subsequently phone screened by a mental health professional. Those deemed likely eligible were invited for the in-person assessment.

In-Person Assessment:

This assessment consisted of a structured diagnostic mental health interview, history and physical exam, laboratory tests, and IQ estimation. The clinical team met weekly to review final eligibility following the in-person assessment. After review, the clinical team used the same SPH items to arrive at a blinded consensus rating of medical and mental health for each volunteer.

Statistical Analysis.

A general linear model was used to assess the relationship between study eligibility decisions and SPH ratings while controlling for age, race, sex, and education. Pearson’s correlation coefficients were calculated to determine relationships between SPH (medical) compared with WHODAS scores, and clinician-rated medical health. Coefficients were also calculated for SPH (mental) compared with DSM-XC scores, and clinician-rated mental health. Intraclass correlation (ICC) coefficients were calculated to determine reliability between volunteer and clinician health ratings. All analyses were conducted using IBM SPSS Statistics (IBM SPSS, 2011).

Results.

910 volunteers consented and completed online measures. Most volunteers were female (66.4%) and white (67.4%), with an age range of 18 to 89 (m=37.6, 15.2). The Pearson’s correlations are in Table 1.

Table 1.

Pearson and Intraclass Correlation Results.

	Pearson	ICC
	SPH Mental
DSM-XC	−.558**
Clinician-Rated Mental	.147**	.124*
	SPH Medical
WHODAS	.314
Clinician-Rated Medical	.275**	.252*

^*p<.05

^**p<.001

Volunteers who did not follow through after completing online forms (n=127) were excluded from subsequent analyses. Of the remaining 783 volunteers, 418 (53%) were deemed likely ineligible, while 365 (47%) were likely eligible.

The mean SPH (medical) for the sample was 80.1 (SD 17.0) and the mean SPH (mental) was 77.2 (SD 20.5). The general linear model revealed a significant main effect of eligibility decision on mean SPH (mental) (F=38.1, p<.001), with ineligible participants rating their mental health lower (m=71.2, 23.1) than eligible participants (m=84.1, 14.4). There was also a significant main effect of eligibility decision on SPH (medical) (F=21.5, p<.001), with ineligible (m=76.6, 18.4) participants rating their medical health lower than eligible participants (m=84.0, 14.4). However, this relationship was moderated by education; the difference was not significant for participants with an advanced/professional degree.

Of the 245 volunteers seen in-person, 221 (90.2%) received clinician-rated medical and mental health ratings following their evaluation. Clinicians generally gave lower average ratings for both medical (m=64.9, 16.0) and mental health (m=64.8, 16.3) compared with volunteer ratings of medical (m=85.2, 14.1) and mental health (m=85.8, 13.4). Pearson’s correlations indicated a positive association between clinician and volunteer-rated health, although ICC revealed low reliability between ratings (Table 1).

Discussion

In this secondary analysis, we evaluated the utility of SPH ratings in screening healthy volunteers for mental health research. Volunteer SPH ratings were correlated with blinded eligibility decisions for participation in research and longer clinical surveys. Furthermore, volunteer SPH ratings were correlated with our clinical team’s rating of health although these correlations were small. This may reflect the fact that our study sought adults in good health, and thus volunteers have been predisposed to overestimate their health to be eligible. While there was low reliability between clinician and volunteer ratings, clinicians utilized all available clinical data and held consensus discussions to make their ratings.

In conclusion, SPH ratings performed reasonably well when compared with more extensive screening methods for research study eligibility. It is important to note that our study demographics do not reflect the general population, limiting the generalizability of findings. Nonetheless, to our knowledge, this is the first study to describe how research volunteers perceive their health and how this relates to other screening methods.

Given our limitations, we recognize that SPH ratings alone are likely insufficient for health screening on their own. However, our results suggest that they are a useful addition to screening methods for research studies. Given the variability in how health is ascertained across studies, robust literature on best practices for determining eligibility for participation in mental health research is needed.