Do claimants over-report behavioral health dysfunction when filing for work disability benefits?

Abstract

BACKGROND

Questions exist related to the best way to use medical evidence relative to self-report as part of the SSA disability determination process.

OBJECTIVE

To examine concordance between provider and claimant responses along the four dimensions of work related behavioral health functioning: Social Interactions, Mood and Emotions, Behavioral Control, and Self-Efficacy.

METHODS

Using secondary data from a larger study, which collected data on individuals reporting difficulties with work (claimants) due to mental conditions, 39 items were completed by claimants and their healthcare provider. Inter-rater agreement was assessed using three techniques: Cohen’s kappa, percent absolute agreement, and folded mountain plots.

RESULTS

A sample of 65 dyads was obtained. Inter-rater agreement was low for most items (k = 0.0–0.20) with a minority of items having fair agreement (k = 0.21–0.40) Percent agreement was fair: Mood and Emotions (46%), Self-Efficacy (44%), Behavioral Control (39%) and Social Interactions (38%). Overall, providers reported lower functioning compared to claimants for the Behavioral Control and Self-Efficacy scales; the reverse trend held for the Mood and Emotions scale.

CONCLUSIONS

Results indicate discordance between provider and claimant report of behavioral health functioning. Understanding reasons for and approaches to reconciling the inconsistencies between claimant and provider perspectives is a complex task. These findings have implications for how best to assess mental and behavioral-health related work disability in the absence of an established gold standard measure.

1. Introduction

Physician assessment and judgment of a claimant’s mental and physical functioning are key components in multiple stages of the Social Security Administration’s (SSA) disability determination process [1]. Literature suggests that a claimant’s physician, via the medical information provided, may substantially influence the outcome at each stage of disability determination [1]. Variation in both the quality and quantity of medical information exists [2]. The health care provider’s perspective and report, if not consistent with how a claimant may perceive his or her disability, may contribute to challenges SSA faces in the disability determination process. Such challenges are related to the substantial and increasing resources spent on appeals, potential misallocation of limited resources, and overall uncertainty and inefficiencies in the current disability determination process [2,3].

A claimants’ self-reported disability and provider reports of claimant disability are among the strongest predictors the disability determination outcome from initial application through appeals [3,4]. SSA uses information from claimants and medical providers to determine benefit eligibility based on the claimant meeting various criterion-based thresholds of disability [5]. These thresholds can be established on regulatory-based medical listings or on medically documented evidence such as symptom severity, laboratory tests, or other disorder-based impairments [5].

Part of the challenge of the current disability determination process is the lack of a gold standard measure for mental/behavioral health related work disability. When relying on self-reported information (from various informants – claimant or physician) the challenge becomes separating the informant’s perceptions of the underlying ability to work from true underlying ability or behavioral health functioning that may be influencing a person’s ability to work [6]. Many aspects of mental and behavioral health are typically assessed by self-report as compared to using more objective, external measures of mental health [7]. Complicating this issue further, previous literature introduces reasons for potential reporting bias of self-reported disability – typically assumed to be in the direction of individuals over reporting their disability level [8,9].

Much of the literature examining agreement in the context of health assessment is at the physician-patient level. The majority of studies focus on aspects of agreement for diagnostic purposes and the utilization of proxy report. Overall, findings suggest that there are higher levels of agreement for more objective, observable aspects of health (e.g. physical limitations) as compared to more subjective, internal health phenomena (e.g. psychological or cognitive symptoms) [10–13]. Aspects of physical health tend to yield higher levels of patient-physician agreement as compared to aspects of mental health or well-being [14,15]. For example, one recent study reports up to a 20% discordance between patient reported anxiety and physician estimation of patient anxiety among oncology patients [13]. This current study builds upon existing literature examining provider-patient agreement to focus on agreement along various dimensions of behavioral health functioning relevant to the context of work.

The goals of this study were to assess the degree to which individuals applying for SSA disability benefits – claimants – agree with their health care providers in terms of the claimant’s behavioral health functioning and to examine trends in potential reporting bias of individuals applying for SSA disability benefits. Specifically, the study examined patterns of agreement among SSA claimants and their matched healthcare providers on scores across four domains of function: Self-Efficacy, Behavioral Control, Mood & Emotions, and Social interactions. Findings from this analysis are expected to assist in determining if there are systematic differences in claimant and his or her providers the perceptions of behavioral health dysfunction in the context of work.

2. Methods

2.1. Study sample

The study used secondary data from a larger study, the SSA Functional Assessment Calibration Study, which collected data on SSA claimants reporting difficulties with work due to mental conditions [16–18]. Recruitment focused on identifying claimant and provider dyads. Claimants who were eligible to participate in the calibration study were asked to identify two healthcare providers who are responsible or very knowledgeable about the condition for which they were applying for work disability benefits.

2.2. Procedure

A research survey organization, Westat, performed all claimant and provider data collection. Interviewers contacted each claimant by telephone to confirm eligibility and willingness to participate in the study. Westat interviewers also contacted two of the claimant’s health care providers that the claimants previously identified, to ask the providers to complete a matching set of questions regarding the claimant’s functioning. Provider data collection was conducted via internet, claimant data was collected via telephone or internet depending on claimant preference. Providers and claimants were recruited simultaneously to reduce the potential for recall bias.

Ethics approval was obtained from the University Institutional Review Board for conduct of this study. All persons who took part in the study provided verbal informed consent. Data were de-identified to protect confidentially of all participants of this study.

2.3. Questionnaire and scoring information

The data used for this study originated from a reduced form of the full 165-item claimant behavioral health functioning survey [16–18]. From the 165 full items in the claimant survey, claimants and providers completed a set of 39 paired claimant and provider items spanning four dimensions of behavioral health functioning: Self-Efficacy (5 items), Behavioral Control (12 items), Mood & Emotions (19 items), and Social interactions (3 items). The items had identical content but the response scales differed for the claimants and providers.

To develop comparable scores for analysis purpose, the claimant responses, which were originally a five level response scale (Strongly Agree, Agree, Disagree, Strongly Disagree, I don’t know), were combined into Agree/Disagree/I don’t know categories to be consistent with the provider survey response category structure. The provider response options were Agree/Disagree/I don’t know. Claimant responses for “Strongly Agree” and “Agree” were combined as “Agree” and responses for “Strongly Disagree” and “Disagree” were combined to “Disagree”. The “I don’t know” responses remained the same. For this study, “I don’t know” items were coded as 0, “Agree items” coded as 1, and “Disagree” coded as 2.Where applicable, responses to items were reverse coded to numerically represent behavioral health functioning scores in ascending order from low functioning to high functioning.

There is lack of consensus as to the best measurement unit to use when measuring constructs in the field of behavioral science. A percent of maximum possible scoring (POMP) method was used to create a scale score for each domain using the following formula: POMP = [(observed − minimum)/(maximum − minimum)] × 100; where observed = the observed score for a single case, minimum = the minimum possible score on the scale, and maximum = the maximum possible score on the scale [19].We chose this method because it allows the each individual’s score to be calculated as a percentage that situates the person on a scale as a percent of the maximum possible score achievable [19]. The percentage-based unit of measurements allows for comparison across the scales to be communicated more clearly than if using a summed score or other technique that is dependent on the number of items in each scale.

2.4. Statistical methods

Frequency distributions were used to examine the study sample characteristics for both providers and claimants. The source of variation and role of uncertainty in physician assessment of patient health is a subject that has been explored in a variety of contexts within the medical literature [20]. Much of the previous literature focuses on variation in diagnosis, practice patterns, and health outcomes. Conceptually, there are several ways to think about variation in health assessment. For the purposes of this study, variability was defined in terms of assessing the inter-rater agreement and concordance of two rater’s evaluation of a single health phenomenon – behavioral health functioning. This technique has been used to examine patients and proxies in the context of their respective assessment of self-reported health [15].

To examine claimant-provider agreement, three methods were used – Cohen’s kappa for inter-rater reliability, percent absolute agreement, and mountain plots to identify potential systematic bias in agreement patterns of claimants versus providers. For calculating the item level kappa coefficient, the claimant and the provider were considered to agree on a given item if both assigned the same rating to it (Agree, Disagree, I don’t know). Thresholds for inter-rater reliability were used, including < 0 No agreement; 0.0–0.20 Slight agreement; 0.21–0.40 Fair agreement; 0.41–0.60 Moderate agreement; 0.61–0.80 Substantial agreement; 0.81–1.00 Almost perfect agreement [21].

Folded mountain plots were developed to graphically depict systematic differences in agreement of provider and claimant responses for each of the four domains of behavioral health functioning. Similar methods as described by Haley et al. have been used for constructing the mountain plots [22]. Steps included calculating difference scores between provider and patient responses, sorting the responses in ascending order, and calculating their percentile rank, then plotting the percentile rank vs. the difference score. To achieve the “folded” nature of the plot, values in the upper 50% of the distribution were folded by subtracting the actual percentile rank from one hundred. SAS statistical software was used for descriptive analyses and Microsoft Excel was used for developing the mountain plots [23, 24].

3. Results

A sample of 70 healthcare providers completed the survey, matched claimant data was missing for 5 of the providers, so the effective sample size resulted in 65 claimant/provider dyads. Of the providers completing the survey, 73% were white, 54% were male, and 74% from an urban area. The majority of providers reported having known the claimants from 1–5 years duration (74%); no providers indicated knowing the claimant for more than 5 years. Additionally 63% of the providers were physicians vs. other health care providers such as therapists, social workers, or psychologists. Lastly, only 30% of providers in this sample identified themselves as a mental health specialist, which included psychiatrists, psychologists, or social workers. Demographic and clinical condition characteristics of both providers and claimants are shown in Table 1.

Table 1.

Provider and claimant demographic and clinical characteristics

	Provider characteristics; N = 70 n (%)	Claimant characteristics; N = 65 n (%)
Gender
Male	38 (54)	20 (31)
Female	32 (46)	45 (69)
Geography
Urban	59 (84)	55 (85)
Rural	11 (16)	10 (15)
Race
White	51 (73)	47 (72)
African American/Black	7 (10)	13 (20)
Other	10 (14)	4 (6)
Missing	2 (3)	1 (2)
Physician vs. Non-physician
MD level	44 (63)	na
Non-MD degree	26 (37)	na
Specialization in mental health
Mental health specialist	21 (30)	na
Non-mental health specialist	49 (70)	na
Duration of care
Less than 1 year	18 (36)	na
1–5 years	51 (74)	na
Missing	1 (0)	na
Information source
Medical record	59 (84)	na
Claimant interview	9 (13)	na
Other	2 (3)	na
Reason for SSI/SSDI* Filing
Mental	na	20 (31)
Physical and mental	na	45 (69)
Mental condition
Bipolar disorder	na	9 (14)
Depression	na	27 (41)
Anxiety	na	9 (14)
PTSD**	na	7 (11)
Other/Not specified	na	13 (20)

^*SSI/SSDI: Supplemental Security Income/Social Security Disability Insurance.

^**PTSD: Post Traumatic Stress Disorder.

Table 2 shows absolute agreement by scale. The Mood and Emotions scale demonstrates the highest percent agreement at 46% followed by Self-Efficacy (44%), Behavioral Control (39%) and Social Interactions (38%). Figure 1 shows the inter-rater agreement and confidence limits for each item used in the measure of behavioral health function. The majority of the items are in the k = 0.0–0.20 range indicating slight agreement, and a few items in the k = 0.21–0.40 range demonstrating fair agreement.

Table 2.

Overall percent agreement by scale

Scale	Agreement n (%)	Disagreement n (%)
Social interactions	74 (37.95)	121 (62.05)
Behavioral control	306 (39.23)	474 (60.77)
Mood and emotions	569 (46.07)	666 (53.93)
Self-efficacy	146 (44.92)	179 (55.08)

Fig. 1.

Inter-rater reliability: Cohen’s Kappa and confidence limits for all items. SI: Social Interactions items (k = 3); ME: Mood and Emotions items (k = 19); SE: Self-Efficacy items (k = 5); BC: Behavioral Control items (k = 12).

The final analysis phase included developing folded mountain plots to examine the magnitude and direction of discordance or agreement between claimant and provider reports of behavioral health functioning for each of the four scales. These results presented in Figs 2a–2d indicate potential reporting bias (over-reporting or under-reporting) of the claimants vs. providers. Along the dimensions of Behavioral Control and Self Efficacy, providers tended to report lower levels of behavioral health functioning than did the claimants. In contrast, for the Mood and Emotions scale, claimants reported lower levels of behavioral health functioning as compared to their matched provider. The Social Interactions scale shows evidence of symmetry, meaning there are no systematic differences in claimant self report vs. provider report of behavioral health functioning in this domain relevant to work.

Fig. 2.

Mountain plots of agreement provider vs. claimant.

4. Discussion

When examining aspects of provider-patient (or in this study claimant) agreement, it is important to understand that no single statistical procedure is able to capture the true nature of any pair of judgments between two individuals [25]. For this reason, three methods were chosen to examine various aspects of provider-claimant agreement (1) Cohen’s kappa, (2) absolute percent agreement, and (3) mountain plots. All three methods of assessing claimant-provider agreement indicated inconsistent agreement as well as systematic differences in how claimants and providers perceive behavioral health function relevant to work. These findings raise questions about how best to assess behavioral health related work disability and which perspective – claimant or provider – provides the best approximation of the true unobservable status.

Both the item-level kappa coefficients and the absolute agreement for each scale indicate less than optimal concordance between claimant report and provider report of work related behavioral health function. Interpretation of the mountain plot graphs is that if there are no differences in the direction or magnitude, the graph would peak at approximately x = 0 and be symmetric around the line x = 0. This can be seen in the Social Interactions scale. If one respondent is less likely to give higher scores than the other (or vice versa) the peak will shift along the x-axis, indicating potential reporting bias. The results suggest that there are systematic differences between how providers and claimants report behavioral health functioning related to work disability. For the Behavioral Control and Self-Efficacy scales, providers systematically reported lower functioning than claimants reported. In contrast, in the Mood and Emotions scale, providers reported higher functioning than the claimant responses. The notion of claimants over-reporting their disability level is not fully supported by this study’s findings. Understanding the nature of underreporting or over-reporting work related behavioral health functioning is an area worthy of future exploration.

The results of this study suggest that for SSA claimants in applying for work disability due to mental or behavioral health difficulties, there is poor concordance and systematic inconsistency between their perceptions of behavioral health functioning in relation to their provider’s report. With the exception of the Mood and Emotions domain, findings in this study are consistent with previous literature discussing the tendency for providers to rate their patients health status lower than patient self-report [26]. The reverse finding that physicians tended to underestimate severity of mood disorders, including depression when compared to patient self-report, is also consistent with the literature [11,27]. These trends in over-reporting versus-under reporting disability level are important to consider in the context of SSA disability determination, as they may be a direct predictor of a claimants probability of applying for, receiving disability benefits, or appealing the SSA disability decision [3].

Both methodological and conceptual explanations can be formulated to address questions of why the agreement between providers and claimants may be low. From a methodological perspective, the kappa inter-rater agreement estimates are partially a function the amount of information provided by the response scale options [28,29]. When using dichotomous categories as compared to a larger 3–6 point scale, the reliability estimates often diminish as a function of the reduced number of response options – there is a significant loss of information when moving from a 5-point scale to a 2-point, dichotomous categories.

Additionally, there may be fundamental differences in how claimants and providers are interpreting the behavioral health functioning constructs being measured. The underlying construct of any given item in the four scales may mean something different depending on the perspective of the claimant, provider, and the context in which the respondent is thinking about the item. In this situation, the construct of behavioral health function may be inherently unstable, in which case little can be done to reduce the measurement error [28]. Depending on the extent to which this conceptual challenge is driving the discordance, alternative methods for measuring behavioral health related work disability may be required.

Some limitations of this study should be noted. First, the sample size of claimant-provider dyads was relatively small and ultimately limited the generalizability and depth of findings. Due to the small sample size, there was insufficient statistical power to conduct meaningful sub-group analyses by provider and claimant characteristics. Although this claimant sample does not differ significantly in reported demographic or condition characteristics from the overall sample of claimants from those who did not have a matched provider, other unmeasured selection factors may be present in this subset of claimants and providers. From a measurement perspective, the reduced form scores resulted in a significant loss of information and potential source of variation in responses. The truncated scales from a four-level to a dichotomous agreement scale limited the ability to perform a more refined analysis of nature of agreement in mental health functioning between providers and claimants. Lastly, the number of items retained in each scale differs. The degree to which the variation in agreement by scale as a function of its number of items cannot be ruled out in this particular analysis. Future work examining a broader set of items with a more robust response scale could allows testing to better understand discrepancies in claimant-provider agreement.

Various measures of agreement were used in this study; however, validity was not addressed. While two imperfect measures can yield a better approximation of the “truth” than either measure alone, the limited sample size of this study prevented application of such methods [30]. Without a true valid measure of behavioral health related work disability, no conclusions can be drawn as to the underlying nature of how these systematic differences relate to a person’s ability to work. In the absence of such a gold standard, careful attention must be paid in selecting informants to provide information for the disability determination process in terms of both number and perspective of the informants. Optimal information selection should triangulate informant responses to maximize information gained from all sources [6]. In this case, information from the provider and claimant are essential in assessing the claimant’s ability to work, but there is most likely additional key information, such as that provided from an employer perspective, which may be helpful in estimating true underlying behavioral health function.

In conclusion, this study illustrates the presence of discordance between provider and claimant reports of behavioral health functioning related to work. Given these findings, additional research is needed to accurately identify persons who are unable to work. Finally, understanding the claimant’s and provider’s perspectives of behavioral health functioning are necessary but not sufficient for determining work disability – aspects of the job demands and workplace environment should be considered when assessing a person’s overall ability to work.