Informatics Research on Mental Health Functioning – Decision Support for the SSA Disability Program

Abstract

The disability determination process of the Social Security Administration (SSA) disability program requires assessing work-related functioning for individual claimants alleging disability due to mental impairment. This task is particularly challenging because the determination process involves the review of a very large file of information, including objective medical evidence and self-reports from claimants, families and former employers. To improve this decision-making process SSA entered into an interagency agreement with the Epidemiology and Biostatistics Section of the Rehabilitation Medicine Department of the Clinical Center at the National Institutes of Health (NIH) intended to use data science and informatics to develop decision-support tools. This collaborative effort over the past decade has led to developing a Work Disability - Functional Assessment Battery and to testing an approach to applying Natural Language Processing to the review of claimants’ files for information on mental health functioning. This informatics research collaboration holds promise for improving the process of disability determination for individuals with mental impairments making claims at the SSA.

The Social Security Administration (SSA) administers two disability benefits programs, which are extremely consequential for millions of Americans, including those who experience a severe mental disorder. The two programs, Social Security Disability Insurance (SSDI) and Supplemental Security Income (SSI), pay benefits to millions of individuals who are determined to be disabled under a set of regulations operationalizing a statutory definition of disability. That definition requires that an individual must be “unable to perform any substantial gainful activity due to a medically determinable physical or mental impairment that has lasted or can be expected last for a continuous period of at least 12 months or result in death.” (1)

To begin the process an individual claimant files an application for disability with the SSA and usually is evaluated by a State Disability Determination Service (DDS). The DDS assembles a dossier on each claimant, including evidence on medical and non-medical issues, which can include information from multiple sources concerning the claimant’s health conditions, impairments and functional limitations. Sources may include the self-reported allegations of the claimant, reports from family and sometimes from previous employers or schools, reports from treating sources, and sometimes the reports from consultative examinations. Often the Medical Evidence Record (MER) extends to hundreds of pages or more of documentation, and it can be a challenge to organize the information so as to fairly and completely evaluate each claimant’s application for disability benefits. At times it is difficult to determine if the claimant has a severe impairment, such as schizophrenia or bipolar disorder, and whether the information in the MER supports a claim of functional limitations that preclude performance of substantial gainful activity (SGA). The disability examiner must be able to extract relevant information from pages and pages of the MER to evaluate a claim. Yet the MER often lacks information on functional limitations, such as in areas of mental functioning like cognition and social interaction. It would be advantageous for the disability examiner to have more information on functional abilities and limitations, such as from a functional assessment tool.

Because the disability determination process is so challenging, it often takes considerable time for deliberation, which can delay decisions and the payment of benefits; and the process is subject to appeals of decisions, hearings and even court cases. Over the past decade SSA has embarked on a program to develop a variety of decision tools to improve the determination process. This paper describes a collaboration between SSA and the Rehabilitation Medicine Department of the Clinical Center at the National Institute of Health, in which a team of epidemiologists, biostatisticians, disability subject-matter experts and data scientists have spent more than a decade working on various aspects of the development of a variety of decision-support tools, designed to improve the disability determination process. Some are tools to extract existing information from the MER, and others are new tools for collecting needed information on work-related functioning. We focus, in particular on the disability claims made on the basis of mental impairments, which pose a challenge to the disability determination process. This collaboration is generally unknown to the readers of Psychiatric Services; its scholarship mostly is published outside of the mental health field. But it is very relevant to research and practice in our field.

The paper outlines the special issues related to mental impairments and evaluation of mental functioning. The work is both conceptual and practical. It explores the ways in which a new paradigm for understanding disability affects research and in which new techniques in information science affect the design of assessment tools to support disability determination. The paradigm shift is consistent with contemporary concepts in disability and functioning, viewing disability and work-related functioning as a multi-factorial phenomenon resulting from the interaction between people and environments. (2 – 4) As such, measurement is challenging and novel approaches are needed. This special topic paper explores the implications of these concepts and describes a set of tools currently under development on the cutting edge of data science and the evaluation of functional limitations and disability. The developing tools are designed to fit into the disability determination process at SSA, but there is no specific assessment profile yet under consideration. While the use-case for tools is the SSA process, the tools may also have broader applicability beyond SSA.

The Special Challenge of Evaluating Disability Due to Mental Impairments

Mental impairments pose a special challenge for the disability determination process, which depends on objective medical evidence for making a decision. Accurate diagnosis depends on observation of signs of mental impairment consistent with self-reported symptoms. While it is challenging to infer ability to perform SGA based on the severity of any impairment for any body system, it is particularly true for mental impairments. The special difficulty arises because the process depends heavily on reconciling self-reported symptoms with observed signs made on mental status examinations or in clinical settings and from the observations of family and others, when available. To address this situation, SSA uses assessments of functional limitations and capacities related to work, such as ability to apply knowledge, interact with others, and ability to concentrate or adapt to new situations. These criteria are set forth in regulations, called Listings of Impairment, and in criteria for assessing what is called residual functional capacity (RFC). These functional measures are the principal criteria for assessment of disability due to mental impairments. A recent report from a consensus committee of the National Academies of Science, Engineering and Medicine (NASEM) on functional assessment reviews these issues in detail. (5)

SSA and its partners at NIH have addressed these issues by developing a new decision-support tool designed to elicit information on functional limitations and capabilities directly from claimants and beneficiaries. They also have been developing other decision-support tools to improve the assessment of functioning using natural language processing (NLP) to extract functional data from the large MERs assembled for each claimant. Developing these tools has necessitated some conceptual work on the relationship of impairment to functioning and the ability to perform SGA. The remainder of this paper describes the products and processes of the collaboration between the SSA and the NIH, with a special focus on assessing mental functioning.

History of SSA and NIH Clinical Center Collaboration

The SSA approached the NIH in 2007 in search of ways to support the disability determination process. The NIH Clinical Center’s Rehabilitation Medicine Department (RMD) had experience with disability assessment and large data analytics, which had direct value for SSA. This led to an interagency agreement (IAA) signed in 2008, which has been renewed annually and will continue in fiscal year 2022. The RMD team supporting this effort has grown over the years and, in order to address the complex challenges of disability assessment, has deliberately included a wide range of expertise such as computer science, epidemiology, medicine, occupational therapy, physical therapy, psychiatry, public health, statistics, and more.

The main goal of the IAA is to address the accuracy, consistency, and timeliness of SSA’s disability determination processes. This goal is supported by two broad objectives: 1) analysis of SSA data to develop data-driven approaches that inform SSA’s decision making processes; and 2) collection of more systematic and comprehensive data on function through the development of the Work Disability Functional Assessment Battery (WD-FAB). The WD-FAB is a self-report instrument to assess functioning relevant to work, across physical and mental spheres of activity. The WD-FAB scales were developed using Item Response Theory, which enables administration using Computer Adapted Testing methods in which the relevant questions are selected and administered, based on prior responses. The WD-FAB assesses work-related functional limitations in four areas of physical activity and four areas of mental functioning. (6 – 11) The development and ongoing research on the WD-FAB are discussed in more detail below.

Conceptual Foundations for Assessing Mental Functioning

Throughout the tenure of the SSA-NIH collaboration, NIH’s focus has been on function. While the statutory definition of disability is rooted in a medical model of disability, more modern models conceptualize disability as the gap between an individual’s functional abilities and their environmental demands, which in the SSA context is work demands. Therefore, we have used the World Health Organization (WHO)’s International Classification of Functioning, Disability and Health (ICF) as a framework for much of the work supporting this collaboration. (2) In particular, we focus on the Activity component of the ICF, which describes function at the level of the individual, rather than at the cellular, organ, or body system level. The team is working on an ontology to organize our thinking and our terminology concerning mental functioning to make it easier to communicate with other information scientists who might want to conduct studies or develop tools in this area.

The scientific literature (2 – 6, 11) and the practical experience of disability evaluators agree that the relationships among symptoms, mental functioning, and work performance are not always clear. Establishing this relationship is particularly challenging for determining if a claimant’s limitations in mental functioning preclude engaging in substantial gainful employment. Numerous personal and contextual factors, in addition to health status, are interconnected and can influence an individual’s overall ability to work. This complex relationship has been increasingly recognized as one of the fundamental challenges in work disability assessment. (3 – 5) For example, someone experiencing a psychotic disorder who may display maladaptive social behavior patterns may function well in a job that is relatively solitary and requires little interaction with others but may not be able to engage in work where collaboration and feedback are essential to the job functions. Additionally, with the emergence of telework and remote work options, there are many unknowns as to how the workplace changes will affect the evaluation of work-related mental functioning.

To that end, we have proposed grounding our work in disability assessment in a multidimensional conceptual foundation that goes beyond symptoms and impairments to include aspects of the work environment, functional abilities, and behaviors. Therefore, the Activity component of the ICF was key in guiding the development of the WD-FAB, focusing on an individual’s functioning related to work. The WD-FAB assesses function in eight areas across the two domains of physical and mental function. It produces a profile of scores in its four scales of mental functioning relevant to work: Communication & Cognition, Self-Regulation, Resilience & Sociability, and Mood & Emotions. (6) WD-FAB scale scores are standardized to the scores from a large general US working-age sample, with a mean of 50 and standard deviation of 10. This means, for example, that a score of 40 in Communication & Cognition is one standard deviation below the population mean. Extensive research was conducted on the measurement properties, such as validity, and reliability and related statistics such as minimal detectable difference. (6–10) The mental functioning measured in these scales also tracks closely with the functional criteria in current use at SSA for determining disability: understanding, remembering and applying knowledge; social interaction; concentration and task persistence; and adaptation and self-management.

In addition to assisting a disability examiner in making a disability determination, the WD-FAB potentially offers a way to assess work-related mental functioning comprehensively. The WD-FAB yields important information on an individual’s abilities and limitations, which can help assess the individual’s capacity for work or to perform other social and environmental roles. The results of a WD-FAB assessment might be useful to a clinician to identify the areas of functional abilities and limitations of a service user in a rehabilitation program. WD-FAB results also might inform a job search by an employment specialist on behalf of a supported employment client.

Current Work on the WD-FAB

After the WD-FAB was developed and its psychometric properties validated (6 – 11), research focused on supporting interpretation of the WD-FAB scores and aligning WD-FAB data with work-related outcomes. Our focus in supporting SSA’s disability determination process is to help understand the alignment of a person’s functional abilities with the functional capacity required to meet work demands. The WD-FAB helps measure functional abilities, and so now we are exploring the relationship between WD-FAB scores and work demands. We are in the process of collecting and analyzing data from three separate efforts to start to characterize this alignment:

The first is a pilot study comparing WD-FAB scores for individuals with self-reported work disability and for individuals who are currently employed. We are collecting data on three key job duties for the individuals’ last held position or current occupation. These data allow us to explore whether there are threshold scores on the WD-FAB indicating when individuals can work, as well as establishing functional profiles for the occupations represented in the study.

We are also collecting WD-FAB data as part of SSA’s Supported Employment Demonstration (SED). This is a longitudinal study to look at whether providing “evidence-based interventions of integrated vocational, medical, and behavioral health services to individuals with behavioral health challenges can significantly reduce the demand for disability benefits and help individuals remain in the labor force.” (12) The WD-FAB data allow us to characterize study participants, study how actual work-place function changes over time, and evaluate whether levels or changes in function, as reflected in WD-FAB scores, are associated with work outcomes. Assessing mental functioning in this way is a critical element in psychiatric services research, such as for evaluating outcomes in intervention studies.

SSA is also conducting a pilot study to look at the potential inclusion of the WD-FAB in their continuing disability review (CDR) process. The CDR is a periodic review of current beneficiaries’ cases to ensure they continue to meet SSA’s definition of disability. Like the SED, a key component of the CDR pilot study is understanding change in function over time as measured by the WD-FAB. New tools, such as the WD-FAB, should find their place in rehabilitation practice and in evaluation research that focuses on mental health and related impairments. (5, 6, 11) It is hoped that the WD-FAB will yield individual profiles characterizing functional abilities and limitations reported by the claimant – in both physical and mental functioning. These profiles would support the decision about the ability of a claimant to perform substantial gainful employment.

Using Natural Language Processing to Support Disability Decision-Making

Function at the level of the individual is also the principal focus of our analytic work. As part of SSA’s five-step determination process, a SSA adjudicator must determine whether or not a claimant can work, either at a past job or in any occupation in the economy. In order to decide whether a person can work, SSA needs information on the claimant’s functional abilities. SSA generally captures this type of information through the residual functional capacity (RFC) assessment, which has separate assessments for physical and mental function. (1) However, the RFC assessment currently is not based on a direct assessment of the claimant, such as with a tool like the WD-FAB, but rather is based on a summary compiled from the available medical evidence (the MER). For example, RFC assessment for functional limitations due to mental impairments involves evaluating information in the MER provided by psychiatrists and other mental health clinicians, as well as by the claimant, family members and former employers, when available. Information on function is often documented in the free text portions of the medical record, and therefore it can be time-consuming to locate all relevant evidence. Informatics methods, such as NLP, offer ways to more efficiently scan medical records and automatically identify, extract, and organize information on function that is relevant to SSA.

The large number of applications that the SSA receives requires accurate evaluation from the adjudicators to locate relevant medical and functioning information that can support the applicant’s claims. Given the significantly time-consuming process for reviewing applications, the adjudicators can greatly benefit from automated solutions that can extract and highlight the relevant information they are searching for (13). NLP models have proven to be very effective in extraction tasks with high performance, if provided with a sufficient amount of training data. When compared to functional information, general medical information extraction is shown to be an easier task given the ICD codes available in the applicant’s records and the ability to use predefined terminologies to extract diagnoses and symptoms (14–16). Functioning information, on the other hand, is more challenging to extract, given the lack of codes; and its presence as free-text that is highly dependent on context (17,18). This challenge is exacerbated in areas such as mental health, where functioning is found in the nuances of the language used in the free-text notes (e.g., “the patient is able to concentrate and follow instructions”) (19). While our team is conducting work in mental health functioning using NLP techniques, NLP research in the broader mental health domain is abundant, and focused on dataset generation (mainly from social media posts and electronic health record reports), automatic risk assessment, and symptoms and diagnosis extraction (e.g., using coded information such as concept unique identifiers (CUIs) [20], dictionaries [21,22], or ontologies [23,24]).

We proposed an information extraction framework for disability determination for mental health functioning (19) derived from a careful examination of the key elements of the SSA statutory definition of disability. To guide the extraction of mental health functioning information, we identified four dimensions to the extracted information that are crucial for disability determination: (i) temporal information including sequence and duration, (ii) level of performance or degree of difficulty in extracted mentions of functioning, (iii) the context with respect to work and work-related information, and (iv) the source of the information. All four dimensions are used to identify and characterize a “span of text” attributed to mental health functioning in the Medical Evidence Record of a claimant. The NLP technology extracts information from the MER that makes specific mention concerning a claimant’s mental functioning, and characterizes information related to when the functioning occurred and its duration, the degree of functional limitation, whether the functional limitation affected work or emerged in some other context, and who was reporting the limitation.

The following statements illustrate the kind of extracts that NLP might produce from the computer review of an outpatient clinic note: “The patient reports that he has been unable to complete simple tasks at home, such as watering his plants, for the past six months.” “The patient’s spouse said that he was very withdrawn at work, and he speaks little with her or with friends.” NLP also can predict or classify information in the MER or other records concerning various aspects of mental functioning, such as severity of a limitation, The approach in the SSA disability determination use-case employs a variety of NLP technical solutions used in other NLP applications, such as (i) temporal reasoning (25–27), (ii) sentiment analysis and risk assessment (28–33), (iii) incorporating the use of environmental characteristics and social determinants of health (34, 35), and (iv) author attribution (36). These NLP techniques have been used to evaluate suicide risk in social media posts and to assess the need for treatment. They also have been used to identify text in electronic medical records that characterizes the severity of symptoms, when they occurred, and who noted them. There is a risk that extracting small bits of narrative text can de-contextualize the information identified, potentially missing cultural or regional contexts, but these NLP extracts are not the sole source of information about a service user or disability claimant. In addition to NLP extracts, disability examiners and other users of NLP should also attend to contextual information available in the MER or other source material.

In 2017, the NIH team began a dedicated effort to develop NLP models and resources to extract information related to individuals’ function, again using the Activity component of the ICF to guide the framing and definition of the project. Recognizing that identifying and extracting information related to mental functioning would be particularly challenging, we formed a working group in the fall of 2018 to support NLP efforts around mental functioning. The first area of mental functioning we chose to model was the Interpersonal Interactions and Relationships (IPIR) chapter of the ICF.

The computer modeling work of IPIR aligned well with one of SSA’s key mental functioning criteria for determining disability in a claimant: interacting with others. The work required developing an annotation schema, which is an approach that allows the computer to capture (i.e., to identify and extract – search and find) IPIR information in clinical documents. The schema is based on a terminology built by extracting IPIR-related terms from SSA’s Mental Residual Functional Capacity assessment, all related to one of SSA’s criteria of mental functioning, interacting with others. Based upon this list of IPIR-related terms, an IPIR schema for annotation and an IPIR annotation guideline were developed. The data set used for annotation and computer modeling came from clinical and medical records from the National Institutes of Health Clinical Center (NIH-CC) and disability applications from SSA. Using the annotated data for training and validation, NLP models were built to automatically detect and extract sentences containing IPIR information from the medical record. Inter-annotator agreement was a challenge prior to training, but a study of agreement among trained raters achieved a Cohen’s kappa of 0.81. (37) Examples of IPIR extracts could include the following: from a note reporting on an occupational therapy group: “The patient did not interact with anyone in the group session today, similar to behavior, off-and-on, during the past year” and a progress report from an inpatient psychiatric unit: “The patient engaged in a heated argument with family members during a family therapy session.”

NLP tools for extracting mental functioning content from narrative text have potential practical benefits beyond supporting SSA disability determination. Such tools might be useful in any review of the health records of a service user, such as in a second opinion review of someone with an extensive clinical history and a voluminous chart. NLP tools might also be used in managed care reviews, searching for specific categories of information, such as mental functioning and types of impairment. And these tools could be used in forensic evaluations and legal depositions to extract all relevant information on a specific topic from testimony and written evidence. Furthermore, NLP tools have been used to extract data to determine clinically significant measures, such as the duration of untreated psychosis. (26)

Potential Impact of the Collaboration between NIH Clinical Center and SSA

The collaboration between SSA and the NIH has yielded numerous current benefits and promises more in the future. It has advanced conceptual thinking on disability and the relationship between impairments and functional limitations, and between those limitations and work-related activities. These are thorny issues, but these new ideas have encouraged SSA to emphasize functioning and work activity along with its focus on impairment severity. Conceptual work has also contributed to thinking about approaches to using informatics techniques, such as NLP, to support disability decision-making. Applying those concepts to tool development has guided NIH and its collaborators in academia to developing the WD-FAB, and to pilot its use in various aspects of SSA’s disability determination processes.

Developing decision-support tools for use in disability determinations holds the promise of more complete evaluations, taking less time, perhaps with fewer appeals. People who use psychiatric services often depend on SSA disability benefits as the sole source of income support. Beneficiary status also confers eligibility for Medicaid and Medicare benefits, which give them access to behavioral health services. Eligibility status for these benefits may also confer eligibility for other social services and subsidies for housing or food. Greater accuracy and faster disability determinations are better for applicants and for the broader society as well - building confidence in the important disability programs at SSA. And beyond their potential use in SSA disability determination, these tools could also provide insights into approaches to rehabilitation that might lead some individuals to employment or to other forms of social participation.

This paper has described ways in which the informatics field has much to offer the field of psychiatric services research and practice in disability assessment and rehabilitation. And it describes a valuable, ongoing collaboration between two venerable U.S. institutions – SSA and the Clinical Center at the NIH – with important implications for the field.

HIGHLIGHTS:

• Assessing mental health functioning is challenging for determining disability due to mental impairments for the two Social Security Administration (SSA) disability programs, Social Security Disability Insurance and Supplemental Security Income.

• The Social Security Administration has been funding collaborative informatics research with the Epidemiology and Biostatistics Section of the Rehabilitation Medicine Department of the Clinical Center at the National Institutes of Health (NIH), designed to use data science to develop decision-support tools.

• To enhance the efficiency of disability determination and to provide decision-support to SSA this collaborative effort has led to developing the Work Disability - Functional Assessment Battery and to testing an approach to applying Natural Language Processing to the review of claimants’ files for information on mental health functioning.