Cross-validation of the screening scale in an adult protective services sample

Abstract

Adult Protective Services (APS) professionals are often called on to assess decision-making capacity when investigating financial exploitation. Previous research found that in consecutive APS cases, a decision-making screening scale (LFDSS) also detected financial exploitation. The purpose of this study was to apply the clinical cutoff scores derived from the previous study to a new sample of APS cases. Using a sample of 105 participants, from APS workers across 5 counties this study investigated the clinical utility of the LFDSS to detect financial exploitation based on ratings by APS professionals using the scale. Results demonstrate that the LFDSS has excellent internal consistency and clinical utility properties. This paper provides support for use of the LFDSS as a reliable and valid instrument. Instructions for use of the LFDSS are included in the article, along with information about online support tools.

Introduction

Lichtenberg et al. (2016, 2017) have demonstrated that when Adult Protective Service (APS) workers substantiate financial exploitation, a lack of informed financial decision-making is frequently identified. Indeed, decision-making capacity is often a key assessment in financial exploitation cases, and the ability to perform such capacity assessments is one of the core competencies for APS workers defined by the National Adult Protective Services Association (2013). The vital nature of this competency is underscored by the prevalence of financial exploitation, which is a significant societal problem: In the United States alone, financial exploitation is estimated to cost older adults $2.9 billion each year (MetLife, 2011). One study found that the one-year prevalence rate of older adult financial exploitation by a family member is 5% (Acierno et al., 2010). Due to the great personal and societal burdens imposed by financial exploitation, many disciplines have sought to understand and assess risk for exploitation. This study seeks to extend empirical investigation to the linkage between financial decision-making deficits and substantiated financial exploitation in APS cases.

Much of the empirical research on financial exploitation risk has focused on measuring financial decision-making capacity—that is, the broad ability to make and execute thoughtful money management decisions. A review by Marson (2016) outlines several models of financial capacity assessment. One such model, the University of Alabama at Birmingham (UAB) clinical model, conceptualizes financial capacity as containing nine domains, including Basic Monetary Skills, Checkbook Management, and Financial Judgment. This model was developed to examine how financial capacity changes in relation to pathology, such as Alzheimer’s disease, and is therefore structured as a series of objective and novel tasks. In his review, Marson concluded that loss of financial capacity is significantly related to the pathology of both Mild Cognitive Impairment (MCI) and Mild Alzheimer’s disease. This finding is critical in light of the prevalence of cognitive impairment among older adults. One study suggests that as many as 30% of those older than 71 years’ experience some degree of cognitive impairment (Plassman et al., 2008), and age remains the greatest known risk factor for developing Alzheimer’s disease (Stern et al., 1994). While the prevalence of dementia may be in decline (Satizabal et al., 2016), Boyle et al. (2012) and Boyle, Wilson, Yu, Buchman, and Bennett (2013) found that poorer decision-making was a consequence of cognitive decline, even in the absence of MCI or Alzheimer’s disease. Even more troubling, difficulties with decision-making were related to shortened longevity. The works of both Marson (2016) and Boyle et al. (2012, 2013) underscore how cognitive decline is related to changes in general financial capacity and decision-making capacity. These findings may not directly translate, however, to real-world financial decision-making, and especially to high-impact, sentinel decisions; that is, those financial transactions that carry the potential to result in significant losses or harmful consequences.

The assessment measures used to evaluate financial decision-making skills have historically employed hypothetical scenarios and vignettes (Boyle et al., 2012; James, Boyle, Bennett, & Bennett, 2012; Marson et al., 2000), and do not assess the capacity to make actual, specific financial decisions. The reasoning applied to a hypothetical situation may not correspond to the real-life financial decisions older adults are required to make regarding personal financial transactions. An older adult who has historically taken responsibility for financial decision-making may persist in that role even after developing cognitive impairment (Hsu & Willis, 2013), and some older adults with cognitive impairment remain able to reason appropriately about a financial decision. This renders nuanced assessment even more critical. An additional drawback to comprehensive financial capacity assessments is that they are typically lengthy, and require administration and interpretation by a qualified mental health professional. While comprehensive financial capacity assessment may be ideal in some instances, this is not always practical or possible; many of the tools used in such assessments require significant time and training to administer.

APS professionals who interact with older adults need standardized financial capacity measures that assess real-world decision-making skills. The Lichtenberg Financial Decision Screening Scale (LFDSS) is unique, in that it is person-centered (i.e. holistic and strengths-based) and multiple choice, and focuses on the actual financial decision(s) the older adult has made or intends to make. In addition, the scale has been examined empirically to determine how well it works. In studies by Lichtenberg et al. (2017) and Teresi et al. (2017), the scale’s reliability, item utility, and validity have been documented, and a specific risk-scoring system has been analyzed and a cutoff score profile established.

In this study, we aimed to examine and apply the risk-scoring system in a new set of 105 APS cases and perform a more stringent test of the scale’s clinical utility.

Conceptual Foundation of the Lichtenberg Financial Decision Screening Scale

The LFDSS was designed based on a dual-framework conceptualization that considers both decisional abilities and person-centeredness. The approach emphasizes the importance of the older adult’s ability to understand a specific financial decision and clearly communicate their rationale for the decision.

Person-Centeredness in Financial Decision-Making

A person-centered approach to working with older adults seeks to support autonomous decision-making by emphasizing their personal strengths and respecting their values, choices, and preferences (Fazio, 2013). Within the domain of assessment, Mast (2011) has developed a method for evaluating individuals with neurocognitive impairment—the Whole Person Dementia Assessment—that seeks to balance standardized assessment procedures with these person-centered principles. The underlying assumptions are that (1) people are more than the sum of their cognitive skills, and (2) traditional assessment approaches are deficit-oriented and do not bolster an individual’s cognitive strengths. This person-centered focus was central to development of the LFDSS for evaluation of actual financial decisions an older adult has made or is considering.

Decisional Abilities Framework

The LFDSS is also anchored in the decisional abilities framework proposed by Appelbaum and Grisso (1988). By examining the legal standard set by various states to determine incapacity, Applebaum and Grisso identified four factors that are essential for informed decision-making: choice, understanding, appreciation, and reasoning. Choice, in this framework, is the older adult’s ability to communicate their autonomous financial decision, free from coercion. The individual must also be able to demonstrate understanding by describing the financial decision and its risks and benefits. Appreciation refers to the individual’s comprehension of the financial decision and its potential consequences, for both themselves and others. This is vital because, according to Appelbaum and Grisso’s findings, the most common causes of impaired appreciation are lack of awareness of deficits and/or delusions or distorted thinking. Finally, reasoning is the ability to describe a clear rationale for pursuing the financial decision, and includes the ability to compare alternative decisions. Both the American Bar Association (ABA) Commission on Law and Aging and the American Psychological Association (APA) support these factors as critical to decision-making. In their joint publication, Assessing Diminished Capacity in Older Adults: A Handbook for Attorneys (ABA/APA, 2008), they encourage attorneys to assess decisional abilities whenever diminished capacity is a concern.

Using the conceptual framework described above, a standardized assessment tool that offers the flexibility to understand the older adult’s decision-making within their unique environmental context was created. It is critical to evaluate the older adult’s ability to express choice, understanding, appreciation, and understanding of the financial decision while simultaneously respecting their autonomy to make choices in line with their personal values and preferences. It is also essential to have a standardized instrument that is rooted in evaluating an actual financial transaction. This approach allows for assessment across different financial decisions, while also ensuring that the same decisional skills are observed across contexts. In order to offer an informed opinion regarding the presence or absence of financial exploitation, a third party, such as APS, must be able to assess financial decision-making abilities using measures that are rooted in a specific financial decision.

Purpose of the Study

The purpose of this study was to examine the case-finding utility of the risk-scoring system created by Lichtenberg et al. (2017) by applying it to a new set of 105 APS cases. Specifically, we sought to use the previously recommended cutoff score of the LFDSS risk-scoring system to differentiate those who had financial exploitation substantiated by an APS worker versus those who did not. We also evaluated the reliability of scale items and the sensitivity and specificity of the scale’s risk-scoring system to detect substantiated exploitation. Finally, we examined the presence of age, gender, and education effects on the LFDSS risk score. Four hypotheses related to the LFDSS were generated:

• Hypothesis One: The LFDSS risk score will differentiate between older adults whose financial exploitation has been substantiated by APS professionals and those whose financial exploitation has not. That is, an older adult whose case is substantiated will have a significantly higher risk score than one whose case is not substantiated.

• Hypothesis Two: The items of the LFDSS will demonstrate adequate reliability as a unidimensional scale.

• Hypothesis Three: The LFDSS will demonstrate adequate sensitivity and specificity to detect substantiated financial exploitation.

• Hypothesis Four: Age, gender, and education will be unrelated to the LFDSS risk score in older adults, regardless of whether their cases were substantiated by APS professionals.

Methods

Participants

APS professionals from five counties (two in the U.S. Midwest and three on the East Coast) assessed 105 participants for financial capacity and/or financial exploitation. Participants were eligible if they were (a) 60 years or older; (b) considering or had recently—i.e., within the past six months—made a significant financial decision (or series of related decisions, such as multiple gifts to the same person); and (c) willing to participate. The LFDSS was administered to consecutive participants who agreed to be interviewed using the SS. No personal identifying information was collected, but the participant’s age, gender, and years of education were recorded. Mean age of the sample was 77.5 years (SD=10.8 years); mean education level was 12.8 years (SD=2.2 years); and 57.1% (n=60) were female. Notably, education was only recorded in 59% (n=62) of cases. For all participants, age, education, and gender were collected, but personal or identifying information was not. Because the data were anonymous, the Wayne State University Institutional Review Board issued a concurrence of exemption. Although written informed consent was not required, the individuals being assessed received an information sheet that included the elements of a consent form.

Training and Implementation of APS Workers

APS sites and training:

APS staff from five U.S. counties participated in the study and received multifaceted training in the use of a new tool to assess informed financial decision–making. Since the tool would be used in the context of aging issues, the training began by introducing concepts such as cognitive decline and cognitive impairment, as well as isolation and depression. Most of the APS supervisors and workers reported that they had never received training on Alzheimer’s disease or geriatric depression; many had worked in either domestic violence or child protective services before joining APS. Also, the training entailed multiple sessions in multiple educational modalities.

First, using a traditional lecture and discussion method, an overview of the LFDSS and its administration were provided. Second, in a follow-up session, multiple case examples were presented, which formed the basis for guided discussion; these sessions were enhanced by APS workers’ descriptions of their own cases. Third, after initial practice in the scale’s use, another training was provided to clarify the tool’s proper and improper uses and administration. Despite extensive training, when the first batches of tools (de-identified) were sent by APS for entry into our database, all had been filled out incorrectly; this was the case across all five settings. Several common misconceptions about the tool were identified, as well as some issues regarding the scale’s layout. The main error was the APS worker’s failure to provide a rating for each item, and instead only record the older adult’s responses. APS workers and supervisors recommended changes to the LFDSS form to include a short set of instruction reminders and prompts for APS workers to record their own ratings. Once these steps had been taken, data collection proceeded smoothly.

Case review with supervisors:

Our relationship with supervisors, and their willingness to have multiple contacts with the tool’s creator, was key to the success of implementation. Supervisors were critical to several aspects of implementation: motivating staff, making the tool a required aspect of any financial-exploitation investigation, learning more themselves about how the tool worked, and providing process-improvement feedback. For example, as one supervisor pointed out initially, some of the problems were with the form itself. APS supervisors identified ways to improve the form, and, as described previously, short instruction reminders and more obvious spaces for APS workers’ ratings were added. In this case, for example, a new column of check boxes for APS worker ratings was added. Another problem was inadequate understanding of the tool on the part of workers and some supervisors. Therefore, communication proved to be a vital component to regular implementation. Specific errors were not identifying the specific decision or set of decisions made by the older adult; and not limiting responses to the primary or single best answer. When cases were discussed one on one with front-line supervisors in a timely manner, the tool’s purpose and utility were easier to grasp. Based on conversations with supervisors, it was clear that front-line workers were becoming more adept at using the tool. As a result, 105 new assessments were collected and the tool became a standard aspect of the financial-investigation process in all five counties.

Measures

Demographic Measures

Age, gender, and education were collected by self-report. It is important to examine potential biases in LFDSS scores, so that it can be adjusted for any known differences based on demographic variables.

Lichtenberg Financial Decision Screening Scale (LFDSS)

The LFDSS is a 10-item scale that contains 7 items from the Lichtenberg Financial Decision-making Rating Scale (LFDRS, 2015) Intellectual Factors subscale and three from the LFDRS Susceptibility to Undue Influence subscale. Two scores were calculated for the APS professional’s use: (1) a dichotomous score that indicates whether or not the case was substantiated and (2) an overall risk score. Whether or not the case was substantiated by the APS worker served as the measure of criterion-related validity for the LFDSS. The overall risk score of financial exploitation was derived from 5 of the 10 scale items. The literature supports the use of an ordinal risk score for these five items. A higher risk score is assigned for riskier financial decisions; for example, an individual whose financial decision would significantly change previously established bequests would be assigned a higher score than someone whose financial decision did not change those bequests. The remaining five LFDSS items are neutral and used to describe the nature of the financial decision, rather than contributing to the overall risk score. For example, establishing a new will is not inherently risker than making a new investment. However, the highest risk score is assigned if the administrator determines that the older adult’s response is inaccurate or the older adult does not demonstrate adequate knowledge about the financial decision. Further information on scoring is available from the corresponding author.

The LFDSS is designed to be administered as a structured interview, with multiple-choice response options. The interview should be conducted in a standardized manner, and questions are to be read aloud as they are written. However, if the older adult responds before all of the options have been provided, the administrator may rate the response without reading the remaining options. The necessity of reading all of the options is determined by the administrator. The older adult should be encouraged to elaborate on any of their responses, but especially in cases in which the initial response is unclear. Elaboration can be requested by the administrator or occur spontaneously. All elaborations should be written down verbatim on the response form.

Scoring each item

The interviewer’s knowledge of LFDSS items and judgment of the older adult’s responses is critical. The LFDSS is a rating scale, and scoring entails two steps:

1. The older adult’s response should be recorded by checking the corresponding box in the older adult’s column for each item.

2. The interviewer should check the corresponding box in the APS worker’s column for the answer the interviewer believes is most accurate. When there is a discrepancy between the older adult’s answer and the worker’s answer, the item will be scored as inaccurate. Seven of the 10 items are used in the scale’s risk-scoring algorithm.

Data Analysis

An independent samples t-test was used to examine differences in LFDSS risk scores for those who had been financially exploited and those who had not. The criterion measure for exploitation was whether the case had been substantiated by APS. To examine the LFDSS’s sensitivity and specificity for detecting financial exploitation, a receiver operator characteristic (ROC) curve analysis was conducted. Cronbach’s alpha was computed for the items of the LFDSS to examine individual items’ reliability and fit to a unidimensional scale. Finally, Pearson’s r correlations were computed for the LFDSS risk score with age and years of education, and a point-biserial correlation was conducted to examine the relationship between gender and LFDSS risk score.

Results

In this sample, APS workers determined that 61% (n=64) of the cases had substantiated financial exploitation; the remaining cases did not (Table 1). The groups did not differ on age t(100)=−.42, p=.67; education level t(58)=.94, p=.35; or gender χ2(1)=.923, p=.34. As can be seen in Table 1, an independent samples t-test found that LFDSS risk scores significantly differed based on the occurrence of financial exploitation t(101)=6.99, p<.001, d=1.37. A post-hoc power analysis demonstrated that at the present effect size, the sample had sufficient power (.999) to detect the effect. That is, older adults whose APS case was substantiated (M=8.88, SD=3.08) had significantly higher LSFSS risk scores than those whose cases were not substantiated (M=3.87, SD=4.16).

Table 1.

Demographic Percentages for Elders Screened at Adult Protective Services (Substantiated vs. Unsubstantiated)

	Total Sample (n=105)	Case Substantiated (n=64)	Case Unsubstantiated (n=39)
Age (mean/SD)	77.58 (10.8)	77.24 (11.67)	77.2 (7.17)	t(100)=−.42, p=.67
Education (Mean/SD)
Years of Education	12.79 (2.23)	12.93 (2.39)	12.35 (1.87)	t(58)=.94, p=.35
Gender (%/n)
Female	57.1% (n=60)	60.9% (n=39)	51.3% (n=20)
Male	42.9% (n=45)	39.1% (n=25)	48.7% (n=19)	χ2(1)=.923, p=.34

ROC curve analysis (Figure 1) found good sensitivity and specificity of the LFDSS score to detect exploitation (AUC=.841). A prior study (Lichtenberg et al., 2017) that examined the ability of the LFDSS to detect financial exploitation found the ideal cutoff score to be 5 or greater. This study also supports a cutoff score of 5, with a sensitivity of 0.922 and a specificity of 0.75 (Table 2). The overall correct classification (85.6%) of participants based on LFDSS score peaks at a cutoff score of 5 or greater. As can be seen in Table 2, however, if the scale were used in a manner that favored higher sensitivity without regard for specificity, a cutoff of 3 could be used, which would detect financial exploitation 98.4% of the time. Also, if the LFDSS score required higher specificity, a more conservative cutoff of 11 could be used, which would specifically detect financial exploitation in 90% of cases. Finally, LFDSS items were found to have adequate reliability as a unidimensional scale, measured by Cronbach’s alpha (α=.729).

Figure 1.

Receiver Operating Characteristic (ROC) Curve for the LFDSS Score Predicting Exploitation

Table 2.

Sensitivity, Specificity, Positive Predictive Value, Negative Predictive Value, and Overall Correct Classification Were Calculated at Each Potential Cutoff Point for the LFDSS

Cutoff	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value	Overall Correct Classification
1 or greater	1.00	.075	.643	1.00	.644
2 or greater	.984	.425	.733	.944	.769
3 or greater	.984	.500	.759	.952	.798
4 or greater	.969	.650	.816	.929	.846
5 or greater	.922	.750	.855	.857	.856
6 or greater	.859	.775	.859	.775	.827
7 or greater	.781	.825	.877	.702	.798
8 or greater	.656	.825	.857	.600	.721
9 or greater	.578	.850	.860	.557	.683
10 or greater	.422	.875	.844	.486	.596
11 or greater	.281	.900	.818	.439	.519
12 or greater	.188	.900	.750	.409	.462
13 or greater	.125	.925	.727	.398	.433
14 or greater	.063	.925	.571	.381	.394
15 or greater	.063	.975	.800	.394	.413

Pearson’s r correlations (Table 3) showed that the LFDSS score was not correlated with age (r=−.076, p=.44) or level of education (r=−.056, p=.67), and a point-biserial correlation revealed that the LFDSS score was also not related to gender (r=.022, p=.82). However, a point-biserial correlation showed that LFDSS score was significantly related to financial exploitation (r=−.571, p<.001).

Table 3.

Pearson’s r Correlation of LFDSS Risk Scores, Demographic Variables, and Financial Exploitation Status (Substantiated vs. Unsubstantiated)

	Age	Education	Gender	Exploitation Status
Risk Score	−0.076	−0.056	0.022*	−0.571*
p-value	0.44	0.67	0.82	<.001

^*Point biserial correlations were used to examine the relationship of Risk Score to Gender and Exploitation Status.

Discussion

The results of this study are consistent with the previous findings of Lichtenberg et al. (2017): The LFDSS successfully differentiates between older adults who have been financially exploited and those who have not. ROC curve analyses support the previously established cutoff score of 5, and scored LFDSS items demonstrated adequate reliability. Finally, the LFDSS risk score did not correlate with the participant’s age, gender, or education level, which suggests that no adjustments are needed in order to interpret scores based on these factors. The LFDSS risk score did correlate, however, with financial-exploitation status. These results, coupled with the brevity of the scale, support the LFDSS as an efficient method of interviewing and scoring for risk of financial exploitation.

APS professionals are on the front line in aiding older adults who experience financial exploitation, and it is well established that assessment of decision-making capacity is a significant element of their work. However, few practical, empirically supported models for assessing financial decision-making are suitable for use by APS professionals; that is, many models of financial decision-making that have been put forth in the empirical literature have focused on broad capacity assessment. Such assessment is costly, both in time and resources, and not always feasible. Also, these decision-making capacity instruments and financial decision-making items, use hypothetical items that are not easily translated into practical use by APS workers. Focused assessment on actual decisions is much better suited for use in the field. This need is highlighted by the lack of training in aging-related issues described by the APS workers in the present study.

The Assessment of Competency for Everyday Decisions (ACED, Lai et al., 2008) instrument focuses, like the LFDSS, on specific, real-world decisions. Unlike the LFDSS, however, the ACED is an open-ended rating scale; our experience with APS workers is that confusion would likely arise regarding how to make ratings on an open-ended scale. Our scale, when administered online (https://olderadultnestegg.com), is not only structured and efficient, but provides risk scores and recommendations for next steps. The LFDSS satisfies APS professionals’ need for a brief, standardized assessment measure of financial decision-making and exploitation risk for a real-world, significant financial transaction(s).

The study has several limitations. Though the sample of 105 participants was large enough for the purposes of this study, it would be valuable to study the LFDSS in a larger sample across more APS sites. We did not examine the types of financial exploitation participants had experienced, and therefore it is not possible to evaluate how sensitive the LFDSS is to severity of exploitation. The study did not examine potentially relevant characteristics of the sample, such as history of prior financial exploitation, cognitive abilities, or psychological vulnerability. Finally, we did not systematically evaluate the acceptability of LFDSS use by APS professionals, and it will be important to understand how the scale is perceived by APS professionals, as well as how the scale is used in practice to assist older adult clients.

Despite these limitations, the study makes a valuable contribution to the professional practice of assessment of decision-making capacity and financial exploitation by Adult Protective Services professionals.