Making and Executing Decisions for Safe and Independent Living (MED-SAIL): Development and Validation of a Brief Screening Tool

Abstract

Objectives

Older adults prefer to remain in their own homes for as long as possible. The purpose of this article is to describe the development and preliminary validation of Making and Executing Decisions for Safe and Independent Living (MED-SAIL), a brief screening tool for capacity to live safely and independently in the community.

Design

Prospective preliminary validation study.

Setting

Outpatient geriatrics clinic located in a community-based hospital.

Participants

Forty-nine community-dwelling older adults referred to the clinic for a comprehensive capacity assessment.

Measurements

We examined internal consistency, criterion-based validity, concurrent validity, and accuracy of classification for MED-SAIL.

Results

The items included in MED-SAIL demonstrated internal consistency (5 items; α = 0.85). MED-SAIL was significantly correlated with the Independent Living Scales (r = 0.573, p ≤ 0.001) and instrumental activities of daily living (r = 0.440, p ≤ 0.01). The Mann-Whitney U test revealed significant differences between the no capacity and partial/full capacity classifications on MED-SAIL (U(48) = 60.5, Z = −0.38, p <0.0001). The area under the curve was 0.864 (95% confidence interval: 0.84–0.99).

Conclusions

This study demonstrated the validity of MED-SAIL as a brief screening tool to identify older adults with impaired capacity for remaining safe and independent in their current living environment. MED-SAIL is useful tool for health and social service providers in the community for the purpose of referral for definitive capacity evaluation.

Remaining in one’s own home, or aging in place, continues to be a primary goal for most older adults. Aging in place may not always be possible or safe,^1,2 however, particularly for those with capacity impairments. Capacity for safe and independent living may be threatened by functional, psychological, and physiological impairments, increasing the risk of long-term care transitions, morbidity, and mortality.^3,4 Though such impairments may impede the ability to meet environmental demands,⁵ vulnerable elders often resist the transition to a more supportive environment, preferring instead to remain in the familiar surroundings of their home and community, which provide feelings of comfort, attachment, control, and independence.²

Difficulty with tasks such as transportation, bill paying, or home care may trigger a request for assistance from community-based health and social service providers (social workers, home health nurses, adult protective services investigators, etc.), who are likely the first point of contact with social services or the healthcare system for elders struggling to maintain independence.⁶ Upon evaluation, it may become evident that these difficulties arise from impaired capacity for independent living. Health and social service providers in the community must be able to quickly and reliably determine whether the elder has needs that can be addressed in the home or if referral for comprehensive capacity testing is needed. Incorrect judgments regarding capacity can result in an older adult remaining in an inappropriate environment without needed supports⁷ or, conversely, result in unjustly limiting an individual’s rights.

Existing methods for assessing capacity have several limitations. Capacity assessments commonly used among clinicians focus on decisional components of capacity such as short-term memory, attention/concentration, and spatial orientation.^7–10 These tools do not address the overlap of function, cognition, and judgment; capacity for independent living lies at the confluence of these domains.^7,9 Even when more comprehensive methods of capacity assessment are utilized, they are often time-intensive and involve multidisciplinary teams in a medical setting.¹⁰

To address these gaps in the literature and practice, Making and Executing Decisions for Safe and Independent Living (MED-SAIL) was developed based on widely cited standards^9,11–13 for assessing medical decision-making capacity and the authors’ previous research.^6,14–18 MED-SAIL is a brief screening tool that allows community-based professionals to quickly determine whether an older adult has sufficient capacity to safely remain in their current environment or if a comprehensive capacity assessment is warranted. This report describes the development of MED-SAIL and presents preliminary validation data from a pilot study, including reliability, criterion-based validity, concurrent validity, and accuracy of classification.

METHODS

The process of developing MED-SAIL included 1) creation of the MED-SAIL scenarios grounded in qualitative evidence;⁶ 2) finalization of the scoring scheme based on an existing criterion based framework;^8,9,12,19 and 3) preliminary validation of MED-SAIL using pilot data. The presentation of our findings was guided by the STARD checklist for the reporting of studies of diagnostic accuracy.²⁰ This study was approved by institutional review boards at Baylor College of Medicine, Harris County Hospital District, and the Michael E. DeBakey VA Medical Center.

Development of the MED-SAIL Scenarios

We first examined the bioethical and clinical literature to develop a comprehensive set of standards to assess whether older adults could make and execute decisions regarding health, safety, and independence.²¹ A series of five focus groups were conducted with community-based health and social service providers to clarify how to operationalize these standards to screen for vulnerability among community-dwelling older adults.⁶ The first focus group drafted case scenarios based on situations an older adult might face in their everyday life related to independent living. In an iterative process, each subsequent group was shown scenarios developed by the previous group(s) as a starting point for further refinement. At the conclusion of the focus groups, the research team finalized a set of seven scenarios, which became the foundation for MED-SAIL:

1. The door to your home is locked and you do not have a key.

2. You run out of a medication that you take regularly.

3. You are at home and suddenly there is a fire in your kitchen.

4. You notice that the cut on your foot is not healing and has become infected.

5. Someone calls you saying you’ve won $100,000 and all they need from you is your social security number to verify your identity.

6. You are driving to the grocery store and you get a flat tire.

7. Your heating unit [air conditioner] breaks down and it is very cold [hot] outside.

The administrator chooses two scenarios most relevant to the respondent’s life. Included in the MED-SAIL training guide are a sample introductory script and a list of questions (e.g., “Do you use a key to get into your home?” or “Do you drive a car?”) that can be asked to determine the most appropriate scenarios if the administrator is uncertain.

MED-SAIL Scoring

Scoring for MED-SAIL is based on four domains derived from established theoretical frameworks for evaluating capacity to consent to medical decisions: understanding, expressing a choice, reasoning (problem solving/consequential reasoning and comparative reasoning), and generating consequences (Fig. 1).^8,9,12,19 The development of a measure of appreciation, another domain of capacity, was deferred until future study (see Discussion section).

FIGURE 1.

Description of MED-SAIL Domains and Questions/Prompts for Assessment.

The first domain (understanding) is scored as either 0 or 1 and if the respondent cannot successfully demonstrate understanding, the assessment is stopped. Consistent with our theoretical framework,^8,9 the remaining four items are scored on a scale of 0 to 2 based on the logic and completeness of the response (0 = no answer, incomplete, or illogical; 1 = logical, but incomplete; 2 = complete and logical). Each scenario has a maximum of nine points, with the final score consisting of the average of the two scenarios chosen to be administered. For the current study, administrators were trained to use MED-SAIL through didactic education and guided practical experience. These materials have since been used to develop a MED-SAIL training manual and accompanying DVD, which includes dramatizations of assessments involving individuals presenting with different levels of capacity. The MED-SAIL manual also provides example scored items by scenario and by item.

Established cognitive screening tools require that the administrator work, without deviation, from a given script and responses are scored with no additional probing or explanation permitted. MED-SAIL administrators are encouraged to prompt respondents to elaborate or clarify answers, which is consistent with literature supporting the necessity of probing within narrow criterion standards to effectively establish capacity.^8,9,12,15 The empiric foundation for the structure and scoring that MED-SAIL uses was first established by the MacArthur Competence Assessment Tool-Treatment,⁸ which has demonstrated validity and feasibility with appropriate instruction.

Preliminary Validation of MED-SAIL

MED-SAIL was pilot-tested at an outpatient geriatrics clinic located within a local community hospital. Over a 2-year period, MED-SAIL was administered to 49 participants as part of routine care for patients referred for the clinic’s existing comprehensive capacity evaluation program.¹⁸

The criterion standard

The criterion standard to which MED-SAIL was compared included both objective, validated instruments coupled with an individualized clinical assessment (e.g., physical examination, lab work, psychosocial assessment) administered by a multidisciplinary team in the geriatrics clinic. These assessments address issues determined through literature review, focus groups, and clinical observation to be critical to a comprehensive capacity evaluation: cognitive abilities, ability to reason/judgment, personal appearance/grooming, safety of the environment, and adequate and safe performance of everyday tasks.⁶ A determination of capacity is made through consensus of the interdisciplinary team based on the standardized test scores in conjunction with clinical assessment. The MED-SAIL administrator was not aware of the capacity determination at the time of assessment.

Five standardized measures are included in the criterion standard capacity assessment. The St. Louis University Mental Status Examination (SLUMS) is an 11-item screening questionnaire designed to test orientation, memory, attention, and executive function. SLUMS is more sensitive than the Mini Mental State Examination in identifying a diagnosis of mild neurocognitive disorder.²² SLUMS scores range from 0 to 30. For an individual with a high-school education, scores of 27–30 are considered normal, scores of 21–26 indicate mild neurocognitive disorder, and scores between 0 and 20 suggest dementia. The Patient Health Questionnaire (PHQ-9) is a nine-item self-report depression scale based on the DSM-IV diagnostic criteria for major depressive disorder.²³ Scores on the PHQ-9 are categorized into several levels of depression: none (0–4), minimal (5–9), mild (10–14), major depression–moderate severity (15–19), and major depression–severe (>20). Functioning and judgment are assessed through Independent Living Scales (ILS), which was developed from the literature on guardianship and interviews and surveys with key stakeholders. ILS is used to determine the respondent’s knowledge of information, ability to perform self-care tasks, and care for property.^17,24 Scores on the ILS indicate low (55–85), moderate (86–100), and high (101–121) likelihood of living independently. We previously demonstrated the relationship of this type of assessment with capacity for independent living.¹⁷ Activities of daily living (ADLs)²⁵ were scored out of six and instrumental activities of daily living (IADLs)²⁶ scores were out of eight, with lower scores indicating greater dependence.

Hypotheses

The research team established five hypotheses surrounding MED-SAIL’s reliability, criterion-based validity, concurrent validity, and accuracy of classification. Our hypotheses were:

• H₁: MED-SAIL will demonstrate internal consistency through a high Cronbach’s alpha coefficient (reliability).

• H₂: MED-SAIL will not have a relationship with physical function (ADLs) and depression (PHQ-9) as demonstrated by the absence of significant correlations between the measures (concurrent validity).

• H₃: Participants with greater skills for living independently (ILS and IADLs) and greater cognitive function (SLUMS) will have higher capacity (MED-SAIL) as evidenced by a strong positive correlation between the measures (concurrent validity).

• H₄: MED-SAIL will show a strong relationship with the gold standard criterion for capacity as demonstrated by a comparison of mean ranks as determined by the Mann-Whitney U test (criterion-based validity).

• H₅: MED-SAIL will illustrate accuracy of classification, as evidenced by the following diagnostic statistics: sensitivity, specificity, positive predictive validity (PPV), negative predictive ability (NPV) and total area under the curve (AUC).

RESULTS

Sample Characteristics

Participant characteristics are presented in Table 1 and scores on MED-SAIL and the standardized assessments included in the criterion standard are presented in Table 2. On average, participants have moderate to advanced cognitive impairment, clinically important functional declines, but mild to no depressive symptoms. The comprehensive capacity assessment clinic determined that 25% (N = 12) of the participants had no capacity, 71% (N = 35) had partial capacity, and 4% (N = 2) had full capacity. We substituted the mean for missing continuously scaled standardized assessments with less than 5% of the data points substituted. Because few participants were determined to have full capacity, the partial and full capacity groups were collapsed into a single group (“partial/full capacity”) for analysis.

TABLE 1.

Participant Characteristics (N = 49)

	N (%)
Female	28 (57.1)
Age, M (SD)	76 (10.9)
Marital status
Widowed	19 (38.8)
Separated/Divorced	18 (36.7)
Single	8 (16.3)
Married	4 (8.2)
Race
Black, non-Hispanic	42 (85.7)
White, non-Hispanic	5 (10.2)
Asian or Pacific Islander	2 (4.1)
Education
Primary school or less	11 (22.4)
Some high school	16 (32.7)
High school diploma/GED	10 (20.4)
Some college/trade school	8 (16.3)
College degree or higher	4 (8.1)
Living arrangement
Own home/apartment	44 (89.8)
With relative/friend	4 (8.2)
Personal care home	1 (2.0)

TABLE 2.

Participant Scores on MED-SAIL and Items Included in Criterion Standard (N = 49)

Item	M (SD)
MED-SAIL	5.4 (2.3)
PHQ-9	4.5 (4.6)
SLUMS	14.8 (5.4)
ILS	70.5 (16.1)
ADLs	3.4 (1.6)
IADLs	1.6 (2.2)

Notes: SLUMS: St. Louis University Mental Status Examination; ILS: Independent Living Scales; PHQ-9: Patient Health Questionnaire (depression); IADLs: instrumental activities of daily living; ADLs: activities of daily living.

Preliminary Validation Study

We calculated Cronbach’s alpha coefficients to determine internal consistency of the five items included in MED-SAIL. The results supported our hypothesis that MED-SAIL would demonstrate high internal consistency. For the first scenario completed by participants, α = 0.77 and for the second scenario, α = 0.78. For the mean score across the two scenarios α = 0.85.

Hypotheses 2 and 3 examined concurrent validity. Our expectation for Hypothesis 2 (discriminant validity) was that MED-SAIL would not have a significant relationship with physical function (ADLs) and depression (PHQ-9) was supported (Table 3). Hypothesis 3 assessed convergent validity and was partially supported. We expected that participants with greater skills for living independently (ILS and IADLs) and those with greater cognitive function (SLUMS) would have higher capacity (MED-SAIL). Pearson’s correlations indicated significant positive correlations for MED-SAIL and ILS (r = 0.573, p <0.001) and IADLs (r = 0.440, p≤0.01). The correlation between MED-SAIL and SLUMS was not significant at the p less than or equal to 0.05 level.

TABLE 3.

Pearson Correlations of MED-SAIL with Items in the Comprehensive Capacity Assessment (N = 49)

	MED-SAIL	95% CI	SLUMS	95% CI	ILS	95% CI	PHQ-9	95% CI	IADLs	95% CI
SLUMS	.178	−0.11 to 0.47
ILS	.573***	0.33 to 0.81	.622***	0.39 to 0.85
PHQ-9	.151	−0.14 to 0.44	.069	0.29 to 0.36	−.015	−0.31 to 0.28
IADLs	.440**	0.18 to 0.70	.338*	0.06 to 0.61	.320*	0.04 to 0.60	.007	−0.29 to 0.30
ADLs	.184	−0.10 to 0.47	.088	−0.20 to 0.38	.224	−0.06 to 0.51	.045	−0.25 to 0.34	.510***	0.26 to 0.76

Notes:

^*p ≤0.05;

^**p ≤0.01;

^***p ≤0.001.

SLUMS: St. Louis University Mental Status Examination; ILS: Independent Living Scales; PHQ-9: Patient Health Questionnaire (depression); IADLs: instrumental activities of daily living; ADLs: activities of daily living; CI: confidence interval.

Hypothesis 4 assessed criterion-based validity. We expected that MED-SAIL would demonstrate a strong relationship with the criterion standard assessment battery through a comparison of mean ranks. This hypothesis was supported. A Mann-Whitney test revealed significant differences between the no capacity group (M = 3.25, SD = 1.60) and partial/full capacity group (M = 6.11, SD = 1.99) classification using MED-SAIL (U(48) =60.5, Z =−0.38, p <0.0001).

Hypothesis 5 evaluated the accuracy of MED-SAIL as a screening tool by examining sensitivity, specificity, and the AUC. Our expectation that MED-SAIL would illustrate good accuracy of classification using these diagnostic metrics was supported. The Receiver Operating Characteristic analysis revealed an AUC of 0.864 (Fig. 2; 95% confidence interval: 0.84–0.99), which indicates good accuracy in distinguishing between no capacity and partial/full capacity. We also examined metrics associated with potential cut points for MED-SAIL scoring (Table 4), including sensitivity, specificity, NPV, and PPV across the range of possible MED-SAIL scores.

FIGURE 2.

Receiver Operator Characteristic Curve for MED-SAIL.

TABLE 4.

Sensitivity, Specificity, Positive Predictive Value (PPV) & Negative Predictive Value (NPV) by MED-SAIL Score (N = 49)

	MED-SAIL Cutoff Score (≤)
	0.5	1.0	1.5	2.0	2.5	3.0	3.5	4.0	4.5	5.0	5.5	6.0	6.5	7.0	7.5	8.0	8.5	9.0
Sensitivity	0	0	0.25	0.25	0.42	0.50	0.75	0.83	0.83	0.92	0.92	0.92	0.92	1.0	1.0	1.0	1.0	1.0
Specificity	1.0	1.0	1.0	1.0	0.92	0.89	0.87	0.78	0.78	0.70	0.54	0.43	0.41	0.32	0.27	0.19	0.11	0.25
PPV	-	-	1.0	1.0	0.63	0.60	0.64	0.56	0.56	0.50	0.39	0.34	0.33	0.32	0.31	0.29	0.27	-
NPV	0.76	0.76	0.80	0.80	0.83	0.85	0.91	0.94	0.94	0.96	0.95	0.94	0.94	1.0	1.0	1.0	1.0	1.0

DISCUSSION

This study demonstrated the development and preliminary validation of MED-SAIL in a pilot sample of community-dwelling older adults. MED-SAIL demonstrated high internal consistency of the five items included to assess each domain of capacity. There was evidence of concurrent validity for MED-SAIL in that the tool demonstrated expected relationships with the battery of tests included in the criterion standard. We found significant differences between the no capacity group and partial/full capacity group on MED-SAIL scores. The relationship between SLUMS and MED-SAIL was not found to be statistically significant. Given the sociodemographic characteristics and overall low SLUMS scores for the current sample, it is possible that this finding is the result of a floor effect in SLUMS, warranting further investigation with a more diverse sample with broader range of cognitive impairment. The high prevalence of cognitive impairment may explain the lack of correlation between depression and capacity impairment.

MED-SAIL also demonstrated accuracy of classification through acceptable sensitivity, specificity, and AUC. With the current data, we can establish a score at which health and service providers in the community can distinguish between no capacity and partial capacity. Given the importance of limiting the number of false negatives (i.e., failing to identify individuals with no capacity), we chose a mean score of 5.0 across two scenarios to maximize sensitivity. Using Bayesian analysis to examine effect of prevalence on PPV, we determined that with the prevalence of no capacity at 25% for the current sample, an older adult with a MED-SAIL score less than 5 has a 79% probability of having no capacity.²⁷

We acknowledge that the current study has important limitations. First, these data were collected as part of a cohort study using a pragmatic design rather than a controlled trial. Though our sample was largely African American and had lower levels of educational attainment, the authors contend that these vulnerable community-dwelling older adults represent the population MED-SAIL was intended to reach and a group that is under-represented in previous research.^28–30 Future work will focus on larger and more diverse study populations to increase generalizability and to reach other equally vulnerable populations, including Hispanics. Second, as briefly mentioned in the Methods section, the standard of appreciation was not included in the current version of MED-SAIL. Feedback collected during early feasibility testing indicated that users did not feel the original appreciation item was critical to the evaluation and was subsequently removed from MED-SAIL. Given the importance of the domain of appreciation^8,19 to an older adult’s capacity to live independently in the community, we have since developed a measure of appreciation to be included in future MED-SAIL studies. Third, variability between administrators is inherent to assessments that do not require administrators to strictly adhere to scripts or allow for flexibility in interpreting responses. Though our structured scoring scheme is intended to counteract such variability, future research will focus on inter-rater reliability in addition to further testing MED-SAIL’s validity in the community and other residential settings. Fourth, because this study included only two participants with full capacity, we were unable to examine whether MED-SAIL could discriminate between all three capacity levels. To create appropriate cut scores to distinguish between no, partial, and full capacity, it is imperative to test effectiveness with a more diverse sample of older adults. Future studies should also compare MED-SAIL with specific measures of executive cognitive function as an additional measure of concurrent validity associated with capacity for independent living.²⁸

The reader can likely conjure a personal or anecdotal story of an older adult living alone in the community struggling to maintain their independence. Too often, these stories involve an individual relying on informal assistance from others or making choices that put them at risk for harm (abuse, exploitation, and medical morbidity) because the suspicion of capacity impairment cannot be quickly and easily verified. We demonstrated that MED-SAIL is an effective screening tool to differentiate between no capacity and partial/full capacity in community-dwelling older adults for the purposes of referral for comprehensive further evaluation and service planning. With a small amount of training, MED-SAIL allows health and social service providers in the community to quickly identify older adults who may be at risk for losing their independence, potentially requiring transitions into more costly and restrictive long-term care settings. Responses to the MED-SAIL scenarios can be shared with family members or caregivers to contextualize capacity impairments in a manner that is meaningful to everyday living, which is not the case when neuropsychiatric measures alone are used in capacity determinations. By identifying older adults with impaired capacity for independent living, supports may be implemented early to improve the likelihood of safely aging in place.