Comparison between a standardized questionnaire and expert clinicians for capacity assessment in stroke clinical trials

CHALLENGE

Research into treatments for acute strokes has dramatically increased in the last decade. Accordingly, the need for testing through randomized clinical trials (RCT) has also increased. Due to the unique combination of factors that are common in acute stroke related research, including narrow treatment windows, ethical concerns regarding research with acute stroke populations and capacity for informed consent, stroke clinical trial enrollment levels have remained stagnant. Given the devastating consequences of acute stroke, researchers are intensifying their efforts to recruit and enroll larger sample sizes in clinical trials.¹

Capacity to consent to medical treatment or medical research is closely related to cognitive functioning, which is frequently impaired in stroke.² Medical decision making capacity (MDC) requires the ability to understand and appreciate diagnostic, treatment, and/or research information and risks and ability to express a choice that is based on adequate reasoning. The treating physician is responsible for the assessment of a patient’s decision-making capacity and clinically estimates their patient’s ability to provide informed consent.³ Many physicians request additional consultative assistance to assess cognitive capacity for consent from psychiatry or neuropsychology, which are considered to be the clinical gold standards⁴ or they perform standardized capacity questionnaires to aid the assessment of capacity.^{5, 6}

One such tool, the Aid to Capacity Evaluation (ACE) had 80–89% agreement with expert clinicians in one study and inter-rater reliability reported as κ=0.79 for medical hospitalized patients.^7,8 The ACE has been validated and found to be one of the best available instruments to assist clinicians in making judgments on MDC.⁴ It is designed to be used by trained non-clinicians and takes about 10 minutes to perform.⁴ For patients who lack capacity medical decisions are deferred to a surrogate decision maker. Surrogate decision makers in acute stroke are able to accurately predict patient treatment preferences, but are significantly less accurate in predicting patient preferences for research participation.⁹

The development of a standardized validated tool to determine medical decision making capacity has been instrumental in identifying patients who lack capacity in hospitalized non-stroke inpatients (see online supplement). While the ACE has been adapted for many medical conditions, it has yet to be extensively evaluated in the acute stroke population.

Presently, there is no standardized tool to evaluate decision-making capacity in stroke patients.¹⁰ The introduction of a standardized tool to rapidly determine MDC of acute stroke patients in the emergency setting would be a significant advancement and would likely increase RCT enrollment – especially when the clinician or study team is uncertain of the patient’s cognitive status in the absence of a legal surrogate decision maker.

INNOVATION

We hypothesized that a modified, stroke specific, version of the ACE would show similar agreement with clinician judgment and would therefore be appropriate for rapid bedside screening for MDC.

METHODS

Design

This was a prospective, pilot study of three different capacity evaluations (ACE performed by a trained rater, psychiatrist (PS) assessment and neuropsychologist (NP) assessment). All assessments were performed independently within the same day or within ± 24-hours. All patients were hospitalized patients in stroke, rehabilitation, or neurological intensive care units of a single tertiary-care university medical center. The study was approved by the Institutional Committee for Protection of Human Subjects.

Study Population

Patients were English speaking, ≥ 18 years old, and diagnosed with either an ischemic or hemorrhagic stroke (NIHSS score ≥ 1) within 10 days of symptom onset. Patients unable to hear despite assisted devices, declared legally incompetent, encephalopathic, severely lethargic or obtunded, diagnosed with dementia or severe cognitive decline, or had a current psychiatric diagnosis (schizophrenia, major depression) that would interfere with study assessment were excluded.

Patients underwent three independent capacity assessments: ACE, PS, and NP. Attempts were made to perform all capacity evaluations while the patient was mentally alert to reduce the variation within the patient’s responses between examiners.

Measurements

To minimize discrepancy among individual examiners every effort was undertaken to ensure that all examiners remained consistent with their technique throughout the study duration. The clinicians and the trained rater were blinded to the results of the other’s assessments as well as methods utilized.

A research assistant (KLF) was trained on how to conduct the ACE, and performed three practice evaluations under the supervision of the principal investigator (ADB) and experienced research coordinators. The ACE questions were modified for stroke (see Figure 1a and online supplement for further detail) by employing a hypothetical scenario (Figure 1b) followed by seven standard questions that tested the ability to: 1) understand the medical problem; 2) understand the proposed treatment; 3) understand treatment alternatives; 4) understand the option of refusing treatment; 5) appreciate the reasonably foreseeable consequences; 6) appreciate foreseeable consequences of refusing the proposed treatment; and 7) make a decision that is not based on delirium or psychiatric disturbances (i.e., severe depression).

Figure 1.

a) Abbreviated version of the modified ACE questionnaire for assessing capacity in stroke patients. b) Hypothetical scenario which is read to the patient prior to asking questions 2–7.

A board certified PS with extensive experience in performing hospital based capacity evaluations, assessed the patient’s capability of consenting to a hypothetical scenario. This scenario concerned patient preference for new versus standard blood thinner treatments. Specific aspects of consent evaluated by the PS included: 1) the ability to communicate and sustain a choice; 2) understand and describe risks, benefits, and alternatives of the proposed medication; and 3) ability to communicate his/her rationale for the choice made.

The neuropsychological exam consisted of a clinical interview and brief formal subtests chosen to evaluate comprehension and judgment. Topics ranged from simple biographical and medical information to hypothetical medical decision-making. Receptive and expressive language, attention, memory, anosognosia and problem solving ability were assessed. This was followed by administration of the Neuropsychological Assessment Battery Judgment subtest and Complex Ideational Material and Syntactic Processing subtests from the Boston Diagnostic Aphasia Exam.^{11, 12} These were sequentially chosen based on the patient’s ability to complete the preceding tests (i.e., presence of visual or hearing impairments, etc.).

RESULTS OF PILOT TESTING

31 patients met study inclusion criteria and 30 agreed to participate. Sixty percent of patients were male with mild-to-moderate ischemic stroke (median NIHSS=6) and a mean age of 67.8 (Table 1). ACE evaluation required approximately 5–7 minutes. Thirty seven percent exhibited aphasia and/or neglect, while the remaining participants lacked these deficits. All 30 completed the ACE, but 2 patients were unable to be assessed by both clinicians. The ACE, NP and PS determined many patients lacked MDC: 70% (21/30), 52% (15/29) and 28% (8/29), respectively (Table 2).

Table 1.

Demographics and Stroke Characteristics.

Variable	N	Summary Statistics
Age, (mean ± SD)	30	67.8 ± 14.9
Male, n (%)	30	18 (60.0)
Race/Ethnicity, n (%)	30
African-American		12 (40.0)
Caucasian		11 (36.7)
Asian		1 (3.3)
Hispanic		6 (20.0)
Education, n (%)	29
No schooling		2 (6.9)
Some high school		8 (27.6)
High school diploma / GED		11 (37.9)
Some college		4 (13.8)
College Degree		3 (10.3)
Advanced Degree		1 (3.5)
Stroke Etiology, n(%)	30
Ischemic (N=27)	27
Cardio-embolic		7 (25.9)
Large Artery Athero		10 (37.0)
Small Vessel		1 (3.7)
Cryptogenic		9 (33.3)
Hemorrhagic (N=3)	3
Aneurysm		1 (33.3)
Unknown		2 (66.7)
Baseline NIHSS, median (Q1, Q3)	28	6 (4, 10)
Minor (1–4), n (%)		9 (32.1)
Moderate (5–15), n (%)		16 (57.1)
Moderate to Severe (16–20), n (%)		1 (3.6)
Severe (21–42), n (%)		2 (7.1)
Pre-capacity Assessment NIHSS, median (Q1, Q3)	30	4 (3, 11)
Aphasia, n (%)	30	8 (26.7)
Extinction/neglect, n (%)	30	5 (16.7)
Aphasia and/or Extinction, n (%)	30	11 (36.7)
Minor (1–4), n (%)		19 (63.3)
Moderate (5–15), n (%)		10 (33.3)
Moderate to Severe (16–20), n (%)		1 (3.3)
Severe (21–42), n (%)		0
Timing between assessments (hours), median (Q1, Q3)
Stroke onset – ACE assessment	30	78.3 (52.2, 128.8)
ACE – Psychiatry assessment (interval time)	29	1.0 (0.6, 1.8)
ACE – Neuropsychology (interval time)	29	1.5 (0.7, 2.3)
Psychiatry - Neuropsychology (interval time)	28	0.8 (0.5, 2.0)

Abbreviations: SD = Standard Deviation; TIA = Transient Ischemic Attack; Athero = Atherosclerosis; Q1 = 1^st quartile; Q3 = 3^rd quartile

Table 2.

Frequency (percentage) of capacity decision by ACE, psychiatrist and neuropsychologist.

Medical Decision-Making Capacity	ACE	Psychiatrist	Neuropsychologist
Capable, n (%)	9 (30.0)	21 (72.4)	13 (44.8)
Definitely capable	3 (10.0)	11 (37.9)	8 (27.6)
Probably capable	6 (20.0)	10 (34.5)	5 (17.2)
Incapable n (%)	21 (70.0)	8 (27.6)	16 (55.2)
Definitely incapable	2 (6.7)	5 (17.2)	4 (13.8)
Probably incapable	19 (63.3)	3 (10.3)	12 (41.4)
Total	30	29	29

We evaluated the sensitivity and specificity as well as positive predictive value (PPV) and negative predictive value (NPV) of the ACE (Table 3). The ACE demonstrated high sensitivity: 93.8% compared with NP, and 100% compared with PS. It also had a high NPV to detect intact capacity vs. clinicians; misclassifying only 1 patient capable when clinicians recorded incapacity (false negative rate of 6.2%). However, it demonstrated a low specificity: 53.8% compared with NP, and 42.9% compared with PS.

Table 3.

Two by two tables of ACE versus PS (n=29) and ACE versus NP (n=29).

		Psychiatrist
		Disease Present Lack of Capacity	Disease Absent Capacity Intact
ACE	Positive Test (Failed ACE)	8	12	PPV = 8/20 40% (95%CI: 19.1–64.0)
	Negative Test (Passed ACE)	0	9	NPV = 9/9 100% (95%CI: 66.4–100)
		Sensitivity = 8/8 100% (95%CI: 63.1–100)	Specificity = 9/21 42.9% (95%CI:21.8–66.0)

		Neuropsychologist
		Disease Present Lack of Capacity	Disease Absent Capacity Intact
ACE	Positive Test (Failed ACE)	15	6	PPV = 15/21 71.4% (95%CI:47.8–88.7)
	Negative Test (Passed ACE)	1	7	NPV = 7/8 87.5% (95%CI:47.4–99.7)
		Sensitivity = 15/16 93.8% (95%CI: 69.8–99.8)	Specificity = 7/13 53.8% (95%CI:25.1–80.8)

CONCLUSIONS

The benefits of using the ACE are that unlike other instruments, the ACE uses treatment information relevant to the patient’s circumstance and is therefore specific. It specifically assesses decision-making capacity using open-ended questions, and at the conclusion of the ACE evaluation capacity is reliably determined.^7,13 The ACE is the only tool that has been evaluated against a gold standard and has also performed strongly on the Journal of the American Medical Association’s (JAMA) Rational Clinical Examination. The ACE can be used in emergency situations and can be performed in less than 10 minutes.⁴

However, the ACE may be overly reliant on intact memory and comprehension as an aspect of the testing. Therefore, the ACE may fail to capture those patients who can make complex decisions but have initial comprehension deficits as a result of an acute stroke. The neuropsychologists considered memory, attention, and expressive and receptive language in their global assessment for MDC by listening to patient responses and using clinical expertise to interpret them. Comparatively, the psychiatrist analyzed capacity on a narrow spectrum by focusing the evaluation on a pre-scripted but simple specific decision at a specific moment in time.

Entering into a randomized clinical trial is a complex decision that requires patients to understand many things – the study purpose, risks and benefits of participation, the concept of blinding, randomization, voluntarism, and withdrawal. Due to its high sensitivity, the ACE appears to be a useful initial screening tool to detect if a patient has complex decision-making capabilities.

There are several limitations to this study. The small sample size limits the conclusions that can be drawn and future studies should include larger sample sizes. Another limitation was the simple scenario used for the psychiatric assessment. Although every attempt was made to create a likely scenario that a patient may face post-stroke, the decision-making capacity for the scenario was not complex and may not be applicable to true research studies.

In conclusion, lack of MDC is very common in mild-to-moderate stroke patients. The ACE can accurately identify those who can participate in stroke trials. However, failing the ACE does not, at present, adequately determine patients who cannot participate. Failing the ACE should trigger supplemental testing to adequately distinguish those who, due to initial comprehension issues or stroke specific deficits appear to lack MDC, but in fact do possess it. Additional refinement and testing of alternative, non-emergent hypothetical scenarios of the ACE for a stroke-specific population might increase its utility for determining participation capacity in RCTs.