What Clinicians Need to Know About Measurement

The August issue of JAMDA features numerous articles that advance measurement in post-acute and long-term care. Measurement is of critical importance in clinical care: it matters for screening, assessing, predicting, treating, dosing, and monitoring. However, few clinicians understand critical issues underlying measurement itself, despite the fact that such understanding is important to digest research findings and conclusions and their application to practice. This editorial presents 5 of those issues: measurement in relation to causality, relevance, adequacy, psychometrics, and inference.

Just Because 2 Things Are Associated Statistically Does Not Mean That One Causes the Other

The logic behind “association does not equal causality” is largely predicated on the fact that a third variable may be responsible for any given association.¹ For example, older adults with less education report fewer benefits from rehabilitation than do those with a college degree²; surely, the physiologic benefits of rehabilitation are not based on an individual’s education. However, in some cases association may suggest causality, and although a third variable may be the operative cause, it may be embedded within the phenomenon. For example, COVID-19 incidence and mortality rates were less in Green House and small nursing homes than other nursing homes, which may in part be due to or caused by their smaller size that limits the number of infectious people who enter the building.³

Although the logic of these examples is likely clear, numerous published articles assert causality based simply on association, which may mislead a casual reader. [“Casual” is not meant to disparage readers; instead, it recognizes that many readers (1) focus on the abstract, which often omits important details, and (2) do not actually have access to entire articles so as to read those important details.]

• Bottom Line: Readers must be cautious when titles assert “outcomes,” “influence,” or “impact” absent a randomized trial^4,5 or based on a small sample size.^6,7

Measurement Does Not Ensure Relevance

The use of data from administrative and clinical records to assess care and outcomes has become common and will continue to grow given the utility of large samples.⁸ Consider, for example, how much we learned about COVID-19 in long-term care thanks to administrative data.^9–11 The National Institute on Aging (NIA) is especially invested in the promise of data existing in health care systems, and funded the IMbedded Pragmatic Alzheimer’s disease (AD) and AD-Related Dementias (AD/ADRD) Clinical Trials (IMPACT) Collaboratory to build capacity and promote pragmatic clinical trials in these systems.¹²

IMPACT recognizes, however, that not all data contained in these databases are relevant to all stakeholders, and conversely, that some data relevant to stakeholders are not contained in these databases. In fact, outcome measures reflecting the lived experience of persons with AD/ADRD and their caregivers are rarely available in large clinical and administrative data sources.¹³ One such limitation might be attenuated based on the findings of an article published in the August issue of JAMDA, which recommended that Nursing Home Compare be improved by adding consumer assessments of quality.¹⁴

• Bottom line: Readers should consider whether the care processes and outcomes being examined are important to the end user; what is easily measured should not dictate what is relevant.

Existing Measures Are Not Always Adequate

In addition to improving existing measures, there is need to develop new measures. As a case in point, shortly after the Alzheimer’s Association set forth the 2018 Dementia Care Practice Recommendations that address care in 9 domains ranging from detection and diagnosis, to care for activities of daily living (ADL), to transitions and coordination of services,¹⁵ it became apparent that existing measures do not sufficiently assess care and outcomes in each of these domains. Consider, for example, that a measure of well-being following a new diagnosis of Alzheimer’s disease may be very different from a measure of well-being related to ADL care or following a transition in care.¹⁶ The NIA funded LINC-AD (Leveraging an Interdisciplinary Consortium to Improve Care and Outcomes for Persons Living with Alzheimer’s and Dementia) to expressly promote new measurement development in these areas.

• Bottom line: Readers should look carefully at whether findings and conclusions regarding a given construct are based on sufficient measurement (ie, use a valid measurement tool).

Not All Research Uses Psychometrically Sound Measures

Simply stated, a measure that is psychometrically sound measures what it intends to measure. Among other criteria, psychometrically sound measures are valid; they (1) appear suitable to their intent (face validity), (2) include all aspects of the concept being measured (content validity), (3) test the concept that is intended to be measured (construct validity), (4) yield results similar to those measured by instruments of the same or related concepts (criterion validity), (5) yield results that differentiate groups that are expected to differ (discriminate validity), and (6) predict a future expected state (predictive validity).

Psychometrically sound measures also are reliable; they (1) generate the same response when used over time (test-retest reliability) and (2) across different raters (inter-rater reliability), and (3) the items within a given measure are scored similarly (internal consistency). In addition, measures must be appropriate for a given population, because if they exhibit floor or ceiling effects—meaning most people score near the bottom or the top—they aren’t useful to indicate decline (if people already score at the floor) or improvement (if people already score at the top). Also, some measures may not be sensitive enough to detect clinically meaningful change. If a measure is not reliable or valid, exhibits floor or ceiling effects, or is not sensitive to change, results are suspect.

Nine of the articles published in the August issue of JAMDA focus on measurement evaluation, related to measures of arousal,¹⁷ frailty,¹⁸ function,^19,20 and prediction.^21–25 However, although many JAMDA authors are expert in measurement, few clinicians are.

• Bottom line: Readers should review the methods section of an article to learn whether the measures have been used extensively in other research (suggesting they are sound), and whether the authors report psychometric properties.

Measurement Itself May Be Misleading

Clinical treatments may have side effects; so too can the process of measurement. The “Hawthorne effect” is a phenomenon with which many are familiar; it refers to behavior change due to the awareness of being observed, typically in the direction of improvement. However, few appreciate that measurement can have a contrary effect—for example, when a falls reduction quality improvement project results in more documented falls after project initiation than at baseline merely because the staff become more diligent about recording falls. In such a case, treatment benefits may be minimized.

A second misleading characteristic is that some measures include items reflecting both care and outcomes, in which case it becomes challenging to disentangle whether the ultimate intent of care was achieved [eg, the Quality of Dying in Long-Term Care (QOD-LTC) scale includes items such as “resident received compassionate physical touch daily” (care) and “resident appeared to be at peace” (outcome)].^26,27

• Bottom line: Readers should reflect on whether human nature may affect measurement, and what items compose a scale.

The Bottom Bottom Line

• As an end user of research, clinicians should be educated consumers regarding measurement, and not draw conclusions without considering how measurement may have impacted the article’s findings and relevance.