Challenges facing COS development for aphasia

Wallace, Worrall, Rose, and Le Dorze (2014) (WWR&L) present a balanced and comprehensive approach to the development of methods for assessing the outcomes of treatments for aphasia.

The thoroughness of their analysis indicates how challenging it is to come up with a core outcome set (COS) that is responsive to conflicting goals, different therapy approaches, competing stakeholder groups, diverse cultural perspectives, and contrasting understandings of the nature of aphasia. Despite these challenges, international standardisation efforts of this type are becoming increasingly important in medicine, education, technology, and elsewhere. It is inevitable that this process will also be conducted in the area of outcome evaluation for the treatment of aphasia. However, to maximise success of the effort, it would be useful to consider potential pitfalls and how they can be avoided.

The first potential pitfall involves the complexity of the problem. One way to deal with complexity would be to define a set of subgoals that could be measured separately in standardised ways. For example, one could articulate goals such as “to increase the accuracy of word naming from pictures” or “to provide responses to a series of 20 questions about personal history”. There could also be higher-level goals, such as “express an improved evaluation of overall ability to communicate” or “feel more comfortable in group interactions”, as well as practical goals, such as “attaining improvement with the lowest cost and time commitment”. The international standardisation effort the authors propose could then focus its work on the development of methods for evaluating each of these goals separately. Work on standardising assessment of the treatment of rheumatoid arthritis has advanced by delineating a wide set of possible outcomes, each of which can then be defined in greater detail and measured separately. It is possible that authors intend for the first stage of the proposed COS process to involve a similar type of decomposition, but they have not yet stated how this would happen.

A second potential pitfall involves the need to emphasise psychometric validity. There are many available outcome measures, but each will have to demonstrate construct validity, predictive validity, and reliability. However, as the authors note, the imposition of basic psychometric requirements will tend to favour measures with easily quantified behavioural or bodily outcomes, rather than those with activity, adjustment, participation, and attitudinal outcomes. The COS approach is particularly susceptible to this problem. If the COS includes only the most easily measured outcomes, it would incorporate a bias against treatments that work on more fundamental, but less easily measured, goals. This problem can be addressed by ensuring that the COS itself be fully balanced, even if some of its components show weaker reliability than others.

A third potential pitfall involves pressures to develop measures tailored towards a particular type of experimental design. For example, if the COS were to focus only on methods for comparing studies using randomised clinical trials, the vast body of outcome research using other methods would be excluded. It is not clear at this point how wide a swath of outcome studies is being considered in the design of the COS. Moreover, it is not clear whether the COS would be applicable equally well to studies that evaluate the retention of long-term changes over time vs. those that examine outcomes at a single time point.

A fourth danger that this effort could face is that excessive decomposition of outcome goals could become counterproductive. For example, the authors mention an analysis by Noonan et al. (2009) of 8 instruments that examine 1,351 meaningful ICF (International Classification of Functioning, Disability, and Health) concepts. Many of these concepts could involve highly intercorrelated formulations of the same underlying outcome dimensions. It would make little practical or theoretical sense to develop a COS that included multiple measures of the same goal, based on reworded or minimally different formulations of the same outcome.

A fifth danger is that this effort could fail to achieve integration with modern methods for data archiving and sharing. In other areas of medicine, education, and technology, evaluation efforts are being closely linked to the development of shared empirical databases against which outcomes can be evaluated in detail. The AphasiaBank Project (Fromm et al., 2011; MacWhinney, Fromm, Holland, Forbes, & Wright, 2010; MacWhinney, Fromm, Forbes, & Holland, 2011; MacWhinney, Fromm, Holland, & Forbes, 2013) that I have organised at http://talkbank.org/aphasiabank is an example of a shared database of this type. If the development of a COS for aphasia rehabilitation research were conducted in close collaboration with systems such as AphasiaBank, it would stand on a more solid empirical footing and thereby make a greater practical and academic contribution.

SUMMARY

The development of a core set standards for evaluating the outcomes of treatments for aphasia is an important and inevitable step forward. To maximise the success of this effort, the organisers need to formulate methods to (1) reduce complexity by analysing outcome subcomponents, (2) balance psychometric validity with coverage scope, (3) consider applications to alternative research designs, (4) avoid measure overlap, and (5) guarantee effective data-sharing and archiving.