“Soft” assessment—an oxymoron?

O wad some Power the giftie gie us, To see oursels as ithers see us!

R Burns, 1798

The desire to assess the “softer” parts of practice is strong. In addressing this ambition, many of the tensions and controversies surrounding the assessment process are thrown into sharp focus. Exactly how one defines soft practice for the jobs carried out by, say, general practitioners or orthopaedic surgeons is open to wild speculation. One key message of the paper by Evans et al is the importance of those being assessed having a stake in their assessment; this may be the “missing link” that could make the assessment process less threatening.

Assessment has to be rigorous, not least because it should be fair. The subtleties of language highlight the difficulties—peer assessment and peer review have different connotations. In an era of individual accountability and clinical governance, many might question whether soft practice can ever be assessed with rigour.

Evans and colleagues in their paper concentrate on what could currently be considered the psychometric gold standard attributes of assessment.¹ Van der Vleuten would argue that acceptability and cost effectiveness are also crucial in laying down a system where busy people are assessing busy people.² No matter how we cut the cake, assessment is expensive in terms of both time and money.

It is interesting that the three papers identified by Evans and colleagues emanate from North America, where a culture of a high level of psychometric skills is possibly encouraged by a litigation conscious community. This may highlight the lack of such skills in other countries and could serve as a warning that, given the high stakes involved in such assessments, the result is either instruments of poor quality, as suggested by the paper, or no instruments at all.

In response to this it may be that in considering assessment of a softer side of practice we need to consider some other measures of validity.³ One example is consequential validity—that is, taking into consideration the consequences on those being assessed as part of the process.⁴ In a world dominated by numbers and the desire to measure them, a wider debate about the strengths and limitations of formal psychometrics when considering the less tangible aspects of practice is urgently needed if a more valid assessment of doctors is to be achieved.

In considering how one might address the assessment of peers, particularly with reference to the complexities of the jobs done, one example is to consider assessments over time—that is, trends such as the longitudinal evaluation of performance (LEP) used for the assessment of dental trainees in Scotland.⁵ It covers a range of competencies on a nine point scale and concentrates on feedback, a crucial element in consequential validity.

Three hundred and sixty degree or multisource feedback provides judgments on strengths and areas for potential improvement as perceived by work colleagues both above and below in the hierarchy. Many see this very practical method as opening new opportunities in exploring humanistic qualities that are not easily assessed by more traditional methods.⁶ Predictably, however, there are limitations, not least with validity, but these are not insurmountable.

Perhaps more qualitative work on defining soft practice is required before designing any more instruments. Themes on areas of soft practice could be collated drawn from groups of doctors with additional input from patients to form the basis for more formal psychometric development and testing. Without this, soft assessment will be seen as a soft option and not given the place it deserves.