Personal experience when it is used to recount unsatisfactory or negative stories of Health IT systems are typically re-labelled as “anecdotes” by defenders of the technology. The aim of this metonymic re-labelling is to negate the reports; to tap into the “anecdotage” value of the meaning of anecdote that portrays these experiences as invalid on the grounds of being the mere reminiscing of people in their dotage. This manoeuvre reaches its zenith with the philosophically positivist aphorism “the plural of anecdotes is not data”, an implied derision that the speaker of personal experience is unscientific and methodologically ignorant, a label that is bound to condemn even the most competent and rigorous scholar. Hence nothing is valid if it isn’t a measurement and collected in some formal fashion. So why are the recountings of experienced users about problems, disadvantages and undesirable characteristics of health information systems treated contemptuously or simply as the reminiscences of the aged. After exploring the reasons why personal experiences are often dismissed an outline of some of the effects of the failure to take seriously these histories follows.
The simplest but not the only explanation is that this practice is used to protect the existing technology by the stakeholder that provides the IT system. But equally, it could be a representative of an institution or corporation that has invested so much money, public or private, in the system and are accountable for it that they find it hard to accept the possibility of negative criticism. There are also times when the expectations of a system are so high people become promoters and believers in it, identifying with it and as such, find it hard psychologically to accept that the HIT has problems or that there might be a better system. They just don’t want to hear it. Furthermore, the scientific method itself does not lend credence to personal experience, as it requires a scientific study usually over some substantial and often unacceptable, period of time to determine “truth”. Thus there are a number of different positions from which the naysayers of personal experience come. Such apologists have a vested interest in discrediting personal experience, even when it is direct and thoughtful evidence of deficiencies in the technologies. These experiences are often the most authentic, congruent and irrefutable form of evidence that we need to scrutinise for their insights.
An heuristic rule of project management is that projects consist of 3 components: cost, quality, and time, and so if there needs to be a compromise it has to be on quality. That is, the quality of the product delivered to the users is diminished. Personal experiences are often early warning signs of such a compromise and one response might be a formal investigation or study. However such an approach may well call into question the choices and decisions made by people in a variety of senior positions.
Rejecting “anecdotes” also defends software manufacturers from fault rectification, and short circuits any need to deliberate on them which if undertaken would mean additional costs. Critics of the value of personal experiences thus end up, sometimes albeit unintentionally, on the side of the faulty and deficient manufacturer and/or service provider. It is worthwhile to note that the National Research Council report on dependable software, in 2007, stated that for software to be safe and dependable the burden of proof should be shifted from the user to prove that the system is not functioning adequately to the vendor to prove that it is.
A closer look at the use of personal experience in HIT in other ways reveals the validity of these assertions. In other industries there are situations in which personal experience is usually accepted as meaningful. A user who experiences an extremely long waiting period when using a program is entitled to phone IT services and ask if the service is faulty or switched off. A user whose system produces unexpected data or impossible data is able to log a “bug” about the system with the supplier or service provider. A user who experiences the steps in a program as requiring more than they consider is necessary is entitled to request the system be changed. A user who has experienced a previous system and is able to learn more quickly the new system is considered to be an “experienced user” and thereby has pertinent knowledge and competency.
Nevertheless it is usual to ignore such rumblings which can be a little like an upset stomach that will go away by either being ignored or given a mild palliative. In fact the situation is often far from benign discomfort. In one Emergency Department that we collaborate with the consequences of an unsatisfactory system led to an interesting process of degradation of morale and a lessening of the utilization of the system matching much of the sequence of processes in the Kubler-Ross grief model of the stages of Disbelief, Anger, Bargaining, Depression, and Acceptance.
The response of the staff moved from openness and a willingness to use the new software and accept the need for more training to disbelief and frustration at having to follow time-consuming redundant steps to the point of anger at both technical and social aspects of the system. This frustration triggered staff to develop workarounds and ad hoc strategies, and to deliberately testing the system to identify and magnify its deficiencies. The bargaining stage commenced with attempts to get redundancies removed and non-intuitive processing simplified by requests to the advisory group. But those attempts failed. Consequently, staff felt disheartened and believed that they had no choice but to withdraw from using the system. Some nursing staff even developed strategies to avoid using the system altogether. Staff resented the negative impact the system had on their work and became disillusioned with it and often contemptuous of it. One staff member recounted that after some time she started to see ridiculous entries in the system, for example someone had added absurdities to the diagnosis list. When questioned about why someone would do that the answer was “because you can.” Later, staff recounted that finding faults in the system became a sport.
As this example demonstrates, ultimately contempt leads to a decaying and moribund information system as staff developed more sophisticated and detailed methods for working around the disadvantages of the system. For example staff in this hospital recorded patient case notes on paper rather than electronically as they had done in their previous information system. Hence the system’s validity is discredited, and only a small portion of its functionality is used because either it is a part of fundamental workflow or the authoritative strictures are powerful enough. But in the end the system will die the death of a thousand abuses, misuses and avoidances, turning a significant amount of money and effort into waste, while multiplying the resentment of staff towards their institution.
This example of the failure of a software system as with any troublesome system reflects the expression of the problems in the personal experiences of the staff who use the system. The recounting of their stories exposes the value of their experiences in both rectifying the system if it is not too far gone, or learning from mistakes of history and endeavouring not to repeat them.
A useful analogy for understanding further the importance of not suppressing but rather engaging with personal experiences in assessing HIT is the classical example of the canary in the mine. The role of the canary was to give early warning of danger from explosive gases. In the case of HIT assessment personal experiences provide an equivalent early warning system of problematic technology that is possibly dangerous and a sign that it may be time for an in-depth study of the problems and issues. The equivalent of refusing to allow the miners to take the canary down into the mine is seen in the HIT industry where authorities sign off to contracts with vendors prohibiting public comment by staff on system installations. This is clearly courting disaster and just as it is an abrogation of responsibility by the company to the miner so it is by the health authorities and professionals to the patient. Agreeing to such prohibitions serves no worthwhile purpose for the staff who have to operate the system and the patients whose records are preserved in the system. When this strategy is combined with diminished content due to minimal system usage and less reliable content due to technical faults and operational barriers, the reusability of the records has to be seriously questioned.
Consequently the denial of recounted personal experiences in discussion and analysis of HIT is biased and specious and has the effect of:
-
1.
Denying early warning signs of problems.
-
2.
Denying a voice for the working clinical community who have to operationalise decisions made by others and thus disempowers them.
-
3.
Denying process improvement within an institution – which is most important for Evidence Based Medicine and incremental review of local processes.
-
4.
Discourages staff from engaging in any form of process improvement hence worsening the sense of disenchantment.
Every legitimate personal experience of a HIT deserves to be considered on its merits lest we wish to retreat from process and product improvement. Mechanisms of censorship both implicit due to contrived processes of disinformation and disempowerment or explicit due to contractual specifications will lead to more waste, lost productivity, contempt for the providers, and distress among frontline staff rather than increased productivity and improved patient health and safety as we all desire.
Conflict of interest
The author declares that he has no conflict of interest in making the comments provided herein. This opinion piece is based upon informal conversations with staff in a number of hospitals across New South Wales and formal discussions with 8 directors of Emergency Departments in the Sydney region and material published at his web site http://www.it.usyd.edu.au/~hitru/.