How does the purchasing staff of an accident insurance organization seek information about treatment effectiveness?

Abstract

Objectives: The objective is to study how the staff who purchase health care services for a large national government accident-compensation system seek information on treatment effectiveness, how they assess the quality of that information, whether they question the information sources they choose, and how familiar they are with the key concepts of evidence-based health care (EBHC).

Method: Staff (22 out of 34) of the health purchasing division of the New Zealand Accident Compensation Corporation (NZ ACC) were interviewed using eight preformatted questions to which they could provide open and multiple answers. Responses were subsequently codified into typologies for quantitative analysis.

Results: Most respondents report that they assess the effectiveness of a treatment by accessing published information (nonhuman sources), by consulting others (human sources), or by both means. They assess the quality of information mostly by consulting others, and the second-highest proportion of responses state that they do not know how to evaluate the quality of information. No clear preference emerges with respect to the types of information needed to determine the effectiveness of treatments. The majority of the staff believes they can access information needed to determine treatment effectiveness through the Internet or information databases such as MEDLINE. Although most said they understand the key concepts of EBHC, only five out of twenty-two were able to accurately describe them.

Conclusions: The findings suggest that there is a low level of awareness among the staff of the NZ ACC regarding the use of evidence and understanding of the key concepts of EBHC. Many surveyed staff members lack the skills or training to directly question information about effectiveness of a treatment. They have little idea of the information required to determine the effectiveness of a treatment, and the majority appears to lack the skills to evaluate the health care literature.

INTRODUCTION

The health insurance industry has particular interests and requirements in terms of evidence-based health care. Decisions about groups of claimants are made according to three factors: (1) evidence, (2) values, and (3) financial or human resources. A population that is increasingly educated about health care has added complexity [1]. It has been argued that

at present, many health care decisions are based principally on values and resources—opinion-based decision making; little attention has been given or is paid to evidence derived from research—the scientific factor. This will change: as the pressure on resources increases, decisions will have to be made explicitly and publicly; those who take decisions will need to be able to produce and describe the evidence on which each decision was based. [2]

Evidence-based health care (EBHC) also provides evidence-based clinical practice and case management for individual claimants. Decisions about claimants are made by the providers based on knowledge about the available evidence. This evidence includes epidemiology, reliability and validity of classification, and effectiveness of interventions.

The growth in demand for health care in industrialized countries has invariably outstripped the rate of increase in resources available for providing it. Therefore, health care resources are under increasing pressure. It has been argued repeatedly that decisions allocating these resources should be made explicitly and publicly, and those who make these decisions need to be able to search for evidence and appraise, present, and describe the evidence on which decisions are based. As the pressure on resources increases, the transition from opinion-based to evidence-based decision making will undoubtedly accelerate. However, it must be recognized that the process of moving to evidence-based decision making itself consumes resources.

In recent years, information needs and availability have been increasing at an ever-accelerating pace. At the same time, the need to keep abreast of and the ability to critically appraise this information has become increasingly important. This is particularly the case in the health services and accident insurance sectors, where access to the best evidence can dramatically alter the quality and cost effectiveness of services provided. The information explosion in this field poses a formidable challenge to both clinicians and purchasers, and the aim of evidence-based medicine (EBM) is to ensure that clinical decision making is now systematically based on information derived from that research.

The EBM approach is defined as the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients [3]. EBM uses a simple scale to determine the quality of information by assigning a level to the evidence [4–7]. The highest level of evidence is from systematic reviews, which summarize randomized controlled trials (RCTs). The lowest level of evidence is expert opinion and the “personal communication” source, also described as “someone once told me.”

Evidence-based health care (EBHC) aims to ensure that clinical decision making and health services purchasing is systematically based on information derived from good-quality research. As such, “evidence-based health care is a discipline centered upon evidence-based decision making about individual patients, groups of patients or populations, which may be manifest as evidence-based purchasing or evidence-based management” [8]. For the purposes of this article, EBHC refers to evidence-based purchasing at the level of groups or populations of claimants.

Systematic appraisal of the best evidence available is a prerequisite for evidence-based decision making by those responsible for purchasing health care services or setting minimum standards for populations of patients [9]. The required skills to achieve this include: an ability to define criteria such as effectiveness, safety, and acceptability; an ability to find articles on the effectiveness, safety, and acceptability of a new test or treatment; an ability to assess the quality of evidence; an ability to assess whether the results of research are applicable to the whole population from which a sample was drawn; and an ability to assess whether the results of the research are applicable to the “local” population.

Despite affecting entire populations of patients, decision-making processes by purchasers of health care services have been little studied. Whereas skills and attitudes of clinicians toward EBM have been well researched [10–15], little is known about the ability and willingness of staff working in accident insurance or health services purchasing to adopt these principles. A literature search revealed a paucity of reported studies on this topic to date.

In New Zealand, the Accident Compensation Corporation (ACC) administers under government legislation an accident compensation scheme that provides comprehensive and compulsory personal injury insurance coverage for individuals injured in New Zealand, whether they are citizens, residents, or temporary visitors. The stated aim of the corporation is “to help injured people return to normal life as soon as possible” [16].

The ACC system is a unique, comprehensive, no-fault, twenty-four hours per day, seven days per week scheme. When introduced in 1975, it replaced common-law rights to sue with entitlements that were supposed to provide equal access to both health care and compensation without any delay or the need for adversarial disputes. New Zealand is a country of 3.8 million people. ACC receives about 1.2 million claims per annum, although most of these are minor and require only a single visit to a treatment provider, such as a family doctor. ACC has a total staff of 1,700, located in one head office and thirty-five branches. ACC pays out 1.4 billion in New Zealand dollars per year, medical treatment costs $190 million, elective surgery $85 million, rehabilitation $690 million, and compensation for people unable to work $620 million [17].

A health purchasing division within ACC negotiates contracts and other purchasing arrangements with providers for a range of treatments, rehabilitation, and other services, funded on an expected outcome basis. These providers are in both the public and private health sectors. They are selected on the basis of quality, accessibility, and price. Contracts are developed to encourage quality improvements and cost effectiveness of these services [18]. Under New Zealand legislation, the ACC has a statutory obligation to ensure that services purchased are of the “quality required” [19]. That is, the purchaser has a responsibility to contribute to the establishment of realistic expectations for claimants and health care providers. Thus, decisions made by this health purchasing division potentially influence the majority of health services provided for accident victims in New Zealand, from emergency transport, primary care services, rehabilitation, to elective surgery procedures.

A survey of the health purchasing division staff at ACC was undertaken. The aim of this survey was to determine their current procedures of assessing the effectiveness of treatments and their familiarity with the key concepts of EBHC. This survey was conducted to provide the basis for a support and training package to be developed by an in-house EBHC Advisory Group, which was created about three weeks before this survey commenced. It is therefore believed that the advisory group did not have a chance to impact the results of this survey. The purpose of this advisory group is to conduct health care technology reviews and promote the application of evidence-based health care principles in the organization.

METHOD

Twenty-two out of thirty-four staff members from the health purchasing division of ACC were interviewed during a two-week period in March 2000. The interviewed staff members included one physician, one physiotherapist, eight nurses, two social scientists, one educationalist, one cultural adviser, three accountants, and one lawyer. Those excluded were five secretaries, the general manager, four staff members who were abroad, and two staff members specializing in promotion of EBHC who authored this article.

The interviews consisted of eight preformatted questions (see question 1–question 8 in the “Results” section) designed to elicit knowledge of EBHC and specific information-seeking behaviors with respect to the effectiveness of treatment. The authors of this article, each of whom had conducted previous surveys with questionnaires, formatted the questions. Six of the questions explored whether, and how, the staff looked for information concerning effectiveness of treatment; which information sources they used; and to what degree, and how, they evaluated the quality of that information. Particular emphasis was placed on discovering whether staff members questioned the information sources they chose, or whether they merely accepted any information given to them. Staff were also asked what they understood by the term “evidence-based health care” and about their training preferences to improve information-seeking skills. To allow an opportunity to exhibit their full range of knowledge, respondents could provide open answers, with no limits to the number or type of answers given and only limited prompting by the interviewer.

All interviews were undertaken and recorded by the same researcher. To allow a quantitative analysis of responses, a first step was to establish typologies of responses to each question. The transcripts were read to identify themes from individual responses, and these were then grouped into categories to produce typologies. To ensure reliability, two researchers analyzed the data independently and the results were compared. Both generated similar systems of categorization, although these categories were initially described in slightly different terms.

Once typologies were established, responses to each question were transformed into a data matrix of response categories (columns) and individual staff members (rows) by codifying answers in each cell as 1 or 0 when they did or did not fit into a given category. Individual responses were assigned to as many categories as relevant, as respondents could provide multiple answers. Frequencies and percentages calculated for each category corresponded to the number and percentage of respondents who gave an answer fitting into a given category.

RESULTS

The mean age of the twenty-two interviewed staff was thirty-eight years (range 24–53 years); 85% were female and 15% male. Their responses to each of the eight interview questions are presented separately below. Note that total frequencies and percentages exceed twenty-two and 100% respectively, as respondents could provide multiple answers to each question.

• Question 1: “How do you know if a treatment is effective?” The twenty-two staff members interviewed provided thirty-three responses. Fifteen (68.2%) indicated that they assessed the effectiveness of a treatment by accessing published information (research results, medical literature, Internet) and ten (45.5%) by consulting others, mostly medical professionals within ACC (27.3%, including 13.6% consulting the staff of the EBHC Advisory Group) or from outside institutions (18.2%). Three relied on their personal knowledge or experience. One response was “don't know,” and one was “when there is no complaint” from the claimant.

• Question 2: “How do you look for information?” Responses to this question showed a clear preference for consulting medical professionals or colleagues (Table 1). Of the forty-six responses, twenty-three involved consulting others (12), medical professionals (6), and the EBHC Advisory Group (5), whereas fifteen involved accessing the medical literature, either by themselves or with the help of the ACC librarian. Three respondents said they do not look for information by themselves; one did not know how to look for information; and one believed that he or she was unable to access information. Three staff members expressed confidence that they were able to look for information by themselves.

• Question 3: “What information source do you use?” The responses to this question indicated a preference for nonhuman sources of information (library, Internet/CD, MEDLINE, journals, books, guidelines) over human sources (medical professionals and others, EBHC Advisory Group, personal knowledgebase). Nonhuman and human sources of information accounted for thirty and twenty-two, respectively (Table 2). Most respondents who looked for nonhuman sources sought it from the library, the Internet, or information databases. Several staff members expressed the importance of having direct Internet access to facilitate their searches for information.

• Question 4: “How do you know what the quality of that information is?” Most respondents indicated that they assessed the quality of information by consulting others, the perceived reliability of the research methodology, or the reputation of the source of information (Table 3). The second-highest proportion of responses stated that they did not know how to evaluate the quality of information. Only one respondent mentioned asking the EBHC Advisory Group. Several respondents reported that they did not know how to critically evaluate medical research and health care information but would like to learn skills to do so.

• Question 5: “What type of information do you need in order to be able to determine if a treatment is effective?” Only twenty staff members responded to this question. No clear preference emerged with respect to the types of information perceived as being needed. Four staff members' sole response was “do not know,” and three answered that they “ask others” (Table 4). This meant that seven of the twenty staff members responding to this question indicated that they had little idea of the information required to determine the effectiveness of a treatment. However, most of the respondents who did identify the type of needed information mentioned more than one type.

• Question 6: “Where do you find this information?” The results indicated that the majority of the staff believed they could access necessary information to determine if a treatment was effective through the Internet or sources such as MEDLINE (Table 5). Their ability to interpret this information accurately is uncertain. Two staff members reported that they “ask others.”

• Question 7: “What do you understand by the term EBHC?” Of the twenty-two respondents, only five (22.7%) gave an explanation that accurately portrayed the discipline of evidence-based health care. Fourteen (63.6%) gave an explanation that did not even approximately describe the basic concepts, and three (13.6%) stated that they did not know at all.

• Question 8: “What would help you to find out if a treatment is effective?” Most respondents emphasized that a training seminar, written instructions, training for search skills, or a mixture of these would be very helpful to improve information-seeking skills to appraise treatment effectiveness (Table 6).

Table 1 Breakdown of forty-six responses by the twenty-two staff members to question 2 “How do you look for information?”

Table 2 Breakdown of fifty-five responses by the twenty-two staff members to question 3 “What information source do you use?”

Table 3 Breakdown of twenty-eight responses by the twenty-two staff members to question 4 “How do you know what the quality of that information is?”

Table 4 Breakdown of twenty-three responses by the twenty staff members to question 5 “What type of information do you need in order to be able to determine if a treatment is effective?”

Table 5 Breakdown of thirty responses by the twenty-two staff members to question 6 “Where do you find this information?”

Table 6 Breakdown of thirty responses by the twenty-two staff members to question 8 “What would help you to find out if a treatment is effective?”

DISCUSSION

This study aimed to elicit the skills at evaluating treatment effectiveness and familiarity with the key concepts of EBHC of the staff in the health services purchasing division of ACC, a large national accident insurer. Decisions about the allocation of health resources are increasingly being made, or at least influenced, by purchasers and funding organizations. The behavior of clinicians needing information when faced with uncertainty about effectiveness of new treatments, products, or procedures has been studied previously. In many cases, their ability to find and critically appraise information has been found lacking [20–23]. To the authors' knowledge, equivalent studies of personnel involved in making administrative decisions about the purchasing of health services have not been reported to date. The aim of this study was to determine how such people differentiated effective from ineffective treatments, which in turn had significant implications for the quality of health care purchasing.

When asked how they knew if a treatment was effective, the purchasing staff responded that they placed a high reliance on published information and asking others. However, it should be noted that the reported importance of looking at published information (nonhuman sources) by staff members was almost certainly overestimated in these results. When staff did not spontaneously report that they looked for published information, the interviewer prompted them by asking if they did so.

When asked how they look for information, the purchasing staff strongly indicated that they either regularly asked others or simply did not seek information by themselves. Moreover the majority acknowledged that they did not know how to look for appropriate information. This finding was consistent with other studies of the medical profession that have demonstrated that “asking others” was the most commonly used information-seeking method. Many studies have found that physicians relied heavily on human rather than published (nonhuman) sources of information to meet their information needs [24–27]. It can be speculated that any tendency to rely on information from others, rather than seeking new published information, might result in purchasing staff adhering to traditional models of health care. If this occurs in place of research knowledge being fully diffused, then purchasers may find themselves at risk of efficiently funding ineffective treatments. This scenario has significant social implications in any country using purchasing staff to administer health care resources.

Responses to the question about specific information sources used contrasted with those obtained when asking staff members how they looked for information. Rather than mostly relying on “asking others,” staff members indicated that they often turned to published sources of information. This finding might be influenced by ambiguity of the term “source” in the question, prompting respondents to consider sources other than interpersonal communication. It might also be an artifact of socially desirable responding. This hypothesis was supported by direct observations obtained from the ACC librarian, who logged requests for information over a six-month period and reported that in fact few staff attended the library in person. Those who stated that they sought nonhuman information as their information source commonly mentioned the library and the Internet. The reason for this could be that going to the library often incorporates assistance from a librarian to access Internet-based information. Staff members expressed the clear expectation that the Internet provided direct access to a wide range of published information.

The fourth question was particularly important, because it asked how staff members determined the quality of information about treatment effectiveness. Content analysis of the staff responses indicated a low level of knowledge about the value of systematic searches, reviews, evaluation of the literature, and relevance of this type of published information. This implied that purchasing staff needed further information about the role of evidence and possibly some training in ranking the quality of available information. This conclusion was supported by the observation that more than three-quarters of staff members were unaware that they could access the services of the in-house EBHC Advisory Group. However, this group had only been created three weeks before the survey. An important finding was that less than a quarter of the purchasing staff believed they had any ability to assess the quality of information about effectiveness by reviewing the research methodology. This finding indicated a general knowledge gap about research methodology relating to health services. Most staff acknowledged that they needed to update their skills for assessing information, evaluating medical literature, and improving understanding of medical research methodology. Four out of twenty respondents clearly did not know what type of information was required to determine if a treatment was effective, and another three admitted that they would have to ask someone else, possibly their own family doctor. More than a third of the purchasing staff members were unable to define any aspects of the type of information needed to determine if a treatment was effective. In addition, the choice of whom to ask about specific treatments was often inadequate. This result indicated a significant gap in their information knowledge and implied the need to bridge that gap through some form of effective training.

Responses to the sixth question inadvertently overlapped with those from the third question. The results suggested that purchasing staff in fact made more use of Internet resources and less use of the library itself. This finding was consistent with the researchers' observation of actual purchasing staff behavior. This might be because the library also provided access to the Internet, and when staff referred to the library they were in fact referring to the Internet. The total number of times the library and the Internet was mentioned in response to question 3 was twenty-one, and with question 6 was twenty. As noted, this indicated substantial overlap between the two questions.

There is a huge amount of medical information available via the Internet, although the quality of this information is known to be variable [28]. Because accessibility to a well-stocked medical library as a source of information is often limited, Internet sources of information represent a potential useful alternative, especially for clinicians and administrators who are remote from a library. The Internet is a relatively cheap way to receive updated information and is accessible either from home or office, twenty-four hours a day. It is suggested that Internet access is a very useful tool for health services purchasing decision makers [29, 30].

When asked to provide their understanding of the term EBHC, most respondents lacked familiarity with the term and had a low level of general understanding about this area. This finding indicated the need for education about the background to EBHC as a discipline, given the importance of these concepts to the core business conducted by purchasing staff in a large accident insurer. To that effect, the most popular request was for training sessions and seminars, possibly because they are perceived as the easiest way to learn. Many of the staff members appropriately asked that the EBHC Advisory Group take a direct role in the educational process and provide important information. This finding indicated a welcome desire and willingness among the purchasing staff to learn more about the subject of EBHC.

Overall, the results of this survey indicated that there was currently a low level of awareness by purchasing staff regarding the use of evidence. The knowledge about the value of guidelines as a type of published medical literature that included systematic searches and review of literature was lacking.

It was concerning to observe that many staff members needed training to directly question information about the effectiveness of treatments. The majority appeared to lack the skills to evaluate the quality of any health care literature they might read. However, it should be noted that the EBHC Advisory Group was formed only three weeks before this survey took place. A consistent theme across all of the results was that while more information about the role of an evidence-based approach was required, this would be readily welcomed as both written and practical training. The incorporation of an EBHC Advisory Group into the decision-making process for purchasing health care is clearly an important issue. Further research into how this can be effectively achieved is warranted.