On Being a Good Listener: Setting Priorities for Applied Health Services Research

In the last decade, explicit priority setting has become an integral part of health care systems. Indeed, there is even an International Society on Priorities in Health Care, created in 1997 (Ham 1997). Whether it is Oregon's priority ordering of symptom treatment pairs to maximize the impact of a limited Medicaid budget (Fox and Leichter 1991), England's National Institute for Clinical Excellence's assessing priorities for therapeutic innovations in the National Health Service (Rawlins 1999), or New Zealand's setting priorities for patients’ access to cardiovascular treatment (Hadorn and Holmes 1997), techniques for judging the relative worth of different health service investments abound.

As these techniques are refined, the most common addition is the incorporation of public values as part of the assessment. Priority setting is increasingly seen as combining an objective assessment of costs and effects with a more subjective assessment of patient or public preferences (Lenaghan, New, and Mitchell 1996; Lomas 1997; National Institute for Clinical Excellence 2002; Stronks et al. 1997).

The incorporation of the service users’ values has led some commentators to conclude that the process is more important than the science. For instance, Klein declared:

Given conflicting values, the process of setting priorities for health care must inevitably be a process of debate. It is a debate, moreover, which cannot be resolved by an appeal to science and where the search for some formula or set of principles designed to provide decision-making rules will always prove elusive. Hence the crucial importance of getting the institutional setting of the debate right … the right process will produce socially acceptable answers—and this is the best we can hope for.

(Klein and Williams 1999, 20–21)

Our article is a response to Callahan's recent observation that “the effort to devise priorities for health care systems is far more advanced and sophisticated than for biomedical research. With the exception of the NIH, it is hard to discover much extended debate anywhere in the world on how best to prioritize research” (Callahan 1999, 116). We describe how the lessons learned from and the approaches to setting priorities for funding health services are translated into setting priorities for funding health services research, in which the paymasters want to use the finite human and financial resources for those areas most likely to improve health services delivery.

We agree with Klein's and Callahan's views that getting the priority-setting process right—preventing domination by any particular interest, being transparent, and reflecting the values of all relevant users—is the most important consideration. Their approaches also reflect the emerging importance of involving users and their values in the process. In the case of research, however, the users include not only patients and providers but also managers and policymakers striving to make the best possible decisions in an evidence-based health service (Walshe and Rundall 2001).

We begin with a review of previous approaches to setting priorities for health research and then state the case for the greater involvement of research users, at least for applied health services research. We use two case studies—one in England and one in Canada—to describe a “listening model” for setting user-driven health services research priorities. Finally, we draw lessons from the case studies before concluding that setting priorities for health services research is more an interpretive than a technical exercise.

Setting Priorities for Health Research

Two broad approaches are evident in the literature on setting priorities for health research: technical assessments, dominated by quantifiable epidemiologic or other needs and costs data, and interpretive assessments, dominated by the consensus views of informed participants.

Technical approaches are obviously highly dependent on data and, therefore, tend to express priorities using those units for which data are readily available: diseases (according to their burden in the population), health problems (related to their prevalence), or treatments/technologies (in regard to costs and extent of use). They are most commonly applied when making priority judgments across potential clinical trial investments (Detsky 1989, 1990; Claxton and Posnett 1996; Weinstein 1983; Weisbrod 1971) or technology assessments (Donaldson and Sox1992; Eddy 1989; Phelps and Parente 1990). For example, Eddy (1989) prescribed a process for the Institute of Medicine's Council on Health Care Technology to collect or estimate the data for the prevalence of the specific condition, the unit cost of the relevant technology, various uses of the technology, the burden of illness addressed by the technology, and the potential of the technology assessment results to change health outcomes and costs. The total impact of funding an assessment is obtained by applying these data in a formula and assigning a series of weights.

The difficulty with this approach is that it tends to bury under a series of assumptions many value judgments that may or may not reflect those of the broader population of users and payers. For instance, the burden of illness is not readily defined in an uncontroversial way. As Resnick pointed out, “There is no set formula for determining the burden of disease … different individuals may arrive at different estimates of the burden of disease as a result of emphasizing different criteria” (2001, 187). Indeed, Gross and colleagues demonstrated this controversy using the funding priorities of the U.S. National Institutes of Health (Gross, Anderson, and Powe1999).

In addition, these technical exercises must adopt a single clear objective to guide their conduct. Most assume that improving the population's health is the reason for the public funding of health research. While no one would deny that this is the ultimate aim, it ignores the reality that intermediate outcomes may sometimes be of greater value. For instance, health research funding agencies often want at least part of their funding to advance knowledge unconstrained by any direct application to health improvement, that is, “blue skies research.” They may also want to develop research capacity in a specific topic or region, redress historical imbalances, answer concerns about cost containment, or improve system governance. These and other intermediary objectives are more closely related to the capacity of the research community or to service delivery and organization than they are to the direct improvement of the population's health.

Interpretive approaches therefore become attractive for their ability to juggle multiple assumptions and objectives. They also help subdue the negative reactions to top/down directives that priority setting elicits in cultures such as that of the United States, which has a history of resistance to centralized planning. Interpretive approaches are more commonly used for agencywide assessments like those of the U.S. National Institutes of Health (Committee on the NIH Priority-Setting Process 1998) or the Science and Technology Council of Australia (Stocker1997). Even though technical data are available for this approach, it relies far more on the subjective judgments of participants expressed through structured exercises.

For instance, England's Health Technology Assessment (HTA) program of the National Health Service (NHS) uses four separate groups, each with a different role, to interpret priority technologies for assessment (Stein and Milne 1998). Potential users of the research generate ideas; university researchers prepare briefing papers and background data; expert advisory panels draw up a short list of priority candidates; and a multistakeholder group “applies criteria … in an interactive process of debate and scoring … [which] leads to a collectively agreed ranked list” (Stein and Milne 1998, 76). As information is collected and passed through each stage, it is guided by criteria and collectively refined by an interpretive process that brings together various objectives and incorporates debates about assumptions.

In a less organized fashion the National Institutes of Health has, under pressure to be more systematic in its priority-setting process, adopted a similar interpretive approach (NIH Working Group on Priority Setting 1998). Callahan described this as “using unranked criteria as its principle guideline and partially informal means of actually using those criteria as its main way of deploying them” (Callahan 1999, 120). The process contains input from the formal Council of Public Representatives (Ag 1998) and was largely endorsed as producing satisfactory results after being examined by a committee of the Institute of Medicine (Committee on the NIH Priority-Setting Process 1998).

Nevertheless, these interpretive processes are only as good as their inputs. Alongside whatever technical data that are made available are the values and assumptions used to interpret those data. Do these approaches reflect the values and assumptions of the stakeholders who should have a voice? Or are they dominated by researchers—with their detached view of the priorities in the health service—or senior bureaucrats—with their attached but somewhat lofty view of the health service?

These are important questions to answer because as the Commission on Health Research for Development stated, “The perception of health problems may differ according to the people consulted” (1990, 39). Sitthi-amorn (1989) demonstrated this for the priority ranking of health problems in Bangkok, by contrasting the views of health professionals (using epidemiologic survey data) with the perceptions of urban slum dwellers. The health professionals listed infection, addiction, and violence as the top three problems, whereas the urban slum dwellers placed sanitation, flooding, and health services at the top of their list. In the London borough of Hackney, researchers Bowling, Jacobson, and Southgate (1993) also found that family physicians and the general public gave health problems different priorities. That is, the general public's top four problems, selected from a menu of sixteen possibilities, were ranked fifth, fourth, eleventh, and twelfth by family physicians.

It may be that more than just the priority rankings is affected by which stakeholders participate. The “currency” used to express priorities also may change. For instance, managers may view the world as a series of issues; policymakers may define it in terms of interest groups; the public may see it as a set of problems; researchers may find disciplines or even methodologies; and clinicians may see only diseases.

Of course, the challenge for priority setting is that there is no gold standard to judge these results against, no “right” answer, and hence the attraction of interpretive rather than technical approaches (Global Forum for Health Research 2000). Such approaches can blend subjective inputs from a variety of stakeholders with whatever data are available. However, whether to privilege, and in what proportion, the views of the different stakeholders—public, clinicians, managers, policymakers, or epidemiologic and research experts—depends on the objective of the research investment decisions under consideration. Put another way, the structure of interpretive research priority setting should reflect its function.

Setting Priorities for Applied Health Services Research

The previous section justified an interpretive approach to setting priorities, which includes selected stakeholder participants reflecting and resolving sometimes competing values and assumptions. We do not reject a role for technical data but, rather, place it in a broader context that allows representative stakeholders to debate and clarify the implicit assumptions upon which technical data are often based. We have not yet defined, however, the appropriate mix of stakeholders, nor have we suggested any differences in their roles. The mix of stakeholders depends on the particular function of the research priority–setting exercise, which, in turn, depends on the particular type of research for which one is setting priorities. Similarly, the role of these stakeholders should be determined by the type of research and the political and organizational context of the health system for which priorities are being set.

In regard to the type of research, we find that existing definitions of health services research do not always agree on whether the research incorporates both applied and basic science elements or only an applied element. For instance, the U.S. Institute of Medicine defined “a multi-disciplinary field of inquiry both basic and applied” (Institute of Medicine 1979, 7). In contrast, the 1988 report by the House of Lords Select Committee that led to the United Kingdom's research and development strategy in the 1990s defined health services research as “all strategic and applied research concerned with the health needs of the community as a whole, including the provision of services to meet those needs” (House of Lords Select Committee on Science and Technology 1988, 26). Notable in this latter definition, and other definitions of health services research, is its focus on directly using the research to improve the delivery of health services. The applied nature of the research connects it more directly than basic research does to those influencing or actually delivering and organizing services. In this article, we use the United Kingdom's applied-oriented definition.

The function of priority setting in applied health services research is not, therefore, to reflect the interests and hunches of the research community or other blue skies perspectives (Stryer et al. 2000). Instead, its central role is to identify those areas in which research investments are most likely to improve service delivery and organization. This implies significant representation in the process from system managers and policymakers, those most closely related to service delivery and organization on a daily basis (e.g., Rogers, Lamont, and Haines 1995). In addition, clinicians and service users such as patients should have a voice, as they will reflect the impact of the managers’ and policymakers’ current strategies (Vella et al. 2000).

Of course, these stakeholders’ backgrounds may vary as the function of a priority-setting exercise becomes more specific. For instance, if there are political concerns about service delivery to specific population groups—the poor, the uninsured, ethnic groups, rural residents, and so on—or when setting priorities for a specific type of health problem or a politically salient disease, stakeholder participants should be oversampled for those with knowledge in these areas. This once again underlines the importance of defining the function or objective of the exercise before deciding on the participants.

Nevertheless, representing political concerns when setting priorities can rub both ways. Their influence on primary research agendas can be negative to the extent that the interests of politicians and of those like senior policymakers closely involved with the political process are short term and ideological. As discussed in more detail later, when primary research—which takes two or three years to come to fruition—is the focus of priority setting, the role of these policymakers may not be as valuable as it is when shorter-term synthesis or other forms of secondary research are the focus. Furthermore, the process must make sure that interest groups and politicians cannot use research priority setting to get a contentious and partisan political issue on the policy agenda as easily as they can use traditional lobbying efforts.

Thus national research funders can become innocently embroiled in politically volatile and dangerous issues—a lesson that was certainly learned at the Agency for Health Care Policy and Research in the mid-1990s, with its low back-pain practice guideline and its Patient Outcomes Research Teams (Gray, Gusmano, and Collins 2003). Unlike lobby groups which do not jealously guard their independence from partisan politics, national research funders need to balance their need to be seen as relevant with their need to be seen as independent. Prudently designed priority-setting processes, with most of the focus on primary research needs for the medium to the long term, can achieve this balance without undue influence from those at the health system's partisan policymaking level. In addition, such priority-setting processes probably should be sponsored by neutral bodies that do not depend on political influence and goodwill for their annual budgetary allocation. Speaking truth (or research) to power can certainly be dangerous. In the United States, the Institute of Medicine, through its Board on Health Sciences Policy, may be a good candidate for this role.

There is a final compelling reason to provide a central role for health system managers and policymakers in setting priorities. Not only do they have specific knowledge of the issues in the system, but they also are the ones who will choose to apply or ignore the results from health services research. Having them help identify the priorities thus increases their sense of ownership and the likelihood that they will adopt and apply the research findings. As Gibbons and colleagues (1994) observed in their study on the social distribution of knowledge for use in research, more interaction is needed among researchers, decision makers, service users, and funders in identifying and formulating research questions. (By decision maker or decision-making organization, we mean those whose actions directly result in the allocation and/or distribution of the health system's resources.) Others have made a similar point about areas such as mental health (Feldman1999), state health policy (Coburn 1998), and education (Huberman 1990)

Indeed, health services research is increasingly seen as a “domain where disciplines and decision makers meet” (Lomas 2000b, 6). There is an emerging model of ongoing “linkage and exchange” between researchers and decision makers throughout the research process (Lomas 2000c), supported by the findings from numerous studies that the best predictor of research use is the early and continued involvement of relevant decision makers (Caplan 1979; Huberman 1994; Landry, Amara, and Lamari 2001; Lavis et al. 2002; Rich 1991; Weiss 1997; Wingens 1990), and that one goal of applied research funding should be to better expose researchers and decision makers to the constraints and realities of each other's worlds (Lomas 2000a; Fox and Oxman 2001; Walshe and Rundall 2001). The underlying model here of the relationship between research and policy/practice is an “enlightenment” model rather than a linear “engineering” model (Bulmer 1982; Weiss and Bucuvalas 1980). Setting the priorities, as perhaps the earliest of all the stages in the research process, is the obvious place to start with such linkage and exchange.

The Listening Model and Two Case Studies

An interpretive “listening model” for priority setting is based on this principle of linkage and exchange between research funders and researchers on the one hand and the research's potential users on the other. We contend that both sides need to listen to each other in order to arrive at research priorities that are relevant to decision makers, feasible within the research community, and eventually incorporated into practice. This listening model has six steps (see table 1):

TABLE 1.

Listening Model for Defining Applied Research Priorities

Model (steps 1 to 6)	English Case Study (September 1999–February 2000)	Canadian Case Study (January 2001–June 2001)
1. Identify the stakeholders to participate.	An “expert forum” of 40 stakeholders advised on the composition of the consultation's focus groups and the main issues to address.	The partners in the consortium talked with one another and their own communities about the participants for the workshops.
	Stakeholder groups to be consulted were identified and ranked into five groups (see step 3).	Environmental scan was made of health services–related organizations (half research and half decision making).
2. Identify and assemble any needed data.	Orientation to the SDO program and the discussion questions were established.	Current priorities of the 56 organizations in the environmental scan were established.
3. Design and complete stakeholder consultation to identify issues likely to be a priority over next three to five years.	Twenty-two focus groups were formed, with a total of 265 participants, reflecting proportions suggested by the expert forum: 65% user reps, clinical staff, and middle managers. 16% policymakers and senior executives. 19% researchers, educators, and research funders.	Six workshops, with a total of 175 participants in the approximate proportions suggested by the consortium, were set up: 30.0% policymakers and clinical associations. 30.4% managers. 39.6% researchers and research funders.
		One survey and two meetings with senior civil servants were held; one survey with researchers was held.
4. Validate priority issues.	Findings from the consultation on priority issues were tested with the expert forum (see step 1) and two groups of policymakers (n= 89).	During the whole process, triangulation was used for the environmental scan, consultation workshops, surveys, and interviews.
5. Translate issues into priority research themes.	The expert forum and the two groups of policymakers (see previous step) identified priority research areas (n= 10).	Panel of experienced applied researchers identified priority research themes (n= 15), using set criteria (see table 3).
6. Validate priority research themes.	Research was conducted to establish priority area not covered elsewhere. Discussions held with multistakeholder commissioning board. Five priority areas were established.	Workshop participants were surveyed on their reaction to the 15 research themes: 90% believed priorities addressed their issues. 85% believed priorities reflected their workshop's debates.

1. Identify the stakeholders to participate in the consultation. A group of experts determine the desired mix of key stakeholders who should be consulted. Most of the participants should represent potential users of research, since the main objective of the consultation is to identify the priorities of these potential users

2. Identify and assemble any data needed for the consultation. Data, such as on current system problems or existing research priorities, are collected to help inform participants’ discussions during the consultation and help them build on existing knowledge during their exchanges

3. Design and complete the consultation with the stakeholders to identify those issues likely to be a priority over the next three to five years. The consultation should be designed to ensure that stakeholders are consulted on issues they feel will remain priorities in the long term, that is, for the next three to five years. This period takes into account the time needed for the design, conduct, and completion of primary research studies. Stakeholders also could be consulted on shorter-term priorities that, if data are available, would help create a secondary research agenda for exercises such as synthesis or meta-analysis. This, however, means another step in the process to determine whether an adequate number of primary studies are, indeed, available for synthesis.

4. Validate the identified priority issues against similar exercises. Validating the results of the consultation against other sources of similar information helps ensure that the priorities are generalizable and not artifactual to the particular process or people involved.

5. Translate priority issues into priority research themes. The priority issues uncovered through the consultation need to be “translated” by research experts from pressing issues into priority areas in which it is possible and/or feasible to do research

6. Validate the priority research themes with stakeholders. A validity check with the consulted stakeholders is needed to guarantee that the final research priorities truly reflect their expressed views.

Before we discuss in more detail each of these steps in the listening model, we will describe two case studies that used this model. Although both cases used all stages of the model, the methods used within each stage differed. The agencies in both countries used the results of the model to influence a predetermined portion of their overall allocation for health services research.

The English Listening Exercise

In England, a listening “exercise” was conducted between September 1999 and February 2000 by the National Health Service's Service Delivery and Organization (SDO) research and development program to set its agenda for research. Established in 1999, the purpose of the SDO program is to produce and promote the use of research evidence to improve the organization and delivery of health services. The program is funded by the National Health Service (NHS) and is one of three national research and development programs (Black 1997). The SDO used the listening model to bring together a range of stakeholders in focus groups around the country, both to understand the issues most important to those delivering, organizing, and using services and to build ownership for the program among these groups (Fulop and Allen 2000).

In the first step, an expert forum of 40 invited specialists from health service managerial, professional, user, policy, and research backgrounds was convened to agree on the composition of the focus groups and the main issues they should address. This forum recommended that the focus groups concentrate on identifying, from their own perspective, local and national issues relating to the organization and the delivery of health services. Hence no additional data beyond the experience of the participants and basic information about the SDO program were deemed necessary.

The expert forum, along with SDO staff, ranked the stakeholder groups in order of importance. In accordance with the purpose of this exercise, the forum gave the highest ranking to those at the front lines of service: patients and other service users, clinicians, and managers. Senior-level decision makers, including policymakers, were given the next ranking, with researchers, educators, and research funders ranked last.

Twenty-two focus groups, with a total of 265 participants, were conducted. Two meetings were held in each of the eight English NHS regions (heterogeneous in terms of stakeholder composition), and six meetings were held for stakeholders such as service user representatives, research funders, and researchers (homogeneous in terms of composition). The discussions within each group were based on questions such as: “If there was one change to the organization of NHS services which you would like to see, what would it be?” and “Thinking forward to five years’ time, what do you think the major issues facing the NHS will be?”

The findings from these focus groups were debated at a second meeting of the expert forum and at two meetings with senior policymakers. These debates largely validated the areas of particular concern identified by the service users and NHS staff who participated in the focus groups. The forum experts and policymakers then translated the issues of most concern into the ten highest-priority research areas (see table 2). Finally, the SDO's multistakeholder research–commissioning board, taking account of the research agendas of other agencies and cognizant of the SDO's need for a manageable agenda, validated five of these priority areas for active commissioning. A total of 354 people were consulted face-to-face during this process.

TABLE 2.

Priority Research Themes from English and Canadian Listening Exercises

Themes from English Listening Exercisea	Themes from Canadian Listening Exerciseb
	Primary Themes
Organizing health services around the patient's needs, e.g., access to health services	Health human resources
User involvement	Financing and public expectations
Continuity of care	Governance and accountability
Coordination/integration across organizations	Driving and managing system change
Interprofessional working	Improving quality
Workforce issues	Health care evaluation and technology assessment
Relationship of organizational form, function, and outcomes	Public advice seeking in the era of e-health
Implications of communication revolution	Improved access for “marginalized” groups
	Secondary Themes
Use of resources, such as ways of disinvesting in services and managing demand	Primary health care
Evaluation of implementation of major national policy initiatives	Globalization
	Regionalization
	Population health
	Continuum of care and delivery models
	Performance indicators, benchmarks, and outcomes
	Evolving role of informal and voluntary care

^aFor more details, see N. Fulop and P. Allen, National Listening Exercise: Report of the Findings (London: NCCSDO, 2000). Available at http://www.sdo.lshtm.ac.uk

^bFor more details, see D. Gagnon and M. Menard, Listening for Direction: A National Consultation on Health Services and Policy Issues. Summary Report (Ottawa: Canadian Health Services Research Foundation, 2001). Available at http://www.chsrf.ca

The Canadian Listening Exercise

The Canadian case was a consortium of five national organizations involved in health services research. A listening exercise was used in 2001 to “listen for direction” from health system managers and policymakers regarding priority health services and policy themes (Gagnon and Menard 2001). These themes were to determine their collective and individual research agendas. The consortium consisted of

• The Canadian Health Services Research Foundation, an independent, not-for-profit corporation established with endowed funds from the federal government and its partners. It promotes and funds management and policy research in health services and nursing to increase the quality, relevance, and usefulness of this research for health system policymakers and managers (Lomas 2000c). The foundation coordinated the activity on behalf of the consortium.

• The Institute of Health Services and Policy Research, one of the 13 institutes within the Canadian Institutes of Health Research, Canada's premier federal agency for health research. It is dedicated to supporting innovative research, capacity building, and knowledge translation initiatives designed to improve the way that health care services are organized, regulated, managed, financed, paid for, used, and delivered, in the interest of improving the health and quality of life of all Canadians.

• The Advisory Committee on Health Services of the Conference of Federal/Provincial/Territorial Deputy Ministers of Health, one of several committees consisting of representatives from the provinces and the federal government that advise the senior levels of health ministries.

• The Canadian Coordinating Office for Health Technology Assessment, a publicly funded agency that responds to the needs of Canadian health care decision makers by providing unbiased, reliable information about health technologies, focusing on evaluations of clinical effectiveness and cost effectiveness.

• The Canadian Institute for Health Information, an independent, not-for-profit organization working to improve the health of Canadians and the health care system by providing quality, reliable, and timely health information.

The first step was to conduct an environmental scan of 56 organizations, split more or less equally between research groups and funding agencies, on the one hand, and decision-making organizations, on the other, to gather data on current priorities in health services research. The results of this scan helped the consortium identify the relevant stakeholders and served as a basis for the subsequent priority-setting discussions in these stakeholders’ workshops.

The organizations in the consortium also consulted with their own communities before jointly agreeing on the desired balance of stakeholder participation in the workshops. Because the principal focus was on the research needs of the system managers and policymakers, these groups were ranked as equally and preeminently important to the consultation. Professional associations representing clinicians or other providers were incorporated as part of the policymaker category. As with the SDO exercise, the participation of researchers, educators, and research funders was deemed less important to the workshop consultation part of the process (although ten selected researchers with specific skills in translating system issues into researchable questions were more extensively involved at a later stage).

Five regional workshops and a national workshop were held, with 175 participants. The workshops were designed to enable participants to concentrate on longer-term (three to five years) issues in health services, by presenting some target “future scenarios” and asking questions like “What are the most pressing issues you will face three to five years from now?” Along with these workshops, separate surveys and interviews were also conducted with senior civil servants and health services researchers.

Findings from the workshops and surveys were cross-referenced and validated with one another and with the original environmental scan. The consortium staff used this validation to integrate the output from the workshops and surveys into a preliminary synthesis of priority issues. This synthesis was then presented to and analyzed by a reference group of ten applied researcher experts who, taking into account such things as research feasibility and the activities of other agencies (see table 3), translated these priority issues into eight primary and seven secondary priority research themes (see table 2).

TABLE 3.

Illustrative Criteria Used to Translate Issues Identified in the Consultation into Priority Research Themes.

• The issue is likely to be a high priority for at least three to five years.

• There is not a large stock of existing relevant research in the area.

• The issue is amenable to a significant number of feasible and generalizable research questions.

• The research capacity exists to respond with high-quality research on this issue.

• Decision makers are receptive to research on this issue.

• Decision makers would be able to use research results on this issue.

• The research would have potentially high impact relative to its costs.

Source: D. Gagnon and M. Menard, Listening for Direction: A National Consultation on Health Services and Policy Issues. Summary Report (Ottawa: Canadian Health Services Research Foundation, 2001). Available at http://www.chsrf.ca

Finally, to validate this translation from the health service issues proposed by the workshop participants into the priority research themes defined by the experts, a survey was sent to the original 175 workshop participants, seeking their feedback on the resulting priorities. Ninety percent of the respondents considered that the priority research themes reflected their own priority issues, and 85 percent felt that the priority research themes reflected the debate in their workshop. Including interviews and validation and translation meetings, more than 200 people were consulted face-to-face in this process, and many more were involved in the surveys.

Lessons from Applying the Listening Model to the Two Case Studies

Step 1

Identify the stakeholders who should participate in the consultation. Both cases reflected the main purpose of their exercises in the desired mix of participants. The English exercise concentrated on the frontline staff and users, whereas the Canadian exercise focused on recruiting the managers and policymakers. In both countries the involvement of researchers, educators, and research funders was recognized as important but requiring less attention than the other stakeholder groups needed.

Despite this judgment about a lesser role for researchers and educators, they tended to be overrepresented compared with the defined optimal mix (see table 1, step 3). Although ranked last in importance in the English exercise, the number of researchers and educators (18.6%) exceeded the number of the higher-ranked senior decision makers (16%). This was even more apparent in the Canadian exercise, in which more researchers (39.6%) were involved than policymakers and association representatives (30.0%) or managers (30.4%). This probably represents a responsiveness bias in which the identification of the eventual aim (applied health services research priorities) leads to the highest attendance rates by those whose interests are most directly affected: researchers. Those conducting such exercises in the future would be well advised to “overinvite” other stakeholder categories relative to researchers in order to adjust for this responsiveness bias.

This overrepresentation of the research world appeared to be at the expense of representation from senior decision makers in the Canadian case and users in the English case. For senior decision makers, this is hardly surprising, given how little control they have over their time and, hence, the difficulty they likely would have in predictably freeing up a set day for the consultation. This problem may be partly resolved by arranging one-on-one interviews with the most senior of these decision makers as a parallel input to the consultation. This was done in the Canadian case, but obviously enough resources must be available to undertake such an expensive compensatory mechanism.

Step 2

Identify and assemble any needed data for the consultation. Priority setting for disease-based research or technology assessment can use data on disease incidence, prevalence, and costs to help guide the deliberations. There are no parallel ready-made sources of data on the incidence, prevalence, and costs of delivery, management, and policy problems in health systems to guide the setting of priorities for health services research. Indeed, some approaches to applied research priority setting have been misled in their conduct by adopting a biomedical perspective and trying to fit the square peg of epidemiologic disease data into the round hole of health services research (Lomas 2000b; Fulop et al. 2001).

If comprehensive surveys on health system issues or content analyses of policy documents are available, they can inform some of the debate among stakeholders. The Canadian exercise provided workshop participants with the results of an environmental scan of 56 decision-maker and research organizations that expressed their views on the prevalent health system issues in need of further research. In practice, the stakeholders in the workshops did not make much use of this information, as they felt more comfortable declaring priorities that were based on their own experience. Perhaps this reaction supports the English listening exercise, which concentrates on the experience and views of participants and the only information given to the focus groups is a basic outline of the SDO program.

Nevertheless, there are at least two other reasons to use an initial environmental scan as part of the listening model. First, it allows for triangulation as a form of validation for the issues identified by the stakeholders (see step 4). Indeed, despite the lack of attention by Canadian participants to the results of the environmental scan, these results and the issues they raised as priorities were similar. One might think that this is a reason to rely solely on the environmental scan, discarding the face-to-face consultations as redundant. This, however, would ignore the significant role of the consultation portion of the listening model in both generating ownership in the stakeholders’ priorities and offering the doers and the users of research an opportunity for linkage and exchange to better understand each other's constraints and practices.

Second, one function of the environmental scan is to identify the health service priorities of the other research funding agencies. This information is useful if one of the selection criteria for the final priorities is to avoid duplicating efforts by agencies with overlapping mandates. This was, in fact, one of the roles for the environmental scan in the Canadian case.

Step 3

Design and complete the stakeholder consultation to identify issues likely to be a priority over the next three to five years. Both case studies tried to focus the stakeholders’ attention on a longer, three-to-five-year term. The natural inclination of those working every day in the system is to attend to immediate concerns and issues (and the often more partisan elements with which they are associated). Many people must recalibrate their planning to encompass time periods beyond the next few months, if not beyond the next few weeks! This longer period is necessary, however, if the exercise is to be insulated from immediate political controversy and if the timelines of the stakeholder community are to be meshed with the structure of the research process. By its nature, primary research takes years, not months. Priorities set now will take more than two or three years to pass through the stages of announcement, proposal writing, review, adjudication, funding, and conduct before results are available to support the health system's decision making. It is of little value, therefore, to talk about today's issues and problems, as they are too late for the attention of primary research that is only now being initiated. Instead, informed “guesses” about the issues and problems of the next three to five years are required. Both exercises, therefore, posed questions that encouraged this longer-term focus. The Canadian case even had a cathartic initial “brain dump” for the participants, to clear their minds of all the current issues and problems at work pressing for their immediate attention.

Both case studies also asked participants directly what their priority issues and problems were, rather than trying to elicit research questions. This was deemed necessary so as not to disadvantage the other stakeholder participants, relative to the researchers. Most of those working in or using the health service did not know what distinguished a research question from either a manager's strategic question or an interest group's advocacy agenda. Research areas were not, therefore, the expected output of the consultations. The translation from stakeholders’ priority issues into priority research areas was left to an expert group with the background and ability to make the required distinctions (see step 5).

A final issue in the design is the use of homogeneous or heterogeneous small groups as part of the consultation. With a variety of stakeholder backgrounds—service users, clinicians, senior policymakers, managers, researchers—the question arises as to whether overall “common good” priorities would be more likely to emerge from separating groups into like backgrounds or from mixing backgrounds together. Both case studies used both types of groups. With the exception of the researchers, who tended to prioritize some themes not considered by others, the output from the homogeneous and the heterogeneous groups did not differ significantly. More exercises may need to explore this issue further.

Step 4

Validate the identified priority issues against similar exercises. Even with large numbers of participants engaged in many focus groups or workshops, we still wonder whether some idiosyncratic or nongeneralizable set of priorities has emerged from the exercise. It is, therefore, advisable to validate the outputs against other sources of similar information. Of course, sources rarely provide identical, “gold-standard” data for validation. But related sources, within the interpretive rather than the technical approach to priority setting, often allow for some checks on face validity.

In the English case study, external reference groups—the expert forum and two groups of senior policymakers—checked the validity of the issues identified as priorities by the stakeholders. They largely confirmed the outputs from the groups. The Canadian case study used triangulation with the results of the environmental scan, with the parallel surveys and interviews, and with three international exercises in countries with similar health systems to confirm the generalizability of the outputs from the workshop consultations.

Step 5

Translate priority issues into priority research themes. Given the purpose of the initial consultation steps to elicit issues, another process is needed to translate them into final research themes or questions that can be used to guide funding agency investments. Criteria such as the capacity of the research community to respond, complementarity with other agencies’ priorities, and the likelihood of results that can influence decisions can be used to guide this translation process (see table 3). In addition, interpretive skills are needed to retain the researchable themes and discard or modify the strategic questions or advocacy agendas. For instance, a strategic or advocacy issue for nurses of overcoming a particular physician group's resistance to their expanded scope of practice may be incorporated into a priority research theme on interprofessional working or on driving and managing system change.

This translation was made by the external reference groups (which included researchers) in the English case study and by an expert panel of ten experienced applied researchers in the Canadian case study. It was not a straightforward task. The danger is the late-stage insertion of these experts’ personal priorities. It may be asking a lot for them to remain fully detached unless processes are in place that significantly constrain them to work only with the priorities presented for translation. This demands relatively rigid procedures (e.g., constantly bringing participants back to the selection criteria) and a very directive chair for the meeting!

Step 6

Validate the priority research themes with the stakeholders. The long journey from the brains and mouths of the stakeholders talking about their soon-to-be pressing issues to the paper-and-ink synthesis of final research priorities allows, if not requires, much interpretation. There is, therefore, the obvious danger that the final product will be only a poor reflection of the raw materials. Thus a check of the eventual research priorities against the initial views expressed by participating stakeholders can uncover the validity from this perspective.

The English case study incorporated this validity check into the work of a multistakeholder-commissioning board, employed by the SDO program to do this and to refine the final set of priorities. The Canadian case study, following the approach of Vella and colleagues (2000), used a resurvey of the 175 stakeholder participants in the original workshops.

Impact and Implications of the Priority-Setting Exercises

In the world of research funding, in which the customers are notoriously critical, the lack of contention, complaints, and condemnation is a significant measure of acceptability. Implementation of the priorities that emerged from both exercises by the agencies involved has been greeted with little controversy and considerable acceptance. Furthermore, in the Canadian case, many of the research teams that competed successfully for funded work on the priorities have had little difficulty involving in the research the managers and other potential users of their results. It is tempting to conclude that this is because they now see the work as relevant to and emanating from not only the researcher's world but also their own.

“Lateral impact,” that is, the adoption of the priorities and/or priority-setting process by other research-funding agencies, has also been found in Canada. Three provincial bodies adopted the declared priorities to guide their own funding activities. One of these bodies repeated the listening model to adapt the priorities to its own setting (Nova Scotia Health Research Foundation 2002); another used it to commission a series of “state of the science” reviews in priority areas (Alberta Heritage Foundation for Medical Research 2003); and yet another used the model to define national priorities for population and public health (Institute of Population and Public Health 2002).

These listening exercises are a useful method to help set the agenda for user-driven research and to add to the priority-setting toolbox. They are not a substitute for, but more a complement to, the traditional investigator-initiated processes that set priorities more by default than by planning. These exercises can identify broad themes for which more research is needed and leave the investigators’ responses to creatively and more specifically fulfill the need for such evidence. In their attention to translating the views of users into the work of researchers, they clearly owe more to the interpretive than to the technical approach to setting priorities for health research. In addition, these exercises are an important step in the ongoing “linkage and exchange” between those who fund and conduct applied health services research and the stakeholders whom the research aims to influence. Other methods for promoting stakeholder involvement in research may be used, such as creating research teams that encompass stakeholders or are overseen by an advisory committee composed of both researchers and stakeholders. In this way, if it is deemed appropriate, stakeholders may influence not only the general research agenda but also the conduct of specific research projects.

The challenge now is to evaluate the impact of user-driven research on decision making, system organization, and service delivery. Do programs based on the listening approach produce research findings that will be useful for managers, professionals, policymakers, and users? Are stakeholders more likely to use research findings from a listening approach and, if so, how? Over and above the involvement of these stakeholders in setting the research agenda, research funders are realizing that adequate linkage and exchange opportunities during the whole spectrum of the research process are essential to facilitate evidence-based decision making.