Defining the Medical Library Association research agenda: methodology and final results from a consensus process

Abstract

Objective:

Using a group consensus methodology, the research sought to generate a list of the twelve to fifteen most important and answerable research questions in health sciences librarianship as part of a broader effort to implement the new Medical Library Association (MLA) research policy.

Methods:

The delphi method was used. The committee distributed a brief survey to all estimated 827 MLA leaders and 237 MLA Research Section members, requesting they submit what they considered to be the most important and answerable research questions facing the profession. The submitted questions were then subjected to 2 rounds of voting to produce a short list of top-ranked questions.

Results:

The survey produced 62 questions from 54 MLA leaders and MLA Research Section members, who responded from an estimated potential population of 1,064 targeted colleagues. These questions were considered by the process participants to be the most important and answerable research questions facing the profession. Through 2 rounds of voting, these 62 questions were reduced to the final 12 highest priority questions.

Conclusion:

The modified delphi method accomplished its desired survey and consensus goals. Future survey and consensus processes will be revised to generate more initial questions and to distill a larger number of ranked prioritized research questions.

Highlights

• The delphi method produced twelve top-ranked research questions to inform the Medical Library Association's current research agenda.

Implications

• The modified delphi method exhibited great versatility. It can be either replicated or adapted for generating ideas and for reaching group consensus in different contexts.

BACKGROUND

A research agenda assists a professional organization such as the Medical Library Association (MLA) in focusing its limited resources on investigating those research topics likely to be most valued by its members. For example, a research agenda might guide the organization's efforts to advocate for funding research on these topics by external agencies. Or the organization might secure the resources itself to fund research on these topics. On an individual member level, the research agenda can provide guidance to researchers who are trying to prioritize their own applied research projects.

The 2007 MLA research policy, The Research Imperative, recommends broad changes in the roles of research for medical librarians' professional practices. Grefsheim et al. have described the process and results that led to the 2007 MLA research policy [1]. As these authors underscore in their article, the new research policy features a pragmatic action plan with two specific recommendations listed in the first section:

• MLA will ask the MLA Research Section to create a forum for identifying research priorities in the field.

• MLA will ask the Research Section to recommend annually to the MLA Board of Directors an MLA research agenda that suggests research topics of highest priority to the association [2].

During 2007, the MLA Research Section charged the new Research Section Research Agenda Committee with implementing these two policy recommendations. Within a seven-week period during June and July 2008, the committee conducted a survey of MLA leaders and all members of the MLA Research Section to elicit the “most important and answerable research questions facing our profession.” Committee members then conducted two rounds of voting involving first the MLA Research Section Research Agenda Committee and next the MLA Research Section Executive Committee to provide results to the MLA Board of Directors by the July 31, 2008, deadline. This paper reports the full research protocol as well as the final results of the consensus methodology. It presents a detailed methodological report that provides the MLA membership with a transparent view of this process. It also enables others to replicate accurately or to adapt this modified delphi method.

METHODS

Definition of the protocol

The MLA Research Section's Executive Committee began discussing implementation of the two MLA research action plan recommendations at its 2006 meeting in Phoenix, while the MLA research policy was still undergoing its final revision. By autumn of 2007, the executive committee had formed the Research Agenda Committee and charged it to survey both the elected and appointed MLA leadership in order to formulate the beginnings of a research priorities list. The Research Section Executive Committee appointed three colleagues with strong research credentials (the authors of this article) to this new committee. MLA Research Section Chair Martha Harris, AHIP, and Chair-elect Susan Lessick, AHIP, consulted with the research agenda committee as well as with the MLA Board of Directors' liaison to the Task Force on MLA's Research Policy Statement, T. Scott Plutchak, AHIP, to refine the charge of the new committee during early 2008. During this time, Research Section Research Agenda Committee Chair Jonathan Eldredge, AHIP, also formulated a research protocol, which was approved by the MLA Research Section Executive Committee and the MLA Board of Directors. The chair then sought and received institutional review board (IRB) approval from the University of New Mexico Human Research Review Committee (number 08-234) to implement this protocol on June 5, 2008.

Review of major methods for reaching consensus

The nominal group technique and the delphi method are two well-established methods for reaching group consensus. The nominal group technique normally involves face-to-face groups with a facilitator who initially encourages all group members to generate as many ideas or statements as possible relevant to the agreed task [3,4]. This technique allows for creative ideas to emerge and for the facilitator to seek immediate clarification as needed. The structure and facilitation of this technique also reduces the likelihood that some group members might pursue an agenda at odds with a consensus view [5]. The nominal group technique facilitator leads the face-to-face group to reach eventual consensus.

The second major means to reach group consensus, the delphi method, allows generation of many diverse ideas or statements in a confidential and nonjudgmental environment. The delphi method can involve participants via either postal, web-based, or email communications. The delphi method consequently can incorporate geographically dispersed groups to work asynchronously but effectively, without using face-to-face group meetings.

The Rand Corporation originally developed the delphi method during the 1950s for panels of experts in a field to predict possible future scenarios. Delbecq and his colleagues attempted to summarize and inventory the many variations of the delphi method for multiple purposes and types of panels that had evolved by 1975. They characterized it in its classic form at that time as a method for bringing together via postal survey a group of interested individuals who would respond to a series of two to four questionnaires until they reached consensus.

The indirect modes of communication in the delphi process mean that, while respondents might be known to one another, their responses in the process remain anonymous to one another. In this manner, the delphi method prevents the forms of interpersonal domination by any persons that otherwise might occur in face-to-face meetings. The coordinators of the delphi method usually pose broad questions on the first survey and then later try to collapse the wide-reaching and even possibly tangential responses into a coherent set of statements in the second round of surveys for the panel to consider. The coordinators repeat the process iteratively for as many as four times until the panelists reach a rough consensus [3].

Adaptations to the classic delphi method

The delphi method continued to evolve to serve the goals of specific applications in many professions and subject disciplines. By 1987, Goodman attempted to characterize core attributes of the delphi method as “anonymity, iteration with controlled feedback, statistical group response, and expert input.” Goodman also mildly criticized the delphi method for its anonymity due to the potential for panelists to not take accountability for their positions. Goodman centered her critique, however, on the concept of “expertise” on delphi panels. She recommended instead that delphi method coordinators recruit for their panels “individuals who have knowledge of a particular topic and who are consequently willing to engage in discussion upon it without the potentially misleading title of ‘expert’” [6]. Similarly, Buck et al. agreed with Goodman on defining the same core characteristics of the delphi method but noted that they found no significant differences between the first and second rounds, which suggested considerable degrees of consistency of views in a panel across votes [7]. In relation to these multiple iterations of panel input, Keeney et al. have observed that a decline in response rate with each successive round of interactions can undermine the representativeness of a delphi process [8].

Alignment of the delphi method with project goals

In discussing the delphi method, Keeney et al. have observed that “flexibility exists in the design and format of the technique and this often depends on the study's aims and objectives” [9]. Indeed, any application of a specific research design must be aligned with the goals of the investigation to remain consistent with the principles of current social science and behavioral research [10–12]. The committee's adaptation of the delphi method evolved to accommodate a number of considerations as well as reported findings in the literature on previous efforts to utilize the delphi method.

Composing a delphi panel of researchers for this project presented two challenges: first, researchers might gravitate toward their existing research areas because these areas would be paramount in their minds; second, it would be difficult to identify a comprehensive list of contemporary health sciences librarian researchers within the limited time frame of the project. A panel of active researchers would, however, be best able to identify those research questions that would be “answerable” via recognized research methods, so their input would be essential to the project. A delphi panel composed of MLA leaders would represent the views of their respective chapters, sections, or subject area interests in MLA. Additionally, these leaders most likely would be aware of contemporary and emerging trends in health sciences librarianship and informatics that might signal areas worthy of research investigations. Using a delphi panel of MLA leaders for successive rounds of voting did pose the risk that panel members would gravitate to their affiliated subject interests, chapters, or sections.

The committee elected to poll MLA leaders on their answers to a single question about what they viewed as the most important research question facing the profession. These questions then would be reviewed and voted on by two different panels originating in the MLA Research Section for the second and third delphi method iterations. The committee assumed that the members of these two panels would be likely to understand how and if the varied research questions submitted might be potentially “answered” by the diverse array of both quantitative and qualitative research methods serving librarianship. These research-oriented MLA members also would either be representative of or be likely to be associated with individuals who would actually attempt to investigate the answers to the highest ranked research questions. Thus, they would have a potentially high stake in what questions constituted the final list of research questions.

Identification and polling of Medical Library Association leaders

In consultation with current Research Section Chair Lessick, the committee defined “MLA leaders” for purposes of the survey as all elected association-wide MLA officers, all appointed association-wide MLA committee members, all elected MLA section and chapter officers, and all MLA section and chapter committee chairs. The committee estimated that this leadership survey population consisted of 827 positions with an indeterminate number of individuals filling multiple leadership roles in MLA. The chair was able to secure from MLA headquarters access to email distribution or discussion lists for most of these MLA leaders. In addition, MLA staff members forwarded the committee's invitation to participate in the survey via staff email address books to the remaining MLA leaders. The committee also distributed the survey to the 237 members belonging to the MLA Research Section email discussion list.

The chair distributed the survey to all 827 MLA leaders and 237 Research Section members through these various email channels on June 5, 2008 (Figure 1). The survey consisted of the single open-ended question, “What is the most important and answerable research question facing our profession?” The survey purposely included no criteria for answering this one question to encourage more spontaneous responses. Some intended recipients might not have received these invitations due to Internet security firewalls at their institutions. The committee also had to rely on others to distribute the survey. For example, certain categories of MLA leaders could only be reached through intermediaries such as section, chapter, and convening chairs. The chair emailed the survey again on June 19, 2008, to all intended MLA leader recipients with a reminder note to complete the survey before the June 22 deadline.

Figure 1.

The research agenda process

The committee managed the question collection phase in a confidential manner. No respondents knew the identities of each other when responding to the broad, open-ended question, “What is the most important and answerable research question facing our profession?” The chair received a total of sixty-two questions and removed any clues to the identities of survey respondents before relaying them to the other two committee members, who examined the format and wording of the questions. Although the survey asked for a single question, five participants submitted two questions and one participant submitted four questions. While forty-two questions were clearly stated and answerable in their original form, twenty questions contained ambiguity needing clarification. After examining these twenty ambiguous questions, the two other committee members asked the chair to query the survey respondents to clarify the meaning or to accept a revised wording of these outstanding questions. The chair then provided the two committee members with the de-identified, clarified versions of questions emailed to him by the original respondents. All sixty-two questions in either original or clarified form appeared on the ballot for the first round of voting as described below. The chair compiled all of the demographic data displayed in Table 1 and then destroyed the original emails and any documents containing the identities of survey respondents to maintain confidentiality.

Table 1.

Demographics of survey respondents*

The post-survey phase of this project included two rounds of voting to produce the final short list of highest ranked research questions for submission to the MLA Board. The chair randomized all sixty-two submitted questions from the order received into a new random order. The first round of voting during July 10 to 11, 2008, involved just the three committee members who cast up to ten votes each for the most important and answerable questions in a process in which all participants knew each other's votes, so this one phase did not offer anonymity to the participants. This process produced a refined list of twenty-two questions, retaining all questions receiving at least one vote. The chair again randomized these twenty-two questions into a new list. The second round of voting, held July 11 to 25, 2008, invited the seventeen eligible members of the MLA Research Section Executive Committee, consisting of both elected officers and committee chairs, to cast up to five votes. These votes cast by fifteen respondents produced the final list of twelve research questions, which had received at least three votes. Two members of the committee reviewed the votes and then independently certified this round of voting on July 30, 2008. The committee chair emailed the final twelve questions to both the Research Section chair and MLA Board liaison on July 31, 2008.

RESULTS

The survey phase of the project generated sixty-two questions submitted by fifty-four MLA leaders. Table 1 provides the descriptive demographic data on the sources for each question, using the questions themselves as the unit of analysis. The first round of voting by the committee on the sixty-two questions produced a list of twenty-two high-priority questions. The second round of voting by members of the MLA Research Section Executive Committee produced a list of twelve top-ranked questions. Committee members noted that although these rounds of voting remarkably did not produce many closely competitive votes between questions, the votes favored certain clusters of questions within each round of voting. While this pattern led the committee to conduct only two rounds of voting, more questions and less agreement resulting from the two rounds of voting would have led to additional rounds of voting. Table 2 lists the top ranked research questions for MLA resulting from this modified delphi method.

Table 2.

Final list of top twelve research questions

DISCUSSION

The delphi method enabled the MLA Research Section's Research Agenda Committee during June and July 2008 to generate sixty-two diverse research questions and then to facilitate the rank prioritization of these questions into the final list of twelve highest ranked questions by July 31, 2008, for the MLA Board of Directors. Because the rounds of voting proceeded smoothly without delays, the committee was able to meet the MLA Board of Director's deadline. Table 2 lists the top twelve questions. The entire list of sixty-two questions appears in the summer 2008 issue of Hypothesis [13].

This article presents the project protocol in a transparent manner for review and replication. This methodology can be replicated by other types of librarians (public, academic, special, etc.) both in the United States and overseas, which could facilitate comparisons across librarian specialties as well as transnational locations in the future. Such comparisons could potentially identify common research questions for possible international collaboration.

LIMITATIONS

The committee was able to meet its project goals and deadlines for the MLA Board of Directors. Committee members agree that they would have attained more survey and ballot responses if they had been able to select a different season for conducting the survey, bypassed the delays in the ethics review procedures, and used more thorough, albeit time-consuming, alternative survey deployments. This section reviews these potential limitations and presents the committee's recommendations for improving future survey and voting processes.

Seasonality

This delphi method process occurred during the summer months of June and July 2008, when many people in the United States take vacation time off from work. The June 5 survey might have reached more MLA leaders had it been conducted during the academic year. In the future, the committee recommends conducting the survey outside either the summer vacation period or the busy month leading up to the MLA annual meeting.

Ethics review

The first author had extensive experience working with the IRB, so this review and approval process went smoothly, except that unexpected delays occurred due to a broad campus reorganization of the IRB that began in mid-April 2008 and took several months to accomplish [14]. Consequently, following IRB approval, the committee had only a brief seven-week time frame to meet the July 31 deadline. The seasonality of the process and the brief process period limited the validity of this study due to an apparent low participation rate. The authors suggest that at least ten weeks should be apportioned in any research study to complete an initial ethics review and approval process.

Alternative survey deployments

The committee chair relied on existing email discussion lists, email distribution lists, and MLA staffers' email address books to distribute the survey. To increase the participation rate in the future, the committee recommends using a more direct method for reaching MLA leaders. At least two approaches could accomplish this goal. First, the MLA-FOCUS email newsletter could carry an announcement asking all MLA leaders to complete the brief two-minute survey. The leaders could list their MLA leadership roles for demographic tracking purposes along with their response (i.e., question) because the survey offers confidential, but not anonymous participation. This survey is intentionally confidential, so that the committee can seek clarification on the meaning of any unclear questions and screen multiple questions from individuals. Respondents' leadership status can be verified on the MLANET Members Directory. A second, more tedious approach would require a member of the committee to compile a list of all MLA members meeting the aforementioned leadership criteria and to email these colleagues directly. Representatives of MLA section and chapter councils might be able to assist with this second option in either the compilation of email lists and addresses or in the distribution of surveys to elected or appointed members in sections and chapters. This second, more personal mode of contacting leaders might yield a higher response rate.

RECOMMENDATIONS

The committee will need to deploy future processes to identify the most important and answerable research questions facing the profession. In this section, the authors offer their recommendations for improving the process or altering the process to incorporate different perspectives or varied emphases to some existing perspectives. The delphi method design currently exhibits great variability in its application in many subject fields and professions [15]. The delphi method usually possesses the core characteristics of anonymity, iteration with controlled feedback, statistical group response (for example, in this study, frequency of votes), and expert input [6,7]. Many of the delphi method studies reported in the library literature reflect these core characteristics, while some reflect variations on the delphi methodology [15–22]. This diversity allows researchers to adapt reasonable variations to those protocols used in this study, if they are consistent with the goals of the specific project.

Alternative panel composition

Time did not permit the committee to identify and recruit a core cadre of researchers in health sciences librarianship to select the “answerable” questions. In the future, such a panel might be composed of authors of original research articles published in the Journal of the Medical Library Association (JMLA), Health and Information Libraries Journal, Medical Reference Services Quarterly, Evidence Based Library and Information Practice, and possibly other key journals relevant to health sciences librarians. In this alternate protocol, MLA leaders would generate the questions, and the panel of researchers then could vote on them.

Volume of questions

The committee received sixty-two questions from fifty-four MLA leaders as part of this delphi method study. In the future, the committee plans to restrict each MLA leader to submitting only one focused question that does not exceed seventy words. This limitation will help avoid unnecessarily verbose, multipart, or rambling questions encompassing several unconnected subjects that would hinder subsequent review and voting. The committee has been considering not attempting to clarify the meaning of any submitted question because unclear questions would likely be removed in successive voting rounds, as happened during this application of the delphi method.

Even with these improvements, the committee still could receive 300–500 questions from MLA leaders. Fortunately, the delphi method would still prove resilient in the face of such a challenge. The delphi method usually preserves anonymity among participants, so no one participant should feel compelled to engage in the kinds of compulsory, cooperative, or competitive coalition behavior observed in other types of decision-making environments [23–27]. An option under these circumstances would be for the committee to invite only the same 300–500 MLA leaders who had participated in submitting in the original questions to vote for up to 2 questions other than the 1 that they had submitted originally. The participants would be anonymous to one another, but the committee would know the identities of all voters, so the committee could enforce the rule of not voting for one's own question. This prohibition on voting for one's own question would eliminate a scenario in which most MLA leaders might vote for their own question, thereby not reducing the numbers of eligible questions for further prioritization. At the same time, the diverse views and interests represented by the MLA leaders would reduce the probability that all participants would vote for the same question or even a handful of questions in common during the 1 round of voting. This single round of votes, or perhaps a second round of votes, most likely would produce a range of approximately 35–65 questions. This smaller number of prioritized questions could then be turned over to the panel of researchers (however composed, as discussed in the previous “Alternate Panel Composition” section) to review and cast their own votes. The researchers on this panel would be able to identify and vote for those questions that would be most “answerable” using current research designs in this final phase of the process.

This unlikely scenario reflects the flexibility of the delphi method to manage even large numbers of MLA leaders participating in the process. The committee will need to experiment in the future in any fine-tuning of the process. Regardless of the eventual process decided on, the committee would need to avoid testing the patience of any collection of voters by conducting more than two rounds of votes by any one group. Such repetitive cycles in the process would risk losing participant representativeness through attrition with each successive round of votes. The committee and the Research Section leadership did not want to tax the patience of the voters with a repeated protocol during 2008, so, following the initial MLA leaders' nominations of questions, the voting involved single rounds each by the committee membership and then the Research Section Executive Committee membership, sequentially.

Frequency

The 2003 JMLA readership survey indicated support for research articles by the readers [28,29]. This support reflected JMLA readers' wish to use research evidence to make decisions in their work. Readers' high interest in research suggests a need to renew this survey process regularly. Even with their eventual rotation out of leadership positions, would MLA leaders wish to be involved so closely with formulating research questions on an annual basis, however? Or should the survey component of the delphi method be conducted every other year? The committee recommends that the survey be deployed annually for the next two years and that the committee only accept those questions submitted anew by MLA leaders to reflect the fresh ideas of the participants. Thus, the committee would not recycle questions retained from the last survey. If identical questions were submitted anew during subsequent annual surveys, however, these questions would be included again. The committee should assess the overall process and resulting trends after two years to determine the optimal survey frequency.

CONCLUSION

The modified delphi method enabled the committee to survey MLA leaders and members of the Research Section, followed by a consensus process for priority ranking of the twelve most important research questions. The initial survey produced sixty-two questions, which were ranked through the delphi method through two rounds of successive votes, first by the MLA Research Section Research Agenda Committee and then by the MLA Research Section Executive Committee, for consensus to attain the final list. The committee's process nevertheless can be improved for future applications of the delphi method. The final twelve questions derived from the process, however, met the goal of providing specific areas for research related to the MLA research agenda.