Abstract
We designed an application to allow respondents to rate components of clinical guidelines on the Internet. Twenty-three invited experts completed the rating followed by a satisfaction survey using a 5-level Likert scale. The experts felt that Web data entry was convenient, acceptable and easily accessible. We conclude that Web-based Delphi rating for consensus development is a convenient and acceptable alternative to the traditional paper-based method.
Background
The Delphi technique, developed by the RAND Corporation in the 1950’s, is a research method that provides a means of assessing the judgments of a group of experts [1]. The technique involves multiple iterative rounds of anonymous responses to a questionnaire until either the opinions converge or until no further substantial change in the opinions can be elicited.
The traditional implementation of the Delphi technique using paper forms is inefficient and error-prone. Analysis of this data requires manual data entry and validation. Computerized techniques that could improve efficiency, however, have been criticized as inconvenient and difficult to use [1]. Our study evaluates the satisfaction of an expert panel with first round Delphi rating using the Internet.
Methods
We designed and implemented a Web-based rating and analysis tool for collecting expert opinion about essential components in clinical guidelines as part of a first round modified Delphi process. Utilizing the RAND/UCLA Appropriateness Method, we determined, for each rated item, the median rating and a disagreement index, defined as the IPR (Interpercentile Range) divided by the IPRAS (Interpercentile Range Adjusted for Symmetry) [1]. Subsequent iteration of ratings were completed at the Conference on Guidelines Standardization in New Haven, CT [2].
The Web application was implemented using Microsoft ASP (Active Server Pages) and the data were stored and analyzed in Microsoft Access. The application displayed items, their definitions and a 9-point rating scale. Following a password-protected login, participants were allowed to enter ratings as well as comments for every item. The application allowed navigation by pages or by jumping directly to a particular question. It also allowed saving session data for completion at a later date.
Members of an expert panel were invited to rate (on a scale from 1 to 9) the necessity of certain components of clinical guidelines. Using a standard Web browser, panelists entered first round ratings. Afterwards they completed a survey regarding their satisfaction with Web rating using a 5-level Likert scale.
Results
From their home bases in 11 states and three countries, all panelists (n = 23) completed the ratings and the Likert survey. All were “comfortable” using the Web. Eighteen had previously participated in a formal consensus development process. The panelists felt that Web data entry was convenient (median 4, interquartile range [IQR] 4.0 – 5.0), acceptable (median 4.5, IQR 4.0 – 5.0) and easily accessible (median 5, IQR 4.0 – 5.0). They were neutral on whether moving back and forth between questions was as convenient on the Web as on paper (median 3, IQR 2.3 – 4.0).
Conclusions
We conclude that Web-based Delphi rating for consensus development is a convenient and acceptable alternative to the traditional paper-based method. It is advantageous because it eliminates errors inherent in manual data entry and the need for data entry validation.
One limitation of our study is the fact that only the first round of the Delphi process was studied (due to conference logistics). Future studies will need to evaluate the convergence of opinion with successive iterations on the Web.