Real-time analysis of Delphi panel data to facilitate iterative rating within meetings

Abstract

Objective

The Modified Delphi panel process involves iterative ratings, often with lengthy lags between rounds to allow time for data analysis. We sought to develop, implement, and evaluate an approach for real-time data analysis and presentation to complete multiple rounds of ratings in a single panel meeting.

Results

We developed statistical code in Python to analyze Delphi panel ratings completed in Slido using a 9-point Likert scale. The code generates histograms of panel ratings. The median score is color-coded pink when formal disagreement criteria are met (regardless of median score), and when there is no disagreement, red for median scores of 1–3, yellow for 4–6, and green for 7–9. We implemented this approach for a Delphi panel on sepsis recovery that rated three sets of care practices. Round 1 rating, panel discussion, and round 2 rating all occurred in one day for each set of practices. Round 1 ratings were presented during discussion. This approach avoided “homework” outside meetings. All panelists either strongly agreed (57%) or agreed (43%) it was easy to understand the presentation of round 1 ratings. Most panelists either strongly agreed (50%) or agreed (21%) that they liked doing both rounds of rating during the meetings.

Introduction

The RAND Appropriateness Method is a Delphi technique used to identify when experts agree and disagree to facilitate a discussion. While some think of this as a “consensus method” it is not designed to yield consensus, but rather, to reduce disagreement, and is commonly used to develop healthcare quality indicators and clinical practice guidelines [1, 5]. Delphi studies often require multiple rounds of ratings and lengthy analysis periods between rounds of ratings [2]. Implementation of real-time analysis and immediate visual feedback of panel ratings may improve Delphi panel efficiency, but is not commonly done [3]. Here we describe the development, implementation, and evaluation of a process for rapid analysis and feedback on Delphi panel ratings.

Methods

Population

The value of the analytic process was assessed by members of a Delphi panel as part of the Optimizing Veteran Recovery from Sepsis (OVeR-Sepsis) study. The panel consisted of 15 multidisciplinary and multispecialty members including physicians, nurses, physical therapists, health care administrators, patient safety and quality experts, and lay members.

The Delphi panel process: The panel rated a total of 18 care practices designed to optimize a veteran’s recovery from sepsis. These care practices were divided into three groups consisting of 7 practices, 6 practices, and 5 practices. Each group of care practices was discussed at a separate virtual meeting, using Zoom, where the panel was initially provided a description of each practice. The panelists then independently rated each care practice using Slido, on a 9-point Likert scale on three dimensions: validity, importance, and resource-intensity. These first round ratings were than analyzed during a 20-minute break. After the break, the summary of the evidence behind each care practice was reviewed, the panel’s first round ratings were shared with the group, areas of disagreement were identified, and a discussion was held to identify why there was disagreement before a second round of voting.

Development of Python code to process Delphi panel ratings

We developed Python code to process the Delphi panel ratings made in Slido. Slido is an online polling software that can be integrated into PowerPoint [4]. Our Python statistical code is publicly available at https://github.com/CCMRPulmCritCare/Delphi-Panel-Method. The code has three core modules: data processing, statistical analysis, and data visualization. The Slido ratings are first downloaded in an Excel file. The data processing module reads in the Excel file; separates the data into multiple smaller datasets (one for each Delphi care practice that was graded); and passes these smaller datasets to the statistical analysis and data visualization modules. The statistical analysis module calculates the median panel rating and assesses whether disagreement criteria are met. Following RAND/Delphi methodology, disagreement was assessed using the InterPercentile Range Adjusted for Symmetry (IPRAS) [5]. This was developed because the Interpercentile Range (IPR), which is a commonly used statistical measure of dispersion of a distribution, provides different ranges for disagreement between ratings that are symmetrically or asymmetrically distributed around the median. The panel ratings are considered to be in disagreement if the IPR is greater than the IRPAS for each care practice [5]. The data visualization module creates PDF (portable document format) figures displaying the panel ratings as vertical histograms. The median score is color-coded pink when disagreement criteria are met (regardless of median score), otherwise they are coded red for a median score of 1–3, yellow for 4–6, or green for 7–9. The Python code uses regular expressions, or a sequence of characters that specifies a text matching pattern, to extract plot titles from the question text, which is included in the Excel output from Slido. The PDF figures are each exported with the title “version_[Number]_Plot_[number] where the version number refers to the run of the code for a given Slido input file. This avoids overwriting prior figures when rerunning the code.

Implementation of Python code to facilitate multiple rounds of Delphi panel ratings within a single panel meeting

We implemented the Python code during the Delphi expert panel. Initial ratings of care practices were completed in Slido. Panelists were provided a QR (quick response) code to access Slido via smart phone, as well as the Slido web address and meeting code to access Slido via internet browser. After initial panel ratings were completed, panel rating data were exported from Slido as an Excel document. The Slido download has a structured format where each Likert-scale response (1–9) was recorded in a separate row, resulting in nine rows per question regardless of the number of respondents.

Using Python, we processed responses by care practice. Each care practice had 27 rows of data: three questions (validity, importance, resource-intensity), each with nine response options. We generated a tri-panel figure for each care practice, output as a PDF, that contained horizontal bar charts of panel ratings for validity, importance, and resource intensity in the design described above.

Evaluation of the real-time analytic process

To assess the acceptability of the process, we surveyed panelists at the conclusion of the Delphi panel. The 6-question survey was deployed in Slido and asked about panelists’ experience, particularly related to completion of iterative ratings within the meetings. Anonymous responses were provided on a 5-point Likert scale.

Analysis

All statistical analyses of the assessment used simple descriptive statistics.

Results

For each of three Delphi panel meetings, we successfully downloaded initial panel ratings, analyzed them in Python, and created PDFs to visually display the initial ratings within the 20-minute break. Panel ratings and summary figure for an example care practice are presented in Fig. 1. The Python code generated PDFs in less than 1 s, and there were no technical difficulties encountered with data analysis or visual presentation during the Delphi panel meetings. The implementation of this method was technically feasible and completed within the allotted 20-minute break period. Among the 54 sets of ratings polls (3 domains for each of 18 practices), none met the formal criteria for disagreement using the IPRAS methodology for either round 1 or round 2. However, the IPRAS score did decrease for both validity (from 5.8 to 4.6) and importance (from 6.8 to 5.4) domains while remaining unchanged for resource intensity (from 4.6 to 4.6).

Fig. 1.

Rapid analysis and visual presentation of Delphi expert panel ratings. Panel A shows output from Slido with ratings for one care practice. Panel B show the summary tri-panel plots of the Delphi panel ratings for validity, importance, and resource-intensity for the care practice. These figures were displayed during group discussion and helped to focus discussion on care practices with formal disagreement or greater spread in rating. The numbers after each bar represent the number of panelists providing that rating. On the bottom, is the median score color-coded for disagreement (pink) and median ratings without disagreement (red for 1–3. Yellow for > 3 to < 7 and Green for ≥ 7)

Survey data on panelists’ experiences with the data visualization and iterative ratings within meetings are presented in Fig. 2. Of 15 panelists, 14 (93%) completed the feedback survey, with 57% strongly agreeing and 43% agreeing that it was easy to understand the presentation of round 1 ratings. Most panelists also either strongly agreed (50%) or agreed (21%) that they liked doing both rounds of rating during the meetings. However, one panelist strongly disagreed with this approach because of the time involved in waiting for every panelist to cast their vote.

Fig. 2.

Delphi panel participant perceptions of experience of rapid Delphi process. This figure displays the results of a panel survey done at the conclusion of the 3rd Delphi panel meeting to assess the acceptability of real-time data analysis, presentation, and completion of multiple rounds of rating within Delphi panel meetings

Discussion

We developed, implemented, and assessed the acceptability of an approach for real-time data analysis and presentation in order to complete multiple rounds of ratings within a single Delphi panel meeting focused on identifying and prioritizing best practices to support recovery from sepsis. We analyzed round 1 ratings within 20-minute break periods and used the results to inform panel discussions. This approach facilitated completion of round 1 and round 2 rating within single Delphi panel meetings. Panelists uniformly agreed the data visualization was helpful and most liked doing round 1 and round 2 rating within the same meeting. Our code could be replicated to be used in other RAND/UCLA Appropriateness Panels.

There are several advantages to real-time data analysis and visualization, as well as iterative rating within single Delphi panel meetings. First, because discussion and round 2 rating occur immediately after round 1 rating, panelists can readily remember the rationale for their round 1 rating, which may yield richer discussion. Second, by eliminating delays between sequential rounds of ratings, panelists can sustain momentum and engagement, which may lead to more efficient discussions. Third, when all ratings occur within meetings, panelist are not required to complete pre-meeting or post-meeting “homework”. This helped to ensure that all round 1 and round 2 ratings were completed and streamlined the timeline to final rating. The drawback of completing voting during the meeting is the idle time involved in waiting for other panelists to vote. Additionally, there may be certain circumstances where additional time between round 1 and round 2 voting is desirable. While most panelists in our study liked doing all voting within the meetings, one strongly disliked this approach due to the idle time.

Limitations

This study should be considered in the context of a few limitations. First, the statistical code in Python was developed specifically for use with Slido polling software output. We selected Slido over other polling platforms because it was already approved for use at our institution which allowed for polls to be integrated directly into PowerPoint slides. Second, our code was developed specifically for 9-item Likert scales, consistent with standard Delphi panel methodology. However, we recognize that alternative response options (e.g., 5-item Likert) may be used in practice. To address these two limitations, we have made our statistical code freely available so that it may be adapted for use with other polling software output or response structure. Third, our survey included responses from only 14 respondents, who may not generalize to other Delphi panelists. Further studies are needed to confirm the acceptibility of iterative rating within meetings.

Conclusion

Our approach for real-time data analysis and presentation was successfully implemented and allowed for completing multiple rounds of ratings within a Delphi panel meeting. The data visualization was universally liked by our Delphi panelists and iterative rating within meetings was acceptable to most panelists.

Author contributions

J.B.W. wrote the programming code and prepared the figures. J.B.W. and H.C.P. drafted the manuscript. C.K.H., T.W., H.C.P, J.S., and S.J.B. conducted the Delphi panel sessions and critically revised the manuscript. All authors contributed to study design and approved the manuscript for submission.

Funding

This work was supported by VA IIR 20–313 (Optimizing Veteran Recovery from Sepsis [OVeR-Sepsis]) from the United States (U.S.) Department of Veterans Affairs, Health Systems Research Service.

Data availability

Python statistical code is publicly available at GitHub (https://github.com/CCMRPulmCritCare/Delphi-Panel-Method).

Declarations

Ethics approval and consent to participate

The study adhered to the Declaration of Helsinki. It was reviewed and approved by the Ann Arbor VA Institutional Review Board with a waiver of written documentation of informed consent for participation in the Delphi panel. The waiver of written documentation of informed consent was approved in accordance with U.S. federal regulation § 46.117(c), which allows waiving the requirement to obtain a signed informed consent form when “the research presents no more than minimal risk of harm to subjects and involves no procedures for which written consent is normally required outside of the research context.”

Consent for publication

Not applicable.

Disclaimer

The views expressed are those of the authors and may not necessarily reflect the position of the U.S. Department of Veterans Affairs or the U.S. government.

Competing interests

The authors declare no competing interests.