A few panel members dominated guideline development meeting discussions: Social network analysis

Abstract

Objectives:

To identify patterns of interactions that may influence guideline panels’ decision-making.

Study Design and Setting:

Social network analysis (SNA) to describe the conversation network in a guideline development meeting in United States.

Results:

We analyzed one two-day guideline panel meeting that included 20 members who developed a guideline using the GRADE (Grading of Recommendations Assessment, Development and Evaluation) approach. The conversation pattern of the guideline panel indicated a well-connected network (density=0.59, clustering coefficient=0.82). GRADE topics on quality of evidence and benefits versus harms accounted for 46%; non-GRADE factors accounted for 30% of discussion. The chair, co-chair and methodologist initiated 53% and received 60% of all communications in the meeting; 42% of their communications occurred among themselves. SNA metrics (eigenvector, betweenness and closeness) indicated that these individuals also exerted highest influence on discussion, controlled information flow and were at the center of all communications. Members were more likely to continue previous discussion with the same individuals after both morning breaks (r=0.54, P<0.005; r=0.17, P=0.04), and after the last break on day 2 (r=0.44, P=0.015).

Conclusion:

Non-GRADE factors such as breaks, and the members’ roles, affect guideline development more than previously recognized. Collectively, the chair, co-chair and methodologist dominated the discussion.

1. Introduction

Despite major investments in standardizing the use of evidence for making recommendations and in providing clear instructions for developing trustworthy clinical practice guidelines, large variations in recommendations exist among guidelines that are developed based on similar evidence [1-4]. Critics charge that obscure process issues are responsible for these discrepancies; current guidelines systems including the leading evidence-based Grading, Assessment, Development and Evaluation (GRADE) approach, represent a “black-box” operation—a process with defined inputs and outputs but without sufficient understanding of its internal mechanisms [5]. Researchers who observed guideline panel meetings that apply the GRADE approach reported marked variation in the level of contribution by each panel member [6,7].

Existing work highlights that guideline development is not just a technical, simple mechanistic process. Rather, it is a human, social process involving relevant stakeholders who actively engage in discussion to formulate their judgements [8,9]. Compared to the development of standardized, formulaic processes for generating evidence-based, trustworthy guidelines [10], investigators have, however, devoted little attention to understanding these social aspects in guidelines decision-making [8]. This social aspect, also referred as the social interaction structure, pertains to the collaborative, dynamic interactions and conversations that occur in guideline panels [8,11].

Guideline panels can be characterized as a network of individuals with diverse backgrounds, qualifications and perspectives, in which information is exchanged and debates may occur. Some actors (panel members) may also contribute to conversations more than others [12,13]. This network involves the flow of information and ideas among individual actors. As is true of other sorts of meetings, the quality of meeting conversations in guideline panels is largely dependent upon the group composition [14], social and professional status of group members [15], small group processes (e.g., compliance, conformity, dominance of discussions) [16], and other extraneous factors, such as the availability of refreshment breaks [17]. Although these factors are not considered within instructions and manuals on how to develop guidelines, they have the potential to significantly impact judgments and decisions.

Despite its essential role in determining the final recommendations, little is currently known regarding the patterns of conversations that occur in guideline development panels. Existing studies do not describe the patterns (content discussed, level of contribution by each member, level of interaction) that occur throughout panel meetings. Examining these patterns can provide important information regarding the mechanisms of formulating judgments and decision-making, ultimately informing how panels make recommendations. We,therefore, undertook a study to identify the patterns of social interaction structures that influence guideline panels’ decision-making. We hypothesized that different actors, depending on their roles, the discussion topic, and different times during the meeting, contribute unevenly to conversations. We also explored whether panel deliberations significantly changed before and after breaks.

2. Methods

2.1. Design

Social network analysis (SNA) is a well-established method that focuses on the relational patterns between ‘actors’ (e.g., individuals, institutions, nations). SNA can be an effective method to visually display the complexity of the conversations and analyze the levels of interaction among panel members in guideline development meetings [18,19]. We used SNA to describe the structures and information exchange that characterize the conversation network in a guideline development meeting. Specifically, we applied SNA on qualitative codes derived from audio recordings.

2.2. Data collection

We collected data from a guideline panel meeting convened in November 2017 by the American Society of Hematology (ASH) in Washington, DC, United States, to develop recommendations for diagnosis and treatment of venous thromboembolism . The panel developed this guideline using the GRADE approach – a widely used evidence-based approach to guidelines development endorsed by over 110 professional organizations. The two-day guideline development meeting was audio-recorded and professionally transcribed. Panel members’ names were anonymized and assigned a number, the identity of the panel was concealed, and all mentioned institutions were de-identified.

2.3. Data analysis

We used inductive and deductive content analysis [20] to code the topics of each speaking turn. A speaking turn is a statement made by a panel member after the previous member finishes speaking and before the next member speaks. A speaking turn can be a word, a phrase, a sentence, or one or more paragraphs [21]. For the deductive analysis, we coded each speaking turn using the discussion topics and their definitions (Appendix Table 1) identified in guideline development meetings that used the GRADE EtD framework [22]. Next, we applied inductive content analysis by actively searching for recurrent concepts or patterns that were not listed in the GRADE EtD framework. This step included rereading the entire transcript without referring to the framework to capture as many additional codes as possible. At least two investigators read any one section of the transcript; new codes were discussed prior to formally including them as final list of codes.

Reviewers coded each speaking turn such that the initiator of that turn was identified as well as the target (for whom the statement is intended) [21]. It was not unusual for the initiator to speak to more than one individual, or to have more than one individual respond. Each speaking turn was coded separately. For example, if Chair (ID=1) addressed a question to Content Experts 2 and 5, each content expert had a line assigned to them (please see Appendix Table 2 for example). Using this who-was-talking-to-whom information, we constructed an overall communication network, and examined the patterns of those interactions among group members in the discussion process. Reviewers coded all the speaking turns in a matrix based on participants’ reactions to previous speaking turns [23]. We determined the frequency of speaking turns for each topic by dividing the total number of speaking turns per topic by the total number of speaking turns for all topics. We determined the proportion of contribution by each panel member by dividing the total number of speaking turns per member by the total number of speaking turns that occurred in the meeting. These descriptive summaries provided the relative proportion of topics discussed and the contribution of each panel member to the meeting discussions.

Four investigators (SAL, GT, QW, YZ) who have education backgrounds in health sciences applied the deductive and inductive content analyses. The most experienced researcher (SAL) developed the coding manual that included definitions and examples for each discussion topic, with instructions and samples on coding. To ensure that all coders understood and were in agreement with the codes, coders held three meetings at which, as a group, they coded three different segments (30 of 466 pages) of the transcripts. Coding was conducted independently and in duplicate; the most experienced researcher (SAL) in the coding team provided adjudications to all disagreements. We report results based on the final adjudicated codes.

2.4. Constructing conversation networks

Using the coded transcripts, we constructed the overall communication network for the two-day meeting. The nodes in the networks are the panel members in the meeting (see Fig 1 for overall network visualization; Appendix Figure 1 for network visualization separated by day). Directed links (lines between nodes) are formed, in which the arrow suggests the direction from the speaker to the target. Each link is assigned a weight, which equals to the frequency of communication between two members.

Fig 1.

Overall network visualization of the guideline development meeting (days 1 and 2 combined). For network visualizations separated by day, please refer to Appendix Figure 1. Abbreviations: CE=content expert; SRT=systematic review team member; PP=patient partner.

The interaction structure of the meeting was analyzed from the network visualizations and SNA metrics of the meeting discussion. Visualizations of the metrics were made with Stata software [24]. We calculated various SNA metrics (see Table 1 for definitions) to assess the network structure and position of panel members for both the overall conversation network and each discussion topic. In particular, we assessed the overall volume and strength of exchanges, influence and effect on information flow, and distance (degree of separation) among participants. We followed a mixed-methods approach to thematically analyze the patterns and structures [25]. Specifically, we developed narratives representing the overall conversation network and for each discussion topic. We also determined both network and individual centrality measures by panel member. To analyze how interactions occurred over the time, the meeting was separated by the assigned refreshment breaks. For instance, T1=time elapsed from beginning of meeting before first break; T2=time elapsed between end of first break to beginning of second break, etc (see Appendix Table 3 for details). We also used Quadratic Assignment Procedure (QAP) procedure [26] to test the null hypothesis that there was no correlation between interactions (ties) among nodes (participants) before versus after the breaks.

Table 1.

Definition of SNA metrics [27,28]

Metric	Definition
Density	The proportion of all possible relations that exist.
Reciprocity	The proportion of relations that are bi-directional. if A chooses B, then B chooses A.
Transitivity	Tendency of network actors to make direct connections with their indirect partners. The number of triangles over the number of paths of length two. If A is connected to B and B is connected to C (path of length is 2), a transitive triangle happens when a is also directly connected to C. The network with high proportion of transitive ties is considered a cohesive network (measures cohesiveness).
Indegree centralization	A proportion indicating the distribution of indegree centrality in the network. The number of ties received, typically useful measure to identify opinion leaders in the network, A centralization of 1 represents a star network (one central actor to which all other actors are connected)
Outdegree centralization	A proportion indicating the distribution of outdegree centrality in the network. Number of ties it sends; indicates to some extent the person’ sociality. A centralization of 1 represents a star network (one central actor who is the only sender of ties to all other actors).
Clustering coefficient	The average density of personal networks, indicating the connectedness of network. The clustering coefficient takes values between 0 and 1 where 1 indicates a fully connected network (e.g. each participant is connected to all the rest). Smaller values of clustering coefficient reveal a rather random pattern of connectivity.
In-degree and out-degree centrality	Number of incoming and outgoing relations for each panel member
(The normalized degree centrality measure varies from 0 to 1)
Closeness centrality	Measures the average distance a node has to all others in the network – shorter values mean greater ease of interaction with all others. The inverse is farness, which is the overall distance of each individual from all other members. At the network level it represents average closeness/farness/nearness of a node to all other nodes in a network
Betweenness centrality	The extent an individual mediates indirect relations among others; measures importance i.e. how important a participant is in connecting others /controlling information flow. At the network level, standardized betweenness indicate percentage of actors who are most influential in the network.
Eiegenvector	Measures the influence of a participant over the entire network not only those directly connected to the participants (by taking into account how well other participants are connected in the network). At the network level, it provides a proportion of high influential participants.
Average path (geodesic distance)	The (maximum/minimum) geodesic distance corresponds to the network’s diameter and it measures the maximum/minimum distance between any two participants in the network (measures distance between any two participants).

3. Results

3.1. Participant characteristics

In November 2017, 20 panel members participated in a two-day guideline development meeting, totaling 20 hours of discussion (466 single-spaced pages of transcription). The group consisted of a chair, co-chair, methodologist; 10 content experts (CEs), four systematic review team (SRTs) members (who conducted evidence synthesis), and three patient partners (PPs). Nine panel members (4 CEs, 3 SRT, 2 PPs) self-identified as female (45%) and the remaining members self-identified as male (chair, co-chair, methodologist; 6 CEs, 1 SRT, 1 PP). One patient partner and one systematic review team member remained silent throughout the meeting after introducing themselves; as such, they did not contribute to the analyses.

3.2. The structure of conversation network

Table 2 shows various structural metrics of overall conversation network, as well as different discussion topics. Please refer to Table 1 for definitions of the network metrics displayed in Table 2. The overall conversation guideline panel displayed a well-connected network, as indicated by the large density (0.59) and clustering coefficient of 0.82. On average, distance between any two participants was less than two steps apart. The reciprocity of 0.86 indicated that the majority of outgoing ties resulted in two-way conversations between pairs of panel members.

Table 2.

Structural metrics of the overall conversation networks and different topics

Theme	Density	Dyad-based Reciprocity	Transitivity	Indegree Centraliza tion	Outdegree Centraliza tion	Clustering Coefficient	Closeness	Betweenness Centrality	Eigenvector Value	Avg. Path
Overall	0.59	0.86	0.70	0.38	0.44	0.82	0.76	0.06	0.22	1.37
Quality of Evidence	0.42	0.73	0.64	0.48	0.42	0.65	0.66	0.28	0.22	1.57
Harm Vs. Benefits	0.41	0.71	0.57	0.56	0.50	0.71	0.67	0.20	0.22	1.54
Accessibility	0.33	0.56	0.36	0.58	0.39	0.30	0.63	0.46	0.34	1.67
Resource Use and Cost	0.31	0.67	0.53	0.74	0.51	0.56	0.63	0.43	0.23	1.63
Orientation/Instructions	0.29	0.73	0.50	0.57	0.51	0.56	0.60	0.32	0.21	1.71
Evidence Summary	0.28	0.55	0.43	0.39	0.46	0.37	0.58	0.22	0.23	1.79
Feasibility	0.26	0.47	0.46	0.63	0.55	0.44	0.62	0.40	0.24	1.65
Values and Preferences	0.25	0.56	0.34	0.45	0.54	0.51	0.57	0.38	0.25	1.81
Health Equity	0.23	0.50	0.39	0.75	0.41	0.40	0.58	0.49	0.24	1.77
Conflicts of Interest	0.26	0.44	0.31	0.46	0.46	0.29	0.60	0.35	0.29	1.71
Decision on Recommendation Wording	0.26	0.62	0.38	0.71	0.64	0.74	0.62	0.47	0.23	1.68
Clinical Experience	0.25	0.59	0.35	0.39	0.28	0.28	0.53	0.18	0.26	1.98
Healthcare System	0.18	0.30	0.43	0.22	0.22	0.15	0.45	0.29	0.31	2.31
Legal Implications	0.50	0.00		1.00	1.00	0.00	1.00		0.71	1.00

We also analyzed the conversation networks for each discussion topic that were extracted through the content analysis (for network maps by discussion topic, please see Appendix Figure 2). Table 3 showed narrative description of network patterns for each discussion topic; Appendix Table 4 displays individual centrality measures by panel member). Overall, topics on ‘quality of evidence’ and ‘benefits versus harms’ accounted for almost half (46%) of the conversations in the meeting (Fig 2). Discussion of these topics also involved the most panel members and have the greatest network densities (0.42 and 0.41, respectively). ‘Orientation/instructions’ topics, which include instructions on how to assess the evidence, orientation to the EtD framework, and other issues surrounding the scheduling of the meetings, accounted for 16% of the conversations. Discussions on ‘resource use and cost’ (11%) and ‘decision on recommendation and wording’ (10%) also proved prominent. GRADE factors accounted for 70% of total discussions.

Table 3.

Narrative description of network patterns for each discussion topic

Topic of discussion	Description of network
Orientation/Instructions**	This network involves all group members. It is highly centralized around chair, co-chair and methodologist. It is a dense network with high reciprocity (two-way conversations). However, the tendency to reach out to several others is less, as many conversations were directed at chair and co-chair. The systematic review team was also involved.
Accessibility*	Chair, Co-chair and Methodologist were involved in this conversation, which proved an ad-hoc discussion activated by Content Expert 5. Patient Partners were considerably involved and were addressed by Content Expert 5. Most content experts and the systematic review team were not involved.
Feasibility*	This network is mostly centralized around chair, co-chair, methodologist, and content expert 5.
Resource Use and Cost*	This is a centralized network, with Chair, co-chair, and Content Expert 5 at the center. It is also a reciprocated network, indicating two-way conversations with people involved. Most of the systematic review team were not involved.
Health Equity*	This is a limited conversation between a few individuals, mostly involving content expert 2, chair, co-chair, and methodologist.
Reporting of Evidence Summary*	This is a dense network, with involvement of most group members with the exception of the patient partners. It is less centralized, and more lateral conversations occurred. Systematic review team, chairs, co-chairs, methodologists and up to 5 content experts engaged in the discussion.
Clinical Experience**	This is limited, ad-hoc conversation occurred among a few individuals, mostly involving content expert 2, chair, and co-chair. Systematic review team members, patient partners, most panel members were not involved.
Harms versus Benefits*	All actors, except patient partners, were involved in these conversations. The network was centralized around chair, co-chair and methodologist, and surrounded by group members. The network is dense, highly reciprocated, and highly transitive.
Patient Values and Preferences*	Few members were involved in conversations regarding values and preferences. The central actors were the patient partners, the chair, and co-chair. The conversations were mostly unidirectional and less reciprocated.
Quality of Evidence*	The quality of Evidence network involves most participants, except the patient partners. It is relatively centralized around the chair, co-chair and methodologist, and a few group members. It is a dense network with high reciprocity (two-way conversations), with low fragmentation (most actors are reachable), and relatively high triangle closure (people reach out to various others).
Legal**	This topic received limited conversation.
Decision on Recommendation and Wording**	This is a centralized network, with chair, co-chair, methodologist, and content expert 5 at the center. It is also a reciprocated network, indicating two-way conversations with people involved. Neither the patient partners nor the systematic review team were involved.
Conflicts of Interest**	This is a limited, ad-hoc conversation occurred between a few individuals, mostly involving content expert 2, chair, co-chair and methodologist.
Healthcare System**	Most conversations happened between four content experts, with involvement of chair and co-chair. it appears to be an ad-hoc conversation.

^*GRADE factors

^**-non-GRADE factors

Fig 2.

Proportion of discussion by topic.

3.3. Role

Figure 3 displays the proportion of discussions by each group member. Combined, the chair, co-chair, methodologist, and one content expert (CE5) accounted for almost two-thirds (65%) of the conversations. They also had the highest indegree and outdegree values in all discussion types, reflecting their role as prominent network participants. This resulted in varying levels of network centralization, depending on the extent other members’ involvement, ranging from ‘health equity’ (indegree centralization: 0.75) and ‘’resource use and cost’ (indegree centralization: 0.74) as most centralized, to least centralized networks, ‘healthcare systems’ (in-degree centralization: 0.22), ‘clinical experience’ (indegree centralization: 0.39), and ‘evidence summary’ (indegree centralization: 0.39). Three of 10 content experts individually accounted for approximately 6% of the conversations. Patient partners accounted for 1% of discussions. For all discussion topics, at least one of chair, co-chair, and methodologist served as the central actor, suggesting that they either initiated conversations or others addressed questions to them.

Fig 3.

Proportion of discussion panel group member.

Note: CE=Content Expert; SRT=Systematic Review Team Member; PP=Patient Partner; Unkn=Unknown (to indicate utterances where the transcriptionist could not identify the content expert due to poor sound quality).

The chair, co-chair or methodologist initiated 53% (1691/3201 turns) and received 60% (1913/3201 turns) of communications, of which 42% (804/1913 turns) occurred amongst themselves and 58% (1103/1913 turns) with others; followed by 10 content experts (43% initiated and 35% received). Patient partners and systematic review team contributed to approximately 1% and 4% of communications, respectively. SNA metrics indicated that the conversation network was centralized towards a few panel members (chair, co-chair, and methodologist). They also had the highest closeness measure, which indicates that they were positioned to reach all participants at greater ease than other panelists. They also had the highest value of eigenvector, confirming that they were most influential participants. Likewise, chair, co-chair, and methodologist scored highest on betweenness, which shows their importance as key actors in controlling information flow in the panel discussion (see Appendix Table 4 showing individual panel members’ centrality measures).

3.4. Effect of breaks on panel discussion

To assess the effect of breaks on the subsequent discussion, we ran QAP correlation tests to compare the similarity of network patterns before and after breaks. We would expect that the same panel members would continue to be more engaged in the discussions before and after breaks, particularly when discussion of a specific issue was required to continue after the break. That is, we would expect to see significant correlation in discussion networks, indicating large overlap among conversation partners before and after breaks. Thus, if the correlation between two networks is 1, it indicates that panel members talked to the same individuals before and after the break. Out of 10 periods, statistically significant correlation was detected in 3 periods: moderate to small correlation after the first break on days 1 and 2 (r=0.54, P<0.005; r=0.17, P=0.04); and moderate correlation (r=0.44; P=0.015) after the last break at the end of day 2. The first two periods occurred at the beginning of each day when discussion was just about “warming up”, while the last break occurred at the end of the day (when the panel members intensified its effort to complete tasks assigned within allocated time). Appendix Figure 3 shows the effect of break on the subsequent discussion.

4. Discussion

Existing guideline development methods, particularly those using evidence-based methods such as GRADE, are normative instructions focusing on which factors the panels should take into consideration when developing recommendations. It typically employs the structured EtD framework [29], a linear-like process in which deliberations related to judgments about the GRADE factors are formulaically converted into strength of recommendations. Our study shows that the process is more complex and that meeting discussions among panel members eb and flow depending on the level of interaction among panel members, the discussion topic, and the timing of breaks.

Indeed, our results indicate an uneven level of engagement of different members and differences in the extent to which the panel attend to various topics. For example, ‘quality of evidence’ and ‘harms versus benefits’ involved the largest number of panel members in conversation, while, ‘health equity’, and ‘patient values and preferences’ received less attention and engaged fewer participants. In other words, each topic can be understood as a different network of interactions with its own social dynamics and influential actors. This is also reflected in the visualization of network maps and inspection of structural metrics by discussion type, which indicate that network connectivity, participation of different roles in conversations, centralization and distribution of conversations across panel members considerably varied between discussion types.

The panel devoted 70% of the discussion to GRADE versus 30% to non-GRADE topics. Over the last 15 years, GRADE has developed detailed methodology regarding how to develop trustworthy guidelines. That about a third of discussion is devoted to the factors not formally identified in the GRADE methodological papers calls for formal assessment of these factors in the guideline development process [30]. Among these non-GRADE factors, the lack of studies addressing the impact of chair, co-chair/methodologists on the way the panels develop their recommendations reflects as a striking knowledge gap.

In our study, three (15%) of the panel members (chair, co-chair, methodologist) accounted for half (52%) of the discussions. At least two of these three individuals were the central actors in all discussion topics, and all SNA centrality metrics confirmed the central role of chair, co-chair or methodologist in all discussion types, suggesting that their contributions had a large influence on the way the panel formulated their judgments, including guideline recommendations. The chair (and co-chair) can play an important role in ensuring equality and inclusiveness of participation that may enable better decision-making [31]; their close involvement likely ensured that GRADE instructions were followed. However, the dominant role of chair, co-chair and methodologist raises a question of the extent to which the final recommendations represent their own, or the entire panels’ judgments – did other panel members simply conform to judgements provided by these central actors [32]?

In addition, exogenous factors such as the scheduled breaks, may also have affected the way the panel engaged in deliberations. Panel members were more likely to continue the previous discussion with the same individuals after the morning break on day 1 and 2 (T1 and T2; T6 and T7), and for the last session on day 2 (T9 and T10). There were significant overlaps before and after breaks, but the correlation coefficients were small to moderate. While we have not collected data to gain more insight into the nature of this phenomenon, the literature suggests that breaks may work by replenishing members’ mental reserve [17]. Further analysis on the content of the conversation data may provide evidence to explain the observed phenomena.

Patient participation in guideline panels can influence the inclusion of patient-relevant topics, outcomes selection, and approaches to recommendation development [33]. It is thus worrisome that the patient partners contributed to only 1% of all discussions in the meeting, despite taking the role as one of the central actors for the values and preferences topic. Conversations pertaining to this topic had the least two-way interactions of all topics, indicating that members who participated in this discussion mainly acknowledged another’s comment, and did not further the discussion. Importantly, one patient partner remained silent throughout the entire panel meeting; this finding is concerning, given that patient partners were invited for articulating their experiences and providing critical insight to the recommendations being made. Patient partner input is valuable and ensures that recommendations are aligned with patient values and preferences. Our results are consistent with existing literature, which has documented the marginal contribution of patient partners during guideline development meetings [34,35]. We highly suggest guideline developers to apply published frameworks [36] or strategies [37] that enhance patient engagement in guideline development, and establish a plan on how to effectively engage patients throughout the deliberation process.

This study has limitations, most of which can lead to important future directions. Analyses were limited to one guideline panel; it is possible that panels that do not use the GRADE approach and/or do not apply the EtD framework can result in different network interaction structures. Future studies should consider comparing the network structures of multiple guideline panels to determine whether application of the GRADE approach and/or EtD frameworks contribute to different conversation patterns and contribution levels based on participants’ different roles. It will also be important to analyze data according to gender. In this study, the chair, co-chair, and methodologist, individuals who contributed the greatest to the meeting discussions, were men. Investigators may wish to examine the differences in the proportion of deliberation between male and female panel members, and the influence of gender on social processes in meetings. A recent systematic review identified six gender-related variables that play critical roles in the interactions that occur in meetings, including: individual gender, sex role orientation, gender composition, gender salience, contextual factors such as task type and organizational settings, and the construction of gender as a social concept. Gender greatly influences the meeting interactions and the amount of deliberation contributed by members [38]. Other factors including panel members’ professional backgrounds, pre-existing professional relationships between members, and self-perceived levels of expertise may also influence these meeting deliberations and are worthwhile to explore. In addition, analyzing non-verbal behaviours and the seating arrangements of these members may also give clue to reasons why certain panel members contributed more frequently than others. These types of investigations call for an analysis of videorecording of future panel meetings. Finally, analysis of orientation sessions and manuals given to members prior to the meeting may bring clarity to the findings of this study.

5. Conclusions

We found that non-GRADE factors play more important roles in the guideline development process than previously recognized. Our findings identified an important gap in the guidelines literature, directing our attention to the mechanisms of the inner, “black-box”, decision-making processes that are beyond the structured frameworks applied during guideline development. Specifically, we found that chair, co-chair or methodologist contributed to most of the discussions and that the engagement of the content experts ebbed and flowed throughout the meeting depending on the discussion topic. These uneven patterns in discussions are likely reflection of differences in judgments displayed as a function of the topic and, extraneous factors such as breaks and the time of day. These (non-GRADE) factors merit more attention in the guideline development and evaluation literature, and in optimizing the current methods for developing guidelines.

What is new?

• Non-GRADE factors play more important roles in the guideline development process than previously recognized.

• Social network analysis is an effective way to characterize the network of conversation patterns in a guideline development meeting.

• Examining these patterns can provide important information on the mechanisms of guidelines development decision-making.

• Chair, co-chair and methodologist contributed to most of the discussions. In fact, at least two of these three individuals were the central actors in all discussion topics. Further research should address the impact of these roles on guidelines development decision-making.

• Non-GRADE factors merit more attention in the guideline development and evaluation literature, and in optimizing the current methods for developing recommendations for clinical practice