Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Feb 1.
Published in final edited form as: J Clin Epidemiol. 2015 Sep 25;70:164–175. doi: 10.1016/j.jclinepi.2015.08.023

Social network analysis identified central outcomes for core outcome sets using systematic reviews of HIV/AIDS

Ian J Saldanha a, Tianjing Li b, Cui Yang c, Cesar Ugarte-Gil d, George W Rutherford e, Kay Dickersin f
PMCID: PMC4733392  NIHMSID: NIHMS725728  PMID: 26408357

Abstract

Objective

Methods to develop core outcome sets, the minimum outcomes that should be measured in research in a topic area, vary. We applied social network analysis methods to understand outcome co-occurrence patterns in HIV/AIDS systematic reviews, and identify outcomes central to the network of outcomes in HIV/AIDS.

Study Design and Setting

We examined all Cochrane reviews of HIV/AIDS as of June 2013. We defined a tie as two outcomes (nodes) co-occurring in ≥2 reviews. To identify central outcomes, we used normalized node betweenness centrality (nNBC) (the extent to which connections between other outcomes in a network rely on that outcome as an intermediary). We conducted a subgroup analysis by HIV/AIDS intervention type (i.e., clinical management, biomedical prevention, behavioral prevention, and health services).

Results

The 140 included reviews examined 1140 outcomes, 294 of which were unique. The most central outcome overall was all-cause mortality (nNBC=23.9). The most central and most frequent outcomes differed overall and within subgroups. For example, ‘adverse events (specified)’ was among the most central but not among the most frequent outcomes, overall.

Conclusion

Social network analysis methods are a novel application to identify central outcomes, which provides additional information potentially useful for developing core outcome sets.

Keywords: outcomes, core outcome sets, social network analysis, systematic reviews, randomized controlled trials, HIV/AIDS

1. Introduction

In clinical research, an outcome is an event or measure in study participants that is used to assess the effectiveness and/or safety of the intervention being studied.[1] The outcome name (e.g., “anxiety”, “death”), is formally called the outcome domain.[2, 3] In this paper, when we say “outcome”, we are referring to the outcome domain. Researchers typically examine multiple outcomes, and the number of outcomes they examine varies widely. For example, clinical trials have been reported as examining between 1 and 71 primary outcomes and between 0 and 122 secondary outcomes.[2] This variation creates inconsistent outcome reporting, threatening credible evidence synthesis because when an outcome is reported for one trial but not another, it is impossible to compare or synthesize results across trials, for example, in a meta-analysis.

The various outcomes examined within a group of related trials or systematic reviews constitute a network of outcomes. For example, a trial of statins for hypercholesterolemia might examine serum cholesterol concentration and stroke. Some less frequently-examined, but not less relevant, outcomes might be cost-effectiveness and quality-of-life. All these outcomes might be examined within trials or reviews, but all would not typically be examined within a single trial or review. For example, a trial addressing mother-to-child transmission of HIV might examine a cluster of outcomes such as the child’s acquisition of HIV, premature delivery, neonatal morbidity, and neonatal mortality, outcomes that could also be examined in other studies on the topic. Trialists and systematic reviewers alike would want relevant outcomes in that cluster to be reported for all studies on the topic, yet this clustering may not be revealed by consensus, survey, literature review, or other methods of deciding on important outcomes.

When a single study examines multiple outcomes, outcomes (or nodes) are said to co-occur. Within a topic area, understanding the underlying outcome co-occurrence patterns in existing research could inform the development of core outcome sets. A core outcome set refers to the minimum outcomes that should be examined in all trials addressing a specific condition.[4] Core outcome sets serve two main purposes: they (1) facilitate decision-making by patients, clinicians, healthcare payers, and guideline developers by promoting consistency in outcomes examined in studies, and (2) reduce the potential for selective reporting of an outcome purely on the basis of results.[37]

Social network analysis, the study of graphs as representation of relationships and patterns of interaction among nodes (in our case, outcomes) within a network,[8] provides methodological tools to understand outcome co-occurrence patterns. To our knowledge, social network analysis methods have not been applied to analyze outcome co-occurrence patterns or to identify potential outcomes for core outcome sets. However, social network analysis methods, with their basis in network theory,[9] have increasingly been applied to other health-related research, including evaluating collaboration among researchers;[1012] evaluating scholarly citation patterns in Alzheimer’s disease;[13] and evaluating the associations between personal relationships and happiness,[14] depression,[15] food choices,[16] physical activity,[17] alcohol use,[18] marijuana use,[19] and smoking.[2022]

Understanding the affinity (or repulsion) between certain outcomes that results in their co-occurrence (or not) would help identify outcomes central to a network of outcomes (i.e., important to the connectedness of other outcomes) in trials or reviews. In a network of persons, for example, centrality would describe the most influential person in a network. Centrality considers which outcomes and groupings of outcomes have been important to researchers. Betweenness centrality is calculated for each node, and indicates the proportion of times, out of the maximum possible, that the outcome (node) occurs in the shortest path between two other outcomes (or nodes). We chose betweenness centrality because it best describes how certain nodes are intermediaries, or important in connecting other nodes in a network.[9]

The Core Outcome Measures for Effectiveness Trials (COMET) Initiative suggests that core outcome sets be developed first by identifying potential outcomes and then establishing consensus among stakeholders.[5] Current approaches to developing core outcome sets vary,[23] although frequency of outcome occurrence in research studies provides key information. In the current study, using Cochrane reviews, we examine whether social network analysis would provide new information on the centrality of outcomes that have been examined in a topic area. Reviews synthesize multiple trials, and directly inform clinical practice guidelines and healthcare policy. Whereas trial outcomes generally relate to patient care and may be constrained by practical considerations, such as costs of measurement and study power,[3] an additional set of issues, such as the usefulness of outcomes to decision-makers, is likely to influence systematic reviewers. Thus, reviews may include both clinical and policy-relevant outcomes.

2. Objectives

We used social network analysis methods to: (1) understand outcome co-occurrence patterns in Cochrane reviews of HIV/AIDS; and (2) identify outcomes that are central to the network of outcomes examined in Cochrane reviews of HIV/AIDS.

3. Methods

3.1 Choice of topic area and reviews

We selected HIV/AIDS for this study because of the variety of interventions and outcomes inherent to the topic area. This variety lends itself nicely to understanding outcome co-occurrence patterns. In addition, a recent review [23] identified only one core outcome set for HIV/AIDS, specifically for interventions addressing mother-to-child transmission among breastfeeding mothers.[24]

All Cochrane reviews published by the Cochrane Review Group on HIV/AIDS in the Cochrane Database of Systematic Reviews as of June 30, 2013 (Issue 6) were eligible. For completed reviews, we included the most recent version. For ongoing reviews, we included the protocol.

3.2 Data extraction

We designed a data extraction form using Google Forms©. One investigator (IJS) extracted the following for each review: status (protocol or completed review), year of publication, and type(s) of interventions assessed. Two investigators (IJS and CUG) independently extracted information about all examined outcomes, as stated in each review. We resolved discrepancies through discussion.

3.3 Categorization of outcomes

Two investigators (IJS and CUG) independently categorized each outcome into one of 14 categories: clinical/biological, behavioral, mental/social, antiretroviral prophylaxis/treatment, health services access/uptake, knowledge, testing/counseling, adverse effects, preference/satisfaction, attitudes, economic, quality-of-life, adherence, and miscellaneous. We revised the categorization system during data extraction, as needed, and we finalized it before we began the social network analysis.

3.4 Social network analysis methods

While the steps of social network analysis might differ based on the objectives of the analysis and the topic area, our social network analysis comprised five steps.

3.4.1 Step 1: Defining the network structure

We defined the network structure using nodes, ties, and isolates (Box 1). We did not include all possible outcomes in our network. Instead, we defined nodes as outcomes that occurred in more than one review, thus reducing the number of nodes in the graphs. We considered two outcomes as tied if they co-occurred in two or more reviews. We generated an adjacency matrix detailing whether each pair of outcomes was tied.[28, 29]

Box 1. Explanation for relevant social network analysis concepts and statistics.
Concept/Statistic Explanation
Concepts related to structure
Node Each individual actor in a network.
Tie or Edge Underlying connection between individual nodes, represented by a line connecting two nodes on a graph.
Isolate A node that is not tied to any other node in a network.

Statistics related to cohesion
Density The number of ties in a network as a fraction of the total number of ties possible.[25]
Component A subset of nodes that are all tied directly or indirectly, but have no ties to nodes outside that subset.[10]

Statistics related to centrality
Geodesic distance The shortest path linking a given pair of nodes in a network via intermediate nodes.[8] Note that there may be multiple geodesic distances between a pair of nodes.
Node betweenness centrality (NBC) This statistic is calculated for each node in the network. First, consider all geodesic distances connecting nodes ‘j’ and ‘k’ in an entire network (gjk). Next, consider the proportion of those distances that pass through a certain other node ‘i’ (gjk(i)). For node ‘i’, its NBC refers to that proportion, summed across all pairs of nodes in the network.[9, 33] NBC for node ‘i’ is computed as follows:[9, 26]
NBCi=j<kgjk(i)gjk
where:
  • gjk = number of geodesics between nodes ‘j’ and ‘k’

  • gjk(i) = number of geodesics between nodes ‘j’ and ‘k’ that pass through node ‘i’


Therefore, for a given node, its NBC reflects the extent to which connections between the other nodes in the network rely on that node as an intermediary.[10, 27]
Normalized node betweenness centrality (nNBC) This statistic is also calculated for each node in the network. The nNBC is a normalized version of the node betweenness centrality, and is calculated by dividing the node betweenness centrality by the number of pairs of nodes in a network.[10]
Network betweenness centralization or global betweenness centralization (“Centralization”) This statistic applies to the network as a whole. Centralization measures the extent to which the network is centered around the most central node (i.e., the node with the highest NBC).[9, 10] Centralization is computed as follows:[9, 26]
Centralization=i=1g[NBC-NBCi]maxi=1g[NBC-NBCi]
where:
  • NBC* = largest individual NBC in the network

  • NBCi = NBC of node i


Therefore, the centralization simply is the sum of the difference in NBC between the most central node in the network and each of the other nodes, normalized by the maximum possible sum of the differences had the network been one where all connections depended on a single node.[26]

3.4.2 Step 2: Graphing the network

We imported the adjacency matrix into UCINET 6,[30] and used NetDraw [31] to produce the graphs, applying principles outlined by Freeman.[32] First, we employed the spring embedding layout, a systematic approach that positions nodes by balancing the attraction of closer nodes (outcomes that co-occurred) with the repulsion of distant nodes (outcomes that did not co-occur).[32] To simplify graph visualization, we did not depict isolated outcomes (i.e., outcomes that did not co-occur with any other outcome in two or more systematic reviews). Next, once nodes were positioned, we incorporated node attributes.[32] Using color coding, we incorporated the main category we had applied to each outcome as that outcome’s attribute.

3.4.3 Step 3: Calculating network descriptive statistics

The third step of our social network analysis was to use statistics to describe the observed network.

3.4.3.1 Statistics related to cohesion (or connectedness) (Box 1)
3.4.3.1.1 Density

We interpreted density to be the observed outcome co-occurrences as a fraction of all possible co-occurrences. Densities of zero and one would mean that no and all possible co-occurrences were observed, respectively.

3.4.3.1.2 Component

Multiple components would suggest that there are silos of outcomes that co-occur together, but do not co-occur with outcomes outside the silo.

3.4.3.2 Statistics related to centrality (extent to which a network depends on intermediary nodes) (Box 1)

Central outcomes connect other outcomes in a network. Connectivity of outcomes is important because core outcome sets should contain outcomes that represent various categories of outcomes; outcomes that contribute to this connectivity might be of increased importance for core outcome sets.

3.4.3.2.1 Node betweenness centrality (NBC)

In our analysis, a high NBC for an outcome would suggest that the outcome is central to the network. The normalized version of this statistic allows the NBC of nodes to be compared across different networks.[9] We calculated normalized node betweenness centrality (nNBC) for each outcome in our overall network and within sub-networks by type of intervention (step 5). For each network in our analysis, we considered the seven outcomes with the highest nNBCs to be the central outcomes of that network. We chose seven because Cochrane recommends including no more than seven main outcomes in reviews.[34] We depicted differences in nNBC across outcomes graphically by resizing each node as proportional to its nNBC.

In order to compare central outcomes identified using betweenness and degree centralities, we also computed each outcome’s normalized node degree centrality.

3.4.3.2.2 Network betweenness centralization or global betweenness centralization (“centralization”)

Unlike NBC and nNBC which are computed for each node in a network, centralization is computed for the entire network. Centralization scores range from zero to one; those close to zero suggest that no single node pervades the network, and those close to one suggest that one node pervades the network.

3.4.4 Step 4: Sensitivity analysis

We subjected our results to sensitivity analysis, by redefining a tie to be co-occurrence of outcomes in three or more reviews, rather than two. A cutoff of one or more reviews meant including too many isolated outcomes, and a cutoff of four or more reviews meant including too few outcomes to meaningfully visualize co-occurrence patterns.

3.4.5 Step 5: Subgroup analysis

We examined our results within sub-networks (or subgroups) of the entire network. We used the Cochrane Review Group on HIV/AIDS’s system for classifying its reviews by intervention type: (a) therapeutics, prognostics, and diagnostics (“clinical management”); (b) biomedical prevention (“biomedical prevention”); (c) behavioral, social, and policy prevention (“behavioral prevention”); and (d) organization and financing of health services and care (“health services”).

To evaluate whether social network analysis provides more information than a simple frequency analysis of outcomes examined in our sample of reviews, we compared, overall and for each sub-network, the seven most central to the seven most frequent outcomes (i.e., outcomes examined in the highest percentage of reviews).

We analyzed descriptive statistics using STATA© version 12 (College Station, TX).

4. Results

4.1 Characteristics of reviews and outcomes

We identified 140 eligible reviews published in 2008 or later, of which 99 (70.7%) were completed (Table 1). Almost half belonged to the intervention subgroup of clinical management (69/140; 49.3%). Box 2 lists examples of interventions assessed by subgroup. The 140 reviews examined a median of 7 outcomes each (interquartile range [IQR] 4–11, range 1–30). Across all 140 reviews, 1140 outcomes were examined, 294 of which were unique. The most frequently-examined outcomes were all-cause mortality (68/140; 48.6%) and cost/cost-effectiveness for patients (50/140; 35.7%) (Table 2).

Table 1.

Characteristics of 140 Cochrane reviews examined

Review characteristic Reviews (N=140)
n (%)

Year of publication
 2008 6 (4.3)
 2009 34 (24.3)
 2010 18 (12.9)
 2011 35 (25.0)
 2012 31 (22.1)
 2013 16 (11.4)

Status
 Protocol* 41 (29.3)
 Completed review 99 (70.7)

Type of intervention assessed
 Clinical management 69 (49.3)
 Biomedical prevention 20 (14.3)
 Behavioral prevention 28 (20.0)
 Health services 23 (16.4)
*

A protocol is a peer-reviewed published document outlining the planned methods of a Cochrane review [including outcomes to be examined]).

Box 2. Examples of interventions addressed in included Cochrane reviews, by type of intervention.

Clinical management
  • Abacavir-based triple nucleoside regimens for maintenance therapy in patients with HIV

  • Stavudine, lamivudine and nevirapine combination therapy for treatment of HIV infection and AIDS in adults

  • Effectiveness of antiretroviral therapy in HIV-infected children under 2 years of age

  • Herbal medicines for treating HIV infection and AIDS

  • Topical treatments for HIV-related oral ulcers

Biomedical prevention
  • Male circumcision for prevention of heterosexual acquisition of HIV in men

  • Nonoxynol-9 for preventing vaginal acquisition of HIV infection by women from men

  • Antiretroviral post-exposure prophylaxis (PEP) for occupational HIV exposure

  • Interventions for preventing late postnatal mother-to-child transmission of HIV

  • Sperm washing to prevent HIV transmission from HIV-infected men but allowing conception in sero-discordant couples

Behavioral prevention
  • Behavioral interventions to promote condom use among women living with HIV

  • Behavioral interventions for preventing HIV infection in homeless or unstably-housed adults

  • Behavioral interventions to reduce risk for sexual transmission of HIV among men who have sex with men

  • Interventions for reduction of stigma in people with HIV/AIDS

  • Interventions to modify sexual risk behaviors for preventing HIV in homeless youth

Health services
  • Home-based care for reducing morbidity and mortality in people infected with HIV/AIDS

  • Integration of HIV/AIDS services with maternal, neonatal and child health, nutrition, and family planning services

  • Setting and organization of care for persons living with HIV/AIDS

  • Interventions to improve adherence to antiretroviral therapy in children with HIV infection

  • Mobile phone text messaging for promoting adherence to antiretroviral therapy in patients with HIV infection

Table 2.

Comparison of seven most central and seven most frequent outcomes, for all interventions and by type of intervention

Type of intervention (number of reviews) Most central outcomes* Most frequent outcomes*

Outcome (outcome code)** nNBC*** Outcome (outcome code)** % of reviews

All (n=140) All-cause mortality (G25) 23.9 All-cause mortality (G25) 48.6
Cost/cost-effectiveness for patients (L2) 16.4 Cost/cost-effectiveness for patients (L2) 35.7
Adverse events (specified) (A3) 14.8 CD4 count (G4) 30.0
Quality-of-life (K2) 10.6 Quality-of-life (K2) 27.9
Acquisition/incidence of HIV (G11) 9.1 Acquisition/incidence of HIV (G11) 18.6
CD4 count (G4) 6.5 Adherence (J2) 18.6
Unprotected sex (type unspecified) (B4) 6.5 Unprotected sex (type unspecified) (B4) 17.9

Clinical management (n=69) All-cause mortality (G25) 38.2 All-cause mortality (G25) 72.5
Adverse events (unspecified) (A2) 13.0 Adverse events (unspecified) (A2) 52.1
Adverse events (specified) (A3) 13.0 CD4 count (G4) 40.6
Quality-of-life (K2) 9.3 Major/severe/serious adverse events (A4) 31.9
Viral load (G5) 8.8 Quality-of-life (K2) 31.9
CD4 count (G4) 4.5 Adverse events (specified) (A3) 21.7
Symptom resolution (GO15) 3.5 AIDS-defining illness/event (G23) 20.3

Biomedical prevention (n=20) Adverse events (specified) (A3) 33.3 Acquisition/incidence of HIV (G11) 60.0
Acquisition/incidence of HIV (G11) 24.3 Adverse events (unspecified) (A2) 45.0
Mother-to-child transmission of HIV (G18) 22.7 Mother-to-child transmission of HIV (G18) 35.0
All-cause mortality (G25) 16.2 Postpartum morbidity (G16) 20.0
Postpartum morbidity (G16) 11.2 Neonatal/infant mortality (G17) 20.0
Neonatal/infant mortality (G17) 0.6 Stillbirth (GO104) 15.0
Premature delivery (GO103) 0.6 Major/severe/serious adverse events (A4) 15.0

Behavioral prevention (n=28) Self-efficacy/empowerment (E2) 14.9 Acquisition/incidence of HIV (G11) 46.4
Cost/cost-effectiveness for patients (L2) 12.0 Acquisition/incidence of STIs (G9) 46.4
Uptake of HIV testing (D2) 9.6 Number of sexual partners (B8) 28.6
Acquisition/incidence of STIs (G9) 7.5 Prevalence of HIV (G13) 28.6
Acquisition/incidence of HIV (G11) 7.5 Cost/cost-effectiveness for patients (L2) 28.6
Condom use (male condoms) (BO35) 7.3 Condom use (male condoms) (BO35) 28.6
Condom use (female condoms) (BO36) 7.3 Condom use (female condoms) (BO36) 28.6

Health services (n=23) All-cause mortality (G25) 14.1 All-cause mortality (G25) 65.2
CD4 count (G4) 8.7 Quality-of-life (K2) 56.5
Quality-of-life (K2) 7.6 CD4 count (G4) 56.5
Progression to AIDS (G22) 6.4 Viral load (G5) 47.8
Viral load (G5) 5.8 Progression to AIDS (G22) 43.5
Cost/cost-effectiveness for patients (L2) 4.6 Cost/cost-effectiveness for patients (L2) 30.4
Hospitalization (G20) 1.7 Adherence (J2) 30.4
*

Within each type of intervention, outcomes in bold text are exclusive (i.e., among the seven most central outcomes in a group of reviews, but are NOT among the seven most frequent outcomes, or vice versa).

**

Alphanumeric outcome codes are provided here to depict categories of each outcome and facilitate comparisons with Figures 1 through 4 and Appendix 1. Each alphabet denotes a separate outcome category.

***

nNBC, the normalized node betweenness centrality, reflects the extent to which connections between the other nodes (outcomes) in the network rely on that node (outcome) as an intermediary. Note that the nNBC is not a percentage.

4.2 Categorization of outcomes

Most of the 294 outcomes fell into one of two of the 14 categories we defined: clinical/biological (160/294; 54.4%) and behavioral (51/294; 17.4%) (Figure 1). Almost half of the outcomes (138/294; 47%) appeared in only one review each. Appendix 1 lists all 294 outcomes with their outcome codes and the number of reviews in which each outcome was named.

Figure 1. Categorization of all 294 unique outcomes into 14 categories.

Figure 1

Examples of outcomes by category

A – Adverse events: major/severe/serious adverse events

B – Behavioral: Unprotected vaginal sex

C – Antiretroviral treatment: Switching of antiretroviral treatment

D – Testing/counseling: Pre-test counseling

E – Mental/social: Depression/depressive symptoms

F – Health services access/uptake: Utilization of healthcare

G – Clinical/biological: All-cause mortality

H – Knowledge: Condom use knowledge

I – Attitude: Sexual risk behavior attitudes

J – Adherence: Adherence

K – Quality-of-life: Quality-of-life

L – Economic: cost/cost-effectiveness for patients

M – Preference/satisfaction: Patient satisfaction with intervention

O – Miscellaneous: Illness intrusiveness

The alphabet before each category name refers to the category code. Each individual outcome within a category was assigned its own alphanumeric code, beginning with the category code and followed by a number.

Inline graphic A – Adverse effects

Inline graphic B – Behavioral

■ C – Antiretroviral treatment

□ D – Testing/counseling

Inline graphic E – Mental/social

Inline graphic F – Health services access/uptake

Inline graphic G – Clinical/biological

Inline graphic H – Knowledge

Inline graphic I – Attitudes

Inline graphic J – Adherence-related

Inline graphic K – Quality-of-life

Inline graphic L – Economic

Inline graphic M – Preference/satisfaction

Inline graphic O – Miscellaneous

4.3 Steps 1 and 2: Defining the structure and graphing the network

After defining a tie as outcome co-occurrence in two or more reviews, we graphed the network (exploratory analysis) (Figure 2). This graph displays 156 of the 294 outcomes; the remaining 138 isolated outcomes are not displayed. Two main clusters were evident (clinical/biological outcomes and behavioral outcomes), though the clustering was not mutually exclusive. For example, some clinical/biological outcomes (e.g., incidence of sexually transmitted infections [code G9]) clustered with behavioral outcomes. A few outcomes (e.g., cost/cost-effectiveness for patients [code L2]) appeared to bridge the two clusters.

Figure 2. Exploratory social network analysis of 156 unique outcomes that co-occurred with at least one other outcome in two or more reviews (color-coded by category of outcome). All node sizes are equal. Outcomes are labeled using their outcome codes.

Figure 2

Alphanumeric codes are provided next to each node to depict categories of each outcome and facilitate comparisons with Table 2, Figure 1, and Appendix 1. Each alphabet denotes a separate outcome category.

The alphabet before each category name refers to the category code. Each individual outcome within a category was assigned its own alphanumeric code, beginning with the category code and followed by a number.

Inline graphic A – Adverse effects

Inline graphic B – Behavioral

■ C – Antiretroviral treatment

□ D – Testing/counseling

Inline graphic E – Mental/social

Inline graphic F – Health services access/uptake

Inline graphic G – Clinical/biological

Inline graphic H – Knowledge

Inline graphic I – Attitudes

Inline graphic J – Adherence-related

Inline graphic K – Quality-of-life

Inline graphic L – Economic

Inline graphic M – Preference/satisfaction

Inline graphic O – Miscellaneous

4.4 Step 3: Calculating network descriptive statistics

After exploratory analysis, we re-sized each node to be proportional to its nNBC (Figure 3). Overall, the most central outcomes were all-cause mortality (code G25) and cost/cost-effectiveness for patients (code L2) (Table 2).

Figure 3. Main social network analysis of 156 unique outcomes that co-occurred with at least one other outcome in two or more reviews (color-coded by category of outcome). Node size is proportional to node betweenness centrality. Outcomes are labeled using their outcome codes.

Figure 3

Alphanumeric codes are provided next to each node to depict categories of each outcome and facilitate comparisons with Table 2, Figure 1, and Appendix 1. Each alphabet denotes a separate outcome category.

The alphabet before each category name refers to the category code. Each individual outcome within a category was assigned its own alphanumeric code, beginning with the category code and followed by a number.

Inline graphic A – Adverse effects

Inline graphic B – Behavioral

■ C – Antiretroviral treatment

□ D – Testing/counseling

Inline graphic E – Mental/social

Inline graphic F – Health services access/uptake

Inline graphic G – Clinical/biological

Inline graphic H – Knowledge

Inline graphic I – Attitudes

Inline graphic J – Adherence-related

Inline graphic K – Quality-of-life

Inline graphic L – Economic

Inline graphic M – Preference/satisfaction

Inline graphic O – Miscellaneous

Because the network centralization (0.27) was closer to zero than to one, it did not provide evidence for the pervasion of the entire network by the most central outcome (i.e., all-cause mortality, nNBC=23.9). The network appeared dense, and there was only one component; nevertheless, the network cohesion was low (density=0.09). This can be explained by clustering of outcomes with only a few bridges between clusters. Further, most outcomes were examined in only a few reviews each (median=2, IQR=1–3), limiting outcome co-occurrence.

4.5 Step 4: Sensitivity analysis

When we redefined a tie to be outcome co-occurrence in three or more reviews, rather than two, the number of tied outcomes dropped from 156 to 96, but the density changed negligibly (from 0.09 to 0.10) and the centralization remained unchanged (0.27). Six of the seven most central outcomes were the same, while the seventh (‘unprotected sex’) was replaced by ‘adherence’.

4.6 Step 5: Subgroup analysis

The seven most central and seven most frequent outcomes differed overall and by intervention-defined subgroups (Table 2 and Supplementary Figure 1). Thus, certain outcomes would be missed if one considered only frequency or only centrality. For example, adverse events (specified), a patient-important outcome, was one of the most central outcomes in the overall network (nNBC=14.8), but not one of the most frequent (17/140 reviews; 12.1%). Similarly, the same outcome was the most central outcome in the biomedical prevention subgroup (nNBC=33.3), but not one of the most frequent (2/20 reviews; 10.0%). This suggests that while this outcome was not frequently used in the reviews, it was important to connecting other outcomes in the networks.

When we compared each network’s seven most central outcomes using betweenness and degree centralities, the lists of outcomes were generally similar. For example, in the health services subgroup, only the sixth and the seventh outcomes in the lists swapped positions (data not shown).

The nNBCs revealed that the evidence supporting the most central outcomes in the sub-networks of clinical management and biomedical prevention was stronger than the evidence supporting the most central outcomes in the other sub-networks. This is a consequence of how centralization is computed; networks with large differences in nNBC between their two most central nodes have high centralization scores. For example, in the most centralized sub-network, clinical management (centralization=0.37, Table 3), the most central outcome (all-cause mortality) had considerably higher nNBC than that of the next most central outcome (38.2 vs. 13.0, Table 2). On the other hand, in the least centralized sub-network, health services (centralization=0.12, Table 3), the difference in nNBC between the two most central outcomes was less remarkable (14.1 vs. 8.7). While all-cause mortality was the most central outcome in both sub-networks, it was more central to the former than the latter sub-network (nNBC=38.2 vs. 14.1, respectively, Table 2). The comparison of nNBCs for the same outcome in different networks is valid because of the normalized nature of this statistic.

Table 3.

Social network statistics for all interventions and by type of intervention (sorted by increasing network density)

Type of intervention Number of reviews Number of outcomes Number of components Density Network betweenness centralization (“centralization”)

All interventions 140 156 1 0.09 0.27
 Clinical management 69 58 1 0.17 0.37
 Biomedical prevention 20 22 2 0.22 0.29
 Behavioral prevention 28 39 1 0.33 0.13
 Health services 23 22 1 0.52 0.12

The inverse relationship between density and centralization for the sub-networks (Table 3) suggests that the more dense sub-networks were less centralized around their most central outcome. All sub-networks included one component each, except for the biomedical prevention sub-network (Supplementary Figure 1b) which contained one additional tiny component.

5. Discussion

Choice of outcomes is critical to trials and reviews, and yet, current approaches to inform that choice vary. Inconsistent outcome use leads to systematic reviewers being unable to compare results across studies or synthesize results in a meta-analysis, threatening the credibility and potential impact of evidence synthesis efforts. In this study, we applied social network analysis methods to identify systematically central outcomes (outcomes important to the connectedness of other outcomes in the network) in all 140 Cochrane reviews on HIV/AIDS. Before beginning our social network analysis, we categorized the 294 outcomes to examine their topic coverage. Whereas others might reasonably categorize these outcomes differently, our categorization system is supported a posteriori by overlap with the spring embedding-based clustering of outcomes on the graphs, and the 15 categories used during a recent survey of outcomes in Cochrane reviews.[35] Future researchers might also use community detection algorithms [36] to formally evaluate the extent of overlap between pre-defined categories of outcomes and social network analysis-identified outcome communities.

We identified the seven most central outcomes for HIV/AIDS interventions overall and separately for each intervention sub-network, considering outcomes with the highest nNBCs as central to a network. For two intervention sub-networks (clinical management and biomedical prevention), we observed evidence supporting pervasion of the networks by their single most central outcomes. Future research should evaluate whether a minimum nNBC cutoff can be used to determine whether or not a given outcome is central to a network. Future research should also evaluate whether seven outcomes is the optimal number for examining centrality.

5.1 What is gained by examining centrality?

To ensure that relevant outcomes are not missed, we suggest that those developing core outcome sets begin by considering both frequent and central outcomes. When we compared the most central to the most frequent outcomes overall or in a sub-network, there was some overlap, but the social network analysis did convey additional information. For example, the list of most frequent outcomes, overall, excludes the outcome of ‘adverse events (specified)’, a patient-important central outcome. So, if frequency were used as the sole determinant for identifying outcomes for a core outcome set, this outcome would have been missed. Similarly, in the sub-networks, all-cause mortality, viral load, and symptom resolution are examples of outcomes that would have been missed if frequency were solely used.

There are various types of centrality. We chose betweenness centrality because it identifies outcomes as central if they are important to the connectedness of the network. Degree centrality measures the number of direct connections that each node has.[8] Eigenvector centrality is the measure of the influence of a node in a network.[8] Closeness centrality is related to the shortest distance between pairs of nodes.[8] While the latter three types of centrality capture important characteristics of outcomes in a network, betweenness centrality captures a concept fundamental to developing core outcome sets, the connectedness of a network and how the outcomes relate to one another.

5.2 Developing core outcome sets using social network analysis

The existence and promotion of core outcome sets will likely enhance the comparability of outcomes across research in a given topic area.[4] Although core outcome sets represent an attempt to standardize outcomes, they are developed using various methods, sometimes arbitrarily. Some of the methods used include the Delphi technique, semi-structured group discussions (e.g., workshops), unstructured group discussions, reviews, and surveys.[23] The COMET Initiative,[23] the Patient-Reported Outcomes Measurement Information System (PROMIS),[37] and the Outcome Measures in Rheumatology (OMERACT) Initiative [38] recommend the use of formal processes.

While it is premature to decide on an optimal method for developing core outcome sets, social network analysis can contribute a perspective that complements traditional frequency, consensus, and survey methods. Central outcomes should not automatically be considered core outcomes. Our study does not test various methods for developing core outcome sets; rather, we explore a method (social network analysis) that could contribute by identifying central outcomes for core outcome sets. Once potential outcomes are identified from both frequent and central outcomes, assuming involvement of all stakeholders, the next step in core outcome set development should be to ensure that all stakeholders, including patients, have a voice.

Because our social network analysis demonstrated different central outcomes across the intervention sub-networks, tailoring core outcome sets by intervention type is likely beneficial. Even when the most central outcome for two sub-networks was the same (i.e., ‘all-cause mortality’ for clinical management and health services), this outcome was more central to the former than the latter sub-network (nNBC 38.2 vs. 14.1). This suggests stronger evidence for including all-cause mortality in a core outcome set for clinical management than for health services in HIV/AIDS.

In our study, most subnetworks included different types of HIV/AIDS interventions and different types of populations. Those developing core outcome sets should consider whether separate core outcome sets are warranted by type of intervention, type of population, or both. The networks of outcomes would need to be constructed accordingly.

5.3 Developing core outcome sets starting with reviews

Systematic reviews are an excellent starting point for identifying central outcomes for core outcome sets. Existing reviews in a topic area, when considered together, appraise a large portion of the evidence, much of it from trials. Although it bears further investigation, deriving central outcomes from reviews is potentially useful for designing trials relevant for reviews and decision-making. Another argument for using reviews is that it is recommended that reviews (e.g., Cochrane reviews [34]) include patient representatives in the process. If followed, this recommendation allows for broader input on outcome inclusion. Further, reviews of intervention effectiveness pre-specify outcomes and indicate where these outcomes are missing from trial reports.[3, 39, 40] For example, choice of outcomes in a trial may be related to what data can be gathered easily (i.e., interim outcomes), and may not always address questions that need to be answered. An example is the focus of glaucoma trials on intraocular pressure instead of visual field, the patient-important outcome that influences visual functioning.[41]

Cochrane reviews use fairly standardized processes within Review Groups. For approximately 36% of Cochrane Review Groups, the editorial team makes decisions about outcome selection for reviews under the Group’s purview.[42] In the HIV/AIDS Review Group, however, review authors propose a list of outcomes, with additions and deletions suggested by the editorial team and peer reviewers, which often include patients and other stakeholders. There is only one core outcome set related to the area of HIV/AIDS, and it is specifically related to mother-to-child transmission of HIV among breastfeeding mothers.[24] Thus, the central outcomes we identified likely reflect the preferences of the larger community of Cochrane review authors in HIV/AIDS, rather than just those of the Review Group. However, in narrower topic areas where the same authors might contribute to a large proportion of the reviews, it is possible that the central outcomes identified might be greatly influenced by those authors’ preferences, rather than the larger community of stakeholders in that topic area.

It is possible that reviews are not the best starting point for developing core outcome sets. Reviews sometimes miss important outcomes reported in trials.[40] Therefore, the number of outcomes examined in trials included in the reviews we examined might be greater than the number of outcomes examined in the reviews. Although Cochrane recommends that systematic reviewers select outcomes without considering trial outcomes,[34] future research should compare the central outcomes of networks developed using the two starting points (trials and reviews). This comparison would help address whether those developing core outcome sets should focus on outcomes examined in reviews, trials, or both.

5.4 Conclusions

The novel application of social network analysis methods to identify central outcomes using Cochrane reviews provides important information needed for the development of core outcome sets in HIV/AIDS. Social network analysis can uncover outcome co-occurrence patterns not discernable using current frequency-based approaches. While there was some overlap, the outcomes identified using methods to identify co-occurrence and centrality were different from those identified using frequency of occurrence alone. Although our results are preliminary, the methods appear feasible and deserve further study.

Supplementary Material

WHAT IS NEW.

Key findings

  • We applied social network analysis methods to outcomes examined in all Cochrane systematic reviews of HIV/AIDS, thus identifying central outcomes, i.e., outcomes important to the connectedness of other outcomes in the network of outcomes in that topic area.

  • Although there was some overlap, the seven most central and the seven most frequent outcomes differed, both overall and within subgroups by type of HIV/AIDS intervention (i.e., clinical management, biomedical prevention, behavioral prevention, and health services).

What this adds to what is known

  • Current approaches to developing core outcome sets vary, and frequency approaches are the most common. We have demonstrated the novel application of social network analysis to identify central outcomes.

What is the implication, what should change now

  • Systematic reviews can be a useful starting point for identifying central outcomes.

  • Social network analysis can contribute a perspective that complements traditional frequency, consensus, and survey methods for developing core outcome sets.

  • Once potential outcomes are obtained from both frequent outcomes and central outcomes, assuming involvement of all stakeholders, the next step should be to develop core outcome sets.

Acknowledgments

We thank Evan Mayo-Wilson, MPA, DPhil; Swaroop Vedula, MBBS, MPH, PhD; and Stanley Wasserman, PhD for comments on previous versions of this manuscript. We thank Xinggang Liu, PhD for assistance with the graphs; and Xue Wang, MBBS, MSPH for assistance with pilot-testing the data abstraction form. This work was funded by the National Institutes of Health grant number EY020140, the Cochrane Eyes and Vision Group, and the Johns Hopkins Center for Global Health.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Ian J. Saldanha, Email: isaldan1@jhmi.edu.

Tianjing Li, Email: tli19@jhu.edu.

Cui Yang, Email: cyang29@jhu.edu.

Cesar Ugarte-Gil, Email: cugarte1@jhu.edu.

George W. Rutherford, Email: george.rutherford@ucsf.edu.

Kay Dickersin, Email: kdicker3@jhu.edu.

References

  • 1.Meinert CL. Clinical trials dictionary: Terminology and usage recommendations. 2. Wiley; Hoboken, NJ: [Google Scholar]
  • 2.Zarin DA, Tse T, Williams RJ, Califf RM, Ide NC. The ClinicalTrials.gov results database–update and key issues. N Engl J Med. 2011;364(9):852–60. doi: 10.1056/NEJMsa1012065. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: An evaluation of completeness and comparability. PLOS One. 2014;9(10):e109400. doi: 10.1371/journal.pone.0109400. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.COMET Initiative. [Last accessed August 10, 2015];COMET Initiative. Available online at: www.comet-initiative.org/
  • 5.Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials. 2012;13(132) doi: 10.1186/1745-6215-13-132. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Medical Research Council Network Hubs for Trails Methodology Research. [Last accessed August 10, 2015];COMET Initiative. Available online at: http://www.methodologyhubs.mrc.ac.uk/upcoming_workshops/comet_initiative.aspx.
  • 7.National Institutes for Health Research. [Last accessed August 10, 2015];COMET database launched to encourage use of core outcome sets in health research. Available online at: http://www.rds.nihr.ac.uk/latest-news/comet-database-launched-to-encourage-use-of-core-outcome-sets-in-health-research/
  • 8.Wasserman S, Faust K. Social Network Analysis Methods and Applications. Cambridge: Cambridge University Press; 1994. [Google Scholar]
  • 9.Freeman LC. Centrality in networks: I. Conceptual clarification. Social Networks. 1979;1:215–39. [Google Scholar]
  • 10.Eblen M, Fabsitz RR, Olson JL, Pearson K, Pool LR, Puggal M, et al. Social network analysis comparing researcher collaborations in two cardiovascular cohort studies. Research Evaluation. 2012;21:392–405. [Google Scholar]
  • 11.Newman ME. Coauthorship networks and patterns of scientific collaboration. Proc Natl Acad Sci USA. 2004;101(Suppl 1):5200–5. doi: 10.1073/pnas.0307545100. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Hughes ME, Peeler J, Hogenesch JB, Trojanowski JQ. The growth and impact of Alzheimer disease centers as measured by social network analysis. JAMA Neurol. 2014;71(4):412–20. doi: 10.1001/jamaneurol.2013.6225. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Greenberg SA. How citation distortions create unfounded authority: analysis of a citation network. BMJ. 2009 Jul 20;339:b2680. doi: 10.1136/bmj.b2680. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Fowler JH, Christakis NA. Dynamic spread of happiness in a large social network: longitudinal analysis over 20 years in the Framingham Heart Study. BMJ. 2008 Dec 4;337:a2338. doi: 10.1136/bmj.a2338. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Rosenquist JN, Fowler JH, Christakis NA. Social network determinants of depression. Mol Psychiatry. 2011;16(3):273–81. doi: 10.1038/mp.2010.13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16.Pachucki MA, Jacques PF, Christakis NA. Social network concordance in food choice among spouses, friends, and siblings. Am J Public Health. 2011;101(11):2170–7. doi: 10.2105/AJPH.2011.300282. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.de la Haye K, Robins G, Mohr P, Wilson C. How physical activity shapes, and is shaped by, adolescent friendships. Soc Sci Med. 2011;73(5):719–28. doi: 10.1016/j.socscimed.2011.06.023. [DOI] [PubMed] [Google Scholar]
  • 18.Rosenquist JN, Murabito J, Fowler JH, Christakis NA. The spread of alcohol consumption behavior in a large social network. Ann Intern Med. 2010;152(7):426–33. doi: 10.1059/0003-4819-152-7-201004060-00007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Tucker JS, de la Haye K, Kennedy DP, Green HD, Pollard MS. Peer influence on marijuana use in different types of friendships. J Adolesc Health. 2014;54(1):67–73. doi: 10.1016/j.jadohealth.2013.07.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Christakis NA, Fowler JH. The collective dynamics of smoking in a large social network. N Engl J Med. 2008;358(21):2249–58. doi: 10.1056/NEJMsa0706154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Valente TW, Unger JB, Johnson CA. Do popular students smoke? The association between popularity and smoking among middle school students. J Adolesc Health. 2005;27(4):323–9. doi: 10.1016/j.jadohealth.2004.10.016. [DOI] [PubMed] [Google Scholar]
  • 22.Green HD, Jr, Horta M, de la Haye K, Tucker JS, Kennedy DR, Pollard M. Peer influence and selection processes in adolescent smoking behavior: a comparative study. Nicotine Tob Res. 2013;15(2):534–41. doi: 10.1093/ntr/nts191. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Gargon E, Gurung B, Medley N, Altman DG, Blazeby JM, Clarke M, Williamson PR. Choosing health outcomes for comparative effectiveness research: A systematic review. PLOS One. 2014 doi: 10.1371/journal.pone.0099111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Alioum A, Dabis F, Dequae-Merchadou L, Haverkamp G, Hudgens M, Hughes J, Karon J, Leroy V, Newell ML, Richardson B, Weverling GJ. Stat Med. 2001;20(23):3539–56. doi: 10.1002/sim.1076. [DOI] [PubMed] [Google Scholar]
  • 25.Valente TW. Social Networks and Health: Models, Methods, and Applications. New York: Oxford University Press; 2010. [Google Scholar]
  • 26.Carrington PJ, Scott J, Wasserman AS. Models and methods in social network analysis. Cambridge University Press; New York, NY: 2005. [Google Scholar]
  • 27.Borgatti SP. Centrality and AIDS. Connections. 1995;18(1):112–114. [Google Scholar]
  • 28.Breiger RL. The duality of persons and groups. Social Forces. 1974;53(2):181–90. [Google Scholar]
  • 29.Everett MG, Borgatti SP. The dual-projection approach for two-mode networks. Social Networks. 2013;35(2):204–10. [Google Scholar]
  • 30.Borgatti SP, Everett MG, Freeman LC. Ucinet for Windows: Software for social network analysis. Harvard, MA: Analytic Technologies; 2002. [Google Scholar]
  • 31.Borgatti SP. NetDraw software for network visualization. Lexington KY: Analytic Technologies; 2002. [Google Scholar]
  • 32.Freeman LC. Graphic Techniques for exploring social network data. In: Carrington PJ, Scott J, Wasserman S, editors. Models and Methods in Social Network Analysis. New York: Cambridge University Press; 2005. pp. 248–269. [Google Scholar]
  • 33.Bonacich P. Power and centrality: a family of measures. Am J Sociol. 1987;92:1170–82. [Google Scholar]
  • 34.Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration; 2011. [Last accessed August 10, 2015]. Version 5.1.0 [updated March 2011] Available online at: www.cochrane-handbook.org. [Google Scholar]
  • 35.Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported. J Clin Epidemiol. 2015;68(3):237–45. doi: 10.1016/j.jclinepi.2014.09.022. [DOI] [PubMed] [Google Scholar]
  • 36.Lancichinetti A, Fortunato S. Community detection algorithms: a comparative analysis. Physical review. 2009;E 80:056117. doi: 10.1103/PhysRevE.80.056117. [DOI] [PubMed] [Google Scholar]
  • 37.PROMIS. [Last accessed August 10, 2015];About PROMIS. Available online at: http://nihpromis.org.
  • 38.Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L. OMERACT: An international initiative to improve outcome measurement in rheumatology. Trials. 2007;8:38. doi: 10.1186/1745-6215-8-38. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365. doi: 10.1136/bmj.c365. [DOI] [PubMed] [Google Scholar]
  • 40.Ioannidis JP, Horbar JD, Ovelman CM, Brosseau Y, Thorlund K, Buus-Frank ME, Mills EJ, Soll RF. Completeness of main outcomes across randomized trials in entire discipline: survey of chronic lung disease outcomes in preterm infants. BMJ. 2015;350:h72. doi: 10.1136/bmj.h72. [DOI] [PubMed] [Google Scholar]
  • 41.Saldanha IJ, Dickersin K, Wang X, Li T. Outcomes in Cochrane systematic reviews addressing four common eye conditions: An evaluation of completeness and comparability. PLoS One. 2014;9(10):e109400. doi: 10.1371/journal.pone.0109400. eCollection 2014. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews? – a survey of the Co-ordinating Editors of Cochrane Review Groups. Trials. 2013;14:21. doi: 10.1186/1745-6215-14-21. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

RESOURCES