Measure of Personal Network Size Using the Known Population Method: A Methodological Guide

Abstract

The network scale-up method (NSUM) has shown promise in measuring the prevalence of hidden public health problems and at-risk populations. The technique involves asking survey respondents how many people they know with the health problem or characteristic of interest and extrapolating this information to the population level.

An important component of the NSUM estimate is the size of each respondent’s network, which is determined by asking respondents about the number of people they know who belong to populations of known size. There is little systematic discussion, however, to guide selection of these questions. Furthermore, many of the most commonly used known population questions are appropriate only in countries with a robust data infrastructure.

Here, we draw from the NSUM literature to present a set of best practices in the selection of NSUM known population questions. Throughout, we address the unique situations that many researchers face in collecting prevalence data in the developing world, where innovative prevalence estimation techniques, such as NSUM, are most needed. (Am J Public Health. 2022;112(5):747–753. https://doi.org/10.2105/AJPH.2022.306731)

The network scale-up method (NSUM) is a prevalence estimation technique designed to measure the size of populations that are hidden from view. The technique was developed to measure the number of deaths caused by the 1985 Mexico City earthquake.^1,2 One resident reportedly said that thousands must have perished because everyone knew at least 1 person who died.¹ Intrigued by this comment, the research team devised the NSUM as a way to use respondents’ knowledge about deaths among their contacts to estimate the total number of deaths caused by the earthquake. Specifically, researchers derive NSUM prevalence estimates by determining the proportion of people who have the characteristic of interest in the social network of respondents and then “scaling-up” to the population level. The NSUM approach is a useful tool for public health researchers, as it is necessary to measure the prevalence of health problems or characteristics that put people at risk to develop effective interventions. Indeed, it has been used to calculate the size of many hard-to-reach populations, including men who have sex with men,³ intravenous drug users,⁴ and sex workers.⁵

A significant barrier to widespread implementation of the NSUM, however, is correct measurement of personal network size. Because people may be unable to report accurately the number of people they know if asked directly,⁶ NSUM researchers ask respondents how many people they know with characteristics whose prevalence is known (populations of known size) and infer the size of their personal network from their responses to these questions (see example in the next section). Inaccurate measurement of personal network size reduces the accuracy of the prevalence estimates, as network size is the denominator in the NSUM equation.⁷ Thus, appropriate selection of the known population questions that serve as the basis of the personal network size estimate is essential. While some research addresses specific aspects of question selection^8,9 and there are examples in the applied NSUM literature of different types of known population questions,^4,10–12 there is little in the way of systematic, practical guidance.

The purpose of this article is to offer a set of best practices on the selection of known population questions used to determine personal network size in NSUM prevalence estimates. This provides needed information to public health researchers who may have considered using the NSUM but were unsure how to proceed. We review the NSUM literature to suggest solutions to common problems with known population questions. We also present information about known population questions and data sources from published NSUM literature in a table for easy reference (Table A, available as a supplement to the online version of this article at http://www.ajph.org). In doing so, our article also serves to introduce the NSUM to public health researchers who are unfamiliar with the method. In suggesting best practices for selecting known population questions, we highlight techniques applicable in the developing world, where indirect prevalence estimation techniques such as the NSUM are most needed.¹³ Increasing understanding of how to select known population questions will not only improve and increase the usage of the NSUM as a prevalence estimation technique, but it will also inform related research in the social sciences that requires estimation of personal network size.^8,14,15 We begin by providing an illustration of the NSUM.

ILLUSTRATION OF THE NETWORK SCALE-UP MODEL

To highlight the role of known population questions in NSUM prevalence estimates, we present an example of how the NSUM would be used to produce an estimate of prescription opioid use. A researcher wanting to estimate the prevalence of prescription opioid use in a given US state would ask respondents how many state residents they know who are members of populations of known size, as well as how many state residents they know who use prescription opioids. The questions about populations of known size are used to estimate the size of the respondent’s network. Surnames are a common type of known population question.^10,13 For example, if there are 20 000 people in the state with the last name Johnson, and the respondent knows 3 of these people, then we would estimate that the respondent knows 0.00015 of the state’s population (3/20 000). If the population of the state is 2 000 000, then we would conclude that the respondent knows approximately 300 people (0.00015 × 2 000 000). Because of nonrandom mixing in the population (e.g., not everyone’s social network is equally represented by persons named Johnson), it is necessary to include a large number of known population questions—such as questions about different surnames, including those that vary by region or ethnicity—and average across these responses to create accurate estimates of personal network size.

The researcher then calculates the proportion of the respondent’s personal network that uses prescription opioids and extrapolates this figure to the population of the state. For example, if a respondent who has an estimated network size of 300 reports knowing 4 people who use prescription opioids, then this suggests that 0.013 of the population (4/300) use prescription opioids. Because no individual’s network is completely representative of the population, estimates of prescription opioid use are averaged across a large number of respondents to produce an overall prevalence rate.¹⁰

As this example illustrates, for the estimates produced by the NSUM model to be accurate, the network size must be correctly estimated. If the estimate of personal network size is erroneously large, then prevalence estimates will be much lower than they should be. Conversely, if the estimate of personal network size is too small, then the prevalence estimates will be inflated. As a result, questions used to calculate personal network size must be carefully chosen.

BEST PRACTICES FOR SELECTION OF KNOWN POPULATION QUESTIONS

Best practices for selecting known population questions involve identifying appropriate data sources on populations of known size, selecting questions from these data sources that avoid recognized biases in the NSUM method, and final selection of questions after data have been collected that maximize estimation accuracy. In this section, we provide guidance for each of these phases.

Identification of Data Sources

The first task for NSUM researchers is to identify data sources from which to select questions about populations of known size. These sources must contain accurate data about the population for which prevalence estimates are sought. The most typical data source is a nationwide census, as some form of census data is gathered in most countries on a regular basis (see Table A for examples of data sources). Probabilistic surveys associated with the census, such as the American Community Survey or the UK Annual Population Survey, can also be useful, as these types of surveys provide more detailed information than the main census.^16,17

Probabilistic surveys collected by external agencies are an alternative to governmental censuses. The Demographic and Health Survey and the Multi-Indicator Cluster Survey, in particular, are appropriate data sources in developing countries where census data are limited. The Demographic and Health Survey and the Multi-Indicator Cluster Survey include information about health behaviors, economic situations, birth rates, religious affiliation, and family structure. To date, however, there are few studies that have used data from these probabilistic surveys to aid in the estimation of hidden populations in the developing world. One notable example is a study that utilized several indicators from the Rwanda Demographic and Health Survey as known population questions, including people who smoked, people who are Muslim, and people who are widowers, to estimate the size of personal networks.¹³ This study highlights the unrealized potential of using data from the Demographic and Health Survey, the Multi-Indicator Cluster Survey, and similar probabilistic surveys in constructing prevalence estimations of hidden populations in the developing world.

To be useful for estimating network size, however, statistics about the known population must be available at the same geographical level (e.g., city, prefecture, region, or country) as the hidden population for which prevalence estimation is sought. To estimate the size of several hidden populations in Nebraska, for example, researchers used state-level census data on names and occupations to construct their known population questions.¹⁸ This allowed the researchers to estimate how many people each respondent knew who lived in Nebraska, which is the relevant personal network for estimating the size of hidden populations in that state.

Finding sufficient data on population characteristics at subnational geographic levels, however, can be challenging in parts of the world where data infrastructure is limited. In these cases, using a combination of data sources might be useful. For example, to obtain data about known populations in Tabriz, Iran, researchers used student registration data from the Tabriz University of Medical Sciences, car fatality data from the local health center, and marriage data from the local marriage registration offices.¹⁹ Here again, the Demographic and Health Survey and the Multi-Indicator Cluster Survey are good resources, as most data are reported at the state or province level, allowing researchers to target hidden populations within relatively narrow geographic areas.

If no appropriate secondary data sources are available, researchers may choose to collect original survey data that establish the size of the populations used to estimate personal network size. It is more feasible to carry out a survey of a smaller geographic area, making this a more viable solution in the cases in which this approach is most likely to be needed. A slightly different approach was taken by a group of researchers in Iran who administered a national-level survey specifically to determine the average network size of Iranian residents, using official data sources to benchmark the size of the known subpopulations.²⁰ Researchers then fielded NSUM-style surveys on a variety of public health topics, using the average network size derived from the national survey in their NSUM prevalence estimation equations.²¹

When none of these options are feasible, researchers could consider the summation method of estimating network size, which avoids known subpopulation questions altogether.²² In this method, respondents are asked to recount the number of people they know in various life domains, such as family, work, and school. The number of individuals in each group is then summed. While this method is not without problems,²³ it is a straightforward approach that could prove useful in areas with poor data infrastructure.

Avoiding Known Biases

Once researchers have identified appropriate data sources for known population questions, they should select questions that minimize the 3 main sources of bias in NSUM prevalence estimates: transmission bias, barrier bias, and recall bias. These biases not only affect responses to questions about the population of interest (e.g., number of prescription opioid users a respondent knows), but they also affect estimates of respondents’ personal network size⁸ (See Table A for full listing of questions used in previous NSUM studies).

Transmission bias

First, transmission bias occurs when a respondent fails to recognize that they know something about people in their social network.⁷ This occurs because the information is not easily observable, such as having diabetes, or is unlikely to be communicated because the condition is stigmatized, such as being HIV positive.^4,24 Transmission bias would occur, for example, if there were 4 people in a respondent’s social network who were HIV positive, but the respondent was aware of only 1. Transmission bias can be reduced by asking about characteristics that are commonly communicated, such as names and occupations, or easily observed, such as physical traits. Using first names, for example, avoids transmission bias in the construction of network size estimates because a first name is the most basic piece of information that one knows about those in their social network.⁸ Researchers further recommend selecting first names that do not typically generate nicknames, as this can introduce transmission bias.⁸

Awareness of some social characteristics, though, is not as straightforward. The extent to which some information is transmitted to people in one’s personal network varies cross-culturally, which means that solutions to transmission bias must be sensitive to the local context. For example, while it is common in the West to talk about one’s job with an acquaintance, occupational identity is less salient in non-Western contexts, making it less likely that acquaintances know one another’s occupations.^25,26 Similarly, it would not be appropriate in Japan to ask about certain health statuses, such as having cancer, because Japanese people rarely reveal information about their health.²⁷

Researchers should also avoid asking about nonvisible stigmatized activities or conditions, as people are unlikely to communicate this information. What is considered stigmatizing varies, though, across contexts. For example, use of tobacco is stigmatized for women in some parts of the developing world, making it an inappropriate question.²⁸ Because some national surveys from which known population questions are drawn focus on “problems,” such as smoking or drug use, candidate questions often refer to stigmatized populations. This is especially true of the known populations that are relatively rare, because conditions that are more present in the population are, by definition, more normalized. Researchers should conduct focus groups on candidate questions before survey administration to determine exactly what characteristics are stigmatized in particular cultural contexts.^9,22,29

It is also possible to adjust statistically for transmission bias. Though this is not typically employed for transmission bias in known population questions, such adjustments are often used to correct for transmission bias relating to the characteristic whose prevalence is being estimated.²⁷ This usually involves surveying people who belong to the population of interest to determine what proportion of people in their social network is aware of their membership in this population (the visibility factor) and then adjusting the prevalence estimate accordingly.³⁰ Applying this approach to each of the known populations could be quite labor-intensive, so we recommend that researchers consult the broader social network literature that models the effects of transmission bias on estimates of personal network size³¹ before concluding that statistical adjustments are necessary.

Barrier bias

Next, barrier bias refers to the physical and social barriers that prevent a respondent from knowing certain kinds of people because of the nonrandom mixing of people in society.⁷ According to the norm of homophily, people are more likely to know people who are similar to them, which may limit their access to particular segments of the population.³² Barrier bias can be a problem, for example, when known population questions ask about characteristics that are more common among people of certain income brackets, given the uneven mixing of people across socioeconomic status. Ideally, known population questions should ask about characteristics that are evenly distributed across the population.¹⁰

Because most personal characteristics are subject to barrier bias, researchers should select a set of known population questions that, together, cover the population to ensure correct estimation of personal network size.³³ For this reason, researchers conducting a recent prevalence study in China selected various surnames that were common in different areas of China so that, in combination, the surname questions would represent all parts of the country.¹⁰ Similarly, another group of researchers selected a variety of first names that represented different racial and age groups.³⁴ Others using first names stress the importance of selecting names whose popularity is consistent across recent generations.³⁵ Here, too, cultural context is important, as different characteristics are subject to barrier bias in different places. For example, researchers did not need to account for ethnic variation in surname use in their prevalence study in China because China is largely monoethnic.¹⁰ In many other countries, though, researchers using questions about surnames to measure personal network size would need to select surnames that adequately covered the country’s ethnic groups.

One interesting technique to address barrier bias involves utilizing known population questions that are related to the population of interest, thus creating an estimate of a more focused personal network where barrier bias is reduced. Here, researchers seeking to measure the prevalence of heroin use established personal network size through questions about criminal victimization, such as being “beaten up, attacked, or hit” or having “their apartment, home, or garage broken into,” as well as engagement in binge drinking or marijuana use.⁴ Because information on the size of these populations is not available in secondary data sources, these researchers collected data about the “known” populations used to estimate personal network size through the same survey that they used to collect information on the population of interest. The collection of data for the known populations through the same survey used to collect data on the population of interest can also help circumvent problems with poor data infrastructure in the developing world, but it does require strict adherence to random sampling and high response rates to ensure that data are representative. Consistent with more general best practices, one must also avoid questions about highly stigmatized activities. Thus, questions about criminal victimization are more appropriate than questions about criminal perpetration.

Recall bias

Finally, recall bias occurs when the respondent does not accurately remember the number of people they know with a certain characteristic.⁷ For example, recall bias occurs if the respondent is aware that there are 5 Johnsons in their social network but remembers only 3 of these individuals at the time of the survey. Psychological research has shown that high levels of cognitive load delay recall³⁶ and increase central tendency bias, where people overestimate values lower than average and underestimate values higher than average.³⁷ Thus, recall bias is more likely to occur when people are asked to provide a precise numerical count about a large population, as this creates substantial cognitive load.⁸

A prime goal in asking questions about populations of known size is to reduce the recall bias that necessarily occurs when people are simply asked, “How many people do you know?”²³ If known populations are too large, however, then cognitive load remains high, increasing the risk of recall bias.²² For this reason, researchers recommend selecting questions about known populations that constitute 0.1% to 0.2% of the whole population, and should never exceed 5% of the total population.^8,10

Postsurvey Question Selection

The final step in known population question selection occurs after the survey has been administered. At this stage, it is important to examine the data collected from participants about the known populations and use only questions in the calculation of personal network size whose response distribution seems reasonable given the prevalence of the characteristic in the population. This step serves as a validation check, as a large number of responses that are out of step with the actual population prevalence may occur because of transmission, barrier, or recall bias that was not recognized at the time that questions were initially selected, or because of data collection errors.

Specifically, research finds that using back-estimation techniques to select final questions for inclusion in network size estimation can greatly improve the quality of these estimates. The most straightforward approach, known as the “hold-out” method, involves an iterative process of holding out 1 known population question at a time when back-estimating the other known populations and then eliminating the questions that produce estimates that deviate significantly from the true size of the known population.³⁸ This process allows researchers to see which questions are producing unrealistic estimates. Using this method, for example, researchers in China found that including the last name “Liu” produced unreasonably large estimates of the size of the population of the other last names used as known population questions, given the actual prevalence of those names in the population.¹⁰ As a result, they dropped the name “Liu” from the list of known population questions used to estimate personal network size.⁹ Using this procedure, it is typical to drop questions that produce ratios of estimated to actual population size that are below 0.5 or above 2.³⁸

An elaboration of this approach also uses back-estimation techniques and “holds-out” questions in an iterative fashion. Researchers implementing this technique note, however, that removing 1 variable at a time in the traditional back-estimation process and using fixed cutoffs for removal of questions, such as ratios below 0.5 or above 2, fails to recognize that questions perform differently as estimators in combination with different questions.¹⁸ They propose, instead, a recursive approach in which the variable that produces the most inaccurate estimate during each iteration is considered for removal and establishing cutoffs for removal that are based on a logarithmic function. This procedure results in cutoffs that vary depending upon the variables included in the back-estimate. This procedure allows researchers to retain a larger number of known population questions, which can improve the accuracy of network size estimation.

CONCLUSION

For prevalence estimates produced by the NSUM to be accurate, the size of respondents’ personal networks must be correctly calculated. Correct calculation of personal network size depends on careful selection of known population questions from data sources that measure the known population at the same geographic level as the level for which the hidden population estimates will be made. The selected questions must avoid transmission, barrier, and recall bias, which are known threats to NSUM estimation. Researchers should also utilize back-estimation techniques after data collection to ensure the appropriateness of the final set of known population questions. We have constructed a checklist of these best practices to complement our narrative discussion of these topics (Table B, available as a supplement to the online version of this article at http://www.ajph.org).

Our analysis of known population question techniques and distillation of the NSUM literature (Table A) also provides a foundation for research beyond traditional NSUM prevalence models. New advances in NSUM techniques, such as Bayesian estimation,⁹ allow for more accurate prevalence estimates, which will likely increase use of the method in public health. These new advances, however, still require accurate personal network size estimates, making it vital that researchers who employ the NSUM engage in best practices for known population question selection. Our recommendations will also be helpful to public health researchers engaged in network size estimation outside of the NSUM context, such as those investigating the relationship between personal network size and health outcomes.^39,40 Thus, our work serves as a resource for a broad array of public health researchers.