Evaluating the impact of broadband access and internet use in a small underserved rural community

Abstract

Having adequate access to the internet at home enhances quality-of-life for households and facilitates economic and social opportunities. Despite increased investment in response to the COVID-19 pandemic, millions of households in the rural United States still lack adequate access to high-speed internet. In this study, we evaluate a wireless broadband network deployed in Turney, a small, underserved rural community in northwest Missouri. In addition to collecting survey data before and after this internet intervention, we collected pre-treatment and post-treatment survey data from comparison communities to serve as a control group. Due to technical constraints, some of Turney's interested participants could not connect to the network, creating an additional comparison group. These comparisons suggest two primary findings, (1) changes in using the internet for employment, education, and health could not be directly attributed to the internet intervention, and (2) the internet intervention was associated with benefits stemming from the ability to use multiple devices at once. This study has implications for the design of future broadband evaluation studies, particularly those examining underserved rather than unserved communities. Recommendations for identifying appropriate outcome variables, executing recruitment strategies, and selecting the timing of surveys are made.

1. Introduction

Broadband access has become a top concern for policymakers as the digital divide threatens to leave rural communities behind. In the U.S., recent Federal Communications Commission (FCC) estimates suggest that at least 9.3 million rural residents have inadequate broadband service (FCC, 2021). This affects rural communities’ ability to retain residents, generate tax revenue and attract employers.

In response to the COVID-19 pandemic, the U.S. federal government authorized $87 billion in funding for broadband access and adoption. This figure includes $65 billion from the Infrastructure Investment and Jobs Act, to address the digital divide, $20.4 billion for funding digital equity policies from the American Rescue Plan, and $1.6 billion from the Consolidated Appropriations Act, for connecting minority communities, connectivity in tribal lands, and general broadband infrastructure deployment (CRS, 2021; NTIA, 2021; Tomer & George, 2021). These funds will be administered by different federal agencies, such as the FCC and the National Telecommunications and Information Administration (NTIA), as well as by the states and U.S. territories.

Given this large increase in available funding, there exists an opportunity to conduct evaluations estimating the impact of related government programs, which can better inform subsequent tranches of funding. Evaluations support evidence-based policy-making efforts to increase the efficiency of government spending. Infrastructure investments have economic as well as social impacts (Stupak, 2018; Zamojska & Próchniak, 2017). The economic benefits have been heavily investigated (Isley & Low, 2022; Kim & Orazem, 2017; Whitacre et al., 2014), even for our study region (Spell & Low, 2021). However, fewer studies examine quality-of-life impacts, which broadly include ease-of-use, mental health, and lifestyle benefits.

Further, most studies to date have estimated the impact of broadband access and adoption in the United States aggregated at national and state levels (Mack, 2019; Mack et al., 2019; see Whitacre & Gallardo (2022) for a meta-analysis of availability and adoption studies). Moving forward, this will shift with access to higher resolution broadband fabric data (Whitacre & Biedny, 2022). Typically, these studies focus on estimating the average effects of broadband on one specific outcome, such as household income, housing value, or student performance in large geographical areas. However, this approach can make it challenging for a particular community to understand potential impacts. Few studies have explored multiple impacts within a single community over time to understand the impacts of improved broadband (for an example at a regional level, see Palmer-Abbs et al. (2021)).

This study presents a pre-post comparison for community members who did and did not receive new wireless internet service in a small rural community in northwest Missouri called Turney. The target community was underserved – rather than unserved – with respect to internet infrastructure, making this evaluation study especially relevant as fewer communities remain unserved. In underserved communities, broadband access does not meet the FCC definition of broadband, which is 25 megabits per second (Mbps) of download and 3 Mbps of upload (or 25/3 Mbps). However, underserved communities’ access is above the FCC unserved level of 10/1 Mbps (FCC, 2021). In addition, these standards are shifting. The Infrastructure Act recently established a new broadband standard of 100/20 Mbps, which dramatically increases the number of communities considered underserved. Turney is representative of many small rural underserved communities. These communities are most likely to be left behind because they are not unserved (a classification typically prioritized by funding opportunities). Yet, they do not have sufficient market dynamics to incentivize private investment.

This paper provides a design, methods, and implications for evaluating broadband interventions in underserved communities, where identifying treatment effects are more complicated than for unserved communities. We find that the internet intervention was associated with benefits for participants stemming from the ability to use multiple devices simultaneously. The impact of internet use for employment, education, and health was more nuanced. Recommendations for future evaluation studies, including identifying appropriate outcome variables, executing recruitment strategies, and selecting the timing of data collection, are discussed.

1.1. Impact of broadband access and adoption

Economic and social outcomes include impacts that allow participants to increase their earning potential and engage in economic development activities (Atkinson & Castro, 2008; Firth & Mellor, 2005). In particular, social impacts aim to reduce social inequities (e.g., health inequities) (Tomer et al., 2020). To evaluate economic and social outcomes, this study measures the use of online applications related to employment, education, and health. In the context of employment, broadband access generally enhances business start-up activity in rural communities, with the highest impact in remote rural counties (Conroy & Low, 2021). High levels of broadband adoption in rural communities reduce unemployment growth and positively impact income growth (Whitacre et al., 2014). Increases in broadband adoption levels are associated with an increase in median household incomes (Whitacre & Gallardo, 2014).

Online learning is increasingly considered an option for K-12 and higher education students. However, those without access to a reliable and robust internet connection cannot participate in virtual school options (Kelley & Sisneros, 2020). Independent of their socioeconomic background, students without access to the internet at home have lower performance in school and on standardized tests, complete homework at lower rates, and are less likely to attend college or university (Hampton et al., 2020, pp. 1–34). Many students with no access to support for their schoolwork live in deep rural communities (Reisdorf et al., 2019). These effects were consistently observed even before schools were forced to move online due to COVID.

In the context of healthcare, broadband access enables patient-centered care, which uses health information technologies (Sun et al., 2013). Patient-centered care encourages personalized care by enhancing collaborative decision-making involving patients and their health service providers. Telehealth can expand access to health services while creating opportunities for cost reductions (American Hospital Association, 2016). However, cost reductions are not consistently observed in rural communities (Rhoades et al., 2022).

In addition, there are less easily quantified, sometimes referred to as quality-of-life, impacts (Barbosa Neves et al., 2018). Quality-of-life is defined as the discrepancy between people's expectations and actual experiences (Calman, 1984). For example, people in rural areas with poor broadband access know that it should be easier to access the internet to take advantage of entertainment options and have information readily available at their fingertips. This frustration can have ripple effects, such as challenges in recruiting and retaining teachers and doctors to rural areas (Valentín-Sívico et al., 2022). Internet access is also associated with mental health benefits, such as social connectivity, emotional support, and reduced isolation (Collins & Wellman, 2010; Kearns & Whitley, 2019). While there can also be negative impacts of internet access (e.g., worsened gambling addiction), the benefits largely outweigh the costs (Valentín-Sívico, 2020).

1.2. Approaches for broadband impact evaluations

As summarized in Table 1 , both quantitative and qualitative approaches have been employed to evaluate broadband impact. Ideally, broadband impact would be evaluated via a randomized control trial to make causal inferences. However, it is not practical to randomly assign communities to receive broadband or not. Instead, a quasi-experimental approach like difference-in-differences can be used – this design controls for changes over time (treated vs. control group) to determine how much of the effect can be attributed to an intervention (pre vs. post-comparison). However, this approach is not practical with small sample sizes, as is the case in a rural community. For correlational analysis, multivariate regression using regional data enables an understanding of the impact at an aggregated level over large geographic areas. However, there is poor resolution at the community level, as federal data are only available at the county level in most instances.

Table 1.

Summary of impact evaluation approaches employed in the literature.

Approach	Benefits	Drawbacks	Key Examples
Difference-in-differences design	Allows for causal inference; can be used at individual or group levels	Need a control population similar to intervened population; not recommended for small sample sizes	Briglauer et al. (2019), Kim and Orazem (2017)
Analysis of federal data sets from FCC, U.S. Census, etc.	High data coverage	Aggregated over large geographic areas	Isley and Low (2022), Whitacre et al. (2014)
Pre-post surveys (within-subject analysis)	Participation by the same participants increases the power in the statistical analysis	Cannot control for changes over time	LaRose et al. (2011)
Mixed methods	Benefits of both quantitative and qualitative data	High cost and time-consuming	Collins and Wellman (2010), Ashmore et al. (2017)
Pre-post interviews (qualitative analysis)	Rich qualitative data	High cost and time-consuming	Rampersad and Troshani (2013)

At the community level, it is possible to conduct pre-post surveys. In this case, the same participants are surveyed before and after an intervention. Using the same participants for a within-subject design increases statistical power (Sansone et al., 2003). In addition, pre-post interviews can be valuable at the community level to collect rich data on experiences and perceptions. However, qualitative data collection and analysis can be costly and time-consuming. Quantitative and qualitative approaches can also be combined in mixed methods to benefit from both. Overall, the design of a broadband impact evaluation varies depending on the intervention, population, and expected impacts.

2. Methods

2.1. Study design

Participants completed a pre-survey before the intervention, and a post-survey, afterwards, to evaluate the impact of a faster, higher bandwidth internet intervention. Participants were recruited from Turney, the target community, as well as 13 additional control communities with similar characteristics (see Appendix A). A between-subject comparison was conducted to compare Turney to the control group for both the pre- and post-survey. Additionally, a within-subject comparison was conducted within the Turney sample. A significance level of α = 0.05 was used in the analysis. The study design was approved by the University of Missouri Institutional Review Board.

2.2. Target community

Turney is a village in Clinton County located approximately 46 miles from Kansas City in Northwest Missouri, see Fig. 1 . It is a small rural community that covers 0.5 square miles with approximately 76 households and a total population of 114 residents. There are no schools in Turney, so students attend the nearby Lathrop R–II School District. In addition, there are no hospitals or healthcare facilities. The closest hospital is the Cameron Regional Medical Center, located 12 miles away.

Fig. 1.

Turney, MO is located approximately 46 miles northeast from Kansas City, MO.

In terms of internet access, Turney is an underserved community because it is at the edge of several existing networks, none of which fully serve the community. The primary providers, which each cover different parts of Turney, are CenturyLink (wired, 40/7 Mbps) and KC Coyote (wireless, 10/1 Mbps max). However, neither meet the new 100/20 Mbps requirement of the Infrastructure Act. A few households are within GRM Networks (fiber) territory and one respondent reported having their service. Turney was targeted for this study because United Fiber, a local internet service provider affiliated with a rural electric cooperative, owns fiber infrastructure two miles from Turney and was willing to partner with us to extend that fiber edge using a novel wireless technology approach.

2.3. Intervention

As part of a larger project, a wireless broadband system was installed in Turney in September 2021. United Fiber installed a point-to-point mmWave link from their fiber infrastructure to the tallest point in Turney on top of a grain elevator. Households were provided routers to connect to the wireless signal, which was distributed as a point-to-multipoint signal via a proprietary wireless standard, Long Term Ubiquiti (LTU). Part of this project involved developing and testing a community-level intelligent router to dynamically allocate bandwidth to increase quality of service. However, this evaluation is focused on the impact of improved internet access more broadly. Each household connected to the wireless network received at least 100/50 Mbps. At installation, average speeds were 280/63 Mbps. User-reported speed tests suggest that users observed average speeds of 161/61 Mbps. All participants in the wireless network received free broadband service during the duration of the study. In addition, connected households received an email every two weeks highlighting local resources and training related to distance education, telemedicine, entrepreneurship, and other ways to leverage their new internet service.

2.4. Recruitment

Households were initially recruited via a mailed pre-survey in August 2021, which could be mailed back via a pre-addressed envelope or completed online (QR code and link provided). All participants were incentivized to participate via a raffle for $10 Starbucks gift certificates. In total, 200 surveys were mailed to reach Turney households, defined as being within a 3-mile radius of the center of Turney (12 returned as undeliverable). As part of the pre-survey, households indicated their interest in receiving free broadband wireless service during the study's duration. Since control community households were not offered free internet service, we oversampled by mailing 700 surveys to randomly selected households (20 returned as undeliverable). To increase participation, we also held in-person events in Turney to increase awareness and answer questions about this project. An ice cream social was hosted in June 2021, and a network kick-off in September 2021. Both events were held in partnership with a countywide grassroots collaborative community development organization implementing a community development approach that bolsters resident awareness and trust in the project (Gollnick, 2022). During the network kick-off event, residents could complete the pre-survey.

Ultimately, 29 households were connected between October 2021 and February 2022 (see Fig. 2 ). Beyond the in-person recruitment events, the internet service provider reached out to eligible households via door-knocking and mailers. Households that had not already completed a pre-survey were asked to complete the survey at or soon after installation. Not all interested households were able to be connected due to technical issues, such as line-of-sight (due to trees, hills, or other homes), that prevented adequate wireless signal from reaching their homes.

Fig. 2.

Map of connected and unconnected households in Turney.

For the post-survey, the same households were mailed a follow-up survey in both Turney and the control communities. The post-survey was mailed in late April 2022 to allow a minimum of three months between the pre- and post-surveys (for participants who were connected in early February 2022). Incentives for participation varied for the three groups, (1) Turney households connected to the network (referred to as connected Turney) could receive an additional month of free service if their survey was returned within three weeks, (2) Turney households that responded to the pre-survey but could not receive broadband service (referred to as unconnected Turney) could receive a $50 gasoline gift card, and (3) households from the control communities as well as any Turney households that did not respond to the pre-survey could enter a raffle for a $50 gasoline gift card. For the post-survey, households that did not participate in the pre-survey were considered to be part of the control group. A small number of responses were received from households that were not from targeted communities, e.g., due to address forwarding. When the response was within the study region, it was included. One survey response was dropped because it was from outside of the region. To increase participation in the connected Turney group, 11 certified letters were mailed to participants who had not yet completed the post-survey in May 2022.

2.5. Measures

The surveys collected information on (1) internet access at home, (2) internet use at home, (3) interest in internet access, and (4) demographics. The primary dependent variables were the use of the internet at home for employment, education, and health purposes.

To measure internet access at home, we asked a series of multiple-choice questions where participants could “check all that apply.” This included challenges using the internet at home, concerns about internet safety, types of computing devices, how they got help using the internet or a device, and how they connected to the internet at home.

To measure internet use at home, we asked a series of binary questions (yes, no, not sure) about how the household had used the internet in the last 3 months. These included activities related to employment (e.g., working remotely), education (e.g., distance learning), and health (e.g., telehealth visits). In addition, we asked about more generic online activities, such as the use of social networks, video/voice calls, streaming, gaming, online shopping, financial services, and government services. In addition, participants reported whether they had earned money or saved money due to the internet. Participants also reported any issues using the internet (multiple choice) as well as their current monthly cost (before receiving the free service as part of this intervention).

To measure interest in internet access, we asked a series of questions about how they would use improved high-speed internet service as well as expectations for their internet provider. Participants reported whether they would engage in high-bandwidth activities (e.g., home-based business, distance education, gaming) on a 5-point Likert scale that ranged from “definitely not” to “definitely would.” To evaluate digital literacy, participants reported their confidence in performing basic internet activities (e.g., searching for information, using word processing, using teleconference applications) on a 5-point Likert scale that ranged from “not confident” to “completely confident.” In addition, participants reported preferences for characteristics of their internet provider and willingness to pay for improved internet service (multiple-choice questions).

Lastly, participants reported demographics for their households as well as for themselves individually. At the household-level, participants reported household size, employment status for adults, school enrollment (K-12 and higher education), ages, and income. At the individual-level, participants reported their age, race, gender, and level of education.

In the pre-survey, Turney residents could indicate interest in participating in the internet intervention. To allow matching between pre and post surveys, participants provided their home address and the first name of the person who completed the survey. In the post-survey, some questions were removed, primarily from the interest in internet access section (confidence in performing tasks was still measured). For connected Turney households, a series of open-ended questions were added to the post-survey to solicit their experiences and how the internet intervention had influenced their use of the internet in the three areas of focus, employment, education, and health.

3. Results & discussion

3.1. Sample

As reported in Table 2 , there was a higher response rate in Turney than the control group, likely due to increased incentives to participate and recruitment efforts. As shown in Fig. 3 , the control group included participants from surrounding communities. Across control communities, response rates were fairly similar – ranging from 0% to 11% – and there was no evidence of a systematic pattern. For the pre-survey, most participants submitted it online in both Turney as well as the control group. For unknown reasons, more post-survey participants from the unconnected Turney group responded by mail, perhaps simply due to convenience. A similar proportion of survey responses were submitted online for the connected Turney group in both the pre- and post-survey.

Table 2.

Response rates for the pre- and post-survey.

	Responses (N)	Online Submissions (%)	Total Delivered (N)	Response Rate (%)
Pre-Survey
Connected Turney	28	64%	188	15% a
Unconnected Turney	26	65%
Control	35	54%	680	5%
Total	89	61%	868	10%
Post-Survey
Connected Turney	25	60%	29b	86%
Unconnected Turney	17	29%	26	65%
Control	51	51%	808c	6%
Total	93	49%	863	11%

^aResponse rate is calculated for the full Turney sample, before we knew which homes could or could not be connected.

^b1 connected Turney household did not complete the pre-survey.

^cSurveys were delivered to the full control sample from the pre-survey as well as all Turney households that did not complete a pre-survey.

Fig. 3.

Map of survey respondents in Control group.

For the Turney sample, in most cases the same individual responded to the pre- and post-survey for both the connected (21/25 = 84%) and unconnected (16/17 = 94%) samples. In the control group, there were only 10 repeat respondents (10/51 = 20%). In total, there were 135 unique respondents across both surveys and all sub-samples. The survey respondents were 54 years old on average (SD = 17, Min = 20, Max = 91). Of the respondents, 90% were White, 56% were women, and 36% had completed a Bachelor's degree or higher. A more detailed breakdown of the demographics across control communities are summarized in the Appendix in Table A1.

Twenty-two percent of all connected Turney and 56% of the unconnected Turney survey participants reported having a Bachelor's degree or higher. This is a much higher rate than the U.S. Census, which reports that 5.2% (margin of error ± 5.0%) of Turney residents 25 years and older graduated from college (Table B1). More educated households may have been more motivated to complete the survey to get improved internet access to increase opportunities to work remotely.

Overall, the Turney sample (inclusive of both the pre- and post-survey responses) was largely representative of the population based on a comparison to U.S. Census data (Table B1). The median income is $56,786, and 11% of the population live below the poverty line (U.S. Census Bureau, 2022). This is consistent with our survey data, which suggest that the median household income is $35,000-$65,000. In addition, the median age is 41 years old (U.S. Census Bureau, 2022). This is also consistent with our survey data, which suggests the median age range is 25–44 years old.

Although a high percent report internet access at home (>90%), cellular hotspots were widely used, indicating inadequate access – classic of underserved communities. The household size, number of children in K-12, and number of residents enrolled in higher education are also consistent between the pre-survey and Census data. As reported in Table B1, there were no differences in household characteristics between the connected Turney, unconnected Turney, and Control groups except for the number of employed adults per household and the respondent having a Bachelor's degree or higher. Turney tended to have more employed adults per household than the Control group and the connected Turney group had the lowest proportion of respondents with a Bachelor's degree or higher.

3.2. Quality of internet intervention

Pre-survey data suggest that Turney had lower quality internet service than the control group, consistent with their classification as an underserved community. Three Turney participants reported not having internet at home, while two participants from the control group reported so. As reported in Table 3 , a higher proportion of both connected and unconnected Turney households relied on slower technologies such as hotspot, satellite, fixed wireless, DSL, and dial-up compared to the control group. Discussions with community members suggest that a large proportion relied on cellular hotspots, specifically. As a result, more Turney households reported that their internet service was slow or unstable compared to the control group. Very few Turney households reported having no challenges using the internet at home (4%). These data suggest that the primary difference between Turney and the control group before the internet intervention was internet quality, rather than access.

Table 3.

Internet quality before and after intervention.

	Connected Turney	Unconnected Turney	Control	χ2 Stat	p
Pre-Survey
Rely on slower tech	82%	85%	47%	12.95	.002
Slow internet	68%	85%	31%	18.87	<.001
Unstable internet	61%	69%	43%	4.56	.102
Post-Survey
Slow internet	38%	82%	48%	8.61	.014
Unstable internet	50%	65%	40%	3.21	.201

In addition, pre-survey data suggest no differences in digital literacy between Turney and the control group. As reported in Appendix C, connected Turney residents reported being fairly confident (M = 4.23, SD = 1.08) on a 5-point Likert scale across a range of basic internet tasks, such as emailing, searching for information, filling online forms, and using word processing applications similar to the unconnected Turney (M = 4.07, SD = 1.04) and control groups (M = 4.14, SD = 0.83). Both groups reported using the internet similarly for applications such as social networks, video/voice calls, streaming, gaming, online shopping, and government services (see Fig. C.1 and Table C1).

The post-survey data suggest that the internet intervention improved service quality in terms of bandwidth and speed when comparing the connected and unconnected Turney groups. Fewer connected Turney households reported experiencing slow internet compared to the unconnected Turney group. However, there was not a significant difference regarding internet stability. In addition, more connected Turney households (42%) reported having no challenges using the internet at home compared to the unconnected Turney group (12%). This suggests that there was a measurable effect of the internet intervention based on improvements in speed but not in stability. The primary benefit reported by connected Turney households was the ability to use multiple devices at the same time due to increased bandwidth.

Over the duration of the study period, there were challenges with the network reliability due to technical limitations of wireless technology. Line-of-sight is required for access, which can be attenuated by rain depending on the distance from the transceiver (Anders, 2022). In addition, there were several hardware malfunctions that caused outages that lasted multiple days. As a result, the improved service quality was inconsistent throughout the study. There was a wide range of feedback (solicited biweekly) as listed below.

• •“Never fails us. Always able to hook up to internet and speed is always good.”
• •“A lot better, with the exception of occasionally losing connection during heavy rain storms.”
• •“Internet was unreliable and slow at our house. Ended up having to give up on it and return to previously used provider.”

These reliability issues may have reduced the impact of the intervention for some households and, more broadly, contribute to preferences for wired rather than wireless internet solutions.

3.3. Impact of an internet intervention

In the pre-survey, Turney residents reported using the internet for education more than the control group, despite having lower quality service. As reported in Table 4 , more Turney households reported engaging in distance learning and using the internet to do homework at home. However, there were no differences in employment and health-related activities. This may be attributed to the timing of the surveys for the two groups, since the data collection period was longer for Turney and extended into the school year. Overall, this suggests that in a between-subjects comparison, there were few differences in how Turney residents used the internet compared to others who had higher service quality. Ultimately, Turney residents reported wanting better internet service to gain quality-of-life benefits. As reported in Fig. 4 , the top intended use for improved internet service was video streaming.

Table 4.

Internet usage for the previous 3 months in the pre-survey.

Internet Use	Connected Turney	Unconnected Turney	Control	χ2 Stat	p
Employment
Work from home	42%	68%	41%	5.02	0.081
Search/apply for job	17%	33%	31%	1.8	0.408
Self-employment at home	39%	21%	13%	5.21	0.074
Education
Distance learning	42%	40%	9%	9.43	0.009
Homework at home	39%	48%	16%	7.35	0.025
Search education-related info	65%	62%	44%	3.15	0.207
Health
Search for health-related info	67%	79%	63%	1.68	0.433
Telehealth	36%	33%	29%	0.33	0.849
Use online patient portal	58%	62%	66%	0.31	0.856

Note: p < .05 is bolded.

Fig. 4.

Intended uses of improved internet by connected and unconnected Turney residents in the pre-survey. Turney residents were primarily interested in quality-of-life benefits like video streaming.

In the post-survey, there was only one significant difference in internet use for employment, education, or health in the three groups. As reported in Table 5 , unconnected Turney residents reported that they worked from home more than the other two groups. However, in general, the unconnected Turney group used the internet at a similar rate as the other groups. This suggests that other factors that are consistent on a regional level, such as social influence, may have a bigger influence on internet usage behavior than access to better service alone.

Table 5.

Internet usage for the previous 3 months in the post-survey.

Internet Use	Connected Turney	Unconnected Turney	Control	χ2 Stat	p
Employment
Work from home	29%	71%	38%	6.88	0.032
Search/apply for job	33%	21%	24%	0.92	0.631
Self-employment at home	25%	43%	17%	3.86	0.145
Education
Distance learning	42%	47%	21%	5.03	0.081
Homework at home	46%	33%	30%	1.82	0.403
Search education-related info	64%	60%	53%	0.83	0.661
Health
Search for health-related info	72%	80%	67%	1.02	0.602
Telehealth	25%	38%	29%	0.74	0.691
Use online patient portal	60%	71%	67%	0.58	0.748

Note: p < .05 is bolded.

In a within-subject comparison, there were no significant differences in employment, education, or health internet use between the pre- and post-survey responses for Turney residents (see Appendix D). Exploratory analysis reported in Fig. 5 suggests that there is weak evidence of increased internet use in the connected Turney group that can be attributed to the internet intervention. For employment, more households reported using the internet to search and apply for jobs in the connected Turney group, but this may be attributed to the shifting labor landscape more broadly. For example, despite having lower quality internet service, unconnected Turney households reported engaging in more self-employment at home in the post-survey. For education, the strongest shift was in the control group, likely due to the timing of the post-survey, which was during the school year. For health, connected Turney households reported using the internet slightly more for searching for health-related information. However, the control group also reported increases for this use, suggesting that external factors such as the evolving COVID-19 pandemic may have driven behavior. Ultimately, this suggests that a complex set of factors influence internet usage behavior beyond access and quality of service alone.

Fig. 5.

Exploratory analysis of pre (white) vs. post (black) internet activity.

Qualitative data on the impact of the internet service suggests that even though the effects were not statistically significant, there is anecdotal evidence that participants perceived benefits of the new internet service. Table 6 provides a summary of the qualitative responses regarding the impact of the improved internet intervention on employment, education, and health. Benefits were observed across all three categories. As reported in Table 7 , more participants reported benefits related to employment and education than health. For health, more participants reported no change or did not respond, than reported benefits.

Table 6.

Qualitative data on perceived benefits of internet intervention.

Employment
Work from home	“I am a teacher so I use the internet to create lesson plans daily. This program has allowed me to prep on the weekends from home.”
Search/apply for job	“Perfect, always able to have connection to find jobs or have zoom interviews”
Self-employment at home	“I have a small online business. I utilize social media platforms and etsy to advertise and sell my products. I've been able to post more and stay in contact with customers more efficiently.”
Education
Distance learning	“My daughter also takes online college classes and she no longer has issues getting assignments submitted due to poor service.”
Homework at home	“Grandson uses it for school.”
Search education-related info	“Watching more ‘how-to’ videos, in regards to cooking, fixing things, not really “higher” learning. Just more related to making one's life better.”
Health
Search for health-related info	“Look things up more about health than before.”
Telehealth	“We have been able to utilize the telehealth visit through our doctor's office. This has saved time and money. Less gas and not needing to take time off work to attend a face-to-face appointment.”
Use online patient portal	N/A

Table 7.

Summary of the qualitative responses regarding the impact of the improved internet intervention on employment, education, and health from connected Turney households.

	Employment	Education	Health
Positive impact	10	12	9
No change	8	8	10
Do not use the internet for this purpose	5	3	3
Did not answer	2	2	3

4. Implications for broadband evaluation design

Compared to unserved communities, it is challenging to evaluate the impact of broadband access in an underserved community where there are competing technologies to facilitate access (e.g., cellular hotspots). However, the difficulty in demonstrating the impact on an underserved community should not prevent decision-makers from funding broadband projects in these communities. Without the proper investments in updating internet infrastructure, underserved communities would continue to experience constraints on their economic opportunities (Philip & Williams, 2019). We have identified several limitations of our study which may help inform future broadband evaluations, (1) identifying appropriate outcome measures, (2) participant recruitment challenges, and (3) survey timing.

In terms of outcomes, our experience suggests that underserved communities may realize more qualitative quality-of-life benefits than typically quantified economic and social impacts, at least in the short-run. In Turney, residents were already accessing the internet for employment, education, and health purposes – indeed, usage behavior was largely consistent between groups and in the pre-post comparison. However, limited bandwidth prevented households from using multiple devices simultaneously (e.g., allowing two people to work remotely) and certain high-bandwidth applications (e.g., streaming). That is, the improved bandwidth created real quality-of-life impacts (such as improved ease-of-use and reduced frustration), however, we did not focus on measuring these outcomes in this study, leading to non-significant results. An implication of our research suggests future studies emphasize quality-of-life outcomes to better measure the benefits that community members perceive. Also, future studies might aim to quantify how much participants use the internet for the applications of interest instead of a binary indicator for focal activities. This can be achieved with objective (i.e., sensors) or subjective (i.e., survey response) methods. Researchers should consider how to protect the privacy of participants if using data from sensors to avoid participant recruitment challenges.

Recruiting and retaining participants is always challenging and is further complicated in small or rural communities, where there is a limited population, necessitating multiple engagement strategies to increase the sample size. For example, LaRose et al. (2011) used door-to-door in-person surveys due to a low response rate in two Texas counties. A study in Colorado combined face-to-face and online recruitment mechanisms (Colwell et al., 2018). In this study, 27 survey responses were initially collected in Turney from a direct mailing. The remaining 27 responses were recruited later after a combination of in-person events, door-to-door engagement by the internet service provider, and word-of-mouth. We extended recruitment over 7 months (August 2021 to February 2022) in Turney to (ideally) achieve a minimum sample size of 30 connected households. As a result, Turney participants were much more aware of the study's purpose, which may have influenced their responses. However, it is unclear if participants were more likely to over-estimate or under-estimate their internet use behavior to justify the provided access. Our study suggests future work use a combination of recruitment strategies, preferably ensuring that the same recruitment methods are used for all groups.

In addition, reliability issues are a major limiting factor for wireless technologies, and this reputation adversely impacts participant recruitment. Many Turney residents were hesitant to participate in this study due to previous negative experiences with wireless technologies. Ultimately, one household dropped out of the study due to frustration with inadequate reliability. However, many households, particularly those in the center of town close to the transceiver, reported very high satisfaction with the service quality.

In terms of timing, the extended recruitment period in Turney shifted the data collection period so it was not consistent between groups. While the pre-survey data collection spanned August 2021 to February 2022 for the Turney sample, all of the control group data were collected in August 2021 (i.e., the end of summer break) in response to the initial mailing. This directly affected the results, leading to significant differences in reported internet usage for the Turney sample in the pre-survey (Table 4) as well as pre-post differences in the control group (Fig. 5). As a result, it is difficult to discern the impacts on education-related internet usage. Future studies should ensure that education-related behavior is only measured during the academic school year to avoid outliers associated with the summer months or clearly specify which time period to consider in the question wording.

Further, the null results reported in this study may be an artifact of our focus on short-term impacts (prior three months), which was driven by the short period of performance required by the funder. However, it is conceivable that it takes longer than three months for impacts to emerge. Future studies would benefit from longitudinal data collection to evaluate which types of impact are observed over time, and funders should consider the need for longitudinal data collection in developing requests for proposals.

Overall, the results of this study have external validity and can be generalized to other small underserved rural communities. According to the 2020 Census, 75% of U.S. residents live in small towns (defined as < 5000 people), and of those, 42% have less than 500 people (U.S. Census Bureau, 2020). These communities (<500 people) are most likely to have insufficient market dynamics to spur private investment in infrastructure and be classified as underserved.

5. Conclusion

This study evaluates the impact of an intervention that provided faster, higher bandwidth wireless internet access in a small underserved rural community. Results suggest that the intervention was associated with quality-of-life benefits, principally reduced frustration due to the ability to use multiple devices at once. We do not find evidence of changes in internet use for employment, education, or health attributable to the intervention. These results have implications for the design of future evaluation studies. This evaluation is relatively unique because it examines economic and social as well as quality-of-life benefits of an internet intervention in a single community, over a short time period. Additionally, the use of pre-post data from demographically similar nearby communities is a contribution to the evaluation literature. Some of the interested participants in the target community (Turney) were unable to be connected to the network due to technical constraints, creating an additional comparison group, further contributing to a relatively unique set of conclusions.

First, there were no significant within-subject differences in internet usage behavior for connected Turney households after the internet intervention. However, there was evidence that households had fewer issues accessing the internet due to the intervention. Moreover, participant feedback indicated that they used the internet more, but in similar ways, as they had before the intervention. For underserved communities, it may be more common to see these types of marginal benefits compared to unserved communities. These results suggest that other factors beyond access and quality of service influence internet usage behavior. Given the consistent internet usage behavior across groups, it is likely that regional forces such as social influence and market conditions are more influential. For example, regions may vary in terms of the expectations regarding how much is accomplished online. In areas with poor internet access, the local school system and local government may not have a system in place to collect information via online forms. Thus, there may not be an option to submit online homework or pay taxes online. In general, the Turney sample had average digital literacy, suggesting that this was not a major barrier to usage. Other studies have found that digital literacy training, access to affordable devices, and subsidies are key programmatic elements for increasing adoption and changing internet usage behavior (LaRose et al., 2011; Whitacre et al., 2015).

Second, instead of achieving impact measures related to employment, education, and health, participants were primarily motivated to get better internet to achieve quality-of-life benefits. Most Turney participants intended to use improved internet service for video streaming and gaming, applications that benefit from higher bandwidth because it means that one household member can be streaming without eliminating access for the remaining household members. Only half of the Turney sample was interested in distance education, telehealth, and telecommuting. Quality-of-life can be an important consideration for small rural towns that are competing with other similar towns for residents and employers. For example, if a young couple is choosing whether to move back to their hometown or the next town over, they are highly likely to consider broadband access in their decision (Low et al., 2022). Similarly, new firms also consider broadband access when making decisions about starting operations in a location (Kim & Orazem, 2017; Krause & Reeves, 2017).

This study had three primary limitations, which have implications for future broadband evaluations: identifying appropriate outcome measures, recruitment challenges, and survey timing. Our results suggest that future studies aim to quantify participant quality-of-life benefits and the quantity of internet use for the applications of interest, to the extent possible. Further, our multi-tiered recruitment approach buoyed participation and should be considered by others doing in-depth, community-level studies.

We hope that the policy implications from our study can help guide additional evaluation research as policymakers strive to improve internet connectivity. High-speed broadband infrastructure is increasingly becoming necessary for communities to remain economically competitive. Strong evaluations are critical to ensure that government funds are effective and can inform subsequent tranches of infrastructure spending. Community-level evaluations such as this one help local elected officials and local decision-makers better understand and anticipate how broadband can and cannot impact their community.

Declarations of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Control community selection

The control communities included: Agency, Cowgill, Easton, Edgerton, Elmira, Holt, Kidder, Kingston, Osborn, Polo, Rayville, Stewartsville, and Trimble. They were identified based on total population, number of households, percent of adults with a bachelor's degree or higher, percent of 16+ who are in the labor force, mean commute time, percent of self-employed individuals, median household income, percent below poverty, and percent of households reporting internet access using cable, fiber or DSL (Digital Subscriber Line) (see Table A1).

Table A.1.

Summary of Turney demographics compared to control communities.

Description	Turney	Osborn	Stewartsville	Easton	Trimble	Edgerton	Holt	Elmira	Rayville	Agency	Cowgill	Kidder	Kingston	Polo
Population	114	374	733	227	573	601	471	39	157	671	168	267	290	509
Households	76	200	269	129	248	239	126	8	95	249	81	130	106	201
Employment rate	65.1%	67.1%	65.9%	49.0%	61.4%	63.8%	61.9%	68.8%	39.5%	73.7%	41.4%	57.5%	36.0%	57.2%
Income ($)a	56,786	75,306	56,250	50,170	63,125	49,135	58.889	N	N	82,981	39,861	49,167	43,500	39,148
Bachelor's degreeb	5.2%	28.9%	16.8%	3%	12.4%	17.9%	13.3%	13.3%	0%	24.7%	10.1%	6.2%	9.0%	8.7%
% population 65+	4.6%	28.9%	9.7%	20.1%	15.2%	20.3%	14.8%	14.3%	8.1%	12.4%	21.5%	13.1%	13.2%	12.7%
% poverty	11.6%	3.9%	12.2%	11.1%	7.7%	18.2%	11.6%	0.0%	0.0%	1.0%	26.3%	16.6%	21.7%	12.7%
% Self-employedc	13%	1.0%	12.3%	24.2%	7.7%	8.7%	4.1%	0.0%	0.0%	4.4%	7.5%	16.2%	11.2%	11.6%
Commute (min)d	21.8	24.5	24.9	31.3	27.6	41.2	34.2	N	N	25.0	31.1	30.0	32.9	32.6
% White	91.2%	91.4%	93.5%	90.0%	90.8%	92.5%	88.5%	84.6%	78.3%	88.7%	89.3%	96.3%	79.7%	91.6%
% internete	39.5%	37.5%	62.5%	28.7%	47.6%	56.1%	59.5%	50.0%	20.0%	75.5%	60.5%	36.2%	49.1%	38.8%

Note: Data from the 2021 American Community Survey 5-Year Estimates; margins of error are not reported but were considered. N = data not available.

^aMedian household income.

^bBachelor's degree or higher.

^cSelf-employed is defined as working in own incorporated or non-incorporated business.

^dAverage travel time to work.

^ePercent with a broadband internet subscription via cable, fiber optic or DSL.

Appendix B. Sample characteristics

Table B.1 summarizes the comparison of household characteristics for the 3 sub-groups versus those reported by the ACS for Turney. The statistical tests reported in Table B.1 compare the 3 sub-groups. When comparing counts, an F-test was used. When comparing proportions, a chi-squared test was used.

The survey data were different than the U.S. Census in 2 ways, which made comparisons less informative. The surveys collected employment status for ages 18+, but the U.S. Census collects data for ages 16+. In addition, the ACS collects educational attainment data for all adults 25 years and older. The pre and post-survey only collected educational attainment for the adult responding to the survey.

Table B.1.

Summary of Turney demographics compared to U.S. Census.

Household characteristic	Turney (Census)	Connected Turney	Unconnected Turney	Control	Stat	p
Household size	3.12	2.8 (1.2)	3.2 (1.5)	2.6 (1.6)	1.00a	.371
Children enrolled in K-12	0.78	0.50 (1.14)	0.85 (1.19)	0.63 (1.06)	0.652a	.524
Residents in higher-ed	0.3	0.39 (0.83)	0.19 (0.49)	0.20 (0.53)	0.926a	.400
Employed adults/household	1.37	1.54 (1.1)	1.69 (0.84)	1.09 (0.95)	3.28a	.042
Bachelor's degree or higher	5.2%	22% (42%)	56% (51%)	42% (50%)	6.30b	.043
Household income of >$35 K	$56,786	68% (48%)	86% (36%)	81% (40%)	2.19b	.335
Internet service at homec	92%	93% (26%)	96% (20%)	94% (24%)	0.278b	.870

Note: values are reported as mean (standard deviation), p < .05 is bolded.

^aF-statistic.

^bChi-squared statistic.

^cAny broadband, including cellular.

Appendix C. Confidence Performing Common Online Tasks

Fig. C1 presents the input from Turney households on how confident they felt performing a series of everyday activities online, such as shopping online, searching for information, and participating in teleconference meetings. Fig. C2 presents the same input from Control households. There were few systematic differences. Table C.1 suggests that there were no differences in how participants used the internet for common online tasks between the two pre-survey groups.

Fig. C1.

Confidence of connected and unconnected Turney households performing common online tasks in the pre-survey.

Fig. C2.

Confidence of Control households performing common online tasks.

Table C.1.

Internet usage for the previous 3 months in the pre-survey.

Internet Use	Connected Turney	Unconnected Turney	Control	χ2 Stat	p
Email	84%	100%	100%	9.59	0.008
Social networks	84%	96%	94%	2.66	0.265
Video/voice calls	72%	83%	68%	1.76	0.416
Streaming	88%	88%	78%	1.43	0.49
Gaming	58%	54%	59%	0.16	0.922
Online shopping	88%	100%	97%	4.23	0.121
Financial services	68%	96%	97%	13.18	0.001
Reading news	76%	84%	81%	0.53	0.768
Government services	54%	71%	63%	1.43	0.489

Note: p < .05 is bolded.

Appendix D. Within-subject comparisons

Turney residents who received improved internet access did not report a change in usage for work, education, or health applications. Table D.1 summarizes the statistical analysis. The McNemar test, also known as a paired chi-square, provides a way of testing hypotheses for subjects who participate in an intervention. The three main assumptions for the test include (1) having a nominal variable with two categories and one independent variable with two connected groups, (2) the two groups in your dependent variable must be mutually exclusive, and (3) the sample must be a random sample.

Table D.1.

Within-subject comparison (pre vs. post) for connected Turney households

Internet Use	Pre	Post	χ2 Stat	p
Employment
Work from home	48%	30%	3.2	0.074
Search/apply for job	15%	30%	1.33	0.248
Self-employment at home	35%	22%	0.8	0.371
Education
Distance learning	38%	39%	0.25	0.617
Homework at home	40%	43%	0	1
Search education-related info	65%	62%	0	1
Health
Search for health-related info	67%	71%	0	1
Telehealth	32%	26%	0	1
Use online patient portal	57%	62%	0.5	0.48

Data availability

The data are available in an Open Science Framework repository https://osf.io/wutxk/.