Broadband Data Forensics: Spatiotemporal Variations of the Download/Upload Speed Metric Commonly Used to Evaluate Potential Telehealth Accessibility

Abstract

Introduction:

The complicated task of evaluating potential telehealth access begins with the metrics and supporting datasets that seek toevaluate the presence and durability of broadband connections in a community. Broadband download/upload speeds are one of the popular metrics used to measure potential telehealth access, which is critical to health equity. An understanding of the limitations of these measures is important for drawing conclusions about the reality of the digital divide in telehealth access. The objective of this study was to assess spatiotemporal variations in broadband download/upload speeds.

Method:

We analyzed a sample of data from the Speedtest Intelligence Portal provided through the Ookla for Good initiative.

Results:

We found that variation is inherent across the states of Vermont, New Hampshire, Louisiana, and Utah.

Conclusions:

The variation suggests that when single measures of download/upload speeds are used to evaluate telehealth accessibility they may be masking the true magnitude of the digital divide.

Introduction

Since the COVID-19 pandemic emerged, telehealth has been at the vanguard of health care due to its potential to serve patients remotely while in-person care was considered high risk.^1–3 As the pandemic unfolded, researchers, clinicians, and health systems worked diligently to evaluate the barriers and facilitators to use, specifically exploring provider and patient experiences, learning that the foundation of continued telehealth practice is grounded in the stability of broadband connectivity serving both the hospital catchment area and the patient communities.^1,4–13

What has not been considered is the inherent variation of the underlying metrics used to characterize broadband stability and what that variation means for accurately capturing potential telehealth accessibility. Often described as the “digital divide,”^5,11,14 a community's experience/use of broadband infrastructure—or lack thereof—can greatly affect the ability for remote, rural patients to receive necessary care in a manner that is accessible, convenient, and reliable. Given that policymakers' commitment to sustaining policies and regulatory protections for telehealth,¹⁵ characterizing this variation is vital for appropriately measuring potential telehealth access.¹⁴

For example, in health services research, one of the proposed methods for assessing geographic access to telehealth is the 2-Step Virtual Catchment Area¹⁶ where the underlying metrics used for evaluating adequate broadband connectivity to support telehealth rely on the “download/upload” speeds that are reported to the Federal Communications Commission (FCC) by Internet Service Providers (ISP) because it is commonly reported that a minimum download/upload speed of 25/3 megabits per second (Mbps) is adequate for delivering telehealth services.^4,17–19 However, there are important limitations associated with this definition of “adequate” download/upload speeds and the FCC data used to support it.

Most notably, FCC data are collected for business purposes related to regulatory rules associated with ISP companies. The FCC does not collect data for measuring spatiotemporal stability for specifically examining delivery of telehealth services. In fact, the FCC requires ISP to provide the “minimum advertised speed” which is a spatiotemporally static metric captured by a broadband company outside of a specified location, reported at periodic intervals throughout the year, and may not necessarily reflect the consumer's realized experience.¹⁹ Additional considerations related to the ability to reliably estimate a consumer's actual broadband experience are factors that affect durability of connection aside from speed. If an instantaneous speed is allegedly “fast,” but the connection is interrupted, it creates an inefficient, frustrating, and potentially ineffective experience for both provider and patient.^4,9

Additional factors associated with consumer experiences include their method for accessing the internet, “fixed broadband” (desktop and/or noncellular Wi-Fi) or “cellular broadband” (mobile phone, tablet, or other remotely connected device), and/or the variability of the underlying physical infrastructure supporting that access (Fiber, Digital Subscriber Line, modem, etc.),²⁰ not to mention the number of users in the household, the number and type of devices, demand on local/regional servers, and seasonal factors (e.g., inclement weather) that may affect reliable use. These additional factors obfuscate the ability to reasonably predict the potential access of either a synchronous or asynchronous telehealth appointment and likely masks the true magnitude of the digital divide¹³ as it relates to health care access.

The extent of variations is evident both spatially, based on various geographies, and temporally through hourly and daily fluctuations. That said, health service researchers have few readily available options other than the download/upload speeds from FCC, or combinations of private and public data that work to augment FCC data such as the National Telecommunications and Information Administrations data (NTIA),²¹ available to characterize the underlying infrastructure that supports telehealth. Given these limitations affecting the durability of broadband connectivity, it is important to demonstrate the inherent variations in the basic metric of download/upload speeds. Understanding the variations will encourage researchers to reach appropriate conclusions, as well as advocate for reporting additional spatiotemporal connectivity measures to improve evaluation of telehealth accessibility.

A Case Study: Consumer-Initiated Ookla for Good™ Speedtest Intelligence^® Data Across Four U.S. States by Day and Time

Since the FCC download/upload data are not available as a temporally detailed dataset (e.g., by subcounty location, hour, week, day, or time), we secured a limited use dataset from Ookla for Good, which included access to their Speedtest Intelligence data portal²² as a means to demonstrate inherent variation in these metrics. The dataset includes a 90-day rolling lookback beginning December 5, 2022, until January 21, 2023, of consumer-initiated internet speed tests from users across Louisiana, New Hampshire, Vermont, and Utah. The selection of these specific four states is to demonstrate the variation of the download/upload speeds across varied geographies (Southeast, Northeast, and Mountain West).

The Speedtest Intelligence data include both fixed and cellular data for the Best 10%, Median, and Worst 10% reported hourly and daily download/upload speeds summarized weekly for each geography. The results of the variation of consumer download/upload speeds are shown below in panel figures to first demonstrate hourly variation in a single day across four states through the following measures: an example of Monday Variation for both cellular and fixed for Best 10%, Median, and Worst 10% (Fig. 1 and 2 ); and then a close look at the interquartile range for Utah's cellular download/upload speeds over the course of a week (Fig. 3). Variation in the download/upload speeds is evident regardless of whether the data are compiled by fixed or cellular broadband, or if summarized hourly or daily (Fig. 1). The majority of the variation for all four states is in the hourly assessment of Best download speed, most notably for cellular data.

Fig. 1.

Figure includes lines that represent the Best 10%, Median, and Worst 10%, and Median 10% upload and download cellular speeds for Louisiana, New Hampshire, Vermont, and Utah. The Best 10% line is a blue solid line, the Median is a red thickly dashed line, and the Worst 10% is thin yellow dashed line.

Fig. 2.

Figure includes lines that represent the Best 10%, Median, and Worst 10%, and Median 10% upload and download fixed speeds for Louisiana, New Hampshire, Vermont, and Utah. The Best 10% line is a blue solid line, the Median is a red thickly dashed line, and the Worst 10% is thin yellow dashed line.

Fig. 3.

Figure is a side-by-side box plot including the variation of the download speeds (left side of the panel) and the upload speeds (right side of the panel). The colors of the box plots are to differentiate the days of the week.

For instance, in Louisiana, the Best 10% download/upload speed for cellular ranges from a maximum of 600/33 Mbps at 3 am to a minimum of 260/7 Mbps at 1 pm. For fixed broadband, the Best 10% upload speeds also show large hourly variation. For example, in Louisiana, the Best 10% download/upload speed for fixed ranges from a maximum of 801/206 Mbps at 3 am to a minimum of 556/277 Mbps at 7 am. While examining the Best 10% download/upload speeds demonstrates variation, these scores are more than adequate to meet the minimum 25/3 Mbps for telehealth access. To demonstrate the potential impact on disparities in telehealth access, this article will focus on the difference from the Best 10% speeds as compared to the Median and Worst 10% download/upload speeds for both cellular and fixed. For example, for every state, the minimum cellular download speeds for Best 10% (LA: 260 Mbps, NH: 119 Mbps, VT: 62 Mbps, UT: 359 Mbps) to Median (LA: 29 Mbps, NH: 31 Mbps, VT: 20 Mbps, UT: 157 Mbps) and Worst 10% (LA: 3 Mbps, NH: 4 Mbps, VT: 2 Mbps, UT: 4 Mbps) demonstrate a dramatic drop in values that exceed the necessary minimum (Best 10%) for broadband connectivity to barely meeting (Median), if at all (Worst 10%), the minimum scores necessary for telehealth access.

Significance and Potential Impact

As demonstrated, there is spatiotemporal variation in the download/upload speed data associated with evaluating the potential access for telehealth services and depending on the metric used for evaluation the assessment may be skewed. For example, if one uses the Best 10% of self-initiated speed test results in evaluating broadband infrastructure, characterization of the potential digital divide may be greatly underestimated, leading to the false impression that access is within an acceptable range in a particular geography. Conversely, if the Worst 10% of download/upload speeds are used, this may result in better characterizing where disparities in telehealth access likely exist. Even Median download/upload speeds demonstrate erratic daily variation that could lead to interruption in the user experience. This variation demonstrates that the current “minimum advertised speeds” in the FCC data, and any FCC data augmentation (e.g., NTIA), using the 25/3 Mbps standard for evaluating telehealth capacity, have limitations that should be taken into consideration when identifying areas for strengthening broadband infrastructure.

While Ookla for Good data are self-reported that suggests a more real-world user experience, it has its own limitations. First, the self-report user is likely different than a typical consumer who may not be familiar with when and how to use a speed test for broadband durability. We are unable to report on these demographics due to the restrictions in the limited use license agreement. Additionally, the Ookla for Good data is for a rolling 90-day period that covers only one “season.” It is possible that additional seasonal variation exists and is not characterized in this case study. Regardless of these limitations, the ability to characterize the inherent variation, and spatiotemporal volatility, of download/upload speeds is crucial to appropriately identifying and characterizing disparities in potential telehealth access.

We conclude that broadband measures that more accurately reflect spatiotemporal durability are needed to avoid unintentionally obfuscating the true magnitude of the digital divide. This is crucial for rural communities that could benefit most from increased telehealth accessibility. Policymakers requiring ISP to report metrics should press regulatory organizations such as FCC to require reportable metrics that better characterize the actual spatiotemporal durability of upload/download speeds. To do so would better position researchers to create measures of accessibility that calculate the variation over time, for example, a “space time cube”²³ as a means to create a metric of “durability” that better characterizes spatiotemporal stability of broadband connectivity. Thus, empowering policymakers, advocates, and researchers to more accurately identify areas of internet disparities can close infrastructure gaps in telehealth access.

Authors' Contributions

J.A.-T.: conceptualization, methodology, data and analysis, writing—original draft, data curation. F.W.: methodology, writing—reviewing and editing. E.M.: writing—reviewing and editing. L.C.: data curation, writing—reviewing and editing. R.E.S.: writing—reviewing and editing. A.N.A.T.: writing—reviewing and editing. T.O.: writing—reviewing and editing, funding acquisition.

Data Statement

The data curated in this brief communication are covered under a limited use agreement with Ookla© through its Ookla for Good program and therefore are not available for this article submission.

Disclosure Statement

No competing financial interests exist.

Funding Information

The work of this article has been supported by a grant from the National Cancer Institute (R01CA267990) and an Ookla^® limited-use license agreement via the Ookla for Good program.