Lessons from West Virginia’s Pandemic Response

Abstract

In this editorial discussion we describe our experience developing and implementing predictive models during the pandemic response in the state of West Virginia. We provide insights the on the importance of communication and the dynamic environment that exists that impacts predictive modeling in situations such as those that we faced. It is our hope that this work brings insight to those who may experience similar challenges while working in public health policy.

“Is there enough to cover hospitals and long-term care facilities?” This is the question that kept us awake at night during the Spring of 2020. Our team that was tasked with forecasting the PPE needs for the state includes academics and researchers from West Virginia University and the West Virginia National Guard. In a state with a high percentage of immunocompromised populations, our priority was to maintain sufficient medical capacity to address the pandemic by ensuring healthcare staff had adequate personal protective equipment (PPE). Our involvement with the West Virginia state PPE pandemic response provides an interesting case-study of the intersection of predictive modeling and decision making in public health policy.

Our experience taught us the importance of three salient issues in building decision-maker trustworthiness in our predictive models that is so critical to grounding public healthcare policy decision making on sound scientific guidance. These include (a) timely and precise communication of modeling features and changes to decision makers, and (b) curating data for a dynamic and uncertain environment. The results of our predictive model are fed to a centralized team which includes the Federal Emergency Management Agency (FEMA), West Virginia Department of Health and Human Resources, and others coordinating PPE requirements at the state level. In addition to our analytical results, this team also considered daily state level projections from federal counterparts along with, albeit to a lesser extent, the ubiquitous media descriptions of publicized projections. Based on these inputs the team would made recommendations to policy makers for resource allocations and purchases around the state using tax payer dollars.

Our model uses a three-week horizon to predict PPE needs for every hospital and long-term care facility in the state of West Virginia. The model takes daily hospital census of COVID-19 patients, both confirmed and symptomatic tests, and weekly PPE supply information to calculate usage rates, and combines with hospital market share information and severity parameters in that given market to forecast PPE needs for a specific hospital. The model details itself are unremarkable. It is important to acknowledge here that access to open source data fueled an explosion of epidemiological models and forecasting tools during the pandemic. Some short-term and long-term modeling approaches made the jump from social- to mass-media gaining acclaim and widespread use attracting the attention of our state officials. However, the local conditions in our state influenced our deliberate selection of a data framework and modeling consideration that led to our unique approach. Specifically, the need for local information necessitated by the state’s unique socio-economic characteristics that drive healthcare disparities that are exacerbated by geographical characteristics, which influences rural and urban population’s locational choices for healthcare interventions. The purpose of accounting for these local characteristics is understanding PPE needs and usage at specific hospitals that are vastly different from other models the state PPE response team were seeing in the mass media.

These differences fueled the realization of our need for precise, timely communicate of the unique modeling considerations necessitated by our state to the PPE response team and policy makers. For example, in certain visualizations, the forecasts from our models would be aggregated to show regional, and overall needs for the state. However, policy makers had a difficult time comparing these more granular forecasts, to mass-media models that masked unique differences in populations behaviors apparent in our state. These include differences between rural and urban populations’ choice of locations for seeking healthcare. Once policy makers understood these modeling differences the true value of the models became clear. Thus greater trustworthiness of our results was achieved and willingness to implement the resulting policy recommendations, which was critical in subsequent data challenges.

Entering the project we found limited state databases with the information needed for our proposed model. We advised the design of a data collection information system that included data types, survey questions, timing information collection, among other data including hospital capacity information. Consequently, we designed our model inputs to match the data collected by this information system. With our model and IS built, national interest in PPE shortages grew increasing federal scrutiny of and guidelines on hospital PPE reporting. To reduce the undue burden of dual reports hospitals in the state began reporting only the Federally mandated data, which resulted in new data structures that impacted our model results. For instance, we originally required hospitals to report actual PPE on hand and PPE days of supply disaggregated at the item level. Federal requirements changed reporting from actual numbers to ranges of values or binary inputs. Employing the new Federal PPE use data categorized into ranges the trends and relationships detected in our model before the data change, became much smaller or disappeared completely. Academics and practitioners are accustomed to consistent data collection standards. However, these standards may be overtaken by other priorities out of the researchers’ control, affecting model’s impacts during a project. Consequently, where feasible strategies must be considered to combat data proliferation issues during implementations. As data change, changing model output, which if not discussed with policy makers and prefaced with background information will erode trust in model results. Thus, trust built from communication with policy makers and the ability to discuss changes in the models (e.g. changes in data vs. changes in policy), and their effect on results and the subsequent policy recommendations cannot be minimized.

In hindsight, employing model creation best practices such as version control of code was critical in understanding the impact data or policy changes had on the model. In addition, our use of open source software enabled a level of inter-organizational transparency that also helped with trust building and auditing changes. All of these tools combined with strategic communication allowed us to have some level of success during this project. Using similar strategies we have been able to build on this success of using open source models and data science tools to assist in vaccine operations. We hope our experiences help others achieve a higher level of success if tasked with similar circumstances.