Advantages and disadvantages of using theory-based versus data-driven models with social and behavioral determinants of health data

Abstract

Objective

Theory-based research of social and behavioral determinants of health (SBDH) found SBDH-related patterns in interventions and outcomes for pregnant/birthing people. The objectives of this study were to replicate the theory-based SBDH study with a new sample, and to compare these findings to a data-driven SBDH study.

Materials and Methods

Using deidentified public health nurse-generated Omaha System data, 2 SBDH indices were computed separately to create groups based on SBDH (0–5+ signs/symptoms). The data-driven SBDH index used multiple linear regression with backward elimination to identify SBDH factors. Changes in Knowledge, Behavior, and Status (KBS) outcomes, numbers of interventions, and adjusted R-squared statistics were computed for both models.

Results

There were 4109 clients ages 13–40 years. Outcome patterns aligned with the original research: KBS increased from admission to discharge with Knowledge improving the most; discharge KBS decreased as SBDH increased; and interventions increased as SBDH increased. Slopes of the data-driven model were steeper, showing clearer KBS trends for data-driven SBDH groups. The theory-based model adjusted R-squared was 0.54 (SE = 0.38) versus 0.61 (SE = 0.35) for the data-driven model with an entirely different set of SBDH factors.

Conclusions

The theory-based approach provided a framework to identity patterns and relationships and may be applied consistently across studies and populations. In contrast, the data-driven approach can provide insights based on novel patterns for a given dataset and reveal insights and relationships not predicted by existing theories. Data-driven methods may be an advantage if there is sufficiently comprehensive SBDH data upon which to create the data-driven models.

BACKGROUND AND SIGNIFICANCE

Social and behavioral determinants of health (SBDH) are known to impact health outcomes.^1,2 SBDH are defined as the environmental context and social conditions in which people live such as geographic location, income, and access to resources.^1,3 Behavioral determinants include but are not limited to diet, exercise, sleep routines, substance use and abuse, and risky behaviors.³ Existing reports have shown approximately 80% of an individual’s health is influenced by SBDH.^3–5 Care delivery models seek to incorporate whole-person health assessments, including SBDH, to provide a basis for tailored interventions to optimize health outcomes and to address social needs (eg, income or food insecurity) that impact overall health.^6–8 Whole-person care takes into account a person’s environment, physical and psychosocial aspects, and health-related behaviors.^6,9 Toward optimal whole-person care, it is essential to consider data collection, measurement, and analysis needs related to SBDH.

In 2014, the Institute of Medicine (IOM) (now the National Academies of Medicine [NAM]) advanced the notion that EHRs should adopt a standardized approach to SBDH documentation and data reuse to advance clinical care and improve population health.^2,10,11 Efforts to achieve this goal have proliferated since that time such as the NAM Futures of Nursing Report 2020–2030: Charting a Path to Achieve Health Equity described SBDH as a key focus for nursing leadership and policy to improve health outcomes for patients and communities.¹¹ Additionally, the revised National Institute of Nursing Research (NINR) framework includes SBDH as a core research lens.¹² Numerous approaches for SBDH assessments have proliferated and are currently in use. Specific examples of SBDH data include the Social Interventions Research and Evaluation Network (SIREN) project, a repository of SBDH assessment tools and the Gravity Project, a national collaborative to build consensus-based data standards for SBDH.^13,14

Despite all these efforts, a recent report identified key challenges to the data collection, integration, and use of SBDH data.¹⁵ These include lack of data standards and integration into patient records; inadequate training and education how to capture, collect, and use SBDH data; and limited use of the data to enable 2-way communication between health providers (systems) and community-based organizations.¹⁵ Standardized nursing terminologies such as the Omaha System have demonstrated effectiveness as informatics approaches for addressing these challenges.^16–18 The Omaha System, multidisciplinary standardized terminology, consists of 3 valid instruments, the Problem Classification Scheme, the Intervention Scheme, and the Problem Rating Scale for Outcomes.¹⁹

The Omaha System Problem Classification Scheme is a rigorous, psychometrically sound assessment used to document all of health, including SBDH signs/symptoms.¹⁹ It is a taxonomy of 42 defined problems and their respective signs/symptoms. There are 335 Omaha System signs and symptoms that assess whole-person health across Environmental, Psychosocial, Physiological, and Health-related Behaviors Domains.^20,21 Of the 335 signs/symptoms, over half (n = 187, 56%) are in the Environmental (40), Psychosocial (85), or Health-related Behaviors Domains (62). By definition, these are social or behavioral determinants of health given that they are outside of the Physiological Domain (n = 148) in which the signs/symptoms are for problems of the body or physical health. The Omaha System is widely used by public health nurses (PHNs) to guide and document care.¹⁹

The Omaha System Problem Rating Scale for Outcomes consists of 3 Likert-type ordinal scales for each of the 42 problems: Knowledge (1 = no knowledge, 5 = superior knowledge), Behavior (1 = not appropriate, 5 = consistently appropriate), and Status (1 = extreme signs/symptoms, 5 = no signs/symptoms). As with the original study, in this replication study, we focused on the Knowledge outcome due the age and overall health status of this population.^20,22,23

The original theory-based study mapped NAM/IOM-recommended SBDH factors to the Omaha System (Table 1) and examined client outcomes for a sample of pregnant/birthing people served by PHNs using existing practice-generated data (N = 4263).²⁰ The clients were stratified into groups by a calculated SBDH index based on the SBDH factors. Several SBDH-related patterns were discovered for this vulnerable group related to racism, income needs, and alcohol use. Building on this work, 2 separate studies addressed the need for algorithm development to identify and serve adults with SBDH and frailty vulnerabilities. Gao et al¹⁶ examined assisted living clients (N = 1618) and identified relationships between SBDH and frailty for older adults along a continuum of frailty. Building on Gao’s work, Lu et al¹⁸ examined community-generated Omaha System data for older adults receiving home care (N = 4322), and confirmed Gao’s findings in creating a new Omaha System-based algorithm for frailty. In addition to the unique findings for each of the studies, important findings were the same across diverse populations and Omaha System data sources: First, as the number of SBDH items increased, interventions increased; and second, as the number of SBDH items increased, outcomes worsened.

Table 1.

Mapping of NAM/IOM SBDH factors to Omaha System signs/symptoms

Problem	Sign/symptom	SBDH measure
Income	Able to buy only necessities	General resource strain (+ for somewhat hard)
Income	Low/no income	General resource strain (+ for very hard or somewhat hard)
Income	Difficulty buying necessities	General resource strain (+ for very hard)
Social contact	Limited social contact	NHANES III (+ for yes answers to Q1 or Q2)
Social contact	Minimal outside stimulation/leisure time activities	NHANES III (+ for yes answers to Q3 or Q4)
Mental health	Loss of interest/involvement in activities/self-care	PHQ-2 (+ for Question 1)
Mental health	Sadness/hopelessness/decreased self-esteem	PHQ-2 (+ for Question 2)
Mental health	Difficulty managing stress	Single item measure of stress symptoms (+ for score of >1)
Abuse	Attacked verbally	HARK (+ for “H”)
Abuse	Consistent negative messages	HARK (+ for “H”)
Abuse	Fearful/hyper vigilant behavior	HARK (+ for “A”)
Abuse	Assaulted sexually	HARK (+ for “R”)
Abuse	Welts/bruises/burns/injuries	HARK (+ for “K”)
Physical activity	Sedentary lifestyle	Exercise vital sign (+ for inactive)
Physical activity	Inadequate/inconsistent exercise routine	Exercise vital sign (+ for insufficiently active)
Substance use	Smokes/uses tobacco products	NHIS (+ for current every day or current some day smoker)
Substance use	Abuses alcohol	AUDIT C (+ for score of greater than 4 for men or 3 for women)

NHANES: National Health and Nutrition Examination Survey; PHQ2: Patient Health Questionnaire; HARK: Humiliation, Abuse, Rape, Kick Assessment; NHIS: National Health Interview Survey; AUDIT C: Alcohol Use Disorders Identification Test.

These studies demonstrate the value of standardization to examine SBDH using the NAM/IOM-recommended SBDH factors. Building on this work, we aimed to compare the NAM/IOM-recommended SBDH index to a data-driven selection of SBDH factors that best predict outcomes. As previously described, the 17 NAM/IOM-recommended signs/symptoms are a small subset of the available 187 Environmental, Psychosocial, and Health-related Behaviors signs/symptoms in the Omaha System.² In this replication and comparison study, we focused on pregnant/birthing people with mental health concerns, given the known importance of SBDH factors for this population.^24–26 This is a priority population in which SBDH factors have long-term health outcomes for both pregnant/birthing people and their children.^24,26 This further aligns with international initiatives to improve maternal health outcomes and support the use of data standards.^25,27

OBJECTIVE

We examined a new dataset that comprised PHN-generated data for pregnant/birthing people with at least one mental health sign/symptom for influences of SBDH on client outcomes using 2 approaches. The objectives were: (1) using the existing SBDH index, to replicate the original theory-based study in the new sample; (2) to create and use a data-driven SBDH index derived from all signs/symptoms and to repeat the replication analysis with the same sample; and (3) to describe the predictive abilities of theory-based and data-driven models for baseline knowledge scores.

MATERIALS AND METHODS

This retrospective, replication, comparison study was deemed exempt from review by the University Institutional Review Board (00004498-2018, 1606E88542-2016). The Clinical & Translational Science Institute Best Practices Integrated Informatics Core (BPIC) granted access to the data shelter on May 27, 2020, where all analyses were completed. The Omaha System Data Collaborative exists within the University’s Academic Health Center Secure Computing Environment.

Sample

The dataset was selected to match the data from the original SBDH study, with an emphasis on mental health as a priority population for public health nursing intervention.^26,28 The original sample included pregnant/birthing people (N = 4263) with an average age of 23.6 years (SD = 6.1), married (21.4%), and white (39.1%). The Health Insurance Portability and Accountability Act (HIPAA)-compliant, deidentified data were generated through routine documentation of PHNs.

Instrument

The Omaha System is a comprehensive multidisciplinary standardized health terminology that is integrated within SNOMED CT and LOINC.¹⁹ It consists of 3 instruments with documented psychometric properties: the Problem Classification Scheme, the Intervention Scheme, and the Problem Rating Scale for Outcomes.¹⁹ The Omaha System provides a framework to support clinical documentation of assessments and interventions, and enables health outcomes measurement.¹⁹ Documentation using the Omaha System generates standardized interoperable data across settings and systems.

The problem classification scheme

The Problem Classification Scheme is a comprehensive, mutually exclusive classification and instrument that consists of 42 nonoverlapping problem concepts taxonomically arranged within 4 Domains (Environmental, Psychosocial, Physiological, and Health-related Behaviors).¹⁹ Each problem is defined and has a unique set of signs/symptoms, ranging from 3 to 19 per problem. For example, the sign/symptom low/no income is found only in the Income problem within the Environmental Domain, while the symptom language barrier is found only in the Communication with community resources problem within the Psychosocial Domain. The Problem Classification Scheme enables practitioners to identify health-related concerns objectively and efficiently, organize data elements, and help identify relationships and trends within the data.¹⁹

Intervention scheme

The Intervention Scheme is a comprehensive classification used to describe practitioners’ actions and activities. Each intervention addresses a problem from the Problem Classification Scheme, described using a category (action) term, a defined target term that further specifies the action, and a care description term (customizable). The 4 category terms are: (1) Teaching, Guidance, and Counseling, (2) Treatments and Procedures, (3) Case Management, and (4) Surveillance.¹⁹ The 75 target terms are arranged alphabetically from anatomy/physiology to wellness. Care description term examples are provided in the Omaha System book, and numerous evidence-based interventions for SBDH are available on the Omaha System Guidelines website.²⁹ An example of an evidence-based SBDH intervention is Social contact (problem)-Teaching, Guidance, and Counseling (category)-interaction (target)-importance of social connection for well-being (care description). Interventions may be analyzed by term or combinations of terms, or by counts in which each problem-category-target-care description string is counted as a one intervention (as in this study).

Problem rating scale for outcomes

The Problems Rating Scale for Outcomes is comprehensive systematic recurring evaluation framework designed to measure client progress related to a specific health problem.¹⁹ This systematic method for evaluation provides the practitioner an objective measure between client problems and delivery of care.¹⁹ For example, a person with the sign/symptom of homeless with no prospects of being sheltered would have a Status score of 1 (extreme signs/symptoms) for the Residence problem. Problem Rating Scale for Outcomes measurements offer a quantitative method of documentation and the ability to track client progress over time. In this study, the mean Knowledge, Behavior, and Status (KBS) scores for all problems on admission and on discharge from services were calculated.

Analytic strategy

The data-driven model was developed using stepwise linear regression with backward elimination to create an alternative data-driven SBDH index. Pearson chi-squared statistical tests were used for understanding the applicability of the various signs/symptoms to the model. The dependent variable was the baseline Knowledge score, and the independent variables were Knowledge on admission, age, race/ethnicity, and 335 signs and symptoms across all problems. These 338 independent variables were included in the first step and were reduced to 18 significant predictors by the fourth backward elimination step. New SBSH subgroups were assigned via backward elimination until all variables were significant.

For both theory-based and data-driven aims, the respective SBDH Index scores were calculated for each client, with 5 or more items condensed into one group (5+-item). Changes in KBS outcomes, numbers of interventions, and adjusted R-squared statistics were computed for both models. All statistical tests were carried out in R Version 1.2.5003 and Excel (Microsoft Corporation, version 14.5.2). Excel spreadsheets were used to create heat maps, line graphs, and other visualizations used in the analysis.

RESULTS

There were 4109 pregnant/birthing people aged 25.7 years (range = 13–40; SD = 5.9) with at least one sign/symptom of the Mental health problem. As with the original study, there was an overall positive change from baseline to discharge.²⁰ The overall knowledge score increased from 2.42 to 3.03 (t = −35.3, P<.001; basic to adequate knowledge). The overall behavior score increased from 2.67 to 3.60 (t = −25.6, P<.001; inconsistently appropriate to usually appropriate behavior). The overall status score increased from 2.59 to 3.90 (t = −21.7, P<.001; minimal signs/symptoms).

The NAM/IOM-recommended SBDH factors varied across 6 SBDH index groups (Figure 1). The highest theory-based SBDH Index score was 13 of 18 possible. The 2 most common signs/symptoms were sadness/hopelessness/decreased self-esteem (n = 2467) and low/no income (n = 2452). Similar to the original study, age decreased across SBDH groups.²⁰

Figure 1.

Heat map of theory-based signs/symptoms frequencies.

For the data-driven model, the highest SBDH Index score was 11 of 18 possible. Unlike the theory-based model, average age was not a significant factor. No signs/symptoms from the IOM-recommended assessments were included in the final data-driven SBDH index (Table 1 and Figure 2). However, signs/symptoms from Income and Mental health problems were present in the data-driven model, along with signs/symptoms from numerous other problems not found within the IOM-recommended signs/symptoms (Figure 2).

Figure 2.

Heat map of data-driven signs/symptoms frequencies.

The signs/symptoms that significantly predicted knowledge outcomes on admission (P<.05) were difficulty with money management (11.4%, Income problem), difficulty understanding roles/regulations of service providers (8.0%, Communication with community resources problem), difficulty providing physical care/safety (32.4%, Caretaking/parenting problem), fails to seek care for symptoms requiring evaluation/treatment (18.2%, Health care supervision problem), and inadequate social support (49.6%, Pregnancy problem).

Comparison of theory-based and data-driven models in predictive ability

The adjusted R-squared for the theory-based model was 0.54 (SE = 0.38). The adjusted R-squared for the data-driven model was 0.61 (SE = 0.35). No signs/symptoms from the theory-based model were significant in the data-driven study. In both models, outcome patterns aligned with the original research: KBS increased from admission to discharge, overall discharge scores decreased as SBDH index increased, Knowledge scores, while lowest at baseline, showed the most improvement (Figure 3), and the number of interventions increased as SBDH increased (Figure 4).²⁰ The slopes of the data-driven model were steeper for KBS outcomes by group, showing clearer trends across data-driven SBDH groups.

Figure 3.

Final knowledge, behavior, and status scores by SBDH Index groups for theory-based and data-driven models.

Figure 4.

Intervention counts by SBDH index groups for theory-based and data-driven models.

DISCUSSION

This retrospective comparative study examining SBDH for PHN clients replicated a previous theory-based study with a new dataset and compared findings to those of a data-driven approach. In general, SBDH patterns in interventions and outcomes were similar across the original research, the replication theory-based model, and the data-driven model. KBS increased with Knowledge showing most improvement; discharge scores decreased as signs/symptoms accrued, and more interventions were provided to those with most signs/symptoms. The finding that signs and symptoms from the theory-driven model differed from those in the data-based study was surprising, given that these were recommended by the NAM/IOM and are generally thought to be a gold standard upon which many SBDH assessments have been developed.^2,30,31 While the 2 models shared some problem concepts in common, there were no signs/symptoms from the theory-driven model in the data-driven model.

There are advantages and disadvantages of both the theory-based and data-driven approaches. The theory-based approach provides a framework to identity patterns and relationships, and may be applied consistently across studies and populations.³² The disadvantage of a theory-based approach is the potential for assumptions and limitations of the theory which may not be applicable to real-world clinical environments, potentially masking factors that truly drive outcomes.³² In contrast, the data-driven approach can provide insights based on novel patterns for a given dataset and reveal insights and relationships not predicted by existing theories.^32,33 In particular, data-driven methods may be an advantage if there is sufficiently comprehensive SBDH data upon which to create the data-driven models.^18,21 Limitations of a data-driven approach are lack of uniformity across studies, the potential for confounding variables, and lack of theoretical explanations for underlying mechanisms related to the findings.³² In this study, we discovered new predictors using a data-driven approach for this dataset. Researchers and informatics teams should consider these advantages and disadvantages when determining which approach (theory-based or data-driven) to employ. For example, these findings suggest that data-driven models should be used to understand health determinants, as they have potential to better inform nurses of unique SBDH considerations, and nurses can thereby target interventions to optimize outcomes for a given group.

Furthermore, if data-driven SBDH algorithms were embedded within an EHR platform, for e-clinical decision support data-driven SBDH algorithms and programming would need to be agile to provide important clinical decision support for each person’s care.^18,34 The advantage with the data-driven model would be realized only if there were sufficient data and resources for continuous algorithm updates. For example, protocols would be needed to address how often the data are updated in the system to keep algorithms current as new data accrue, which in turn could affect associated clinical decision support systems in other parts of the EHR. This both improves data-driven decision making and makes EHR and algorithm updating more challenging.¹⁸

The finding that lower baseline knowledge scores were related to a broad range of signs/symptoms and problems that were present in the Omaha System but not typically recommended for SBDH assessments suggests the need for a more comprehensive and complete SBDH assessment. In particular, such a comprehensive assessment should be given special consideration for pregnant/birthing people with mental health signs/symptoms and other vulnerable populations.^16,18,27,30 Such assessments may be completed by a nurse or, to decrease documentation burden and promote client engagement, by the use of a consumer-facing application that generates data that can be provided to the practitioner electronically or on paper.^35,36

The findings of this study suggest that SBDH are complex and multifaceted and may differently impact health. The specificity of the Omaha System assessments has provided critical data that nurses use to inform their practice. Informatics nurses have a moral imperative to use such data optimally in order to create clinical decision support systems guiding targeted or tailored interventions to improve outcomes for pregnant/birthing people and potentially other vulnerable groups.^37–39 Given the complexity of SBDH, there is a need for a comprehensive and yet standardized approach to assess and address SBDH, and standardized nursing terminologies such as the Omaha System offer opportunities to assess a broad array of SBDH factors that will in turn provide high quality data for data-driven models and clinical decision support. It will be important to regularly examine such approaches to ensure both methods and outcomes align with population needs.⁴⁰

This work aligns with the Futures of Nursing report and provides foundational knowledge to support nurses in addressing SBDH.¹¹ This study provides insight into signs and symptoms that best predicted baseline Knowledge scores in this sample of pregnant/birthing people with at least one mental health sign/symptom. Future research should examine other datasets to determine whether data driven SBDH models differ for other populations of interest. Commonalities across data driven models may aid in further refining SBDH assessments for use in clinical decision support algorithms.

This secondary data analysis of a deidentified practice-generated dataset may be biased or limited by programmatic documentation requirements or time constraints. However, the SBDH-related patterns aligned with those of the original study, increasing confidence in the findings. Findings are not generalizable beyond the study population and specific health condition (mental health). Further research is needed to confirm and extend these findings with other samples and settings. As with the original study, this study focused on client Knowledge. Future work should predict Behavior ratings, Status ratings, or other measures to provide comprehensive insight into optimal SBDH indices.

CONCLUSION

In this replication study of standardized SBDH data, patterns in intervention effectiveness were consistent with those of the original study: As SBDH accrued, outcomes decreased while interventions increased. Use of a data-driven SBDH index identified different predictors than typical SBDH assessments. This research aligns with national initiatives and provides an example of how nurses can use standardized data to address SBDH toward improving health outcomes. The findings underscore the complexity of SBDH and demonstrate the importance of a comprehensive holistic assessment. The advantages and disadvantages of both theory-based and data-driven approaches should be carefully considered in future research as well as in clinical decision support development for practicing nurses.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

HUMAN SUBJECTS PROTECTIONS

The study was performed in compliance with the University of Minnesota and was reviewed by University of Minnesota Institutional Review Board.

AUTHOR CONTRIBUTIONS

RRA, TMM, and KAM devised and conducted the study. DSP and TA advised the research. All authors collaborated in writing the manuscript and have read and approved the final version.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

The Omaha System data are deidentified data and will be shared on a case-by-case basis based upon request to the corresponding author.