Psychometric Validation of the Adolescent Opioid Safety and Learning (AOSL) Scale Using Confirmatory Factor Analysis: A Nationally Representative Sample

Abstract

Background:

The opioid overdose epidemic has worsened during the COVID-19 pandemic. Recent data revealed a 28.5% increase in drug-related overdose deaths from 2019 to 2020. Adolescents often misuse family members and friends’ prescription opioid medications. Furthermore, adolescents may not possess the knowledge or understanding to safely manage opioid medications. There is a need for a validated scale to effectively measure adolescents’ opioid misuse knowledge, attitudes, and interest in learning about prescription opioid safety.

Objective:

The purpose of this study was to validate the Adolescent Opioid Safety and Learning (AOSL) scale with a nationally representative sample of adolescents and confirm the factor structure of the scale using confirmatory factor analysis.

Methods:

Adolescent participants (aged 13–18) completed the 16-item AOSL scale in Qualtrics from November to December 2020. A total of 774 responses were analyzed. A confirmatory factor analysis (CFA) was performed to determine the fit of the data to the four-factor model proposed by a prior exploratory factor analysis of the AOSL scale. Fit was assessed using the chi-square test, comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean-squared error of approximation (RMSEA).

Results:

Participants were 50% male and 62% white non-Hispanic. The CFI was 0.984, TLI was 0.980, and RMSEA was 0.048 (95% CI: 0.041 to 0.054, p-value that RMSEA ≤ 0.05 = 0.712). The chi-square test results were X² = 268.752 on 98 degrees of freedom (p < 0.001). Cronbach’s alpha, a measure of internal consistency, was high within each factor. CFA indicated good fit of the current study’s data to the four-factor model.

Conclusion:

We found the AOSL scale measures adolescents’ knowledge of opioid misuse, knowledge of opioid harm, interest in learning about prescription opioids, and likelihood to practice misuse behaviors. This scale can help researchers understand adolescent perceptions and opinions about opioid safety.

Background

The COVID-19 pandemic has worsened the United States drug overdose epidemic. According to the latest preliminary data from the Centers for Disease Control and Prevention (CDC), approximately 100,000 people in the United States died from a drug-related overdose in 2020, which is a 28.5% increase from 2019.^1,2 Opioid-involved overdose deaths involving prescription opioids rose from 3,442 in 1999 to 17,029 in 2017.³ Additionally, between 1999 and 2016, approximately 9,000 children and adolescents died from prescription and illicit opioid poisonings in the United States, and the pediatric mortality rate for these drugs increased nearly 3-fold.⁴ A recent study showed most of those pediatric deaths were unintentional, with only 5.0% related to suicide, and the majority occurred outside of a medical setting; 38.0% occurred at home.⁵

One in four adolescents in the United States have reported misuse of a prescription opioid at least once in their life, and roughly one in four high school students in the United States are exposed to opioids.⁶ Research shows adolescent prescription opioid use is associated with increased health-risk behaviors and premature death.⁷ Adolescents who use prescription opioids are more likely to be diagnosed with an opioid use disorder by the time they turn 25 years old and are prone to suicide, overdose, and substance-related crime.^7–10 Additionally, early opioid prescription use in adolescents and young adults is associated with a 30% to 40% increased risk of subsequent substance-related morbidity.¹¹

Access to family members’ opioid prescriptions, continued use of opioids after surgery, and using high dosages of opioids are correlated with increased use of prescription opioids among youth.^12–16 A main contributing factor to opioid misuse among adolescents is that their parents/guardians do not think their child would misuse opioids and therefore do not prioritize storing their opioids correctly in the home or dispose of opioids responsibly.^17–20 However, studies show opioids prescribed to family members are the most frequently misused opioid by adolescents, both with and without the permission of the prescribed person.¹⁷ Only 11.7% of parents with an older child in the home report storing their opioids in a locked cabinet, even though research shows older adolescents are more likely to misuse opioids and share them with their peers.^7,21–24 Additionally, parents do not intend to dispose of expired or unused opioids for the following reasons: (1) they paid for them, (2) they are unsure of how to properly dispose of the prescribed medication, and (3) they think their children may need the drug in the future.²⁰

Studies have established that adolescents have gaps in their knowledge or understanding of correct opioid use and safety; many adolescents cannot accurately identify what medications are opioids and which are not.^25–29 It is important to educate adolescents on the safe and proper management of opioid medications because they are in a developmental stage where they can learn about healthy lifestyle behaviors and can control health-risk behaviors.³⁰ Beginning opioid safety education at an early age is critical for curbing the opioid epidemic, decreasing opioid misuse among youth, and preventing future opioid-related deaths. To educate adolescents effectively and efficiently on proper opioid use, researchers first need to understand adolescents’ interest in learning about opioids, opinions and knowledge of opioid harm, and opioid misuse behaviors.

The Adolescent Opioid Safety and Learning (AOSL) scale was developed from a 68-item survey measuring adolescents’ knowledge of opioid misuse behaviors, how much harm misuse of opioids can cause, adolescents’ likelihood to engage in misuse behaviors, and their interest in learning more about opioid medication topics. Exploratory factor analysis (EFA) was conducted to understand the underlying factor structure and develop the AOSL scale.³¹ To our knowledge, the AOSL scale is the only scale that measures adolescents’ knowledge of and preferences for opioid safety education, likelihood to misuse opioids, and the consequences of misuse.³¹ However, before the AOSL scale can be widely used, it must first be validated using a nationally representative sample of adolescents.

Objective:

The purpose of this study was to validate the AOSL scale with a nationally representative sample of adolescents and verify the factor structure of the scale using confirmatory factor analysis (CFA).

Methods

The Adolescent Opioid Safety and Learning (AOSL) scale

The original survey was designed by the investigator and consisted of 68 instrument items and 16 demographic questions (Appendix A). Survey items were adapted from a statewide survey designed to increase awareness of prescription drug misuse and abuse in Wisconsin, which was adapted from a survey used by the State of Maryland.^32,33 Questions from the Wisconsin state-wide survey were revised in structure and content to meet the needs of high school aged adolescents as well as the study objectives. Exploratory Factor analysis (EFA) was used for item reduction and to understand the underlying factor structure. Four factors emerged: ‘Opioid Learning Interest’, ‘Opioid Misuse Behavior’, ‘Likelihood to Misuse Opioids’, and ‘Opioid Harm’. The final scale resulted in 16 items and was named the Adolescent Opioid Safety and Learning (AOSL) scale. The AOSL scale was used in this study (Appendix B).

Confirmatory Factor Analysis (CFA) to determine Validity of the AOSL scale

This study used confirmatory factor analysis (CFA) to measure the validity of the AOSL scale. The 1999 Standards for Educational and Psychological Testing defines validity as “the degree to which evidence and theory support the interpretation of test scores entailed by proposed uses of tests”.³⁴ Validity is established not by a single method, but by providing multiple examples of evidence. CFA provides accurate results in testing the validity and reliability of an instrument and has been used to test the validity and reliability of survey instruments in prior work.^35,36 Our previous work used EFA to determine the four-factor structure which included 16-survey items. Therefore, by using CFA as a second method of analysis, the validity of the ASOL scale can be determined.

Procedure

Data were collected from November to December 2020. Parents of potential adolescent participants were recruited via Qualtrics panels and quota sampling methods were used to create a sample based off the 2010 US Census distributions.^37,38 Eligible participants were adolescents aged 13 to 18 who could read English and lived in the United States. Once the online screening process was complete, parent consent and child assent were collected. The adolescent was then directed to take the online survey independently. Participants who completed the survey were financially compensated through their pre-determined contract with Qualtrics Panels (e.g., airline miles, gift cards, sweepstakes entrance, vouchers, coupons for food, e-cash, or more). This study was approved by the University’s Institutional Review Board.

The online survey consisted of close-ended questions with “yes/no” response options or a 5-point Likert scale. Demographic information was also collected, including participant’s age, gender, grade, race/ethnicity, and total number of children under 18 in the household. To ensure data quality, the survey contained two attention check questions. The survey immediately ended for participants who incorrectly responded to these question(s). Participants could skip any survey questions they did not want to answer, except for attention check questions or age.

Statistical Analysis

A total of 774 survey responses were included in analysis. Participants who missed attention checks, straightlined responses, and/or whose IP addresses were located outside of the United States were removed prior to data analysis. Survey questions in the factor “Opioid Misuse Behavior” had “yes/no” response options and were mapped with 1 for “yes” and 0 for “no”. Questions in the three other factors had Likert answers and were mapped to a 1 through 5 scale. A confirmatory factor analysis (CFA) was conducted to determine the fit of the data to the four-factor model proposed by prior exploratory factor analysis.³¹ Fit was assessed using the chi-square test, comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean-squared error of approximation (RMSEA). Factor loadings were estimated, and Cronbach’s alpha was calculated within each factor. CFI and TLI values greater than 0.95 and RMSEA values less than 0.06 were indicators of good model fit.³⁹ Cronbach’s alpha of 0.7 to 0.8 were considered to represent satisfactory internal consistency, with greater values as more desirable.⁴⁰ All analyses were conducted using the lavaan 0.6–9 and ltm 1.1–1 packages for R version 4.1.1 (2021-08-10).

Results

Sample Characteristics

The sample consisted of 377 females (48.7%), 389 males (50.3%), 480 white non-Hispanic participants (62.1%), 142 Hispanic participants (18.3%), 89 Black or African American participants (11.5%), and 32 Asian participants (4.1%) with a mean age of 15.16 years (SD = 1.42) Participant characteristics are shown in Table 1.

Table 1.

Participant characteristics.

Variable	n (%)
Sample Size	774
Grade
7th grade	47 (6.1)
8th grade	109 (14.1)
9th grade	151 (19.5)
10th grade	172 (22.2)
11th grade	154 (19.9)
12th grade	141 (18.2)
Gender a
Female	377 (48.7)
Male	389 (50.3)
Other	8 (1.0)
Race/ethnicity b
Asian	32 (4.1)
Black or African American	89 (11.5)
Hispanic	142 (18.3)
Other or multiple categories	31 (4.0)
White	480 (62.0)
	Mean (SD)
Age	15.16 (1.42)

^aThree options were presenting to participants to select for their gender; “male”, “female”, and “other”. Thus, “male” is defined as selecting “male” while not selecting other choices, and “female” is defined as selecting “female” while not selecting other choices; “other” was defined as all other combination of choices that occurred.

^bSeven different categories were available to select from for race/ethnicity, including “Black or African American”, “Hispanic or Latino”, and “White.” Participants were defined as “Black”, “Hispanic”, or “White”, if they only selected the associated category and no other selection was made; all other combinations of selects were defined as “Other”.

Confirmatory Factor Analysis

The confirmatory factor analysis values of CFI, TLI, and RMSEA indicated good fit of the current study’s data to the four-factor model, although the chi-square test was significant. The CFI was 0.984, TLI was 0.980, and RMSEA was 0.048 (95% CI: 0.041 to 0.054, p-value that RMSEA ≤ 0.05 = 0.712). The chi-square test results were X² = 268.752 on 98 degrees of freedom (p < 0.001). The chi-square test results would indicate a poor fit; however, this could be due to the large sample size (n = 774) rather than a true poor fit. Considering the consistent good performance described by CFI, TLI, and RMSEA, we feel the four-factor model is appropriate for describing our data. Factor loadings are shown in Figure 1, and survey questions are in Appendix B. All factor loadings of individual questions were significantly greater than zero, with standardized estimates ranging from 0.81 to 0.95. Covariances were not significantly greater than zero for relationships between ‘Opioid Misuse Behavior’ with each of ‘Opioid Learning Interest’ (p = 0.637) and ‘Likelihood to Misuse Opioids’ (p = 0.059). ‘Opioid Harm’ and ‘Likelihood to Misuse Opioids’ were inversely related (p < 0.001). All other factor pairs were positively related (p < 0.001).

Figure 1.

Factor structure with standardized loading for the 16-item Adolescent Opioid Safety and Learning scale.

Cronbach’s alpha, a measure of internal consistency, was high within each factor: Opioid Learning Interest, α = 0.941 (95% CI: 0.931 to 0.949), Misusing Opioids, α = 0.903 (95% CI: 0.884 to 0.920), Likelihood to Misuse Opioids, α = 0.961 (95% CI: 0.954 to 0.967), Opioid Harm, α = 0.900 (95% CI: 0.881 to 0.918).

Discussion

This study demonstrated that the AOSL scale can be adapted for use with adolescents in the United States. To the authors’ knowledge, this is the first study reporting on the validity of an instrument aimed at assessing adolescents’ knowledge of opioid harm, likelihood to misuse opioids and misuse behaviors, as well as interest in opioid education with a national sample. Additionally, this study confirmed the investigator’s previous work, which examined the AOSL scale on a sample of Wisconsin adolescents.³¹ Factors ‘Likelihood to Misuse Opioids’ and ‘Opioid Misuse Behavior’, along with ‘Opioid Misuse Behavior’ and ‘Opioid Learning Interest’ were not related. The chi-square test results would indicate a poor fit; however this could be due to the large sample size. Considering the consistent good performance described by CFI, TLI, and RMSEA, we feel the four-factor model is appropriate for describing our data. Our values of CFI and TLI exceeded 0.95 and our RMSEA was less than 0.6, indicating a good model fit.³⁹

The AOSL scale is an important survey instrument that provides a tool for capturing adolescents’ perspectives on opioid misuse. Replicating the use of this instrument in future research will be useful for both researchers and educators. Once the scale has been used in a variety of studies, the validated scale will allow for researchers to gain better insight on adolescents’ opioid safety, misuse behaviors, and opioid education preferences. This information can guide the advancement of opioid misuse prevention interventions designed specifically for adolescents, which will help combat the opioid epidemic. Furthermore, this scale can be used by pharmacists during opioid medication counseling or in schools to assess adolescents’ perceptions and knowledge before offering an opioid safety education in health-related courses.

Using the AOSL scale to understand what factors contribute to opioid misuse among adolescents is essential to designing and identifying adolescent-targeted educational materials. The Pediatric Pharmacy Advocacy Group (PPAG) recognizes the crucial role pharmacists can play in reducing opioid misuse.⁴¹ The PPAG recommends pharmacists educate their pediatric patients on pain management, including the safe use of opioids, non-opioid alternatives for pain, (over-the-counter (OTC) pain medications) and when to recommend prescribing naloxone (Narcan®).⁴¹ Opioid safety education by pharmacists to adolescents and their families regarding proper administration, storage, and disposal, as well as the awareness of opioid misuse by youth, is key to prevention. Future research should be conducted using the AOSL scale to aid in the development of adolescent-specific educational materials.

Study Limitations

This study is not without limitations. First, use of quota sampling and recruitment through Qualtrics panels may limit generalizability and increase bias as the sample was limited to participants who had an interest in participating in research. Second, this study relied on the use of self-report surveys for data collection and trusted participants to give honest responses. Such study designs are limited by forgetfulness, dishonesty, fraud, and reporting-bias. Third, we are unable to determine if adolescents completed the surveys independently and without the presence of a family member who may have affected their responses. To ensure accuracy, another national survey should be distributed to adolescents in a way that confirms they are the ones taking the survey.

Conclusion

We confirmed that the AOSL scale measures adolescents’ knowledge of opioid misuse and harm, their interest in learning about prescription opioids, and their tendency to practice misuse behaviors. This survey instrument may help researchers understand adolescent perceptions and opinions about opioid safety. The AOSL Scale may also be used to help develop adolescent-targeted educational materials.

Key Points:

What was already known

• There is a need for a scale to measure adolescents’ opioid knowledge of and preferences for opioid safety education, likelihood to misuse opioids, and knowledge of the consequences of misuse.

• The AOSL scale was previously tested with adolescents in Wisconsin and was found to be an effective instrument.

What this study adds

• A validation of a survey used to measure adolescents’ opioid knowledge, opioid misuse and harm, likelihood to misuse opioids, and interest in learning about opioids.

Abbreviations (in order of appearance):

CDC

Centers for Disease Control and Prevention

AOSL

Adolescent Opioid Safety and Learning

EFA

Exploratory Factor analysis

CFA

Confirmatory Factor analysis

CFI

Comparative Fit Index

TLI

Tucker-Lewis Index

RMSEA

Root Mean-Squared Error of Approximation

PPAG

Pediatric Pharmacy Advocacy Group

OTC

over-the-counter