Implementation and evaluation of a secret shopper methodology to examine access to prenatal care for patients with opioid use disorder

Abstract

Background

Maternal opioid use disorder (OUD) has risen exponentially in the US, impacting pregnancy outcomes. This analysis describes and evaluates a secret shopper methodology for prenatal capacity and quality care research, addressing key considerations for study design. In recent years, this innovative methodology has gained popularity in research to obtain unbiased data on patient experiences in real-time. However, there is a lack of research on accessing prenatal care for pregnant patients with OUD. This analysis provides a comprehensive assessment of the secret shopper methodology’s application in the prenatal care setting for patients with OUD.

Methods

A trained group of racially and ethnically diverse callers followed a standardized script and used Google Voice numbers to call randomly selected prenatal care providers throughout Florida. The sample was obtained via geocoding providers, assessing the sampling frame for equal probability of provider selection, stratified randomized sampling in each of the 11 Agency for Health Care Administration regions, assigning an order of calls for providers with multiple locations, and blocked randomization of providers. The approach was pilot tested, and the protocol iteratively refined based on feedback loops from callers and a reflective Advisory Committee of parents with lived experience.

Results

Evaluation of this methodology was guided by key domains and constructs from the Consolidated Framework in Implementation Research. The methodology’s relative advantage over traditional research methods, trialability, adaptability, cost-effectiveness, and telephonic approach allowed for a greater reach of physicians statewide, providing unique insights into healthcare access and quality.

Conclusion

The study identified lessons learned during development and implementation, emphasizing the need for piloting, stakeholder support, and multidisciplinary collaboration to address unforeseen issues and ensure effective implementation. The methodology uncovered the challenges faced by pregnant individuals with OUD when seeking prenatal care and demonstrates the utility of secret shopper methods in assessing various facets of healthcare.

Background

Prenatal opioid use is common and poses immediate and long-term risks for mothers and infants [1–3]. Maternal opioid use disorder (OUD) increased by over 100% from 2010 to 2017 in the United States (US) [3]. The standard of care for OUD during pregnancy includes a combination of behavioral therapies and medication for OUD (MOUD), such as buprenorphine or methadone [4, 5]. Adherence to MOUD is associated with a decreased risk of maternal overdose and improved health outcomes for mother-infant dyads [4, 5].

Pregnancy provides a rare opportunity to offer interventions and support for mothers with OUD when and if prenatal care is attained [4, 6, 7]. Generally, adequate prenatal care is fundamental for a healthy pregnancy [8], but is particularly crucial for people with OUD [2, 9]. Prenatal care visits allow providers to prescribe MOUD and monitor dosage, screen for commonly co-occurring health conditions, identify gaps in social or financial support, and provide interventions to foster mother-infant wellness beyond pregnancy [2, 9]. Numerous publications have outlined considerations for optimal prenatal care for pregnant people with OUD [5, 6, 10], and a few have examined access at the population- or patient-level [7, 11, 12]. However, there are critical gaps in literature regarding barriers to prenatal care from the patient-level perspective of pregnant people with OUD.

In recent years, secret shopper methodologies have become more commonly used in research to obtain unbiased, patient level experiences in real time [13–15]. A secret shopper study includes the use of a trained “simulated patient”, “standardized patient”, or “mystery shopper” that poses as a person seeking clinical care to gain unique insight into healthcare access and quality. The secret shopper study design can be implemented in a wide array of healthcare settings or systems to quickly inform quality improvement and intervention priorities [14]. More traditional research methods to evaluate patient level quality of care include direct patient observation and patient exit interviews or satisfaction surveys [15–17]. However, each comparable method has potential for bias and logistical disadvantages that may be unethical or impractical in certain healthcare settings [15–17]. Thus, the secret shopper methodology is a promising approach to examining healthcare from the patient perspective.

Literature examining access to prenatal care among people with OUD using the secret shopper methodology is sparse. Previous secret shopper studies have examined contraceptive counseling and long acting contraceptives [18, 19], access to MOUD among the general population [20–22], and pharmacy access to an opioid overdose antidote, naloxone [21]. Two additional secret shopper studies examined access to MOUD in specialty treatment settings for pregnant and non-pregnant people focused on appointment availability and insurance acceptability [23, 24]. In contrast, our study applies the methodology to prenatal care settings, an area that remains understudied in the context of maternal opioid use disorder and evaluates its use guided by a theoretical framework. Therefore, critical gaps in the literature remain regarding the implementation and evaluation of the secret shopper methodology within the context of prenatal care. The purpose of this analysis is to provide a comprehensive evaluation guided by the Consolidated Framework in Implementation Research (CFIR) of a secret shopper methodology for research in the prenatal care setting and pinpoint crucial considerations when designing studies using this approach. Patient-level results from our secret shopper study were previously described [27].

Methods

Implementation of the secret shopper methodology

This project was initiated by champions at the state Department of Health, Department of Children and Families (DCF), and Agency for Health Care Administration (AHCA), who were inspired by a study conducted by Patrick et al. and reached out to the University of South Florida to conduct a similar study in 6 months [24]. We implemented a cross-sectional secret shopper data collection method in which five callers who were trained members of the study team presented themselves as a pregnant patient with OUD attempting to seek prenatal care. The study was conducted by a multi-disciplinary research team from the University of South Florida consisting of an obstetrician, public health systems researcher, pediatrician, general practice provider, epidemiologists and methodologists, patient advocates, and research assistants. The study included feedback from a community advisory committee consisting of individuals with lived experience. Advisory members were recruited from maternal and family-serving programs, including peer recovery programs. We met with the advisory committee virtually via Microsoft Teams for one-hour meetings twice a month and they provided iterative input on study design, implementation processes, and interpretation of study findings. An overview of the implementation method is shown in Fig. 1.

Fig. 1.

Key steps and considerations for designing and implementing a secret shopper study

Creating the patient persona

We carefully developed the standardized patient persona to reduce caller variability that may influence access to prenatal care. We set the patient age to 21 years old to avoid bias by age. The week of pregnancy was set at 14 weeks to match the typical time our patient population establishes prenatal care [25]. Callers calculated the last menstrual period using an online calculator to reflect 14 weeks pregnant when the call was conducted to ensure accurate information was shared during each interaction. The simulated patient was currently prescribed MOUD through a local methadone clinic. The patient also had no pregnancy complications, prior pregnancies, or previous prenatal care, and was currently on Medicaid for pregnancy. We chose to focus on access among Medicaid insured patients since nearly 80% of mothers with OUD are insured by Medicaid at delivery [3].

We took multiple measures to protect caller identities by creating a fully fabricated patient contact profile including name, phone number, and email. Callers chose fictitious patient names from a list of popular baby names in the US census of those born in the year 2000 to match their respective racial/ethnic backgrounds. Each caller also registered a phone number under Google Voice and chose a new email and home address based on geographic location to maintain anonymity. We determined patient race/ethnicities based on the caller’s actual background which generally represented the epidemiological profile of new mothers with OUD in Florida [3]. Two callers identified as White, one caller as Black, one as Latina, and one as Asian.

Developing the caller script

Next, we developed the standardized caller script [27] based on Patrick et al.’s previous secret shopper study [24]. The script outline included a patient introduction, prenatal care appointment request, insurance information, disclosure of MOUD prescription (methadone), and an excuse to end the call and/or cancel the appointment if offered. Callers asked for the first available appointment or inquired as to why they could not be seen if an appointment was not offered. If the option for a waitlist was given, callers were to ask about the potential wait time.

The script was reviewed and verified by the study’s Patient Advisory Committee to ensure it supported a realistic conversation between a patient and prenatal care provider (i.e. tone of voice, wording of patient characteristic description, and caller reaction to stigma from the provider). The team incorporated clinical, practical, and social perspectives to maintain the balance between patient urgency, reaction to stigma, receptionist attitude, and physician availability. The script was continuously reviewed for flexibility and adjusted to accommodate individual issues as they were encountered (Fig. 2).

Fig. 2.

Caller encounters, provider responses, and real-time adaptation of the secret shopper methodology during implementation

Identifying sample of prenatal care providers

We identified our sample of Medicaid enrolled prenatal care providers to whom the simulated calls would be made (Fig. 3). Using a list of Medicaid obstetric providers from the Florida AHCA [26], we implemented a sampling strategy to stratify results by AHCA region. ArcGIS Pro was used to geocode and standardize provider addresses and categorize them into AHCA Medicaid Managed Care regions. Duplicate provider practices were removed to ensure equal selection probability. The final sampling frame included 3,034 providers across 11 AHCA regions. We implemented a stratified random sampling algorithm to select the final sample of providers from the complete sampling frame. Additional methodological details have been previously published [27].

Fig. 3.

Identifying a random sample of prenatal care providers by Agency for Healthcare Administration regions in Florida

Piloting the secret shopper methodology

We conducted a pilot test of the patient persona and caller script to selected prenatal care providers in Atlanta, Georgia, and Tampa, Florida from a Google search within those areas. Our secret shopper callers contacted 10–12 obstetric physician offices in those locales and attempted to schedule a prenatal care appointment by implementing the standardized call script. Callers used a Qualtrics survey to document physician practice details such as insurance acceptance, scheduler interaction, information requested during the calls, ability to schedule an appointment, and reason for denial if no appointment was offered. Based on the pilot testing results, we completed necessary changes to our caller script and sampling methodology before implementing them in Florida. One main issue callers encountered during the pilot was difficulty reaching and speaking to providers. Many callers noted they couldn’t reach providers at the phone numbers listed, despite numerous attempts and those that did were placed on hold for long wait times. Also, some providers indicated that callers needed to call back due to the nature of their “high-risk” pregnancy. Therefore, we decided to set five as the maximum number of call attempts for each provider and a maximum wait time of 15 min per call if placed on hold. The pilot also identified provider requests for patient persona information we had not previously accounted for, including type of Medicaid insurance and patient address. In response, we had callers pre-verify the type of Medicaid insurance accepted by each provider and determine patient address using Google maps. Callers were instructed to identify an address at a large apartment complex within 5–8 miles of each specific healthcare provider office. Addresses and zip codes were inputted into the overall Microsoft Excel sheet for tracking.

Implementing the secret shopper study

We began conducting calls to the randomly selected prenatal care providers by AHCA region in Florida. The information collected during the call was entered into a Qualtrics survey for data collection. Each caller was given a predetermined ordered list of providers by AHCA region and instructed to contact providers in the specific order. Callers were given specific instructions for several situations including: disconnected phone numbers, phone numbers that reached a hospital, or instances where different providers were listed at the same practice with the same phone number. To avoid callers contacting the same office for different providers, another caller was assigned the duplicate number for a different provider in the same practice.

This study was reviewed and approved by the University of South Florida Institutional Review Board, and the study was determined to be exempt.

Results

Evaluation of the secret shopper methodology

We utilized constructs from the Consolidated Framework in Implementation Research (CFIR) to evaluate our implementation of the secret shopper methodology. The key CFIR domains and constructs that guided our evaluation included innovation, outer setting, inner setting, and implementation process (Table 1) [28]. The evaluation results presented focus on lessons learned during development and implementation of this secret shopper study.

Table 1.

Application of the Consolidated Framework for Implementation Research (CFIR) for evaluation of the secret shopper methodology

CFIR Domain	Summary
Innovation Characteristics	▪ More ideal methodology to study sensitive topics and populations that are difficult to recruit compared to other conventional research methods. ▪ Utilizes trained simulated patients minimizing the Hawthorn effect, selection bias, and recall bias. ▪ Simulated patients are able to capture contexts and nuances of the patient experience. ▪ Can be pilot tested prior to implementation to identify any design weaknesses. ▪ Allows for design modifications to fit study context or needs as they arise. ▪ Can be implemented with feasible costs and time frame.
Outer Setting and Inner Setting	▪ Buy-in from state health department and the Florida Agency for Health Care Administration (AHCA) in addressing barriers to healthcare access and policy reform. ▪ Availability of external funding for implementation. ▪ Multidisciplinary partnerships to facilitate implementation efforts to address access to care. ▪ Affordable operating costs.
Implementation Process	▪ Feasible operating costs. ▪ Allows multidisciplinary team collaborations to engage in specific roles and implement secret shopper methodology. ▪ Secret shopper methodology offers ease of implementation and flexibility to revise aspects of the protocol during implementation. ▪ However, issues may emerge during implementation despite pilot testing. ▪ Depending on study goals, some modifications needed may not be feasible once implementation has begun. ▪ Adequate amount of simulated callers and time for study should be considered during design and prior to implementation.

Innovation

We considered the relative advantage of the secret shopper methodology which utilized trained simulated patients to be more advantageous to assess prenatal care access for patients receiving MOUD compared to conventional research utilizing real patients. Typically, it is challenging to recruit hard-to-reach populations such as pregnant individuals with OUD as participants for research studies [2]. We are also mindful of the burden of participating in research, particularly a study of this scope, for actual patients enduring the hardships of navigating pregnancy while receiving MOUD. The secret shopper methodology allowed us to use simulated patients who were able to record call responses and also avoided the Hawthorn effect, selection bias, and recall bias that could occur with real patients. The telephonic approach within secret shopper mimicked the real-world interaction a patient would have when seeking a prenatal appointment, and provided a large, geographically diverse sample of obstetric physicians while requiring less resources. Callers were able to qualitatively capture context and nuances of their experiences, such as perceived stigma and discrimination when interacting with schedulers and providers. In general, the secret shopper methodology offered an objective alternative of assessing access to prenatal care for pregnant patients with OUD across Florida at a reasonable cost and time commitment.

Our evaluation revealed several lessons learned regarding trialability (pilot testing) and adaptability of the secret shopper prior to implementation. Pilot testing of the secret shopper methodology in conjunction with feedback from our Advisory Committee was beneficial because it helped identify weaknesses and facilitate adaptations to the study protocol prior to implementation. Because many providers had multiple locations within the same region, once a provider was chosen for the sample, our original intent was to randomly select a single location and phone number to place the call. However, during pilot testing, we realized that the physician list was not entirely up-to-date, and many addresses and phone numbers were incorrect. Therefore, to maximize the likelihood of successfully contacting the sample physician’s office, when a physician had more than one location/phone number, we kept all available for our callers. In addition to incorrectly listed phone numbers and addresses for physicians, numerous calls had to be made in an attempt to reach some physician offices, and for some, once a physician’s office was reached, callers sometimes had excessively long wait times when placed on hold. This hindered the time allocated for calling other physicians. We therefore adapted the study protocol to restrict to a maximum of five call attempts to reach each physician’s office and 15 min of call-time when placed on hold. We also learned that to progress through some physician appointment systems, callers were required to provide the scheduler with either their Medicaid plan type and/or number, social security number (SSN), or both. The protocol was then revised to have callers pre-verify the Medicaid plan types accepted by each physician’s office they were contacting. Otherwise, if a Medicaid number and/or SSN were requested, the caller was instructed to ask for a potential appointment date given that they would call back with the requested information. The caller script was adapted to reflect these changes. In general, we found the trialability and adaptability of the secret shopper methodology to be easy and comprehensive, and methodology weaknesses identified emphasized the need for piloting prior to implementation.

Outer setting and inner setting

Buy-in and support from major stakeholders were critical in facilitating the implementation of this study. The state health department and AHCA both have a personal stake in addressing barriers to healthcare access and policies. We received grant funding from the state health department. The state DCF and Drug Policy Council also expressed interest in and support of the project. Furthermore, conducting this study from a large academic public institution that is research and clinically oriented, allowed for a streamlined process in creating a multi-disciplinary research team.

Implementation

Despite conducting pilot testing, it is possible for unforeseen issues to emerge during implementation. However, we recommend collaboration with multidisciplinary research team members and regular team debriefing sessions that we incorporated throughout the development and implementation process to facilitate identifying and addressing such issues. We identified discrepancies within the AHCA physician list where some physicians were not obstetricians, and some phone numbers and clinic locations were inaccurate. This was rectified early during implementation by verifying physician information prior to calls, but this added more time to completing calls. Some physician schedulers were unaware of what Methadone was, so callers had to take time to explain and sometimes were put on hold or told they would receive a callback once the scheduler confirmed that the physician accepted such patients. This also contributed to further delays in callers reaching other physicians’ offices. Some callers missed callbacks from schedulers while engaged in other secret shopper calls. On a few occasions, callers’ voices, phone numbers, or addresses were recognized by scheduling staff when calling multiple physicians at the same clinic location within a region. Caller training prepared callers to improvise their responses and contact details were amended without arousing further suspicion. Moving forward, we encourage having sufficient callers, each utilizing several different personas and contact details which would facilitate achieving more calls in a set period and reduce the likelihood of caller recognition. It is possible for issues to emerge late in the implementation process. However, depending on how modifiable the issue is, careful considerations should be made so as not to introduce bias into the study. For example, we realized that a gestational age of 14 weeks pregnant was considered “too far along” by some physician offices, and our team decided that it was too far into the implementation process to modify that characteristic without biasing our study results. Overall, the ease of implementation and flexibility to revise aspects of the protocol during implementation are major benefits in utilizing the secret shopper methodology to assess prenatal care access for pregnant patients receiving MOUD.

Discussion

This study evaluated the use of the secret shopper methodology as a research tool in assessing timely access to prenatal care for pregnant people receiving MOUD. Overall, we found the secret shopper methodology to be an appropriate and feasible approach for examining access to prenatal care statewide. The use of trained patient callers provided a unique perspective on the barriers and challenges that pregnant patients receiving MOUD may encounter when seeking a prenatal care appointment in Florida.

Key strengths of the secret shopper methodology identified in our study include its ability to capture real-world patient experiences, particularly during the COVID-19 pandemic when traditional in-person research methods were challenging. The standardized approach allowed us to examine prenatal care access across Florida and identify complex barriers to care, such as the discriminatory practice of not accepting patients on MOUD [24, 29] Additionally, using callers closely resembling pregnant patients receiving MOUD avoided the Hawthorne effect associated with scheduling staffs’ awareness of evaluation [30]. These strengths inform targeted interventions to improve healthcare delivery and address patient needs and disparities.

The secret shopper methodology proved cost-effective by utilizing trained graduate research assistants as callers, minimizing training, travel, and participant incentive costs. However, resource limitations for time, efforts, and funds for training, caller placement, and data analysis should be considered. Adequate number of secret shoppers are essential to ensure timely completion and study generalizability. Careful planning of study design, sample selection, and pilot testing are advised to ensure representation, validity, and address potential design weaknesses. This will also ensure that valuable healthcare resources (i.e., healthcare providers’ work with real patients) are not diverted while still gathering essential information from these studies. Finally, the secret shopper methodology raises essential ethical considerations, consistent with general principles of ethical research. Researchers must consult their Institutional Review Board to appropriately address the need for informed consent and confidentiality to protect the rights and well-being of all study participants [14, 31].

Conclusions

Designing secret shopper studies for improving access to treatment and supporting recovery and optimal care for patients with perinatal substance use has several implications for policy and practice. Well-designed secret shopper studies should avoid scheduling actual appointments that disrupt healthcare providers’ work with real patients and should prioritize the development of an efficient call script to minimize scheduling staff interaction. Additionally, these studies should utilize frequent feedback from implementers and ideally be supported by policy leaders and practice champions, such as state agencies, healthcare groups or insurers, and quality oversight committees.

Abbreviations

AHCA

Agency for Health Care Administration

CFIR

Consolidated Framework in Implementation Research

DCF

Department of Children and Families

MOUD

Medications for Opioid Use Disorder

OUD

Opioid Use Disorder

Authors’ contributions

CNR, lead author, assisted with study design and protocol development, coordinated and supervised calls, conducted analysis, and contributed substantially to the manuscript (all drafts). AE completed analysis and contributed substantially to the manuscript (all drafts). SM conducted a substantial portion of the calls and contributed substantially to the manuscript. MPG conducted a substantial portion of the calls and contributed substantially to the manuscript. JS designed provider selection algorithm and tracking system, monitored call completion and data quality, conducted analysis, and contributed to this manuscript. WMS coordinated project partners, contributed substantially to study design and implementation, and contributed to this manuscript. KF, OBGYN, co-PI of the project, contributed to study design, protocol development, project oversight, analysis, and manuscript development. JM, Principal Investigator, designed the project, provided oversight of team, state partners, and all stages of design and implementation, and contributed substantially to the manuscript.

Funding

This study was funded by the Florida Overdose Data to Action (FL-OD2A) program which is supported by the Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services (HHS) as part of a financial assistance award totaling $60.6 million with 100% funded by CDC/HHS. The contents are those of the author(s) and do not necessarily represent the official views of, nor an endorsement, by CDC/HHS, or the U.S. Government.

Data availability

No datasets were generated or analysed during the current study.

Declarations

Ethics approval and consent to participate

This study was reviewed and deemed exempt by the University of South Florida Institutional Review Board.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.