A Review of Unsolicited Comments on the CAHPS 5.0 Health Plan Survey

Abstract

The CAHPS Health Plan Survey (CAHPS 5.0) collects invaluable information regarding consumer experiences with their health plans, and these data inform healthcare policies at both the state and federal levels. The purpose of this paper was to explore unsolicited comments provided on the CAHPS 5.0 survey of one state's Medicaid program. Secondary data analysis was conducted of unsolicited, written comments received from Medicaid recipients who completed the CAHPS 5.0 adult or child postal survey between 2016 and 2018. The majority of unsolicited comments were moderately or very negative in attitude (or tone) for adult and child surveys. Analysis of unsolicited comments yielded 3 themes: positive experiences with Medicaid, limitations of coverage, and direct requests for assistance. Providing space for Medicaid patients to share comments and receiving further guidance for content analysis would provide valuable context for interpreting overall survey results. Comments may also help Medicaid program administrators respond to the frequently complex and challenging experiences of navigating a continually evolving state health insurance program by the most vulnerable populations.

Introduction

The purpose of this paper was to explore the unsolicited comments provided on The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) surveys of one state's Medicaid program. Secondary data analysis was conducted of unsolicited, written comments received from Medicaid recipients who completed the CAHPS 5.0 adult or child postal survey between 2016 and 2018. Content analysis of comments was performed to identify general sentiment and response themes. Results and a discussion of their implications are provided.

CAHPS is a series of surveys designed to assess healthcare experiences in various settings and populations. ¹ One of these surveys, the CAHPS Health Plan Survey (CAHPS 5.0), is used to collect standardized information on Medicaid enrollees’ experiences with their insurance health plans and services.^2,3 Public reporting of survey results is intended to incentivize health plans to improve the quality of their customers’ experiences, promote provider accountability, and increase care cost transparency. ⁴

CAHPS 5.0 asks patients about key aspects of their care, such as whether or how often they experienced communication with their doctors. ⁵ Questions are forced-choice, scale-measure options (eg, Never to Always, 0 to 10), and measures are statistically adjusted to correct for differences in patient mix and survey modes. Previous studies of CAHPS surveys have analyzed issues such as care quality ⁶ ; consumer choice ⁷ ; and perceived differences in care delivery by race and ethnicity ⁸ ; using the enumerated data. Despite the survey's scale rating design, a number of consumers also include unsolicited satisfaction-related comments when completing the postal version.

Background

To frame this article, our team adopted a general perspective of empowerment theory. “Empowerment” is viewed as a mechanism through which individuals attempt to exert control and influence over decisions that affect their lives. Frequently, enacting empowerment involves identifying sources of power, exploiting potential resources, creating connections to issues of concern, and modifying the factors that influence decision making. ⁹ For this discussion, empowerment theory speaks to an individual's ability to advocate for their own self-interests regarding healthcare. ¹⁰

CAHPS 5.0

By design, CAHPS 5.0 is not structured to accept open-ended text entries from respondents. Nevertheless, Medicaid recipients often write comments in the margins of the CAHPS 5.0 survey or submit comments as supplementary text (ie, handwritten or typed notes) to provide relevant context, explanations, or concerns. Neither the Agency for Healthcare Research and Quality (AHRQ), the organization that manages CAHPS, nor the National Committee for Quality Assurance (NCQA), the organization that certifies vendors, offers guidance regarding the tracking or use of unsolicited consumer comments. Previous researchers have used content analysis of unsolicited comments and free-text entries to explore a wide range of topics in various settings.^11–14 The question that guided this investigation was: What do unsolicited written comments on the CAHPS 5.0 survey reveal about Medicaid patient experiences with healthcare, especially in relation to differences among the adult and child Medicaid insured?

Methods

Sample

For 3 years from 2016 to 2018 (3 years inclusively), the annual CAHPS survey among adult (ages 18-75 +) and child (ages <1-17) recipients of Medicaid were collected. For child surveys, parents or guardians were instructed to complete the survey on behalf of their child. The state sample of adult survey respondents self-identified as female (67%), White (68%), ages 65 to 74 (24%), with a high school diploma or GED (41%). This compared to the national dataset: female (51%), White (48%), ages 45 to 54 (15%), with a high school diploma or GED (29%). ¹⁵ The state sample for child survey responses were characterized as male (55%), White (45%), ages 13 to 18 (53%), and non-Hispanic (90%). This compared to the national dataset: male (53%), ages 12+  (37%), and non-Hispanic (70%). ¹⁶

Data Entry Procedures

Upon receipt of postal surveys, trained staff of the service center sorted survey forms by client type (adult or child), conducted a visual inspection of forms for survey completeness, and uploaded survey forms into a database to scan for potential errors. The service center's optical mark recognition software cannot process comments that are written in the margins of a machine-readable form (ie, Scantron) or submitted as separate attachments. Therefore, these “artifacts” were stored in an Excel database by the service center and analyzed separately.

Our team conducted secondary data analysis of unsolicited written comments received from one state's Medicaid recipients who completed the postal version of the CAHPS 5.0 adult or child survey between the years 2016 and 2018. We used NVivo 12 to code, arrange, sort, and manage sections of text as well as autocode sentiment references using the built-in lexicon dictionary (QSR International Pty. Ltd., Version 12). Autocoding considers words and phrases in isolation within passages, and, as such, particular comments may have been coded with both positive and negative segments. ¹⁷ However, not all comments left by Medicaid participants contained enough depth to be autocoded based on sentiment.

Additionally, coding followed procedures outlined by Ivankova ¹⁸ and allowed the team to extrapolate recurring words and ideas from Medicaid respondents. Emergent codes and themes were reviewed by co-authors, and disagreements were discussed until consensus was reached. Consistent with best practices in qualitative design, we used multiple methods of verification to ensure qualitative rigor. ¹⁹ Under the Common Rule, this project was not considered research but rather public health surveillance; therefore, it was not subject to review by the Institutional Review Board. Informed consent was not applicable.

Findings

In Table 1, we report the total number of surveys received by type as well as the number of unsolicited written comments for the years under review. On average, 12.9% of adult and 5.6% of child postal surveys had unsolicited comments appended to them.

Table 1.

Sample Size and Unsolicited Comment Response Rates for Postal Surveys.

	2016	2017	2018
CAHPS 5.0 Adult	7057	6421	6980
Postal only	5214	3584	5964
Comments	680 (13.0%)	210 (5.8%)	297 (5.0%)
CAHPS 5.0 Child	4639	4949	4152
Postal only	3688	3622	3858
Comments	143 (3.9%)	129 (3.6%)	354 (9.2%)

Note. Comments were only received on postal surveys. Percentages were based solely on comments submitted in writing.

In Figure 1, we display autocoded sentiment references for the period of 2016 through 2018 with respondent attitude (or tone) reflecting one of the following 4 descriptors: very positive, moderately positive, moderately negative, or very negative. As previously noted, not all comments were detected by autocoding, therefore, the total number of comments in Table 1 is greater than the number of passages presented in Figure 1.

Figure 1.

Respondent Attitudes as Reflected by Sentiment Type.

The majority of unsolicited comments (greater than half) were moderately negative or very negative in attitude on the CAHPS 5.0 adult and child surveys. A chi-square test showed a significant association between survey type and sentiment type, Χ² (3, N = 306) = 16.72, P < .001. The Cramer's V = 0.23, which represents a small-to-medium effect size for the association between the 2 variables. Unsolicited adult comments tended to be more negative than child comments, 78.4% (n = 174) compared to 57.2% (n = 48), respectively.

Analysis of unsolicited comments on the CAHPS 5.0 survey yielded the following 3 themes: positive experiences with Medicaid, limitations of coverage, and direct requests for assistance. For each theme, we provide sub-themes and representative quotes to highlight findings. A summary of themes and sub-themes can be seen in Table 2.

Table 2.

Themes and Sub-Themes.

Themes	Sub-themes
Positive experiences with Medicaid	Caring providers
	Safety net service
Limitations of coverage	Scarcity of providers
	Poor dental coverage
	Restrictions on other services
Requests for assistance	Needs not adequately addressed
	Challenges associated with children with special healthcare needs

Positive Experiences With Medicaid

Many of the written comments from Medicaid recipients conveyed positive experiences with the state health insurance program, as demonstrated by the following statement, “We have been treated with respect and the best possible care.” In addition to positive comments and notes of appreciation, Medicaid recipients highlighted experiences with specific providers and systems.

Caring Providers

There were numerous examples in which individuals identified healthcare providers by name and commented on their care and compassion for patients. For example, one set of caregivers stated, “This child is niece we (have) been raising most all her life. (Name of hospital) and (Name of physician) has been nothing short of fantastic to us.” Likewise, in diagnosing her child's allergy to corn and corn syrup, one mother suggested that the physician probably saved her daughter's life, “None of the other doctors cared to check and see why she threw up and cried all the time. He (physician) cared enough to find out and work(ed) hard to fix the problem … She is now 15.” Medicaid recipients also expressed appreciation for how their physicians made them feel, “He takes time to answer and explain everything I ask;” “(Name of physician) makes you informed of benefits of even mild exercise and activity … and always treats (us) with respect.”

Safety Net Services

Survey respondents frequently noted that they would not have health insurance were it not for Medicaid. One caregiver stated:

I want to say thank you to everyone for the medical help and to the government for providing Medicaid to my kids. At least I know my kids are provided with great care when needed by doctors and nurses. It's a service I did not have growing up as a kid but was greatly needed. My kids’ (2) doctor is intelligent and knows his medicine. I am very grateful for him for healing my kids when sick. Thanks.

For individuals with limited financial resources or on fixed incomes, Medicaid seemed to fulfill its role as a safety net service provider. One Medicaid recipient simply stated, “I don't know what I would have done without it.” Similarly, an individual who self-identified as a single mom of 3 sons said, “I am truly blessed for the situations I have been in … the state-funded program has helped me so much and I appreciate it.”

Limitations of Coverage

Despite positive experiences with Medicaid, survey respondents described limitations with Medicaid coverage, including difficulty finding healthcare providers who accept Medicaid, gaps in coverage regarding dental services, and restrictions of other types of health services. Medicaid recipients frequently expressed anger and frustration with access to healthcare and dissatisfaction with the quality of care.

Scarcity of Providers

Survey respondents were critical of the low number of providers in the state who accepted Medicaid reimbursement for services, and regularly noted the distances they had to travel to see a provider who would accept their insurance. One survey respondent observed, “The trouble with Medicaid is not with the insurance, (but) it's the doctors. The personal care or family doctor won't take Medicaid. You can't find a good doctor. They won't take it (Medicaid insurance).” Another respondent stated, “The reason I have not seen a doctor, the doctor that was assigned to me is about 50 miles from me. I called (and) they told me to go to (different location) but he is never in his office there so I just stopped trying.”

Poor Dental Coverage

The most common complaint received from Medicaid recipients was the inadequacy of dental coverage through Medicaid. In this state, Medicaid does not cover dental care for adults and only covers routine preventative and restorative services for children under the age of 21 who have full Medicaid eligibility. ²⁰ Adults with Medicaid seemed acutely aware of this limitation: one survey respondent clearly stated, “You all need to have dental coverage. Need it badly. Have very few teeth.” Another respondent provided a rationale for providing this type of service for adults, “Infections in teeth cause other major issues, but Medicaid won't cover it!” Parents and caregivers also expressed frustration with access and coverage of dental services for their children. One stated, “I have a daughter needing wisdom teeth pulled and it's been over a month and Medicaid has NOT approved treatment.” Another said, “Our burden has been dental work, thousands of dollars. When will Medicaid figure out that poor dental status can cause many other health problems?”

Restrictions on Other Services

Medicaid recipients also described frustrations regarding access to equipment, services, and prescriptions. One individual noted that he had recently gotten a new pair of glasses that had broken after 3 months. He was informed that Medicaid only pays for one pair of glasses every 3 years. He stated, “I can't see anything without my ‘second pair of eyes.’ I am blind and can't see without glasses … one pair of glasses every three years, what do you do in case of emergency?” Another survey respondent simply listed his concerns that had not yet been addressed by Medicaid, including: “help with paying for medication, oxygen therapy and supplies, help with required over-the-counter medications/supplies required, duties of Medicaid care network team, services to help with the above that are not wild goose chases, and a wheelchair.” One individual challenged the current policy on prescription medications:

Why didn't you ask about people who need more than the allowed monthly prescriptions and can't afford to pay for the rest? Like me, I use heart meds and psych. meds … Keep taking medications we need off the list that y’all pay for (and) you won't have to worry about some people too long, they are just going to pass away because of government greed.

Additionally, a caregiver described the challenges of accessing mental health services in the state, “(My daughter) sees her psychiatrist via Skype for 15 minutes every 3 months. Very difficult getting refills filled when out of psych meds!”

Direct Requests for Assistance

In addition to comments targeted to specific issues, many of the written comments were simply direct requests for further assistance or appeals for help, which may suggest that current or future needs are not being adequately met. Parents and caregivers also identified unique challenges of caring for children with special healthcare needs.

Needs Not Adequately Addressed

Several survey respondents viewed the CAHPS 5.0 survey as an opportunity to ask for help beyond current levels of support. For example, one individual chronicled her various health conditions since 1983 and said, “I have chronic pain and in need of a physician that can treat me. Please help me find a good doctor.” Similarly, a mother noted changes in her daughter's health and indicated that her physician would not run a thyroid test despite the prevalence of hyperthyroidism in her family. In these text entries, individuals frequently left their names and contact information in the hopes that someone would respond to their needs.

Challenges Associated With Children With Special Healthcare Needs

Parents and caregivers were outspoken in their criticism of Medicaid in caring for their children with special healthcare needs. One individual wrote:

The main problem we have with Medicaid is that it does not give her enough days to see a doctor. We always run out. (Name of child) has multiple disabilities and she has to see her regular doctor and all her specialists. Please see what you can do to help us.

Similarly, a parent described the multiple challenges of providing services for her daughter:

(Name of child) is autistic and many dental offices are not properly equipped to handle cleaning or caring for her properly. They all want to strap her down or yell at her. But she HATES brushing her teeth. Dentists are not understanding and offer no solutions.

Discussion

The majority of unsolicited comments were either moderately negative or very negative in attitude (or tone) for adult and child surveys. This finding is consistent with research by Poncheri and colleagues who noted that unsolicited comments, like open-ended responses, are more likely to be disproportionately negative in tone. The authors explained that individuals completing surveys may concentrate on negative information “to bring attention to areas where they perceive improvement is needed.” ²¹ (p. 616) Yet, we noticed that individuals who shared positive comments frequently named specific healthcare providers or care coordinators who provided excellent service, particularly in cases involving children. Individuals who wrote negative comments, on the other hand, were more inclined to criticize systems and institutions. ²²

Respondents expressed very real concerns about their unmet healthcare needs, especially related to dental care and challenges among children with special healthcare needs.^23,24 Currently, Medicaid provides limited dental care for adults and children. ²⁵ Medicaid recipients in this study framed the lack of dental care as a quality of life issue and noted that dental health is often associated with or a precursor to more serious health concerns. Similarly, parents and caregivers of children with special healthcare needs suggested that Medicaid is simply not designed to meet the needs of children who require more than a typical patient. Comments by these individuals indicated that the one-size-fits-all approach to state health insurance lacks the flexibility to address patients with medically complex health concerns. ²⁶

With respect to the varying percent of surveys with unsolicited comments from year to year, there is nothing in the comments per se that indicates a rationale. That more comments were received on the CAHPS 5.0 adult survey in 2016 (13.0%) as compared to 2017 (5.8%) and 2018 (5.1%) could be related to the state's decision not to expand Medicaid coverage under the Patient Protection and Affordable Care Act (ACA). ²⁷ Similarly, more comments were received in 2018 (9.2%) on the CAHPS 5.0 child survey as compared to 2016 (3.9%) and 2017 (3.6%). Again, it is unclear if this greater number of comments can be attributed to a specific program change or was simply a sample characteristic. Therefore, comparing unsolicited responses across states that have significantly different features or populations in their Medicaid programs would likely be useful for policymakers.

Conclusions

The CAHPS Health Plan Survey (CAHPS 5.0) collects invaluable information regarding the experiences of consumers with their health plans, and these data inform healthcare policies at both the state and federal levels. However, the forced-choice format of the survey does not provide space for individuals to elaborate or further explain their responses. As demonstrated by this current study, Medicaid patients who had concerns or comments about their health plans or healthcare providers found creative ways to share this information by writing comments in the margins or adding supplemental notes to the survey. These actions are consistent with empowerment theory, which suggests that individuals will use whatever tools are at their disposal to advocate for their own self-interests. Medicaid recipients represent some of the most vulnerable citizens due to socioeconomic and/or disability status. By providing written comments on the survey, respondents ensured that their stories did not get “lost” within the standardized scoring rubrics of CAHPS 5.0.^28,29

We believe that deliberately providing space for Medicaid patients to share comments, as well as receiving further guidance for qualitative content and sentiment analysis, would provide invaluable context for interpreting overall survey results. Moreover, a robust monitoring and evaluative process would allow programs to respond to patient comments in meaningful ways and to continue to respond to the frequently complex and challenging experiences of navigating a continually evolving state health insurance program by the most vulnerable populations.

Limitations

There are limitations to secondary data analysis that must be considered. Unsolicited written comments were submitted voluntarily by individuals who completed the postal version of the adult and child CAHPS 5.0 survey between 2016 and 2018; however, responses were anonymous and did not allow for follow-up questions or clarification. Findings cannot be generalized beyond survey respondents who submitted unsolicited comments. Additionally, findings were limited to Medicaid recipients from one state. Future researchers are encouraged to build upon the results of this study and consider a larger sample size including Medicaid recipients from more than one state.