Leveraging Free-Form Comments to Assess and Improve Patient Satisfaction

Abstract

This study employed a text-analysis methodology to identify themes within patient comments and measure the relationship of those themes to patient satisfaction. Using these findings, a spreadsheet tool was created to allow a large sample of comments to be readily analyzed. The tool was validated using patient comment data provided by the Family Medicine Residency of Idaho. The tool gives clinicians the ability to easily analyze patient comments and identify actionable measures of patient satisfaction. Additionally, this tool will allow researchers to reduce vast sets of comment text into numerical data suited for quantitative analyses.

INTRODUCTION

Health care organizations frequently use patient-reported satisfaction data to measure performance and identify opportunities to improve the quality of care.^1-3 Accurately assessing patient satisfaction has long vexed health care clincians.² Often clinicians attempt to measure satisfaction using free-form comments from patients.^4,5 Software tools, which streamline the assessment of comments, have been developed; however, these tools typically require users with advanced data analytics skill sets.⁶

A team of 3 operations management professors and 2 family medicine physicians collaborated to develop a system for clinicians to easily analyze large sets of patients comments and identify drivers of patient satisfaction. We employed text-analysis techniques to find recurring themes within patient narratives and to assess their relationship with satisfaction. Recurring themes were then used to build a spreadsheet-based comment analysis tool.

METHODS

Identification of Themes Within Comments

Manifest text analysis leverages an analytical technique known as Centering Resonance Analysis. It was used to identify recurring themes in a sample of 4,024 online patient comments (see Corman et al).⁷ This analysis reduces comments into a network of nouns and adjectives and identifies the influence of the words (measured as the frequency with which a word is positioned such that it connects other words).⁷ The resulting data set comprises the words in the comments, each with an influence score. Factor analysis of the data set revealed 15 themes of frequent key words (SAS JMP 8 User Guide; SAS Institute Inc). An external panel (4 family practitioner physicians, 1 hospital pharmacist, 1 internal medicine physician, and 1 medical office manager) reviewed the validity of the themes and helped name them.

We then examined how themes related to quantitative assessments of patient satisfaction, using a rating included with each comment. Ratings ranged from 1 to 5. The distribution was bimodal, so we used logistic regression where the dependent variable (the rating score) was recoded into a binary variable in which 1 denoted a rating of 4.5 or higher (very satisfied) and 0 represented a rating less than 4.5 (less satisfied). Regression found 14 themes had significant relationships with the patient rating (7 positive, 7 negative). The themes, key words, their odds ratios (likelihood that the theme is related to a satisfied rating relative to the probability that it is related to a less-satisfied rating), and representative comments are in Table 1.

Table 1.

Themes and Key Words

Theme No.	Full Theme Name	Abbreviated Theme Name	Theme Key Words	Relationship to Satisfaction (P Value) Odds Ratioa	Sample Review Comment
1	Appointment scheduling difficulty and excessive waiting time	Scheduling	Quantitative number; minute; appointment; month; day; hospital	Negative (<.001) 0.701	“This is the worst office in regards to patient care. Expect long wait times and it’s nearly impossible to get the staff to answer the phone. One week I called 25 times and no one answered the phone. The last time I saw the doctor he did some extensive blood work and said to follow up in three months. His schedule was out almost four months and then the day before my appointment he decided to go out of town, so my appointment was canceled. The office didn’t reschedule the appointment, they said they would ‘work me in.’ That was a month ago. I asked for a copy of my blood work and the results are three times the normal limits. I won’t be going back to this office. If I was having a serious medical problem that needed to be addressed I don’t think I would ever be able to see the doctor in a timely manner. He is very knowledgeable but it doesn’t do any good if you can’t ever get an appointment or speak to the nurse.”
2	Professionalism: compassionate and friendly care	Professionalism	Professional; friendly; comfortable; adult dependent	Positive (<.001) 1.661	“Dr. _ _ _ _ _ is very professional both my spouse and I are very comfortable in his office and care. I would recommend him to anyone. His staff is Top notch very friendly as well as professional.”
3	Bedside manner and information exchange	Bedside manner	Question; time; bedside manner; care; due; good	Positive (.008) 1.212	“Excellent bedside manner, always takes the time to answer questions and is very good with kids.”
4	Staff is rude and unhelpful	Rude staff	Office; staff; service; rude	Negative (<.001) 0.712	“Did not help, insisted on multiple useless appointments and is rough and rude. Office staff rude and impatient.”
5	Problem with treatment process (issue with laboratory tests and/or diagnosis)	Treatment concern	Test; treatment; diagnosis; result; way	Negative (.001) 0.766	“Initially ordered many tests, but never looked at test results until my next exam, when I had to brief the doctor on what and why we were doing tests and treatments. Staff forgot to order last test series. Didn’t even see doctor on last visit, only nurse and PA. Wanted to treat me for sleep apnea with no tests or diagnosis, other than a visual look at me. Ended with no diagnosis and I won’t return.”
6	Caring and knowledgeable treatment	Caring and knowledgeable	Only; caring; knowledgeable; health; year	Positive (<.001) 1.522	“Excellent, very caring doctor. Very knowledgeable and kind.”
7	Problems with management and communication of medical condition	Management and communication	Medical; problem; call; female; condition	Negative (.005) 0.806	“Had two common conditions that were undiagnosed for several years. One was self-diagnosed and finally confirmed by testing. Resulting treatment required follow up testing on a periodic basis by established medical protocol. The doctor did not order any follow up testing at all after nearly one and a half years of treatment. A new doctor had ordered six follow up tests in year resulting in the discovery that I was being over treated by nearly 100% as the result of a simple calculation error. Condition is stable and being treated appropriately. Ironically, I was called by his office staff as a reminder for the same type of follow-up I needed, but was for another patient. Based on the phone call, that patient requested in writing for the follow up testing to be done since the doctor had not done so. The other more significant condition was diagnosed by another doctor on the second visit. The condition has a couple dozen symptoms as listed in many resources of which I have nearly all of them.”
8	Excellent experience with children	Experience with children	Wonderful; amazing; experience; child dependent	Positive (<.001) 1.363	“Dr. _ _ _ _ _ is wonderful and does great with my kids!”
9	Word of mouth (recommend to friends and family)	Word of mouth	Friend; family	Positive (<.001) 1.974	“Dr. _ _ _ _ _ was great doctor. He answered all my questions and made me and my family feel comfortable. I would recommend Dr. _ _ _ _ _ to any of my friends and family.”
10	Treated rudely in the examination room	Treated rudely (exam room)	Rude; room; different; long; phone	Negative (<.001) 0.681	“Horrible, he is not even a doctor, he is a physician’s assistant. I hate how the medical field is replacing doctors with PA-Cs, nurses with medical assistants, and so on. He was incredibly rude and left me in the exam room for 20 minutes while he went out and talked about me with another co-worker, I could hear through the walls. It was extremely unprofessional, and I would have much rather seen a real doctor. This guy was an absolute joke and they should make it known when they enter a room that they are not a physician but a physician’s assistant and the patient should have the option to be seen by a physician, since we are the one’s paying the money.”
11	Pain untreated	Pain untreated	Pain; back	Negative (.009) 0.837	“Doctor was very vague and could not help me with my back. I understand doctors sometimes do not always have the answers. I suffer from horrible low back pain, tried many different remedies such as chiro, physical therapy, steroid injections, nothing was helping. I was very clear I wanted nothing to do with narcotics but still made me feel as though I was making up the pain and would not hear of any ideas I had regarding research I did on my own out of desperation. Going for a second opinion!”
12	Knowledgeable and willing to treat ailment	Knowledgeable and willing	Willing; helpful; visit,; knowledgeable	Not significant (.587) 0.956	“Great visit, always know he is helpful and willing to look into my questions or concerns.”
13	Great job	Great job	Great; job; nice	Positive (<.001) 1.531	“Very professional and very good at their jobs. Easy to work with. Very kind and nice people too. I couldn’t ask for a better experience. I read the previous negative review. WOW. What a wacky job of a person. These are great dentists and great people. Me and my family have had a great experience there. Reasonably priced and always willing to work us in.”
14	Doctor overemphasizes money versus care	Overemphasis on money	Patient; doctor; new; money; medication	Negative (<.001) 0.601	“Dr. _ _ _ _ _ is the worst kind of human. My mother was being treated by him. He makes numerous prescription errors. Wrote RX’s for medicine that was either on my mom’s allergy list or was known to cause severe or LETHAL side effects due to her allergies. She lives in chronic pain due to 4 failed neck surgeries. The only pain medication she can take to control this is Demerol. He refused to renew the RX so my mom started looking for a new doctor. When Dr. _ _ _ _ _ found out this he wrote a letter to other doctors stating that my mom was addicted to pain killers (even though her prescription had NEVER changed in quantity or dosage in 7 years) and was an alcoholic (she doesn’t drink). Now she can’t find another doctor since they all believe this. Dr. _ _ _ _ _ has made comments to her that ‘he doesn’t make any money off her because she is on Medicaid and Medicare.’ In my opinion Dr. _ _ _ _ _ is unstable and thinks that patients don’t deserve the best care possible. AVOID HIM AT ALL COSTS!”
15	Positive procedure outcome by surgeon/doctor	Positive outcome	Hospital; procedure; surgeon	Positive (.003) 1.230	“The best of the best in this field. Highly skilled as a surgeon, well respected by surgical/hospital staff. Made sure you understood what procedures you were going through and truly cared how you felt physically and emotionally. Her office staff were always very friendly and very caring. I would highly recommend Dr. _ _ _ _ _ if you need a plastic surgeon.”

PA = physician assistant; PA-C = certified physician assistant; RX = medical prescription.

^aOdds ratio values are Exp(B) = the exponential value of B.

This study was determined to be exempt from each of the 4 authors’ institutional review boards as it did not involve human subjects.

Developing a Comment Analysis Tool

We leveraged the findings of the theme identification process to create the clinician Review Comment Analysis Tool using Microsoft Excel (Microsoft Corp) which allows users to analyze up to 5,000 comments. To use the tool, users simply paste in patient comments. The tool then automatically measures the prevalence of each theme’s key words. This is used to create a summary (Figure 1), that applies the odds ratios from the regression, to predict the likelihood that a patient will leave a positive review, as well as the relative prevalence for each theme.

Figure 1.

Family Medicine Residency of Idaho comment analysis tool results.

RESULTS

The tool was tested using 721 patient comments collected during 2019 by the Family Medicine Residency of Idaho (FMRI). As shown in Figure 1, FMRI patients were more likely to leave a highly satisfied review than a less-satisfied review (by a ratio of 1.5 to 1). The finding that Theme 1 (Scheduling) was the highest-loading negative theme validated ongoing efforts to improve these processes. Feedback from the FMRI staff indicated they appreciated the ease with which the tool allowed them to quickly analyze comments and provide insights into areas of success and potential improvement.

DISCUSSION

The findings of this study can be used in several ways. First, by understanding which themes have the greatest influence on patient satisfaction, clinicians can monitor their interactions with patients to ensure that satisfaction is emphasized within their practices. Second, given the proliferation of online reviews, clinicians can read through comments to see which themes frequently appear. For larger sets of comment data, the spreadsheet tool, which will be provided free-of-charge to interested clinicians, gives users (without specialized skills) actionable measures of patient satisfaction.

A limitation of this study is that patient comments typically reflect how their expectations were met, which does not directly equate to the quality of care received.¹ Nevertheless, patient experiences and outcomes are not unrelated, as satisfaction may lead patients to be more involved in their care.² An understanding of patient satisfaction may also highlight systemic issues and identify underlying drivers of outcome-related underperformance.⁵ Thus, measuring how comments reflect the themes provides insights ranging from specific, addressable actions of clinicians to breakdowns of organizational processes. The tool we developed, however, represents only 1 input to a comprehensive quality program, as patient satisfaction should not be the sole focus of improvement efforts.³

The tool developed can also be leveraged by future researchers. Researchers can reduce vast sets of comment text into numerical data, suited for quantitative analyses, using the thematic scoring method. The tool also facilitates longitudinal analyses of patient feedback; for example, a follow-up to this effort is an investigation of how FMRI’s patient satisfaction was impacted by COVID-19. Additionally, the use of this tool for future research is suited for today’s family medicine environment, as the broad dimensions of patient experiences encompassed by the themes share commonality with aspects of patient-centered care.⁸