When Patient-Reported Experience Does Not Match Change In Clinical Outcomes: A Perplexing View From The Inside Of A Diabetes Distress Intervention

Abstract

Aims:

To assess between-group differences in participant experiences in a two-arm diabetes distress (DD) reduction RCT and to determine their relationship to clinical outcomes (reductions in DD and HbA1C).

Methods:

For high DD adults with Type 1 diabetes and HbA1c ≥ 7.5% participating in T1-REDEEM, we evaluated post intervention 5-point ratings of overall program “helpfulness” and program component “helpfulness,” along with open-ended feedback statements using 10 qualitative codes. We compared responses of those in OnTrack, a distressed-focused intervention, with KnowIt, an education/management intervention.

Results:

Those in OnTrack reported significantly higher levels of overall program helpfulness and greater helpfulness of each component of the program, greater group support, far fewer negative experiences, and more active and meaningful group engagement than those who participated in KnowIt. Ratings of helpfulness were unrelated to reductions in DD and HbA1C in both study arms. As previously reported, these findings occurred despite significant reductions in both DD and HbA1C in both arms with no between-group differences.

Conclusions:

Findings highlight the importance of addressing the personal experience of diabetes interventions in clinical care as separate, distinct outcomes. Personal experience may not always be related to changes in traditional clinical indicators.

1. Introduction

Given the accumulating evidence of a significantly negative impact of high diabetes distress (DD) on self-management, glycemic control and quality of life among adults with type 1 diabetes (T1D)^1–3, new efforts have focused on developing programs to reduce high DD in clinical settings. Most studies have been randomized clinical trials that compare educational, behavioral or emotion-focused interventions with usual care or with other appropriate control groups^{3, 4}. The results have been encouraging: DD has been shown to be malleable with intervention, with results often showing dramatic reductions in DD that are sustained over time^{5, 6}. Most interventions, however, are complex and are delivered using multiple modalities, making it difficult to identify the components that are most critical in alleviating DD and informing clinicians about how these programs “work.” Furthermore, there are little data reflecting how participants experience interventions, what they find helpful and why⁷. This information is crucial as new programs to reduce DD are developed in ways that capture what participants find to be most helpful and compelling. We addressed these problems by gathering information from the “inside,” by asking highly distressed adults with T1D and HbA1C ≥ 7.5% (58 mmol/mol) to describe their “lived experience” regarding their participation in a DD-reduction intervention called T1-REDEEM.

T1-REDEEM (Reducing Distress and Enhancing Effective Management) was a randomized control trial to identify which of two interventions, one focused on diabetes education/management (KnowIt) and the other on the emotional side of diabetes (OnTrack), most effectively reduced DD among highly distressed, adults with T1D and HbA1C ≤ 7.5%⁶. Both interventions led to striking reductions in DD and modest reductions in HbA1c at three months, both of which were sustained at nine months.

This report focuses on a mixed-methods analysis of participant responses to both structured and open-ended questions, comparing the experience of participation and the perceived helpfulness of each program component in each intervention group. We also explored the relationship between ratings of program “helpfulness” and reductions in DD and HbA1C over time. This information is essential to inform the development and evaluation of future pragmatic interventions for use in clinical care⁸.

2. Materials and Methods

2.1. Study design and participants

Details of recruitment and methods have been presented previously⁶. Briefly, adults with T1D who scored ≥ 2.0 on the T1-Diabetes Distress Scale (T1-DDS) and whose most recent HbA1C result was ≥ 7.5% (58 mmol/mol) were recruited from multiple community and academic diabetes centers across several western U.S. states and Toronto, Canada to assure a highly diverse sample. Participants were ≥ 19 years of age, were diagnosed with T1D ≥ 12 months and displayed no severe diabetes complications or other major health problems that were functionally limiting. Human subjects approval was received from the appropriate Institutional Review Board (IRB) at each site, including the UCSF Institutional Review Board, and recruitment followed a combination of opt-in and opt-out procedures. Using opt-out procedures, the research team mailed letters to patients identified on their clinic registry informing them of the study and telling them that a project representative would contact them by phone within 2 weeks unless they opted out of the call by returning an enclosed postcard, emailing us, or calling a toll-free telephone number. Using opt-in procedures, individuals receiving letters were encouraged to call our toll-free number or send an e-mail expressing interest. During initial contact with those identified by either recruitment procedure, the project was explained, informed consent was obtained, and initial screening commenced, including administration of the T1-DDS and permission to obtain their latest clinic-recorded HbA_1c.

Once recruited, participants completed an online baseline assessment and either provided consent to obtain their most recent clinic HbA1C result or were provided with a form to obtain an HbA1C test at a local laboratory. Participants then were randomized to one of the two arms of the trial: KnowIt or OnTrack. Both study arms required the same time commitment: a one-day workshop with a trained leader, followed by participation in four, one-hour, online group video meetings with the group leader over the succeeding three months. Participants also received a personal phone call from their group leader approximately one week prior to each of the four online group meetings to see how they were progressing with their action plan and to encourage attendance at the online meeting. At 3- and 9-months after the workshop, each participant was asked to complete an online survey and additional HbA1C test. Both structured and open-ended survey items about their experience in the program were completed at 3-month assessment. The 3-month data were considered primary because of their temporal closeness to the intervention. Data were collected between 2015 and 2017, and analyzed in 2018.

KnowIt was an information-based educational/management program that provided an intensive diabetes update on the causes and management of T1D. Each of the four subsequent online meetings reviewed participant action plans for management change and addressed a specific topic: continuous glucose monitoring, islet and pancreas transplantation, hypoglycemia, and travel. In contrast, OnTrack focused on ways of dealing with the emotional side of diabetes and to develop a personalized action plan that addressed DD directly to help get “unstuck” about management change. Specific scenarios and exercises were presented to enhance discussion and to illustrate methods for better management of DD. The four post-workshop online group meetings addressed: dealing with T1D 24 hours a day, coping with frustrating blood glucose numbers, dealing with family and friends, and addressing fears of hypoglycemia. Thus, the two arms enabled a comparison between an educational/behavioral change approach to reducing DD (KnowIt) and a program that addressed high DD and emotion management directly (OnTrack). Upon completion of each of assessment, participants received a $25 gift card.

2.2. Measures

Participant demographics and disease status were recorded at baseline: age, gender, years with T1D, and HbA1C. DD was assessed by the T1-DDS, a 28-item scale (alpha = .91)⁹ that assesses overall level of DD, with items scored using six response options, from “not a problem” to a “very serious problem.” A mean score across all items was calculated.

Structured items:

Eight structured “helpfulness” items were included in the 3-month assessment. These items were scored on a 5-point scale from “not at all helpful” to “extremely helpful,” and included ratings of the overall program, day-long group workshop, information presented by the facilitator, individual telephone calls with the facilitator, online group meetings, and group support. Participants also were asked if they communicated with other group members outside of the program (yes/no) and if they had plans to do so (yes/no).

Open-ended items:

At the completion of the 3-month online assessment, participants also were asked, “Is there any other feedback you would like to share about your experience in T1-REDEEM?” An unlimited free-text box was provided.

2.3. Data analyses

Using SPSS version 19 software, student t tests or χ², as appropriate, compared the two treatment conditions on participant characteristics and baseline values of outcome variables, along with tests for differences between dropouts and completers of three month assessment. Student t-tests examined differences in helpfulness ratings by intervention group. Repeated Measures ANOVA (RM-ANOVA) tested for differences in perceived intervention helpfulness among intervention components, differences between intervention groups in helpfulness, and the interaction between intervention group and specific components in perceived helpfulness. For open-ended responses, all comments, other than “thank you,” etc., were recorded onto an ATLAS.t1 (v 8.0) data base and reviewed to harmonize and organize the comments into generic categories based on content until saturation was reached. i.e., no new categories emerged. Comments were further refined into sub codes for each category so that each comment could be classified into a single generic category and sub code¹⁰. The number of comments in each category and code were then calculated. Coding of the entire sample was undertaken by a single coder (VB), with a sample of 20% of responses coded by a second coder (LF) to assess reliability, resulting in 89% of individual coding instances given by one coder, replicated by the second coder, with a minimum of 85% agreement on any single code. Reconciliation occurred where the two coders differed.

3. Results

From the original sample of 301 participants, 267 (88.7%) completed the 3-month assessment that included both the structured and open ended items (KnowIt n = 136; OnTrack n = 131). A description of both samples appears in Table 1. As previously reported⁶, those who did not complete the 3-month assessment were significantly younger, had higher baseline DD and HbA1c level, and reported more complications than those who completed the 3-month assessment. There were no between-group differences in attrition from baseline to 3 months; however similar to the baseline sample, in the 3-month sample KnowIt participants were slightly older than OnTrack participants (KnowIt = 47.4 [14.4]; OnTrack = 43.2 [14.8] years, p=.05). Of the 267 completing the 3-month assessment, 172 participants (KnowIt n = 81; OnTrack n = 91) provided an open-ended statement about their experiences in the program (72%). Those who did vs. did not provide open-ended statements did not differ by intervention group nor by any patient characteristic studied.

Table 1.

Participant Characteristics.

	Survey (structured) respondents n=267 Mean (SD) or %	Open-ended item respondents n=172 Mean (SD) or %
Age (years)	45.5 (14.9)	46.6 (15.0)
Age at type 1 diagnosis	20.8 (16.0)	21.2 (14.0)
Sex (% female)	70.4% (188)	72.1% (124)
Education
% >=12 years	15.4%	14.0%
% 13–16 years	46.0%	44.7%
% 17+ years	38.6%	41.3%
% married/living with partner	66.5% (177)	66.1% (113)
Ethnicity (% non-Hispanic White)	80.5% (215)	80.8% (139)
No. of complications	2.7 (2.5)	2.8 (2.6)

Note: No significant differences between the two groups of respondents were found.

3.1. Responses to structured items

Mean ratings of the structured “helpfulness” items are presented in Table 2. Although the majority of participants rated their experience as helpful overall, OnTrack participants rated overall helpfulness of the program and group support, as well as each of the three components of the program (in person workshop, group web meetings, and individual phone calls) as significantly more “helpful” than KnowIt participants (p < .01 in all cases). We also examined which of the three components of the program was rated as more or less helpful than the others for each group using RM-ANOVA. In addition to confirming the omnibus between intervention group effect (F=14.96, p < .001), there was a significant difference in ratings of helpfulness among program components (F=119.24, p < .001): for both groups, the day-long in-person workshop received significantly higher ratings of helpfulness compared to both the online group meetings and the individual phone calls (both p < .001); likewise, the online group meetings received significantly higher ratings than the individual phone calls (p < .001). The interaction between intervention group and program component was not significant, indicating that this pattern was consistent across both intervention groups. Thus, OnTrack participants rated each aspect of the program as significantly more helpful than KnowIt participants, and both groups rated the live, day-long workshop as the most helpful of the three program components.

Table 2.

Mean (SD) ratings of intervention program “helpfulness” by study arm.

Structured Helpfulness Items	KnowIt Mean (SD)	OnTrack Mean (SD)	p value	Effect size (Cohen’s d)
Helpfulness of Overall Intervention Program	3.13 (1.1)	3.62 (1.1)	<0.001	0.45
Helpfulness of Group Support	3.19 (1.3)	3.72 (1.2)	0.001	0.42
Helpfulness of Intervention Components
In-person group workshop	3.68 (1.1)	4.22 (0.9)	<0.001	0.53
Online group meetings	2.96 (1.2)	3.38 (1.2)	0.006	0.35
Individual phone calls with the facilitator	2.69 (1.3)	3.16 (1.3)	0.004	0.36

We also assessed the relationship between ratings of overall program helpfulness and group support with reductions in DD and HbA1c from baseline to three months for each group. No correlation reached statistical significance. Thus, perceived program helpfulness and group support were unrelated to reduction in DD and reduction in HbA1c over three months as a result of intervention.

Finally, responses from the structured items indicated that 32.1% of OnTrack participants reported having contact with other participants outside of the program, whereas only 12.7% of KnowIt participants did so (Chi-square = 14.37, p < .001). Likewise 53.4% of OnTrack participants reported a plan to contact other group members, whereas only 35.1% of KnowIt participants had similar plans (Chi-square = 9.06, p = .003). Actual or planned contact with other participants was unrelated to reductions in either DD or HbA1C over time. OnTrack participants therefore, appeared to be more actively engaged with other group members than KnowIt participants.

3.2. Responses to the open-ended question

Open-ended comments were classified into five categories, with a further division into 11 sub codes (Table 3). Many of the comments in both programs were positive, with several participants specifically calling out the supportive and beneficial experience of being in a group with other adults with type 1 diabetes.

Table 3.

Number and percent of participants providing coded responses to the open-ended question.

Category	Sub-code	KnowIt n (%)	OnTrack n (%)
Positive program experience	Supportive or beneficial connection with rest of group	11 (10.8)	19 (15.4)
	Learned from/shared with group	6 (5.9)	4 (3.3)
	Positive feedback about facilitator	6 (5.9)	13 (10.6)
	Learned something new from content	5 (4.9)	6 (4.9)
	Total	28 (27.5%)	42 (34.2%)
Negative program experience	Negative online group meeting experience	15 (14.7)	4 (3.3)
	Knew all the material already	24 (23.5)	7 (5.7)
	Lacked a connection with the group	7 (6.9)	4 (3.3)
	Total	46 (45.1%)	15 (12.3%)
Did something different	Did something different in regard to their diabetes management as a result of participating	4 (3.9%)	12 (9.8%)
New perspectives	Being around other T1D adults normalized the struggles of living with T1D	4 (3.9)	23 (18.7)
	Gained new perspective on diabetes	2 (2.0)	21 (17.1)
	Total	6 (5.9%)	44 (35.8%)
Wants more contact	Wants more contact with group or facilitator	16 (15.7%)	14 (11.4%)

OnTrack male, age 42: “Having a chance to see what other people with diabetes were experiencing was a great help to me.”

Striking between-group differences in number of comments made occurred in two coding categories. Negative comments about the program were much less frequent among OnTrack than KnowIt participants (12.3% vs. 45.1%). The majority of negative comments focused on how they were already familiar with program material and content, and that they learned nothing new.

KnowIt female, age 55: “The material presented was all stuff I had seen before, and easily available online or in books. Nothing new.”

In contrast, many more OnTrack than KnowIt participants reported gaining greater perspective about their diabetes (35.8% vs. 5.9%). OnTrack participants shared how powerful it was to have their lived experience of type 1 diabetes normalized.

OnTrack female aged 26: “It was comforting to know that I am not alone with my struggles and that people like me live day to day without drama to their everyday life. It felt normal to be in a room full of other t1ds.”

OnTrack female, aged 63: “The information presented was positive and believable (doable) especially reinforcing getting away from negative (guilty, unhealthy) feelings about various aspects of diabetes.”

Last, more OnTrack than KnowIt participants noted changes in their management as a result of participation (9.8% vs 3.9%).

OnTrack male, age 47: “Gaining some perspective about how diabetes was making me feel, I was able to try some different ways of managing my glucose levels. I felt a bit freer to try things and to see how they worked.”

4. Discussion

4.1. Conclusions

Our earlier findings indicated that both OnTrack and KnowIt participants displayed significant and dramatic reductions in DD over three months as a result of intervention, with no between-group differences⁶. The current findings, however, indicate important differences in the reports of personal experiences of OnTrack and KnowIt participants: those in OnTrack report significantly higher ratings of overall program helpfulness and group support than those in KnowIt. Furthermore, OnTrack participants report fewer negative experiences, greater perspective about diabetes, more actual and intended contact with group members, and more management changes than those in KnowIt. Finally, each program component was rated as significantly more helpful by OnTrack than by KnowIt participants, although both groups rated the full-day group workshop significantly more helpful than other program components.

Although both interventions required the same time commitment and included the same program components, the content of the two interventions differed dramatically. OnTrack focused entirely on the emotional side of diabetes. Emphasis was placed on identifying the common emotional demands of living with diabetes, labeling and addressing difficult feelings (anger, frustration, guilt), facing the effects of management on self-esteem, and dealing with sometimes insensitive friends, co-workers and family members. Discussions focused on placing these difficult feelings in context, normalizing them and separating feelings from actions. In contrast, although KnowIt also capitalized on group interaction, it focused exclusively on a diabetes information update that included the known causes of the disease and a review of best practices for good diabetes management. Participants worked to identify a specific self-management problem and to develop a structured action plan to resolve the problem.

We suspect that, in addition to the lack of new information among already well-informed and well-educated KnowIt participants, the greater emotional intensity and content of OnTrack may underly the between-group differences of helpfulness and other positive program experiences. Many were surprised by the affective focus and shared that even after decades of having diabetes, no one had ever asked them about what it was like for them to struggle with the disease, how diabetes affected their lives and what their personal experience of diabetes had been. Group discussion often was intense and quite personal, with many sharing very private experiences coupled with tears. Although active group discussion also occurred in KnowIt, it did not reach the level of intensity and emotional intimacy that we saw in OnTrack, focusing instead mainly on specific management experiences. Thus, the greater emotional focus and intensity of the OnTrack experience may have contributed to the reports of more positive personal reactions to OnTrack than KnowIt.

Given these findings, it is surprising that we found no between-group differences in DD reductions from baseline to 3-month assessment, as reported earlier. One might expect that those who report a more positive intervention experience might display larger reductions in DD and/or HbA1C over time. Three considerations are worthy of note. First, as documented in previous research⁸, the impact of group process most likely contributed to the dramatic reductions in DD that occurred in both groups. That is, both the emotional focus of OnTrack and the management focus of KnowIt led, perhaps in different ways through different mechanisms, to equally substantive DD reductions. Thus, it is most likely that significant reductions in DD can occur as a result of different types of interventions. Furthermore, the considerable reductions in DD, most likely a result of group process, may have left little room for the more subtle influence of program content to drive even greater change in DD and HbA1C in OnTrack than in KnowIt.

Second, we wondered if the somewhat complementary literature that explores the relationship between treatment satisfaction and change in clinical outcomes as a function of intervention might help explain our findings. While the relatively sparse literature that compares treatment satisfaction directly with improvements in clinical outcomes points to significant linkages in some studies, other studies show only inconsistent and modest linkages and still others show no relationship at all^11–14. We suggest, therefore, that there are often unrecognized, subject perceptions and experiences that contribute to ratings of treatment satisfaction or, in our case, “helpfulness,” that may or may not be related to changes in specific clinical outcome measures, such as DD.

Third, these findings suggest that it may be worthwhile for future studies to include more highly targeted patient experience measures as assessment parameters than we included in T1-REDEEM¹⁵. Rather than including only global or generic measures, as is generally the case, such measures need to be clearly defined by the substance and content of the intervention so that they have relevance to the specific changes that the protocol hoped to produce¹⁶. It makes little sense, for example, to employ only a broad and generic measure of quality of life, such as the WHO5, when studying the psychosocial impact of a new continuous glucose monitor; chances are great that meaningful and specific indicators of participant experience will be missed. This problem shows itself clearly in the current study and it limits what we can say about how participants experienced the intervention. For example, we only asked about ‘helpfulness’ as a generic term and never explored in what specific ways the interventions might have been helpful; for example, helpful in feeling better about one’s life, feeling better about one’s relationship with diabetes, feeling freer to try new approaches to diabetes management, or perhaps something entirely different. Thus, our assessments, while informative, were too limited, which, in turn, prevented a more thorough understanding of participant experience that was directly relevant to the intervention. Furthermore, perhaps ongoing contacts with a small subset of participants throughout the study would have better informed the post-intervention assessment protocol by identifying more critical, participant-defined experiences that we as investigators may not have been fully unaware of at the outset¹⁶.

4.2. Study limitations

Several study limitations need to be kept in mind. First, the study sample was generally white and well-educated, with many using insulin pumps and continuous glucose monitors. Inclusion of a more diverse sample might have yielded a broader range of subjective responses. Second, the structured questions were limited and the open-ended statements were based on a very general question: “Is there any other feedback you would like to share about your experience in T1-REDEEM?” More focused questions or, more importantly, one-on-one interviews might have enabled participants to provide more detailed and comprehensive responses regarding different aspects of their personal experience. Last, although the active intervention period had ended and efforts were made to minimize bias, some participants may have been selective in sharing their impressions of different aspects of the programs.

4.3. Summary

In conclusion, we find that despite no between-group differences in overall reduction of DD as a result of intervention, those who participated in OnTrack, an emotion-focused intervention, reported significantly higher levels of overall program helpfulness and helpfulness of each component of the program, greater group support, far fewer negative experiences, and more active and meaningful group engagement than those who participated in KnowIt, an educational and management intervention. Ratings of program helpfulness were unrelated to reductions in DD and HbA1C following intervention. These findings highlight the unique benefits of addressing the personal experience of diabetes interventions in clinical care as separate, distinct outcomes. Personal experience may not always be related to changes in traditional clinical indicators.

Highlights.

• Findings highlight the importance of including ratings of personal experience in intervention trials.

• Improvements as a result of clinical intervention may not be associated with ratings of participant satisfaction with or helpfulness of the intervention.

• Participant ratings of personal experience can provide important information not reflected by traditional clinical measures.