Abstract
Background
For intervention studies that require the use of participant self-reports, the quality and accuracy of recorded data and variability in participant adherence rates to the treatment can cause significant outcome bias.
Purpose
To assess the quality and accuracy of participant documentation of daily self-monitoring of leg skin temperature, adherence to a graduated cooling treatment protocol to prevent venous leg ulcers, and the potential for bias in treatment effect in a randomized controlled trial that included a population with chronic venous disease.
Methods
Individuals were randomized to a leg cooling intervention or placebo treatment group to daily self-monitor and record lower leg skin temperature over a 9-month period on monthly paper study logs. Returned study logs for the first 100 completed participants (n=54 cooling intervention, n=46 control) were reviewed for quality and accuracy. Adherence was determined from evaluating the accuracy of participant documentation. To examine potential outcome bias in treatment effect, mean between group and within group comparisons of the before and after treatment differences were conducted using an intention-to-treat (ITT) versus a modified intention-to-treat (mITT) analysis approach with an 85% accuracy cut-off rate. Data were collected in 2011–2014.
Results
Of the expected 900 study logs, 91.8% (826/900) were returned and 8.2% (74/900) were not. Non-mutually exclusive main error types in returned documentation included: 59.2% (489/826) white-outs, cross-off and/or overwrites, 34.9% (288/826) entries omitted, 29.4% (243/826) no performance of daily self-monitoring, 28.7% (237/826) no performance of the treatment intervention per the prescribed protocol regime, 26.8% (221/826) extraneous data, 8.6% (71/826) suspected fabrication, and 7.6% (63/826) questionable validity. Under ITT analysis, 38.4% (346/900) of all returned logs were <85% accurate, 25.0% (225/900) were 85%–99% accurate, and 36.6% (329/900) were 100% accurate. Mean overall participant adherence rates were: 22.0% at <85% accuracy, 53.0% at 85%–99% accuracy, and 25.0% at 100% accuracy. Under the mITT analysis, 54.0% (483/900) of returned logs were deemed adherent with ≥85% accuracy.
Conclusion
This study found good rates of adherence. Under ITT analysis, 78.0% of participants were deemed adherent to the study protocol with ≥85% accuracy in documenting daily self-monitoring of skin temperatures in response to a topically applied experimental cooling cuff intervention for the prevention of venous leg ulcers.
Keywords: self care, lower leg, symptoms, prevention intervention, diary logs, chronic illness
Introduction
Among patients with chronic illnesses, it has been estimated that approximately 50%–60% do not take their medications or perform medical treatments as prescribed.1,2 Adherence rates among patients with acute conditions are only slightly higher than those with chronic conditions.3 The effects of poor adherence can lead to increased morbidity and lower quality of life, contributing to additional health care costs of over ≥100 billion per year in the US alone.4
Self-monitoring is considered to be critical in the management of many chronic illnesses. However, the therapeutic benefits and maintenance of successful long-term outcomes are only achievable through frequent patient self-monitoring and adherence to the treatment regime. Many studies have examined the accuracy of self-monitoring and self-reporting using paper diaries or study logs when compared with electronic monitoring devices with mixed results.5 It is estimated that upon average, 20%–40% of self-reports are inaccurate due to a combination of missing data, patient errors in recording, and the suspected fabrication of data.6–8
In clinical trials, the accuracy of participant self-monitoring and reporting as a form of data collection is of considerable concern to researchers. Poor self-reporting can have a profound effect on the quality of the data and confidence in the results. Erroneous documentation, missing data, and non-adherence to the experimental treatment can potentially bias both the estimate of treatment effect9,10 and the safety profile of the study.11 Furthermore, the handling of missing data for analysis can further compound this issue.12 Although problems associated with missing data in clinical studies can, to some degree, be mitigated through statistical imputation methods,13,14 executing intention-to-treat (ITT) analysis can be especially difficult in longitudinal studies that use self-reported data. Due to the frequency of missing data from participant attrition and documentation errors, and suspected fabrication of entries on diaries or logs, it is recommended that researchers perform sensitivity analyses to examine the robustness of the data and the assumptions made regarding the handling of missing data in the primary analysis, and to observe for outcome bias.15
Self-monitoring and self-reporting for ulcer prevention
There is sufficient evidence to suggest that self-monitoring the temperature of the skin can prevent ulcer formation by detecting areas of inflammation.16–18 Now, incorporated into clinical guidelines,19,20 patients with diabetes and neuropathy of the lower extremities are instructed to take the temperature of various areas of the plantar surface of the feet.16 When elevation is detected, they are instructed to self-manage the inflamed areas by offloading the feet to reduce high-pressure areas by wearing properly fitted protective footwear during ambulation. These guidelines and results from our previous studies have provided the basis for the study of skin temperature elevation among a patient population with chronic venous disease (CVeD) and a history of venous leg ulcers (VLUs).21–23 A randomized controlled trial (RCT) was undertaken to examine a VLU recurrence prevention model of daily skin temperature self-monitoring and use of a graduated cooling regime applied topically to the inflamed lower leg skin area. Daily participant record keeping using paper study logs (self-reports) for 9 months was the data collection method to evaluate adherence to study treatment.
The aims of this study were to assess the quality and accuracy of participant self-monitoring, as reported through study logs (Aim 1), and to explore any potential for outcome bias in treatment effect using an ITT versus a modified intention-to-treat (mITT) sensitivity analysis approach with an 85% adherence rate cut-off (Aim 2).
Patients and methods
Study design
This was a longitudinal RCT of participants with stage 4 (skin damage) and 5 (healed VLUs) CVeD recruited from specialty wound clinics and the general population of two southern US states. Flyers and brochures were placed in participating hospitals and wound clinics; television and billboard advertisements were purchased. One of the goals of the trial was to invite participation from rural, underserved, and minority populations, so as to maximize the generalizability of findings.
Participants
Eligible individuals were adults ≥21 years of age, had an ankle brachial index of 0.8–1.3 mmHg (measure of adequate arterial blood flow), intact protective, and thermal sensation of the lower leg, a working freezer and telephone, agreed to wear a compression wrap on the affected leg during waking hours, and could both speak and write in English. Those having recent surgery on the affected leg in the past year, an open ulcer, a diagnosis of peripheral arterial disease, cognitive impairment, or chronic inflammatory or vascular disorders were excluded. Participants received US ≥400.00 compensation for their participation in this study. The study was approved by the Institutional Review Board of the Medical University of South Carolina. Written informed consent was obtained from each participant.
Self-reporting instructions
Participants who provided informed consent were randomized to receive either a gel-filled cooling cuff (intervention group) or a non-cooling placebo cuff (control group) to apply topically to the affected skin for 30 minutes each day. The cuffs were to be placed in a sealed plastic bag in the freezer at 0°C between treatments. Participants were provided with detailed study instructions (including daily study log completion), completed a post-study instruction knowledge assessment, and received an instructional DVD to take home. They were also given study materials such as a handheld contact infrared dermal thermometer, ThermoTrace (Diabetica Solutions, San Antonio, TX, USA) with which to measure skin temperatures each morning, before and after treatments, the gel or placebo cooling gel-cuff, a freezer thermometer to monitor freezer temperature, a clipboard with attached pen for use with the specially designed monthly (28 days) study logs (Figure 1) to record leg skin temperatures and feelings (such as tingling, pain, or numbness).
Figure 1.
Monthly study log.
Each participant that was self-monitoring was instructed to record each day the: morning leg skin temperatures of both legs; length of time the treatment was conducted; freezer temperature; and skin temperature of the treatment leg before and after the cooling treatment, including time of day. All leg skin temperatures were measured at the same location (5 cm above the malleolus) that was initially marked with a surgical skin marker at the baseline visit. The cooling regime varied monthly per the protocol: daily during month 1, twice a week during months 2 and 3, once a week during months 4, 5, and 6, and as needed during months 7–9. When the cooling regime became graduated (less than daily), participants were given the autonomy to choose which days of the week they performed the cooling intervention; however, they were asked to not perform cooling on 2 consecutive days.
The investigators maintained supportive telephone contact with the participants at 24, 48, and 72 hours after enrollment to reinforce study instructions, weekly during month 1, biweekly during months 2 and 3, and monthly during months 4–9 to assist with any questions or problems they may have experienced with adhering to the protocol and to solicit adverse events (VLU occurrence and/or reoccurrence). Participants also received monthly study visit reminder cards in the mail and were instructed to return the monthly study logs at each scheduled clinic study visit. At these visits, the returned logs were reviewed with the participant by study personnel, and new logs were given out to be completed by the next scheduled study visit. Study visits coincided with changes in the cooling regime at months 1, 3, and 6 at which time the participants received instruction on the regime change and also completed standardized quality of life, leg pain, and self-efficacy surveys. At month 9, an end-of-study satisfaction survey was completed.
Data collection and analysis
Data were collected, managed, and exported using Research Electronic Data Capture (REDCap, Nashville, TN, USA), and analyzed using Statistical Analysis Software (SAS) 9.4 (SAS Institute Inc., Cary, NC, USA). Data collected included: baseline demographic characteristics such as age, sex, race, ethnicity, marital status, education level, employment, type of job held, residence, and whether he or she had been a research volunteer in the past, study visit outcome data such as blood flow, pain, quality of life, skin temperature, the development of a VLU during the study, participant satisfaction, self-monitoring monthly logs, and satisfaction ratings.
Prior to analysis of the log data, a literature search was performed to develop a list of commonly reported patient errors in the use of study logs and diaries and to determine cut-off scores for accuracy.24–27 Three study team personnel independently reviewed each returned participant monthly study log and coded errors by typology and overall accuracy (<85%, ≥85%, and 100% accurate); ≥85% accuracy was considered adherent to the protocol and was defined as accurately recording 24 days (≥85%) or more on a returned monthly study log (28-day period). To be accurate, a participant had to document daily self-monitoring of both leg temperatures in the morning and before and after performing the 30-minute cooling intervention per the treatment regime. Any day that a participant failed to properly document any of these activities or perform the prescribed cooling treatment when required counted against the number of days of accuracy in that month. Suspected fabrication was defined as a returned monthly log with an observable pattern of columned data entry, or contained illogical data (eg, recorded before-treatment times occurring after recorded after-treatment time), or nonsensical data (eg, a high frequency of impossible recorded skin temperature values). Recorded skin temperatures outside the range of 16.0°C–37.0°C were coded as “suspicious” because these would be improbable temperatures, thus were of questionable validity. All temperatures recorded as 22.0°C (the default setting for the thermometer) were excluded. Inconsistencies in study personnel codes of errors and/or ratings of accuracy were remedied through joint review until a majority consensus was achieved. For the determination of the overall average accuracy, each month’s quality of <85%, 85%–99%, or 100% accuracy was assigned a value of 0, 1, or 2, respectively, then averaged and rounded to the nearest integer.
Data analyses
For Aim 1, we report the quality of participant self-monitoring by evaluating the frequency distribution of error types and the accuracy (<85% accurate, 85%–99% accurate, and 100% accurate) of returned monthly study logs. For Aim 2, to explore the potential for outcome bias in treatment effect caused by poor documentation and variable participant adherence, an ITT versus an mITT (≥85% accuracy with no suspected fabrication) comparative analysis scenario was conducted of the average monthly mean difference on returned logs for the temperature taken before and after the cooling treatment. It was anticipated that adherence would also be reflected through an observed negative difference between leg skin temperatures among the intervention group when compared with the placebo group. Under the ITT analysis, all participant data were analyzed as randomized, regardless of the quality of their self-reported adherence to the intervention. Under the mITT analysis, the 85% adherence cut-off value was defined as being accurate (performing and documenting daily morning monitoring of leg skin temperature and/or the cooling intervention) for ≥24 days out of 28 days on each returned monthly study log without suspicion of fabricated data. Missing data under both the ITT and mITT analyses were considered missing completely at random (MCAR), and outliers were excluded. For comparisons between treatment groups, t-tests and chi-square tests were used for continuous and categorical variables, respectively. All longitudinal models were analyzed using generalized linear models (GLMs). For multiple comparisons, P-values were adjusted using Bonferroni correction. For comparison of monthly logs between treatment groups, models were clustered by participant to control for dependence of repeated measurements.
Results
Data were analyzed for the first 100 participants (n=54 intervention; n=46 control) who completed the 9-month study. Table 1 shows no statistically significant differences were observed between the intervention and control groups for: mean age, 64.6 and 61.7 years (P=0.19); female, 57.4% and 65.2% (P=0.42); black, 72.2% and 73.9% (P=0.85); and residence in rural area, 38.9% and 37.0% (P=0.84), respectively.
Table 1.
Demographics
| Demographicsa | Intervention group n=54 | Control group n=46 | P-value |
|---|---|---|---|
| Age, years, mean (SD) | 64.6 (11.0) | 61.7 (10.8) | 0.19 |
| Sex | 0.42 | ||
| Male | 23 (42.6) | 16 (34.8) | |
| Female | 31 (57.4) | 30 (65.2) | |
| Race | 0.85 | ||
| White | 15 (27.8) | 12 (26.1) | |
| Black/African American | 39 (72.2) | 34 (73.9) | |
| Employment | 0.73 | ||
| Employed (full or part time) | 11 (20.4) | 10 (21.7) | |
| Not employed | 12 (22.2) | 13 (28.3) | |
| Retired | 31 (57.4) | 23 (50.0) | |
| Education | 0.51 | ||
| High school or less or missing | 21 (38.9) | 16 (34.8) | |
| High school graduate | 25 (46.3) | 19 (41.3) | |
| Some college or college graduate | 8 (14.8) | 11 (23.9) | |
| Marriage status | 0.12 | ||
| Not married | 28 (51.9) | 31 (67.4) | |
| Married | 26 (48.2) | 15 (32.6) | |
| Residence | 0.84 | ||
| Rural | 21 (38.9) | 17 (37.0) | |
| Urban | 33 (61.1) | 29 (63.0) |
Notes:
N (%) unless otherwise noted; there are no significant differences between groups, P>0.05. Due to rounding, percentages may not sum to 100%.
Abbreviation: SD, standard deviation.
Aim 1: self-monitoring log quality
Error typology
Figure 2 provides a study CONSORT flow diagram of participant study logs. Of the expected 900 study logs to be collected (1 log ×9 months ×100 participants =900 logs) over the 9-month study, 91.8% (826/900) logs were returned. Table 2 shows the main non-mutually exclusive error types associated with the returned participant logs. Main self-reporting errors included: 59.2% (489/826) contained white-outs, cross-off and/or overwrites, 34.9% (288/826) had entries omitted (missing data), 26.8% (221/826) contained extraneous data, 11.1% (92/826) were recorded on the wrong monthly log or weekly sheet, 8.6% (71/826) of logs were suspected to contain fabricated entries, and 8.2% (68/826) had illegible handwriting. The main error types associated with the documentation of self-monitoring and adherence to the protocol that affected accuracy included: 29.4% (243/826) not performing morning leg skin temperature self-monitoring, 28.7% (237/826) not performing the cooling intervention per the prescribed protocol regime, and 25.2% (208/826) not documenting performance of the cooling intervention for the prescribed length of time of 30–40 minutes. The distribution of error types was consistent across both study groups (intervention and control), P≥0.05.
Figure 2.
CONSORT flow diagram.
Abbreviation: CONSORT, consolidated standards of reporting trials.
Table 2.
Returned participant study logs error typology
| Error typea | Returned logs
|
|
|---|---|---|
| N=826
| ||
| n | % | |
| White-outs, cross-offs and/or overwrites | 489 | 59.2 |
| Entries omitted (temperatures and/or before or after treatment times) | 288 | 34.9 |
| Morning leg skin temperature monitoring not performed | 243 | 29.4 |
| Cooling intervention not performed per protocol | 237 | 28.7 |
| Extraneous data recorded | 221 | 26.8 |
| Recorded cooling intervention time not per protocol (<30 or >40 minutes) | 208 | 25.2 |
| Wrong monthly/weekly sheet used | 92 | 11.1 |
| Suspected fabrication of data (patterns of columned handwriting, etc) | 71 | 8.6 |
| Not all monthly/weekly sheets returned | 71 | 8.6 |
| Illegible hand writing | 68 | 8.2 |
| Questionable validity of data (severely extreme values, etc) | 63 | 7.6 |
| Use of dittos or arrows | 55 | 6.7 |
| Data recorded in wrong box | 38 | 4.6 |
| Freezer temperature not maintained per protocol at 0°C (±2°C) | 38 | 4.6 |
| Monthly log returned but completely unusable | 36 | 4.4 |
| Weekly sheets returned in wrong order | 29 | 3.5 |
| Multiple data entry in the same box | 17 | 2.1 |
| Monthly logs returned blank | 16 | 1.9 |
Note:
Error types are not mutually exclusive, therefore percentages do not sum to 100%.
Accuracy and adherence
Under ITT analysis, 38.4% (346/900) of all returned logs were <85% accurate, 25.0% (225/900) were 85%–99% accurate, and 36.6% (329/900) were 100% accurate. Mean overall participant adherence rates were: 22.0% at <85% accuracy, 53.0% at 85%–99% accuracy, and 25.0% at 100% accuracy. A total of 78.0% of participants were adherent to the protocol with ≥85% accuracy in returned self-reporting documentation. Table 3 reports demographic characteristics by average accuracy across the 9 months for each group. No significant differences were found between groups, P.0.05 (Table 4).
Table 3.
Average accuracy of participant monthly study logs by demographics
| Intervention group
|
Control group
|
All groups
|
||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| N | <85% | 85%–99% | 100% | N | <85% | 85%–99% | 100% | N | <85% | 85%–99% | 100% | |
| Total sample | 54 | 10 (18.5%) | 30 (55.6%) | 14 (25.9%) | 46 | 12 (26.1%) | 23 (50.0%) | 11 (23.9%) | 100 | 22 (22.0%) | 53 (53.0%) | 25 (25.0%) |
| Age, years, mean (SD) | 54 | 70.0 (9.3) | 65.0 (12.0) | 59.8 (8.3) | 46 | 61.8 (12.8) | 60.7 (10.4) | 63.7 (10.1) | 100 | 65.5 (11.9) | 63.2 (11.4) | 61.5 (9.1) |
| Sex | ||||||||||||
| Male | 23 | 3 (13.0%) | 16 (69.6%) | 4 (17.4%) | 16 | 5 (31.3%) | 6 (37.5%) | 5 (31.3%) | 39 | 8 (20.5%) | 22 (56.4%) | 9 (23.1%) |
| Female | 31 | 7 (22.6%) | 14 (45.2%) | 10 (32.3%) | 30 | 7 (23.3%) | 17 (56.7%) | 6 (20.0%) | 61 | 14 (23.0%) | 31 (50.8%) | 16 (26.2%) |
| Race | ||||||||||||
| Black/African American | 39 | 8 (20.5%) | 22 (56.4%) | 9 (23.1%) | 34 | 9 (26.5%) | 18 (52.9%) | 7 (20.6%) | 73 | 17 (23.3%) | 40 (54.8%) | 16 (21.9%) |
| White | 15 | 2 (13.3%) | 8 (53.3%) | 5 (33.3%) | 12 | 3 (25.0%) | 5 (41.7%) | 4 (33.3%) | 27 | 5 (18.3%) | 13 (48.1%) | 9 (33.3%) |
| Marital status | ||||||||||||
| Not married | 28 | 5 (17.9%) | 17 (60.7%) | 6 (21.4%) | 31 | 10 (32.3%) | 16 (51.6%) | 5 (16.1%) | 59 | 15 (25.4%) | 33 (55.9%) | 11 (18.6%) |
| Live with partner | 26 | 5 (19.2%) | 13 (50.0%) | 8 (30.8%) | 15 | 2 (13.3%) | 7 (46.7%) | 6 (40.0%) | 41 | 7 (17.1%) | 20 (48.8%) | 14 (34.1%) |
| Education | ||||||||||||
| Less than high school | 13 | 5 (38.5%) | 7 (53.9%) | 1 (7.7%) | 9 | 2 (22.2%) | 5 (55.6%) | 2 (22.2%) | 22 | 7 (31.8%) | 12 (54.5%) | 3 (13.6%) |
| High school graduate | 25 | 4 (16.0%) | 15 (60.0%) | 6 (24.0%) | 19 | 5 (26.3%) | 9 (47.4%) | 5 (26.3%) | 44 | 9 (20.5%) | 24 (54.5%) | 11 (25.0%) |
| More than high school | 16 | 1 (6.3%) | 8 (50.0%) | 7 (43.8%) | 18 | 5 (27.8%) | 9 (50.0%) | 4 (22.2%) | 34 | 6 (17.6%) | 17 (50.0%) | 11 (32.4%) |
| Employment | ||||||||||||
| Not employed | 43 | 9 (20.9%) | 23 (53.5%) | 11 (25.6%) | 36 | 11 (30.6%) | 16 (44.4%) | 9 (25.0%) | 79 | 20 (25.3%) | 39 (49.4%) | 20 (25.3%) |
| Employed | 11 | 1 (9.1%) | 7 (63.6%) | 3 (27.3%) | 10 | 1 (10.0%) | 7 (70.0%) | 2 (20.0%) | 21 | 2 (9.5%) | 14 (66.7%) | 5 (23.8%) |
| Type of job helda | ||||||||||||
| Professional | 21 | 2 (9.5%) | 11 (52.4%) | 8 (38.1%) | 21 | 5 (23.8%) | 12 (57.1%) | 4 (19.0%) | 42 | 7 (16.7%) | 23 (54.8%) | 12 (28.6%) |
| Technical | 13 | 3 (23.1%) | 9 (69.2%) | 1 (7.7%) | 14 | 4 (28.6%) | 6 (42.9%) | 4 (28.6%) | 27 | 7 (25.9%) | 15 (55.6%) | 5 (18.5%) |
| Manual | 16 | 3 (18.8%) | 8 (50.0%) | 5 (31.3%) | 8 | 2 (25.0%) | 4 (50.0%) | 2 (25.0%) | 24 | 5 (20.8%) | 12 (50.0%) | 7 (29.2%) |
| Other | 4 | 2 (50.0%) | 2 (50.0%) | 0 (0.0%) | 3 | 1 (33.3%) | 1 (33.3%) | 1 (33.3%) | 7 | 3 (42.9%) | 3 (42.9%) | 1 (14.3%) |
| Residence | ||||||||||||
| Rural | 21 | 1 (4.8%) | 12 (57.1%) | 8 (38.1%) | 17 | 4 (23.5%) | 10 (58.8%) | 3 (17.6%) | 38 | 5 (13.2%) | 22 (57.9%) | 11 (28.9%) |
| Urban | 33 | 9 (27.3%) | 18 (54.6%) | 6 (18.2%) | 29 | 8 (27.6%) | 13 (44.8%) | 8 (27.6%) | 62 | 17 (27.4%) | 31 (50.0%) | 14 (22.6%) |
| Volunteered beforeb | ||||||||||||
| Yes | 5 | 3 (60.0%) | 1 (20.0%) | 1 (20.0%) | 5 | 1 (20.0%) | 2 (40.0%) | 2 (40.0%) | 10 | 4 (40.0%) | 3 (10.0%) | 3 (30.0%) |
| No | 49 | 7 (14.3%) | 29 (59.2%) | 13 (26.5%) | 41 | 11 (26.8%) | 21 (51.2%) | 9 (22.0%) | 90 | 18 (20.0%) | 50 (55.6%) | 22 (24.4%) |
Notes:
Current or previous job held.
Volunteered for research study before. There are no significant differences between groups, P>0.05.
Abbreviation: SD, standard deviation.
Table 4.
Average accuracy of participant monthly study logs by end-of-study satisfaction survey
| Difficulty of procedures | Intervention group
|
Control group
|
||||||
|---|---|---|---|---|---|---|---|---|
| N | <85% | 85%–99% | 100% | N | <85% | 85%–99% | 100% | |
| Total sample | 54 | 10 (18.5%) | 30 (55.6%) | 14 (25.9%) | 46 | 12 (26.1%) | 23 (50.0%) | 11 (23.9%) |
| DVD instruction score | ||||||||
| <10 | 0 | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) | 0 | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| 10–15 | 7 | 0 (0.0%) | 6 (85.7%) | 1 (14.3%) | 9 | 3 (33.2%) | 3 (33.2%) | 3 (33.2%) |
| 16 (perfect score) | 45 | 10 (22.2%) | 23 (51.1%) | 12 (26.7%) | 35 | 8 (22.9%) | 19 (54.3%) | 8 (22.9%) |
| Treatment | ||||||||
| Easy/somewhat easy | 48 | 10 (20.8%) | 26 (54.2%) | 12 (25.0%) | 45 | 11 (24.4%) | 23 (51.1%) | 11 (24.4%) |
| Difficult/somewhat difficult | 5 | 0 (0.0%) | 3 (60.0%) | 2 (40.0%) | 1 | 1 (100.0%) | 0 (0.0%) | 0 (0.0%) |
| Leg-cuff use | ||||||||
| Easy/somewhat easy | 46 | 8 (17.4%) | 25 (54.4%) | 13 (28.3%) | 43 | 10 (23.3%) | 22 (51.2%) | 10 (34.5%) |
| Difficult/somewhat difficult | 7 | 2 (28.6%) | 4 (57.1%) | 1 (14.3%) | 3 | 2 (66.7%) | 1 (33.3%) | 0 (0.0%) |
| Comfortable to elevate | ||||||||
| Yes | 34 | 7 (19.4%) | 15 (41.7%) | 12 (33.3%) | 27 | 7 (24.1%) | 10 (34.5%) | 11 (25.6%) |
| No | 2 | 0 (0.0%) | 2 (5.6%) | 0 (0.0%) | 2 | 0 (0.0%) | 2 (6.9%) | 0 (0.0%) |
| Thermometer use | ||||||||
| Easy/somewhat easy | 52 | 10 (19.2%) | 28 (53.9%) | 14 (26.9%) | 46 | 12 (26.1%) | 23 (50.0%) | 11 (23.9%) |
| Difficult/somewhat difficult | 0 | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) | 0 | 0 (0.0%) | 0 (0.0%) | 0 (0.0%) |
| Study log use | ||||||||
| Easy/somewhat easy | 48 | 9 (18.8%) | 25 (52.1%) | 14 (29.2%) | 44 | 11 (25.0%) | 22 (50.0%) | 11 (25.0%) |
| Difficult/somewhat difficult | 4 | 1 (25.0%) | 3 (75.0%) | 0 (0.0%) | 2 | 1 (50.0%) | 1 (50.0%) | 0 (0.0%) |
Note: There are no significant differences between groups, P>0.05.
Aim 2: ITT versus mITT sensitivity analysis and outcome bias
Under mITT analysis, 54.0% (483/900) of logs were ≥85% accurate with no suspected fabrication. When the participants were observed for adherence to performing and documenting both self-monitoring and the cooling intervention per the prescribed protocol regime, no statistically significant differences were found between the groups (P>0.05 for each month). Overall, a mean average of 51.9% (28.0/54) of participants in the intervention group and 50.7% (23.3/46) in the control group was adherent at the ≥85% accuracy level with no suspected fabrication of data over the 6-month prescribed cooling intervention portion of the trial. The monthly mean level of participant adherence at ≥85% accuracy with no suspected fabrication of data for both groups showed a slight positive trend as the frequency of the treatment regime decreased and participant autonomy in the selection of the days on which to perform the cooling intervention increased toward the last months of the trial: everyday (45.8% month 1), twice a week (51.0% month 2 and 51.0% in month 3), once a week (56.2% month 4; 56.1% month 5; and 47.9% in month 6). Mean monthly leg skin temperature differences before and after the cooling treatment under ITT and mITT analysis are shown in Figure 3 within the control group (placebo cuff) and in Figure 4 within the intervention (cooling gel-cuff) group, respectively. Mean treatment group differences between leg skin temperatures measured before and after the cooling treatment under ITT and mITT analyses show that the participants in the intervention group consistently recorded lower after-treatment leg skin temperatures than the before-treatment values, whereas those in the control group showed almost no change in temperature from the before-treatment measurement. The use of mITT when compared with ITT analysis of returned participant logs resulted in lower mean differences in monthly temperatures (higher cooling treatment effect) for the intervention group and more positive readings for the control group (lower cooling treatment effect), respectively. However, no significant differences were found between these two approaches to suggest an outcome bias in treatment effect within either group by month or over time (P=1.00).
Figure 3.
Mean monthly leg skin temperature difference before and after cooling treatment within the control group (placebo cuff) under ITT and mITT analysis.
Note: No statistical difference found between mITT and ITT analyses (P-value 1.0).
Abbreviations: ITT, intention-to-treat; mITT, modified intention-to-treat.
Figure 4.
Mean monthly leg skin temperature difference before and after cooling treatment within the intervention group (cooling gel-cuff) under ITT and mITT analysis.
Note: No statistical difference found between mITT and ITT analyses (P-value 1.0).
Abbreviations: ITT, intention-to-treat; mITT, modified intention-to-treat.
Discussion
This study assessed the quality and accuracy of participant documentation of daily self-monitoring of leg skin temperature, adherence to a graduated cooling treatment protocol to prevent VLUs, and the potential for bias in treatment effect in a RCT that included participants with CVeD. Documentation on paper logs was selected as the method to assess adherence, because an electronic tool for capturing data over multiple time points was unavailable. The thermometer used to measure skin was not equipped with a memory card that would have enabled it to record, date and time stamp three temperature readings per day. Due to these limitations, paper and pencil self-report logs were provided to participants.
In Aim 1, the quality and accuracy of the logs were found to be acceptable. The majority of participants filled out the logs as directed. Adherence to the treatment was also found to be acceptable; differences in before and after the experimental cooling treatment showed a sustained decrease in skin temperature over 6 months of active cooling compared with placebo. There was little variability in monthly adherence rates in both groups. These outcomes were unexpectedly higher than anticipated, considering the high burden of treatment and self-report requirements, including recording three different skin temperature readings and the time for each, the temperature of the adjacent leg once each day, the freezer temperature where the cuff was stored, and comments.
These findings are in contrast to published studies of self-reports of other skin related research. Patient-applied topical regimens to treat or ameliorate skin disorders show varying adherence rates based on self-report measures and logs. Shergill et al27 found that 88% of participants applying a topical therapy for actinic keratosis over 12 weeks were either non-adherent, non-persistent or both. Duration of treatment was considered to be a major factor contributing to non-adherence. However, Yentzer et al28 showed better adherence to a 4-week topical therapy for atopic dermatitis in which 50% of participants were reported to be adherent to the treatment protocol. Similarly, Hix et al29 reported almost 40% of patients were adherent to a 5-day topical treatment course for atopic dermatitis. These latter two studies suggest that even with short duration of therapy, adherence rates were somewhat low. The fact that participants in our study were actively followed via clinic visits and received phone calls may have prompted greater adherence.
Participants in this study were on average, 60 years of age, and found the study protocol easy to perform. Perfect or nearly perfect scores on a post-test administered after viewing an instructional video suggested that participants understood the study procedures. The majority in both groups found the treatment to be easy, and the leg cuffs, thermometers, and logs easy to use. All of the procedures were pilot tested with participant input for refinement prior to study implementation, which we believed positively influenced adherence.
While the logs in this study were revised during early feasibility testing to make the font larger and columns wider, several documentation mistakes made it difficult to translate hand-written entries, such as illegible and overwrites. Errors of omission included missing times, dates, temperatures on one or both legs, the freezer temperature, and failure to return all or part of the monthly log sheets. Other errors include the use of extraneous data recordings, ditto marks, arrows, multiple data entries in the same box, data recorded in the wrong box, use of the wrong weekly or monthly log sheets, and failure to keep the freezer set at 0°C. Some participants made unsolicited notations in margins. Entries that directly affected the quality of data were not included in the analysis.
In Aim 2, we conducted a comparative ITT versus an mITT (85% accuracy with no suspected fabrication cutoff) sensitivity analysis where missing data were assumed to be MCAR and outliers were excluded. The analysis revealed no significant differences (P=1.0) between these two approaches in both the before and after treatment mean leg skin temperatures for both the control (placebo cuff) and intervention group (cooling gel-cuff) suggesting no outcome bias in the observed treatment effect or the underlying assumptions made about handling of missing data as MCAR, the removal of outliers, and of data suspected of fabrication. Additionally, both analyses were also able to demonstrate that participant self-reported adherence to the cooling intervention on the returned study logs was observable between the groups; participants in the intervention group, as expected, showing consistently lower mean leg skin temperatures after performing the cooling treatment than before with the gel-cuff when compared with those in the control group with the placebo cuff. Within both the control and intervention groups, the ITT analysis provided a more conservative estimate of the cooling treatment effect compared with the mITT approach. Under the control group analysis (Figure 3), the mean before and after leg skin temperature difference was consistently higher each month under mITT, with an overall study monthly mean of 0.18°C for mITT and −0.23°C for ITT analysis, respectively. Under the intervention group analysis (Figure 4), the mean before and after leg skin temperature difference was consistently lower each month under mITT than ITT analysis, with an overall study mean monthly temperature difference of −3.05°C for mITT and −2.15°C for ITT analysis, respectively.
Although our sensitivity analysis results demonstrated consistency between the ITT and mITT analysis approaches, they are limited by a number of factors. First, we did not consider missing data in the analyses, but treated them all as MCAR. Several imputation methods could have been applied.13 Second, we used only one comparative efficacy subset scenario rather than used multiple scenarios.30 Finally, the use of the 85% accuracy cut-off exclusion for comparative mITT analysis has not been validated, and selection bias in this efficacy subset could bias the ITT comparison.31 Despite these limitations, the demonstrated consistency in findings between these two analysis approaches to this study dataset provides the researchers with confidence that participant adherence to the treatment intervention is observable within and between the two study groups from the quantity, quality, and accuracy of returned study logs.
Limitations
A limitation of this study was the use of only one self-report measure. One or more secondary approaches such as an adherence questionnaire or electronic device such as a thermometer with memory would have been helpful to verify findings. In some chronically ill populations, patient-reported adherence to self-monitoring has been reported to be higher than or at least similar to objective measures of adherence such as sleep apnea and cardiovascular conditions32–34 and lower in others such as adherence to HIV prophylaxis.35 Self-report data from diaries, log, and journals have been successfully used to capture information on the use of medications, and a number of therapies from which rates of adherence have been calculated.36 We believe our findings provide insights into real-world adherence to a long-term VLU prevention treatment strategy among a CVeD population.
Conclusion
This study found good rates of participant adherence. Under ITT analysis, 78.0% of participants were deemed adherent to the study protocol with ≥85% accuracy in recorded self-monitoring of skin temperatures in response to a topical skin cooling intervention for the prevention of VLUs. Successful self-management of chronic conditions is contingent upon adherence to a regimen of self-monitoring. Self-report via “pencil and paper” logs continues to be an acceptable approach for documenting adherence. These study findings suggest daily documentation on specially designed logs yielded sufficient evidence from which to draw conclusions about adherence as noted by a reduction in skin temperature in the treatment group. Methods to enhance adherence to self-monitoring regimens for the self-management of chronic conditions continue to be heavily investigated, however best practice models remain elusive. The consequences of non-adherence, especially in research trials, are substantial resulting in wasted resources, poor research outcomes, reduced scientific integrity, and lack of generalizability of findings. Self-report documentation methods to enhance adherence that best fit the type of therapy should consider the research question, study design, and patient population.36
Acknowledgments
This work was funded by the US National Institutes of Health (NIH) National Institute of Nursing Research (NINR) Award no. R01NR012237, and supported by the South Carolina Clinical & Translational Research (SCTR) Institute, with an academic home at the Medical University of South Carolina, through NIH Grant Number UL1 TR000062. The use of REDCap was supported by NIH/NCATS UL1TR000062. The ideas and opinions expressed herein are those of the authors and not necessarily reflective of the NIH/NINR.
Footnotes
Disclosure
The authors report no conflicts of interest in this work.
References
- 1.Anglada-Martinez H, Riu-Viladoms G, Martin-Conde M, Rovira-Illamola M, Sotoca-Momblona JM, Codina-Jane C. Does mHealth increase adherence to medication? Results of a systematic review. Int J Clin Pract. 2015;69(1):9–32. doi: 10.1111/ijcp.12582. [DOI] [PubMed] [Google Scholar]
- 2.Sabate E. Adherence to Long-Term Therapies: Evidence for Action. Geneva: World Health Organization; 2003. [Google Scholar]
- 3.Hingorani R, Mahmood M, Alweis R. Improving antibiotic adherence in treatment of acute upper respiratory infections: a quality improvement process. J Community Hosp Intern Med Perspect. 2015;5(3):27472. doi: 10.3402/jchimp.v5.27472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Roebuck MC, Liberman JN, Gemmill-Toyama M, Brennan TA. Medication adherence leads to lower health care use and costs despite increased drug spending. Health Aff (Millwood) 2011;30(1):91–99. doi: 10.1377/hlthaff.2009.1087. [DOI] [PubMed] [Google Scholar]
- 5.Daniels T, Goodacre L, Sutton C, Pollard K, Conway S, Peckham D. Accurate assessment of adherence: self-report and clinician report vs electronic monitoring of nebulizers. Chest. 2011;140(2):425–432. doi: 10.1378/chest.09-3074. [DOI] [PubMed] [Google Scholar]
- 6.van Dulmen S, Sluijs E, van Dijk L, de Ridder D, Heerdink R, Bensing J. Patient adherence to medical treatment: a review of reviews. BMC Health Serv Res. 2007;7:55. doi: 10.1186/1472-6963-7-55. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Patel M, Perrin K, Pritchard A, et al. Accuracy of patient self-report as a measure of inhaled asthma medication use. Respirology. 2013;18(3):546–552. doi: 10.1111/resp.12059. [DOI] [PubMed] [Google Scholar]
- 8.Yoong J, Traeger LN, Gallagher ER, Pirl WF, Greer JA, Temel JS. A pilot study to investigate adherence to long-acting opioids among patients with advanced lung cancer. J Palliat Med. 2013;16(4):391–396. doi: 10.1089/jpm.2012.0400. [DOI] [PubMed] [Google Scholar]
- 9.Bell ML, Fairclough DL. Practical and statistical issues in missing data for longitudinal patient-reported outcomes. Stat Methods Med Res. 2014;23(5):440–459. doi: 10.1177/0962280213476378. [DOI] [PubMed] [Google Scholar]
- 10.Burzykowski T, Carpenter J, Coens C, et al. Missing data: discussion points from the PSI missing data expert group. Pharm Stat. 2010;9(4):288–297. doi: 10.1002/pst.391. [DOI] [PubMed] [Google Scholar]
- 11.Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. doi: 10.1136/bmj.d5928. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Altman DG, Bland JM. Missing data. BMJ. 2007;334(7590):424. doi: 10.1136/bmj.38977.682025.2C. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Sterne JA, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393. doi: 10.1136/bmj.b2393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.White IR, Horton NJ, Carpenter J, Pocock SJ. Strategy for intention to treat analysis in randomised trials with missing outcome data. BMJ. 2011;342:d40. doi: 10.1136/bmj.d40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Vanderweele TJ, Arah OA. Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and con-founders. Epidemiology. 2011;22(1):42–52. doi: 10.1097/EDE.0b013e3181f74493. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Houghton VJ, Bower VM, Chant DC. Is an increase in skin temperature predictive of neuropathic foot ulceration in people with diabetes? A systematic review and meta-analysis. J Foot Ankle Res. 2013;6(1):31. doi: 10.1186/1757-1146-6-31. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Sibbald RG, Mufti A, Armstrong DG. Infrared skin thermometry: an underutilized cost-effective tool for routine wound care practice and patient high-risk diabetic foot self-monitoring. Adv Skin Wound Care. 2015;28(1):37–44. doi: 10.1097/01.ASW.0000458991.58947.6b. quiz 45–36. [DOI] [PubMed] [Google Scholar]
- 18.Sousa P, Felizardo V, Oliveira D, Couto R, Garcia NM. A review of thermal methods and technologies for diabetic foot assessment. Expert Rev Med Devices. 2015;12(4):439–448. doi: 10.1586/17434440.2015.1032251. [DOI] [PubMed] [Google Scholar]
- 19.Bus SA, van Netten JJ, Lavery LA, et al. IWGDF guidance on the prevention of foot ulcers in at-risk patients with diabetes. Diabetes Metab Res Rev. 2015 doi: 10.1002/dmrr.2696. [DOI] [PubMed] [Google Scholar]
- 20.Crawford PE, Fields-Varnado M, Society W. Guideline for the management of wounds in patients with lower-extremity neuropathic disease: an executive summary. J Wound Ostomy Continence Nurs. 2013;40(1):34–45. doi: 10.1097/WON.0b013e3182750161. [DOI] [PubMed] [Google Scholar]
- 21.Kelechi TJ, Johnson JJ, Yates S. Chronic venous disease and venous leg ulcers: an evidence-based update. J Vasc Nurs. 2015;33(2):36–46. doi: 10.1016/j.jvn.2015.01.003. [DOI] [PubMed] [Google Scholar]
- 22.Kelechi TJ, McNeil RB. A prospective, descriptive study of hour-to-hour and day-to-day temperature variability of skin affected by chronic venous disorders. Ostomy/Wound Manage. 2008;54(4):18–34. [PubMed] [Google Scholar]
- 23.Sayre EK, Kelechi TJ, Neal D. Sudden increase in skin temperature predicts venous ulcers: a case study. J Vasc Nurs. 2007;25(3):46–50. doi: 10.1016/j.jvn.2007.06.002. [DOI] [PubMed] [Google Scholar]
- 24.Carroll CL, Feldman SR, Camacho FT, Manuel JC, Balkrishnan R. Adherence to topical therapy decreases during the course of an 8-week psoriasis clinical trial: commonly used methods of measuring adherence to topical therapy overestimate actual use. J Am Acad Dermatol. 2004;51(2):212–216. doi: 10.1016/j.jaad.2004.01.052. [DOI] [PubMed] [Google Scholar]
- 25.Menon C, Gipson K, Bowe WP, Hoffstad OJ, Margolis DJ. Validity of subject self-report for acne. Dermatology. 2008;217(2):164–168. doi: 10.1159/000136655. [DOI] [PubMed] [Google Scholar]
- 26.Almeida H, Jr, Cecconi J, Duquia RP, Souza PR, Breunig J. Sensitivity and specificity of self-reported acne in 18-year-old adolescent males. Int J Dermatol. 2013;52(8):946–948. doi: 10.1111/j.1365-4632.2011.05443.x. [DOI] [PubMed] [Google Scholar]
- 27.Shergill B, Zokaie S, Carr AJ. Non-adherence to topical treatments for actinic keratosis. Patient Prefer Adherence. 2013;8:35–41. doi: 10.2147/PPA.S47126. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Yentzer BA, Camacho FT, Young T, Fountain JM, Clark AR, Feldman SR. Good adherence and early efficacy using desonide hydrogel for atopic dermatitis: results from a program addressing patient compliance. J Drugs Dermatol. 2010;9(4):324–329. [PubMed] [Google Scholar]
- 29.Hix E, Gustafson CJ, O’Neill JL, et al. Adherence to a five day treatment course of topical fluocinonide 0.1% cream in atopic dermatitis. Dermatol Online J. 2013;19(10):20029. [PubMed] [Google Scholar]
- 30.Ye C, Beyene J, Browne G, Thabane L. Estimating treatment effects in randomised controlled trials with non-compliance: a simulation study. BMJ Open. 2014;4(6):e005362. doi: 10.1136/bmjopen-2014-005362. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Nuesch E, Trelle S, Reichenbach S, et al. The effects of excluding patients from the analysis in randomised controlled trials: meta-epidemiological study. BMJ. 2009;339:b3244. doi: 10.1136/bmj.b3244. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Hood MM, Corsica J, Cvengros J, Wyatt J. Impact of a brief dietary self-monitoring intervention on weight change and CPAP adherence in patients with obstructive sleep apnea. J Psychosom Res. 2013;74(2):170–174. doi: 10.1016/j.jpsychores.2012.12.006. [DOI] [PubMed] [Google Scholar]
- 33.Jones CD, Holmes GM, DeWalt DA, et al. Self-reported recall and daily diary-recorded measures of weight monitoring adherence: associations with heart failure-related hospitalization. BMC Cardiovasc Disord. 2014;14:12. doi: 10.1186/1471-2261-14-12. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Englert H, Muller-Nordhorn J, Seewald S, et al. Is patient self-report an adequate tool for monitoring cardiovascular conditions in patients with hypercholesterolemia? J Public Health (Oxford) 2010;32(3):387–394. doi: 10.1093/pubmed/fdq013. [DOI] [PubMed] [Google Scholar]
- 35.Agot K, Taylor D, Corneli AL, et al. Accuracy of self-report and pill-count measures of adherence in the FEM-PrEP clinical trial: implications for future HIV-prevention trials. AIDS Behav. 2015;19(5):743–751. doi: 10.1007/s10461-014-0859-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Clifford S, Perez-Nieves M, Skalicky AM, Reaney M, Coyne KS. A systematic literature review of methodologies used to assess medication adherence in patients with diabetes. Curr Med Res Opin. 2014;30(6):1071–1085. doi: 10.1185/03007995.2014.884491. [DOI] [PubMed] [Google Scholar]