Development and Validation of the Kidney Transplant Understanding Tool (K-TUT)

Abstract

Background

Several educational interventions have been designed to improve patient knowledge before and after kidney transplantation. However, evaluation of such interventions has been difficult because validated instruments to measure knowledge-based outcomes in this population have not been developed.

Objective

To create a tool to measure patient knowledge of kidney transplantation and to evaluate its validity.

Methods

The Kidney Transplant Understanding Tool (K-TUT) was created using a stepwise iterative process. Experts in the field and transplant recipients were consulted to establish content validity. The K-TUT consists of 9 true/false and 13 multiple-choice questions, and scores are based on the number correct answers [YES/NO format] of 69 items. The questionnaire was piloted in a study that also measured health literacy (via the Short Test of Functional Health Literacy) in transplant candidates, whereas the main survey was mailed to transplant recipients. Test-retest was performed, and completed surveys were analyzed for internal consistency, construct validity, floor and ceiling effects, and reproducibility.

Results

Surveys were offered to 106 pretransplant patients and 235 in the posttransplant period, and response rates were 38.7% (41/106) and 63.4% (149/235), respectively. The mean corrected scores were 53.1 ± 8.5 (77%) and 56.2 ± 6.3 (81%), respectively. Test-retest was performed over 20% of both cohorts and percent agreement ranged between 70% and 100% in the pretransplant group and 66% and 100% in the posttransplant group. Cronbach α ranged from 0.794 to 0.875 in all cohorts indicating favorable internal consistency. Increased health literacy in the pretransplant group was significantly associated with increased knowledge (r = 0.52; P < 0.001), suggestive of construct validity, and the absence of floor and ceiling effects was positive. The majority of transplant recipients (98/148, 67%) believed the questionnaire adequately assessed transplant knowledge, about a quarter (36/148, 24.3%) were “unsure,” and 85% (126/148) agreed that no questions should be removed.

Conclusions

Although more study is warranted to further assess psychometric properties, the K-TUT appears to be a promising tool to measure transplant knowledge.

Nearly 80 000 renal transplants are performed worldwide annually, and this number is on the rise.¹ For patients with end-stage renal disease, transplantation offers the opportunity to discontinue dialysis, improve quality of life, and prolong survival.^2-4 Although transplantation is the treatment of choice for most patients with end-stage renal disease, the complexities involved with this procedure should not be understated. Several steps are involved in the transplant process, including the workup process to determine candidate suitability, placement on the waitlist, and the surgical procedure itself. Patient knowledge is important at every interface of care. For example, lack of information regarding kidney transplantation is associated with decreased access to transplantation,⁵ whereas improving patient knowledge may influence the likelihood of being transplanted.⁶

According to the program logic model described by Osborne and colleagues⁷ in chronic conditions, knowledge is a proximal health care outcome, which may impact distal outcomes, such as decreased morbidity and mortality, and decreased health care expenditures. This is particularly relevant in the posttransplant setting, where management is complex and organ recipients have much to learn. In addition to understanding the proper timing, administration, and ongoing titration of immunosuppressive medications, transplant recipients must know how to adhere to lifestyle changes, hygiene practices, and infection prevention, must understand how to monitor for signs and symptoms of rejection and more. Inadequate patient knowledge could lead to dire consequences, such as decreased patient adherence and organ rejection.

Patient education can be described as the process of enabling individuals to make informed decisions about their personal health-related behavior.⁸ It aims to encourage adherence with medical treatment regimens and promote healthy lifestyles, thereby improving health.⁸ Several strategies have been suggested to improve patient education for both transplant recipients and candidates on the waitlist. These include cognitive strategies, such as teaching videos, electronic education, group information sessions, or one-on-one counseling.^9-12 Unfortunately, the lack of available and validated instruments to measure outcomes of education-based interventions has hindered the ability to evaluate these strategies.¹³ Robust outcome measures for assessing patient knowledge are necessary to determine the effectiveness of educational interventions, as well as evaluate the potential predictive value of patient knowledge on adherence and self-efficacy.

A systematic review regarding educational interventions in kidney transplantation identified knowledge as the most frequently measured endpoint.¹³ In general, the instruments were vaguely described and reports of content validity were present in only 2 of the 6 studies. Available studies suggest longer duration of kidney disease is significantly associated with knowledge, whereas longer time on dialysis and complications negatively impact scores.¹⁴ A more recent study undertaken by Ahsanuddin and colleagues¹⁵ indicated that educational level and older age are independent risk factors for poor comprehension. However, both of these studies were conducted using tools that had not undergone formal psychometric testing. Further development of these types of tools is critical to advancing progress in this important research area.

Validation is the process of confirming that an assessment tool produces accurate, reliable, and generalizable results. In addition to content validity, it also includes the assessment of internal consistency, criterion validity, construct validity, reproducibility, responsiveness, floor and ceiling effects, and interpretability.¹⁶

This study sought to develop and validate an instrument to assess patient knowledge of kidney transplantation. We identified 2 purposes for the tool a priori: (1) to be used as a tool for targeting education deficiencies (by identify gaps in transplant information, and highlighting areas of patient misunderstanding) and (2) to measure knowledge as an outcome in educational research. We envisioned a tool that could be used both in patients waiting for a transplant, as well as those who have received a kidney, because disease state knowledge is important in all stages of the transplant process.

MATERIALS AND METHODS

The study occurred over 2 phases as outlined in Figure 1. Phase 1 included the steps involved in questionnaire development, whereas phase 2 involved the assessment of psychometric properties of the tool in 2 separate cohorts: adult candidates on the kidney transplant waitlist (pretransplant) and posttransplant recipients. Approval to carry out the study was granted by the regional ethics board (ethics protocol BEH-14-475, and BEH-15-09).

FIGURE 1.

Process for development and validation of the K-TUT.

Phase 1: Tool Development

To establish content validity, instrument development occurred in a multistep process.¹⁷ The first step involved the determination of the concepts of interest and the formulation of the questions and was undertaken by 2 pharmacists who work in transplantation, a pharmacy summer student, and a research expert in questionnaires development and assessment. An extensive review of the literature was undertaken to identify previously published knowledge questionnaires as well as articles reporting on required knowledge for transplant recipients. Because the literature was sparse in the latter area, a focus group meeting was conducted to investigate this topic from the views of transplant recipients.¹⁸

Next, the draft questionnaire was reviewed for accuracy, appropriateness, and completeness by 39 members of the Saskatchewan Transplant Program and Saskatoon Health Region, including 3 transplant nephrologists, 3 transplant surgeons, 16 nurses, and 3 pharmacists with extensive knowledge of transplant. The others were members of the Saskatchewan Transplant Program with knowledge of transplantation, but they were managers, administrative, and support staff rather than health care providers. Modifications and additions to the tool were made based on this feedback. The readability of the instrument was analyzed using the Flesh-Kincaid formula¹⁹ and the SMOG grading formula.²⁰ Because the first version required a reading level approximately equivalent to that of a ninth grade student, the questions were iteratively reworded until an acceptable reading level of grade 6.1 was achieved.²¹ Finally, the tool was piloted on 10 transplant recipients. Minor edits were performed as a result of this patient feedback.

Phase 2: Testing of the Kidney Transplant Understanding Tool

The Kidney Transplant Understanding Tool (K-TUT) was tested in 2 separate cohorts of patients aged 18 years and older: (i) a pretransplant cohort and (ii) a posttransplant cohort. The pretransplant cohort consisted of patients on the kidney transplant waitlist who received the K-TUT as part of a study aimed to characterize health literacy, beliefs of medicine, knowledge of kidney transplantation, and satisfaction with current methods of education.²² Briefly, all patients on the kidney transplant waitlist between April 1, 2015, and September 30, 2015, were eligible to participate and a research assistant administered a 4-part questionnaire consisting of the K-TUT, a questionnaire to determine satisfaction with current education, the Beliefs about Medicines Questionnaire (BMQ),²³ the Short Test of Functional Health Literacy,²⁴ and the numeracy section of the original Test of Functional Health Literacy.²⁵ The research assistant remained present throughout the entire assessment, but did not assist the participant with completing the questionnaire. Demographic information included age, sex, race, education, occupation, income, residence, marital status, whether English was the primary language, presence of a support person and length of time waiting for a transplant. A Tim Horton's Canadian dollars (CAN) $15 coffee card was provided as incentive for participation. Test-retest was performed in a subgroup of 10 participants who remained on dialysis 6 months after the initial study. This convenience sample was chosen by reviewing the list of participants that received their dialysis at St. Paul's Hospital, and performing the test in the first 10 patients that agreed to participate. The K-TUT was readministered to the subgroup and then repeated again approximately 1 week later under the supervision of a research assistant during a dialysis session.

The posttransplant cohort consisted of kidney transplant recipients attending clinic appointments in the Saskatoon location of the Saskatchewan Transplant Program. In this cohort, the K-TUT was completed at home rather than at the clinic. The questionnaire was distributed by mail using the Dillman method.²⁶ An advance letter preceded the cover letter and questionnaire by approximately 1 week. A reminder letter was mailed to nonresponders approximately 1 month later. A CAN $5 gift card was offered as an incentive for participation for respondents that chose to share their contact information. To test the reliability, a repeat questionnaire was mailed to the first 50 participants who returned the survey with contact information (within 3 weeks after the initial mailout). Demographic information collected during this phase included sex, race, education, residence, length of time since transplant, whether or not English was their primary language. Questions were also added at the end of the questionnaire to assess whether or not they perceived the questionnaire to be an accurate assessment of transplant knowledge and if any questions should be added or removed.

Statistical Analysis

Data analysis was carried out with SPSS version 23 (IBM Corp., Armonk, NY). The raw data from responses of each participant were coded numerically. (Correct and incorrect answers were coded “1” and “0,” respectively). Results were described as percentages and means ± SD. The K-TUT scores were calculated using intent to treat analysis: all participants who returned the questionnaire and all questions were included in the results. A sensitivity analysis was undertaken to determine whether the results significantly changed by excluding participants with missing data. The percentage of participants that achieved the highest and lowest possible score was calculated to assess for floor and ceiling effects.

Differences between demographic variables (such as sex, race, English as a first language, occupation, education level, marital status, income, presence of a support person, and residence) and the K-TUT scores were calculated using t test or 1-way analysis of variance (for more than 2 groups). In the pretransplant cohort, relationships between the K-TUT and health literacy, numeracy literacy, and BMQ scores were also examined using Pearson correlation.²² Because a high proportion of participants displayed adequate health literacy and numeracy, these variables were treated as continuous, rather than using categorical variables of “adequate,” “marginal,” and “inadequate literacy.” Cronbach α was calculated to determine the internal consistency. Acceptable α values range between 0.70 and 0.95, with higher scores indicating increased interrelatedness.^27,28 Interclass correlation coefficient (ICC) (1-way random effects) was used to assess the stability of the mean scores of the K-TUT re-administered on 2 separate occasions. ICC values greater than 0.75 were interpreted as excellent reproducibility.²⁹ Cohen κ was used measure test-retest of each statement, because these variables were dichotomous in nature.³⁰ These values were defined as the following: 0.01 to 0.20 as none to slight agreement, 0.21 to 0.40 as fair, 0.41 to 0.60 as moderate, 0.61 to 0.80 as substantial, and 0.81 to 1.00 as almost perfect agreement.³¹ Percent agreement was also calculated to assess reliability, because the stability of κ can be compromised when there is a low prevalence of one response.³² Percent agreement of at least 66% was classified as fair.³³

RESULTS

A transplant recipient focus group was held during the tool development phase of the project. This group generated many ideas on important aspects of knowledge in all areas of the transplant process, including more specific ideas about the waitlist and surgery.¹⁸ To ensure that the questionnaire would be relevant for both transplant candidates and recipients, however, the final questionnaire consisted of general questions pertaining to lifestyle, medication and rejection, and transplant terminology, all which were perceived to be necessary for optimal transplant outcomes. The final version of the K-TUT consisted of 9 true and false, and 13 multiple-choice questions (each with more than 1 potential answer). The true and false items at the beginning of the questionnaire were included based on patient feedback, as a way for participants to “get warmed up,” rather than starting with the longer multiple choice questions that take longer for patients to process. Each statement was dichotomized (“correct” or “incorrect”). A point was given for each correct answer chosen, and the scores were summed and converted to percentages. A perfect score (100%) equated to a total of 69 points.

Missing Data

Almost all patients in the pretransplant cohort provided responses to at least 90% of the questionnaire. In the posttransplant cohort, however, approximately 76% and 80% completed at least 90% of the questionnaire in the initial study and test-retest, respectively. The questions that were most commonly left blank were the ones pertaining to pregnancy, sexual health, or the first few months after kidney transplant (questions 20a-e, 22a-d, and 18a-d, respectively) (Table 1). The sensitivity analysis revealed that mean scores in both groups did not significantly change when the participants with missing data were excluded from the analysis.

TABLE 1.

Psychometric properties of the K-TUT in both cohorts

Pretransplant Cohort

Of the 106 eligible patients on the kidney transplant waitlist who were invited to participate, 41 completed the study providing a response rate of 38.9%. The score on the K-TUT was 53.1 ± 8.5/69, which converted to 77%. Using Pearson bivariate correlation a positive association was noted between K-TUT score and health literacy (r = 0.52, P < 0.05). No other significant associations were noted between K-TUT scores and demographics.

Cronbach α of the items in the questionnaire ranged from 0.79 to 0.88. ICC between mean score of test and re-test was 0.937 (95% confidence interval [CI], 0.763-0.985). According to κ scores for the individual items, over three quarters (54/69) had substantial to perfect agreement, 17% (12/69) had fair to moderate agreement, and 4% (3/69) had no agreement. Percent overall agreement ranged from 70% to 100% for each item (Table 2).

TABLE 2.

Reliability of the K-TUT at item level

Posttransplant Cohort

The K-TUT was mailed to 235 transplant recipients, and 148 returned the questionnaire, equating to a response rate of 63%. The mean score of 56.2 ± 6.3 (81%) was slightly higher than that of the pretransplant group (77%), but this was not statistically significant. Thirty-two of the 50 transplant recipients that received the K-TUT a second time returned the re-test questionnaire (response rate = 66%). Percent overall agreement ranged from 60-100%, while κ values ranged from Cronbach α was 0.83 and 0.87 in the test and retest groups, respectively. The ICC for the mean score was 0.762 (95% CI, 0.566-0.877), whereas κ values for each item indicated that 50% (34/69) had substantial to perfect agreement, 33% (23/69) had fair to moderate agreement, and 17% (12/69) lacked agreement (Table 2).

The majority of transplant recipients (98/148 = 67%) believed the questionnaire adequately assessed transplant knowledge, 12 respondents (8%) did not, and about a quarter (36/148 = 24.3%) were not sure. Eighty-five percent (126/148) felt that no questions should be removed, whereas 10 respondents did not provide a response to the question. Mean corrected knowledge score was significantly associated with female sex (P = 0.042), white race (P = 0.01), English as a first language (P = 0.015), higher levels of education (P = 0.02), but not time since transplant, or perceived impression of whether the K-TUT accurately assesses knowledge or questions should be added or removed.

Summaries of the patient demographics and testing undertaken to establish validity are presented in Tables 3 and 4, respectively.

TABLE 3.

Respondent demographics

TABLE 4.

Summary of validity testing undertaken on the K-TUT

DISCUSSION

The K-TUT was created to assess an individual's understanding and knowledge of kidney transplantation. A tool that could be used in patients on the waitlist or in patients who have received a kidney transplant would be extremely valuable as it could serve many purposes. The ability to knowledge across the continuum of transplant care would allow health care providers to identify whether educational interventions performed before transplant could contribute to knowledge sustainability after transplant. Hence, we set out to validate the tool in both populations.

Content validity is a measure of the instrument's comprehensiveness and whether the items measure what they are supposed to.¹⁷ This type of validity was assessed and determined by a panel of experts and kidney transplant recipients. Content validity was reaffirmed in the posttransplant cohort, because only 8% of participants believed that the K-TUT did not adequately assess knowledge, and the majority (85%) felt that no questions should be removed.

The reliability of the questionnaire was tested by Cronbach α, which was greater than 0.7 in all cohorts. This indicates that all items within the instrument measure the same construct (which in this case, was transplant knowledge) and attests to the internal consistency of the K-TUT. Floor or ceiling effects can reduce the reliability of the instrument, because participants with the lowest or highest scores cannot be distinguished from one another.¹⁶ Fortunately, no ceiling or floor effects were noted within either cohort.

The reproducibility of the tool was examined by administering the test to the same group of patients on 2 consecutive occasions and comparing the change in scores over time. The mean scores increased by 2% when the test was readministered within a short timeframe (1-3 weeks). The K-TUT scores in the pretransplant group improved by 6%, however, during the 6-month lag time that occurred between the pilot and the test-retest. This could speak to the importance of performing reliability testing on this instrument within a reasonable time frame, because knowledge can improve over time. The ICC for mean scores was greater than 0.75, indicating excellent agreement in both cohorts. The κ statistic and percent agreement were also calculated to assess the reproducibility at item level. The majority of items had at least fair agreement between test and retest, and all except 1 item in the posttransplant cohort had fair percentage of overall agreement of 67%, which was defined a priori (item 22a).

Criterion validity is the degree to which the instrument correlates with an outcome. This type of validity testing was not applicable in this case, because there is no criterion standard for which to compare knowledge. We were, however, able to assess the construct validity, or the extent to which the questionnaire scores relate with another measure derived from a theoretically driven hypothesis in the pretransplant cohort.³⁴ We used health literacy as the comparator because studies have identified a direct correlation between health literacy and medical knowledge.^35-41 In a study of 401 adult patients with chronic kidney disease (stages 1-5), disease state knowledge scores were significantly associated with health literacy measured by the Rapid Estimate of Adult Literacy in Medicine (β = 0.06; 95% CI, 0.03-0.10; P = 0.001).³⁵ Similar correlations were discovered in a study investigating the relationship between diabetes knowledge and health literacy in 125 patients with diabetes (β = 0.55; 95% CI, 0.29-0.82).³⁶ In another study of patients with chronic diseases (n = 653), respondents with inadequate health literacy (as measured by the Short Test of Functional Health Literacy) knew significantly less about their illness (asthma, diabetes, congestive heart failure) than those with adequate health literacy, even after adjusting for the potential confounders of older age, disease duration, and prior attendance at education sessions.³⁷ The positive relationship between health literacy and knowledge has also been noted with medications^38,39 and other disease states, such as human immunodeficiency virus/acquired immune deficiency syndrome.⁴⁰ Consistent with these aforementioned studies, significant correlation was shown between mean knowledge scores and health literacy in the pretransplant cohort (r = 0.52; P < 0.05), indicating construct validity in our questionnaire.

Future studies should investigate the interpretability and responsiveness of the K-TUT. The interpretability refers to how well meaning can be applied to the given quantitative scores,⁴¹ whereas the responsiveness describes how accurately the instrument can detect clinical important changes over time.⁴² One way to examine the responsiveness would be to administer the K-TUT to a group of individuals before and after an education session on transplant and determine how much the scores change. Although this particular study was not aimed for this purpose, some interesting correlations lead us to speculate that the K-TUT could also be used to identify subgroups of patients with knowledge deficits that require targeted education. Higher K-TUT scores in the posttransplant cohort were significantly associated with female sex, white race, English as a first language, and higher levels of education in the posttransplant cohort (P < 0.05). These associations were not significant in the pretransplant cohort (n = 41), and we speculate that this may be an effect of sample size. More study is warranted in this area.

Because the present study took place in several stages and in 2 separate cohorts, the process varied slightly with each cohort, depending on the overall purpose. For instance, the K-TUT was initially piloted in a study that also aimed to evaluate health literacy, beliefs of medicine, and education satisfaction. This provided an opportunity for the K-TUT to be administered by a research assistant in the clinic setting. The questionnaire was mailed to the posttransplant group, however, because the sole purpose was to evaluate the questionnaire. This also provided an opportunity to undertake a more in-depth analysis of the questionnaire, such as gathering feedback on whether or not the participants believed the K-TUT was an accurate measure of transplant knowledge.

A research assistant was not present to observe the test in the posttransplant cohort, and we wondered whether the participants in this group would look up the answers. We tried to minimize this possibility by including a cover letter encouraging participants to fill out the questionnaire as honestly as possible and by distributing the questionnaire by mail (we reasoned that it might be tempting to search out answers on the internet, if filling out the questionnaire electronically). Interestingly, the mean scores of 56.2 ± 6.3 (81%) in the posttransplant cohort were only 4% higher than in the pretransplant group 53.1 ± 8.5 (77%). Hence, it seems reasonable to assume that the K-TUT could be successfully administered in a clinic setting or distributed by mail.

There are inherent limitations to measuring disease state knowledge as an outcome. It is important to note that disease state knowledge does not directly translate into behavior change, self-efficacy, or adherence. However, measuring patient knowledge may help to characterize these elusive, yet important concepts. Given the importance of education in the field of transplantation, educational research should be commended and encouraged. The influence of education on transplant outcomes cannot be properly explored, however, until we have accurate way to measure its effect. To perform sound, scientific research and produce trustworthy results, the measurement tool must be proven to be reliable. Using a flawed questionnaire in the world of behavioural science could be considered analogous to using a miscalibrated scientific instrument in a laboratory. In order for educational research to be respected, it should be held to the same rigorous standards as other research. Developing a validated tool to measure transplant knowledge is a first step in this process.

In addition to research, a validated tool to assess transplant knowledge could be an important resource in the clinical setting. Health care providers, such as nurses, physicians, or pharmacists, could use the tool in the pretransplant setting to identify areas of patient confusion. Targeted education could then be applied accordingly. For instance, although the question: “True or false? 'Every person who receives a kidney transplant feels better than they did before the transplant’ ” may seem subjective, this item can provide very important information regarding patient perception and knowledge of transplantation. In consultations with patients that occurred before the tool development, we discovered an important theme: “patient expectation may not always match outcome.” This specific question was designed to help identify patients that may be at risk of not internalizing all potential consequences of transplantation. Participants (especially in the pretransplant stage) who answer this question “true” should be provided with more education on the risks and benefits of the procedure and the lifelong immunosuppression required posttransplant, along with potential side effects. Likewise, the tool could be used in the posttransplant setting to identify misconceptions about required medications and lifestyle changes. For instance, if a patient incorrectly answers “true or false? ‘Herbal supplements are generally safe to take with your transplant, since they are natural’ ” more education could be provided around the potential risks for drug interactions and/or potentially stimulating the immune system with some herbal products.

There are limitations to this study that deserve acknowledgement. We had identified one of the potential uses of the questionnaire to be identifying gaps in patient knowledge (for instance so that health care providers can tailor patient education). As such, we wanted to ensure that the instrument was comprehensive and did not use item reduction to minimize the number of questions. A questionnaire that takes too much time to complete may promote user fatigue and missing data, and be challenging to use in a busy clinic setting. It is encouraging, however, that the majority of respondents believed that no questions should be removed from the instrument.

Because a tool that could measure knowledge across the continuum of care would be extremely beneficial in the research setting, we opted to include content that was applicable to both transplant candidates and transplant recipients. In doing so, however, we omitted some specific aspects of knowledge that our focus group¹⁸ identified as relevant in the before transplant (such as information about the surgical procedure). The questionnaire was developed and studied in a single center in Canada, and particularly high levels of health literacy were noted in the pretransplant group. Although every effort was made to keep the items general and widely applicable to other populations and patients with low health literacy, the validity of this instrument should be confirmed in other settings. Furthermore, this finding should be confirmed in a cohort of posttransplant recipients.

The process of establishing the validity of a questionnaire is lengthy. We have developed a questionnaire to measure transplant disease state knowledge and have undertaken several steps to examine its validity. Although more study is warranted to further assess psychometric properties (such as interpretability, and responsiveness), the K-TUT appears to be a promising tool to measure transplant knowledge.