Reliability & validity of the LIBRE Profile

Abstract

Background:

The Life Impact Burn Recovery Evaluation (LIBRE) Profile^© was developed using Item Response Theory methods to assess social participation after a burn injury. The LIBRE Profile measures six areas of social participation: Relationships with Family & Friends, Social Interactions, Social Activities, Work & Employment, Romantic Relationships, and Sexual Relationships. It can be administered through a computerized adaptive test or through fixed short forms. The goal of this study was to further examine the psychometric properties of the LIBRE Profile, including reliability and validity.

Methods:

We examined the validity of the LIBRE Profile by administering the six LIBRE Profile scales as well as legacy measures that assessed similar constructs. We calculated the Pearson correlations between the LIBRE Profile scales and the scores on the same-domain and crossdomain legacy measures to evaluate convergent and divergent validity. We then administered the LIBRE Profile scales a second time, seven to ten days after the first administration, to a sample of adult burn survivors to evaluate test-retest reliability. We calculated repeatability coefficients, standard error of measurement, and minimal detectable change to establish the threshold beyond which the amount of change observed across an episode of care cannot be explained as measurement error.

Results:

For reliability, the repeatability coefficients ranged from 7.31 to 9.27 and SEMs ranged from 2.62 to 3.39 for all six scales. MDC₉₀ values ranged from 6.08 to 7.86 points, and MDC₉₅ values ranged from 7.26 to 9.40 points. All correlations between the LIBRE Profile scales and legacy measures are significant (p < 0.05) and in the expected directions for both convergent and divergent validity.

Conclusions:

This study provided evidence for the reliability and validity of the LIBRE Profile, one of the first tools that measures exclusively the social participation after a burn injury.

1. Introduction

The Life Impact Burn Recovery Evaluation (LIBRE) Profile is a patient reported outcome measure (PROM) that assesses the impact of a burn injury on the social participation of burn survivors. This tool was developed because medical advances have led to reduced mortality and concomitant complications from a burn injury, and there is now a need to measure and address the psychosocial ramifications of such a traumatic event, as well as to optimize the pathways to a strong recovery [1,2]. Other PROMs such as the Burn Specific Health Scale and the Burn Outcome Questionnaires are important for assessing health related quality of life for burn survivors, but the LIBRE Profile is the first to use Item Response Theory (IRT) methods for development and to exclusively focus on social participation [3-8]. Previous research, as well as feedback from burn survivors and family members, have identified this area as important to the recovery process [9-12].

The LIBRE Profile provides information that can potentially inform treatment plans and evaluate programs addressing social participation after severe burn [13]. It can be administered through a computerized adaptive test, called the LIBRE Profile-CAT or fixed short forms, called the LIBRE Profile-SF, which are both on the same metric. Through these CAT and SF administrations, fewer questions can be administered than traditional legacy measures without loss of scoring precision. The computerized adaptive testing technology used in the LIBRE Profile-CAT allows for the items administered to be responsive to the burn survivor’s answers in real time. For example, if a burn survivor responds that he or she has difficulty with the situation posed by the item, then the next item will ask about a situation that is less difficult. The LIBRE Profile-CAT will administer a subset of items from the item bank until a level of precision is reached. Although the LIBRE Profile-SF is a set of fixed questions, it assimilates the computer-generated version well as discussed in previous works [14]. By administering fewer questions, the LIBRE Profile can be used in a fast-paced clinical setting. Out of the entire item bank, the LIBRE Profile-CAT administers only six to ten items for each scale, and the LIBRE Profile-SF administers ten items for each scale. Depending on how many scales are applicable, the participant would therefore respond to a range of 18–60 items when completing the LIBRE Profile-CAT, and between 30 and 60 items when completing the LIBRE Profile-SF. Other published works describe the conceptual basis and psychometric development of the LIBRE Profile [12-15].

The goal of the present study was to conduct a psychometric evaluation of the LIBRE Profile by examining the (1) test–retest reliability and (2) convergent validity. To do this, we administered the LIBRE Profile-SF at two different time points and compared the results of each LIBRE Profile scale with established legacy measures that assess a similar construct. We also reviewed the standard error of measurement (SEM) to quantify the precision of scores at the individual level. Lastly, we analyzed the minimal detectable change (MDC₉₀ and MDC₉₅) to determine the lowest threshold beyond which the amount of change observed across an episode of care cannot be explained as measurement error and provide a basis for understanding if it is clinically important.

2. Methods

2.1. Procedure

Participants were asked to complete two online surveys to assess test–retest reliability and convergent validity. The first survey (T1) administered contained the six LIBRE Profile-SFs, the well-established Veterans RAND 12 Item Health Survey (VR-12), which evaluates the physical and mental health-related quality of life of the sample, as well as five other well-known legacy measures described below [5,16-22]. The legacy measures were included to assess convergent validity of the LIBRE Profile. The measures are psychometrically sound and valid, and they assess similar constructs as the LIBRE Profile. Smilkstein’s Family System APGAR Items were used to assess the validity of the Relationships with Family & Friends LIBRE Profile scale. APGAR stands for the five aspects of family functioning measured by the tool: Adaptability, Partnership, Growth, Affection, and Resolve [16,17]. Three domains from the Young Adult Burn Outcome Questionnaire (YABOQ) were used to assess three of the LIBRE Profile scales: Social Function Limited by Appearance Compared to Social Interactions, Social Function Limited by Physical Function Compared to the Social Activities scale, and Work Reintegration was compared to the Work & Employment scale [5]. In addition, the Stanford Presenteeism Scale was administered in order to compare with the Work & Employment scale. The Stanford Presenteeism scale is a measure of the impact of health-related issues on a person’s work performance and productivity [18]. The Relationship Assessment Scale was administered to assess the LIBRE Profile Romantic Relationship scale [19]. Two domains from the Patient-Reported Outcomes Measurement Information System (PROMIS) Sexual Function scale were compared to the LIBRE Profile Sexual Relationship scale: Global Satisfaction with Sex Life and Interest in Sexual Activity. The Global Satisfaction with Sex Life scale is a general assessment of one’s sex life. The Interest in Sexual Activity scale measures if a person does or does not want to engage in sex [20].

To assess test–retest reliability, the research team administered the LIBRE Profile-SF at two different time points online. As mentioned, T1 contained the six SFs in addition to other legacy measures. The second survey (T2) contained the six SFs for test–retest purposes. T2 was administered seven days after completion of T1, and had to be completed within 72h. The seven to ten day timeframe is commonly used so that there is enough of a time lag to lessen the potential for recall effects based on T1 responses, but short enough so that respondent’s health will not have changed substantially [23,24]. Eligible participants were burn survivors 18 years of age or older, with an initial discharge from the hospital, a TBSA of 5% or greater or a burn to a critical area (face, hands, feet, or genitals), able to read and understand English, and with access to email and the ability to navigate the Internet. Subjects were recruited from the Phoenix Society for Burn Survivors, the LIBRE Clinical Advisory Board representing hospital burn units across the U.S. and Canada, from the LIBRE Project’s contact list, and by word of mouth.

2.2. Sample size

We set the target sample at 250 participants at T1 based upon prior work [11]. We estimated approximately two-thirds of the sample would respond to the Work & Employment, Romantic Relationships, and Sexual Relationships scales. If at least 100 individuals responded to each scale, an assumed correlation of 0.65 would yield a 95% confidence interval (CI) of sufficient precision (±0.27) for convergent validity analyses, which require higher power than the test-retest component. That same sample size for the test–retest portion of the study with an assumed intraclass correlation coefficient (ICC_3,1 of 0.6 would yield a 95% CI of ±0.29, also of sufficient precision. To estimate our target sample size, we assumed a credible ICC, denoting ICC values <0.4 as poor, 0.4–0.59 as fair, 0.60–0.74 as good, and >0.75 as excellent [25].

2.3. Data analysis

The LIBRE Profile-SF raw summed scores were converted to a T-score for each subject. The T-score was the standardized score based on the previously published LIBRE Profile national calibration sample of 601 burn survivors with a mean of 50 and a standard deviation of 10 [13]. The higher the score, the better the burn survivor is performing in that area [14]. If a burn survivor has a score above 50, then the interpretation would be that the burn survivor is faring better in that scale than the mean of the sample from the calibration study. If the burn survivor scores below 50 then the burn survivor is faring worse when compared with the mean of the calibration study sample.

To assess the change in scores from T1 to T2, we examined the magnitude of the repeatability coefficients. With the SD of change defined as the square root of the average of squared change scores, the repeatability coefficient is 1.96 × SD of change. The percentage of subjects exceeding these limits were determined. We also generated Bland Altman plots and calculated the ICC_3,1 for respondents that completed surveys T1 and T2 as ICCs are widespread in the psychometric literature despite criticism in the statistical literature [27-29]. For convergent validity, we assessed the convergent and divergent Pearson correlations between the legacy measures and the LIBRE Profile scales taken at T1. We hypothesized that each LIBRE Profile scale would be strongly and significantly correlated with the legacy measure identified as assessing a similar construct to that specific LIBRE Profile scale. In addition, we hypothesized that each LIBRE Profile scale would have a weak or moderate correlation with the legacy measures not identified as assessing a similar construct. For example, the Relationships with Family & Friends LIBRE Profile scale should have a correlation of at least 0.6 with the Smilkstein Family System APGAR, but a correlation of less than 0.6 with the Work Reintegration YABOQ subdomain [30].

We also calculated additional properties related to reliability, SEM, MDC₉₀, and MDC₉₅. SEM is a measure of the precision of an individual’s score [31]. It is calculated by S_B*√(1–ICC) where S_B is the standard deviation at baseline [32]. MDC₉₀ and MDC₉₅ indicate the minimal threshold for change in an individual’s score that would represent a change beyond measurement error with 90% and 95% confidence limits respectively [33]. MDC₉₀ is calculated by 1.64*√2*SEM and 1.64 is derived from the standardized normal distribution in which 90% of observations lie within ±1.64 SDs of the mean. MDC₉₅ is calculated by 1.96*√2*SEM. MDC₉₀ and MDC₉₅ are both widely used in studies, and we used both to strengthen our results [34,35].

3. Results

3.1. Sample

The total sample included 259 burn survivors who completed at least T1 (Table 1). Subsets of this group, 153 for Sexual Relationships, 162 for Romantic Relationships and 154 for Work and Employment also completed the LIBRE Profile scales at T2 based on their responses to three filter questions asking if they were currently working for pay, in a romantic relationship, or sexually active.

Table 1 –

Respondent demographics.

Characteristic	N	%
Age mean [SD]	45.5 [14.6]
Gender
Women	152	58.7%
Men	105	40.5%
Missing	2	0.8%
Race
White	221	85.3%
Black/African American	16	6.3%
Other	21	8.1%
Missing	1	0.4%
Hispanic ethnicity
Yes	19	7.3%
No	240	92.7%
Missing	0	0
Education
Less than high school	5	1.9%
High School/GED	90	34.8%
Greater than high school	164	63.3%
Missing	0	0
Time since burn injury	<3 years	61 (23.6%)
	3–10 years	70 (27.0%)
	>10 years	128 (49.4%)
TBSA mean [SD]	44.5 [25.1] N=199
VR-12: PCS mean [SD]	46.6 [10.6] N=259
VR-12: MCS mean [SD]	45.3 [12.2] N=259

Out of the 259 burn survivors, 58.7% were female, with an average age of 45.5 and an age range of 19 to 81. Mean TBSA was 44.5% and just over half of the sample were injured within 10 years. Out of 391 people who expressed interest in the study, 15 individuals were deemed ineligible, 115 were lost to follow-up or were unresponsive, and two declined participation. Mean VR-12 scores were 46.6, SD of 10.6 for the physical summary (PCS) and 45.3, SD of 12.2 for the mental summary (MCS).

3.2. Reliability analyses

Fig. 1 displays the Bland Altman plots illustrating the agreement between each participant’s LIBRE Profile scale score for T1 and T2. The figures show that across all scales, the average percentage of subjects that exceeded the upper or lower bound, ±1.96 SD, is 6.25%, and the range is 4.9%–7.14%. Although this is slightly more than the 5% distribution, there appears to be no systematic bias towards higher or lower differences when there is poor agreement. Further, those cases above and below the upper and lower limits of agreements tended to be for participants whose scores were above the mean of the scale, which could suggest that scores are slightly less reliable for higher scoring and performing individuals [36].

Fig. 1 –

Bland Altman Plots of LIBRE Profile T1 and T2 scores.

For the six LIBRE Profile scales, the repeatability coefficients ranged from 7.31 to 9.27, and the ICCs ranged from 0.84 to 0.91 (Table 2). The SEMs obtained for the six scales ranged from 2.62 to 3.39. The Social Interactions scale had the lowest SEM and the Work & Employment scale had the highest. MDC90 values ranged from 6.08 to 7.86 points, and MDC₉₅ values ranged from 7.26 to 9.40 points. When examining the magnitude of the change scores, we found the only significant change was for the Sexual Relationships scale (t value=−3.25, p < 0.01), however the magnitude was small (effect size=−0.11).

Table 2 –

Test-retest reliability of the LIBRE Profile.

LIBRE Profile Scale	N	Repeatability coefficients	ICC (95% CI)	SEM	MDC90	MDC95	t Value (df), p	Effect size of the difference (difference/SD at baseline)
Family & Friends	246	7.58	0.89 (0.86, 0.92)	2.83	6.55	7.83	−1.24 (245), 0.22	−0.04
Social Interactions	245	7.31	0.91 (0.89, 0.93)	2.62	6.08	7.26	−2.04 (244), 0.04	−0.05
Social Activity	245	8.31	0.88 (0.85, 0.91)	3.04	7.05	8.43	−0.66 (244), 0.51	−0.02
Work & Employment	154	9.27	0.84 (0.78, 0.88)	3.39	7.86	9.40	−0.22 (153), 0.82	−0.01
Romantic Relationships	162	9.10	0.88 (0.84, 0.91)	3.32	7.70	9.20	1.68 (161), 0.09	0.06
Sexual Relationships	153	7.78	0.90 (0.86, 0.93)	2.85	6.60	7.89	−3.25 (152), <0.01	−0.11

3.3. Convergent validity

Table 3 gives mean scores for each of the six LIBRE Profile domains. Scores are standardized with a mean of 50 and a SD of 10 based upon the LIBRE Profile calibration study burn survivor sample. Average scores ranged from 47.70 for Social Interactions to 51.73 for Social Activities. The average scores across the scales were around 50, which indicates that the average function levels of this validity sample is similar to those in the LIBRE Profile calibration sample. Table 3 also has the mean scores of the legacy scales used to measure validity. All correlations between the LIBRE Profile scales and legacy measures are significant (p <0.05) and in the expected directions (Table 4). We also examined scatter plots for the LIBRE Profile scale scores and legacy measure scores for outliers, and found they were not a problem (data not shown).

Table 3 –

Mean scores on all measures administered.

Measure	N	Mean (SD)
LIBRE Profilea
Relationship with Family & Friends	259	48.41 (8.47)
Social Interactions	259	47.70 (8.97)
Social Activities	259	51.73 (8.93)
Work & Employment	164	48.46 (8.48)
Romantic Relationships	175	48.85 (9.69)
Sexual Relationships	163	48.53 (9.14)
YABOQb
Social Function Appearance	259	79.4 (34.1)
Social Function Physical	259	80.4 (32.7)
Work Reintegration	164	56.7 (22.8)
Smilkstein’s Family System APGARc	259	7.0 (3.0)
Stanford Presentism Scaled	164	24.3 (5.6)
Relationship Assessment Scalee	167	28.2 (5.8)
PROMIS Sexual Functionf
Global Satisfaction with Sex Life	163	55.4 (8.5)
Interest in Sexual Activity	163	56.6 (8.2)

^aScores are all T scores with mean of 50, SD of 10. Higher scores mean better performance.

^bHigher scores mean better health.

^cScores range from 0 to 10. Higher scores mean more satisfaction with family functioning.

^dScores range from 6 to 30, higher scores indicate higher presenteeism at work.

^eScores range from 7 to 35, higher scores mean higher relationship satisfaction.

^fScores are T scores with mean of 50, SD of 10. Higher scores mean more satisfaction/interest.

Table 4 –

Pearson correlations (r) between established and LIBRE Profile measures.

	LIBRE Profile Scales
Legacy Measure	Family & Friends	Social Activities	Social Interactions	Romantic Relationships	Sexual Relationships	Work & Employment
YABOQ: Social Function Appearance	0.29*	0.49*	0.60*	0.29*	0.41*	0.32*
YABOQ: Social Function Physical	0.30*	0.67*	0.33*	0.23*	0.31*	0.43*
YABOQ: Work Reintegration	0.30*	0.36*	0.27*	0.23*	0.23*	0.31*
Smilkstein’s Family System APGAR	0.77*	0.42*	0.37*	0.39*	0.32*	0.38*
Stanford Presentism Scale	0.44*	0.57*	0.33*	0.37*	0.44*	0.66*
Relationship Assessment Scale	0.22*	0.29*	0.24*	0.82*	0.45*	0.32*
PROMIS: Global Satisfaction with Sex Life	0.30*	0.34*	0.42*	0.47*	0.81*	0.372*
PROMIS: Interest in Sexual Activity	0.25*	0.23*	0.32*	0.22*	0.58*	0.33*

Cells in grey denote convergent validity, and unshaded cells denote divergent validity.

^*All coefficients were significant at the p < 0.05 level.

4. Discussion

This study examined the reliability and validity of the LIBRE Profile, a measure of social participation of burn survivors. We found that the LIBRE Profile displays for the most part highly credible test–retest reliability in the sample of adult burn survivors, with repeatability coefficients that ranged from 7.31 to 9.27 and ICCs all above 0.8. Widely-used legacy measures tend to display ICCs of at least 0.7 [37-39]. The SEM, which indicates the reliability of individual scores, was between 2.62 to 3.39 for all scales, reflecting good discriminant validity. The lower the SEM, the more precise the score [23]. The MDC₉₀ ranged from 6.08 to 7.86 points, and the MDC₉₅ ranged from 7.26 to 9.40 points. The MDC₉₀ and MDC₉₅ represent the smallest amount of change in an individual’s score that is not due to measurement error and characterizes measurable true change. For example, the MDC₉₀ for Work & Employment is 7.86. If the respondent’s score changes by at least that amount, we can conclude that the individual’s performance in that domain has changed, and the difference in the score is not due to measurement error. However, the clinical importance of that change will require future work in clinical settings to better understand the implications of those changes on the outcomes of burn survivors.

We also found good convergent validity by comparing the LIBRE Profile scales to established legacy measures that assessed similar constructs. The correlations were all in the expected direction, and all but two of the matching scales had strong correlations (>0.8) [30]. The YABOQ Work Reintegration domain had the lowest correlation (0.31) with the LIBRE Profile Work & Employment scale. This weaker correlation may be due to questions in this subdomain that are not applicable to all respondents. More specifically, this domain asked about experiences returning to work or school, but some participants were infants when burned. The other legacy measure correlated with the LIBRE Profile Work & Employment scale, the Stanford Presenteeism Scale, had a strong correlation 0.66, which suggests that the low correlation with the YABOQ Work Reintegration subdomain may be because of how the questions are worded.

The LIBRE Profile Sexual Relationship scale had moderately strong correlations with the PROMIS Global Satisfaction with Sex Life and Interest in Sexual Activity Scales, 0.81 and 0.57, respectively. The LIBRE Profile Family and Friends, Social Interactions, and Social Activities scales correlated well with their legacy measure counterparts, all above 0.60.

For tests of divergent validity, as postulated, correlations were for the most part lower than the correlations of those LIBRE Profile scales matched against legacy measures of similar constructs. There were low correlations (typically in the 0.20–0.30 range) among the LIBRE Profile scales and the legacy scales measuring other domains of content. The YABOQ Work Reintegration domain has a higher correlation with the LIBRE Profile Social Activities scale (0.36) than with the LIBRE Profile Work & Employment scale, but otherwise all legacy measures have the highest correlation with the LIBRE Profile scale identified to be assessing similar constructs.

VR-12 scores were collected to assess health related quality of life scores for a population of adult burn survivors. The scores are lower than average with a mean of 50, SD of 10, for physical and mental health functioning. These results indicate a less than average healthy population that is about 0.35 and 0.50 of one SD lower than a general U.S. population, which is between a small and moderate effect size [40].

4.1. Limitations

There are a few limitations to this study. The Work Reintegration subdomain from the YABOQ was not applicable to all participants because of how the items were worded, which may have affected the validity correlations. Also, the study sample is a convenience sample, so these findings may not be generalizable to other audiences. However, our sample is quite heterogeneous and represents a range of characteristics in terms of demographic and clinical variables. In addition, we found that although there were more women than men in our sample, this fact did not bias our results. Future work can investigate the selection biases that may contribute to these findings.

Since we recruited from the LIBRE Project using a contact list, there was some overlap of about 100 participants in this study who had participated in a previous LIBRE Project study. This may overestimate the reliability of the LIBRE Profile tool. The LIBRE Profile should be tested in other samples in future work in order to confirm or corroborate its validity. Also, there are other psychometric tests that can be done on the LIBRE Profile that will add to these findings by examining the reliability and validity of the measure even further. Lastly, the LIBRE Profile can be tested in future clinical settings for evidence promoting the tool to capture clinically relevant information.

4.2. Conclusion

This study presents evidence that supports the validity and reliability of the LIBRE Profile as a PROM that measures the social participation of burn survivors. PROMs are a growing area in burn care and the LIBRE Profile enables future measurement of interventions meant to address these issues [41]. The LIBRE Profile can be used in clinical practice to measure the effect of the increasing number of interventions that address social participation of burn survivors, such as peer support groups and social skills training.