Abstract
Objective
To assess the reliability, criterion and construct validity of the self-administered Brief Index of Lupus Damage (SA-BILD), a patient-reported measure of organ damage in systemic lupus erythematosus (SLE).
Methods
The validity of the SA-BILD was assessed using data from the Georgians Organized Against Lupus (GOAL) survey. GOAL is longitudinal cohort of SLE patients predominantly derived from the Georgia Lupus Registry, a population-based registry established in Atlanta, Georgia, United States (US). Seven hundred eleven participants with documented SLE completed the SA-BILD. To test reliability, the SA-BILD was re-administered to 32 patients. Criterion validity was examined in 150 respondents for whom the SLICC/ACR Damage Index (SDI) was also completed. Construct validity was assessed among 711 GOAL participants dividing the SA-BILD scores into quartiles and examining the association with demographics, health status and health care utilization.
Results
The test-retest correlation score was 0.93 (p<0.0001). The item-by-item agreement with the SDI was over 80% for most SA-BILD items. The Spearman rank correlation for SDI and SA-BILD was moderately high (r=0.59, p<0.0001). SA-BILD scores showed significant associations in the expected directions with age, disease duration, disease activity, overall health, comorbidity index, and physician visits.
Conclusion
The SA-BILD is reliable and has very good or good criterion validity compared to the SDI when tested in a predominantly African American cohort of US SLE patients. Associations of SA-BILD scores with sociodemographics and health status are consistent with previous studies. These findings support the use of SA-BILD as a valid measure of patient-reported damage in SLE.
Keywords: Systemic Lupus Erythematosus, disease outcomes, patient-reported damage, instrument validation
As the life expectancy of patients with systemic lupus erythematosus (SLE) has improved in the past decades, comorbid conditions have become important determinants of outcomes. SLE patients are frequently affected by irreversible organ damage that can occur as a consequence of disease activity, disease-related chronic inflammation, and/or side effects of the drugs used to treat the disease (1–4). Quantifying organ damage of SLE populations has therefore become a relevant dimension of outcomes research (1, 4–10).
Damage is most commonly assessed with the validated SLICC/ACR Damage Index (SDI)(1, 11–13). The SDI has been found to predict mortality (5, 7, 8, 14), physical function (15, 16), work disability (17, 18), health care utilization (19, 20) and societal burden (21). In its original form, this 41-item questionnaire was to be completed by a trained physician (12, 13). However, the need to expand the available SDI tool to assess SLE-related damage with a patient-reported measure arose along with the development of community-based cohorts. Efforts led by investigators from academic centers in the United States (US) resulted in two validated instruments originating from the SDI: the Lupus Damage Index Questionnaire (LDIQ) and the Brief Index of Lupus Damage (BILD) (22, 23). LDIQ consists of 56 questions administered as a written survey. BILD was designed to be a shorter (28 questions) patient-reported tool administered by an interviewer with SLE patients over the telephone or in person.
A preliminary validation study found that the BILD was a suitable proxy of the SDI (23). The BILD was highly acceptable to respondents and administered efficiently by telephone interview. The criterion validation of BILD was conducted with a relatively small sample of predominantly young, nonwhite patients from two university-affiliated SLE clinics, while the construct validity was analyzed with data from the Lupus Outcomes Study (LOS), a community-based cohort of predominantly middle-class, well-educated SLE patients, 66% of whom were non-Hispanic whites (23).
Because SLE patients from minority groups suffer worse disease outcomes, having a cost-effective tool to quantify organ damage in patients from vulnerable groups is essential to better understand the burden of the disease at the population level. However, these groups are typically underrepresented in measure development and validation research. The promising validity findings of the BILD, along with the low administration burden and high acceptability by LOS patients, encouraged us to adapt the BILD as a self-administered written version that could be mailed to SLE patients with diverse sociodemographic backgrounds. Here, we describe the adaptation of the BILD to a self-administered format (SA-BILD) and assess its reliability, criterion and construct validity in a large SLE cohort from the Southeastern US that includes a representative proportion of high-risk individuals.
Patients and Methods
Study population
Data from the Georgians Organized Against Lupus (GOAL) cohort were used to assess reliability, criterion and construct validity of the SA-BILD. GOAL encompasses a large cohort of adult English speaking SLE patients from metropolitan Atlanta, Georgia (GA). The overall aim of GOAL is to examine the impact of sociodemographic and health care factors on outcomes that are relevant to patients, health care providers and policymakers. Recruitment and data collection methods, as well as the sociodemographic characteristics of SLE participants have been described (24)_ENREF_24. Briefly, the primary source of SLE enrollees is the Georgia Lupus Registry (GLR), a population-based registry funded by the Centers for Disease Control and Prevention in order to better estimate the incidence and prevalence of SLE in Atlanta, an area with large number of African Americans at high risk for SLE (25). Implemented through a partnership between the Georgia Department of Public Health (GA DPH) and Emory University, the GA DPH enabled Emory investigators to review medical records without patient consent to meet the public health goal of determining the incidence and prevalence of lupus (under the HIPAA Privacy Rule, 45 CFR parts 160 and 164). Furthermore, the GA DPH allowed Emory investigators to recruit SLE patients into the GOAL Cohort. Thus, adult lupus patients, who received medical care at community- and university-based practices, were recruited by mail, telephone, and in person to complete annual self-administered surveys. Over 70% of lupus patients in the GOAL cohort were ascertained from the GLR. Other patients came from lupus clinics at Emory University, the indigent care hospital in Atlanta (Grady Memorial Hospital) and community rheumatologists from metropolitan Atlanta. There were 850 participants with a documented diagnosis of SLE: 688 fulfilled > 4 Revised ACR Criteria for the Classification of SLE and 162 fulfilled 3 ACR Criteria and had a diagnosis of SLE by the attending Board certified rheumatologist.
The Emory University Institutional Review Board, Grady Health System Research Oversight Committee, and the GA DPH Institutional Review Board approved the GOAL study protocol. All GOAL participants provided informed consent.
Data collection
Data are collected for GOAL through annual surveys that include questions on sociodemographic characteristics, health care utilization, and validated measures of health status, disease activity and comorbidities. Given the high proportion of socioeconomically disadvantaged subjects in the GOAL Cohort, the GOAL survey was designed for a population with limited health literacy and targets an eighth-grade reading level. Additionally, we offered flexible administration modes (self- or interviewer-administered GOAL survey) and delivery methods (mail, telephone, in person). Among 751 SLE respondents to the baseline GOAL survey, 711 completed the written survey (primarily by mail). The remaining 40 participants chose to complete GOAL surveys by telephone interview, and are not included in this study. Sociodemographic and clinical characteristics were similar between respondents to the mail-based and to the telephone-based surveys, with the exception of disease duration, which was significantly shorter for respondents to the written survey (mean=13.3 years, SD 9.0) than for the group interviewed by the phone (mean= 16.8 years, SD 8.0) (data not shown). No significant differences were found between survey respondents (n=751) and non-respondents (n=99) in sociodemographic and clinical characteristics.
The self-administered BILD (SA-BILD)
The original BILD was designed for administration by telephone interview. The description of the development and validation of the BILD has been published (23). In order to adapt the 28-question BILD tool to a self-administered form, we used the same table-based format as the other instruments included in the GOAL survey. To reduce a potential high rate of unknown answers, we adopted an approach similar to the LDIQ questionnaire, providing only ‘yes’ or ‘no’ response for each item (22). As in the LDIQ, after a brief introduction on the purpose of the questionnaire, we added: “Don’t worry if there are some medical words you don’t understand. This usually means that you don’t have the problem the question is asking about”. Additionally, to enhance understanding of each question by readers of the SA-BILD (as opposed to listeners of the original BILD), we integrated into the questions or added in parenthesis the optional notes of the original BILD. Then, the SA-BILD was piloted among 5 adult female patients with SLE from the Grady Lupus Clinic targeting wide age and educational attainment. Patients’ feedback on the clarity and the meaning of the questions, and the appropriateness of the response choices served to revise the questionnaire. The only modification needed was the addition of a note to clarify that transient ischemic attack -or TIA- is not included as a positive response for stroke (item 5). Then, the SA-BILD was included into the baseline GOAL survey and mailed to a subset of 247 SLE patients. After responses were evaluated, questions related to premature gonadal failure (only for women) were moved to the end of the table in order to reduce potential missing responses by males of two items originally placed at the bottom: diabetes and malignancy. The annual GOAL survey with the final SA-BILD was then mailed to the remaining GOAL participants. Like the original telephone form, the SA-BILD contains 28 questions that capture information on 26 of the original SDI items (Appendix A).
Statistical Analysis
Reliability was tested among 32 patients who completed the SA-BILD survey two times between December 2011 and June 2012. Spearman rank correlation coefficient was used to measure test-retest reliability.
Criterion validity was examined in 150 consecutive SA-BILD SLE respondents who had a follow up visit at the Grady Health System affiliated Lupus Clinic (n= 93) or the Emory University affiliated Lupus Clinic (n=57) within 6 months of the annual GOAL survey completion date. Between February and June 2012, staff rheumatologists completed the SDI blinded to the subject’s SA-BILD responses. We compared the SA-BILD item responses to the corresponding SDI responses by calculating the item-by-item percent-observed agreement (po) with the SDI. The prevalence-adjusted bias-adjusted kappa (PABAK; 2po - 1) has been proposed as a better measure of agreement than kappa when prevalence varies or when the prevalence of each method or instrument differs (26). Like kappa, a PABAK value of −1 indicates perfect disagreement, 0 indicates no agreement, and 1 indicates perfect agreement. We also compared the distributions of the overall SDI and SA-BILD scores, calculating the Spearman rank correlation coefficients between both measures.
Construct validity was assessed using GOAL data collected by July 31, 2012. Because SA-BILD scores did not show a normal distribution, we divided the GOAL sample into quartiles based on SA-BILD scores to examine the correspondence of SA-BILD scores with demographics, disease duration, health-related measures and health care utilization, which are measures previously associated with SDI disease damage. For demographic measures, we included age, sex, ethnicity (African American versus White), education (high school or less versus some college education or greater), annual household income below the federal poverty threshold (adjusted for number of individuals in the household), and work status. Disease and health status measures included the Systemic Lupus Disease Activity Questionnaire (SLAQ) (27), the Medical Outcomes Study Short Form 12 (MOS SF-12) (28) and the Rheumatic Diseases Comorbidity Index (RDCI) (29). We assessed annual mean number of outpatient visits (to rheumatologists and to all physicians) over the past year as measures of health care utilization. We used Cochran–Armitage trend test to compare the distribution of categorical measures across quartiles of SA-BILD (30, 31). Likewise, we compared continuous measures using analysis of variance. Additionally, to explore the independent association of the sociodemographic measures with SA-BILD, we modeled the SA-BILD score as a function of age, disease duration, sex, ethnicity, education and employment. We used logistic regression and multiple regression analyses to test the SA-BILD score as binary (> 4 versus 0, 2–3 versus 0, 1 versus 0) and continuous (raw score) variables, respectively. Response rates for individual items were used as a proxy for acceptability.
Results
All SLE patients who responded to the self-administered GOAL survey (n=711) completed the SA-BILD. Figure 1 shows the subsets of GOAL respondents examined for the different validation assessments.
Figure 1.
Schematic diagram of GOAL participants included in the SA-BILD validation assessments.
The sociodemographic and health characteristics of the SA-BILD participants and the subset examined for criterion validity are depicted in Table 1. Among the 711 respondents to the SA-BILD, 94% were women and 78% were African American (Table 1). Thirty-five percent attained a high school or less than high school education, 45% reported an annual household income below the federal poverty threshold, and 43% were unemployed or disabled at the time of survey completion. Like the overall sample, the subset tested for criterion validity was predominantly represented by females (94%) and African Americans (88%). Forty-two percent attained a high school or less than high school education, 66.7% reported an annual household income below the federal poverty threshold and 54.7% were unemployed or disabled.
Table 1.
Description of SLE GOAL participants in the SA-BILD validation
| Characteristic | Self-administered BILD participants n=711 |
Criterion validity assessment subset n=150 |
|---|---|---|
| Sociodemographics | ||
| Age at survey (years), mean ±SD | 45.9 ± 13.4 | 42.4 ± 13.1 |
| Gender | ||
| Female | 666 (93.7) | 525 (93.6) |
| Ethnicity | ||
| African American | 554 (77.9) | 422 (75.2) |
| White | 143 (20.1) | 126 (22.5) |
| Other ethnicity | 14 (2.0) | 13 (2.3) |
| Educational Attainment | ||
| High school or less | 248 (35.0) | 185 (33.1) |
| Some college | 226 (31.9) | 171 (30.6) |
| College or higher | 234 (33.1) | 203 (36.3) |
| Household income below poverty level | 305 (45.3) | 209 (39.5) |
| Unemployed or disabled | 306 (43.0) | 224 (39.9) |
| Disease status | ||
| Disease duration (years), mean ±SD | 13.3 ± 9.0 | 14.1 ± 8.8 |
| Disease activity (SLAQ score), mean ±SD | 17.2 ± 9.4 | 17.0 ± 9.5 |
| Mental health (SF-12 MCS), mean ±SD | 43.8 ± 11.2 | 43.9 ± 11.5 |
| Physical health (SF-12 PCS), mean ±SD | 39.3 ± 10.4 | 39.6 ± 10.5 |
| Comorbidity Index (RDCI), mean ±SD | 2.9 ± 1.9 | 3.0 ± 2.0 |
| Health status | ||
| Fair/poor | 362 (51.0) | 276 (49.3) |
| Health Care Utilization, mean ±SD | ||
| Annual rheumatologist visits | 3.2 ± 2.9 | 3.0 ± 3.0 |
| Annual physician visits, mean ±SD | 9.6 ± 12.1 | 9.9 ± 13.0 |
Values are presented as n (%) unless otherwise specified. SLE= Systemic Lupus Erythematosus; GOAL= Georgians Organized Against Lupus; SLAQ= Systemic Lupus Activity Questionnaire; SF-12= Medical Outcomes Study Short Form 12; PCS= Physical Component Summary; MCS=Mental Component Summary; RDCI= Rheumatic Disease Comorbidity Index.
The mean score for self-reported disease activity was moderately high (>17) in both groups, the SA-BILD participants and the individuals assessed for criterion validity. Half of the participants from both groups reported fair or poor health status, with mental and physical component scores below the general population average. In both groups, the average number of annual visits to the rheumatologist and overall physicians was approximately 3 and 10, respectively.
Test-retest reliability was assessed among 32 GOAL respondents who were re-administered the SA-BILD questionnaire. The median time between the first and second assessment was 30.8 days (range 14–53). All these patients were female and their average age was 38 years; 56% were African Americans and 41% were Whites. The Spearman test-retest coefficient was 0.95 (p<0.0001).
Criterion validity was assessed among GOAL respondents for whom the SDI was completed during a regular lupus clinic visit. The mean time between the SA-BILD and the SDI assessments was 162 days (SD=75). Table 2 depicts the percentage agreement between each SDI item and the corresponding SA-BILD item. The observed item-by-item agreement ranged from 81% to 99%, while PABAK ranged from 0.73 to 0.99, except for retinal change or optic atrophy (PABAK 0.63) and thrombosis (PABAK 0.67). Table 3 shows that the distributions of the SDI and SA-BILD scores were similar, and the correlation of the overall score between both instruments was moderately high (rho=0.59, p <0.0001).
Table 2.
Item-by-item comparison of the SDI and SA-BILD in a subset of 150 SLE GOAL participants
| Organ System and Item | SDI n (%) |
SA-BILD n (%) |
SDI vsSA-BILD |
|
|---|---|---|---|---|
| Agreement (%) |
PABAK | |||
| Ocular | ||||
| Any cataract ever | 12 (8.0) | 20 (13.3) | 92 | 0.84 |
| Retinal change or optic atrophy | 2 (1.3) | 27 (18.1) | 81 | 0.63 |
| Neuropsychiatric | . | |||
| Cerebrovascular accident/resection | 6 (4.0) | 13 (8.7) | 91 | 0.81 |
| Cognitive impairment or psychosis | 7 (4.7) | 8 (5.3) | 91 | 0.83 |
| Cranial/peripheral neuropathy | 5 (3.3) | (omitted) | . | |
| Seizures requiring therapy 6+months | 8 (5.3) | 17 (11.3) | 94 | 0.88 |
| Transverse myelitis | 2 (1.3) | 7 (4.7) | 95 | 0.91 |
| Renal | . | |||
| ESRD (dialysis/transplant) | 5 (3.3) | 6 (4.0) | 98 | 0.96 |
| GFR <50 | 6 (4.0) | (omitted) | . | |
| Proteinuria >3.5g/24h | 3 (2.0) | (omitted) | . | |
| Pulmonary | . | |||
| Pleural fibrosis | 0 (−) | (omitted) | . | |
| Pulmonary fibrosis | 7 (4.7) | 8 (5.4) | 96 | 0.92 |
| Pulmonary hypertension | 4 (2.7) | 17 (11.4) | 87 | 0.73 |
| Pulmonary infarction or resection | 0 (−) | (omitted) | . | |
| Shrinking lung | 0 (−) | (omitted) | . | |
| Cardiovascular | . | |||
| Angina or coronary artery bypass | 3 (2.0) | 4 (2.7) | 99 | 0.97 |
| Cardiomyopathy | 4 (2.7) | (omitted) | . | |
| Myocardial infarction | 3 (2.0) | 11 (7.4) | 93 | 0.87 |
| Pericarditis 6+months | 1 (0.7) | 13 (8.8) | 89 | 0.77 |
| Valvular disease | 2 (1.3) | (omitted) | . | |
| Peripheral vascular | . | |||
| Claudication, 6+mos | 0 (−) | (omitted) | . | |
| Minor tissue loss (pulp space) | 5 (3.3) | 14 (9.3) | 90 | 0.80 |
| Significant tissue loss, ever | 0 (−) | 0 (−) | 99 | 0.99 |
| Venous thrombosis | 3 (2.0) | 24 (16.0) | 83 | 0.67 |
| Gastrointestinal | . | |||
| Chronic peritonitis | 0 (−) | 0 (−) | 98 | 0.96 |
| Infarction/ abdominal organ resection | 3 (2.0) | 8 (5.3) | 95 | 0.91 |
| Mesenteric insufficiency | 0 (−) | (omitted) | . | |
| Pancreatic insufficiency | 0 (−) | (omitted) | . | |
| Stricture or upper GI tract surgery | 1 (0.7) | 5 (3.3) | 96 | 0.92 |
| Musculoskeletal | . | |||
| Avascular necrosis | 4 (2.7) | 8 (5.3) | 93 | 0.87 |
| Deforming or erosive arthritis | 10 (6.7) | (omitted) | . | |
| Muscle atrophy or weakness | 11 (7.3) | (omitted) | . | |
| Osteomyelitis | 0 (−) | 1 (0.7) | 99 | 0.99 |
| Osteoporosis with fracture | 2 (1.3) | 15 (10.1) | 91 | 0.81 |
| Ruptured Tendon | 0 (−) | (omitted) | . | |
| Skin | . | |||
| Extensive scarring/panniculitis | 17 (11.3) | (omitted) | . | |
| Scarring chronic alopecia | 20 (13.3) | (omitted) | . | |
| Skin ulceration (not thrombosis) | 3 (2.0) | 15 (10.1) | 91 | 0.81 |
| Others | . | |||
| Diabetes | 9 (6.0) | 14 (9.3) | 93 | 0.85 |
| Malignancy | 4 (2.7) | 9 (6.0) | 97 | 0.93 |
| Premature gonadal failure (age <40) | 4 (4.7) | 10 (11.6) | 92 | 0.84 |
SDI= Systemic Lupus International Collaborating Clinics/American College of Rheumatology Damage Index; SA-BILD= Self-administered Brief Index of Lupus Damage; GOAL= Georgians Organized Against Lupus; PABAK = prevalence-adjusted bias-adjusted kappa.
Table 3.
SLE damage scores in a subset of 150 GOAL participants
| Score | Scores >0 N (%) |
Mean | Median | IQR | Maximum score |
Correlation |
|---|---|---|---|---|---|---|
| SDI (n=150) | 88 (58.7%) | 1.37 | 1 | 0–2 | 7 | - |
| SA-BILD (n=150) | 107 (71.3%) | 2.05 | 1 | 0–3 | 10 | 0.59 (p<0.0001) |
Abbreviations: IQR=interquartile range, SDI= SLICC/ACR Damage Index; SA-BILD= self-administered Brief Index of Lupus Damage (SA-BILD)
The distribution of the SA-BILD score in the overall GOAL sample of 711 patients was very similar to the sample used to determine criterion validity, with a median of 2.0 (IQR=0–3) and maximum score of 18 (data not shown). The acceptability of the SA-BILD questionnaire in the overall GOAL sample was high, with only 3 items having over 1% of missing values as a consequence of missing responses. These items were pericarditis (7 missing), chronic peritonitis (9 missing), and premature gonadal failure (7 missing).
Construct validity was assessed by analyzing the association of sociodemographic, disease status, health status, and health care utilization measures across quartiles of SA-BILD score (Table 4). As expected, higher SA-BILD scores, which represent progressively greater SLE damage, were associated with greater age and longer disease duration, as well as with progressively worse health scores (higher SLE activity index, lower physical health score, higher comorbidity index). As SA-BILD scores increased, the proportions of unemployed or disabled individuals and of patients reporting poor or fair health also increased. Similarly, the annual number of physician visits increased progressively as SA-BILD scores increased. We also found lower SA-BILD scores in women and in patients who attained < high school education. Ethnicity, poverty, mental health, and annual visits to the rheumatologist were not associated with increasing SA-BILD scores.
Table 4.
Comparison of demographic, health status and health utilization by quartiles of SA-BILD in SLE patients from the GOAL Cohort (N=711)
| Characteristic | Quartiles of SA-BILD Score | ||||
|---|---|---|---|---|---|
| 0 (n=192) |
1 (n=149) |
2–3 (n=205) |
≥4 (n=165) |
P value* | |
| Sociodemographics | |||||
| Age at survey (years), mean ± SD | 40.6 ± 12.6 | 44.7 ± 13.1 | 47.6 ± 13.2 | 51.2 ± 12.5 | <0.001 |
| Female | 185 (96.4) | 139 (93.3) | 193 (94.1) | 149 (90.3) | 0.036 |
| African American | 131 (70.1) | 127 (87.6) | 167 (82.7) | 129 (79.1) | 0.055 |
| High school education or less | 132 (69.5) | 103 (69.6) | 122 (59.5) | 103 (62.4) | 0.048 |
| Household income below poverty level | 75 (41.0) | 65 (45.1) | 92 (47.7) | 73 (47.7) | 0.17 |
| Unemployed or disabled | 53 (27.6) | 59 (39.6) | 97 (47.3) | 97 (58.8) | <0.001 |
| SLE disease status | |||||
| Disease duration (years), mean ± SD | 10.7 ± 8.2 | 12.5 ± 9.0 | 13.6 ± 8.7 | 16.7 ± 9.3 | <0.001 |
| SLAQ Score, mean ± SD | 13.0 ± 8.7 | 16.9 ± 9.2 | 18.7 ± 9.1 | 20.5 ± 8.9 | <0.001 |
| General health status | |||||
| Fair/poor self-reported health | 63 (32.8) | 73 (49.0) | 114 (55.9) | 112 (67.9) | <0.001 |
| SF-12 PCS, mean ± SD | 45.2 ± 10.4 | 39.8 ± 9.8 | 37.7 ± 9.3 | 34.0 ± 8.7 | <0.001 |
| SF-12 MCS, mean ± SD | 44.7 ± 11.1 | 43.9 ± 11.8 | 43.1 ± 11.4 | 43.3 ± 10.8 | 0.49 |
| RDCI, mean ± SD | 1.4 ± 1.3 | 2.5 ± 1.6 | 3.1 ± 1.6 | 4.7 ± 1.7 | <0.001 |
| Health care utilization | |||||
| Annual rheumatologist visits, mean ± SD | 2.9 ± 2.7 | 3.4 ± 2.7 | 3.4 ± 3.2 | 3.0 ± 3.1 | 0.28 |
| Annual physician visits, mean ± SD | 7.0 ± 5.9 | 8.6 ± 8.3 | 9.6 ± 9.3 | 13.8 ± 19.8 | <0.001 |
Values are presented as n (%) unless otherwise specified.
Calculated with one-way ANOVA test for continuous variables and Cochran–Armitage trend test for categorical variables.
SA-BILD = Self-administered Brief Index for Lupus Damage; SLE= Systemic Lupus Erythematosus; SLAQ= Systemic Lupus Activity Questionnaire; SF-12= Medical Outcomes Study Short Form 12; PCS= Physical Component Summary; MCS=Mental Component Summary; RDCI= Rheumatic Disease Comorbidity Index
To further examine the relationship between sociodemographic characteristics and damage, we constructed several multivariable models in which the outcome was the SA-BILD score (either as binary or continuous variables), and predictors included age, sex, ethnicity, disease duration, education and employment. All analyses yielded similar conclusions, with age, disease duration and unemployment emerging as significant predictors of damage.
Discussion
In this study, we validated a self-administered version of the Brief Index of Lupus Damage (BILD) in an independent community-based cohort of SLE patients from the Southeastern US. Questionnaire length and delivery mode have a significant impact on both survey acceptability and response rates (32), particularly when the target population includes vulnerable patients, such as the predominantly African American population in this study. We adapted the previously validated interviewer-administered BILD to a written survey that can be mailed to SLE patients, while retaining the psychometric properties and low administrative burden of the original tool. Our findings suggest that the SA-BILD is reliable and acceptable to SLE respondents with diverse sociodemographic backgrounds, and that it has criterion and construct validity comparable to the interviewer-administered BILD.
Like the original BILD, we found very good or good item-by-item agreement between the SA-BILD and the SDI. The PABAK surpassed 0.80 for most of the items. The only exceptions were retinal change or optic atrophy (PABAK= 0.63); venous thrombosis (PABAK=0.67); pulmonary hypertension (PABAK= 0.73) and pericarditis (PABAK=0.77). Interesting, all SA-BILD items were overreported by SLE patients, compared to rheumatologist assessment. Such patient overreporting has also been found with the Lupus Damage Index Questionnaire (LDIQ) (22), but not with the original BILD (23), suggesting that it may be associated with the self-administered method. Without the assistance of a trained interviewer, overreporting may be the result of patients’ unfamiliarity with the medical terminology used in the questions. Alternatively, physician underreporting can also explain patient-physician disagreement. As pointed out by the developers of the LDIQ, given the lack of universal medical records in the US health system, physicians might not be aware of all chronic manifestations accrued in individual patients since the disease onset (22). We do not believe that a delay between administration of the SA-BILD and the SDI impacted item-by-item agreement in relation to the occurrence of new damage manifestations. The average time between the patient and physician assessment was only 5 months, and the SDI was conducted after the SA-BILD in all cases. If new damage manifestations had occurred after the SA-BILD administration, we would expect the SA-BILD items to be under-reported in relation to the SDI. However, as discussed above, all discrepancies were based on more frequent patient reports (or physician underreporting)
We should emphasize that rather than developing a substitute for the SDI, the goal of the BILD instruments is to have a patient-reported measure that would differentiate between greater and lesser degrees of SLE damage with minimal administration burden to patients. Consequently, both the BILD and the SA-BILD include only 26 of the 56 items originally developed for the SDI. When we examined the correlation between the overall SA-BILD and SDI scores, we found moderately high Spearman’s rank correlation (rho=0.59, p<0.0001), which is consistent with findings previously reported with the original BILD (rho=0.64). We did not aim to compare the performances of the SA-BILD and the LDIQ, which is the 56-item patient-version of the SDI (22). However, it is noteworthy that despite the larger minority representation in our community-based cohort than in the academic-based sample used to validate the LDIQ, the overall correlation with the SDI seemed to be somewhat better for the SA-BILD (rho=0.59) than for the LDIQ (rho=0.48).
Data from over 700 SLE patients from the GOAL cohort also showed significant associations in the expected directions between SA-BILD scores and age, disease duration, socioeconomic and health status, as well as with disease outcomes and health service utilization measures. Consistent with prior studies using the SDI, the SLAQ, or the original BILD, increasing SA-BILD scores were positively associated with poorer scores of self-reported physical health, greater disease activity, greater comorbidity index and higher annual average of visits to physicians (16, 23, 27, 33–35)_ENREF_31_ENREF_31. Similarly, higher SA-BILD scores were associated with higher rates of unemployment or disability, and poor or fair overall health (21). Thus, our findings suggest that the SA-BILD may have a role in predicting long-term outcomes, although such longitudinal studies are yet to be completed. It was unexpected, however, that the number of annual visits to rheumatologists did not increase along with higher SA-BILD scores, as occurred with participants of the LOS cohort. Given the fact that there was a larger proportion of unemployment among GOAL participants with more severe damage, it is plausible that these patients may face barriers to specialized health care access and be monitored in the primary care setting.
A number of caveats need to be noted regarding the present study. First, the SA-BILD is not a substitute for the physician assessment of organ damage in the clinical setting; the gold standard instrument to assess organ damage for SLE outcomes research remains the physician-reported SDI, when it is feasible. Second, the SA-BILD questionnaire is only available in English, and further efforts are warranted to translate and validate the instrument for use among non-English speaking SLE patients. Finally, because our study was cross-sectional, we do not have insight on the value of the instrument for longitudinal studies. To establish the sensitivity to change, prospective cohort studies correlating physician and patient assessments are necessary.
The two-fold strengths of our study are the population-based nature and diverse sociodemographic characteristics of the targeted sample, which provided external legitimacy to this instrument as a self-administered measure of patient reported damage for epidemiological research in SLE. Survey respondents were predominantly African American and also represent socioeconomically disadvantaged subgroups that are most affected by SLE. Not surprisingly, SLE patients from minority groups accrue higher organ damage at earlier ages than Whites or those from more privileged social classes (9, 36, 37). However, minority groups are generally underrepresented in outcome measure development and validation studies (22, 23, 32, 38). The SA-BILD promises to be an economical alternative to capture organ damage for epidemiological studies in US communities with large representation of high-risk subjects.
Second, rather than developing a new instrument, this collaborative effort took advantage of an existing short instrument validated among SLE patients with a different sociodemographic profile. Despite the distinctive ethnic and socioeconomic characteristics of the Georgian Cohort, and the modifications needed to tailor the BILD to a written version, the SA-BILD showed moderately high criterion and construct validity performances comparable to the original interviewer-administered version _ENREF_23(23).
In conclusion, the SA-BILD was acceptable to SLE patients from a predominantly African American population who responded to a self-administered annual survey. Reliability, criterion validity and construct validity coefficients of the SA-BILD were within the range of previously validated patient-reported tools. Thus, the SA-BILD may be a practical and cost-effective option for collecting patient-reported damage for epidemiological SLE research when vulnerable SLE populations are targeted and physician assessment is not feasible. Further research is warranted, including longitudinal studies to assess sensitivity to change. Translation to other languages would also be desirable to determine the potential value of this instrument among other high-risk minorities with SLE.
Significance and Innovation.
Although having validated tools to measure patient-reported outcomes among vulnerable groups with SLE is essential to better understand the burden of the disease at the population level, disadvantaged SLE groups are typically underrepresented in measure development and validation research.
We validated a self-administered patient-reported tool to quantify organ damage in a unique population-based cohort of SLE patients from the Southeastern US, which includes a representative proportion of high-risk SLE individuals.
Our findings support the use of a novel cost-effective tool for collecting patient-reported damage for epidemiological SLE research when physician assessment is not feasible.
Acknowledgments
Human Genome Science Inc. and GlaxoSmithKline, Research Triangle, NC supported this research, study number GHO-11-3366.
J Yazdany, PP Katz, and L Trupin are supported by NIH/NIAMS grant P60 AR053308 and by the Rosalind Russell Medical Research Center for Arthritis at the University of California, San Francisco. J Yazdany is supported by the NIH/NIAMS grant K23 AR060259
Appendix A. Self-administered Brief Index for Lupus Damage (SA-BILD) as it is included in the GOAL survey
Table 5.
LUPUS-RELATED DAMAGE
| This survey collects information about symptoms you may have experienced related to your lupus. Don′t worry if there are some medical words you don′t understand. This usually means that you don′t have the problem the question is asking about | ||
|---|---|---|
| Eyes | Yes | No |
| 1. Has an eye doctor ever told you that you had something wrong with the retina of your eye because of your lupus? (The retina is the back of your eye) |
□1 | □0 |
| 2. Has a doctor ever told you that you had a cataract in your eye? | □1 | □0 |
| Brain: Has a doctor ever told you that you had any of the following symptoms? | Yes |
No |
| 3. A psychotic episode? | □1 | □0 |
| 4. Seizures? | □1 | □0 |
| If YES Seizures: Did you ever have to take medication for seizures for at least 6 months? |
□1 | □0 |
| 5. A stroke? (This does not include TIA or transient ischemic attack) | □1 | □0 |
| If YES stroke: Did you ever have more than 1 stroke at least 6 months apart? | □1 | □0 |
| 6. Paralysis in your arms or legs that was so severe that you needed to be hospitalized? (This is also known as transverse myelitis, a rare condition caused by inflammation of the spinal cord). |
□1 | □0 |
| If YES Paralysis: Was this paralysis from a stroke or multiple sclerosis? | □1 | □0 |
| Kidneys | Yes | No |
| 7. Have you ever had a kidney transplant? | □1 | □0 |
| 8. Have you ever been on dialysis for 6 months or longer? | □1 | □0 |
| Lungs: Has a doctor ever told you that you had any of the following conditions? | Yes | No |
| 9. Pulmonary hypertension, which is high blood pressure in the lungs? (This is different from regular hypertension or high blood pressure. The diagnosis starts with an echocardiogram or ultrasound of the heart, not with a blood pressure cuff) |
□1 | □0 |
| 10. A serious condition of your lungs, such as fibrosis or interstitial lung disease? (This does not include pneumonia, asthma, emphysema, pleurisy, COPD, or bronchitis) |
□1 | □0 |
| Heart | Yes | No |
| 11. Have you ever had coronary or heart bypass surgery? | □1 | □0 |
| 12. Have you ever been told by a doctor that you had heart disease, including angina or congestive heart failure? |
□1 | □0 |
| 13. Have you ever been told by a doctor that you had a heart attack? | □1 | □0 |
| If YES heart attack: Did you ever have more than 1 heart attack at least 6 months apart? |
□1 | □0 |
| 14. Have you ever been told by a doctor that you had an episode of pericarditis, which is an inflammation in the sack around the heart, that lasted 6 months or longer? |
□1 | □0 |
|
Blood Vessels: Because of your lupus, have you ever had any of the following symptoms? |
Yes |
No |
| 15. Loss of flesh or thinning on the ends of your fingers? | □1 | □0 |
| 16. Loss of a finger, toe, or part of an arm or leg not due to an accident? | □1 | □0 |
| 17. Deep vein thrombosis -- DVT -- or a blood clot in your arm or leg? | □1 | □0 |
|
Stomach and Bowels 18. Because of your lupus, have you ever had abdominal surgery of your: |
Yes | No |
| Esophagus | □1 | □0 |
| Stomach | □1 | □0 |
| Small Intestine | □1 | □0 |
| Large intestine/colon | □1 | □0 |
| Spleen | □1 | □0 |
| Liver | □1 | □0 |
| Pancreas | □1 | □0 |
| Gall bladder | □1 | □0 |
| Other (e.g. kidney, appendix, uterus or reproductive organs). __________________________________ |
□1 | □0 |
| 19. Has a doctor ever told you that you had peritonitis that lasted 6 months or longer? (Peritonitis is an inflammation of the lining of your abdomen) |
□1 | □0 |
| Muscles and Bones: Has a doctor ever told you that you had? | Yes | No |
| 20. Osteoporosis, or thin bones, that resulted in a fracture? | □1 | □0 |
| 21. Avascular necrosis? (This is when part of a bone dies) | □1 | □0 |
| 22. Osteomyelitis? (This is an infection in a bone) | □1 | □0 |
| Skin: Has a doctor ever told you that you had? | Yes | No |
| 23. A skin ulcer, which is an open sore on your skin, that lasted 6 months or longer? (Note: this is not an oral ulcer or a ‘cold sore’) |
□1 | □0 |
|
Diabetes |
Yes | No |
| 24. Has a doctor ever told you that you had diabetes? | □1 | □0 |
|
Cancer |
Yes | No |
| 25. Has a doctor ever told you that you had cancer? If Yes, what kind of cancer? (List all) __________________________________________________________ |
□1 | □0 |
|
Menopause Questions 26.a and 26.b are for WOMEN ONLY |
Yes | No |
| 26.a This question is only if you now are 40 or older: Did your menstrual periods stop before you turned 40? |
□1 | □0 |
| If Yes, was this due to a hysterectomy? | □1 | □0 |
| 26.b This question is only if you now are younger than 40: Do you still get your menstrual periods? |
□1 | □0 |
| Are you pregnant or nursing? | □1 | □0 |
Footnotes
Disclosures: ther commercial relationships
Author Contributions
CD, JY, LT, CDT and SSL contributed to the study design. CD, JY, LT, PPK, CDT, GB, SSL contributed to acquisition of data, statistical analysis and interpretation of data. CD, JY, LT, PPK, CDT, GB, SSL contributed to manuscript preparation.
References
- 1.Gladman DD, Goldsmith CH, Urowitz MB, Bacon P, Fortin P, Ginzler E, et al. The Systemic Lupus International Collaborating Clinics/American College of Rheumatology (SLICC/ACR) Damage Index for Systemic Lupus Erythematosus International Comparison. J Rheumatol. 2000;27:373–376. [PubMed] [Google Scholar]
- 2.Haque S, Gordon C, Isenberg D, Rahman A, Lanyon P, Bell A, et al. Risk factors for clinical coronary heart disease in systemic lupus erythematosus: the lupus and atherosclerosis evaluation of risk (LASER) study. J Rheumatol. 2010;37:322–329. doi: 10.3899/jrheum.090306. [DOI] [PubMed] [Google Scholar]
- 3.Calvo-Alen J, McGwin G, Toloza S, Fernandez M, Roseman JM, Bastian HM, et al. Systemic lupus erythematosus in a multiethnic US cohort (LUMINA): XXIV. Cytotoxic treatment is an additional risk factor for the development of symptomatic osteonecrosis in lupus patients: results of a nested matched case-control study. Ann Rheum Dis. 2006;65:785–790. doi: 10.1136/ard.2005.040428. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Zonana-Nacach A, Yanez P, Jimenez-Balderas FJ, Camargo-Coronel A. Disease activity, damage and survival in Mexican patients with acute severe systemic lupus erythematosus. Lupus. 2007;16:997–1000. doi: 10.1177/0961203307083175. [DOI] [PubMed] [Google Scholar]
- 5.Nived O, Jonsen A, Bengtsson AA, Bengtsson C, Sturfelt G. High predictive value of the Systemic Lupus International Collaborating Clinics/American College of Rheumatology damage index for survival in systemic lupus erythematosus. J Rheumatol. 2002;29:1398–1400. [PubMed] [Google Scholar]
- 6.Cardoso CR, Signorelli FV, Papi JA, Salles GF. Initial and accrued damage as predictors of mortality in Brazilian patients with systemic lupus erythematosus: a cohort study. Lupus. 2008;17:1042–1048. doi: 10.1177/0961203308093829. [DOI] [PubMed] [Google Scholar]
- 7.Chambers SA, Allen E, Rahman A, Isenberg D. Damage and mortality in a group of British patients with systemic lupus erythematosus followed up for over 10 years. Rheumatology (Oxford) 2009;48:673–675. doi: 10.1093/rheumatology/kep062. [DOI] [PubMed] [Google Scholar]
- 8.Rahman P, Gladman DD, Urowitz MB, Hallett D, Tam LS. Early damage as measured by the SLICC/ACR damage index is a predictor of mortality in systemic lupus erythematosus. Lupus. 2001;10:93–96. doi: 10.1191/096120301670679959. [DOI] [PubMed] [Google Scholar]
- 9.Alarcon GS, McGwin G, Jr, Bartolucci AA, Roseman J, Lisse J, Fessler BJ, et al. Systemic lupus erythematosus in three ethnic groups. IX. Differences in damage accrual. Arthritis Rheum. 2001;44:2797–2806. doi: 10.1002/1529-0131(200112)44:12<2797::aid-art467>3.0.co;2-9. [DOI] [PubMed] [Google Scholar]
- 10.Mok CC, Ho CT, Wong RW, Lau CS. Damage accrual in southern Chinese patients with systemic lupus erythematosus. J Rheumatol. 2003;30:1513–1519. [PubMed] [Google Scholar]
- 11.Gladman D, Ginzler E, Goldsmith C, Fortin P, Liang M, Urowitz M, et al. Systemic lupus international collaborative clinics: development of a damage index in systemic lupus erythematosus. J Rheumatol. 1992;19:1820–1821. [PubMed] [Google Scholar]
- 12.Gladman D, Ginzler E, Goldsmith C, Fortin P, Liang M, Urowitz M, et al. The development and initial validation of the Systemic Lupus International Collaborating Clinics/American College of Rheumatology damage index for systemic lupus erythematosus. Arthritis Rheum. 1996;39:363–369. doi: 10.1002/art.1780390303. [DOI] [PubMed] [Google Scholar]
- 13.Gladman DD, Urowitz MB, Goldsmith CH, Fortin P, Ginzler E, Gordon C, et al. The reliability of the Systemic Lupus International Collaborating Clinics/American College of Rheumatology Damage Index in patients with systemic lupus erythematosus. Arthritis Rheum. 1997;40:809–813. doi: 10.1002/art.1780400506. [DOI] [PubMed] [Google Scholar]
- 14.Alarcon GS, McGwin G, Jr, Bastian HM, Roseman J, Lisse J, Fessler BJ, et al. Systemic lupus erythematosus in three ethnic groups. VII [correction of VIII]. Predictors of early mortality in the LUMINA cohort. LUMINA Study Group. Arthritis Rheum. 2001;45:191–202. doi: 10.1002/1529-0131(200104)45:2<191::AID-ANR173>3.0.CO;2-2. [DOI] [PubMed] [Google Scholar]
- 15.Mok CC, Ho LY, Cheung MY, Yu KL, To CH. Effect of disease activity and damage on quality of life in patients with systemic lupus erythematosus: a 2-year prospective study. Scand J Rheumatol. 2009;38:121–127. doi: 10.1080/03009740802415527. [DOI] [PubMed] [Google Scholar]
- 16.Wang C, Mayo NE, Fortin PR. The relationship between health related quality of life and disease activity and damage in systemic lupus erythematosus. J Rheumatol. 2001;28:525–532. [PubMed] [Google Scholar]
- 17.Baker K, Pope J, Fortin P, Silverman E, Peschken C. Work disability in systemic lupus erythematosus is prevalent and associated with socio-demographic and disease related factors. Lupus. 2009;18:1281–1288. doi: 10.1177/0961203309345784. [DOI] [PubMed] [Google Scholar]
- 18.Bertoli AM, Fernandez M, Alarcon GS, Vila LM, Reveille JD. Systemic lupus erythematosus in a multiethnic US cohort LUMINA (XLI): factors predictive of self-reported work disability. Ann Rheum Dis. 2007;66:12–17. doi: 10.1136/ard.2006.055343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Clarke AE, Petri M, Manzi S, Isenberg DA, Gordon C, Senecal JL, et al. The systemic lupus erythematosus Tri-nation Study: absence of a link between health resource use and health outcome. Rheumatology (Oxford) 2004;43:1016–10124. doi: 10.1093/rheumatology/keh229. [DOI] [PubMed] [Google Scholar]
- 20.Sutcliffe N, Clarke AE, Taylor R, Frost C, Isenberg DA. Total costs and predictors of costs in patients with systemic lupus erythematosus. Rheumatology (Oxford) 2001;40:37–47. doi: 10.1093/rheumatology/40.1.37. [DOI] [PubMed] [Google Scholar]
- 21.Bultink IE, Turkstra F, Dijkmans BA, Voskuyl AE. High prevalence of unemployment in patients with systemic lupus erythematosus: association with organ damage and health-related quality of life. J Rheumatol. 2008;35:1053–1057. [PubMed] [Google Scholar]
- 22.Costenbader KH, Khamashta M, Ruiz-Garcia S, Perez-Rodriguez MT, Petri M, Elliott J, et al. Development and initial validation of a self-assessed lupus organ damage instrument. Arthritis Care Res (Hoboken) 2010;62:559–568. doi: 10.1002/acr.20193. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Yazdany J, Trupin L, Gansky SA, Dall'era M, Yelin EH, Criswell LA, et al. Brief index of lupus damage: a patient-reported measure of damage in systemic lupus erythematosus. Arthritis Care Res (Hoboken) 2011;63:1170–1177. doi: 10.1002/acr.20503. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Drenkard C, Rask KJ, Easley KA, Bao G, Lim SS. Primary preventive services in patients with systemic lupus erythematosus: Study from a population-based sample in Southeast U.S. Semin Arthritis Rheum. 2013 doi: 10.1016/j.semarthrit.2013.04.003. E-pub ahead of print. [DOI] [PubMed] [Google Scholar]
- 25.Lim SS, Bayakly RA, Helmick CG, Gordon C, Easley K, Drenkard C. The Incidence and Prevalence of Systemic Lupus Erythematosus, 2002–2004: The Georgia Lupus Registry. Arthritis Rheum. 2013 doi: 10.1002/art.38239. in press. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol. 1993;46:423–429. doi: 10.1016/0895-4356(93)90018-v. [DOI] [PubMed] [Google Scholar]
- 27.Karlson EW, Daltroy LH, Rivest C, Ramsey-Goldman R, Wright EA, Partridge AJ, et al. Validation of a Systemic Lupus Activity Questionnaire (SLAQ) for population studies. Lupus. 2003;12:280–286. doi: 10.1191/0961203303lu332oa. [DOI] [PubMed] [Google Scholar]
- 28.Hurst NP, Ruta DA, Kind P. Comparison of the MOS short form-12 (SF12) health status questionnaire with the SF36 in patients with rheumatoid arthritis. British journal of rheumatology. 1998;37:862–869. doi: 10.1093/rheumatology/37.8.862. [DOI] [PubMed] [Google Scholar]
- 29.Wolfe F, Michaud K, Li T, Katz RS. Chronic conditions and health problems in rheumatic diseases: comparisons with rheumatoid arthritis, noninflammatory rheumatic disorders, systemic lupus erythematosus, and fibromyalgia. J Rheumatol. 2010;37:305–315. doi: 10.3899/jrheum.090781. [DOI] [PubMed] [Google Scholar]
- 30.Cochran W. Some methods for strengthening the common tests. Biometrics. 1954;10:417–451. [Google Scholar]
- 31.Armitage P. Tests for linear trends in proportions and frequencies. Biometrics. 1955;11:375–386. [Google Scholar]
- 32.Baker K, Pope J, Fortin P, Silverman E, Peschken C. Work disability in systemic lupus erythematosus is prevalent and associated with socio-demographic and disease related factors. 2009;18:1281–1288. doi: 10.1177/0961203309345784. [DOI] [PubMed] [Google Scholar]
- 33.Rivest C, Lew RA, Welsing PM, Sangha O, Wright EA, Roberts WN, et al. Association between clinical factors, socioeconomic status, and organ damage in recent onset systemic lupus erythematosus. J Rheumatol. 2000;27:680–684. [PubMed] [Google Scholar]
- 34.Fortin PR, Abrahamowicz M, Neville C, du Berger R, Fraenkel L, Clarke AE, et al. Impact of disease activity and cumulative damage on the health of lupus patients. Lupus. 1998;7:101–107. doi: 10.1191/096120398678919813. [DOI] [PubMed] [Google Scholar]
- 35.Vila LM, Alarcon GS, McGwin G, Jr, Friedman AW, Baethge BA, Bastian HM, et al. Early clinical manifestations, disease activity and damage of systemic lupus erythematosus among two distinct US Hispanic subpopulations. Rheumatology (Oxford) 2004;43:358–3563. doi: 10.1093/rheumatology/keh048. [DOI] [PubMed] [Google Scholar]
- 36.Cooper GS, Treadwell EL, St Clair EW, Gilkeson GS, Dooley MA. Sociodemographic associations with early disease damage in patients with systemic lupus erythematosus. Arthritis Rheum. 2007;57:993–999. doi: 10.1002/art.22894. [DOI] [PubMed] [Google Scholar]
- 37.Toloza SM, Roseman JM, Alarcon GS, McGwin G, Jr, Uribe AG, Fessler BJ, et al. Systemic lupus erythematosus in a multiethnic US cohort (LUMINA): XXII. Predictors of time to the occurrence of initial damage. Arthritis Rheum. 2004;50:3177–3186. doi: 10.1002/art.20578. [DOI] [PubMed] [Google Scholar]
- 38.Yazdany J, Yelin EH, Panopalis P, Trupin L, Julian L, Katz PP. Validation of the systemic lupus erythematosus activity questionnaire in a large observational cohort. Arthritis Rheum. 2008;59:136–143. doi: 10.1002/art.23238. [DOI] [PMC free article] [PubMed] [Google Scholar]