Abstract
Background
The World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) is a widespread measure of disability and functional impairment, which is bundled with the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) for use in psychiatry. Administering psychometric scales via the Internet is an effective way to reach respondents and allow for convenient handling of data.
Objective
The aim was to study the psychometric properties of the 12-item self-report WHODAS 2.0 when administered online to individuals with anxiety and stress disorders. The WHODAS 2.0 was hypothesized to exhibit high internal consistency and be unidimensional. We also expected the WHODAS 2.0 to show high 2-week test-retest reliability, convergent validity (correlations approximately .50 to .90 with other self-report measures of functional impairment), that it would differentiate between patients with and without exhaustion disorder, and that it would respond to change in primary symptom domain.
Methods
We administered the 12-item self-report WHODAS 2.0 online to patients with anxiety and stress disorders (N=160) enrolled in clinical trials of cognitive behavior therapy, and analyzed psychometric properties within a classical test theory framework. Scores were compared with well-established symptom and disability measures, and sensitivity to change was studied from pretreatment to posttreatment assessment.
Results
The 12-item self-report WHODAS 2.0 showed high internal consistency (Cronbach alpha=.83-.92), high 2-week test-retest reliability (intraclass correlation coefficient=.83), adequate construct validity, and was sensitive to change. We found preliminary evidence for a three-factorial structure, but one strong factor accounted for a clear majority of the variance.
Conclusions
We conclude that the 12-item self-report WHODAS 2.0 is a psychometrically sound instrument when administered online to individuals with anxiety and stress disorders, but that it is probably fruitful to also report the three subfactors to facilitate comparisons between studies.
Trial Registration
Clinicaltrials.gov NCT02540317; https://clinicaltrials.gov/ct2/show/NCT02540317 (Archived by WebCite at http://www.webcitation.org/6vQEdYAem); Clinicaltrials.gov NCT02314065; https://clinicaltrials.gov/ct2/show/NCT02314065 (Archived by WebCite at http://www.webcitation.org/6vQEjlUU8)
Keywords: disability, Internet, psychometrics, questionnaire, validity, WHODAS
Introduction
The World Health Organization Disability Assessment Schedule 2.0 (WHODAS 2.0) [1,2] is an assessment tool developed by the World Health Organization (WHO) to measure disability and functional impairment in accordance with the International Classification of Functioning, Disability and Health [3]. The WHODAS 2.0 comes bundled with the Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition; DSM-5), and is endorsed as a new and useful measure of functional impairment in psychiatric disorders [4]. The WHODAS 2.0 measures average functioning in everyday situations for the last 30 days, and surveys six domains of functioning: (1) cognition (understanding and communicating), (2) mobility (ability to move and get around), (3) self-care (eg, with regard to hygiene, dressing, and eating) (4) getting along with others, (5) life activities (ability to attend to everyday responsibilities), and (6) participation in society [1]. The most widespread and evaluated form of the WHODAS 2.0 is the 36-item structured interview version, which takes approximately 20 minutes to complete and has excellent psychometric properties [2]. This study, however, concerns the shorter 12-item self-assessment questionnaire version of the WHODAS 2.0. In the WHODAS 2.0 field trials, the reduced 12-item scale, despite only taking approximately 5 minutes to complete, explained 81% of the variance in the 36-item scale [2]. Several large-scale studies have suggested that the 12-item WHODAS 2.0 is a reliable and valid instrument when administered as an interview [5,6] or in a pencil-and-paper format [7,8]. As to dimensionality, both a one-factor structure [6,8] and a second-order model that specifies the six WHODAS 2.0 domains of functioning as subfactors to an overarching disability variable [5,6] have been suggested.
Administering psychometric questionnaires via the Internet is rapidly becoming more common in both research and routine mental care. Compared with conventional pencil-and-paper administration, there are many advantages of this online approach. Respondents can complete the necessary questionnaires wherever an Internet connection is available, and for the clinician or researcher data are quickly and easily stored, scored, analyzed, and interpreted with less risk for human error. Questionnaires are easily integrated with routine care software for evaluation and record keeping, as well as digital monitoring systems and Web-based psychological treatments. Loss of individual item scores may be prevented entirely, time of measurement may be registered and determined by prespecified time schedules, and respondents may be readily contacted via automatic email or text-message reminders. Although it has often been found that well-established scales do well regardless of administration format, online adaptions of validated scales should preferably undergo separate validation [9,10]. There is thus a need for separate validation of the self-rated 12-item WHODAS 2.0 when administered via the Internet and, to the knowledge of the authors, no study has yet investigated the convergent or discriminant validity, responsiveness, test-retest reliability, or factor structure of the self-rated 12-item WHODAS 2.0 when administered online to individuals with common mental disorders.
Based on data from two clinical trials of cognitive behavior therapy (CBT) for anxiety and stress disorders, we aimed to present estimates of test-retest reliability and thoroughly investigate item score distributions, convergent and discriminant validity, as well as the factor structure of the 12-item online WHODAS 2.0 when administered to individuals with anxiety and stress disorders. We expected the scale to be unidimensional, possibly with the six domains of functioning as subfactors (see previous), and with high internal consistency (Cronbach alpha>.80) as seen in previous studies. We expected strong baseline Pearson correlations (approximately .50 to .90) between the WHODAS 2.0 and other measure of disability or functional impairment, as well as substantial, yet slightly weaker, baseline Pearson correlations (approximately .30 to .70) with the measures of depression and general anxiety. We expected the WHODAS 2.0 to discriminate well between chronic stress patients with and without International Classification of Diseases, Tenth Revision (ICD-10) exhaustion disorder, a disorder characterized by burnout-like symptoms including fatigue and cognitive weariness, and typically regarded as highly disabling [11]. We also hypothesized that the WHODAS 2.0 would be sensitive to within-group change in primary psychiatric symptom domain.
Methods
Design
This was a psychometric study of the WHODAS 2.0 administered online to patients with anxiety and stress disorders. Data were collected from clinical trials of CBT for severe health anxiety (n=60) and stress disorders (n=100) conducted at Karolinska Institutet and Gustavsberg primary care clinic, Stockholm, Sweden. Both trials were approved by the Stockholm regional ethics review board (2015/415-31/5, 2014/1530-31/2), registered at ClinicalTrials.gov (NCT02540317, NCT02314065), and participants provided informed consent. Data used for this study were collected between September 2015 and July 2016.
Recruitment
Both clinical trials employed patient self-referral via the Internet, and advertised in newspapers as well as on online social media networks. Study applicants completed a series of online screening symptom questionnaires before a diagnostic interview with a licensed psychologist. This interview primarily served to survey eligibility criteria and lead up to a decision regarding inclusion or exclusion (ie, this decision was based on the psychiatric interview), but also served to collect important clinical data (eg, comorbid diagnoses). After the pretreatment assessment, which was conducted online, patients underwent randomization and subsequent treatment. All included patients were at least 18 years of age. The severe health anxiety sample had a principal diagnosis of DSM-5 somatic symptom disorder or illness anxiety disorder, whereas the stress disorders sample had a principal diagnosis of DSM-5 adjustment disorder or ICD-10 exhaustion disorder (for a brief introduction to this disorder, see [11]). We intend to provide an in-depth description of the methods of the clinical trials, including the recruitment process, in the primary publications.
Procedure
All questionnaires were completed through a simple Web-based interface with white background, radio buttons, and checkboxes. All 60 patients in the severe health anxiety sample and 50 patients in the stress disorders trial received CBT (12 weeks, disorder-specific) for their principal disorder. The WHODAS 2.0 was administered before and after CBT. In addition, patients with severe health anxiety completed the WHODAS 2.0 at screening, thus allowing for estimates of test-retest reliability. Other measures used to validate the WHODAS 2.0 were also administered before and after treatment.
Clinical Instruments for Diagnostic Assessment
Both DSM-5 somatic symptom disorder and illness anxiety disorder were assessed with the Health Preoccupation Diagnostic Interview, which exhibits excellent interrater reliability [12]. Both ICD-10 exhaustion disorder and DSM-5 adjustment disorder were assessed with a clinical interview developed specifically for the stress disorders trial that closely followed the diagnostic criteria of ICD-10 and DSM-5. Comorbid psychiatric disorders were surveyed with the Mini-International Neuropsychiatric Interview, which is a reliable and valid instrument for assessing psychiatric disorders [13].
Self-Rated Measures of Functional Impairment
The self-report 12-item WHODAS 2.0 [2] instructs the respondent to determine his or her difficulty in engaging in particular activities (eg, “taking care of [...] household responsibilities” and “maintaining a friendship”), as rated on a scale from “none” (no difficulty) to “extreme or cannot do” and corresponding to six domains of functioning (see Introduction). We employed the WHO simple scoring method [1] that gives a 12-item WHODAS 2.0 score range from 12 to 60, where higher scores indicate higher disability or loss of function. This type of straightforward additive scoring has been shown to correlate strongly (r>.98, ρ=.999) with more complex scoring methods incorporating weights based on item response patterns [5,14]. Just like the pencil-and-paper version, the WHODAS 2.0 was presented over two pages with items 1 to 5 on the first page, and items 6 to 12 on the second page.
The Sheehan Disability Scale (SDS) is a well-established three-item measure of psychiatric symptom-related functional impairment with a sum score range from 0 to 30, with a higher score indicating a higher degree of functional impairment [15,16]. The Work Ability Index (WAI) measures work ability with a sum score range from 7 to 49, and higher score indicating higher work ability [17-19]. The SDS and WAI were used as indexes of functional impairment.
Self-Rated Measures of Primary Psychiatric Symptoms
The Health Anxiety Inventory (HAI) is a 64-item questionnaire that measures health anxiety on a scale from 0 to 192, with a higher score indicating more health anxiety [20-22]. The Perceived Stress Scale (PSS) is a 14-item measure of perceived stress with a range from 0 to 56, with a higher score indicating more stress [23]. The HAI and PSS were used to describe the samples in terms of primary symptom domains (ie, health anxiety and stress).
Self-Rated Measures of General Psychiatric Symptoms
The self-reported Montgomery-Åsberg Depression Rating Scale (MADRS-S) is a widely used nine-item questionnaire that measures depressive symptoms on a scale from 0 to 54, with higher scores indicating more symptoms of depression [24-26]. The seven-item Generalized Anxiety Disorder scale (GAD-7) measures general anxiety from 0 to 21, with higher scores indicating more general anxiety [27]. The MADRS-S and GAD-7 were used to assess the common symptom domains of depression and general anxiety and facilitate comparison of the two samples.
Statistical Analysis
Analyses were done in SPSS version 23.0.0.2 (IBM Corp, Armonk, NY, USA) and R 3.3.2 [28] with lavaan 0.5-22 [29]. We pooled the severe health anxiety and stress disorders samples (total N=160), dropped three multivariate outliers, and employed structural equation modeling to assess the validity of the simple one-factor model of disability endorsed by the WHO [1], as well as the second-order model fitted in two previous studies [5,6]. The latter had latent variables corresponding both to the six WHODAS 2.0 dimensions of functioning and an overarching latent disability variable. We employed weighted least squares means and variance adjusted estimation, which is adequate for categorical data and nonnormal manifest variables [30]. Based on the recommendations of Byrne [31], we established a priori criteria for acceptable model fit in terms of a comparative fit index (CFI) greater than 0.90, a Tucker-Lewis index (TLI) greater than 0.90, and a root mean square error of approximation (RMSEA) of 0.08 or lower. Post hoc exploratory factor analysis was based on principal axis factoring with promax rotation (ie, factors were assumed to be correlated).
Internal consistency was investigated in terms of Cronbach alpha, complemented by adjusted item-total correlations (ITCs), which are not as strongly affected by the number of scale items. For instruments of typical length, Cronbach alpha≥.9 is usually regarded as excellent, ≥8 as good, and ≥7 as acceptable. Test-retest reliability was estimated based on a two-way mixed-effects model absolute agreement intraclass correlation coefficient (ICC) and data from a subsample (n=25) from the severe health anxiety trial that had completed the screening and pretreatment assessments within 14 days (mean 6.8, SD 3.2, range 1-13).
We used an independent samples t test to assess if patients with and without a clinical diagnosis of exhaustion disorder differed with regard to WHODAS 2.0 score, and then performed a receiver operating characteristic analysis to assess the ability (area under the curve [AUC]) of the WHODAS 2.0 to identify cases of exhaustion disorder. Pearson correlations were used to investigate baseline associations between the WHODAS 2.0 and other self-rated measures.
To evaluate sensitivity to change, we compared pretreatment and posttreatment mean scores using paired samples t test and calculated effect sizes as the difference in means at t1 and t2, divided by the t1 standard deviation. Based on the Jacobson and Truax reliable change index [32], we classified patients as either reliably improved (reduction in the HAI>29.9 or reduction in the PSS>7.1) or not reliably improved. As recommended in the assessment of responsiveness, we also further differentiated between reliably improved patients based on whether they improved more than a minimal important difference in their primary symptom domain [33]. In other words, for both samples we estimated change in functional impairment in three strata: (1) patients who did not made a reliable improvement on the HAI or PSS, (2) patients who reliably improved but only to a slight to moderate degree, and (3) patients who reliably improved to a large degree. However, it was necessary to set the threshold for changing more than a minimal important difference relatively high (1.5 SD) due to the precision of the instruments requiring a relatively large score change (1.21 SD in the case of HAI, 1 SD in the case of PSS) for patients to have experienced true change with 95% certainty, as assessed with the reliable change index [32]. Item-level change was assessed with the Wilcoxon signed-rank test.
The WHODAS 2.0 response rate was 100% for the screening and pretreatment assessments. Because there was a small proportion of missing posttreatment data (for the WHODAS 2.0: 3%, 3/100 in the stress sample; 5%, 3/60 in the severe health anxiety sample), and we did not aim to investigate treatment effects but rather the responsiveness of the scale, data were used on a complete case (not intention-to-treat) basis.
Results
Sample Characteristics
Sample characteristics for the severe health anxiety sample, the stress disorders sample, and the pooled sample are presented in Table 1.
Table 1.
Sample characteristics.
| Measure | Severe health anxiety (n=60) | Stress disorders (n=100) | Total (N=160) | ||
| Demographics | |||||
| Age (years), mean (SDa), range | 36.4 (11.9), 18-78 | 46.2 (8.8), 26-65 | 42.5 (11.1), 18-78 | ||
| Gender (female), n (%) | 40 (67) | 85 (85) | 125 (78) | ||
| Psychiatric symptomsb | |||||
| HAI, mean (SD), range | 105.6 (24.7), 51-164 | — | — | ||
| PSS, mean (SD), range | — | 36.8 (7.1), 17-52 | — | ||
| MADRS-S, mean (SD), range | 13.9 (7.4), 1-34 | 19.7 (7.5), 3-40 | 17.5 (7.9), 1-40 | ||
| GAD-7, mean (SD), range | 12.0 (5.3), 2-21c | 10.8 (4.8), 2-21 | 11.2 (5.0), 2-21c | ||
| MDD, n (%) | 11 (18) | 13 (13) | 24 (15) | ||
| ≥1 anxiety disorder/OCDd, n (%) | 34 (57) | 19 (19) | 53 (33) | ||
| Functional impairmente | |||||
| WHODAS 2.0, mean (SD), range | 21.1 (6.3), 12-36 | 24.7 (8.5), 12-51 | 23.4 (7.9), 12-51 | ||
| SDS, mean (SD), range | 10.4 (6.7), 0-26 | — | — | ||
| WAI, mean (SD), range | — | 33.0 (8.4), 13.0-47.0 | — | ||
| On sick leave, n (%) | 3 (5) | 14 (14) | 17 (11) | ||
aSD: standard deviation.
bHAI: Health Anxiety Inventory; PSS: Perceived Stress Scale; MADRS-S: Montgomery-Åsberg Depression Rating Scale self-report version; GAD-7: Generalized Anxiety Disorder 7-item scale; MDD: major depressive disorder; OCD: obsessive compulsive disorder;.
cGAD-7 data only available from a subsample of the severe health anxiety sample (n=43).
dAt least one anxiety or obsessive compulsive disorder that is not severe health anxiety.
eWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0; SDS: Sheehan Disability Scale; WAI: Work Ability Index.
Factor Structure
Neither the one-factor model endorsed by the WHO (χ254=1699.1, P<.001; CFI=0.29, TLI=0.14, RMSEA=0.44, standardized root mean square residual [SRMR]=0.52) nor the second-order model presented by previous studies achieved acceptable fit (χ248=132.7, P<.001; CFI=0.96, TLI=0.95, RMSEA=0.11, SRMR=0.09), and we did not see implementing any of the modification indexes as theoretically justifiable.
Post hoc exploratory factor analysis (Kaiser-Meyer-Olkin test=0.85, Barlett test<.001) was suggestive of a three-factor solution, with one very strong factor (eigenvalue=5.4, 45.0% of variance explained) and two weak factors (eigenvalues=1.1-1.7, 9.3%-14.2% of variance explained).
After rotation (Table 2) it was clear that one of the two weak factors was primarily associated with items 1 and 7 (“getting around”), and the other weak factor was primarily associated with items 8 and 9 (“self-care”), whereas the strong factor was associated with the remaining items (“understanding and communicating,” “getting along with others,” “life activities,” and “participation in society”). Interfactor correlations were substantial (r=.40-.46). Dropping items 1, 7, 8, and 9 had little impact on interindividual sum score variance, and the resulting mean was heavily correlated with the conventional one (r=.97). Thus, the factor analysis indicated that a majority of items had high loadings on a general disability factor, but that “mobility” and “self-care” emerged as distinct, but weak, factors.
Table 2.
Factor loadings of the WHODAS 2.0a.
|
Item |
Factor 1: psychosocialb | Factor 2: self-careb | Factor 3: mobilityb | A priori dimensionc |
| 1. Standing long periods | –.008 | –.097 | .877 | Mobility |
| 2. Household responsibilities | .540 | –.027 | .264 | Household |
| 3. Learning new tasks | .740 | –.002 | .035 | Cognitive |
| 4. Joining community activities | .848 | –.046 | .042 | Society |
| 5. Emotionally affected | .652 | –.087 | .074 | Society |
| 6. Concentrating | .683 | –.010 | –.040 | Cognitive |
| 7. Walking long distance | –.003 | .329 | .597 | Mobility |
| 8. Washing whole body | –.068 | .893 | .132 | Self-care |
| 9. Getting dressed | .078 | .909 | –.126 | Self-care |
| 10. Dealing with strangers | .704 | .167 | –.164 | Social |
| 11. Maintaining friendships | .618 | .128 | –.101 | Social |
| 12. Work/school activities | .814 | –.059 | .037 | Household |
aWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.
bFactor loadings (regression coefficients) based on principal axis factoring with promax rotation.
cCognitive: understanding and communicating; household: life activities; mobility: getting around; social: getting along with others; society: participation in society.
Item Score Distributions
Parameters related to item score distributions are presented in Table 3. Overall, patients scored low on items that concerned functional impairment in self-care, and higher on items that concerned difficulties participating in society and everyday life activities, as well as items that concerned cognitive impairment. Unlike the summary score, several item distributions were skewed and showed high kurtosis. There was also evidence of a floor effect with regard to several items.
Table 3.
Web-based self-report 12-item WHODAS 2.0 item scoresa.
|
Item and subscale |
Mean (SDb) |
Median (range) |
Floor |
Ceiling |
Skew |
Kurtosis |
ITCc | |
| Item | ||||||||
| 1. Standing long periods | 1.54 (0.89) | 1 (1-4) | 66% | 0% | 1.58 | 1.47 | 0.44 | |
| 2. Household responsibilities | 2.07 (1.02) | 2 (1-4) | 38% | 0% | 0.47 | –1.00 | 0.64 | |
| 3. Learning new tasks | 1.87 (1.02) | 2 (1-5) | 49% | 1% | 0.85 | –0.37 | 0.70 | |
| 4. Joining community activities | 2.23 (1.19) | 2 (1-5) | 38% | 4% | 0.55 | –0.77 | 0.78 | |
| 5. Emotionally affected | 3.21 (1.14) | 4 (1-5) | 13% | 6% | –0.69 | –0.60 | 0.59 | |
| 6. Concentrating | 2.37 (1.05) | 2 (1-5) | 26% | 1% | 0.17 | –0.90 | 0.61 | |
| 7. Walking long distance | 1.51 (0.82) | 1 (1-5) | 66% | 1% | 1.63 | 2.29 | 0.52 | |
| 8. Washing whole body | 1.20 (0.55) | 1 (1-4) | 86% | 0% | 3.12 | 10.22 | 0.48 | |
| 9. Getting dressed | 1.16 (0.47) | 1 (1-4) | 88% | 0% | 3.44 | 12.87 | 0.48 | |
| 10. Dealing with strangers | 1.83 (1.01) | 2 (1-5) | 49% | 1% | 1.10 | 0.43 | 0.66 | |
| 11. Maintaining friendships | 1.85 (1.07) | 1 (1-5) | 53% | 1% | 0.97 | –0.17 | 0.59 | |
| 12. Work/school activities | 2.54 (1.20) | 3 (1-5) | 25% | 6% | 0.27 | –0.88 | 0.75 | |
| Total score | 23.37 (7.91) | 23 (12-51) | 8% | 0% | 0.68 | 0.52 | — | |
| Subscale | ||||||||
| Psychosocial | 12.60 (4.70) | 12.76 (5.60-24.84) | 8% | 0% | 0.31 | –0.49 | — | |
| Self-care | 2.12 (0.86) | 1.80 (1.80-7.21) | 84% | 0% | 3.20 | 11.23 | — | |
| Mobility | 2.25 (1.15) | 1.47 (1.47-6.49) | 56% | 0% | 1.56 | 1.76 | — | |
aWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0. Based on baseline data from two clinical trials of severe health anxiety and stress disorders (total N=160, items scored from 1 to 5).
bSD: standard deviation.
cITC: adjusted item-total correlation.
Internal Consistency
Adjusted baseline ITCs are presented in Table 3, where the mean adjusted ITC was 0.60 (SD 0.11). Cronbach alpha values were good to excellent for both the total sum score and the proposed psychosocial subscale (Table 4).
Table 4.
Web-based self-report 12-item WHODAS 2.0a internal consistency.
| Sample | Screening | Pretreatment | Posttreatment | ||||
| Cronbach alpha | n | Cronbach alpha | n | Cronbach alpha | n | ||
| Total score | |||||||
| Severe health anxiety | .83 | 60 | .86 | 60 | .87 | 56 | |
| Stress disorder | — | — | .90 | 100 | .92 | 97 | |
| Psychosocial subscaleb | |||||||
| Severe health anxiety | .82 | 60 | .88 | 60 | .87 | 56 | |
| Stress disorder | — | — | .89 | 100 | .91 | 97 | |
aWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.
bPsychosocial subscale with regression weights applied.
Test-Retest Reliability
The test-retest reliability of the WHODAS 2.0, based on the severe health anxiety sample data (n=25), was estimated at ICC=0.83 (95% CI 0.62-0.92) and the correlation between the measurements was r=.71. The test-retest reliability of the proposed psychosocial subscale was similar (ICC=0.85, 95% CI 0.67-0.94; r=.75).
Construct Validity: Associations With Other Measures
In the stress disorders sample, patients with a principal diagnosis of exhaustion disorder had a significantly higher WHODAS 2.0 mean score than those who had a principal diagnosis of adjustment disorder (mean difference 8.84, 95% CI 5.96-11.72; t98=6.10, P<.001), and the WHODAS 2.0 discriminated well between stress patients with and without exhaustion disorder (AUC=0.81, 95% CI 0.73-0.89; P<.001). The proposed psychosocial subscale fared about as well (AUC=0.79, 95% CI 0.70-0.87; P<.001). Baseline correlations between the WHODAS 2.0 as well as the subscale and other self-assessment questionnaires are shown in Table 5.
Table 5.
WHODAS 2.0 associations (bivariate Pearson correlations) with other self-rated questionnairesa.
| WHODAS 2.0b |
Psychosocial subscale |
MADRS-Sb | GAD-7b | SDSb | ||
| WHODAS 2.0 | — | .96c | .60c | .58c,d | .66c | |
| Psychosocial subscale | .97e | — | .65c | .54c,d | .67c | |
| MADRS-S | .65e | .64e | — | .54c,d | .59c | |
| GAD-7 | .45e | .45e | .71e | — | .40c | |
| WAI | –.71e | –.71e | –.55e | –.28e | — |
aAll bivariate Pearson correlations significant at α=.05.
bWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0; MADRS-S: Montgomery-Åsberg Depression Rating Scale-self-report version; GAD-7: Generalized Anxiety Disorder 7-item scale; SDS: Sheehan Disability Scale; WAI: Work Ability Index.
cSevere health anxiety sample (n=60).
dGAD-7 data only available from a subsample of the severe health anxiety sample (n=43).
eStress disorder sample (n=100).
Sensitivity to Change
Effect sizes and tests pertaining to responsiveness are presented in Table 6. Wilcoxon signed-rank tests showed that all but one WHODAS 2.0 item (item 9, related to self-care) changed in the severe health anxiety sample, whereas 9 of 12 items changed (but not items 1, 8, or 9—one mobility and both self-care items) in the stress disorders sample.
Table 6.
Responsiveness stratified by change in primary symptom domaina.
| Change in primary symptom domainb |
n (%) |
ESc |
Mean change (95% CI) |
Pd | ||
| Severe health anxiety sample | ||||||
| HAI reliably improved >1.5 SDe | 32 (5) | |||||
| WHODAS 2.0 | 1.21 | 7.19 (5.17, 9.22) | <.001 | |||
| SDS | 1.00 | 7.28 (4.59, 9.98) | <.001 | |||
| HAI reliably improved ≤1.5 SD | 8 (14) | |||||
| WHODAS 2.0 | 0.52 | 2.75 (–0.85, 6.35) | .11 | |||
| SDS | 0.53 | 1.88 (–1.07, 4.82) | .18 | |||
| HAI not reliably improved | 18 (31) | |||||
| WHODAS 2.0 | 0.29 | 2.00 (–0.13, 4.13) | .06 | |||
| SDS | 0.22 | 1.39 (–0.59, 3.37) | .16 | |||
| Stress disorder sample | ||||||
| PSS reliably improved >1.5 SD | 33 (34) | |||||
| WHODAS 2.0 | 0.80 | 6.42 (4.27, 8.58) | <.001 | |||
| WAI | –0.49 | –4.14 (–5.75, –2.52) | <.001 | |||
| PSS reliably improved ≤1.5 SD | 12 (12) | |||||
| WHODAS 2.0 | 0.39 | 2.83 (0.09, 5.58) | .04 | |||
| WAI | –0.25 | –2.29 (–5.10, 0.51) | .10 | |||
| PSS not reliably improved | 52 (54) | |||||
| WHODAS 2.0 | 0.19 | 1.75 (-0.10, 3.60) | .06 | |||
| WAI | –0.13 | –0.96 (–2.66, 0.73) | .26 | |||
aAll estimates based on data from patients that completed the posttreatment assessment. Status as improved or not improved based on the Jacobson and Truax reliable change index [32].
bHAI: Health Anxiety Inventory; WHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0; SDS: Sheehan Disability Scale; PSS: Perceived Stress Scale; WAI: Work Ability Index.
cStandardized effect sizes (ES) are calculated with the stratum-specific pretreatment standard deviation as nominator.
dP values based on paired samples t test.
eSD: standard deviation.
Discussion
Summary of Principal Results
To our knowledge, this was the first in-depth analysis of the reliability and validity of the self-rated 12-item WHODAS 2.0 when administered online to patients with anxiety and stress disorders. In line with our hypotheses, the WHODAS 2.0 exhibited high internal consistency, acceptable test-retest reliability, and was demonstrated to identify cases of exhaustion disorder. As expected, the WHODAS 2.0 also showed substantial associations with other measures of functioning, as well as slightly weaker but yet substantial associations with primary symptom measures. The instrument was sensitive to change in the primary psychiatric symptom domain, as illustrated by a convincing gradient in change effect size over the nonimproved versus slightly improved versus much improved strata (Table 6). It might be noted that there was no significant change in WHODAS 2.0 for the minimally changed severe health anxiety group, but this is not surprising given the very small size of this subsample (n=8). Taken together, our findings suggest that the self-rated 12-item online WHODAS 2.0—when administered to individuals with anxiety and stress disorders—is a valid measure of disability and functional impairment. The results are important because online administration facilitates handling of data in a wide range of research and routine care settings. Key strengths of this study are that two psychiatric samples with different primary conditions could be examined, that other scales could be used to validate the WHODAS 2.0 scores in both samples, and that the ability of the WHODAS 2.0 to identify patients with a clinical diagnosis of exhaustion disorder could be investigated in the stress disorders sample.
Factor Structure
Regarding factor structure, we could neither confirm the expected one-factor WHO solution nor the second-order model put forward by previous investigators [5,6]. A post hoc model derived from exploratory analysis saw most items loading heavily (>.5) on one strong factor, and four items (8, 9, 1, and 7) loading heavily on two other factors. The results are somewhat similar to those seen with the WHODAS 2.0 administered to individuals with musculoskeletal pain [34], and suggests that items 8 and 9 (“self-care”), as well as items 1 and 7 (“getting around”), largely tap into sources of variance other than that responsible for most of the sum score (see Table 3). Our subjective impression is that these four items seem more related to physical incapacity than the rest of the scale. Because (1) these items showed apparent evidence of a floor effect (their median sum being zero), (2) the sum score primarily relied on another very strong factor, and (3) the interfactor correlations were moderately strong, the impact of multidimensionality is likely to be very small in the study of change in otherwise healthy samples with anxiety or stress disorders. On the other hand, the interfactorial correlations were small enough to complicate direct comparisons of scores between studies. Especially when anxiety or stress disorder samples differ in terms of somatic comorbidity, it is probably informative to compare the “getting around” and “self-care” subdomains separately. That is, based on the results of this study our tentative recommendation would be for studies on anxiety and stress disorders to report both the self-rated 12-item WHODAS 2.0 sum score, as well as the three subfactors, here referred to as “psychosocial,” “self-care,” and “getting around” (Table 2). However, we wish to emphasize that this factor solution was solely based on an explorative analysis, that the second-order model with six subfactors [5,6] was not far off the mark, and that additional confirmatory factor analyses based on data from similar samples are warranted to arrive at more firm conclusions.
Score Distribution
On an item level, the most striking aspect of the score distribution was that both items 8 and 9 (ie, the “self-care” items) had a mode score of 1 and appeared to pick up on very little variance in the two samples. This may suggest that individuals with a principal diagnosis of severe health anxiety or a stress disorder are typically not much impaired in terms of ability to carry out everyday self-care tasks such as washing oneself or getting dressed.
Internal Consistency and Test-Retest Reliability
Although the internal consistency and test-retest reliability estimates were in line with our expectations, and also fulfilled commonly accepted quality criteria for health status questionnaires [35], it is conceivable that the estimates of test-retest reliability were deflated because a psychiatric interview and the decision to include patients in a clinical trial took place between the screening and pretreatment measurements. However, the means from these two measurement points were not significantly different (t59=1.55, P=.13), which for example speaks against patients having altered their responses so as to be included in one of the trials.
Convergent Validity and Sensitivity to Change
In this study, correlations with other measures of psychiatric symptoms and functional impairment corroborate the construct validity of the self-rated 12-item WHODAS 2.0. As this study was based on data from two clinical trials, we do not find it unfit or surprising that the WHODAS 2.0 was highly correlated with measures of anxiety and depression which were likely to be the primary reasons for functional impairment.
Due to the limited sample size and variability in the size of substrata (nonchanged vs minimally changed vs much changed) we wish to emphasize that the significance tests pertaining to change (Table 6) are of limited value, and emphasize that all P values ought to be interpreted alongside their corresponding effect sizes. Over and above the effect size gradient over the nonchanged versus minimally changed versus much changed strata, a general trend was also that the 12-item WHODAS 2.0 changed slightly more from pretreatment to posttreatment than the WAI. This may suggest that the WHODAS 2.0 covers dimensions of functional impairment that, compared to work ability, are more likely to change in a clinical trial of CBT for chronic stress disorders.
Limitations
The primary limitation of this study is that over and above clinical diagnoses we did not have access to “hard” measures of functional impairment, collected by other means than self-assessment (eg, register data on disability status) that could be used to validate the WHODAS 2.0. One consequence of this is that all indexes of change were to some degree susceptible to social desirability bias or the possibility that patients reported change so as to please their therapist rather than as a consequence of real change in symptoms or disability. However, it has been demonstrated that Web-based survey administration is relatively robust to desirability bias [36], and all scales used to validate the WHODAS 2.0 (ie, the HAI, PSS, MADRS-S, GAD-7, SDS, and WAI) have, at least in conventional form, been shown to be associated with non-self-reported validators such as clinical diagnoses, sick leave, and health care consumption (eg, [16,37-41]).
We also had no control over what equipment the patients used to access the Web-based WHODAS 2.0, meaning we could not determine the significance of filling in the questionnaire via a mobile phone app rather than a conventional browser on a desktop computer, for example. The results showing good psychometric properties of the WHODAS 2.0 also suggest that there was no substantial measurement error related to the type of device used. Another threat to the generalizability of our findings is that the two samples were relatively homogenous due to the eligibility criteria for the two clinical trials, which were not primarily designed to study the psychometric properties of the WHODAS 2.0. Therefore, it is preferable to validate the findings of this study, particularly with regard to factor structure, in anxiety and stress disorder samples recruited through other means.
Comparison With Prior Work
The item mean score profile seen in this study—with substantial functional impairment in the “understanding and communicating,” “life activities,” and “getting along with others” domains but very low functional impairment in the “self-care” domain—is highly similar to the item mean score profile seen in studies of interview and pencil-and-paper versions of the WHODAS 2.0 administered to individuals with common mental disorders [8,14]. This suggests that the response pattern does not change much due to the online format.
Before this study, the 12-item WHODAS 2.0 had also been administered online to individuals with common anxiety, stress, and mood disorders in several clinical trials (Table 7, see Multimedia Appendices 1 and 2 for details).
Table 7.
Previous studies that have administered the 12-item self-report WHODAS 2.0 online to respondents with common mental disordersa.
|
Author (year) |
N |
Diagnosisb |
Mean |
Cronbach alpha |
WHODAS 2.0c change over time? |
||
| 12-60 scaled | 0-48 scaled | 0-100 scaled | |||||
| Allen et al (2016) [42] | 63 | PD | 26.0 | 14.0 | 29.2 | .89 | Yes |
| Andrews et al (2011) [43] | 37 | SAD | 26.6 | 14.6 | 30.4 | — | Yes |
| Mason & Andrews (2014) [44] | 173e | Mixed CMD | 25.0 | 13.0 | 27.1 | — | Yes |
| Mewton et al (2012) [45] | 588 | GAD | 25.7 | 13.7 | 28.5 | .90 | Yes |
| Newby et al (2016a) [46] | 2109 | Mixed CMD | 26.3 | 14.3 | 29.8 | >.88 | Yes |
| Newby et al (2016b) [47] | 16 | SHA | 20.3 | 8.3 | 17.3 | .83 | Yes |
| Perini et al (2008) [48] | 13 | MDD | —f | —f | —f | — | Yes |
| Spence et al (2011) [49] | 244 | PTSD | 32.1 | 20.1 | 41.9 | .92 | — |
| Titov et al (2008a) [50] | 105 | SAD | 26.3 | 14.3 | 29.8 | — | Yes |
| Titov et al (2008b) [51] | 88 | SAD | 25.0 | 13.0 | 27.1 | — | Yes |
| Williams et al (2013) [52] | 69 | MDD | 41.2 | 29.2 | 60.8 | — | Yes |
| Williams et al (2014) [53] | 560 | SAD | 39.3 | 27.3 | 56.9 | — | Yes |
| Williams et al (2015) [54] | 75 | MDD | 44.8 | 32.8 | 68.3 | .83 | Yes |
| Wootton et al (2011) [55] | 118 | OCD | 30.4 | 18.4 | 38.3 | .91 | — |
aInformation from articles complemented by personal communication via email.
bPD: panic disorder; SAD: social anxiety disorder; CMD: common mental disorder; GAD: generalized anxiety disorder; SHA: severe health anxiety; MDD: major depressive disorder; PTSD: posttraumatic stress disorder; OCD: obsessive compulsive disorder; . See Multimedia Appendices 1 and 2 for details.
cWHODAS 2.0: World Health Organization Disability Assessment Schedule 2.0.
dThe 12-60 scale has items scored 1-5, the 0-48 scale has items scored 0-4, and the 0-100 scale is the 0-48 scale divided by 48 and then multiplied by 100.
eAnother sample (n=135) in this study completed the WHODAS 2.0 on a computer, but not via the Internet.
fUnknown; author could not be reached.
Although the primary aim of these trials was not to study the psychometric properties of the WHODAS 2.0, these studies presented estimates of both internal consistency (Cronbach alpha≥.83) and baseline mean scores that are very much in line with data in this trial, and thus lend further support to the validity and generalizability of our findings.
Conclusions
This is, to date, the most extensive investigation into the psychometric properties of the self-rated 12-item version of the WHODAS 2.0 when administered via the Internet to individuals with anxiety and stress disorders. When administered online to individuals with anxiety and stress disorders, the WHODAS 2.0 exhibits high internal consistency, high convergent validity, adequate test-retest reliability, and is sensitive to change. We conclude that the psychometric properties of the self-rated 12-item version of the WHODAS 2.0 are acceptable when the instrument is administered via the Internet to individuals with anxiety and stress disorders, but suggest that the three subfactors found in this study be reported alongside the sum score to facilitate comparisons between studies.
Acknowledgments
This work was supported by Karolinska Institutet and Stockholm County Council, which are public institutions that did not influence the study design, execution, or publication process.
Abbreviations
- AUC
area under the curve
- CBT
cognitive behavior therapy
- CFI
comparative fit index
- DSM-5
Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition)
- GAD-7
Generalized Anxiety Disorder seven-item scale
- HAI
Health Anxiety Inventory
- ICC
intraclass correlation coefficient
- ICD-10
International Classification of Diseases, Tenth Revision
- ITC
item-total correlations
- MADRS-S
Montgomery-Åsberg Depression Rating Scale-self-report
- PSS
Perceived Stress Scale
- RMSEA
root mean square error of approximation
- SDS
Sheehan Disability Scale
- SRMR
standardized root mean square residual
- TLI
Tucker-Lewis index
- WAI
Work Ability Index
- WHODAS 2.0
World Health Organization Disability Assessment Schedule 2.0
Search strategy and data collection for brief review.
Flowchart illustrating study selection process for brief review.
Footnotes
Authors' Contributions: All authors made significant contributions to the planning, recruitment, execution, analysis, and publication of this study.
Conflicts of Interest: None declared.
References
- 1.Üstün TB, Kostanjsek N, Chatterji S, Rehm J, editors. Measuring Health and Disability: Manual for WHO Disability Assessment Schedule (WHODAS 2.0) Malta: World Health Organization; 2010. [Google Scholar]
- 2.Üstün TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, Saxena S, von Korff M, Pull C, World Health Organization. National Institutes of Health Developing the World Health Organization Disability Assessment Schedule 2.0. Bull World Health Organ. 2010 Nov 1;88(11):815–823. doi: 10.2471/BLT.09.067231. http://europepmc.org/abstract/MED/21076562. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.World Health Organization . International Classification of Functioning, Disability and Health: ICF. Geneva: World Health Organization; 2001. [Google Scholar]
- 4.American Psychiatric Association . Diagnostic and Statistical Manual of Mental Disorders. Arlington, VA: American Psychiatric Association; 2013. [Google Scholar]
- 5.Andrews G, Kemp A, Sunderland M, Von KM, Ustun TB. Normative data for the 12 item WHO Disability Assessment Schedule 2.0. PLoS One. 2009 Dec;4(12):e8343. doi: 10.1371/journal.pone.0008343. http://dx.plos.org/10.1371/journal.pone.0008343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Weeks M, Garber BG, Zamorski MA. Disability and mental disorders in the Canadian Armed Forces. Can J Psychiatry. 2016 Apr;61(1 Suppl):56S–63S. doi: 10.1177/0706743716628853. http://europepmc.org/abstract/MED/27270743. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.McEvoy PM, Mahoney AE. Achieving certainty about the structure of intolerance of uncertainty in a treatment-seeking sample with anxiety and depression. J Anxiety Disord. 2011 Jan;25(1):112–122. doi: 10.1016/j.janxdis.2010.08.010. [DOI] [PubMed] [Google Scholar]
- 8.Luciano JV, Ayuso-Mateos JL, Fernández A, Serrano-Blanco A, Roca M, Haro JM. Psychometric properties of the twelve item World Health Organization Disability Assessment Schedule II (WHO-DAS II) in Spanish primary care patients with a first major depressive episode. J Affect Disord. 2010 Feb;121(1-2):52–58. doi: 10.1016/j.jad.2009.05.008. [DOI] [PubMed] [Google Scholar]
- 9.Buchanan T. Internet-based questionnaire assessment: appropriate use in clinical contexts. Cogn Behav Ther. 2003;32(3):100–109. doi: 10.1080/16506070310000957. [DOI] [PubMed] [Google Scholar]
- 10.Gosling SD, Vazire S, Srivastava S, John OP. Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. Am Psychol. 2004 Mar;59(2):93–104. doi: 10.1037/0003-066X.59.2.93. [DOI] [PubMed] [Google Scholar]
- 11.Grossi G, Perski A, Osika W, Savic I. Stress-related exhaustion disorder - clinical manifestation of burnout? A review of assessment methods, sleep impairments, cognitive disturbances, and neuro-biological and physiological changes in clinical burnout. Scand J Psychol. 2015 Dec;56(6):626–636. doi: 10.1111/sjop.12251. [DOI] [PubMed] [Google Scholar]
- 12.Axelsson E, Andersson E, Ljótsson B, Wallhed Finn D, Hedman E. The health preoccupation diagnostic interview: inter-rater reliability of a structured interview for diagnostic assessment of DSM-5 somatic symptom disorder and illness anxiety disorder. Cogn Behav Ther. 2016 Jun;45(4):259–269. doi: 10.1080/16506073.2016.1161663. [DOI] [PubMed] [Google Scholar]
- 13.Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, Hergueta T, Baker R, Dunbar GC. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry. 1998;59 Suppl 20:22–33;quiz 34. [PubMed] [Google Scholar]
- 14.Sjonnesen K, Bulloch AG, Williams J, Lavorato D. Characterization of disability in Canadians with mental disorders using an abbreviated version of a DSM-5 emerging measure: the 12-Item WHO Disability Assessment Schedule (WHODAS) 2.0. Can J Psychiatry. 2016 Apr;61(4):227–235. doi: 10.1177/0706743716632514. http://europepmc.org/abstract/MED/27254415. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Leon AC, Shear MK, Portera L, Klerman GL. Assessing impairment in patients with panic disorder: the Sheehan Disability Scale. Soc Psychiatry Psychiatr Epidemiol. 1992 Mar;27(2):78–82. doi: 10.1007/bf00788510. [DOI] [PubMed] [Google Scholar]
- 16.Leon AC, Olfson M, Portera L, Farber L, Sheehan DV. Assessing psychiatric impairment in primary care with the Sheehan Disability Scale. Int J Psychiatry Med. 1997;27(2):93–105. doi: 10.2190/t8em-c8yh-373n-1uwd. [DOI] [PubMed] [Google Scholar]
- 17.De Zwart BC, Frings-Dresen MH, van Duivenbooden JC. Test-retest reliability of the Work Ability Index questionnaire. Occup Med (Lond) 2002 Jun;52(4):177–181. doi: 10.1093/occmed/52.4.177. [DOI] [PubMed] [Google Scholar]
- 18.Radkiewicz P, Widerszal-Bazyl M, NEXT-Study Group Psychometric properties of Work Ability Index in the light of comparative survey study. 2nd International Symposium on Work Ability; Oct 18-20, 2004; Verona, Italy. Elsevier; 2005. [Google Scholar]
- 19.Ilmarinen J. The Work Ability Index (WAI) Occup Med. 2007 Mar 01;57(2):160. doi: 10.1093/occmed/kqm008. [DOI] [Google Scholar]
- 20.Salkovskis PM, Rimes KA, Warwick HMC, Clark DM. The Health Anxiety Inventory: development and validation of scales for the measurement of health anxiety and hypochondriasis. Psychol Med. 2002 Jul;32(5):843–853. doi: 10.1017/S0033291702005822. [DOI] [PubMed] [Google Scholar]
- 21.Hedman E, Ljótsson B, Andersson E, Andersson G, Lindefors N, Rück C, Axelsson E, Lekander M. Psychometric properties of Internet-administered measures of health anxiety: an investigation of the Health Anxiety Inventory, the Illness Attitude Scales, and the Whiteley Index. J Anxiety Disord. 2015 Apr;31:32–37. doi: 10.1016/j.janxdis.2015.01.008. [DOI] [PubMed] [Google Scholar]
- 22.Hedman E, Lekander M, Ljótsson B, Lindefors N, Rück C, Andersson G, Andersson E. Optimal cut-off points on the health anxiety inventory, illness attitude scales and whiteley index to identify severe health anxiety. PLoS One. 2015;10(4):e0123412. doi: 10.1371/journal.pone.0123412. http://dx.plos.org/10.1371/journal.pone.0123412. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983 Dec;24(4):385–396. doi: 10.2307/2136404. [DOI] [PubMed] [Google Scholar]
- 24.Svanborg P, Åsberg M. A new self-rating scale for depression and anxiety states based on the Comprehensive Psychopathological Rating Scale. Acta Psychiatr Scand. 1994 Jan;89(1):21–28. doi: 10.1111/j.1600-0447.1994.tb01480.x. [DOI] [PubMed] [Google Scholar]
- 25.Svanborg P, Åsberg M. A comparison between the Beck Depression Inventory (BDI) and the self-rating version of the Montgomery Asberg Depression Rating Scale (MADRS) J Affect Disord. 2001 May;64(2-3):203–216. doi: 10.1016/s0165-0327(00)00242-1. [DOI] [PubMed] [Google Scholar]
- 26.Carlbring P, Brunt S, Bohman S, Austin D, Richards J, Öst LG, Andersson G. Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Comput Hum Behav. 2007 May;23(3):1421–1434. doi: 10.1016/j.chb.2005.05.002. [DOI] [Google Scholar]
- 27.Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006 May 22;166(10):1092–1097. doi: 10.1001/archinte.166.10.1092. [DOI] [PubMed] [Google Scholar]
- 28.R Core Team . R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2016. [2017-11-23]. https://www.r-project.org/ [Google Scholar]
- 29.Rosseel Y. lavaan: an R package for structural equation modeling. J Stat Softw. 2012;48(2):1–36. doi: 10.18637/jss.v048.i02. [DOI] [Google Scholar]
- 30.Li C. Confirmatory factor analysis with ordinal data: comparing robust maximum likelihood and diagonally weighted least squares. Behav Res Methods. 2016 Sep;48(3):936–949. doi: 10.3758/s13428-015-0619-7. [DOI] [PubMed] [Google Scholar]
- 31.Byrne BM. Structural Equation Modeling with AMOS: Basic Concepts, Applications and Programming. Mahwah, NJ: Lawrence Erlbaum Associates; 2001. [Google Scholar]
- 32.Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991 Feb;59(1):12–19. doi: 10.1037/0022-006x.59.1.12. [DOI] [PubMed] [Google Scholar]
- 33.Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008 Feb;61(2):102–109. doi: 10.1016/j.jclinepi.2007.03.012. [DOI] [PubMed] [Google Scholar]
- 34.Saltychev M, Mattie R, McCormick Z, Laimi K. Confirmatory factor analysis of 12-Item World Health Organization Disability Assessment Schedule in patients with musculoskeletal pain conditions. Clin Rehabil. 2017 May;31(5):702–709. doi: 10.1177/0269215516652930. [DOI] [PubMed] [Google Scholar]
- 35.Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, Bouter LM, de Vet HC. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007 Jan;60(1):34–42. doi: 10.1016/j.jclinepi.2006.03.012. [DOI] [PubMed] [Google Scholar]
- 36.Kreuter F, Presser S, Tourangeau R. Social desirability bias in CATI, IVR, and web surveys: the effects of mode and question sensitivity. Public Opinion Quarterly. 2009 Jan 26;72(5):847–865. doi: 10.1093/poq/nfn063. [DOI] [Google Scholar]
- 37.Eklund M, Bäckström M, Tuvesson H. Psychometric properties and factor structure of the Swedish version of the Perceived Stress Scale. Nord J Psychiatry. 2014 Oct;68(7):494–499. doi: 10.3109/08039488.2013.877072. [DOI] [PubMed] [Google Scholar]
- 38.Seivewright H, Salkovskis P, Green J, Mullan N, Behr G, Carlin E, Young S, Goldmeier D, Tyrer P. Prevalence and service implications of health anxiety in genitourinary medicine clinics. Int J STD AIDS. 2004 Aug;15(8):519–522. doi: 10.1258/0956462041558122. [DOI] [PubMed] [Google Scholar]
- 39.Ericsson M, Poston II WS, Linder J, Taylor JE, Haddock CK, Foreyt JP. Depression predicts disability in long-term chronic pain patients. Disabil Rehabil. 2002 Apr 15;24(6):334–340. doi: 10.1080/09638280110096241. [DOI] [PubMed] [Google Scholar]
- 40.Kroenke K, Spitzer RL, Williams JB, Löwe B. The Patient Health Questionnaire Somatic, Anxiety, and Depressive Symptom Scales: a systematic review. Gen Hosp Psychiatry. 2010;32(4):345–359. doi: 10.1016/j.genhosppsych.2010.03.006. [DOI] [PubMed] [Google Scholar]
- 41.van Den Berg TI, Elders LA, de Zwart BC, Burdorf A. The effects of work-related and individual factors on the Work Ability Index: a systematic review. Occup Environ Med. 2009 Apr;66(4):211–220. doi: 10.1136/oem.2008.039883. [DOI] [PubMed] [Google Scholar]
- 42.Allen AR, Newby JM, Mackenzie A, Smith J, Boulton M, Loughnan SA, Andrews G. Internet cognitive-behavioural treatment for panic disorder: randomised controlled trial and evidence of effectiveness in primary care. BJPsych Open. 2016 Mar;2(2):154–162. doi: 10.1192/bjpo.bp.115.001826. http://europepmc.org/abstract/MED/27703768. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Andrews G, Davies M, Titov N. Effectiveness randomized controlled trial of face to face versus Internet cognitive behaviour therapy for social phobia. Aust N Z J Psychiatry. 2011 Apr;45(4):337–340. doi: 10.3109/00048674.2010.538840. [DOI] [PubMed] [Google Scholar]
- 44.Mason EC, Andrews G. The use of automated assessments in internet-based CBT: the computer will be with you shortly. Internet Interv. 2014 Oct;1(4):216–224. doi: 10.1016/j.invent.2014.10.003. [DOI] [Google Scholar]
- 45.Mewton L, Wong N, Andrews G. The effectiveness of internet cognitive behavioural therapy for generalized anxiety disorder in clinical practice. Depress Anxiety. 2012 Oct;29(10):843–849. doi: 10.1002/da.21995. [DOI] [PubMed] [Google Scholar]
- 46.Newby JM, Mewton L, Andrews G. Transdiagnostic versus disorder-specific internet-delivered cognitive behaviour therapy for anxiety and depression in primary care. J Anxiety Disord. 2017 Mar;46:25–34. doi: 10.1016/j.janxdis.2016.06.002. [DOI] [PubMed] [Google Scholar]
- 47.Newby JM, Mahoney AE, Mason EC, Smith J, Uppal S, Andrews G. Pilot trial of a therapist-supported internet-delivered cognitive behavioural therapy program for health anxiety. Internet Interv. 2016 Nov;6:71–79. doi: 10.1016/j.invent.2016.09.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Perini S, Titov N, Andrews G. The Climate Sadness program of Internet-based treatment for depression: a pilot study. EJAP. 2008 Dec 23;4(2):18–24. doi: 10.7790/ejap.v4i2.135. [DOI] [Google Scholar]
- 49.Spence J, Titov N, Solley K, Dear BF, Johnston L, Wootton B, Kemp A, Andrews G, Zou J, Lorian C, Choi I. Characteristics and treatment preferences of people with symptoms of posttraumatic stress disorder: an internet survey. PLoS One. 2011;6(7):e21864. doi: 10.1371/journal.pone.0021864. http://dx.plos.org/10.1371/journal.pone.0021864. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Titov N, Andrews G, Schwencke G, Drobny J, Einstein D. Shyness 1: distance treatment of social phobia over the Internet. Aust N Z J Psychiatry. 2008 Jul;42(7):585–594. doi: 10.1080/00048670802119762. [DOI] [PubMed] [Google Scholar]
- 51.Titov N, Andrews G, Schwencke G. Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry. 2008 Jul;42(7):595–605. doi: 10.1080/00048670802119820. [DOI] [PubMed] [Google Scholar]
- 52.Williams AD, Blackwell SE, Mackenzie A, Holmes EA, Andrews G. Combining imagination and reason in the treatment of depression: a randomized controlled trial of internet-based cognitive-bias modification and internet-CBT for depression. J Consult Clin Psychol. 2013 Oct;81(5):793–799. doi: 10.1037/a0033247. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3780629&tool=pmcentrez&rendertype=abstract. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 53.Williams AD, O'Moore K, Mason E, Andrews G. The effectiveness of internet cognitive behaviour therapy (iCBT) for social anxiety disorder across two routine practice pathways. Internet Interv. 2014 Oct;1(4):225–229. doi: 10.1016/j.invent.2014.11.001. [DOI] [Google Scholar]
- 54.Williams AD, O'Moore K, Blackwell SE, Smith J, Holmes EA, Andrews G. Positive imagery cognitive bias modification (CBM) and internet-based cognitive behavioral therapy (iCBT): a randomized controlled trial. J Affect Disord. 2015 Jun 01;178:131–141. doi: 10.1016/j.jad.2015.02.026. https://linkinghub.elsevier.com/retrieve/pii/S0165-0327(15)00119-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Wootton BM, Titov N, Dear BF, Spence J, Kemp A. The acceptability of Internet-based treatment and characteristics of an adult sample with obsessive compulsive disorder: an Internet survey. PLoS One. 2011;6(6):e20548. doi: 10.1371/journal.pone.0020548. http://dx.plos.org/10.1371/journal.pone.0020548. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Search strategy and data collection for brief review.
Flowchart illustrating study selection process for brief review.