Reliability and validity of the self-reported Activities of Daily Living Scale for people with mental illness

Short abstract

Background/objectives

Patient-reported outcome measures have been found to be an effective method of reflecting client perspectives on their personal health condition. The primary aim of this study was to determine the reliability and validity of the self-reported Activities of Daily Living Scale (sf-ADLS) using Rasch analysis in Taiwan.

Methods

A total of 455 people were included in this study; 224 were persons with mental illness and 231 were healthy adults. We applied Rasch analysis as the means of testing the psychometrics of the scale.

Results

The final version of the sf-ADLS used in this study included 14 items, with no differential item functioning being discernible on the gender variable. The scale was found to be of use in classifying the subjects into four levels of independence.

Conclusions

The revised sf-ADLS conforms to the Rasch measurement model in the formulation of a unidimensional scale. The scale can be used to measure the level of independence with acceptable reliability (internal consistency as 0.9) and validity.

Introduction

Mental illness poses a significant and widespread threat to our population, with the high prevalence rate found in the US suggesting that in 2012, an estimated 43.7 million adults aged 18 years or above were suffering from some form of mental illness (18.29% of adults struggle with a mental health problem annually) (Mental Health America website, http://www.mentalhealthamerica.net/issues/2017-state-mental-health-america-prevalence-data). Indeed, in an earlier report, the World Health Organization (WHO) indicated that neuro-mental illness made a significant contribution (33.2%) to the total disability burden for the US and Canada (NIMH, 2008).

According to a ‘World Mental Health’ survey carried out by the WHO, mental illness has particularly destructive effects on social life and personal relationships, which can create psychological barrier to everyday functions (Ormel et al., 2008; Reichenberg et al., 2009). Furthermore, severe mental illness invariably leads to serious functional disability, substantially limiting or interfering with one or more major life activities (Murray & Lopez, 1997). Thus, the evaluation of psychiatric patients with functional disabilities, and appropriate interventions to deal with such disabilities, are crucially important issues.

The definition of functional disability refers to the limitations that are imposed upon the independent performance of daily life tasks; these tasks are often further divided into basic ‘activities of daily living’ (ADLs) and ‘instrumental activities of daily living’ (IADLs), with the latter including activities that are necessary for dwelling within a community, such as shopping, managing finances, housekeeping and meal preparation (Spector & Fleishman, 1998).

Numerous experts have been striving for some time to develop appropriate means of evaluating ADL and IADL performance among persons with mental illness (Holmqvist & Holmefur, 2017; Tornquist & Sonn, 2014). The following two instruments were developed by occupational therapists for use in persons with mental illness. For example, the ‘Kohlman Evaluation of Living Skills’ (KELS) provides a standardised tool for evaluating both basic and instrumental ADLs (Thomson & Robnett, 2016), whilst the ‘Milwaukee Evaluation of Living Skills’ provides a method of evaluating patients in long-term psychiatric care (Askew, 1990; Haertlein, 1993).

Although a few similar instruments have been proposed and developed through the years, differences have been observed in the cross-cultural use of such tools for the measurement of daily life activities. For example, there is an item in KELS which asks if the client is able to obtain and/or continue the food stamp (Teixeira, Pereira, Maenner, & Albuquerque, 2018) which would not be applicable in Taiwan due to different welfare system.

Furthermore, some of the existing questionnaires exhibit limitations; for example, the ‘activities of daily living rating scale III’ (ADLS III), the most widely applied ADL questionnaire in Taiwan. There were overall 82 items in the ADLS III of which quite a few items required blank filling answers; therefore, it took about 20–30 minutes for clients to complete. The ADLS III was found to be very time-consuming and easily influenced by educational levels, thereby limiting its application in clinical settings (Chu & Hsieh, 2004). Accordingly, there is a clear need for the development of an efficient method of comprehensively assessing self-reported daily living activities among persons with mental illness, and indeed, the method must be well developed and focus on functional perspective, if we are to provide appropriate interventions for such patients. Because of this, we aim to develop a shorter and functional orientation scale as an alternative to apply.

Nowadays, client-centred practice is congruent with, and inseparable from, clinical decision making (Tickle-Degnen, 2002), with one very important client-centred profession, ‘occupational therapy’, providing help to people and groups experiencing difficulties by re-establishing the balance between their occupational life and independence, despite the existence of symptoms and diseases.

Given the lack of an appropriate patient-reported ADL outcome measure for psychiatric patients in Taiwan, Pan, Chung, and Chen (2005, 2006) began developing a comprehensive self-reported daily living scale based on the ‘uniform terminology-III’ and the ‘Occupational Therapy Practice Framework’ produced by the American Occupational Therapy Association. Following discussions with experts and clinical trials, a preliminary self-reported Activities of Daily Living Scale (sf-ADLS) was subsequently tested. The original scale, which comprised of just 12 items, was found to be too easy for people with mental illness (Pan et al., 2005); thus, a further eight items relating to community living were subsequently added. These additional eight items in the sf-ADLS proved to be more difficult than the original items (Fan & Pan, 2007; Pan et al., 2006).

The sf-ADLS has the strengths of knowledge-based, developed and discussed by experienced occupational therapists in Taiwan, and tested for its psychometric qualities on groups of persons with mental illness in Taiwan. Furthermore, the client has the choice of ‘I don’t have the need’ to express their unique status on certain activity. Each item was rated depending on the level of difficulty to complete the activity. All of these would make the scale being more culture-free and applicable in Taiwan.

Our primary aim in the present study is to examine the validity and reliability of the current sf-ADLS, using a Rasch measurement model for our analysis of the dataset. In contrast to classical test theory, Rasch analysis has the following unique characteristics: (i) the ability to convert ordinal raw data into interval data by using the logarithm of the odds ratio of persons to items; (ii) the ability to provide a difficulty level for each item and match these with a specific patient’s ability, thereby enabling therapists to adjust their treatment goals; (iii) the hypotheses of the model addressing the item-free and sample-free issues of a scale to propose a true measure of ability and (iv) the resultant item calibrations and person measures can be used to further develop ‘computerized-adaptive testing’ (Bone, 2016; Velozo, Kielhofner, & Lai, 1999).

Methods

Participants

The sample for this study comprised of psychiatric patients and a healthy control sample group. All the patients suffering from mental illness were recruited from the inpatient ward and outpatient visit of the psychiatry department of a medical centre in northern Taiwan, whilst the healthy control sample was made up of students recruited from two colleges in Taiwan. The study was approved by the Institutional Review Board of the affiliated institution. The criteria for inclusion in the sample were that all participants must be: (i) above 18 years of age; (ii) willing to participate in the study and (iii) fully able to complete the questionnaire. The persons with mental illness received the official diagnosis by attending psychiatrists based on DSM-IV.

Instruments

The sf-ADLS developed by Pan et al. (2005) included a total of 12 items in original version and 20 items in the second version, comprising of daily activities such as grooming, taking medication, transportation, healthcare, sleep patterns, and so on (Pan et al., 2005). Clients self-report their perceived level of difficulties (the level of assistance required) based on their experiences over the previous two-week period. The responses use a four-point Likert-type rating scale, where a score of 1 represents total dependence, whilst a score of 4 represents total independence. It takes less than 10 minutes to complete the self-reported scale. A pilot test of the sf-ADLS and revision of the scale revealed that it had good internal consistency of 0.92 and 0.88 (version II-20 item) (Fan & Pan, 2007; Pan et al., 2005, 2006).

Statistical analysis

Rasch analysis was applied in this study based on the use of the WINSTEPS Rasch computer program, Version 3.63.3 (Linacre, 2006) and the adoption of a partial credit model. The partial credit model indicates that each item has its own rating scale structure. By using the partial credit model, we are provided with the opportunity to estimate the item parameters independently of the characteristics of the calibrating sample; such an approach is also indifferent to the characteristics of the individual being assessed (Masters, 1982). The demographic data were analysed using the PASW computer program, Version 18.0.0 (SPSS, 2009). The six-step process of the data analysis is described below (Chen, Pan, Chung, & Chen, 2015).

Phase 1: Rating scale analysis

We adopted the general rules that are widely suggested within the related literature, with the requirements being: (i) a minimum of 10 observations in each category and regular observation distribution across the categories; (ii) monotonically increasing average measures across the categories; (iii) monotonically increasing step calibrations and (iv) category outfit mean square values of <2.0. Given that a few disordered thresholds were observed on certain categories of the items, the decision was taken to collapse those categories and re-run the analysis until a qualified rating scale structure was successfully formulated (Linacre, 2002).

Phase 2: Unidimensionality

There is a requirement for several criteria to be examined in order to achieve the characteristic of unidimensionality. The criterion for evaluating item goodness-of-fit is to accept infit MnSq logit values between 0.6 and 1.4 and an associated Zstd value of <2 or >−2 (Arnadottir & Fisher, 2008; Chen et al., 2015), where MnSq refers to the mean-square statistics and Zstd denotes the mean-square t statistic fit standardised to approximate a theoretical ‘unit normal’.

The criteria of the principle component analysis of the residuals require that the observations are capable of explaining at least 60% of the unidimensionality (Arnadottir & Fisher, 2008). The first contrast in unexplained variance is found to be less than 5%, which demonstrates a lower probability of explaining a potential secondary dimension. The criteria of unidimensionality were set in such a way that the eigenvalues are less than 3 (Linacre & Wright, 1999).

Phase 3: Targeting

The item-person map has the distinct advantage of aligning person measures with item calibrations on the same interval scale (Bond, 2004). The items should cover a sufficient range of difficulty, and thus, be capable of discriminating between groups of people with different abilities (Duncan, Bode, Lai, & Perera, 2003), essentially because this facilitates the identification of those items that are generally found to provide poor assessment of areas of ability.

As noted by Lai et al. (2005), the ‘Test Information Function’ (TIF) is defined as the reciprocal of the radical of the ‘standard error measurement’ (SEM) function (TIF = 1/√SEM), which represents the maximum level of accuracy to which the independence level of an individual can be measured along the latent continuum (Chen et al., 2015).

Phase 4: Reliability analysis

In order to describe the reliability of the instrument for our sample, we produced a separation index (G) to indicate the number of distinct strata of people discerned within each domain (Duncan et al., 2003). With a separation of 3, as represented by a coefficient of 0.9, reliability is considered to be excellent. A separation of 2 enables the distinction of three strata of people based on computation of the strata formula = (4G + 1)/3 (Duncan et al., 2003; Nilsson, Sunnerhagen, & Grimby, 2005).

Phase 5: Differential item functioning

Differential item functioning (DIF) occurs when items display bias for a particular subgroup of the sample (Amin et al., 2012). For the study, we made DIF comparison for the diagnosis group.

Phase 6: Rasch stratified score

The number of statistically different function levels can be measured by computing the standard errors of each raw score, with the principle being to advance each time by twice the joint standard error (the square-root of the sum of the squared standard errors) of the start and end measures until there is no room for another level (Wright, 2001).

Results

Participants

A total of 475 subjects who had completed the SF-ADLS were included in our subsequent Rasch analysis. The baseline characteristics of the psychiatric patients and the healthy control group are provided in Table 1. The mean age was 46.42 (SD = 12.99) for people with mental disorder and 21.59 (SD = 1.44) for the healthy controls.

Table 1.

Demographic characteristics of the subjects (N = 455).

Variables	People with mental disorder (N = 224)	Healthy control group (N = 231)
	No. (%)	No. (%)
Age (range), Mean (SD)	46.42 (12.99) Range 18–78	21.59 (1.44) Range 18–26
Gender
Male	68 (30.3)	79 (34.2)
Female	156 (69.7)	152 (65.8)
Status
Student		231 (100)
In-patient	40 (16.5)
Out-patient	203 ( 83.5)
Diagnosis
Schizophrenia	10 (4.5)
Affective disorder	197 (87.9)
Anxiety disorder	4 (1.8)
Eating disorder	1 (0.5)
Somatoform disorder	5 (2.2)
Personality disorder	1(0.5)
Others	3 (1.3)
Combination	3 (1.3)

Phase 1: Rating scale analysis

Items 1–6, 8–17 and 20 failed to meet the minimum requirement of 10 observations per category; after considering their theoretical significance, six of the items (items 1, 10, 11, 12, 13 and 14) were eliminated. Condensing the item-rating categories also facilitated the elimination of underutilised categories in the remaining 14 items. Firstly, the modifications eliminated the redundancy of the rating categories that were underutilised in the original scale; and secondly, they preserved the purpose of the rating scale by ensuring that each rating was distinctly separated from every other rating category (Pallant & Tennant, 2007). As shown in Table 2, except items 9 and 15 which were collapsed by combining rating scale of 1, 2 and 3 into score of 1. Item 2 was collapsed by combining 2 and 3 into score of 2, score of 4 as score 3. The other items were collapsed by combining rating scale 1 and 2 to form score of 1.

Table 2.

sf-ADLS item statistics, sorted by frequency of endorsement.

Items (Cronbach’s ɑ = 0.96)		Item-total correlation	Item calibration	Infit		Scoring structure
				MnSq	ZSTD
16	Sleep	0.55	1.28	1.30	4.0	1123
17	Child raising	0.62	0.97	1.08	0.8	1123
19	Emergencies	0.68	0.89	0.86	–2.0	1123
18	Care giving	0.66	0.51	0.83	–2.2	1123
8	Finances	0.59	0.35	0.88	–1.4	1123
3	Meal preparation	0.45	0.20	1.28	3.3	1123
20	Healthy lifestyle	0.64	0.13	0.80	–2.6	1123
4	House chores	0.49	–0.03	1.10	0.9	1123
9	Nutrition	0.45	–0.19	0.92	–0.7	1112
15	Sex	0.36	–0.43	1.02	0.2	1112
5	Laundry	0.45	–0.61	0.93	–0.4	1123
6	Transportation	0.48	–0.62	0.99	0.0	1123
2	Shopping	0.44	–1.17	0.97	–0.2	1223
7	Medication	0.32	–1.26	1.24	1.3	1123

As shown in Table 2, as a result of the revised scoring structure, all of the remaining 14 items have a good fit with the criteria.

Phase 2: Unidimensionality

The results of the fit analysis showed that all remaining items fit with the model with sleep and child-rearing items represented the most difficult items. The shopping and medication management items are the easiest items of all. The 14 items accounted for 54.4% of the explained variance for the major factor, and also for 5% of the unexplained variance on the first contrast. Although this does not meet all of the criteria for unidimensionality, we decided that this was acceptable on a theoretical basis because each item represents a unique domain which a person requires to be independent (see Appendix 1).

Phase 3: Targeting

The easiest item of the sf-ADLS was item 7, ‘Taking medication’, whilst the most difficult item was item 16, ‘Sleep’. There was a ceiling effect for ceiling for the sf-ADLS as illustrated in Figure 1 (Pallant & Tennant, 2007). These items were too easy for measuring the abilities of people with mental illness unless they were very ill (Velozo, Magalhaes, Pan, & Leiter, 1995).

Figure 1.

Item-person map of sf-ADLS.

Note: The vertical line represents the measure of difficulty, with logit values given on two sides. The item difficulty levels perceived by a person are represented as ‘X’ symbols and aligned to the left of the corresponding measure. Items are aligned on the middle of the corresponding values. The accumulation of patients in the uppermost area of the scale indicates a ‘ceiling’ effect.

The sf-ADLS provides a reliable measure (SEM <0.5) of people with mental illness, with the scale scores of between −1.88 and 0.98 from the 14 items capturing 17% of the sample, as illustrated in Figure 2.

Figure 2.

SEM and TIF for sf-ADLS.

X-axis represents level of item difficulty from left (easiest) to right (most difficult); left side of Y-axis represents test information function, right side of Y-axis represents standard error of measurement (SEM).

Note: The reliably measured sample (SEM <0.5) accounted for 17% of the total.

Phase 4: Reliability analysis

The results show excellent item separation index and item reliability (5.12; 0.96) representing seven strata of items.

Phase 5: DIF

The results of the DIF analysis showed that there was no DIF item across diagnosis (normal vs. patient) groups.

Phase 6: Rasch cut-off score

The resultant person measures ranged between –4.69 and 4.87, and we subsequently developed a conversion table based on the Wright (2001) method – which enables clinicians to identify the level of independence. As shown in Table 3, there were four statistically distinct levels of independence.

Table 3.

sf-ADLS cut-off scores.

Level	Person calibration	Raw score
1	–4.69 to –0.93	0–21
2	–0.93 to 0.34	22–28
3	0.34 to 1.66	29–34
4	1.66 to 4.87	35–40

Discussion

The results of the study support the reliability and validity of the sf-ADLS via Rasch measurement model. It is noted that the six items deleted were items related to basic ADL except for making a phone call. The reasons for omitting these items are because of the misfit of these items plus the fact that the persons with mental illness were different from persons with physical disability in that motor deficits were not the main reason for functional deficits (Pan & Fisher, 1994); thus, the basic ADLs do not provide information to differentiate the functional level of persons with mental illness. The order of item difficulty of the sf-ADLS showed that the top three easier items were medication management, shopping and transportation. The top three harder items were sleep, rearing children and handling emergency. With the order of activity difficulties in mind, the occupational therapist is able to adjust the treatment planning to fit the client’s needs (Gantschnig, Page, & Fisher, 2012). Since there were a few rating scale categories used less often, a few of them were combined to form a practical rating scale for use in clinical practice according to Chen et al. (2015).

The current analysis adapted Rasch measurement model which enables a detailed understanding of the psychometric qualities of the sf-ADLS. By adapting the cut-off score of the persons, we are able to apply the sf-ADLS to classify people with mental illness into different level of functioning. Although six items were deleted from the original sf-ADLS (de Morton, Keating, & Davidson, 2008), the remaining items enable the formulation of a unidimensional scale for measuring the functional levels of people with mental illness.

We did find that some of the subjects achieved maximum score (5.71%) which is acceptable. A limitation of the sf-ADLS analysis in the present study was the lack of people with more severe levels of functioning; thus, we suggest that acute stage patients or persons with severer functional deterioration should also be included in any future studies. The other limitation is the use of the level of difficulties as the rating criteria, which would hinder the identification of specific deficit of the performance of the task. On the other side, it would provide an overall picture of the client’s performance, disregarding specific task elements or steps which would be variable from culture to culture and from time to time.

Conclusion

In conclusion, the sf-ADLS can be readily applied to people with mental illness with appropriate validity and reliability. Occupational therapists can use the scale to measure the functional level status of people with mental illness and then apply the outcome measure when making their intervention decisions.

Appendix 1. Final items of the sf-ADLS (English translation)

Revised sf-ADLS
1. Shopping
□ I don’t have the need □ I have no difficulty going shopping	□ I have some difficulty going shopping □ I can not go shopping independently
2. Meal preparation
□ I don’t have the need □ I have no difficulty preparing my meals	□ I have some difficulty doing household chores □ I have extreme difficulty preparing my meals
3. Household chores
□ I don’t have the need □ I have no difficulty doing household chores	□ I have some difficulty doing household chores □ I have extreme difficulty doing household chores
4. Laundry
□ I don’t have the need □ I have no difficulty doing my laundry	□ I have some difficulty doing my laundry □ I have extreme difficulty doing my laundry
5. Transportation
□ I don’t have the need. □ I have no difficulty using public transportation	□ I have some difficulty using public transportation □ I have extreme difficulty using public transportation
6. Medication
□ I don’t have the need □ I have no difficulty taking medication	□ I have some difficulty taking medication □ I have extreme difficulty taking medication
7. Finances
□ I don’t have the need □ I have no difficulty managing my finances	□ I have some difficulty managing my finances □ I have extreme difficulty managing my finances
8. Nutrition
□ I don’t have the need □ I have no difficulty eating healthy foods	□ I have difficulty eating healthy foods
9. Sex
□ I don’t have the need □ I have no difficulty engaging in intercourse	□ I have difficulty engaging in intercourse
10. Sleep
□ I don’t have the need □ I have no difficulty sleeping	□ I have some difficulty sleeping □ I have extreme difficulty sleeping
11. Child raising
□ I don’t have the need □ I have no difficulty raising my child	□ I have some difficulty raising my child □ I have extreme difficulty raising my child
12.Care giving
□ I don’t have the need □ I have no difficulty taking care of others	□ I have some difficulty taking care of others □ I have extreme difficulty taking care of others
13. Emergencies
□ I don’t have the need □ I have no difficulty handling emergency situationsand staying safe	□ I have some difficulty handling emergency situationsand staying safe □ I have extreme difficulty handling emergencysituations and staying safe
14. Healthy lifestyle
□ I don’t have the need □ I have no difficulty maintaining a healthy lifestyle	□ I have some difficulty maintaining a healthy lifestyle □ I have extreme difficulty maintaining a healthy lifestyle

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: The study was funded by the Ministry of Science and Technology of Taiwan (MOST 106-2511-S-002-016-MY2).