The study of the validity and reliability of the Occupational Self-Assessment-traditional Chinese version

Abstract

Introduction

Health professionals who work in mental healthcare settings need to use standardised, objective instruments; however, it is also extremely important that they maintain a client-focused perspective. The purpose of this study was to investigate the validity and reliability of the ‘Traditional Chinese version of the Occupational Self-Assessment’ (TC-OSA).

Methods

A total of 593 participants with mental illnesses participated in the study. The data were analysed using confirmatory factor analysis as well as the Rasch measurement model.

Results

The results of our analysis revealed that the TC-OSA encompasses four domains: self-performance (11 items), self-habituation (5 items), self-volition (5 items) and environment (8 items). Most of the items within each domain were found to have a good fit with the Rasch measurement model, whereas the CFA index was found to have a good fit for only three of the domains, the one exception being the environment domain.

Conclusions

We suggest applying the scale in clinical practice to identify the priority of intervention and as a measure for changes in outcomes. Further development and refinement of the environmental domain is guaranteed.

Introduction

There is growing awareness that people are faced with greater stress and environmental pressures than ever before (American Psychological Association, 2017), which result in such people being increasingly vulnerable to stress-related diseases. Occupational therapists (OT), as important members of the psychiatric professional team, have a long history of providing psycho-social evaluations and programs to clients with psychiatric disorders; thus, the involvement of OT practitioners in the treatment of clients with mental illnesses could provide additional care perspectives in psychiatric healthcare delivery. Since clients with mental disorders, such as mood disorders or anxiety disorders, often differ from those with psychosis, in terms of both their characteristics and functional prognosis (Wang et al., 2003), the development of adequate assessment tools for the evaluation of clients with mental illnesses is crucial for OT practitioners.

An appropriate assessment tool for use in evaluating the strengths and limitations of clients is essential to the provision of quality care; furthermore, it is important that the instrument contents are both consistent with professional philosophy and applicable to the clients being served (Kielhofner et al., 2009). Since clients with mental disorders represent a group of people with a wide variety of problems, prognoses and functional outcomes, they need to be viewed from a holistic viewpoint, and should be empowered so that they can maintain control of their own lives. There were a few reviews of assessment tools for use in mental health occupational therapy practices (Hsiao et al., 2000); among those assessment being reviewed, we only adapted the comprehensive occupational therapy evaluation (Chiu et al., 2019; Wang et al., 2012), the Canadian Occupational Therapy Measure (Pan et al., 2007) and the kinetic house tree person drawing (Li et al., 2014) in Taiwan. We do have our own assessment tools developed in our own culture such as the Chu’s hand function text, attention test and activity of daily living test and Pan’s self-reported daily living test (Chu & Hsieh, 2004; Pan et al., 2018). However, none of the aforementioned tools were developed based on client-centred approach and using self-reported format.

Thus, suitable approaches to be adapted for this population are the ‘model of human occupation’ (MOHO; original developer Prof Keilhofner) (Taylor, 2017) and a client-centered methodology (Law et al., 2005). MOHO, which addresses issues of occupational functioning from a systemic view whilst also providing a paradigm for OT practitioners, proposes that volition, habituation, performance and environment systems are all extremely important elements for individuals to function competently in society.

The volition system is conceptualised as a collection of thoughts and feelings pertaining to an individual’s abilities and effectiveness; personal causation, interests and values are the three components of volition delineating a person’s sense of capacity and efficacy, their preferences and their everyday life patterns. The habituation system comprises of role scripts and habit maps which are rules and expectations that enable an individual to act automatically and fit in with environmental conditions. The performance system represents the capabilities that are fundamental to the skilled performance of an individual; these capabilities comprise of motor, process and communication skills. Finally, the environment system includes social and physical aspects that enable or constrain the adaptability of individuals in their daily lives (Taylor, 2017).

A client-centred approach is an additional theoretical thought process linked to occupational self-assessment (OSA) (Baron, 2006) which places the emphasis on consumer rights and active participation. When a client-centred approach is adopted within clinics, it has the effect of empowering the clients to choose their own goals and set up treatment priorities for themselves (Law et al., 2005).

OSA was developed from the ‘Self-Assessment of Occupational Functioning’, which was underpinned by the MOHO and client-centred approach. Although the OSA has been shown by researchers to have appropriate psychometric qualities (construct validity, sensitivity, reliability, ability to detect change and scale validity) (Kielhofner et al., 2009, 2010; Taylor et al., 2011), it was felt that it would be more appropriate to fit the 29 items into four domains, since this would make it compatible with the MOHO.

The OSA is a self-reported questionnaire which addresses self and environmental domains, each item is rated on both competence and value perspectives on a four-point rating scale. The items for each system within the MOHO comprise of self-performance (11 items), self-habituation (5 items), self-volition (5 items) and environment (8 items). The importance of translating OSA into traditional Chinese for use in Taiwan is to provide a client-centred and self-reported comprehensive assessment for use in treatment planning and progress evaluation. Furthermore, for the translated scales to be comparable and equally adapted as the original tool, the validation of the translated scales is important to ensure the proper psychometrics of the scale (Fisher et al., 1992). The primary aim of the study was to validate the translated Traditional Chinese version of the Occupational Self-Assessment (TC-OSA) (focus on the competence part) on a group of clients with mental illnesses in Taiwan.

The hypotheses of the study are listed below:

1. There was accepted internal consistency of the translated OSA subscales.

2. There were four unidimensional constructs for OSA-defining volition, habituation, performance and environment domains of MOHO by Rasch analysis and confirmatory factor analysis (CFA).

3. The level of item difficulties for each domain was adequate.

4. There was adequate item and person separation for each domain (at least above 2).

Methods

Translation

We obtained the translation permit and translated the OSA into traditional Chinese. A bilingual person subsequently carried out a reverse translation of the OSA forms, after which we examined the differences between the two versions and carried out appropriate changes to ensure the reconciliation of any differences between the two versions. The final version of the OSA was determined after experts agreed on its readability, comprehensibility and congruence with the original version.

Participants and procedures

Our study sample, which was obtained from the Department of Psychiatry of a university affiliated hospital, comprised of a total of 593 patients suffering from mental disorders, their diagnoses include schizophrenia, affective disorder, anxiety disorder and ‘others’ (Table 1). The subjects were recruited from the inpatient ward with stable medication use and symptoms, and from outpatient clinics which they attended follow-ups by psychiatrists regularly. The study was approved by the Institutional Review Board of a related institution. We analysed a total of 232 males (39.1%) and 361 females (60.9%), with a mean age of 37.79 (SD = 15.03). The sample for analysis was based on the self-reported TC-OSA and data on personal characteristics (such as gender and age).

Table 1.

Demographic data on participants (N = 593, observation = 702).

	N	%	Mean	SD
Age	593		37.79	15.03
Gender	593
Male	232	39.1
Female	361	60.9
Diagnosis	526
Schizophrenia	129	21.75
Affective disorder	382	64.42
Anxiety disorder	5	0.8
Somatoform disorder	9	1.52
Personality disorder	1	0.2
Missing	67	11.3

Traditional Chinese version of Occupational Self-Assessment

The TC-OSA questionnaire comprises of 29 items scored on a four-point Likert scale, indicating the level of competence in the performance of an activity. The scale ranges from 1 (no discernible problem) to 4 (serious problem), with the 29 items being organised into four domains comprising of self-performance (11 items), self-habituation (5 items), self-volition (5 items) and environment (8 items). These four domains are designed to evaluate personal occupational capabilities (for example, ‘physically doing what I need to do’) and environmental resources (such as ‘the things I need to be productive’).

Data analysis

In order to determine both the reliability and validity of the TC-OSA scale for people with mental disorders, we carried out an evaluation of the internal consistency of the scale by calculating the Cronbach’s coefficient alpha for the total scale scores (Cronbach, 1951). The data sample was subsequently examined based upon ‘confirmatory factor analysis’ (CFA) using ‘structural equation modelling’ (SEM) and the LISREL statistical program (Jöreskog & Sörbom, 2017). The ‘comparative fit index’ (CFI > 0.9), ‘goodness-of-fit index’ (GFI > 0.9), Adjusted goodness of fit index (AGFI > 0.9) and the ‘root mean square error of approximation’ (RMSEA ≤0.08) were used for the criteria (Bentler, 1990; Steiger, 1990). The confirmatory factor analytic models were evaluated using the ‘maximum likelihood’ (ML) estimation, an approach known to be robust to departure from normality. Given that our approach in this study was to confirm the hypothesised four domains of the TC-OSA scale: self-performance, self-habituation, self-volition and environment, our research methodology involves the evaluation of a total of four models.

We employed the WINSTEPS software (Linacre, 2005) to carry out a Rasch model analysis (Rasch, 1960), with this model being used to estimate both item and person fit on a common scale. The appropriateness of the rating scale for each item was examined according to the following criteria: (Chen et al., 2015) (1) at least 10 observations in each category and a regular observation distribution across categories, (2) monotonically increasing average measures across categories, (3) monotonically increasing step calibrations and (4) category outfit mean square (MnSq) values <2.0.

Results

The demographic data of the participants are presented in Table 1 (as discussed above). The results on the reliability for each domain and the overall TC-OSA are reported in Table 2. The Cronbach’s alpha coefficients in this study were found to be 0.853 for the self-performance domain, 0.823 for the self- habituation domain, 0.856 for the self-volition domain and 0.811 for the environment domain, whilst the Cronbach’s alpha for the overall TC-OSA scale was 0.934. Thus, hypothesis one is supported. These results clearly indicate that the scale items have good internal consistency. The item-total correlation coefficients are also presented in Table 2, with the results showing that there were moderate to high correlations between the items (that is, 0.36 or above).

Table 2.

Reliability of the TC-OSA scale.

	Item	Cronbach’s Alpha	Item-total correlation range
Self-performance	1–11	0.853	0.446–0.612
Self-habituation	12–16	0.823	0.564–0.648
Self-volition	17–21	0.856	0.576–0.709
Environment	22–29	0.811	0.435–0.604
TC-OSA scale	1–29	0.934	0.360–0.675

TC-OSA: Traditional Chinese version of the Occupational Self-Assessment.

We carried out tests on the goodness-of-fit for the four different models and the empirical outcomes. The results are summarised in Table 3, which shows that the environmental domain did not fit, whilst self-performance, self-habituation and self-volition did. Thus, we are willing to accept the domain of self-performance, self-habituation and self-volition as an unitary construct within the MOHO model.

Table 3.

Comparative goodness-of-fit statistics for each domain.

Fit indices	Recommended value	Self-performance (1–11)	Self-habituation (12–16)	Self-volition (17–21)	Environment (22–29)
χ 2	N/A	2694.88	1332.17	1636.75	2087.58
df	N/A	55	10	10	28
CFI	≥0.90	0.86	0.96	1.00	0.71
GFI	≥0.90	0.87	0.96	0.99	0.79
AGFI	≥0.90	0.86	0.89	0.98	0.62
RMSEA	≤0.08	0.121	0.135	0.047	0.224

CFI: comparative fit index; GFI: goodness of fit index; AGFI: adjusted goodness of fit index; RMSEA: root mean square error of approximation.

Whilst the first model was rejected due to the large sample size (χ² = 2649.88, df = 55, p <0.001), the CFI, GFI, AGFI and RMSEA for this model barely met the criteria. The second model fit the criteria except for AGFI and RMSEA. The third model also fit the criteria. The final model of the TC-OSA did not fit with the criteria. The second hypothesis is partially supported by CFA.

The rating scale structure for all four models fit based on the suggested criteria. The results of the self-performance domain showed that the internal consistency values for items and persons as 0.97 and 0.83, these values were satisfactory; as shown in Table 4, all items in the self-performance domain fit well. The most difficult item was ‘Taking care of others for whom I am responsible’, whilst the easiest item was ‘Managing my basic needs’. The gender variable revealed differences between males and females in the ‘differential item functioning’ (DIF) for this domain. The scale can classify clients into seven levels of performance ability.

Table 4.

Self-performance domain item statistics in item measure order.

Item		Measurea	S.E.b	Infit		Outfit		DIFc,d
				MnSq	ZSTD	MnSq	ZSTD
8.	Managing my basic needs	–1.00	0.07	1.07	1.20	1.01	0.10	0.16
4.	Taking care of myself	–0.25	0.06	0.90	–1.90	0.87	–2.30	0.04
6.	Getting where I need to go	–0.12	0.06	1.16	2.90	1.18	3.10	–0.05
9.	Expressing myself to others	–0.02	0.06	0.95	–0.80	0.92	–1.40	0.06
3.	Taking care of the place where I live	0.02	0.06	0.94	–1.20	0.93	–1.30	–0.11
10.	Getting along with others	0.09	0.06	0.91	–1.70	0.90	–1.80	0.18
7.	Managing my finances	0.14	0.06	1.18	3.30	1.22	3.40	–0.17
2.	Physically doing what I need to do	0.17	0.06	0.96	–0.70	0.94	–1.10	0.09
11.	Identifying and solving problems	0.28	0.06	0.88	–2.50	0.88	–2.20	–0.40*
1.	Concentrating on my tasks	0.32	0.06	1.01	0.20	1.05	0.90	0.18
5.	Taking care of others for whom I am responsible	0.38	0.06	0.98	–0.40	0.97	–0.40	0.06

^aMeasure value refers to personal ability.

^bModel S.E. refers to the standard error of the estimate.

^cThe ‘differential item functioning’ (DIF) refers to the difference between males and females.

^d*Significance of the Bonferroni multiple-comparison correction: p = 0.05 divided by the DIF value (0.05/11 = 0.0045).

The results of the self-habituation domain showed that the internal consistency values for items (0.97) and persons (0.79) indicated satisfactory to very high internal consistency; as shown in Table 5, all items in the self-habituation domain fit well. The most difficult item was ‘Relaxing and enjoying myself’ whilst the easiest item was ‘Handling my responsibilities’. No DIF was discernible for this domain. The scale can classify clients into six levels of habitation capability.

Table 5.

Self-habituation domain item statistics in item measure order.

Item		Measurea	S.E.b	Infit		Outfit		DIFc
				MnSq	ZSTD	MnSq	ZSTD
15.	Handling my responsibilities	–0.73	0.07	0.98	–0.30	0.95	–0.80	2.00
13.	Getting done what I need to do	–0.15	0.07	0.92	–1.50	0.90	–1.70	2.00
16.	Being a student, worker, volunteeror family member	0.13	0.07	0.96	–0.70	0.95	–0.90	–2.00
14.	Having a satisfying routine	0.17	0.07	0.96	–0.80	0.96	–0.70	–0.70
12.	Relaxing and enjoying myself	0.58	0.07	1.14	2.60	1.17	3.00	–2.20

MnSq: mean square; ZSTD: z-standardized.

^aMeasure value refers to personal ability.

^bModel S.E. refers to the standard error of the estimate.

^cThe ‘differential item functioning’ (DIF) refers to the difference between males and females.

The results of the self-volition domain showed that the internal consistency values for items (0.94) and persons (0.82) indicated satisfactory to very high internal consistency; as shown in Table 6, all items in the self-volition domain fit well. The most difficult item was ‘Working towards my goals’, whilst the easiest item was ‘Doing activities I like’. No DIF was discernible for this domain. The scale can classify clients into four levels of volitional status.

Table 6.

Self-volition domain item statistics in item measure order.

Item		Measurea	S.E.b	Infit		Outfit		DIFc
				MnSq	ZSTD	MnSq	ZSTD
17.	Doing activities I like	–0.58	0.07	1.25	4.20	1.28	4.40	0.10
21.	Effectively using my abilities	0.01	0.07	0.96	–0.70	0.94	–1.10	–3.90
19.	Making decisions based on what I think is important	0.04	0.07	0.98	–0.30	0.97	–0.50	–1.30
20.	Accomplishing what I set out to do	0.17	0.07	0.90	–1.90	0.86	–2.50	2.90
18.	Working towards my goals	0.36	0.07	0.89	–2.00	0.86	–2.40	2.10

MnSq: mean square; ZSTD: z-standardized.

^aMeasure value refers to personal ability.

^bModel S.E. refers to the standard error of the estimate.

^cThe ‘differential item functioning’ (DIF) refers to the difference between males and females.

The results of the environment domain showed that the internal consistency values for items (0.99) and persons (0.80) indicated satisfactory to very high internal consistency; as shown in Table 7, all items in the environment domain fit well. The most difficult item was ‘A place to live and take care of myself’, whilst the easiest item was ‘The basic things I need to live and take care of myself’. The gender variable revealed DIF for this domain. The scale can classify clients into eight levels of environmental resources.

Table 7.

Environment domain item statistics in item measure order.

Item		Measurea	S.E.b	Infit		Outfit		DIFc
				MnSq	ZSTD	MnSq	ZSTD
22.	A place to live and take care of myself	–0.58	0.06	1.12	2.10	1.11	1.90	0.04
26.	People who support and encourage me	–0.18	0.06	1.03	0.70	1.05	0.90	0.34*
28.	Opportunities to do things that I value and like	0.16	0.06	0.97	–0.50	0.98	–0.40	0.07
25.	The things I need to be productive	0.27	0.06	1.04	0.70	1.06	1.10	–0.28
23.	A place where I can be productive (work, study and volunteer)	0.33	0.06	1.03	0.60	1.05	1.00	–0.26
27.	People who do things with me	0.36	0.06	0.87	–2.50	0.87	–2.60	0.13
29.	Places where I can go and enjoy myself	0.51	0.06	0.93	–1.50	0.92	–1.60	0.00
24.	The basic things I need to live and take care of myself	0.87	0.07	0.98	–0.30	0.97	–0.60	0.02

MnSq: mean square; ZSTD: z-standardized.

^aMeasure value refers to personal ability.

^bModel S.E. refers to the standard error of the estimate.

^cThe ‘differential item functioning’ (DIF) refers to the difference between males and females.

^*Significance of the Bonferroni multiple-comparison correction: p = 0.05 divided by the DIF value (0.05/8 = 0.0063).

The second, third and fourth hypothesis are supported by Rasch analysis.

Discussion

The TC-OSA was developed as a mean of assessing the occupational competence and values for specific tasks undertaken by people with various needs, whilst our study demonstrates that the OSA is a four-domain scale encompassing self-volition, self-habituation, self-performance and environment. The results of the confirmatory factor analysis, internal consistency and item-total correlations provide evidence of unidimensionality of the self-performance, self-habituation, and self-volition domain from a classical test theory perspective. Nevertheless, since the results of the factor analysis could vary due to sampling differences, the results of the CFA should be adapted carefully (Wang, 1997). The Rasch analyses demonstrated that all four domains of the TC-OSA fitted well, with gender DIF found for self-performance (item 11) and environment (item 26). Since this study is one of several studies that applied translated OSA to a group of clients with mental illnesses in Taiwan and demonstrated that there are acceptable cross-cultural psychometric qualities for TC-OSA, it means that OTs who work with Taiwanese clients can readily apply it to generate client-centred treatment goals and monitor their progress.

The TC-OSA has important implications in clinical settings since the order of item difficulty can be used as an index to guide treatment priorities. For example, if a client can take care of his or her own needs, then the next task to train would be ‘Getting where I need to go’. Another example would be if a client has no problem of having ‘People who support and encourage me’, then the therapist would move to identify ‘People who do things with me’. Our approach reveals that the occupational competence of an individual can be seen from the perspectives of volition, habituation, performance and environment. For example, a client with Schizophrenia may have low motivation toward activity participation (volition), but he or she might be able to compensate by establishing daily routine or skill competence which both contribute to competent functional performance. Thus, the TC-OSA offers a way to identify the strengths and weaknesses of our clients and provides a framework to target treatment.

Our results were different from previous literature (Kielhofner et al., 2010; Taylor et al., 2011) in that our study results support the four dimensions (domains) including volition, habituation, performance and environment, whilst the environmental domain will need further development. Therefore, OTs can adapt OSA scores to understand the volitional, habitational or performance status of the client. Our argument is that volition, habituation, performance and environment are related, but demonstrate distinct domains.

In conclusion, our study shows that the reliability and validity of the TC-OSA are acceptable; thus, we suggest that therapists use the domain scores to represent the capabilities of their clients. However, the environment domain clearly needs to be modified to become an independent construct. The limitations of our study include the limited variety of the diagnostic group and the absence of measures of the cognitive competence of the participants who completed the self-reported questionnaire. There was also a lack of cross-cultural study of test–retest reliability, criterion-related validity and minimally detectable changes. In the future, there are needs to conduct research which include a more diverse sample with different capability level to demonstrate the sensitivity of the TC-OSA. Furthermore, it is necessary to modify the environmental scale so that the OT can apply it to allocate resources for our clients.

Contributorship

The first author drafted the article, analysed the data and finalised the article. The second author analysed the data and approved the results. The third author designed the acquisition of the data and contributed to the idea of the research. The fourth author assisted to acquire the data and collaborated on the data analysis and results interpretation.

Ethical approval

The data source was from several studies listed below. NTUH 9461701109 2005, NTUH 200804025R 2008, NTUH: National Taiwan University Hospital.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was funded by the Ministry of Science and Technology of Taiwan (MOST 106-2511-S-002-016-MY2; MOST 108-2511-H-002-015-; MOST 103-2314-B-002-179-MY3).

ORCID iD

Ay-Woan Pan https://orcid.org/0000-0002-4793-2505