Effect of brief training on reliability and applicability of Global Assessment of functioning scale by Psychiatric clinical officers in Uganda

Abstract

Background

The Global Assessment of Functioning (GAF) is the standard method and an essential tool for representing a clinician's judgment of a patient's overall level of psychological, social and occupational functioning. As such, it is probably the single most widely used method for assessing impairment among the patients with psychiatric illnesses.

Objective

To assess the effects of one-hour training on application of the GAF by Psychiatric Clinical Officers' in a Ugandan setting.

Method

Five Psychiatrists and five Psychiatric Clinical Officers (PCOs) or Assistant Medical Officers who hold a 2 year diploma in Clinical Psychiatry were randomly selected to independently rate a video-recorded psychiatric interview according to the DSM IV-TR. The PCOs were then offered a one-hour training on how to rate the GAF scale and asked to rate the video case interview again. All ratings were assigned on the basis of past one year, at admission and current functioning. Interclass correlations (ICC) were computed using two-way mixed models.

Results

The ICC between the psychiatrists and the PCOs before training in the past one year, at admission and current functioning were +0.48, +0.51 and +0.59 respectively. After training, the ICC coefficients were +0.60, +0.82 and +0.83.

Conclusion

Brief training given to PCOs improved the applications of their ratings of GAF scale to acceptable levels. There is need for formal training to this cadre of psychiatric practitioners in the use of the GAF.

Introduction

The Global Assessment of Functioning (GAF) is the standard method and an essential tool for representing a clinician's judgment of a patient's overall level of psychological, social and occupational functioning¹. As such, it is probably the single most widely used method for assessing impairment among the patients with psychiatric illnesses^1,2,3. The GAF is a 100-point scale and is listed as axis V in the 4^th edition of the Diagnostic and Statistical Manual (DSM IV-TR) ^4,5,6. Scores are used to monitor clinical progress, justify the level of care, determine eligibility for treatment, communicating to the courts of law in forensic psychiatry, as well as a tool for clinicians to communicate to their patients' clinical status and progress¹. Generally in clinical guidelines, any one scoring 50 and below may need admission for psychiatric evaluation and/or treatment.

Various studies have suggested that the GAF is a reliable clinical scale ^{7, 8}. The GAF is promoted as an assessment tool that is easily and quickly administered with minimal training. However, research has shown that training specific to GAF is a key element to its accurate use⁹.

The effects of brief training on the application of the GAF have been assessed and reported in some previous studies¹⁰. In a study by bates and others , 31 staff members of U.S Veterans Affairs Medical Centre assessed the GAF scores for patients in two case vignettes before and after one-hour . The authors reported that before training the raters, there was inter- and intra-rater inconsistency and that training improved consistency¹⁰. In Uganda neither the inter-rater reliability of this scale nor the effect of brief training is known.

Psychiatric clinical practice in Uganda

According to the Health care delivery structure in Uganda, clinical psychiatry services is included as one of the twelve components of the National Minimum Health care Package to be provided at all levels of health care¹¹. In order to ensure that mental health services are provided at all levels of care, several strategies have been adopted by Ministry of Health, one of which is recruiting and posting mental health professionals at various levels. There are 12 regional referral hospitals supposed to be headed by a psychiatrist, however, most of them are headed by a Psychiatric Clinical Officer (PCO) in an effort to task-shift because of shortage of psychiatrists¹². The PCOs are a cadre of mental health professionals who were originally nurses or general clinical officers with additional two year training in clinical psychiatry. The PCOs bridge the gap between nurses and psychiatrists in mental health services but where no psychiatrist exists, they assume the role of Psychiatrist albeit in a limited way.

During their training, the PCOs are learning to assess level of functioning of psychiatric patients using the GAF. This is useful for making patient management plans as well as for inclusion in forensic reports required by other institutions such as courts. However owing to their busy schedule and limited role in management planning, PCOs frequently do not do this assessment necessitating brief retraining when they are required to use GAF. It is imperative that use of GAF by PCOs and Psychiatrists be encouraged in order to ensure good clinical practice and quality of care.

This study examined the reliability of GAF, highlighting whether there are significant differences in the assessment of GAF scores by psychiatrists and Psychiatric Clinical Officers (PCOs), and whether a brief training of one hour given to PCOs in using the scale improves inter rater reliability.

Methods

The study was conducted at the Department of Psychiatry, Makerere University, Faculty of Medicine and Butabika National Mental Referral Hospital, Kampala. Ethical approval was obtained from the relevant University and national research regulatory bodies in Uganda. Video recording of a full psychiatric interview according to the DSM IV-TR was done^{1, 6}.

The patient, who had a diagnosis of schizophrenia and was known to the psychiatric services, was informed of the purpose of the interview and video recording to which he consented a day before the psychiatric interview. The psychiatrist in charge of the ward (interviewer), the video person, the first author and observer (another psychiatrist); were present in the interview room. The role of the observer was to listen and take notes of any parts of the interview that may have been missed. At the end of the interview, the observer and the first author asked questions to clarify issues that were not clear. The patient was then allowed to also ask questions and air out his concerns. The whole process lasted 1 hour and 15 minutes.

The video was then studied by five different Psychiatrists and five PCOs selected randomly. They then rated the GAF of the same patient interview independently. The PCOs were then offered a one-hour training that consisted of the introduction to the concepts of GAF and its categories. Practical examples using two case vignettes were given to illustrate how to give the ratings on severity of symptoms or level of functioning,, whichever was worse. The PCOs were asked to individually rate the video again. The raters did not communicate with each other on their respective ratings. The psychiatrist who was in charge of the patient and had evaluated him also rated the video recorded psychiatric interview. This was used as a ‘gold standard’. In this study, all ratings were assigned on the basis of current functioning, functioning at admission and in the past one year. This patient had been on the ward for two months. The ratings generated nine groups; the ratings of the psychiatrists for current functioning, rating at admission and that at one year prior to admission and similar PCO rating both before and after training.

Data analysis

One way between subjects analysis of variance (ANOVA) statistics were performed to compare the effects of one hour training on the application by PCO mean GAF score for the video recorded psychiatric interview. Table 1 gives the results of ANOVA for the 9 groups.

Table 1.

Results of ANOVA test for the 9 groups: Psychiatrist and PCO before and after training, each rating at one year, at admission and current

Source of variation	DF	Sum of Squares	Mean Square	F ratio	P value
Between groups	8	4052	506.5	3.703	0.003
Within groups	36	4925	136.8
Total	44	8977

Intra-class correlations (ICC) were computed to examine inter-rater reliability using SPSS version 11.0 software. The study specifically focused on two sets of fixed raters (the Psychiatrists and the PCOs), all of whom rated a defined video recorded psychiatric interview, thus a two way mixed ICC model based on consistency measurement was appropriate¹³. We considered ICC coefficients to be excellent when higher than +0.74, good if between +0.60 and +0.74 and as fair if between +0.40 and +0.60 ¹⁴.

Results

There was a statistically significant difference in rating among the nine groups (p=0.003). However, we did not know which groups were different as ANOVA statistics does not indicate the differences.

Table 2 below shows means, standard deviation and 95% Confident Intervals (CI) of the individual groups. Taking the upper limits of 95% CI, it is evident that there are differences in all the groups. The differences are greater between psychiatrist and PCOs ratings before training than after training. The inter-rater reliability was lowest in the past one year ( +0.48). Followed by that at admission (+0.51) and highest at current time (+0.59). This scenario between psychiatrist and PCOs rating changes after training. ICC coefficients changed +0.60 for the past one year, to,+0.82 at admission and to +0.83 at current time.

Table 2.

Means, Standard Deviation and 95% Confidence Intervals following the ANOVA test

	Psychiatrist			PCO before training			PCO after training
Group	1 year	Admission	Current	1 year	Admission	Current	1 year	Admission	Current
Mean	40	22	39.4	43	36	52.4	32.2	21.4	42.4
SD	11.9	3.39	9.84	22.8	15.2	13.3	4.49	2.19	4.34
95%CI	29.8–51.0	11.4–32.6	28.8–50.0	32.4–53.6	25.4–46.6	41.8–63.0	21.6–42.8	10.8–32.0	31.8–53.0

To examine if low inter-rater reliability between the psychiatrists group and PCO group was caused by only a few ratings, all cases with GAF score discrepancies of 20 or more points were identified. This threshold was selected because a difference between two raters of 20 or more implies that the two rated the patient in a clearly different category on the qualitatively defined intervals of the scale ². Out of 45 ratings, seven had deviating scores according to the threshold established above. The ratings with a discrepancy of 20 or more points included those between two ratings from the psychiatrists (1 at past one year and 1 current) and five PCOs all before training (1 past one year, 2 current and 2 at admission). Only one PCO had three deviating scores. The rest of the deviating scores were by deferent raters.

To examine how far away or near the ratings were to the ‘gold standard’, we compared the group mean ratings to the ratings given by the psychiatrist who was in charge of the patient. His rating was as follows: Past one year=37, Admission=22 and Current=39.

The psychiatrists' mean ratings were closest to the ‘gold standard’, followed by PCOs after training, while PCO mean ratings before training deviated the most from ‘gold standard’.

Discussion

To our knowledge, this is the first study on the effects of one hour training on applicability of the GAF scale by PCO s in a Ugandan setting, comparing the GAF score assessment by medical professionals at different levels of training. The results indicate that PCOs assigned variedly higher GAF scores than psychiatrists before training in the video recorded psychiatric interview. Theoretically, a higher GAF score estimation could lead to patients receiving lower levels of care than needed.

Our results suggest that with brief training, there is improvement on the inter-rater reliability of the GAF scores. This finding is similar to that of Bates and others (7). With severe shortage of psychiatrists in Uganda, PCOs take responsibility for most patients with mental health problems in the country, including psychiatric assessment and treatment. These results suggest that with training, PCOs' assessment of GAF scores could be appropriately used by the PCOs to rate patients.

The strengths of this study are that the raters assigned the GAF scores independently. Furthermore, the rating was done on one constant video recorded psychiatric interview indicating that there was no change in behavour and symptoms. The most important strength was that we were able to compare participants' GAF ratings to some sort of “gold standard.”

In spite of the aforementioned strengths, this study has several noteworthy limitations that could have impacted on the results. The first is the use of a limited sample since all individuals rating the GAF were working in Mulago and Butabika hospitals, both of which are training institutions. This was a threat to the external validity of the study. In addition, the small sample size may have limited the statistical power of some tests. Another potential limitation of this study was the use of only one video recording of a patient with severe mental illness. All of these limitations might have had an impact on our results. Nevertheless, our results suggest further studies in the applicability of GAF scale by different medical workers other than psychiatrists.

Conclusion

Brief training given to PCOs improved the inter-rater reliability of GAF scale to acceptable levels. The study points to the need for formal training to PCOs in the use of the scale.