Assessing the value of and contextual and cultural acceptability of the Strength and Difficulties Questionnaire (SDQ) in evaluating mental health problems in HIV/AIDS affected children

Abstract

Background

The Strengths and Difficulties Questionnaire (SDQ) is a robust, powerful and internationally recognised diagnostic screening tool for emotional and behaviour problems among children, with the particular advantage that it can be used by non-health professionals. This makes it useful in a South African context characterized by shortages of professional mental health carers. However the cultural and contextual acceptability and potential uses of the SDQ have not yet been examined in the South African context.

Methods

The aim of the current study was to evaluate the acceptability of the SDQ in a Sesotho speaking area of South Africa. As part of a larger study to standardise the SDQ for use among Sotho speakers, teachers were asked to use the tool to assess learners in their class. Ten teachers were then asked to write a report on their experience of the SDQ and how useful and applicable they found it for their school setting. These findings were discussed at two later meetings with larger groupings of teachers. Reports were analysed using a modified contextualised interpretative content analysis method.

Results

Teachers found the SDQ very useful in the classroom and easy to administer and understand. They found it contextually relevant and particularly useful in gaining an understanding of the learners and the challenges that learners were facing. It further allowed them to differentiate between scholastic and emotional problems, assisting them in developing relationships with the pupils and facilitating accurate referrals. There were very few concerns raised, with the major problem being that it was difficult to assess items concerning contexts outside of the school setting. The teachers expressed interest in obtaining further training in the interpretation of the SDQ and a greater understanding of diagnostic labels so as to assist their learners.

Conclusion

The SDQ was found to be acceptable and useful in the context of this very disadvantaged community. The teachers felt it assisted them in their role as teachers by providing a greater understanding of emotional and behaviour problems among learners. However, lack of places for referral and their own lack of appropriate skills and time did generate frustration.

Background

The Strength and Difficulties Questionnaire (SDQ) has been used extensively, the results are trusted (Achenbach, 1991; Mullick and Goodman, 2001), and it has been shown to be an effective quick measure of emotional and behavioural problems in multiple settings (Goodman, 1997, Goodman, 2001; Mathai, 2003; Obel et al., 2004, Goodman et al., 2000, Marzocchi et al., 2004; van Widenfelt et al., 2003). One of the main advantages of the SDQ is that it can be used by non-professional people with low levels of additional training to make provisional diagnoses (Goodman, Ford, Simmons, Gatward & Meltzer, 2003; Goodman, Renfrew & Mullick, 2000). These characteristics make it ideal for use in South Africa (SA), where there is a distinct lack of mental health professionals. It has been used successfully in SA as part of a study looking at the impact of being orphaned by AIDS (Cluver, 2011), but there has been no previous attempt at standardising it for local use or to ascertain its cultural acceptability.

This study, from which this material was drawn as a subcomponent was aimed at evaluating the psychometric properties of the SDQ among the Sotho population (Sharp, Venta, Marais, Skinner, Lenka and Serekoane, 2014). While such an evaluation is important, it is also necessary to examine the cultural and contextual acceptability and the potential for its use in a community context (Canino and Alegria, 2008). Items and measures may be understood differently (Cluver and Gardner, 2007a) and individuals may be incorrectly categorised to a label based on an inaccurate understanding of responses and behaviour, the ‘category fallacy’ (Kleinman et al., 1978).

One of the primary groups who have regular contact with children is teachers who see them in the school environment and develop a particular knowledge of and insight into each child in their class. The aim of the current paper was to assess the acceptability of the SDQ by teachers at schools that enrol high numbers of vulnerable children. A similar approach was used in assessing the local acceptability of the Diagnostic Interview Schedule, by Sotho speaking mental health professionals, for Sotho speaking children (Sharp, Skinner, Serekoane & Ross, 2011). For this paper we are defining acceptability as the tool been seen as culturally appropriate for a Sotho speaking population; easy to use and apply, and useful to the teachers in their work.

The study was conducted in in a large township in the urban centre of Mangaung, which is the capital of the Free State, located in the heart of SA and is one of the areas affected most by HIV with a prevalence rate of 14.9% (Shisana and Simbayi, 2002). The Free State is the third most urbanized province, with the Mangaung Local Municipality in the Motheo District being the most densely populated. Of the 752,906 people living in Mangaung, 618,408 (82%) are Black (mostly Sesotho). In 2008, 31% of children in the Motheo district were orphaned (Municipality, 2008), and is likely to have increased since then.

Methods

This paper is nested within a larger study aimed at evaluating the psychometric properties of the SDQ for use amongst a Sotho speaking population in SA (Sharp et al, 2014). As part of the study the educators of the children who were enrolled in the study were asked to use the SDQ to assess the emotional and behavioural problems of children in their classes. Those educators whoagreed to participate were given a brief training in the use of the SDQ. Each educator was given a list of pupils in their class that they were asked to observe and assess using the SDQ. The teachers were paid a small honorarium of R100 per pupil. They were given up to two weeks to complete the assessment and the forms were then collected by the fieldworkers.

Participants

After having completed about two thirds of the total assessments for the study, so having given the educators a number of opportunities using the tool, ten educators, who had each assessed at least five pupils, were approached to write an evaluation report on their experience of using the SDQ. The teachers were selected on the basis of obtaining a spread across multiple schools in the communities where the study was done and having a spread across different education levels of pupils, and those. Each educator was paid R500 ($50) to each write a report of about 5 pages. Scholars themselves were not asked to reflect on the instrument as they were felt to be too young, aged 7 – 11, and not to have the personal resources to be able to reflect insightfully on the instrument.

Measures

The terms of reference (Sharp et al, 2011) that the teachers responded to, included the following key questions, each of these questions had sub-questions that served to focus their responses further.

1. How did you experience administering the SDQ on learners in your class?

2. Was the SDQ useful to you as a teacher?

3. Do you feel that you need additional support to be able to use the SDQ better?

4. How do you feel the SDQ could be used in the school or more broadly in the community?

5. Is there anything that you think we should know that we have not asked?

Educators were also asked to make specific comments on aspects of SDQ and its use in general, and to assess both at the strengths and weaknesses of the instrument.

Qualitative Analysis

The reports were analysed using a modified interpretive content analysis approach (Skinner, 2007), which drew together teachers’ comments into the key areas as organised in the results section below. No personal details on any of the teachers or learners are reported to protect confidentiality.

The provisional results from the 5-page reports were presented to two meetings of teachers, of about 35 teachers each, as part of our feedback to the community. In the meeting the teachers affirmed the results as reported here and in the open discussion after the presentations added some details.

Ethics

The full study was reviewed by the IRBs of Houston University, Free State University and Stellenbosch University.

Results and discussion

The dominant theme was one of support for the SDQ and of appreciation that it had been passed onto the teachers as a tool for use in their classrooms. “I personally give an SDQ a go ahead; it has really worked for me”. There appeared to be a huge gap in the tools currently available for teachers to be able to assess and understand their pupils. A key issue was that the use of the SDQ made teachers aware of the problems that many of the children faced, leaving them feeling empowered.

Experience of using the SDQ with learners

The educators felt comfortable with the SDQ. It was easy to use and there were no experiences of discomfort or concern. The instructions were felt to be clear and the questions easy to respond to. All elements of the SDQ were felt to be important and of value. Some of the material covered was new to the teachers, but the SDQ provided a structure for them to understand and use the new information. The tool was felt to provide a very constructive overall view of the children in the classroom.

The elements in the questionnaire were clear and simple. They addressed the problems that the teachers often experience with certain learners who have behavioural problems.

Usefulness of the SDQ

All the reports addressed the usefulness of the tool and testified to the value of the tool in assessing and understanding the children in their classes. The teachers wanted to be able to identify children in trouble and thereby have some power to deal with them. The SDQ was thus seen as offering a valuable contribution giving teachers greater understanding of what was restricting the children or why they were generating problems in the classroom.

The questionnaire was useful and an eye opener to me because I was not aware of some of the problems children encountered at home and at school. I now know that I should treat each individual with respect and differently considering their environment as well.

The educators felt that through using the tool they gained personally and grew in their professional capacity by increasing their awareness and understanding of the aspects of the mental health of the pupils that may impinge of the learners’ capacity to work in the classroom. There appeared to previously be a very low awareness of the nature of the mental health and emotional problems that the pupils faced and the teacher had felt impotent in trying to respond. So the SDQ was felt to empower them and give considerably greater insight.

I have used the SDQ to identify particular problems amongst learners in my class. Everything was clear. The SDQ has opened my eyes and added to my knowledge in identifying children with emotional and behavioural difficulties in my class.

The tool added structure to their evaluation and observation of the learners, identifying issues that teachers previously would have missed. Teachers learned how to view pupils individually, holistically and in depth. They saw the need to focus not just on schooling, but also on emotions, social interactions, behaviour and capacity. Some of the comments spoke directly to this with teachers stating how they would come back to observing the children while completing the SDQ to look at aspects of their behaviour that they had not considered or had overlooked before, e.g. to assess difference between developmental and emotional problems. There was also potential for the SDQ to be used to assist in assessments such as when a child is ready to return to school following trauma, or in a school readiness assessment.

It will also reduce stress and confusion since now educators will know exactly what kind of learners they are working with and which learning barriers to treat. I think since the educational needs of every learner differs from one another, so for teachers not to end up treating a gifted child like a learner with any learning barrier.

Teachers also noted that it opened avenues for them to directly support the learner with their problems. This applied both in the direct assistance to the learner and the organisation of the class so that they could work more effectively. Importantly for the teachers was the contribution the SDQ made towards developing their relationships with pupils in their classes. The new relationship allowed for a closer connection to develop and greatly raised the empathy and understanding that the teachers felt for the learners. This empathy had the knock-on effect of facilitating the teachers being able to provide greater assistance and support to the learners, and so reduced conflict in the classroom. It further reduced the stereotyping of pupils by educators. Insight was also gained into the strengths of the learners, so gave the teachers additional tools to work with. The additional insights allowed for a more positive insightful relationship to develop. It was noted that as the learner realised that the teacher understood them better they also related better to the teacher and began to trust this relationship.

My experience in doing SDQ measure on learners has brought me closer to them. It has helped me to be more careful, broadminded and protective than before. It has also helped me to try and make life easier for them.

The following excerpt shows how the SDQ directly facilitated the teacher’s work with a child in her class, allowing the teacher to better understand the stressors the learner was under.

I started to know her better, her parents, as well as the conditions under which the learner lives. For example: The learner is coming from a very poor family. The mother is HIV positive. She is a single mother with three children. The mother is unemployed. So the type of life has already affected the learner though she is not infected with HIV. The behaviour of the learner is affected by the way and conditions of the family.

It is particularly important in situations where the teacher is overworked and has too many learners in the class. Some teachers described class sized of up to 50 or 60 learners, many of which had special needs and emotional problems. The schools are dominantly under- resourced and teachers operate with little or no support from the Department of Education. The use of the SDQ assisted teachers to assess pupils individually and to see them each as separate individuals, rather than a mass that needed to be controlled. The teachers were also empowered by being able to use the SDQ to elicit broader assistance for the children. This included both referrals to outside agencies and their capacity to work with the child’s parents in order to assist the learner.

In township schools teachers teach many children and they are unable to give individual attention. Some children are not identified so the instrument will help teachers identify children with emotional and behavioural problems earlier and support systems will be applied to help the learner. At present, schools are not using any instrument and it will be of great assistance if the SDQ can be applied at schools.

It was noted that the SDQ was a tool. It does still require the educator to be proactive and to put in the effort to use the information gained to assist the child. Great emphasis was put on the effort that the educators put into observing the child and to considering the behaviour of the children in relation to the questions raised by the SDQ.

Do you feel that you needed additional support to be able to use the SDQ better?

The dominant theme expressed by the teachers was that the SDQ was generally easy to implement on their own. The clear structure provided by the SDQ provided teachers with a framework to observe children’s behaviour. The structure of the forms and the clear content of the documentation made it easy to use and score. Teachers expressed that with this tool the learners would not be able to hide their problems and the teachers had the capacity to recognise the mental health symptomatology and concerns.

There were no components that I felt I do not understand. Everything was clear and understandable.

Training requests that arose focused both on the content on the SDQ and a desire to understand and interpret the individual components. One area that was emphasized for additional training was to assist teachers in moving from the domain of the cognitive training into the arena of emotion. Similarly, there were expressed needs to be able to interpret and to use the SDQ themselves, especially if it was to be used by the educators independently. The teachers wanted to be able to use the SDQ as a starting point for providing actual on-going assistance to the learners. For many educators this would require a conceptual shift on their part. The requests also reflected an excitement around the tool and a commitment to it being implemented on an on-going basis. There was a real sense from some that they wanted a greater involvement in the use of the tool and have it connected to broader operations with the education dept.

Yes, to me it would have been useful to have received some additional training in the use of the instrument. I am prepared to undergo and also to be involved in training in order to use the SDQ.

As an extension to the above concern, some teachers felt the need to be open about their use of the SDQ with the learner. Teachers asked for assistance in how to explain the SDQ to the learners, to use the results effectively and how to work with the learner on a collective basis using the findings.

A key point of frustration was that the tool still left the teacher without the tools to intervene. While it was important to be able to understand what the learner’s problems were, the more important need was to find solutions and a way forward to assist the learner.

…… it helps to be able to diagnose and understand the child, but there remained huge frustration in terms of not having anywhere to refer the children who need assistance. Neither the health services nor the Dept of Education took the issue of mental health problems among school children seriously and few if any real services existed. Those services that did exist were heavily overextended.

How do you feel the SDQ could be used in the school or more broadly in the community?

There was strong support for the use of the SDQ, both in schools and in the community. They felt that there were currently no tools to help the teachers to perform such tasks. The teachers usually rely merely on closely observing the learner’s behavioural changes. The instrument’s holistic approach was identified as a further strength over current referral forms. The SDQ could also act as an advocacy document to promote the need for additional resources for schools to assist learners with mental health problems.

Could the strengths and difficulties questionnaire be used in the school or more broadly in the community, yes I fully agree in the sense that the school and the community plays a major role in the upbringing of the child. ……. In the question whether we need the tool, I will vocally say yes we need it as I have alluded to the fact that the tool is user friendly because it addresses a holistic approach of the problematic learner.

Teachers felt that the SDQ was easy to use, and that this would allow for it have a broad use in the school context. Teachers mainly addressed the use of the SDQ in assisting them with the process of referral and explaining the problems that they had noted to other professional, service providers and to the children’s parents. Many pupils with mental health problems are incorrectly sent to remedial schools, while their problems are actually primarily emotional. This was born out in discussion with teachers at the follow-up workshop.

It will also stop wrong or let me say “quick referring” of learners if need be. It will help teachers to provide coherent reasons for a referral and to provide adequate information about the referral to the family.

The school is also connected to the community, so the application and benefits can be broader than just the school. The teachers focused on their own role, but raised some additional service providers who could apply it including social workers and church leaders.

I feel it should be used in both the school and in the community. The very same learner that I am dealing with is the member of a certain church. And at church there are organizations that deal with or experience similar problems with learner as us at school. Now if the program could be extended to the broader community we will be able to deal with the learner at all spheres.

Translation issues of the instrument in Sotho

The teachers were generally satisfied with the translation given to them and expressed no problems in understating it. They felt that the questions were structured in such a way that each question does not need a follow-up explanation. It was recognised that some confusions may arise as many different dialects exist in Sotho, so there will be differences between the version of the language spoken in academic circles to that spoken in the poorer communities and differences based on where the family comes from geographically. These differences can extend to the point of terms and entire sentences not being understandable. However, the teachers felt that the structure and general clarity of the instrument would contribute to reducing even those problems.

The translation is 99.9% correct. The Sotho speaking child and the English speaking child who have similar problems will answer the questionnaire more or less the same. They will provide almost similar answers to all questions.

An item related to Attention-deficit Hyperactivity Disorder of fidgeting or squirming in their seats was the only item found to be difficult as they did not use the term often and did not have a clear meaning in the Sotho lexicon.

Concerns or problems

Very few concerns or problems were experienced in the use of the SDQ. It was felt to cover the major issues faced by the teachers in assessing children in their classrooms and in extra curricula activities. One concern common in many critiques from Africa of diagnostic schedules is the lack of coverage of somatic symptoms, which received brief, but no consistent mention.

The learner often complains of headaches, stomach aches or any other sickness that is true and this matter can never be disputed

Some of the questions were difficult for teachers to respond to, based on their restricted contact with the children, e.g. covering information from the home situation or requiring knowledge of the child over a period longer than the year that they spent in their class. The teachers felt that they would have to involve the parents or care givers to assist them in answering these questions. Some additional training could facilitate their consultation.

Yes there were elements that I felt created confusion, like that question from the SDQ asking as to whether my learner is kind to younger children or not.

Limited concerns were raised about the grading system in terms of the symptoms with some struggling to with the construct of a symptom happening sometimes. They felt it either happened or it did not. One concern raised above was on the use of the terms such as “somewhat true”, which most found difficult to respond to.

The context of the teacher was raised as an issue. The overcrowded classrooms and multiple demands on the teacher made diagnosis even with an instrument such as the SDQ difficult. This may present particular difficulties in allowing time to establish trust with a learner so they will feel comfortable talking about their concerns.

In our school the classrooms are over-crowded. The more learners there are in the classroom that need individual vocational attention, the less the teacher is going to concentrate on their emotional issues unless that particular learner exhibits severe emotional and behavioural changes.

One teacher raised a number of concerns related to the developmental and contextual issues of adolescent children. It was also felt that children respond differently depending on context. They felt uncertain about how to respond to some questions. Some of the issues raised included:

Consideration of other people’s feelings: Most children are self-centred, they only think about themselves, they do not consider other people’s feelings.

General obedience: Children behave differently in different situation. They change their behaviour in order to adapt to what is happening around them at that particular moment. A child can misbehave at home but be obedient at school or the other way around.

Thinks things out before acting: Many children do not think before they act, they just do things and regret it afterwards especially 7–9 yrs age group.

One critique raised was that the SDQ did not deal with some of the issues that teacher’s faced in enough depth. In this case they would have liked to see more focus on the issue of disruptive behaviour which is a common problem faced by the teachers. The issue of the learners challenging behaviour was felt by some to warrant a separate more intensive focus. Similarly some questions that are directed at the specifics of the teaching situation were suggested.

Has the child participated in class activities

Raising hands when a question is asked

Does the child like going out now and then

Is the child attention seeker

Conclusions

There was generally high acceptance of the SDQ as a diagnostic screening tool. The tool appeared be culturally appropriate with minimal concerns being raised. Even the concern expressed around the translation of the one items on fidgeting in the classroom could be addressed by improving the translation. The SDQ also appeared to be easy to use and useful for the teachers in the classroom situation. Primarily the teachers spoke posivitively of the tool, both as a basis for understanding the children in their classrooms that are having problems and as a tool for making effective referrals. This support was evident both in the reports written by the teachers and in the discussions held as part of the reporting back process. The tool clearly addressed a significant gap in their expertise and responded to an important need, especially as the learners raised multiple problems for the teachers. The value of the SDQ lay in the teachers being able to better understand what was affecting the children and thereby being able to respond more effectively. It also improved their capacity to refer the child effectively or at least to be able to discuss the issue surrounding the child with the parents\caregivers.

Some concerns were raised with the instrument, but these were limited to having difficulty with the graduation system in the response set and the desire to tailor it specifically for their needs by adding focused questions for the classrooms. Additional tools would have to be developed to deal with those concerns.

Beyond the formal role of the instrument, the SDQ appeared to generate additional awareness in the teachers around mental issues and the potential impact of these on their pupils that they had not considered before. This additional insight went beyond the direct diagnostic power of the tool. So it serves an awareness raising function. These roles do raise the possibility for the use of the instrument beyond the immediate context of the schools by lay people in the broader community to identify children with problems and to facilitate referral; with use amongst others social workers, Church leaders, youth group leadership being identified to take on this role.

Of primary concern to the teachers was the lack of referral sources for children with emotional and behavioural problems. The teachers felt that the Dept of Education did not provide adequate support services and there are limited referral sources within the health services as well. While this is not a problem with the SDQ, the value of a tool to identify emotional and behaviour problems is reduced if no referral sources are available. For instance, in an earlier part of this study (Sharp et al., 2011) we identified only six Sotho speaking psychologists in the Mangaung area. The teachers expressed the need for additional professional services such as psychologists and social workers to whom they could refer children in need; specialised facilities to treat and support the children, especially those most affected; assistance to households to be able to cope with the social pressures placed on them and the children living there; and support and training for the teachers themselves to be able to assist directly the children in their classrooms with their social and mental health problems as appropriate. As indicated in Sharp et al (2014), the needs of this community are very high, and will take considerable resources to provide these. In the interim the SDQ does at least provide the teachers as one front line set of workers with a tool that can assist them.

Limitations

A number of limitations need to be acknowledged in this study. The sample of teachers selected was small and represented those who were more committed than most to their professional roles and the children that they cared for. As such, they may represent more commitment than other teachers who had not participated would show. Given the low levels of institutional and state support that they receive, this support could in part be making a desperate grab for anything that is on offer. Given the extremely high loads that they face on a daily basis in the schools this energy and commitment to using the SDQ may be difficult to maintain.

List of Abbreviation

SDQ

Strength and Difficulties Questionnaire

SA

South Africa