Using outcome measures with young people from different ethnicities and in different countries
by CORC Research Associate Dr Florence Ruby
As part of our commitment to contributing to conversations on racism and inequalities within the child and youth mental health sector, the CORC team has started to reflect on the ways race, ethnicity and culture play a role when using outcome measures.
One of the first aims we set was to improve our understanding of the literature currently available regarding the use of outcome measures with different ethnic groups. We know that standard measurement tools can be prone to systematic racial bias, with many questionnaires developed based on the expected behaviours and attitudes of Western, White children, young people and their families, frequently underrepresenting the experiences of people of colour and needing to take into account potential cultural differences. Research aiming to validate those questionnaires also reinforce this racial bias, with samples consisting predominantly of White participants and few studies exploring how the validity and reliability of a measure may change for different ethnic groups. But with 20% of children and young people being from a minority ethnic background (Office for National Statistics, 2011), it is essential that a variety of experiences are taken into account and that mental health difficulties are assessed equally well across different ethnic groups.
Over the summer, we started looking at the evidence available regarding the psychometric properties of common mental health measures when used with different ethnicities or cultures. We focused on exploring the evidence available for the Strengths and Difficulties Questionnaire (SDQ) and the Revised Child Anxiety Depression Scale (RCADS), two measures that are frequently used with children and young people in mental health services.
We were particularly interested in finding evidence for measurement invariance across ethnicities, i.e. whether a scale measures a specific construct or trait in the same way across different ethnic groups. In statistical terms, measurement invariance can be established when all groups respond to particular items on a scale in a similar way, and when factors (sub-scales) identified by factor analysis are comparable across ethnic groups. If evidence for measurement invariance does not exist, then we cannot be sure that a scale actually measures a construct in the same way across different groups, and therefore any differences we may see between ethnic groups may not be meaningful.
What did we find for the SDQ?
A number of studies have investigated measurement invariance for the SDQ, but so far showed mixed results. Findings across studies could not easily be compared, because different respondents completed the SDQ (young people, parents or teachers) and because different studies focused on different countries and ethnicities (e.g. New Zealand, the Netherlands, Germany). Despite these differences we tried to identify emerging patterns in the findings. Below we provide a short summary of papers we have identified as helpful examples of the evidence currently available.
Several studies found for example that the five sub-scales that are commonly calculated from the SDQ (Emotional symptoms, Conduct problems, Hyperactivity/Inattention, Peer Relationship Problems and Prosocial behaviour) were not comparable across ethnic groups. For example, Stevanovic and colleagues (2015) found that the five-factor model was not adequate for the self-report SDQ in samples from India, Indonesia, Nigeria, Serbia, Turkey, Bulgaria and Croatia; although a simpler model with three factors (Externalising, Internalising and Prosocial sub-scales) was found to be common across countries, the authors reported that individual items (especially on the peer problems and hyperactivity scales) were perceived differently across groups. Altogether they concluded that the SDQ lacked measurement invariance and did not allow for meaningful comparisons across the samples investigated.
Other studies found that by removing items that were ‘behaving’ differently for different groups, they could create a shorter version of the SDQ that was able to measure meaningful differences between groups (Duinhof et al. 2020).
We could only identify two studies that explored measurement invariance in a UK-based sample. One focused on Indian British and White British children (Goodman et al. 2010) and found that models including two factors (reflecting Internalising and Externalising problems) could be compared across ethnicities. The other study mainly included Pakistani British and White British children (Stochl et al. 2016) but could not explore invariance across ethnic groups due to small sample size and focused on invariance across age.
What do the SDQ findings mean?
Overall, the studies currently available indicate mixed results regarding whether the SDQ is invariant to ethnicity or country. Some studies have found the measure to be invariant while others report that it may not allow for a meaningful comparisons of mental health difficulties across ethnicities or countries. The current evidence seems to be very dependent on the context of the study, in particular who is completing the SDQ and the country or ethnicity of the children and young people. Lack of invariance could be due to a number of factors, for example cultural or language differences that may influence the ways items on a scale are perceived and answered, or the relative relevance of the items in a specific socio-cultural setting.
Evidence is also missing for many contexts. For example for the UK, we could not find studies looking at measurement invariance across all representative ethnic groups, for example Black British youth, and therefore we do not know whether the five SDQ sub-scales currently used are meaningful for all children and young people. More research is therefore needed to better understand how the SDQ can be used meaningfully to draw comparisons across ethnicities.
What did we find for the RCADS?
So far, we came across three studies looking at measurement invariance across different ethnic groups for the RCADS. One study by Stevanovic and colleagues (2016) analysed RCADS data from 11 countries. They found that a few items were non-invariant between countries (e.g. Romanian children generally scored higher on item 1 “I worry about things”, compared to children from other countries) but on the whole, the six factors traditionally obtained for the RCADS (Social Phobia, Major Depression, Separation Anxiety, Generalised Anxiety, Panic disorder and Obsessive Compulsive disorder) could be replicated across individual countries. The authors therefore concluded that the RCADS sub-scales could be used to draw comparisons across the countries investigated.
Two other studies focused on African American and White children and young people in the US (Latzman et al. 2011; and Trent et al. 2012) and reported that RCADS sub-scales were generally invariant i.e. were measuring the same constructs across both ethnic groups. No studies on a UK sample could be identified in the literature.
What do the RCADS findings mean?
Overall, these studies provide initial evidence that the RCADS may be a helpful measure to compare levels of anxiety across different countries or ethnicities. But as with the SDQ, more research is needed to explore whether this would hold true for different samples, for example different age groups and different ethnicities.
A final note
As we can see here with the SDQ and the RCADS, the number of studies exploring the relationship between outcome measures and ethnicity or culture remains relatively small. More work needs to be done to better understand where and how measures can be used to make meaningful comparisons across different contexts. In addition, we also need to ensure that scales are relevant for a diverse range of children and young people and actually consider more diverse lived experiences.
Over the coming months we will continue to review the evidence available and share updates with you in our blogs and on the outcome measure webpage. We’d love to hear from you if you have come across relevant studies on the subject. Get in touch with us at CORC@annafreud.org.
References
Duinhof, E. L., Lek, K. M., de Looze, M. E., Cosma, A., Mazur, J., Gobina, I., ... & Stevens, G. W. J. M. (2020). Revising the self-report strengths and difficulties questionnaire for cross-country comparisons of adolescent mental health problems: the SDQ-R. Epidemiology and psychiatric sciences, 29.
Goodman, A., Patel, V., & Leon, D. A. (2010). Why do British Indian children have an apparent mental health advantage?. Journal of Child Psychology and Psychiatry, 51(10), 1171-1183.
Latzman, R. D., Naifeh, J. A., Watson, D., Vaidya, J. G., Heiden, L. J., Damon, J. D., ... & Young, J. (2011). Racial differences in symptoms of anxiety and depression among three cohorts of students in the southern United States. Psychiatry: Interpersonal & Biological Processes, 74(4), 332-348.
Stevanovic, D., Bagheri, Z., Atilola, O., Vostanis, P., Stupar, D., Moreira, P., ... & Dodig-Ćurković, K. (2017). Cross-cultural measurement invariance of the Revised Child Anxiety and Depression Scale across 11 world-wide societies. Epidemiology and Psychiatric sciences, 26(4), 430-440.
Stevanovic, D., Urbán, R., Atilola, O., Vostanis, P., Balhara, Y. S., Avicenna, M., ... & Petrov, P. (2015). Does the Strengths and Difficulties Questionnaire–self report yield invariant measurements across different nations? Data from the International Child Mental Health Study Group. Epidemiology and psychiatric sciences, 24(4), 323.
Stochl, J., Prady, S. L., Andrews, E., Pickett, K., & Croudace, T. J. (2016). The psychometric properties of the strengths and difficulties questionnaire in a multi-ethnic sample of young children. Acta Universitatis Carolinae Kinanthropologica [AUC Kinanthropologica], 15-37.
Trent, L. R., Buchanan, E., Ebesutani, C., Ale, C. M., Heiden, L., Hight, T. L., ... & Young, J. (2013). A measurement invariance examination of the Revised Child Anxiety and Depression Scale in a southern sample: Differential item functioning between African American and Caucasian youth. Assessment, 20(2), 175-187.
Office for National Statistics (2011): 2001 Census aggregate data (Edition: May 2011). UK Data Service. DOI: http://dx.doi.org/10.5257/census/aggregate-2001-2