Assessing the need of young people using online counselling services: how useful are standardised measures?
For this research, we collaborated with our CORC members at Kooth, to explore the use of standardised measures in the online environment. Evidence related to the use of these measures online to identify need, is limited, so our aim was to explore this. The YP-CORE, SDQ and SWEMWBS were the focus of the research and are all used by Kooth counsellors to assess the needs of the young people accessing Kooth prior to a counselling session.
Given that these three measures were developed – and are primarily used – in face to face settings, we wanted to determine a marker of the reliability of using the measures online. To do that, we compared young people’s scores on the YP CORE, SDQ and SWEMWBS to the clinical judgement of the counsellors, based on case notes information. When we compared the young person and counsellor ratings of the assessment measures, we found fair agreement. This suggests that, when using clinical judgement as a comparator, the measures are good indicators of young people’s need, when used in an online setting. The highest levels of agreement between ratings were found for the YP CORE. Additionally, when we compared the ratings on the measures overall to previous research, we found that generally, Kooth young people had higher ratings of need than demonstrated in face to face clinical settings. This suggests that the measures may be completed differently by young people online, compared to in face to face settings.
Co-author and CORC Research Lead, Dr Jenna Jacob said about the research:
"It was great to collaborate with our Kooth members on this research, to help them to explore the use of assessment and outcome measures online, which are routinely used in face to face settings. Whilst Kooth is an established online counselling service, the question of the use of measures in the online setting is a topical area at the moment. The questions we sought to answer were whether there are differences in the way young people respond to these measures in different settings, and how we might determine a measure of reliability to young people’s ratings.
We compared the young people’s ratings on these measures to case notes, and found variation in ratings, which highlights the complexities of using standardised measures alone to assess need. We reflected that the use of such measures provides little insight into the goals or motives of young people when self-referring to an online counselling site. However, the reasonable correlation between the YP-CORE ratings and counsellors’ clinical judgement demonstrated the merit of using a standardised measure to identify the initial needs for a proportion of the population. It is likely that the combination of a standardised measure with an idiographic measure will strengthen the assessment of need. We reflected that the higher levels of need identified by the young people online compared to face to face settings might be in part due to the ‘disinhibition effect’ of an online space (i.e. people are more likely to be more open about difficulties in an online setting).
Our suggestion is that future research should focus on the motivations of young people to complete assessment measures online and seek to further explore the investigate true differences between assessment measures in face-to-face and online populations. We hope this research goes some way to help other members think about their use of measures in their own services, and the data that is derived particularly at the moment, when much clinical work has been moved to remote provision."
Access the research paper here.
CORC member Kooth also published a Theory of Change paper for their online counselling service for children and young people
Web-based counseling and support has become increasingly commonplace for children and young people. Currently, there is limited research that focuses on the mechanisms of change within complex telepsychology platforms, a factor that makes designing and implementing outcome measures challenging.
This paper reports a study conducted as a collaboration between professionals working for Kooth and academic researchers working within the United Kingdom. It is anonymous at the point of access, with young users typically finding out about the service through educational providers and only having to provide limited information on registration. In 2019, it received approximately 1700 log-ins per day, a figure that increased to just over 3000 log-ins on the day in which the COVID-19 pandemic caused schools to be shut down and the government to restrict movement of the UK population (March 20, 2020). Of these log-ins, just under 1000 were a result of new registrations, which represented an increase of over 50% in the trend for registrations earlier in that week.
During the evolution of the Kooth service, it became clear that existing tools for measuring outcomes did not prove fit for purpose, with both the format of the tools and the concepts that they measure proving to be problematic. In this regard, further scrutiny of the work that was undertaken on the website proved to be necessary. As a first step, it was considered necessary to review the work of the service as a whole and develop a theory of change (ToC) so as to consider what might be the most appropriate means of evaluating the work that the service engages in. This was then divided into the conditions that lead to individuals wanting to access web-based therapy and support (eg, individuals wanting support there and then or quickly), the mode of service delivery (eg, skilled and experienced professionals able to build empathetic relationships with children and young people), and the observed and reported changes that occur as a consequence of using the service (eg, individuals being better able to manage current and future situations).
Figure 1. Descriptions of the 4 service pathways for Kooth. CYP: children and young people.
Developing the ToC helps to shed light on how web-based therapy and support services aid the mental health and wellbeing of children and young people. Furthermore, it helps to understand the development of positive virtual ecosystems and can be used to devise evaluative tools for children and young people telepsychology providers.