Rethinking an assessment of Children's Social Care Services: blog on a recent feasibility study
As a researcher, when you start a feasibility study you hope that you’ll get some insight into the best ways of implementing the larger project you are working towards. At the very least, you hope to find that the larger study is possible. Well spoiler alert for those who haven’t read our report, but sadly the conclusion of our feasibility study for a project that would have focused on the processes through which children’s social care services (CSCS) could be improved was that the main study was not, in fact, feasible.
Our original idea was to look at the processes that different CSCS went through in order to try and improve, and to compare a sample of services where that process had been successful with a sample where the process hadn’t been. Our intention had been to use Ofsted ratings as a measure of success. We applied to the Nuffield Foundation for a grant to support the project, but the Foundation felt we needed a more robust measure of success than Ofsted ratings and so agreed to fund a feasibility study to look at that issue.
No link between Ofsted rating and Department of Education outcomes data for CSCS
So, we set out to explore how to combine the Ofsted ratings with the outcomes data collected nationally by the Department for Education (DfE). Our analysis of this data came to a surprising conclusion: there seemed to be no association between Ofsted ratings and how successful CSCS were according to the DfE outcomes data. Services deemed to be ‘failing’ services according to Ofsted were just as likely to appear in the top 10% of ‘succeeding’ ones according to the outcomes data, and the same was true the other way round.
In science one should always favour the most parsimonious explanation, i.e. the simplest one that fits the available evidence. In this case, the simplest explanation is based on accepting the DfE data and Ofsted ratings at face value, in other words that the Ofsted ratings are a good measure of how well CSCS’s are performing and the DfE data are a good indication of the outcomes experienced by children being supported by those services. Based on those assumptions, the explanation for the finding is that an effective CSCS is not sufficient and may not even be necessary for good children’s outcomes.
A shocking conclusion?
This is a rather shocking conclusion, but can’t be dismissed out of hand. Currently, there is very little evidence that can contradict it and if CSCS were being piloted, the data certainly wouldn’t support extending that pilot. However, there are serious issues with the data that raise doubts about accepting the parsimonious explanation. A new Ofsted inspection framework has recently been introduced, so only about half of England’s CSCS had an up-to-date rating that could be used in the analysis; the DfE outcomes data was not designed as a holistic set to be used to track children’s progress and wellbeing, and the data that is collected is incomplete and collected in inconsistent ways. In addition, at a basic analytical level the analysis didn’t take into account the characteristics of different areas (such as the degree of social deprivation).
Nevertheless, when we presented these findings to an expert audience, including leading academics, representatives from the Department for Education, Ofsted and the Association of Directors of Children’s Services, there was an audible (and rather satisfying) collective gasp. It was not that the findings were necessarily hugely surprising to anyone in the room, but the analysis hadn’t actually been done before and to see it presented in black and white was clearly something of a jolt.
Where to next
The seminar concluded, as did the research team, that the nature of the outcomes data meant it couldn’t be relied upon to assess success or otherwise of CSCS and therefore our original study wasn’t feasible. Instead, the advice was to explore further what data is needed to have a comprehensive understanding of outcomes for children using CSCS and how that data could be collected. Following that advice, the research team applied to the Nuffield Foundation for a project looking at those issues and we’re very pleased to have received funding from the Foundation to do so. Our hope is that once we’ve completed this study, we’ll be one step closer to being able to understand the effectiveness of CSCS and whether steps to improve them have had the desired impact.