Using outcome and experience measures when providing a remote service

Services are now delivered in a range of ways, with remote support widespread.  However, many outcome and experience measures were originally developed with in-person delivery in mind.  

Here we share our guidance and advice on working with feedback and outcome measures for services working remotely with children, young people and families either online or by phone.  

Download guidance here

Principles to bear in mind

Measuring and monitoring the outcomes of care is just as important for services delivered remotely as for those delivered face to face. This is part of offering responsive and person-centred care, improving and developing services, and being accountable and transparent.

There is even more of a case for working with feedback where there has been a change in the way support is provided. However you approach this, good practice principles are:

• ‘no measurement without meaning’ – know why you are asking, what you are going to do with the information, and explain this clearly
• always acknowledge and respond to the feedback shared through questionnaires
• ask questions as they were designed to be asked – if you adapt an outcome measurement questionnaire, you can’t be confident that it will still be a good tool for measuring what it is supposed to measure.

Approaches to capturing questionnaire feedback in remote provision

We look at this in terms of three over-arching approaches:

1. The questionnaire is completed at the service user’s end: the service user will download and complete a questionnaire and send it back to the service (e.g. by email)
2. The service makes a shared electronic space available: this might be a portal or platform where measures can be accessed, completed and saved or submitted by the service user, without the service user needing to ‘send’ anything to the service
3. The service user shares their responses to a questionnaire verbally, and these are recorded by the practitioner who stores them in the appropriate place in the service’s care record system.

Our full guidance includes a table (page 4) that breaks down how these three approaches can work in practice and some of the infrastructure, data protection, back office and practice considerations that you might like to take into consideration in planning your approach.

Helpful tips and practical pointers 1: versions of questionnaires that can be edited by practitioners or service users

If you have access to PDF versions of the questionnaire you want to use:

• The free Adobe Acrobat Reader does have commenting tools that allow you to add, circle or highlight text and then save the PDF with these 'comments'
• If you are a practitioner going through the questionnaire verbally online and can share screen, we feel it works well for the child or young person to be able to see you edit the questionnaire as you go through it
• The questionnaire can also be edited this way at the service user’s end, although do experiment with this yourself to check its suitability for those you work with - not everyone will find this equally user-friendly and practice or advice may help.

Microsoft office formats, e.g. word, excel:

If you are considering reproducing the questionnaire in another format, please bear in mind possible copyright considerations, and the research findings about how questionnaire responses might be influenced by particular contexts or formats.

Copyright considerations

If the copyright for a measure is held by a measure developer (or other party), specific terms of use may have been set, for example regarding the purposes for which the measure is used or the ways in which it is reproduced. If you are thinking about reproducing or modifying a questionnaire – for example so it can be completed in a particular electronic format – you should check whether this is compatible with the terms of use.

In response to the impact of coronavirus, some developers are taking a more flexible approach: where we have information about this we have highlighted it in our section on ‘specific measures’ below, as well as putting details on the page for the relevant measure in our outcome and feedback measures directory, which signposts to information about terms of use where possible.

Please note that NHS Digital have negotiated licences for a wide range of measures (although there may still be restrictions e.g. to a particular geographic territory): to find out more about a specific measure you may wish to contact NHS Digital (clin.licences@nhs.net).

Modifying standardised measures might affect your ability to flow data from that measure to the Mental Health Services Dataset or to CORC. It may also have an impact on how far your data can be meaningfully compared with data captured using the unmodified measure.

Confidentiality and data protection

Considerations around confidentiality, informed consent and data protection apply in the normal way. Furthermore, when completing and discussing questionnaires with service users remotely, it may be helpful to suggest going to an area where conversations are less likely to be overheard. You may wish to discuss with service users any new potential risks to the confidentiality of information which you as a service will not be able to control or mitigate – for example:

• If questionnaires need to be saved or printed on devices owned by or shared with others (e.g. parents, carers, siblings)
• If you will be corresponding about questionnaires with both parents and young people using the same email address.

In thinking about any online data transfer (e.g. email, secure messaging app) or online data collection platform, you should consider similar data security questions as for other systems for example what is the sensitivity of the data being collected, where would password-protection of documents be appropriate, how are user accounts set up and secured, who has access to what data, etc. These aspects should be reviewed regularly as part of your service’s approach to data security. Useful resources include ICO guidance, Cyber Essentials, and the Data Protection and Toolkit Standard.

Research relevant to using outcome measures online

There are two aspects of research which may be helpful to bear in mind when you are working with outcome measures online.

If you are considering adapting questionnaires, or paraphrasing questions:
Developers put a lot of expertise into developing measures and the psychometric validation of these – copyrights aim to protect the validity and reliability of the questionnaire:

• validity is the extent to which something measures what it is intended to measure, or how accurate it is
• the reliability is how consistent a measure is, for example, over time or between respondents.

If research shows the questionnaire is valid and reliable, we know it can be meaningfully used with multiple young people, and that we can compare responses across young people and across teams or services. Even small tweaks can mean the questionnaire it is no longer a valid or reliable measurement tool. It would no longer be sound to look at scores against clinical thresholds or norms, or to benchmark the data or to draw meaningful conclusions when looking at multiple scores together at an aggregate level.

When interpreting data from questionnaires completed online

Even when the questions are not changed, research suggests different formats can have an impact on the way people respond to mental health and wellbeing questionnaires. The full version of our guidance discusses the published findings (page 7) and references (page 12) – however these are not clear-cut and there is a need for more research.

Given we do not fully understand the way questionnaires function differently in different settings, we recommend that practitioners and services bear in mind when they are interpreting questionnaire results that there is potentially a difference in the way that young people (and others) respond to measures remotely (compared to completing a measure on paper).

When given a choice, the majority of adults and young people opt for electronic versions of measures over paper-based versions (Buchnell Martin & Parasuraman, 2003).

Helpful tips and practical pointers 2: filling in a questionnaire together, verbally

Our full guidance (page 8-9) talks through some of the tips we have come up with when we have been using measures online.

Summary suggestions if you are filling in a questionnaire together, verbally:

• go straight through all of the questions in order without stopping to discuss individual items – explain you will do this and then discuss at the end
• if you can, it is best if both of you can see the questionnaire you are filling in e.g. by sharing screen or emailing the blank questionnaire in advance
• use the exact language in the questionnaire –this can feel clunky until you have worked out what is fluent for you and done a couple of practice runs
• When you reach the end of the questionnaire, talk together about the responses given.

Measure choice and guidance on specific measures

A change in the way your service is delivered might cause you to review your choice of measure. If you are talking through a measure in a session you might want to consider:

• the length of the questionnaire: if you have been using longer measures, it is worth bearing in mind that research suggests briefer measures can also be psychometrically robust as well as simple to use (e.g. the Child Outcomes Rating Scale (Casey et al, 2019) or Young Person’s CORE (Twigg et al., 2009).

• Goal-based measures may feel more natural to integrate into a remote session. Recent research on the Goals and Goal-based Outcome tool (GBO; Law, 2011) has found good levels of internal consistency between goals which suggests that even though goals vary in content, the goal ratings work together in a more cohesive way than previously thought - similar to standardised measures of mental health and wellbeing outcomes (Edbrooke-Childs et al., 2015). 

We encourage you to test out the questionnaires you use with a colleague and see if they still feel like a good fit in your new way of working. Choosing an outcome measurement questionnaire always involves balancing a number of considerations and we welcome conversations with CORC members to review or talk though their outcome measure choices or options.

You can find any information we have about specific measures recorded on the page for that measure in our measures hub including any guidance we may have on using it remotely, or any current relaxation in its terms of use. Helpful to note:

• CORE Systems Trust (YP CORE and CORE 10) have shared versions of their measures to support practitioners working remotely at this time. They also have fillable PDF forms for the CORE measures available. 
• There are developer-approved video guides available discussing how to administer the Outcome Rating Scale and Session Rating Scale in remote work.

• To help with using the GBO in online work, Goals in Therapy have developed an editable PDF version that allows you or your client to rate the GBO electronically and share results. The interactive PDF versions rating sheets can be downloaded and used for free here. 

CORC members share their experience of providing support remotely

Sheffield Children's NHS Trust pilot: Experience of Service Questionnaire (ESQ) via text

Read case study here:

ESQ via text case study

CORC members share their experience of transitioning to provide support remotely as a result of the pandemic.

Download PDF here:

Transitioning to provide support remotely