Asking graduates how they feel

It’s been really interesting to see the discussions taking place in the sector regarding questions in the Graduate Outcomes survey about subjective wellbeing (or SWB for short). We’ve also had lots of queries from HE providers and researchers about these questions, so I hope this blog post is both a useful addition to the discussion, and answers some of those queries. Of course if this piece raises more questions, please do get in touch with our Liaison Team.

Getting your hands on the data

At the end of May, HESA provided access to the raw survey data to providers, and for some there was a section of data missing… where are the subjective wellbeing answers? Well, it wasn’t forgotten, rather it was left out due to the nature of the data. One of the themes I’m reading in discussions of the survey is the sensitive nature of this data - and this is the exact reason it’s not included.

As many of you will already know, the HESA approach to special category personal data is to aggregate it and apply disclosure controls as standard practice. This will ensure we suppress any small numbers and ensure individuals can’t be identified. We also need to be mindful that the data we supply to the sector is meaningful so you can make appropriate use of it. So, to provide aggregated data that’s meaningful, we need plenty of it! This means we need to wait until all of the data has been collected so it will form part of the final release in Spring 2020.

Where did the SWB questions come from?

Subjectiv​e Wellbeing questions

Measure	Question
Life Satisfaction	Overall, how satisfied are you with your life nowadays?
Worthwhile	Overall, to what extent do you feel that the things you do in your life are worthwhile?
Happiness	Overall, how happy did you feel yesterday?
Anxiety	On a scale where 0 is “not at all anxious” and 10 is “completely anxious”, overall, how anxious did you feel yesterday?
Source: Office for National Statistics

The questions are a core part of the survey at the request of HE regulators, and the same questions were in the last iteration of the DLHE Longitudinal survey.

The questions themselves are those that the ONS and other survey organisations have been using to measure and understand ‘personal wellbeing’ for many years through national surveys. Using the same questions in Graduate Outcomes should provide comparable results for recent graduates, and complement other measures of the impact of higher education on social capital, mobility and cohesion.

We did cognitive testing to ensure that the target audience for Graduate Outcomes could interpret and understand the questions. This testing didn’t explicitly cover emotional reactions to the survey, but there were opportunities to raise this issue. Some testers did query whether the questions were appropriate and the purpose of gathering the data, while others commented that SWB issues are relevant to assessing their overall higher education experience.

No one raised the likelihood of causing distress or anxiety during cognitive testing but there is always the possibility for approaches to handling respondent anxiety (if any) to vary. The open centralisation model gives us the opportunity to set and embed best practice if or when we experience complaints from graduates about these questions.

HESA completed a Data Protection Impact Assessment prior to the survey taking place which assessed the impact of the survey on the privacy and rights of individuals. We also undertook a full ethical review which made recommendations about how the questions are implemented in the survey. This is a really important issue, so I’ve covered it in the next section.

How do we ask the SWB questions?

Although the questions are within the core part of the survey, they are optional. This means they can be skipped on either mode (online or telephone), the same as any of the other optional questions. No marker is added to respondents who’ve skipped optional questions and graduates who don’t answer them are still included in the data set.

IFF Research have a full training and development programme for their telephone interviewers. Interviewers are trained to explain the purpose of these questions, to understand that they are potentially sensitive, and to signpost graduates to the Samaritans if they feel this is appropriate.

Both online and telephone surveys give this introduction to the SWB questions:
"The following four questions ask you about your feelings on aspects of your life. There are no right or wrong answers."

Additional prompts for phone interviewers can include reassurance that individual responses will not be followed up. Interviewers can also record and feed back to HESA any comments or complaints made by respondents.

The SWB questions are asked near the end of the survey after all of the compulsory questions have been asked. This ensures that respondents are still able to provide a complete response to the survey even if they don’t want to answer these questions, and ensures that these questions don’t have a negative impact on response rates.

A graduate can choose to answer any, all, or none of the SWB questions. From the data we’ve collected so far, of all graduates who completed the survey, only 0.6% stopped just before the wellbeing questions. A further 1% skipped these questions and moved on to the next section. 98.4% answered at least one of the four questions.

What could you use this data for?

The data can be used to answer a wide range of research questions, including but not limited to:

• What is the state of personal wellbeing among graduates?
• Is graduate wellbeing changing over time?
• How does the wellbeing of recent graduates compare with the rest of the population (using ONS data for example)?
• How does the wellbeing of graduates vary by geographic region?

One probably wouldn’t try to draw causal relationships between higher education experience and wellbeing 15 months after graduation since there are so many factors that influence our state of wellbeing. But one could certainly develop a more thorough definition of ‘graduate success’ by incorporating graduates’ personal wellbeing. This may provide crucial evidence to HE providers considering more holistic approaches to course delivery, such as the inclusion of coaching or personal development for students.

Under all circumstances, use of this data should comply with data protection legislation and the privacy information that HESA has provided to graduates.

The future of SWB?

Who knows? This is the first year of Graduate Outcomes and the first year of wellbeing questions being part of the annual graduate destinations survey. The utility of these questions will be determined by the users of the data. As with everything else, we welcome your feedback. In time, we would love to see case studies on how you’ve used this data to inform policy or practice at your institution or in the wider community.

Of course, all this will only be possible once you have access to the data, in 2020!