Ready for the weekend? How data relating to the Graduate Outcomes survey process can be used in assessing the validity of subjective wellbeing information

In this insight, HESA researchers examine the association between the day of the week the Graduate Outcomes survey was submitted and different measures of wellbeing.

Summary and key findings

• It is well-established that replies to questions on wellbeing may vary depending on how one is asked to complete the survey (e.g. telephone versus face-to-face).
   
• However, wellbeing responses could also be related to the day of the week on which the individual takes part in the survey.
   
• Theory suggests that we should anticipate a greater association between day of the week and present emotions (e.g. happiness), as opposed to life evaluations (e.g. life satisfaction).
   
• We draw upon Graduate Outcomes data linked to information on survey completion dates to carry out an exploration in this area. 
   
• To the best of our knowledge, this is the first UK study to examine this relationship.
   
• Our results are in line with theoretical expectations, with a stronger link observed between the day of the week the survey was completed and present emotions.
   
• This provides some support for the validity of the wellbeing data we collect.
   
• We note that graduates tend to report higher levels of happiness at (or in the run up to) the weekend.

Introduction

The last decade has seen increased debate about how nations measure economic and social progress, with the population’s wellbeing continuing to be given greater focus in policy circles. Indeed, many papers now look at what factors are associated with higher levels of wellbeing (e.g. income, education etc).

Very few, however, consider how responses to wellbeing questions vary depending on the day of the week an individual responds to a survey. An exception to this is a report in the US, which details how this examination can be used to understand the validity of wellbeing information.[1] As a statistics agency that adheres to the Code of Practice of Statistics, one of HESA’s core objectives is to continuously seek out innovative ways to investigate the quality of our data.

With no UK study having analysed how wellbeing fluctuates on a day-to-day basis (to the best of our knowledge), we exploit Graduate Outcomes data linked to information on when survey responses were submitted to start to fill this gap in the literature. By doing so, we can offer an insight into the quality of the wellbeing information collected by HESA to data users with an interest in this subject matter.

Measuring wellbeing: why does it matter?

One of the great (and ongoing) debates among social scientists is how we quantitatively explore economic and social progress, with the level of attention received by this matter particularly gathering pace following the financial crisis of 2008.[2] In the UK, the former Prime Minister David Cameron set out the importance of measuring wellbeing in a speech back in 2010[3], tasking the Office for National Statistics (ONS) to develop suitable statistics on this issue. Over the past few years, the ONS have introduced regular bulletins that explain the key trends in our wellbeing.[4]

More recently, the UK government published the ‘Levelling up the United Kingdom’ White Paper.[5] Mission 8 highlights an ambition to see wellbeing improve in every area of the UK by 2030, with there being a commitment to publish regular statistics to enable progress to be tracked. In Wales, the Wellbeing of Future Generations Act 2015 outlines seven goals and one of these (‘a healthier Wales’) relates to maximising physical and mental wellbeing among the population.[6] The National Strategy for Economic Transformation recently circulated by the Scottish Government details an aspiration to create an economy that is focused on delivering fair work and greater wellbeing.[7] Furthermore, following the restoration of the executive in Northern Ireland at the start of this decade, a pledge was made to improve wellbeing across the nation.[8] Indeed, the recently published ‘A 10x economy’ by the Department for Economy notes a desire to raise societal wellbeing.[9]

Given the growing importance of this data in national policy, we look to investigate a research question that can provide an insight into the validity of the wellbeing information we gather.

How would we expect emotions and life evaluations to vary on a day-to-day basis?

One of the new aspects of the Graduate Outcomes survey (relative to its predecessor, the Destinations of Leavers from Higher Education Survey) was the introduction of the following four subjective wellbeing questions, which align with those currently asked by the ONS in the Annual Population Survey.[10]

1. On a scale of zero (extremely dissatisfied) to ten (extremely satisfied), how satisfied are you with life nowadays?
    
2. On a scale of zero (not at all worthwhile) to ten (extremely worthwhile), to what extent do you feel the things you do in your life are worthwhile?
    
3. On a scale of zero (extremely unhappy) to ten (extremely happy), how happy did you feel yesterday?
    
4. On a scale of zero (not at all anxious) to ten (extremely anxious), how anxious did you feel yesterday?

The first half of this set of questions ask the individual to make an evaluation of their life, while the latter two relate to very recent emotions experienced by the graduate. In the US study we referred to earlier, an examination is undertaken into how the day of the week a survey is completed is associated with wellbeing. The authors hypothesise that they would observe greater differences by the day of the week (i.e. short-term life conditions) for present emotions, with life evaluations more dependent on overall economic and social circumstances. Their empirical investigation supports their predictions, which they note improves trust in the validity of such data, given the alignment with theoretical expectations. We therefore look to carry out a similar exploration for data relating to the UK.

Our Graduate Outcomes sample consists of UK domiciled graduates who completed a higher education course in either 2017/18 or 2018/19 and whose most important activity was paid work for an employer. We link this to data we receive on the survey process (also known as ‘paradata’), such as the day of the week that the graduate submitted their survey response, as well as mode of completion. As graduates do not have to finish the survey in one visit, it is possible that they did not necessarily answer the four wellbeing questions on the same day they sent their final response to HESA. However, the likelihood of this is reduced by the fact that the subjective wellbeing questions appear at the very end of the survey.

As the questions on happiness and the degree to which life is satisfying/worthwhile use the same scale, we will focus on these in our analysis for ease of comparison. Hence, we restrict our dataset to those who answered all three of these wellbeing questions and for whom we have the associated data on day and mode of survey completion. The final sample is comprised of 334,095 graduates.[11] The hypothesis we seek to explore empirically is as follows:

Hypothesis: Happiness data will show greater variation over the course of the week than responses on the extent to which life is satisfying/worthwhile.

Results

Figure 1 illustrates that – as expected – we see greater fluctuations by day of the week in self-reported happiness relative to life evaluations. Those who respond to the survey on a Sunday (and hence are asked about their happiness levels on Saturday) display the highest mean scores, followed by those who replied either on a Saturday or Monday. Our findings are therefore in line with the US paper on this matter, with happiness levels rising at (or as we approach) the weekend. Though we are unable to describe why this may be the case through our data, the US study suggests that this can be explained by individuals having more time to spend with friends and family.

Figure 1: The relationship between day of the week the survey was completed and different measures of wellbeing.

As we also have data on mode of completion available to us, one of the additional investigations we are able to undertake is to examine whether these results continue to hold when we control for how the individual responded to the survey. Previous work looking at the relationship between mode of collection and personal wellbeing responses has identified that answers are sensitive to how an individual completed the survey.[12] In the Graduate Outcomes sample we analyse, we note that a greater proportion of those submitting over the weekend participated via telephone (Figure 2).

Figure 2: The relationship between mode of survey completion and day of the week

Figures 3 to 5 in Appendix 2 demonstrate that the relationship we observe between day of the week and wellbeing continues to hold, irrespective of the mode of completion. While personal wellbeing responses are lower among those who respond via desktop or mobile, this only shows that the two variables are correlated – that is one cannot infer that personal wellbeing and completion mode are causally related. The reason behind this is that individuals are able to self-select their mode of response, hence there could be differences in the observable and unobservable characteristics of those who respond by telephone/online.

Concluding remarks

With the findings of our empirical analysis lining up with theoretical expectations on how day of the week would correlate with personal wellbeing, our work here has provided some support to the validity of the data we are collecting.

One of the unique aspects of the Graduate Outcomes survey is the inclusion of questions on subjective wellbeing, as well as asking graduates about the extent to which they believe their activities are meaningful, utilise their skills and in line with their aspirations. In future research therefore, we will be looking at the relationship between our design and nature of work measure and the different items of subjective wellbeing (something that was noted as being of interest by some of our stakeholders who responded to our 2021 consultation on this newly created variable).[13]

The expectation will be that a positive association will be observed between this specific dimension of employment quality and subjective wellbeing. However, given the discussion in this insight, a further area of consideration will be to assess whether the extent of the correlation varies depending on whether one is considering the life evaluation questions or emotions. If indeed overall life circumstances have a greater association with life evaluation relative to present emotions, we may anticipate that the design and nature of work displays a stronger relationship with questions on life being satisfying/worthwhile relative to one on happiness.

Next - Appendix 1: Tables of summary statistics

Related content

• Graduate Outcomes : A statistical measure of the design and nature of work