Review topics and main data items
In DLHE, the survey was focused on employment and employment activity. Graduate Outcomes aimed to give a wider understanding of the graduate journey.
We identified the LDLHE questions as falling into the following broad categories:
- Types of activity: employment, study, travelling, etc. (all and the most important one).
- Employment (including the job title, duties, salary, employer and location, motivations and how the role was obtained). The Standard Industrial Classification (SIC) of the employer and Standard Occupational Classification (SOC) of the role are both derived from this section.
- Further study (what is being studied, where and how funded).
- Graduates working in regulated professions (for linking purposes).
- The HE experience and preparedness for future activity
We asked about the continuing appropriateness of these areas in the first consultation and received strong support for continuity, subject to iterative improvement. Detailed consultation responses and feedback from other sources helped us to improve on the DLHE question set. We also committed to commissioning cognitive testing of the proposed question set, to ensure that it was fit for purpose. The details of this are covered in part two of this methodology statement, including links to the published findings.
One exception was for the hypothetical questions on work preparedness. These were deemed to be unnecessary and we resolved to remove them.
LDLHE / DLHE questions had been in use for many years and had in most cases proved their reliability. The model we designed in large part aimed to set these same questions in a more logical routing structure, which would preserve the best of the LDHLE / DLHE approach, while allowing richer data to be accumulated. It would also address known deficiencies of those surveys, such as the absence of almost all information about graduates not currently in employment as one of their activities.
A new measure for graduate success
The design that was produced (which was adapted slightly following cognitive testing) is available in the: New DLHE review archive.
We sought feedback on outline proposals for developing a new measure of graduate success. The first consultation had revealed a very high level of support for a new measure, coupled with a range of conflicting viewpoints on how best to achieve this. The working group had initially explored feasible additional mechanisms for capturing different types of self evaluations of outcomes from HE, in ways that are amenable to quantitative analysis. Our outline proposals were as follows:
- The application of a skills framework could help us understand the extent to which graduates are deploying learned skills at work (or whatever path they are following), and would add the voice of graduates to the debate about the skills (or perhaps attributes or competencies) required for graduates to thrive. This would also contribute a significant source of information to debate and research about the skills requirements of jobs and employers.
- The use of a widely-adopted subjective wellbeing framework would help us understand and demonstrate the extent to which HE has a positive impact on attitudes, sense of worthwhileness and satisfaction with life, comparable to other segments of the population. We might also look at other related areas, such as social and cultural capital; autonomy; self-actualisation or resilience.
- Net Promoter Score (NPS) which measures loyalty, offering a proxy for satisfaction as well as having predictive power around the potential for growth. We are all familiar with being asked this question (“would you recommend…”) in market research contexts. It is an approach commonly used in commercial benchmarking and is starting to be used in HE.
- A link back to previous surveys or activity which could offer the chance to observe change in selfperceptions, goals, capabilities or attitudes over time, measured in consistent ways.
- A new self-evaluative question seeking to measure outcomes from the graduate’s viewpoint and according to their own success criteria.
We also sought alternative proposals. We published our reflections on the responses to this question in the synthesis of responses to the first consultation which demonstrated substantial support in principle, but fragmented views on how to proceed. As with other national surveys (e.g. the Labour Force Survey and Annual Population Survey), it is timely to consider adding wellbeing-related questions to Graduate Outcomes, to support policymaking and decision-making that take wellbeing into account.
The model we developed synthesised a great number of interweaving viewpoints to realise a data source that would use data from a number of sources, including a centralised census survey, LEO data and HESA’s Student record, as well as the possibility of using other sources if appropriate. We also reflected extensively on the literature published as a result of the ONS’ project to develop measures of subjective wellbeing1.
Some of the ideas for alternative measures of graduate success were eventually adopted as optional banks of questions, for example the Net Promoter question, which is a well-defined evaluative tool widely used in industry and of which we discovered increasing use in the HE sector.
At the request of HESA’s statutory customers, the ONS’ Subjective Wellbeing question set, which was one of the proposed options for capturing graduate outcomes, and which had already been deployed successfully in the final iteration of LDLHE, was made part of the core Graduate Outcomes survey. This question set had been tested extensively and offers a measure of affect by gathering the respondent’s subjective experience of wellbeing “yesterday”.
However, users sought a measure of wellbeing that connected the respondent’s current situation with their experience of HE. Since respondents to our consultation had given a mixed response to the straightforwardly evaluative Net Promoter question, and the ‘hedonic’2 Subjective Wellbeing set; while offering a wealth of viewpoints and commentary, we determined that a set of questions that sought to capture a ‘eudemonic’ measure of worthwhileness was also required.
Development effort was therefore focused on synthesising a question set that reflected the needs of stakeholders to capture qualitative reflection by graduates responding to the survey on their outcomes so far. In doing so, we reflected on the following matters:
- Higher education is intended to develop skills and attributes that contribute to both instrumental measures of success and human flourishing. Measuring HE’s contribution to perceptions of this was deemed a priority. This was especially the case since the idea of utilising a skills matrix approach (point one above) was deemed unwieldy, expensive and unworkable.
- The Taylor3 review had revealed the importance of work in both giving purpose and meaning, and also as a source of rewards across a range of value measures. Our working hypothesis was that developing and utilising intellect, mastery of knowledge and skills, and self-knowledge were all promoted by HE, and we might therefore expect good outcomes from HE to be correlated with better scores on a measure of “good work”.
- Research4 shows that ‘eudemonic’ measures of wellbeing correlate with self-acceptance and environmental mastery: characteristics that HE is instrumental in developing. Furthermore, since higher education is associated with life-wide benefits (in employability, health, etc.) that can support positive outcomes over the lifecourse (and since there is evidence of appreciation of the value of HE increasing with the time from graduation) we wanted a measure that recognised the longer-term impacts expected of HE and so sought to quantify progress toward the graduate’s own future goals.
We could not identify an existing model for gathering data of this kind and so HESA undertook to develop a proposal question set that would synthesise and serve the variety of needs expressed by stakeholders. We are grateful to the many individuals who supported us in this creative reflection, too numerous to mention. The new ‘graduate voice’ measures we developed are intended to capture the attributes of ‘eudemonic’ wellbeing relevant to graduates from HE in three dimensions:
- Meaningfulness or importance of the activity to the graduate.
- Skills utilisation.
- The graduate’s progress towards future goals.
The three core questions are asked of all graduates and there are versions tailored for those in work, further study, or doing something else. These questions were cognitively tested and received a very positive response, as well as some advice for refinement, which was taken on board. Details of them are available in the Graduate Outcomes Survey Results coding manual.
Consultation respondents wanted to understand the link between qualitative self-assessments of outcomes with factors like salary, location, job role, and degree class. This could play a role in better evaluating the impact of careers education and personal development (whether separate or embedded in the curriculum). By deploying versions of these questions to all graduates, no matter what their current activity is, the Graduate Outcomes survey will also allow us to gather deeper insights into graduates pursuing nontraditional career paths, such as those developing creative portfolios or setting up a business.
It will also help us understand more about the experiences of graduates who are not engaged in economic work, for instance who are travelling or who have caring responsibilities. The questions we developed received a positive response in the second consultation. These questions will provide a richer picture of the diversity of graduate outcomes and will help redefine how we understand graduate success.
- For instance Dolan, P., Layard, R., & Metcalfe, R. (2011). Measuring Subjective Well-being for Public Policy. Office for National Statistics, 21. http://eprints.lse.ac.uk/35420/1/measuring-subjective-wellbeingfor-public-policy.pdf
- Measures of hedonic wellbeing aim to understand positive and negative affect, using questions that promote recall of recent experience of feelings. The ideal data collection instrument for hedonic wellbeing would therefore be something akin to a brain scan. Conversely, eudemonic wellbeing questions attempt to measure human flourishing in a more evaluative and reflective way. For a discussion of these concepts, in various contexts see, for example, the following articles:
Vanhoutte, B. (2015). Hedonic and eudemonic wellbeing. In: ESS ERIC (2015) Measuring and Reporting on Europeans’ Well-Being: Findings from the European Social Survey. https://www.research.manchester.ac.uk/portal/en/publications/hedonic-and-eudemonic-wellbeing(e0343dce-8207-4262-8288-5bdf8fd39a89).html
Dolan, P., Layard, R., & Metcalfe, R. (2011). Measuring Subjective Well-being for Public Policy. Office for National Statistics, 21. http://eprints.lse.ac.uk/35420/1/measuring-subjective-wellbeing-for-public-policy.pdf
Keyes, C. L. M., & Annas, J. (2009). Feeling good and functioning well: distinctive concepts in ancient philosophy and contemporary science. The Journal of Positive Psychology, 4(3), 197–201. https://doi.org/10.1080/17439760902844228
- Taylor, M. (2017). Good work: the Taylor review of modern working practices. https://www.gov.uk/government/publications/good-work-the-taylor-review-of-modern-working-practices
- See Ryff, C. D., & Keyes, C. L. M. (1995). The Structure of Psychological Well-Being Revisited. Journal of Personality and Social Psychology, 69(4), 719–727. http://midus.wisc.edu/findings/pdfs/830.pdf