Skip to main content

Assessment of gaps

The Graduate Outcomes survey was designed, as far as possible, to meet likely user needs for data on what graduates do after finishing HE and how they feel about their careers so far. We maintain a watching brief on policy issues that are of relevance to data about graduate outcomes, and report internally on emerging trends. Since the development of this survey was an extended project, however, there may be some gaps between our outputs and user needs. This could be because needs are changing faster than the survey development process, or because there are trends of which HESA is not yet aware. Since the publication of the first year of Graduate Outcomes data, we have been soliciting further feedback from users of the data via a variety of channels, including collecting feedback submitted via the website and conducing an extensive programme of sector engagement, in order to assess how well the initial publication has met user needs. As we collect user feedback, we aim to incorporate it, as appropriate, into any adjustments we make to the survey and resulting statistical outputs. Our approach to evaluation is covered in the Survey methodology[1].

HESA is already aware of some areas in which we are working to make improvements in the data which we collect and publish. Regional employment and skills gaps are important areas of current policy interest; in the spring 2020 budget speech, the government emphasised a policy of ‘levelling up’ across the UK, aimed at providing opportunities in under-served regions and reducing regional disparities, and a recent OfS initiative offered grants to universities to work with local employers to develop graduate jobs[2]. While the Graduate Outcomes survey collects data on location of domicile, HE provider, and place of work (for those graduates in work), the year one statistical outputs for the survey analysed graduate outcomes only by country of provider and in some cases domicile; the year two data provides more granular geographic detail, including analysis of place of work at the level of Government Office region[3]. For the second year of Graduate Outcomes publications, we have in addition carried out a review of the base population used for response rates (so as to include for the sake of consistency all seriously ill or deceased graduates) and the salary outlier thresholds.

Additionally, the ongoing data collection process continues to bring to light areas in which adjustments to the survey questionnaire have the potential to improve data quality and our ability to produce outputs which meet user needs. Significant changes to the survey questionnaire require approval from the Graduate Outcomes Steering Group; as we continue to assess the survey and integrate feedback from users, HESA will continue proposing changes to the Steering Group wherever it seems that modifications will enable us to produce data which better serves the needs of our stakeholders. At the end of 2020 we commenced an extensive review of Graduate Outcomes, aimed at identifying improvements to various aspects of the survey such as data collection instruments and survey methodology. As part of this we are currently reviewing the survey questionnaire with the aim of re-establishing data requirements of our users by making sure questions in the survey are still fit for purpose and reducing respondent burden by removing redundant questions. Several changes have already been identified as a result and they will be confirmed over the coming months and implemented in the next collection, i.e., 2020/21.

During the second year of surveying, we also considered the impact of Covid-19 on both survey operations and the instruments themselves[4]. On 19 May 2020, we issued an update explaining the issues we had considered and the decisions we had come to with respect to the pandemic[5]. Our focus was on ensuring that graduates could self-administer the survey to accurately reflect their personal situation and that interviewers could support participants sensitively and appropriately. We added a clarification to the list of graduate activities to clarify that furloughed employees should still identify themselves as undertaking paid work for an employer. We also added supportive text to the survey (both on the online and CATI version) signposting participants to mental health and wellbeing organisations across the world (the Samaritans, Befrienders Worldwide and Mind). We also identified other areas where the impact of the pandemic would need to be taken into account  either in outputs (as was the case for salary, which is covered in the Reliability of sensitive data section of this report, on the handling of sensitive data) or in the future design of the survey (in particular questions about the location of work, since working locations have been impacted so significantly for many workers). These latter issues have been taken forward in the survey review activity currently being conducted by the Steering Group supported by HESA.

Next: Accuracy and reliability


[2] Rishi Sunak. Budget Speech 2020. 11 March 2020. https://www.gov.uk/government/speeches/budget-speech-2020
OfS. 2020. No place like home – new fund to boost local graduate opportunities. https://www.officeforstudents.org.uk/news-blog-and-events/press-and-media/no-place-like-home/

[3] See the Location of work data - handling free text section for discussion of the work which has been done to improve the quality of data on location of work.

[4] Further detail on the impact of the Covid-19 pandemic on the 2018/19 Graduate Outcomes data can be found in The impact of the Covid-19 pandemic section of this report, and in the accompanying insight brief: https://www.hesa.ac.uk/insight/20-07-2021/impact-covid-19-graduate-outcomes

[5] For information on our assessment of gaps as a result of the Covid-19 pandemic, see: https://www.hesa.ac.uk/news/coronavirus#acc1