Assessment of gaps

The Graduate Outcomes survey was designed, as far as possible, to meet likely user needs for data on what graduates do after finishing HE and how they feel about their careers so far. We maintain a watching brief on policy issues that are of relevance to data about graduate outcomes, and report internally on emerging trends. Since the development of this survey was an extended project, however, there may be some gaps between our outputs and user needs. This could be because needs are changing faster than the survey development process, or because there are trends of which HESA is not yet aware. Since the publication of the first year of Graduate Outcomes data, we have been soliciting further feedback from users of the data via a variety of channels, including collecting feedback submitted via the website and conducing an extensive programme of sector engagement, in order to assess how well the initial publication has met user needs. As we collect user feedback, we aim to incorporate it, as appropriate, into any adjustments we make to the survey and resulting statistical outputs. Our approach to evaluation is covered in the Survey methodology[1].

HESA is already aware of some areas in which we are working to make improvements in the data which we collect and publish. Regional employment and skills gaps are important areas of current policy interest; in the spring 2020 budget speech, the government emphasised a policy of ‘levelling up’ across the UK, aimed at providing opportunities in under-served regions and reducing regional disparities, and the OfS has offered grants to universities to work with local employers to develop graduate jobs[2]. While the Graduate Outcomes survey collects data on location of domicile, HE provider, and place of work (for those graduates in work), the year one statistical outputs for the survey analysed graduate outcomes only by country of provider and in some cases domicile; the data for subsequent years provides more granular geographic detail, including analysis of place of work at the level of Government Office region[3]. We have also worked to create a new graduate mobility marker, which allows us to analyse patterns of graduate mobility beyond the regional level[4]. For the second year of Graduate Outcomes publications, we carried out a review of the base population used for response rates (so as to include for the sake of consistency all seriously ill or deceased graduates) and the salary outlier thresholds.

Additionally, the ongoing data collection process continues to bring to light areas in which adjustments to the survey questionnaire have the potential to improve data quality and our ability to produce outputs which meet user needs. Significant changes to the survey questionnaire require approval from the Graduate Outcomes Steering Group; as we continue to assess the survey and integrate feedback from users, HESA will continue proposing changes to the Steering Group wherever it seems that modifications will enable us to produce data which better serves the needs of our stakeholders. At the end of 2020 we commenced an extensive review of Graduate Outcomes, aimed at identifying improvements to various aspects of the survey such as data collection instruments and survey methodology. Part of this activity involved reviewing the survey questionnaire with the aim of re-establishing data requirements of our users by making sure questions in the survey were still fit for purpose and reducing respondent burden by removing redundant questions. Several changes have already been identified through this process  and were subsequently implemented for the 2020/21 collection. A second phase of the survey review began in late spring 2022, involving further assessment of how well existing survey questions meet user needs. This phase of the review will also include horizon scanning and desk-based policy research, aimed at identifying areas of existing or emerging user need which may require the development of new survey topics.  

Starting in the second year of surveying, we have had to consider the impact of COVID-19 on both survey operations and the instruments themselves[5]. On 19 May 2020, we issued an update explaining the issues we had considered and the decisions we had come to with respect to the pandemic[6]. Our focus was on ensuring that graduates could self-administer the survey to accurately reflect their personal situation and that interviewers could support participants sensitively and appropriately. We added a clarification to the list of graduate activities to clarify that furloughed employees should still identify themselves as undertaking paid work for an employer. We also added supportive text to the survey (both on the online and CATI version) signposting participants to mental health and wellbeing organisations across the world (the Samaritans, Befrienders Worldwide and Mind). We also identified other areas where the impact of the pandemic would need to be taken into account  either in outputs (as was the case for salary, which is covered in the Reliability of sensitive data section of this report, on the handling of sensitive data) or in the future design of the survey (in particular questions about the location of work, since working locations have been impacted so significantly for many workers). These latter issues are being taken forward in the survey review activity currently being conducted by the Steering Group supported by HESA.

Next: Accuracy and reliability