Graduate Outcomes is a national survey, now in its third year of publication, of students completing courses of higher education (HE). It is conducted across the UK and seeks to survey the entire graduate population. It is the largest annual social survey in the country and is run by the Higher Education Statistics Agency (HESA) – a producer of official statistics. In this third year of publication from the survey, we have removed the ‘experimental statistics’ badge from our Graduate Outcomes outputs, as they are no-longer newly-developed, and we have published extensive evaluation of them. In March 2021, we were pleased that the Office for Statistics Regulation (OSR) wrote to us noting a range of features that demonstrate the trustworthiness, quality, and value of the Graduate Outcomes statistics and which may support our eventual application for National Statistics designation. This quality report has been written to help analysts evaluate the quality and coverage of any data they intend to use in the context of the intended application to ensure that it is fit for their purpose. Although we are no longer badging our Graduate Outcomes outputs as experimental statistics, we remain committed to assessing and improving the quality of our survey and the outputs derived from it. We are keen to hear what users think of the products. Contact our Official Statistics team ([email protected] or (0)1242 388 513 [option 2]) with feedback and suggestions.
Graduate Outcomes data has been designed to be relevant to a wide range of user needs. The data reflects what we know about the requirements of prospective and current students; graduate employers; the HE sector and its funders and regulators; national, devolved and local governments; the press; and civil society, to have access to an independent and trusted source of information about graduates. It covers longstanding areas of interest in the activities graduates are doing, including whether they are in work or further study, and what their job or course is about. The survey also collects newer data where respondents are asked to reflect on the experience of being a graduate, their subjective wellbeing, and offer information about the characteristics of self-employment.
Accuracy and reliability
The survey offers information sourced directly from graduates, and this report explores the extent to which this can be relied upon as accurate. While no social survey can offer the individual-level precision of an administrative data source, the scope of topics in Graduate Outcomes is much broader than such sources. Graduate Outcomes’ sample size and response rate is much larger than for other surveys. We have found the data to be representative of the population for most statistical purposes. Our decision not to weight the data during the first year of publication was based on rigorous research reviewed by our peers. We have since commissioned an external study during the second year to evaluate whether weighting should be applied, and our contractor likewise concluded that there would be no advantage to weighting our data. A full-scale review assessing the need for weighting will now take place every five years, though surveillance activity will be carried out on an annual basis to see if there is any potential evidence to suggest that this date may need to be brought forward.
Table 0: Overall 2017/18 survey response rates by group (full responses)
|Main target group||Target||2017/18 response rate||2018/19 response rate||2019/20 response rate|
|UK domiciled, full-time||60%||52.3%||53.6%||53.4%|
|UK domiciled, part-time||60%||48.7%||49.5%||51.5%|
Many users wish to analyse sub-samples of data about graduates. Sample sizes are important when using disaggregated data. Analysts should consider the sample sizes, and any uncertainty that generates. HESA has published confidence intervals on key data tabulations to assist in understanding how reliable the data is. This quality report also explains sources of known or potential bias we have identified, to help analysts decide how they should use the data, safely. We offer specific advice around using the data for regional or sub-regional geographic analysis. We also describe our survey instrument and processing approach in detail. One such section explains the creation of occupational and industrial classifications, and our high confidence in what has been produced.
Timeliness and punctuality
Through extensive consultation with users and stakeholders, the census week at 15 months after course completion was determined as the best point at which to balance the need to generate meaningful insights into career and other outcomes with the need to deliver good rates of survey response. One implication of this is that our statistics include those who went straight on to postgraduate studies after their bachelor’s degree and who may only just have finished at time of survey. Depending on onward use it may not be appropriate or timely to compare those who have spent 15 months in the labour market with those just graduating from a further qualification and graduate responses to the survey may be driven more by the second qualification achieved. Our publications make filtering these individuals from the data easy to achieve. Given our decision to amend the publication date for the Statistical Bulletin, we also comment on the punctuality of production in this section.
Accessibility and clarity
The Graduate Outcomes data is designed to be accessible, and users can view the data on our website, and download our data to perform their own analysis and visualisations. Open data is released under a Creative Commons 4.0 CC-BY license. We are also making aggregate and disaggregated survey data available through our data processor, Jisc. To find out more about how Jisc data analytics can help you, see: www.jisc.ac.uk/data-and-analytics. In addition we supply microdata.
The data release is accompanied by a comprehensive range of supporting information. Besides this quality report, users can find a Survey methodology, coding manuals, reports, blogs, and detailed guidance on our website. There is a lot of information available, which can be daunting. We provide clear access routes to this information from the publications themselves and for visitors navigating to our website. For more expert analytical and technical users of Graduate Outcomes data, we have developed a user guide. The user guide has been designed to make navigating and accessing the large body of supporting information easier. We particularly welcome feedback on the approach we have taken to presenting the User Guide, to help us improve it.
Coherence and comparability
Graduate Outcomes forms the newest member of a family of exceptionally rich information about the HE sector. It coheres with the HESA Student records (and other data about HE in further education (FE) settings) to which it can be linked. We have begun a study comparing information relating to further study activity collected from respondents to the Graduate Outcomes survey, with similar variables available within the Student records. This work remains underway at the time of writing and a summary of current progress is offered within section 3.5. Further detail will be published in due course.
The Graduate Outcomes survey from which this year’s statistical releases have been derived was carried out entirely during the coronavirus pandemic. Following on from last year’s research into the impact of the pandemic on the 2018/19 survey data, especially on whether the data is comparable over time, we carried on a new programme of analysis of the 2019/20 data . An overview of the results of that research can be found in The impact of the Covid-19 pandemic section, and further detail can be found in the accompanying insight brief covering the impact of the Covid-19 pandemic on Graduate Outcomes 2019/20.
Graduate Outcomes survey results can also be used in conjunction with other data HESA collects about HE providers, their staff, finances, estates, and interactions with business and society. This survey replaces the former Destinations of Leavers from Higher Education (DLHE) survey and differs from it significantly in a number of ways that are explored later in this report. Data and statistics drawn from these two surveys are not directly comparable. HESA advises all data users against attempting to directly compare data between Graduate Outcomes and DLHE. Any such comparisons are likely to generate highly questionable results that are open to misinterpretation.
The Graduate Outcomes survey offers a rich and regular source of information collected directly from graduates themselves, offering their perceptions of their outcomes to date, as well as factual information about the kind of work they are doing, their salary and contractual status, or their further study options. This presents a breadth and level of detail about graduate experiences unparalleled in any other data source. It offers context to the tax and benefits data at the core of the Longitudinal Educational Outcomes (LEO) data from the Department for Education. It also complements the Labour Force Survey (LFS) by shining a spotlight on recent graduates and their activities. Our future plans include assessing how comparable our data is with similar variables in these other data sources and over time. This will provide users further understanding of the quality of the Graduate Outcomes data, to increase trust in our data source and methods, and to demonstrate the value the survey offers to our understanding of society.
 Mark Pont to Jonathan Waller: Higher Education Graduate Outcomes Data. 2021.