Comparability and time series
When the new Graduate Outcomes survey was being designed, the outputs developed from the DLHE data were seen to have value, and it was therefore decided to begin iterating from the DLHE approach in designing the new survey. Nevertheless, Graduate Outcomes is an entirely new survey, and important differences in timescale, methodology, and survey questions between Graduate Outcomes and DLHE make it impossible for direct comparisons to be made between data from the two surveys.
For the DLHE survey, graduates were contacted six months after the completion of their qualifications; Graduate Outcomes surveys graduates 15 months after the completion of their qualifications, that is, nine months later than they would have been surveyed for DLHE. Graduates surveyed for Graduate Outcomes are therefore at a very different stage in their post-HE careers than those who were surveyed for DLHE, which means that comparing the outcomes of respondents to the two surveys will not be a like-for-like comparison.
Methodological differences between DLHE and Graduate Outcomes are another reason to avoid direct comparisons between the two surveys. DLHE was a survey of graduates conducted by providers, whereas Graduate Outcomes is a survey of graduates directly; thus where DLHE was administered by providers who then returned data to HESA for processing and analysis, Graduate Outcomes is administered centrally. For DLHE, SOC coding was done by providers, whereas SIC and SOC coding for Graduate Outcomes is outsourced to the business data services company Oblong. The central administration of both the Graduate Outcomes survey itself and its SIC and SOC coding ensures a greater degree of consistency than was possible with the DLHE survey.
Finally, although the two surveys cover similar ground, the specific questions asked by the two surveys are different. The list of activities which can be selected by respondents to Graduate Outcomes is different from the list available to DLHE respondents; the Graduate Outcomes survey gives respondents more options and, in particular, allows graduates who are in work to be more specific about the type of work they are doing. In addition to asking for more detail about areas which received less emphasis in the DLHE survey, Graduate Outcomes also includes new questions, such as the graduate voice questions, which reflect the new survey’s emphasis on providing metrics for graduate success beyond employment and salary; similarly, the SWB questions, which were previously used in the final iteration of LDLHE, have been made part of the core Graduate Outcomes survey. Given these differences in survey design, much of the Graduate Outcomes data will have no direct equivalent in DLHE.
Having decided to replace DLHE with a new and fundamentally different survey, HESA has taken the decision not to undertake, publish, or otherwise disseminate any comparisons of data between the Graduate Outcomes survey and the DLHE survey. We likewise advise all users of the two surveys to avoid making any direct comparisons between the two datasets. The two surveys are not directly comparable and any attempts to make direct comparisons are likely to lead to questionable results which are open to misinterpretation.
The first UK cases of Covid-19 were confirmed at the end of January 2020, about two thirds of the way through the Cohort A survey period for year two of Graduate Outcomes. The World Health Organization declared that the outbreak of Covid-19 was a pandemic on 11 March 2020, shortly after the end of the Cohort B census week. The first UK lockdown, which was agreed in all four nations, was announced on 23 March, and, despite some easing during summer 2020, different levels of pandemic-related restrictions remained in force throughout the rest of the second year of Graduate Outcomes surveying. Given the far-reaching effects of the Covid-19 pandemic on all aspects of daily life in the UK and around the globe, including employment and study, HESA endeavoured to respond appropriately to the pandemic in its handling of the Graduate Outcomes survey and publications.
Although no changes could be made to the survey for Cohort B, Cohort C offered an opportunity to consider whether any changes could be made which would improve the quality of data collected under pandemic circumstances. From the start of Cohort C, two changes were made to the survey with the goals of allowing interviewers to support respondents and helping respondents describe their current circumstances accurately. First, supportive text was added to the wellbeing questions, signposting mental health and wellbeing organisations around the world. Second, additional guidance was added to the activity questions, instructing furloughed graduates to select the option ‘paid work for an employer’. Changes to the place of work and salary questions were also considered, given the shift to remote working and the possibility of furloughed graduates reporting lower salaries, but it was decided that changing these questions in the short term would be likely to introduce additional uncertainty rather than adding clarity; these questions, however, are currently undergoing a process of fundamental review.
After the second year of Graduate Outcomes data was received, HESA undertook a programme of analysis to determine the impact of the pandemic on data quality. We compared year two response rates to those from year one, both overall and for graduates with different characteristics, in order to determine whether the circumstances of the pandemic seem to have made some groups less likely to respond to the survey. Since the effects of the pandemic were likely to have been more pronounced for the later cohorts of year two, we also looked at response rates by cohort, comparing year two cohorts with the equivalent cohorts from year one.
Our analysis showed that the year two Graduate Outcomes data remained robust despite the changing circumstances under which it was collected. Response rates for year two remained for the most either stable or slightly higher than equivalent rates for year one; this was true both overall and when response rates were broken down by personal characteristics. Although there were some differences by cohort, these differences for the most part matched the cohort-level differences which were visible in year one. While there were some changes in graduate activities between year 1 and year 2, including a 1.5 percentage point rise in graduates in reporting themselves as unemployed and a 47% decrease in the percentage of graduates taking time out to travel, these changes are likely to reflect real changes in what graduates were doing during the pandemic rather than problems with the quality of the Graduate Outcomes data.
A fuller discussion of the results of our investigation into the effects of the Covid-19 pandemic on Graduate Outcomes can be found in the accompanying insight brief, The impact of the Covid-19 pandemic on Graduate Outcomes 2018/19, published on the HESA website.
 See the section of the Graduate Outcomes Survey methodology on the review of the data items collected in DLHE and LDLHE: https://www.hesa.ac.uk/data-and-analysis/graduates/methodology/review-topics
 For more detail on SIC and SOC coding methodology, see https://www.hesa.ac.uk/definitions/operational-survey-information#data-classification-sicsoc
 See the dissemination section of the Graduate Outcomes Survey methodology: https://www.hesa.ac.uk/data-and-analysis/graduates/methodology/dissemination