Skip to main content

Comparability and time series

Graduate Outcomes and DLHE

When the new Graduate Outcomes survey was being designed, the outputs developed from the DLHE data were seen to have value, and it was therefore decided to begin iterating from the DLHE approach in designing the new survey.[1] Nevertheless, Graduate Outcomes is an entirely new survey, and important differences in timescale, methodology, and survey questions between Graduate Outcomes and DLHE make it impossible for direct comparisons to be made between data from the two surveys.

For the DLHE survey, graduates were contacted six months after the completion of their qualifications; Graduate Outcomes surveys graduates 15 months after the completion of their qualifications, that is, nine months later than they would have been surveyed for DLHE. Graduates surveyed for Graduate Outcomes are therefore at a very different stage in their post-HE careers than those who were surveyed for DLHE, which means that comparing the outcomes of respondents to the two surveys will not be a like-for-like comparison.

Methodological differences between DLHE and Graduate Outcomes are another reason to avoid direct comparisons between the two surveys. Where DLHE was administered by providers who then returned data to HESA for processing and analysis, Graduate Outcomes is administered centrally to graduates. For DLHE, SOC coding was done by providers, whereas SIC and SOC coding for Graduate Outcomes is outsourced to the business data services company Oblong.[2] The central administration of both the Graduate Outcomes survey itself and its SIC and SOC coding ensures a greater degree of consistency than was possible with the DLHE survey.

Finally, although the two surveys cover similar ground, the specific questions asked by the two surveys are different. The list of activities which can be selected by respondents to Graduate Outcomes is different from the list available to DLHE respondents; the Graduate Outcomes survey gives respondents more options and, in particular, allows graduates who are in work to be more specific about the type of work they are doing. In addition to asking for more detail about areas which received less emphasis in the DLHE survey, Graduate Outcomes also includes new questions, such as the graduate voice questions, which reflect the new survey’s emphasis on providing metrics for graduate success beyond employment and salary; similarly, the SWB questions, which were previously used in the final iteration of LDLHE, have been made part of the core Graduate Outcomes survey. Given these differences in survey design, much of the Graduate Outcomes data will have no direct equivalent in DLHE.

Having decided to replace DLHE with a new and fundamentally different survey, HESA has taken the decision not to undertake, publish, or otherwise disseminate any comparisons of data between the Graduate Outcomes survey and the DLHE survey. We likewise advise all users of the two surveys to avoid making any direct comparisons between the two datasets. The two surveys are not directly comparable and any attempts to make direct comparisons are likely to lead to questionable results which are open to misinterpretation.[3]

The impact of the COVID-19 pandemic

The first UK cases of COVID-19 were confirmed at the end of January 2020, about two thirds of the way through the Cohort A survey period for year two of Graduate Outcomes. The World Health Organization declared that the outbreak of COVID-19 was a pandemic on 11 March 2020, shortly after the end of the Cohort B census week. The first UK lockdown, which was agreed in all four nations, was announced on 23 March, and, despite some easing during summer 2020, different levels of pandemic-related restrictions remained in force throughout the rest of the second year of Graduate Outcomes surveying. Given the far-reaching effects of the COVID-19 pandemic on all aspects of daily life in the UK and around the globe, including employment and study, HESA endeavoured to respond appropriately to the pandemic in its handling of the Graduate Outcomes survey and publications.

Although no changes could be made to the survey for Cohort B, Cohort C offered an opportunity to consider whether any changes could be made which would improve the quality of data collected under pandemic circumstances. From the start of Cohort C, two changes were made to the survey with the goals of allowing interviewers to support respondents and helping respondents describe their current circumstances accurately. First, supportive text was added to the wellbeing questions, signposting mental health and wellbeing organisations around the world. Second, additional guidance was added to the activity questions, instructing furloughed graduates to select the option ‘paid work for an employer’. Changes to the place of work and salary questions were also considered, given the shift to remote working and the possibility of furloughed graduates reporting lower salaries, but it was decided that changing these questions in the short term would be likely to introduce additional uncertainty rather than adding clarity; these questions, however, are currently undergoing a process of fundamental review.

After the second year of Graduate Outcomes data was received, HESA undertook a programme of analysis to determine the impact of the pandemic on data quality. We compared year two response rates to those from year one, both overall and for graduates with different characteristics, in order to determine whether the circumstances of the pandemic seem to have made some groups less likely to respond to the survey. Since the effects of the pandemic were likely to have been more pronounced for the later cohorts of year two, we also looked at response rates by cohort, comparing year two cohorts with the equivalent cohorts from year one.

Our analysis showed that the year two Graduate Outcomes data remained robust despite the changing circumstances under which it was collected. Response rates for year two remained for the most either stable or slightly higher than equivalent rates for year one; this was true both overall and when response rates were broken down by personal characteristics. Although there were some differences by cohort, these differences for the most part matched the cohort-level differences which were visible in year one. While there were some changes in graduate activities between year 1 and year 2, including a 1.5 percentage point rise in graduates in reporting themselves as unemployed and a 47% decrease in the percentage of graduates taking time out to travel, these changes are likely to reflect real changes in what graduates were doing during the pandemic rather than problems with the quality of the Graduate Outcomes data.

The 2019/20 Graduate Outcomes survey was also conducted under pandemic circumstances, with each survey cohort finishing their studies and being surveyed at a different point in the pandemic. While cohorts A and B finished their higher education courses before the start of the pandemic, the COVID-19 outbreak was declared a pandemic on 11 March 2020, as cohort C was finishing their studies; cohort D completed their higher education courses between May and July 2020, during the gradual easing of the first national lockdown. Conversely, cohorts A and B were surveyed between December 2020 and May 2021, while Covid restrictions of one sort or another were in place across the UK. Cohort C was surveyed as restrictions were gradually phased out, while cohort D was surveyed against a backdrop of rising case numbers but few legal restrictions on activity.

Given the changing pandemic circumstances of the 2019/20 survey year, we conducted a further programme of analysis to identify any effects of the pandemic on the year three survey data. As we found in year 2, response rates for year 3 remained steady or increased slightly in year 3, both overall and for graduates with different personal characteristics. In general, we saw fewer and smaller changes in graduate activity between 2018/19 and 2019/20 than we had seen between 2017/18 and 2018/19. We saw a decrease of about 1 percentage point in the rate of graduates reporting themselves as unemployed, which aligns with data from the ONS suggesting that the labour market in 2021 was beginning to recover following increases in unemployment early in the pandemic. While we expanded our analysis of the possible effects of the pandemic on the 2019/20 data to include occupational classification, industry of employment, and the relationship between industry of employment and responses to the subjective wellbeing and graduate voice questions, we saw no notable year-on-year changes in any of these areas.

The pandemic context of the 2020/21 survey year was different; although some restrictions remained in place as graduates in cohorts A and B finished their qualifications, most formal restrictions (except for some around international travel) had been lifted by the time surveying began in December 2021. Given both the decreasing magnitude of changes which might be related to the pandemic in the 2019/20 data and the increasing difficulty of distinguishing between potential pandemic effects and the effects of other factors, we took the decision that we would carry out one final programme of analysis focusing on potential pandemic impacts, but that in subsequent years we would simply include COVID-19 amongst a range of contextual factors which may have an impact on our data.

Our final investigation of the impact of the pandemic, as expected, showed few changes which could be attributed to the pandemic. While we did see a change in response rates between 2019/20 and 2020/21, that change stemmed not from the pandemic, but from the cessation of international calling. Where we saw changes in graduate activities, these changes continued the 2019/20 trend towards recovery; in particular, the percentage of 2020/21 graduates in full-time employment was the highest since the start of surveying for the 2017/18 academic year. We saw very little overall change in graduates’ reflections on their activities; graduate subjective wellbeing, after a dip in the first year of the pandemic, has remained stable since the second year of surveying.

Fuller discussions of the results of our investigations into the effects of the COVID-19 pandemic on Graduate Outcomes can be found in three insight briefs, published on the HESA website alongside the 2018/19, 2019/20, and 2020/21 Graduate Outcomes statistical releases.[4] 

Next: Respondent burden


[1] See the section of the Graduate Outcomes Survey methodology on the review of the data items collected in DLHE and LDLHE: https://www.hesa.ac.uk/data-and-analysis/graduates/methodology/review-topics

[3] See the dissemination section of the Graduate Outcomes Survey methodology: https://www.hesa.ac.uk/data-and-analysis/graduates/methodology/dissemination