Research concludes there's no need to weight
Research finds no benefit in weighting Graduate Outcomes data.
Receiving the recent recommendation from our external research contractor that the latest Graduate Outcomes survey data should remain unweighted, was very reassuring. This recommendation from some of the UK’s leading experts in survey design and weighting provided a strong endorsement of HESA’s design and operation of the survey.
The recommendation mirrors the decision HESA made last year not to add weighting to the 2017/18 survey data.
However, we didn’t want to rely on analysis of a single set of survey data, so we committed to assessing the next iteration of data for non-response effects once it became available for 2018/19. This year we decided to commission experts from outside HESA to get an independent perspective and we were delighted to contract the team at the Institute for Social and Economic Research (ISER) at the University of Essex.
Understanding the impact of non-response on Graduate Outcomes data
Graduate Outcomes aims broadly to survey everyone who qualified from higher education in each academic year. It’s not a compulsory survey, like the UK population census for example, so graduates we contact don’t have to respond. There may be many reasons why they don’t, and the resulting level of non-response has the potential to adversely affect the quality of data collected through the survey.
For instance, it can cause non-response bias. This is when survey results show systematic differences between those who responded to a survey and those who didn’t. This can have a knock-on effect for key statistics we derive from the data, in that statistics derived from the sample of graduates who responded to the survey may not accurately reflect the wider population. Weighting is an approach that can mitigate these impacts by applying a numerical adjustment to compensate for systematic bias in the data. Good survey design and operation can be used to reduce the need for weighting. For example, as the Graduate Outcomes survey is being collected we dynamically test response levels within key groups of graduates and re-order survey calling queues to improve consistency in response rates across those groups. However, for most large-scale surveys with appreciable levels of non-response, weighting is often the default expectation.
Given the scale of the Graduate Outcomes survey (we aim to contact over 750,000 graduates each year), and the range of important uses made of the resulting data it was very important for HESA to ensure that a thorough assessment was made of the need to apply weighting.
How the researchers reached their conclusion
Weighting models tested on key survey outcomes
As they say in shampoo commercials – “here comes the science bit…”
First, the ISER researchers focused on UK domiciled graduates and developed a suite of weighting models (of varying complexity) that could be tested on two key survey outcome statistics. These were:
- The proportion in employment and/or study
- The proportion in highly skilled employment and/or study
These outcome statistics were chosen because they are central to many onward uses of Graduate Outcomes data. The models were formed through identifying variables that correlated with both non-response and the two outcomes of interest – the necessary condition for weighting to assist with improving the quality of statistics derived from the survey data
As with the analysis of 2017/18 data, survey statistics were generated at the overall level, as well as by specified sub-groups (e.g. provider, subject, subject within provider, ethnicity, sex, age and disability).
Conclusion: there is no significant difference between weighted and unweighted statistics
For both outcomes (proportion in employment and/or study and proportion in highly skilled employment and/or study), the ISER researchers found that there was no significant difference between the weighted and unweighted versions of statistics.
This finding was robust to the types of variables included in the weighting procedure and when introducing non-UK graduates into the analysis.
On the basis of the findings of this research, ISER recommended to HESA that all uses of this data in HESA published statistics and open data should not be weighted. HESA has accepted this recommendation, so none of our data products using Graduate Outcomes data will feature weighting this year.
Whilst this conclusion provides validation of the quality of Graduate Outcomes data this year, we will not rest on our laurels. We will continue to assess possible non-response effects with each new annual set of survey data, to ensure the Graduate Outcomes survey remains a high-quality and trusted source of information.
Forthcoming technical report
For those of you who would like more detail on the findings, we are currently working on a technical report alongside a non-technical summary, which will both be published soon.
The first release of data from the 2018/19 Graduate Outcomes survey will be published on 20 July 2021.
Edit on 20 July 2021: The ISER technical report on statistical weighting is now available to download: