Future proofing our approach to weighting

Regular users of the Graduate Outcomes survey data will no doubt be aware of the extensive analysis undertaken in the first two years of the survey to assess the impact of non-response on data quality. This work aimed to determine whether the level of non-response was likely to create ‘bias’, i.e. rendering data collected through the survey unrepresentative of the whole population of graduates. We also sought to test whether the application of statistical weighting could improve the quality and representativeness of the data.

In the first survey year (results published in 2020) HESA analysts completed this work and we published a final report on our website. In the second year of the survey (published in 2021), HESA sought an independent expert perspective on the topic, commissioning analysis from the Institute for Social and Economic Research (ISER), based at the University of Essex. The final report by ISER was published in June 2021.

Both sets of analysis reached a similar conclusion – that there was no evidence of substantial non-response bias in the survey data, and that the application of statistical weighting (calculated using a variety of weighting models and approaches) did not significantly improve the quality of estimates derived from the data. HESA therefore reached the decision not to recommend the application of statistical weighting on any data published from the survey.

Furthermore, the final report by ISER made the following recommendation:

“Given the strength of the findings of the current study, and their stability over two years of the survey, it would seem unnecessary to do this every year, but once every several years may be advisable.”¹

A longer-term plan for weighting assessment

HESA has considered this recommendation and discussed it with the Graduate Outcomes Steering Group. Taking into account the findings from both weighting studies with the ISER recommendation, along with the highly resource-intensive nature of the analysis undertaken, HESA has determined the longer-term plan for assessment of weighting within the survey. HESA will schedule further in-depth assessments of representativeness and effects of non-response once every five years, with the next assessment expected in 2026.

During the periods between these in-depth assessments HESA will undertake an annual smaller-scale ‘surveillance assessment’ process. This will involve the ongoing analysis of response rate patterns that is routinely undertaken during the survey field work. In addition, following the completion of surveying, we will apply the preferred weighting model which was determined by ISER to derive a set of test weightings. We will then compare weighted with unweighted estimates across a range of graduate sub-groups to test whether the weighted estimates deliver significant quality improvements.

Graduate Outcomes data published by HESA will remain unweighted until such time as we can determine and demonstrate that weighting significantly improves the quality of survey estimates. Data from the third year of the survey, to be published at the end of May 2022, will therefore remain unweighted.

Maintaining a watchful eye

There is, of course, a possibility that any of the annual surveillance assessments undertaken between the five-yearly in-depth assessments may indicate that substantial non-response bias may be emerging and that the application of weighting could improve the quality of survey estimates. In that eventuality, HESA will bring forward the date of the next full assessment so that the issue can be explored thoroughly. That set of circumstances may cause HESA to revisit the need for statistical weighting. However, we would aim to provide as much advance notice to users as possible before weighting is applied to HESA statistical products.

One benefit of this approach is that the period needed for a full assessment – at least six weeks – can be removed from the production schedule for the Graduate Outcomes statistical release. For 2022, this has enabled HESA to bring forward the publication schedule significantly from previous years. We hope that users will welcome publication on this much earlier schedule, together with a higher degree of confidence that statistical weighting is unlikely to be required over the next few years, subject to the findings of the surveillance assessments.

I look forward to sharing more about the content and labelling of this year’s Graduate Outcomes outputs in an upcoming blog.

¹ https://www.hesa.ac.uk/files/ISER-Graduate-Outcomes-weighting-report-20210720.pdf, Conclusions, page 20.