The ultimate guide to Graduate Outcomes - HESA publishes survey methodology statement
Today HESA has released a two-part methodology statement on Graduate Outcomes which provides a comprehensive overview of the survey’s history and operations.
Visit the Graduate Outcomes methodology statements
If you’ve ever wondered why we started conducting this survey, how we collect data from thousands of graduates across the UK and what do we do with it afterwards, then this guide is for you. It is split into two parts, the first shines a light on the history and background of the survey; the second provides a step by step description of every stage in the life cycle of Graduate Outcomes (based on our first year of operation).
Who should read this guide and why?
• Are you a current or prospective user of Graduate Outcomes data?
• Are you an analyst with an interest in survey research methods?
• Are you interested in large change projects with a focus on topics of national significance?
• Are you simply interested in the latest HESA publications?
It will answer these key questions and more and offer operational detail that may not be available elsewhere on the HESA website. You may use this as a reference while exploring our datasets and to help you decide how best to utilise our published statistics.
It may also make you think about, evaluate and challenge some well-known or emerging concepts in the field of survey research and the measurement of graduate outcomes. We welcome your ideas as it would help inform future enhancements to the survey and help us meet our users’ requirements for this data.
What is included in the methodology statement?
Part one of the methodology statement summarises our current position and how we got here. It describes the predecessor to Graduate Outcomes (Destinations of Leavers of Higher Education survey or ‘DLHE’), the transition to a new model and the stark differences between the old and the new approaches; a theme which sets the scene for part two.
The second part offers a detailed description of each of the following stages:
• Survey coverage and sampling frame
• Survey design
• Data collection
• Data processing
• Data analysis
• Data dissemination and
• Evaluation
It includes useful signposts to various locations on our website which contain further details and documents relevant to different topics.
In addition to providing an overview of survey operations, this guide also includes a summary of significant pieces of developmental work which HESA has carried out since the survey was launched in 2018.
Our approach to survey weighting
Survey weighting is one such area of work. This is a correction technique that aims to make a sample more representative of the population. If the characteristics of the sample (those who respond to a survey) are very different to those who do not respond, survey results may be biased. Weighting makes an adjustment for this bias.
Like every survey, Graduate Outcomes also has a level of non-response i.e. people who do not take part in the survey and in theory this creates a risk of bias. As mentioned previously, bias occurs when the sample is not representative of the population. Having studied the responding sample in Graduate Outcomes we found it to be representative of the population across a range of variables that we tested. We also tried applying a range of different approaches to weighting and we found they generally made very little difference to the quality of statistics we calculated from the survey.
Following an extensive review of various approaches to weighting, a thorough assessment of our data and a series of discussions with subject matter experts, HESA has agreed with the Graduate Outcomes Steering Group (the governing body which oversees survey operations and delivery) that weighting will not be applied to statistics published by HESA from this first year of survey data.
Further details of this work are available in part two of methodology statement and for those with greater expertise in this field, a more comprehensive technical description of the analysis and conclusions on the weighting methodology will be published shortly.
Who has written this guide?
Everyone involved in the design, development and implementation of the survey. The list includes technical developers, researchers, analysts, communications experts and our suppliers. We wanted to cover all aspects of the methodology and this could only be achieved through close collaboration with the entire multi-disciplinary team that is responsible for the delivery of this survey.
It will be a ‘live’ record of the methodology, at least in the first few years as we make improvements to our data collection, processing, analysis, and dissemination policies.
Who do I contact if I need more information?
The methodology statement will undoubtedly raise questions among those who want to find out more about certain aspects of the survey. You can contact our Liaison team on +44 (0) 1242 388 531 or [email protected].
Equally, if you feel something is missing from this statement then please do get in touch with our Liaison team. We’re always keen to hear from potential users of these new data outputs, as well as existing users of our other data outputs.
