It is important to ensure that response rates do not fall significantly over time, especially if they are likely to reduce the usability of survey results (e.g. through the generation of less precise estimates for smaller groups of population). Target response rates could therefore be one possible measure of survey performance.
The DLHE survey was administered with target response rates, which HE providers were required to meet. Target response rates for 2016/17 were set at:
- 80% for UK-domiciled HE leavers who previously studied full-time.
- 70% for UK-domiciled HE leavers who studied part-time.
- 80% for Research Council-funded students.
- 50% for all other EU HE leavers.
- 20% for non-EU international HE leavers.
To set target response rates for Graduate Outcomes, response rates from DLHE and Longitudinal DLHE (LDHLE) were analysed; the former being a destinations survey six months after graduation and the latter being its longitudinal element conducted 42 months after graduation. In the absence of a precedent for a 15-month destination survey, the two existing surveys were deemed as the most suitable benchmarks.
As expected, the response rates achieved in the LDLHE survey were lower than those achieved in the DLHE survey. Although many factors may contribute to this decrease, time-lag is a major determinant.
The timing of the Graduate Outcomes survey is 15 months after graduates complete their studies. This is exactly one quarter of the way between the timing of the DLHE (six months) and the timing of the LDLHE (42 months). Assuming the drop in response rates over time is linear, expected response rates for Graduate Outcomes can be calculated by taking a quarter of the drop away from the response rate achieved in DLHE. This is adjusted to account for the fact that response rates may drop in a non-linear fashion and the aim to provide a more cost effective yet effective overall solution to the sector.
The resulting targets for Graduate Outcomes response rates, by main groups are:
UK domiciled full-time: 60%
UK domiciled part-time: 60%
Research funded: 65%
EU domiciled: 45%
Non-EU domiciled: 25%
These targets are mainly applicable for the entire set of survey results across a year but are also used to monitor performance of individual cohorts. This is to acknowledge the diverse composition of different cohorts in terms of graduate characteristics and how it impacts the response rates that can be realistically achieved.
The ‘headline’ response rates for the Graduate Outcomes survey are defined as:
- Numerator: Count of records with a valid response to a minimum1 (pre-determined) set of core questions (classified with a status of ‘survey completed’).
- Denominator: Count of all records in the target population, excluding those marked as dead or seriously ill.
In addition to responses classified as ‘survey completed’, a status of ‘partially completed’ has been assigned where some of the core questions are missing but the first two questions have been answered. Although partially completed responses do not contribute to the ‘headline’ response rates, these are used alongside ‘survey completed’ responses in statistical outputs and form part of the base population for this purpose. Data from such responses will appear in published statistics with unknown values for questions that were not answered.
It should be noted that achieving a high response rate is not sufficient in itself to ensure good quality data has been collected. In later parts of this methodology statement, we will be discussing the other checks and processes we have undertaken in assessing and ensuring the quality of the data.