How does the sampling frame relate to the population?

This section deals with what we know about coverage error. The population of interest is graduates from HE-level courses. The exclusions from this are explicit and intentional (see footnote 8 of the Accuracy and reliability section). The survey does not attempt to contact students who did not graduate – these individuals are counted elsewhere in HESA’s Student data. Where students graduate with a different award than that they originally intended at the beginning of studies, they will be included in the sampling frame (except where they fall into the exclusions we list).

The administrative data described in the previous section comprises all publicly-funded and/or regulated HE providers in the UK. There are known instances of duplication of student identifiers between providers within the Individualised Learner Record (ILR)[1] (which does not have a globally unique identifier akin to the HESA unique student identifier, or HUSID) and between the ILR and HESA data (where ‘franchise’ arrangements exist). The Office for Students is the expert in handling both types of duplication and has isolated and removed these within their dataset prior to sharing data with HESA. Where other administrative data sources are concerned the separation of reporting environments militates against duplications occurring.

One legitimate question is how complete the administrative data is: could there be any under-coverage of HE graduates, because the provider they studied at is not included in the administrative data? In short, our sampling frame represents the overwhelming majority (probably in excess of 99% based on Hunt and Bolliver’s figures[2]), but not absolutely all, UK HE students.

While there is no definitive answer to how many are missing, it is known that a small amount of HE-level provision remains outside the formally-regulated sector. Research commissioned over nine years ago by the former Department of Business, Innovation and Skills identified a minimum estimate ‘of 674 named privately funded HE providers operating in the UK. […] Most providers identified [were] relatively small in scale; 217 of the 674 had fewer than 100 students. Only 35 providers had over 1000 students, with five of these having over 5000 students.’[3] Subsequently, the Higher Education and Research Act 2017 has had the effect of expanding the sphere of HE regulation in England to include a group of organisations referred to as ‘Alternative Providers’[4]. While this terminology is no longer used officially, the providers brought into the regulated sphere under this designation are now included in administrative data returns used to create the sampling frame, and include the majority of larger organisations identified by the literature[5]. In the Graduate Outcomes open data, we provide some information at provider-level, and users can therefore see the list of providers, where data on their graduates is included in our outputs.

Volatility in the segment of the HE marketplace comprising the very smallest providers means that some will not have provided full data for inclusion in the sampling frame, nor would they have shouldered their share of the costs of surveying, having undergone market exit. Further research  indicates that there are ‘some 813 private providers in operation in the UK – a significant increase on the 732 and 674 recorded in 2014 and 2011 respectively.’ Many of these ‘are small scale, concentrating on sub-degree or postgraduate qualification across a narrow band of subjects – often characterised as being popular but with low overheads[6].’ For the most up-to-date documentation on what is known about the scale and scope of this part of the HE sector, readers are directed to the Hunt and Bolliver paper listed in the references. In HESA’s published data no attempt has yet been made to provide estimates that include this population, as we know too little about the characteristics of students and graduates from this part of the sector.

We therefore estimate that the list of graduates in the sampling frame comprises in excess of 99% of the population of interest, and that the impact of this slight undercoverage is therefore likely to be very slight in England, and negligible in Wales, Scotland, and Northern Ireland[7]. However, we remind users of the discussion about the provenance of contact details collected against the sampling frame. The practical effect of missing contact details, and those found to be unusable or ineffective during fieldwork, reduces the effective size of the sample, and limits the achievable number of responses.

Next: The sample