Although Graduate Outcomes is a population survey in practice, it is inevitable that there is some under-coverage: some graduates cannot be surveyed, perhaps because their contact details were unavailable, or because they are seriously ill, or have died. We call the list of all cases we can include, the “sampling frame”. Using the coverage criteria outlined above, a sampling frame is developed for each collection year. This contains contact details (email addresses, phone numbers and in rare cases residential addresses) for all graduates eligible to take part in the survey.
A bespoke data collection portal (the ‘Graduate Outcomes provider portal’) enables the collection of contact details to be used for the sampling frame. Contact details for graduates are supplied by providers using this portal. Using the eligibility criteria (as defined under survey coverage), a population file containing unique identifiers for all graduates in coverage is made available to each provider in the portal. For those providers who return student data to HESA (part of Jisc), the population file is automatically generated from that data. For other providers, i.e. FECs in England and Northern Ireland, the information for the population file will be taken from the appropriate student data collection (the Individualised Learner Record (ILR) in England and the Consolidated Data Return (CDR) from the Department for the Economy Northern Ireland (DfENI)). HESA supplies a separate report to provide the relevant derived fields which will support providers in identifying graduates for inclusion in this return.
Once the Graduate Outcomes data collection portal opens, providers begin uploading contact details for graduates, one cohort at a time, based on dates provided in the data collection schedule. A quality assurance period runs from when the collection system opens until the ‘final updates’ deadline. This is the process in which a provider’s contact details submissions are validated on the provider portal to ensure they are of sufficient quality for use. This is facilitated by a set of quality rules developed by HESA, alongside providers’ own activity to ensure contact details are useable and accurate. Contact details should be continuously updated (if necessary) during the survey field period.
Once compiled and approved by the provider, the set of contact details for each cohort are exported to our survey supplier platform (Confirmit) through which we manage all aspects of survey design and data collection. Contact details for each cohort are compiled and processed individually, prior to data collection.
Our model of open centralisation means that HESA is responsible for delivering the survey, including ensuring graduates are surveyed, with the support of a number of suppliers. HESA depends critically on providers to fulfil a number of roles to ensure the survey is delivered successfully, response rates are maximised, and bias is not introduced.
The primary role of providers can be summarised in three activities:
- Collecting and maintaining contact details to support the creation of an accurate and comprehensive sampling frame
- Submission of contact details through a bespoke data collection portal
- Promotion of the survey to create brand awareness among prospective respondents (which we discuss in greater detail in a later section of this methodology)
Providers are instructed to refrain from contacting the graduates during the survey window to minimise risks of introducing bias.
These and other provider responsibilities are explained in more detail on our website.
Methodological decisions about the survey design and implementation are made by HESA (Jisc), with input from regulatory and funding bodies, relevant sector bodies and providers.
All HESA collections have a coding manual, which contains the relevant operational information and guidance to help aid a provider in compiling and submitting appropriate data. For Graduate Outcomes, there are two relevant coding manuals for the different aspects of the collection:
- The contact details coding manual contains all the necessary information and guidance required to enable providers to submit their contact details correctly and on time. It contains several different areas of information, ranging from population coverage, data protection and the physical data structure of the collection, to the data items (required to be returned), the data quality rules (which must be adhered to), and the user guide (which sets out, step by step, the overall processes required during the collection and use of the provider portal).
- The survey results coding manual contains the necessary information for providers to download and analyse their raw survey results (during a cohort) and the final data delivery (when appropriate).
HESA has also provided a suite of communications materials for providers to use to build brand awareness for the survey. This includes logos (in various formats), engagement materials, social media content and much more.