Methodology: Details

HESA was tasked with developing a methodology for collecting graduate outcomes data that was efficient, future-proof, fit for purpose, and supportive of legislation. This methodology also needed to provide us with rich and robust outcomes data.

Open centralisation

A key aim of the NewDLHE review is ensuring that we gather data which is perceived as objectively impartial and of a high standard. As part of the review, we have undertaken an Official Statistics appraisal of DLHE using the UK Statistics Authority’s toolkit (available here). The outcome of this was that the current DLHE survey does not meet the criteria for comprehensive assurance. Having explored various options around the design of the survey, we have concluded that recommending a form of open centralisation for the NewDLHE model will best achieve our goals.

Our work on the design phase has taught us that the term ‘centralisation’ is broad and can refer to many points on a spectrum between full-centralisation and full-devolution. In the sector at the moment, there are many approaches taken to survey processes, including the current DLHE, Longitudinal DLHE, DLHE at FE colleges, the Postgraduate Research and Postgraduate Taught Experience Surveys, the National Student Survey and the UK Engagement Survey.

Our proposed model of open centralisation can be demonstrated by comparing its positioning with existing surveys in the sector.

Open centralisation consists of the following:

• Data processing. The quality assurance and coding of the data will be carried out by the survey contractor. HE providers will have opportunity to comment if anomalies in the data are identified. The survey contractor will be required to demonstrate appropriate levels of validity, completeness and consistency in their coding, through the governance process.
• Data access. One of the key aspects which has been highlighted to us throughout the review is providers’ requirement to have near real-time access to the data, in order to allow careers services to support struggling graduates and to plan based on immediate data. Further detail about how this will work is available in the data supply model.
• Model governance. Providers will have a key role in the overall governance of the model. This includes the ability to add their own unique questions to the end of the survey, and to propose additional optional question banks. Further information is available in the governance model.
• Administration. Responsibility for carrying out the survey will lie with the survey contractor.

Contact methods

The main methodologies will be telephoning and online survey. Postal surveys will no longer be used, to align with changing survey practice. The survey design has been based on these methods, and therefore maximises opportunities for routing graduates and piping earlier answers in to maximise response rates. Third party telephone responses will still be permitted, subject to the same limitations as in DLHE.

Similarly, third party responses from HE providers will still be permitted at a limited level. HE providers will be able to feed these responses back to the survey contractor and will be required to provide evidence to the source of these responses. These will only be accepted as a response where the contractor has not managed to contact the graduate directly.

Coverage

NewDLHE will have the same population profile as current DLHE:

“The HESA Destinations of Leavers from Higher Education (DLHE) target population contains all students reported to HESA for the reporting period 01 August to 31 July as obtaining relevant higher education qualifications and whose study was full-time or part-time (including sandwich students and those writing-up theses). Awards from dormant status are only included in the target population for postgraduate research students. Relevant qualifications exclude intercalated degrees, awards to visiting students, students on post-registration health and social care courses, and professional qualifications for serving school teachers”.

The NewDLHE population will include both UK and non-UK domiciled students. Full details of the coverage and exclusions can be seen here.

Survey timing

Graduates will be surveyed at approximately 15 months after completing their studies. We will ensure we capture those graduates who are in further study by making the census week the first week of the month (e.g. graduates who are surveyed in the September – November period will be asked about the activities they were undertaking during the first week in September). We will ask for career history and further study history in order to retain some information on first destinations.

We will be explicit about which study period the survey relates to, by piping in information about the course a graduate studied into the survey questions. This will also enable a personalised approach. We are aware that this survey window could be problematic for PGCE students who may not have yet started a post and are in discussions with NCTL about our approach for these graduates.

For each academic year, there will be four survey points, in the first weeks of December, March, June and September. Estimations from students who fell into the DLHE population from the 2015/16 Student and AP Student data demonstrate that the September survey is likely to have the largest cohort, with 69% of graduates falling into this time period (see figure below). However, this trend may change over time, with changes in provision. This also only provides an average across the sector and will differ by provider.

Response rates

Response rates below 70% per provider will not be accepted, although our expectation is that higher rates will be reached (demonstrated by the Australian online-only survey attaining 39% at 12 months and rising online response rates for Longitudinal DLHE C12019). This level provides sufficient detail for onward use of the data at course level, as demonstrated by the National Student Survey (NSS). The survey contractor will be required to ensure the responses are representative across different courses, as HE providers are required to do by the current DLHE.

Achieving high response rates will require collaboration between HESA, HE providers and the survey contractor. HE providers will be responsible for maintaining channels of communication with their alumni to ensure that they have accurate contact details. There would also be a value in providers raising the prominence of the NewDLHE survey in any alumni contact undertaken prior to the survey period. We will make digital materials available to support this promotion.

We will work with the survey contractor to develop a robust communications plan to support the achievement of high response rates. This plan will include a range of central activities as well as the provision of resources to support local communications activity.

We will also maximise response rates through:

• Utilising best practice in survey design. This includes making questions personalised to the graduate depending on their activity and avoiding question types which commonly lead respondents to drop out, such as grid questions. The survey design will undergo cognitive testing to ensure the questions are clear to respondents.
• Utilising technological advances. This will include routing the survey to ensure graduates are only required to answer questions which are relevant to them, and piping data such as the course they have undertaken in order to provide a personal experience. The interviewers who conduct the survey over the telephone will be able to use this system in order that all respondents get the same experience.
• Personalisation of the survey and contact methods to each provider. Providers will have options to advise their relationship manager at the survey contractor of characteristics of their graduates which may influence the contact method. They will also be able to work with the third party supplier to personalise the contact details and survey visually to ensure graduates understand the links with the provider. This will build on examples of good practice in current outsourcing across the sector.
• Maximising the use of linked data to avoid asking questions which are seen as intrusive (such as salary) and to ensure the survey is kept brief.

Contact details

Providers will be responsible for maintaining contact details for their graduates, as now. HESA will provide a platform through the data collection system where providers can return and update their graduate’s contact details. Contact details will be quality checked to ensure levels and quality (e.g. institutional email addresses only, incomplete telephone numbers). Data validation quality rules will be handled through familiar, standard HESA processes.

Optional question banks

Two types of optional question bank will be permitted:

1. Question sets which have been through governance process and can be opted into. These will have been through the governance process and details of the optional question banks for the first NewDLHE survey are included in the survey design, such as those for research students, and alternative measures of graduate outcomes including subjective well-being and net promoter score. These questions can be asked at any point in the survey (e.g. overseas salary information sitting in the employment section) and will fall under our main collection notice. Optional banks can be made available online-only (as part of the core service level) and also through the telephone survey, in which case standard additional unit charges will apply.
2. Providers adding their own questions to the survey. These can be added at the end of the survey, and will be clearly divided from the core question set in order to retain response rates to the main survey and demonstrate that these are not subject to the same collection notice. The survey contractor will work with providers to ensure where similar questions are being asked by different providers, these are implemented consistently to minimise costs.

If providers wish to add questions to the question sets described in option 1 they will be able to propose them through the governance process.

External coding frames

NewDLHE will continue to use external coding frames. The use of these coding frames will be kept under review through the governance process.

Standard Occupational Classification (SOC)

The Standard Occupational Classification (SOC) coding frame will continue to be used to classify the occupations of graduates. The first year of NewDLHE will use the existing SOC(DLHE) coding frame. The Office for National Statistics (ONS) is currently running a review of the SOC coding frame for 2020. We hold a position on the review group and are representing the views on SOC shared throughout the NewDLHE review.

Once SOC2020 has been published, we will investigate whether this can be used as is for NewDLHE, or whether a fifth digit version is required to replace SOC(DLHE).

Standard Industrial Classification (SIC) and company information

The Standard Industrial Classification 2007 (SIC2007) coding frame will continue to be used to classify the industry the graduate is working in. The SIC coding process will provide some derived data about the company, such as company size. We will aim to derive if an organisation is part of the NHS, but if this is not feasible we will include the current NHSORG question in the survey.

JACS/HECoS

We will link NewDLHE survey data to the HESA Student record (see the linked data model) using subject classification frames. The JACS (Joint Academic Coding System) subject coding frame will be replaced by its successor, the Higher Education Classification of Subjects (HECoS), in the academic year 2019/20. This means that linked HESA Student further study data will contain JACS codes for graduates studying up until summer 2019, and so supplied data for the first year of NewDLHE will contain mainly JACS data. From the second full year of operation onwards, HECoS will be the main coding frame. You can find out more about HECoS here

Currency

The optional question bank for salary (for students overseas) will utilise the ISO4217 standard currency coding frame.

Methods of coding

Coding for both SOC and SIC will operate through:

1. Respondents enter free text responses to questions, for example job title. Their response is saved in the system.
2. Respondents are then presented with a drop down, for example of occupations, which match most closely the data they have entered. They will have the opportunity to select from this list, enabling some self coding.
3. Both the respondent’s free text response and their list selection will be returned to HESA. If a respondent’s job does not have a match, only the free text response will be returned.
4. The data will be continuously quality assured by the survey contractor who will provide reports for the steering group on this coding. Providers will also be able to monitor this coding through their dashboards.

Additional surveys

The NewDLHE survey will be a census of graduates at 15 months. However, there will be interest in additional information about some groups of graduates, for example understanding the early destinations of those who have undertaken teacher training, or understanding the longitudinal outcomes of research students. About a third of respondents to our previous consultation (and about a quarter of all HE providers) indicated these and similar interests. While we have not explicitly catered for their preferences, we have designed the proposal to be potentially extensible. The survey platform could be adapted to allow additional surveys to run. However, any request for additional surveys would be subject to the governance process and at additional cost.

Updates June 2017

We consulted on the model in March/April 2017, and published a synthesis of consultation responses. We have also published a number of responses and clarifications on points raised by respondents, including points on methodology, survey timing, response rates, contact details, coding and the relationship between the HE provider, the survey contractor and HESA.