Operational survey information
As the Graduate Outcomes survey is live, we’ve shared some useful information for providers which outlines how HESA is delivering the Graduate Outcomes survey.
The content will grow and change over time but to respond to provider queries, we will add to this information gradually, so it can be made available to you sooner. In addition, as we learn from each cohort, we may also change our approaches as outlined on these pages. We will endeavour to update this page as relevant.
How are IFF introducing themselves at the start of the interviews?
IFF interviewers introduce the Graduate Outcomes survey as the reason for the call and state they are calling on behalf of the provider for the particular graduate. If they are challenged further, they will explain that they are a research agency that have been appointed by HESA to carry out this work. If required, the interviewer can also advise that the survey has been commissioned by the UK higher education funding and regulatory bodies.
How do you ensure that where a graduate is in a different time zone, this is reflected in the time of call?
Where we have collected the data fields for international telephone numbers, the computer-assisted telephone interviewing (CATI) system automatically allocates the appropriate time zone to ensure the graduate is contacted at the correct time of day. The CATI system determines the location of the telephone number and assign it to the correct call queue. This is another reason for personal contact data needing to be as accurate as possible.
What does the telephone number come through as? Will it look like a spam call?
We have assigned a relevant geographical telephone number in the Confirmit system to reflect each institution’s geographical location.
What happens when a graduate answers, but is unable to take the call?
In these instances, where we know the contact details are correct but the timing of the call is inconvenient, the interviewer will ask for the graduate’s preferred date and time to undertake the survey. The interviewer will log these details into the CATI system and the graduate then will not be made available to be called until the date and time specified. The graduate is also given the option of completing the survey online.
How does the CATI system react to a graduate switching modes (phone to online and vice versa)?
If a graduate is called and states that they would prefer not to take the survey over the telephone, the IFF interviewer asks whether they would be willing to complete the survey online instead. If they consent, the interviewer checks the email address with the graduate and closes the interview.
On closing the interview, the graduate will be automatically sent an invite to participate in the online survey, via the email address checked on the call. Where a graduate has already started the survey online, they will be withheld from telephone interviewing until such a point where it is deemed unlikely that they will go on to complete the survey.
At this point, the graduate will be made available in the CATI system for telephone interviewing. An IFF interviewer will call the graduate, and upon introducing the survey and gaining the consent to participate, will start the survey at the point the graduate reached in the online survey.
What’s the approach to quality assurance?
All interviews are recorded digitally to keep an accurate record of interviews. A minimum of 5% of each interviewers’ calls are reviewed in full by one of IFF’s Team Leaders. Quality control reviews are all documented using a series of scores. Should an interviewer have below acceptable scores, this will be discussed with them along with the issue raised, an action plan agreed and signed, and their work further quality controlled.
Team Leaders rigorously check for tone/technique, data quality and conduct around data protection and information security. In addition, where Oblong are unable to code verbatim responses, these will be returned to IFF who will take steps to obtain and supply better quality verbatim by listening back to the interview and where necessary calling the graduate again.
How opt outs are handled?
When a graduate informs that they wish to opt-out on the call, then the IFF call handler will action this on the system so they will be set to not be contacted by any form of communication.
Where are IFF’s interviewers based?
IFF operates a CATI centre in their offices in London, which is supported by interviewers working remotely within the United Kingdom. These remote workers access the same system and interface as London operators and they are treated no differently to interviewers based in the London office. Each machine used by home workers is under the complete control of IFF’s IT department.
What’s the approach to training IFF’s interviewers?
Interviewers receive full training covering practical, theoretical and technical aspects of the job requirements. For quality control purposes, Team Leaders provide ongoing support throughout, harnessing interviewer skills and coaching around areas for improvement. This is done through top-up sessions, structured de-briefs and shorter knowledge sharing initiatives about “what works”.
For Graduate Outcomes, all interviewers receive a detailed briefing upon commencing interviewing, covering the purpose of the survey, data requirements (for example level of detail needed at SIC and SOC questions), running through each survey question, and pointing out areas of potential difficulty so objections and questions can be handled appropriately and sensitively.
This page was updated in November 2019 to include additional Q&As and a deadline for receipt of coding feedback for the 2017/18 collection. The deadline is Friday 6 December 2019.
Who is completing the data classification coding for Graduate Outcomes?
Oblong is our supplier for the coding of occupations and industries that graduates are working in (known as SIC and SOC coding). They are business data experts, with their main focus being the classification of businesses and database cleaning/enhancement. They have been SIC coding DLHE for the past 6 years, using specially developed coding software, in combination with a highly experienced manual research team.
The classification of Graduate Outcomes is key to allowing analysis and understanding of this large data source, and accuracy and consistency are paramount given the scrutiny and importance of the data. Learn more about Oblong.
What’s the approach to SIC coding?
Over the years, Oblong has developed self-learning software to deal with the classification of company data. This software has been finely tuned to work with HESA data. Graduate Outcomes will use their Business Data, Unity matching software suite and AutoSIC software to add industry classifications - SIC codes - to companies that employ graduates. The dedicated manual research team quality check most of the data and infill the gaps where the system can't add a SIC.
The fields Oblong will used for SIC coding are:
- Company Name
- Company Town/City
- Company Postcode
- Company Description
- Job title (to help with School/Healthcare classifications)
- Course title
- JACS level 3 grouping
- Level of qualification
What’s the approach to SOC coding?
As part of the Graduate Outcomes survey, Oblong has also been contracted to add occupation classifications - SOC codes - to summarise the type of job each graduate undertakes for the company they work for. It will use their new self-learning AutoSOC software to add classifications. This will be followed up by manual quality checks on most of the data and manual infill on those the system can't classify.
The fields Oblong will used for SOC coding are:
- Company Name
- SIC code
- Job title
- Job Duties Description
- Qualification Required?
They will also take into account if the company is an NHS organisation, if the graduate is self-employed, freelance, running their own business, supervising staff, or own the business.
The coding system uses various different methods to SOC code a record. It looks for keywords in the job title and job duties field, and takes into account if the qualification was required or not, before choosing the SOC. The system learns from data that has been previously coded (including manual SOC coded records), so if it sees a record with similar details to one that was seen before, it can be assigned the same SOC as last time. Oblong are manually reviewing all of the SOC codes the system produces, and then following this up with a second manual quality check and a final consistency check at the end of each cohort.
What is the ongoing data quality process for SIC / SOC data?
The automated coding system uses various methods to try and SOC code a record using different combinations of all the available data. The system learns from data it has previously coded (including manual SOC coded records). Therefore, when it encounters a record with similar details to one it has seen before it can be assigned the same SOC as the last time.
Currently, all SOC codes produced by the system are manually reviewed, and then followed up with a second manual quality check and a final consistency check at the end of each cohort.
A final data quality review will take place at the end of the collection. This will involve consistency checks across all cohorts to make sure no single cohort within the collection looks different to the rest. This is particularly important as most provider queries only started emerging towards the end of cohort C, by which point more than two cohorts had already been coded. The end of collection consistency checks will ensure that any amendments are applied across all four cohorts.
By the end of the process, every SOC and SIC code will have been manually checked at least twice – the thoroughness of the check on each record depends on the confidence the system has in the code allocated.
For telephone interviews, Oblong has an open channel with IFF Research (our contact centre supplier) to discuss data collection improvements to assist with collecting better data for more accurate coding. Of the responses collected through telephone interviewing, any uncodable records identified by Oblong are sent back to IFF for a follow-up interview where there is a reasonable case for going back to the respondents. Over the last few cohorts, the number of uncodable telephone responses has reduced.
What can providers report to HESA?
Continuous improvement is a key principle for Graduate Outcomes and as such, we want to continually build on and improve the coding of data. HESA would welcome feedback from providers where a provider believes that there is a widespread or systemic error in the coded data that they have received. However, HESA’s capacity to engage meaningfully in all coding queries is limited and we do not have capacity to check all individual record queries raised by providers, especially where the overall impact is low.
What is a systemic error?
The most common example of systemic error is the incorrect coding of an entire professional group. At times, providers may not identify this using data from just one cohort. This may require a review of data across multiple cohorts.
In addition, it’s important to note that any form of coding is impacted by subjective judgment. The reason for undertaking consistency checks is to reduce this impact and make the results as objective as possible. To this effect, we advise that any judgment about systemic errors should entirely be based on data provided by the respondent. Subjective opinion and/or contextual, extraneous information will not be considered when reviewing the case for systemic issues.
How can providers locate systemic errors?
Over the past few months, we have received feedback from providers that the process of pinpointing systemic errors is challenging without having access to a wider dataset.
We know that some sector groups such as AGCAS have created a valuable community approach to this exercise. If you are a member of such a group, you might find this a useful community to help develop your approaches to Graduate Outcomes.
Whilst we welcome provider collaboration in finding the best ways to locate systemic errors, we must remind providers to adhere to the data usage sign off form. This prohibits providers from sharing actual survey results data in order to do this.
How do providers report a systemic issue to HESA?
To report an issue that a provider deems to be an error in the system, not an individual case, please email Liaison with the a few examples relating to the professions in question. We do not require student identifiers as correcting individual records is not in scope.
When is the deadline for reporting errors to HESA?
To ensure we can meet our strict timeline for the dissemination of 17/18 Graduate Outcomes data, we must place a deadline for providers to submit systemic errors to HESA. The deadline is Friday 6 December 2019. Following this deadline, HESA and Oblong will carry out their own strict quality assurance processes on this data.
Has HESA taken action following any provider reports of systemic errors?
Yes. We have received feedback from a sector group and a small number of providers which has resulted in several coding changes. A few examples of these are auditors, civil servants and library staff. This has enabled HESA and Oblong to improve the data classification process for 17/18 and future collections.
To aid providers’ communication with graduates, we’ve outlined the general engagement plan below that will be implemented for each cohort. See more information on survey timings (course end dates, contact periods and census weeks).
Our full engagement strategy is an internal only document and incorporates an intricate plan which includes a range of methods, timings and scenarios. This has been crafted using best practice data collection and research methods. We have also worked with experts from the ONS, Confirmit (and liaised with the Graduate Outcomes steering group) to create a robust strategy which carefully balances the need to gain responses using a blend of methods across all of the contact details we hold for each graduate.
The strategy will be reviewed over time and we will make changes and refinements where needed.
|The primary engagement method will be email - we will also send email reminders at regular intervals.||Graduates will receive text messages (SMS) from ‘GradOutcome’ which also contain links to the online survey (where we have mobile numbers).||Graduates will also receive phone calls from IFF Research (our contact centre) on behalf of providers.|
The engagement plan for cohort D can be found below. See more on response rates.
|Survey fieldwork starts:
|Week 3/4||Email and SMS reminders sent to non-respondents|
|Week 5||Email and SMS reminders sent to partial-respondents|
|Week 6/7||Email and SMS reminders sent to non-respondents|
|Week 8/9||Email and SMS reminders sent to partial-respondents|
|Week 10||Email and SMS reminders sent to non-respondents|
|Week 11||Email and SMS reminders sent to partial-respondents|
|Week 12/13||Final email and SMS reminders|
|Week 13||Survey fieldwork ends|
You can view the sample email and SMS message being sent to graduates in survey materials below.
Encouraging response rates
Administration of invitations and reminders is carefully managed and considers timing, frequency, volume and journey of respondents. Every successful cycle of reminders informs the delivery of future reminders. All graduates across the entire sector are treated equally in terms of the communications they receive from us.
Partials to complete
Encouraging the graduates who have started the survey to finish it is a key part of our engagement strategy. To share more about our approach, Neha (our Head of Research and Insight) has shared some key aspects of our engagement strategy.
Focusing on the mandatory questions
The majority of the mandatory Graduate Outcomes questions are the same that were required for a valid response in DLHE. Maintaining a set of mandatory questions ensures we have the correct routing in place and the required data for SIC/SOC coding purposes.
To ensure we receive maximum response rates, we have taken a number of steps to ensure each graduate is encouraged to complete these mandatory questions as a priority:
- non-mandatory questions are optional and can be skipped
- the subjective wellbeing and graduate voice questions are placed at the end of the survey, after the mandatory questions have been completed
- opt-in question banks are also placed at the end of the survey, after the mandatory questions have been completed.
How are contact details prioritised for surveying?
For C18071, we’ve provided some guidance in relation to the number of contact details required by provider and how they are returned to us. This is all detailed in the C18071 coding manual. We've summarised it here:
EMAIL: We send the email invitations and reminders to EMAIL01 first, followed by EMAIL02-10. It is recommended that the ‘best’ email address for a graduate is returned in the first position either in the webform or in the XML file.
UKMOB / UKTEL / INTTEL: It is recommended that the ‘best’ telephone numbers for a graduate are returned in the first position for UKMOB / UKTEL / INTTEL either in the webform or in the XML file. Graduates will be called on their first telephone number before subsequent numbers are tried, and once successful contact is made, this contact detail is prioritised for any future calls.
Mobile numbers are more likely to result in successful contact and therefore, UKMOB (mobile) numbers are called before UKTEL (landlines), followed by INTTEL (international) numbers, where applicable.
Can you explain how to determine the ‘best’ contact details?
What we mean by ‘best’ is the supply of contact details that are most likely to elicit a response to the survey. This can be determined by recent contact with the graduate via this contact detail.
Some providers will be using email clients (e.g. mailchimp, raisers edge) to despatch their communications to graduates which often provide rich insight into the behaviour of the email recipient. This includes where the recipient has opened the email, clicked on a link or replied to it. This provides the evidence required to determine that the graduate is actively using this contact detail and is therefore useful in terms of survey engagement. If the provider is using these systems to carry out the suggested graduate contact during the 15 months post-course completion period, this provides a regular feed of information about this contact detail to help make this determination.
Where providers do not have access to this information, and multiple email addresses for a graduate are known, then provided these remain accurate for the graduate, it is recommended these are all returned to give HESA additional opportunities to contact the graduate.
Can you share the rationale for the pre-notification strategy for cohort D?
We are implementing a new system called MailJet to help us with the issues related to internet service providers (e.g. Gmail, Office 365) blocking our email invitations. We have also been working to trial new activity that aims to improve our response rates and a pre-notification email (a warm up) is one strategy that we have been considering.
From 14 August to 30 August, we will stagger the delivery of a pre-notification email to approved cohort D graduates that shares key information about the survey and lets them know they’ll receive their unique survey link in early September. Read more in the email issued to providers on 8 August 2019.
Can you provide more detail about the subjective wellbeing survey questions, including how you ask graduates and how the data will be used?
You can read more about this in our blog - 'Asking graduates how they feel' by Neha Agarwal, Head of Research & Insight.
Can you share more detail about what each of the statuses in the provider portal progress bar mean?
The provider portal user guide explains what each status means and who's included within each one.
Are providers allowed to contact graduates within a cohort (once it has opened for surveying)? (Updated October 2019)
We have previously supplied guidance that once a cohort has commenced surveying, providers should not be making direct contact with graduates for risk of over-communication, creation of potential bias and the fact that we cannot report on those who’ve chosen to opt out (providers should be respecting their wishes).
We have been exploring this in more detail within our response rate strategy work and have determined that there is considerable risk of bias and more importantly, HESA will not be able to measure and/or control it. This outweighs the possible benefits of getting some engagement from a hard to reach group. Therefore, the guidance above remains our position and we will not be pursuing the reporting of opt-outs.
What providers should not do:
- Under no circumstances should providers make direct contact (for example by email or phone) with graduates currently being surveyed.
What providers can do:
- Focus on brand recognition communications using non-direct channels such as provider websites and social media platforms including Linkedin and Twitter. We have provided a suite of communications materials for this purpose.
- If you’re sending a regular email campaign with the primary purpose of sharing a general update, for example your Alumni newsletter sent to a mass audience, you could include a small feature about Graduate Outcomes. Just make sure the content included about Graduate Outcomes is low in prominence – it should not be more than 50% focused on Graduate Outcomes.
Providers play a vital part in the success of Graduate Outcomes. It’s important that providers are engaging with the cohort population directly in the build up to the contact period commencing as well as other key points in the student to graduate lifecycle. So, whilst contact mid-cohort is not advised, there are plenty of other opportunities to build awareness of the survey. Find out more.
Can you share survey paradata or response rates with providers?
In the creation of a centralised survey, our strategy does not include individualised reporting of this kind, so we cannot share survey paradata (data about the process by which the data was collected) at a provider level. This means that our approaches to making any developments to the survey design or engagement strategy will be carried out across the entire survey by reviewing the entire data set.
We will also not be looking to share any overall sector statistics as any release would require considerable contextual information and we are unable to prioritise the required activity at this time. We want providers to rest assured that this analysis and activity is taking place but in a measured and controlled manner. HESA is approaching this first year of Graduate Outcomes in an agile way and taking every step to ensure within the active survey management we are reacting to all of the information available. At the same time, we need to ensure the integrity of the live survey.
How are you encouraging graduates who've started the survey but not yet completed to finish the survey?
This is a central part of our engagement strategy. Broadly speaking, graduates in the ‘started survey’ group will receive regular email and SMS reminders encouraging them to finish the survey. In addition, they will be allowed appropriate time to complete it online before they are followed up via telephone. Read more about how we aim to turn partials into completes.
Can you share your stance on provider incentives?
We have taken feedback from the sector and assessed the internal requirements to implement incentives versus the potential impact on response rates. Due to this, and the other initiatives we have deemed higher priority, work to roll out incentives to increase online responses has been deprioritised. We will review this decision in the next collection (18/19).
Despite this, we strongly recommend that providers do not create their own incentives. Depending on the type of population breakdowns a provider may have, incentives could bias the results in favour of that population group. For a centrally run survey like Graduate Outcomes, incentives must be rolled out across the entire population so that all respondents are treated equally.
How many times will a graduate be contacted to complete the survey?
It is impossible to provide a simple answer to this question as it depends on what type of engagement we have with each graduate across the cohort and across modes. If you review the outline engagement plan, it shows the proposed contacts we will make with the online survey (via email and SMS message) and calls are carefully scheduled to complement this. Before a call is made, we will ensure that where a graduate has already started the survey online, that they have had time to complete it on this mode. It then depends how successful the contact details are and whether any contact is made using them (i.e. do they pick up).
You can view our engagement statistics e.g. average number of emails / text messages sent, within the end of cohort reviews on the supplied infographic.
In discussion with the Graduate Outcomes steering group, HESA has set the following response rate targets for Graduate Outcomes:
|Target group||Response rate target|
|UK domiciled full-time||60%|
|UK domiciled part-time||60%|
The above targets are applicable at a national, provider, undergraduate and postgraduate level. In addition to these, we will also monitor response rates for a range of socio-demographic and course characteristics such as age, gender, ethnicity and course.
We understand the importance of good quality outputs for the sector as well as other users of our data. We will continuously monitor the progress of Graduate Outcomes against the targets set out above. We will employ robust research and statistical methodologies to ensure we are able to produce estimates that meet the required statistical quality standard as well as our users’ requirements.
To enable providers to support the survey, final versions of the survey materials and communications with graduates can be viewed below.
There are a number of emails in the engagement strategy:
A pre-notification (warm up) email is being trialled for cohort D in our first collection. This will be sent to all graduates prior to the start of the contact period to provide key information about the survey and lets them know that they’ll receive their unique survey link in early September.
The email invitations sent within the contact period are derived from the core email text which is shared below. This has been updated for cohort D (17/18) and includes some look and feel changes. The other iterations vary depending on the nature of the email and its role in the engagement strategy. Changes are mainly limited to the subject line and first paragraph.
SMS / text message
There is one version of the SMS text which is generic so it can be used across the engagement plan.
You can view the survey questions on the page below split by type. The data items and routing diagram can be found in the Graduate Outcomes Survey Results coding manual.