Operational survey information
As the Graduate Outcomes survey is live, we’ve shared some useful information for providers which outlines how HESA is delivering the Graduate Outcomes survey.
The content will grow and change over time but to respond to provider queries, we will add to this information gradually, so it can be made available to you sooner. In addition, as we learn from each cohort, we may also change our approaches as outlined on these pages. We will endeavour to update this page as relevant.
How are IFF introducing themselves at the start of the interviews?
IFF interviewers introduce the Graduate Outcomes survey as the reason for the call and state they are calling on behalf of the provider for the particular graduate. If they are challenged further, they will explain that they are a research agency that have been appointed by HESA to carry out this work. If required, the interviewer can also advise that the survey has been commissioned by the UK higher education funding and regulatory bodies.
How do you ensure that where a graduate is in a different time zone, this is reflected in the time of call?
Where we have collected the data fields for international telephone numbers, the computer-assisted telephone interviewing (CATI) system automatically allocates the appropriate time zone to ensure the graduate is contacted at the correct time of day. The CATI system determines the location of the telephone number and assign it to the correct call queue. This is another reason for personal contact data needing to be as accurate as possible.
What does the telephone number come through as? Will it look like a spam call?
The contact centre operates using a geo-dialling system, whereby the geographical location of providers is taken into consideration. This use of familiar area codes serves to mitigate concerns about spam calling. This approach is supported more generally by existing best practice within the market research sector.
Despite the benefits of a geo-dialling system, the use of phone numbers that are visible but unknown to respondents does increase the likelihood that they will repeatedly ignore or even bar the calls, especially where they are called multiple times from the same number. Therefore, during the second half of year one (17/18), the approach was further enhanced by changing the telephone numbers used for some of the fieldwork period. This approach is being replicated in year two (18/19).
What happens when a graduate answers, but is unable to take the call?
In these instances, where we know the contact details are correct but the timing of the call is inconvenient, the interviewer will ask for the graduate’s preferred date and time to undertake the survey. The interviewer will log these details into the CATI system and the graduate then will not be made available to be called until the date and time specified. The graduate is also given the option of completing the survey online.
How does the CATI system react to a graduate switching modes (phone to online and vice versa)?
If a graduate is called and states that they would prefer not to take the survey over the telephone, the IFF interviewer asks whether they would be willing to complete the survey online instead. If they consent, the interviewer checks the email address with the graduate and closes the interview.
On closing the interview, the graduate will be automatically sent an invite to participate in the online survey, via the email address checked on the call. Where a graduate has already started the survey online, they will be withheld from telephone interviewing until such a point where it is deemed unlikely that they will go on to complete the survey.
At this point, the graduate will be made available in the CATI system for telephone interviewing. An IFF interviewer will call the graduate, and upon introducing the survey and gaining the consent to participate, will start the survey at the point the graduate reached in the online survey.
What’s the approach to quality assurance?
All interviews are recorded digitally to keep an accurate record of interviews. A minimum of 5% of each interviewers’ calls are reviewed in full by one of IFF’s Team Leaders. Quality control reviews are all documented using a series of scores. Should an interviewer have below acceptable scores, this will be discussed with them along with the issue raised, an action plan agreed and signed, and their work further quality controlled.
Team Leaders rigorously check for tone/technique, data quality and conduct around data protection and information security. In addition, where Oblong are unable to code verbatim responses, these will be returned to IFF who will take steps to obtain and supply better quality verbatim by listening back to the interview and where necessary calling the graduate again.
How opt outs are handled?
When a graduate informs that they wish to opt-out on the call, then the IFF call handler will action this on the system so they will be set to not be contacted by any form of communication.
Where are IFF’s interviewers based?
IFF operates a CATI centre in their offices in London, which is supported by interviewers working remotely within the United Kingdom. These remote workers access the same system and interface as London operators and they are treated no differently to interviewers based in the London office. Each machine used by home workers is under the complete control of IFF’s IT department.
What’s the approach to training IFF’s interviewers?
Interviewers receive full training covering practical, theoretical and technical aspects of the job requirements. For quality control purposes, Team Leaders provide ongoing support throughout, harnessing interviewer skills and coaching around areas for improvement. This is done through top-up sessions, structured de-briefs and shorter knowledge sharing initiatives about “what works”.
For Graduate Outcomes, all interviewers receive a detailed briefing upon commencing interviewing, covering the purpose of the survey, data requirements (for example level of detail needed at SIC and SOC questions), running through each survey question, and pointing out areas of potential difficulty so objections and questions can be handled appropriately and sensitively.
18/19 SOC coding update
HESA has been exploring the potential use of the newly released SOC2020 coding framework and, working with the Steering Group, has taken the decision to use this framework for the coding of year two (18/19) survey data. To enable this, we are currently undertaking the planning and implementation activity required to make this change. This means that we have delayed the provision of the raw feed of SIC / SOC data until this work is complete.
We’re keen to share the lessons learned from year one and how this has helped to inform what will be our finalised approach to the coding of year two data. In the coming months, we will provide a full overview, to include our approach to quality assurance and how we intend to engage with the sector.
We would like to thank providers for their patience while we develop this approach and to those who’ve engaged with HESA regarding occupation coding and other aspects of the survey during this unprecedented time for the sector.
Outcomes of 17/18 SOC coding assessment
We would like to express our sincere thanks to all providers for their cooperation and patience during the assessment process. We have now provided an overview of the process, the action we’ve taken, and our next steps - this has been sent to all providers for their information. Please click the button below to view this:
Which occupation groups have been identified in the 17/18 data as having systemic errors?
During the process outlined above, the following occupation groups have been identified as having systemic issues. These groups have been addressed in the data delivery to providers in late March 2020.
Secretary in educational establishment
Investment banking associates
Research administrator (higher education)
IT and software engineer
Administrators at HE providers
Merger and acquisition analyst
Support worker (housing)
When can providers expect delivery of additional SOC codes for 17/18?
Following the delivery of final provider survey data in late March, we received questions regarding a number of SOC records with a code of “00010 - insufficient information to code”. HESA’s coding methodology is underpinned by the need for data quality. As such, we aim to collect the requisite amount of data (all four fields outlined below) and this data must be of sufficient quality to allow correct coding, as determined by our coding partner. A large percentage of the records coded 00010 did not meet these criteria and were therefore not sent for coding, as per our methodology.
Following an investigation, we delivered a new iteration of the collection results data into the provider portal on 22 May. This delivery included additional SOC coding of a subset of partial responses where we are satisfied that accurate coding can be achieved. This is, for example, where responses of sufficient quality have been provided in job title and job duties, even if the employer’s name and/or duties are missing.
Further information on coding processes
Who is completing the data classification coding for Graduate Outcomes?
Oblong is our supplier for the coding of occupations and industries that graduates are working in (known as SIC and SOC coding). They are business data experts, with their main focus being the classification of businesses and database cleaning/enhancement. They have been SIC coding DLHE for the past 6 years, using specially developed coding software, in combination with a highly experienced manual research team.
The classification of Graduate Outcomes is key to allowing analysis and understanding of this large data source, and accuracy and consistency are paramount given the scrutiny and importance of the data. Learn more about Oblong.
What’s the approach to SIC coding?
Over the years, Oblong has developed self-learning software to deal with the classification of company data. This software has been finely tuned to work with HESA data. Graduate Outcomes will use their Business Data, Unity matching software suite and AutoSIC software to add industry classifications - SIC codes - to companies that employ graduates. The dedicated manual research team quality check most of the data and infill the gaps where the system can't add a SIC.
The fields Oblong use for SIC coding are:
- Company Name
- Company Town/City
- Company Postcode
- Company Description
- Job title (to help with School/Healthcare classifications)
- Course title
- JACS level 3 grouping
- Level of qualification
What’s the approach to SOC coding?
As part of the Graduate Outcomes survey, Oblong has also been contracted to add occupation classifications - SOC codes - to summarise the type of job each graduate undertakes for the company they work for. It will use their new self-learning AutoSOC software to add classifications. This will be followed up by manual quality checks on most of the data and manual infill on those the system can't classify.
The fields Oblong use for SOC coding are:
- Company Name
- SIC code
- Job title
- Job Duties Description
They will also take into account if the company is an NHS organisation, if the graduate is self-employed, freelance, running their own business, supervising staff, or own the business.
The coding system uses various different methods to SOC code a record. It looks for keywords in the job title and job duties field, and takes into account if the qualification was required or not, before choosing the SOC. The system learns from data that has been previously coded (including manual SOC coded records), so if it sees a record with similar details to one that was seen before, it can be assigned the same SOC as last time. Oblong are manually reviewing all of the SOC codes the system produces, and then following this up with a second manual quality check and a final consistency check at the end of each cohort.
What is the ongoing data quality process for SIC / SOC data?
The automated coding system uses various methods to try and SOC code a record using different combinations of all the available data. The system learns from data it has previously coded (including manual SOC coded records). Therefore, when it encounters a record with similar details to one it has seen before it can be assigned the same SOC as the last time.
Currently, all SOC codes produced by the system are manually reviewed, and then followed up with a second manual quality check and a final consistency check at the end of each cohort.
A final data quality review will take place at the end of the collection. This will involve consistency checks across all cohorts to make sure no single cohort within the collection looks different to the rest. This is particularly important as most provider queries only started emerging towards the end of cohort C, by which point more than two cohorts had already been coded. The end of collection consistency checks will ensure that any amendments are applied across all four cohorts.
By the end of the process, every SOC and SIC code will have been manually checked at least twice – the thoroughness of the check on each record depends on the confidence the system has in the code allocated.
For telephone interviews, Oblong has an open channel with IFF Research (our contact centre supplier) to discuss data collection improvements to assist with collecting better data for more accurate coding. Of the responses collected through telephone interviewing, any uncodable records identified by Oblong are sent back to IFF for a follow-up interview where there is a reasonable case for going back to the respondents. Over the last few cohorts, the number of uncodable telephone responses has reduced.
What is a systemic error?
The most common example of systemic error is the incorrect coding of an entire professional group. At times, providers may not identify this using data from just one cohort. This may require a review of data across multiple cohorts.
In addition, it’s important to note that any form of coding is impacted by subjective judgment. The reason for undertaking consistency checks is to reduce this impact and make the results as objective as possible. To this effect, we advise that any judgment about systemic errors should entirely be based on data provided by the respondent. Subjective opinion and/or contextual, extraneous information will not be considered when reviewing the case for systemic issues.
To aid providers’ communication with graduates, we’ve outlined the general engagement plan below that will be implemented for each cohort. See more information on survey timings (course end dates, contact periods and census weeks).
Our full engagement strategy is an internal only document and incorporates an intricate plan which includes a range of methods, timings and scenarios. This has been crafted using best practice data collection and research methods. We have also worked with experts from the ONS, Confirmit (and liaised with the Graduate Outcomes steering group) to create a robust strategy which carefully balances the need to gain responses using a blend of methods across all of the contact details we hold for each graduate.
The strategy will be reviewed over time and we will make changes and refinements where needed.
|The primary engagement method will be email - we will also send email reminders at regular intervals.||Graduates will receive text messages (SMS) from ‘GradOutcome’ which also contain links to the online survey (where we have mobile numbers).||Graduates will also receive phone calls from IFF Research (our contact centre) on behalf of providers.|
The engagement plan for the fieldwork period can be found below. See more on response rates.
|Week 0||Pre-notification emails sent to all graduates with approved contact details|
|Survey fieldwork starts:
You can view the sample email and SMS messages being sent to graduates in survey materials below.
Encouraging response rates
Administration of invitations and reminders is carefully managed and considers timing, frequency, volume and journey of respondents. Every successful cycle of reminders informs the delivery of future reminders. All graduates across the entire sector are treated equally in terms of the communications they receive from us.
Partials to complete
Encouraging the graduates who have started the survey to finish it is a key part of our engagement strategy. To share more about our approach, Neha (our Head of Research and Insight) has shared some key aspects of our engagement strategy.
Focusing on the mandatory questions
The majority of the mandatory Graduate Outcomes questions are the same that were required for a valid response in DLHE. Maintaining a set of mandatory questions ensures we have the correct routing in place and the required data for SIC/SOC coding purposes.
To ensure we receive maximum response rates, we have taken a number of steps to ensure each graduate is encouraged to complete these mandatory questions as a priority:
- non-mandatory questions are optional and can be skipped
- the subjective wellbeing and graduate voice questions are placed at the end of the survey, after the mandatory questions have been completed
- opt-in question banks are also placed at the end of the survey, after the mandatory questions have been completed.
How are contact details prioritised for surveying?
For C18071, we’ve provided some guidance in relation to the number of contact details required by provider and how they are returned to us. This is all detailed in the C18071 coding manual. We've summarised it here:
EMAIL: We send the email invitations and reminders to EMAIL01 first, followed by EMAIL02-10. It is recommended that the ‘best’ email addresses are returned for a graduate.
UKMOB / UKTEL / INTTEL: It is recommended that the ‘best’ telephone numbers for a graduate are returned for UKMOB / UKTEL / INTTEL. Graduates will be called on their first telephone number before subsequent numbers are tried, and once successful contact is made, this contact detail is prioritised for any future calls.
Mobile numbers are more likely to result in successful contact and therefore, UKMOB (mobile) numbers are called before UKTEL (landlines), followed by INTTEL (international) numbers, where applicable.
Can you explain how to determine the ‘best’ contact details?
What we mean by ‘best’ is the supply of contact details that are most likely to elicit a response to the survey. This can be determined by recent contact with the graduate via this contact detail.
Some providers will be using email clients (e.g. mailchimp, raisers edge) to despatch their communications to graduates which often provide rich insight into the behaviour of the email recipient. This includes where the recipient has opened the email, clicked on a link or replied to it. This provides the evidence required to determine that the graduate is actively using this contact detail and is therefore useful in terms of survey engagement. If the provider is using these systems to carry out the suggested graduate contact during the 15 months post-course completion period, this provides a regular feed of information about this contact detail to help make this determination.
Where providers do not have access to this information, and multiple email addresses for a graduate are known, then provided these remain accurate for the graduate, it is recommended these are all returned to give HESA additional opportunities to contact the graduate.
Can you share the rationale for the pre-notification strategy?
We have implemented a new system called MailJet to help us with the issues related to internet service providers (e.g. Gmail, Office 365) blocking our email invitations. We have also been working to trial new activity that aims to improve our response rates and a pre-notification email (a warm up) is one strategy that we have been considering.
From 14 August to 30 August, we staggered the delivery of a pre-notification email to approved cohort D (17/18) graduates that shares key information about the survey and lets them know they’ll receive their unique survey link in early September. Read more in the email issued to providers on 8 August 2019.
As we saw a positive return on email invitations in cohort D, we've determined that we will continue to use MailJet in Year 2 (18/19) and we will also continue with the pre-notification strategy for future cohorts.
Can you provide more detail about the subjective wellbeing survey questions, including how you ask graduates and how the data will be used?
You can read more about this in our blog - 'Asking graduates how they feel' by Neha Agarwal, Head of Research & Insight.
Can you share more detail about what each of the statuses in the provider portal progress bar mean?
The provider portal user guide explains what each status means and who's included within each one.
Are providers allowed to contact graduates within a cohort (once it has opened for surveying)? (Updated October 2019)
We have previously supplied guidance that once a cohort has commenced surveying, providers should not be making direct contact with graduates for risk of over-communication, creation of potential bias and the fact that we cannot report on those who’ve chosen to opt out (providers should be respecting their wishes).
We have been exploring this in more detail within our response rate strategy work and have determined that there is considerable risk of bias and more importantly, HESA will not be able to measure and/or control it. This outweighs the possible benefits of getting some engagement from a hard to reach group. Therefore, the guidance above remains our position and we will not be pursuing the reporting of opt-outs.
What providers should not do:
- Under no circumstances should providers make direct contact (for example by email or phone) with graduates currently being surveyed.
What providers can do:
- Focus on brand recognition communications using non-direct channels such as provider websites and social media platforms including Linkedin and Twitter. We have provided a suite of communications materials for this purpose.
- If you’re sending a regular email campaign with the primary purpose of sharing a general update, for example your Alumni newsletter sent to a mass audience, you could include a small feature about Graduate Outcomes. Just make sure the content included about Graduate Outcomes is low in prominence – it should not be more than 50% focused on Graduate Outcomes.
Providers play a vital part in the success of Graduate Outcomes. It’s important that providers are engaging with the cohort population directly in the build up to the contact period commencing as well as other key points in the student to graduate lifecycle. So, whilst contact mid-cohort is not advised, there are plenty of other opportunities to build awareness of the survey. Find out more.
Can you share survey paradata or response rates with providers?
In the creation of a centralised survey, our strategy does not include individualised reporting of this kind, so we cannot share survey paradata (data about the process by which the data was collected) at a provider level. This means that our approaches to making any developments to the survey design or engagement strategy will be carried out across the entire survey by reviewing the entire data set.
We will also not be looking to share any overall sector statistics as any release would require considerable contextual information and we are unable to prioritise the required activity at this time. We want providers to rest assured that this analysis and activity is taking place but in a measured and controlled manner. HESA is approaching this first year of Graduate Outcomes in an agile way and taking every step to ensure within the active survey management we are reacting to all of the information available. At the same time, we need to ensure the integrity of the live survey.
How are you encouraging graduates who've started the survey but not yet completed to finish the survey?
This is a central part of our engagement strategy. Broadly speaking, graduates in the ‘started survey’ group will receive regular email and SMS reminders encouraging them to finish the survey. In addition, they will be allowed appropriate time to complete it online before they are followed up via telephone. Read more about how we aim to turn partials into completes.
How many times will a graduate be contacted to complete the survey?
It is impossible to provide a simple answer to this question as it depends on what type of engagement we have with each graduate across the cohort and across modes. If you review the outline engagement plan, it shows the proposed contacts we will make with the online survey (via email and SMS message) and calls are carefully scheduled to complement this. Before a call is made, we will ensure that where a graduate has already started the survey online, that they have had time to complete it on this mode. It then depends how successful the contact details are and whether any contact is made using them (i.e. do they pick up).
You can view our engagement statistics e.g. average number of emails / text messages sent, within the end of cohort reviews on the supplied infographic.
Can you share your stance on provider incentives?
We have taken feedback from the sector and assessed the internal requirements to implement incentives versus the potential impact on response rates. Due to this, and the other initiatives we have deemed higher priority, work to roll out incentives to increase online responses has been deprioritised. We will review this decision in the next collection (18/19).
Despite this, we strongly recommend that providers do not create their own incentives. Depending on the type of population breakdowns a provider may have, incentives could bias the results in favour of that population group. For a centrally run survey like Graduate Outcomes, incentives must be rolled out across the entire population so that all respondents are treated equally.
When will you permit providers to include their own questions in the survey?
In June 2018, we took the decision to postpone the inclusion of the provider questions in the survey due to the legal, financial and operational complexities which surround these and to allow providers to manage the potential demand for these. Many of these complexities still exist and, above and beyond normal operational activity, we are currently focusing internal resource on improving email invitation deliverability, continuous survey improvement and strategies to boost online response rates. We have therefore decided to postpone provider questions for another collection. We will review this decision again in the next collection (18/19).
View the FAQs provided to support providers with the delivery of their 17/18 Graduate Outcomes data:
We have prepared some frequently asked questions about response rates. We will add to these over the coming months. View the response rate targets.
Why has HESA not hit all of the response rate targets?
To reflect on this first year of surveying we must look at the landscape in which this collection existed. Not only have we been impacted by known issues such as the unknown brand; a culture of data sensitivity with the introduction of GDPR (making the collection and use of personal data more challenging than ever before); we have also increased the gap between graduation and data collection to 15 months (impacting quality of contact details). In addition, people are generally over-surveyed these days making it harder to tackle survey-fatigue.
The quality of contact details prevented us from making contact with some graduates and, even where we have high quality details, email providers such as Microsoft and Gmail have developed sophisticated ways to block email delivery to protect their users from unwanted or unexpected emails. The survey suffered from the latter of these issues in the first half of the year, but things improved significantly during the second half. If the technological improvements we implemented later were available from the start, we would have been able to close the gap between the response rate and the targets even further.
It is also worth noting that prior to starting Graduate Outcomes we did not have a baseline to set realistic targets for this brand new survey. The assumptions around online completion rate and impact of the shift from a provider-led to a centralised survey with a 15-month gap were not based on a prototype. Outputs form the first year of Graduate Outcomes should inform the discussion on targets moving forward.
While response rates are an indicator of data quality, they are not the only indicator. Having a responding sample that is representative of the population in terms of characteristics is more important. This has been our main objective in Graduate Outcomes – a high response rate from a representative sample.
What does the lower response rate mean for data quality of outputs?
Response rates are one of many indicators of data quality. They are not necessarily predictive of non-response bias. That is to say that even with low response rates, survey data may be representative of the population. The impact of non-response is what we aim to alleviate as far as possible through case prioritisation and weighting of the responses we achieve, so that outputs can be more representative of the population.
How will HESA be using weightings in the final data?
We’re still developing our methodology but our intention is to explore use of weighted data as opposed to raw data for all of HESA’s Graduate Outcomes data outputs. The purpose of using weighted data is to present information that is as representative as possible of the entire population of graduates without bias. This is achieved by statistically adjusting the survey results to bring them in line with known information about the population.
Are you planning to make any changes to the targets for the next collection?
The first year of Graduate Outcomes has provided a helpful baseline for the survey. Given some of the data collection issues we experienced early on, we hope to iron them out in the second year and get a revised baseline. We may revisit the targets at that stage and discuss them with our Steering Group.
My provider’s response rates do not reflect the overall response rates for the collection – how will this impact my final data?
The quality of the data at provider level cannot be determined by comparing the provider response rate with the overall response rate. Even with low response rates, survey data may be representative of the population.
All graduates at all providers have received the same level of engagement. There can be multiple reasons why a provider’s response rates are lower than others in the sector. Some of the factors responsible for high or low response rates are quality of contact details, prior awareness of Graduate Outcomes among graduates and a general preference towards participating in surveys.
How many times did you contact my graduates? How do I know you contacted them enough?
On average, it took 5 calls to achieve a completed survey response. The number of calls made to non-respondents were a lot higher. More detailed statistics are made available in the end of cohort reports.
The amount of contact depends on what type of engagement we have with each graduate across the cohort and across modes. The outline engagement plan shows the proposed contacts we will make via email and SMS message and calls are carefully scheduled to complement this. Before a call is made, we will ensure that where a graduate has already started the survey online, that they have had time to complete it on this mode. It then depends how successful the contact details are and whether any contact is made using them (i.e. do they pick up). Successful contact with graduates heavily relies on the quality of contact details. Invalid and inactive email addresses and phone numbers will inevitably result in no contact.
Did you stop making contact with the target groups once you hit the target?
No, we carried on contacting people even if the main targets had been achieved. This is because we aim to get a representative sample of respondents whose demographic characteristics are similar to the population. As these characteristics are not mutually exclusive, we cannot make survey decisions based on just one target group.
Did HESA take valid answers from third parties (i.e. the graduate's parents, spouse, etc) as part of the approach?
Yes, data from a third party were collected for all providers except English FECs. This was introduced in the later stages of a cohort to allow sufficient amount of contact with the primary respondents; as data from third party is likely to be of a lower quality. The volume of third-party data has been limited to no more than 10% of a provider’s total responses (if they had at least 10 graduates in the population; if not, the provider was excluded from third party data collection). Subjective questions such as salary, graduate voice and wellbeing are excluded from interviews with third parties.
Are you including partial responses in the overall response rate?
Partial responses (that did not meet the minimum criteria but contain sufficient information required for key statistical outputs) will be included in the final data deliveries to providers and Statutory Customers and published outputs. They are not included in the overall response rates. These will only be based on completed surveys that meet the criteria for a minimum response.
What can I do to improve my response rate?
There are many ways for a provider to improve their response rates and they are focused around two key themes:
Increase brand awareness - Providers play a vital part in the success of Graduate Outcomes. It’s important that providers are engaging with the cohort population directly in the build up to the contact period commencing. We know from our work looking at response rates, that engagement with the online survey is dependent on recognition of the Graduate Outcomes brand. Providers play a key role in legitimising the online survey via their own direct and trusted engagement routes.
To support providers, we’ve created a contact plan to follow and we have supplied a whole range of communications resources in the Graduate Outcomes brand that can be used and personalised by a provider. We are also collating provider case studies to share ideas where tactics have worked and could be replicated.
Improve graduate contact details – HESA can only engage with gradates where contact details are accurate and up to date. This relies on a holistic approach step to this process which includes the collection and maintenance of contact details and submission. We have provided lots of information on how to approach this using the links provided. Additional support material can be found in the Contact Details coding manual, in particular, the following pages:
- Quality rules - help providers to identify duplicated data, shared data and non-permitted data
- Additional notes detail in data items including UKMOB, UKTEL, INTTEL and EMAIL help providers ensure the data submitted is in the format required
- Contact details guidance for providers - outlines what HESA classifies as accurate personal contact details.
Will you be looking to change your guidance to allow providers to make contact with graduates being surveyed to help increase the number of responses?
No, we will not be reviewing this guidance. Under no circumstances should providers make direct contact (for example by email or phone) with graduates currently being surveyed. This is to prevent the survey results being biased towards one or more groups of graduates which in turn reduces the precision of survey estimates. View our operational FAQs.
When will you be introducing incentives to increase the response rate?
We have taken feedback from the sector and assessed the internal requirements to implement incentives versus the potential impact on response rates. Due to this, and the other initiatives we have deemed higher priority, work to roll out incentives to increase online responses has been deprioritised. We will review this decision in the next collection (18/19). Under no circumstances must providers create incentives for their own graduates. View our operational FAQs.
When will providers know what is going to be in the publication?
This information will be provided in the new year which includes the detail of what will be included in the releases. Visit our dissemination timeline.
When will I get my final data?
As per the dissemination timeline, HE providers will get the quality assured dataset for their graduates only in February 2020. A specific date will be provided to operational contacts in due course.
Will providers have early sight of the data before it is released?
The datasets released in February will be the final deliveries and providers will not have access to the final data before then.
In terms of HESA’s open data releases, providers will not have early sight of these before they are released. This is because these are official statistics subject to strict rules regarding pre-release access.
What can I do with my data?
We will be issuing clear guidance about how the data and statistical outputs can be used in early 2020. This will be very different to DLHE so it’s vital that providers read and understand this.
In discussion with the Graduate Outcomes steering group, HESA has set the following response rate targets for Graduate Outcomes:
|Target group||Response rate target|
|UK domiciled full-time||60%|
|UK domiciled part-time||60%|
The above targets are applicable at a national, provider, undergraduate and postgraduate level. In addition to these, we will also monitor response rates for a range of socio-demographic and course characteristics such as age, gender, ethnicity and course.
We understand the importance of good quality outputs for the sector as well as other users of our data. We will continuously monitor the progress of Graduate Outcomes against the targets set out above. We will employ robust research and statistical methodologies to ensure we are able to produce estimates that meet the required statistical quality standard as well as our users’ requirements.
To enable providers to support the survey, final versions of the survey materials and communications with graduates can be viewed below.
There are a number of emails in the engagement strategy:
We will be carrying out the pre-notification strategy (introduced for cohort D in 17/18) for all cohorts in this collection. This will be sent to all graduates prior to the start of the contact period to provide key information about the survey and lets them know that they’ll receive their unique survey link very soon.
The email invitations sent within the contact period are derived from the core email text which is shared below. The other iterations (reminders) vary depending on the nature of the email and its role in the engagement strategy. Changes are mainly limited to the subject line and first paragraph.
SMS / text message
We are using a number of messages at key points in the engagement plan.
You can view the survey questions on the page below split by type. The data items and routing diagram can be found in the Graduate Outcomes Survey Results coding manual.