Location data

Analysis of Employment location questions – Postcode and town/city

Introduction and context

Graduates in certain types of employment who state that their place of work is in England, Scotland, Wales, or Northern Ireland during the survey are asked to provide the postcode for their place of work. Whilst this is a mandatory question, there is a response option of ‘don’t know’ that is available for graduates to select. However, graduates who select ‘don’t know’ or provide a short postcode will be required to provide the nearest city or town to their place of work in the next question, whereas the town/city question is optional for graduates who provide a full postcode. Respondents to surveys can be reluctant to provide personal information, particularly if they feel that this information may not be kept confidential or if the questions are administered by an interviewer (Tourangeau and Yan, 2007). This may be exacerbated by more people working from home, potentially making this data feel more personal to them. Equally, respondents may not know certain information about their place of work, and this may be a particular problem for postcode in the CATI completion mode if respondents do not have access to this at the time of the call.

In year two, validation was added to the postcode question to check the first two digits of the postcode and to ensure the formatting and length were correct. An assessment of the year two data in the 2^nd edition of the Graduate Outcomes Survey Quality Report highlighted very positive improvements in the quality of the data collected, likely as a result of the validation but also potentially other factors such as a change in working patterns. However, there was a slight increase in item non-response and whilst levels of ‘don’t know’ selection clearly reduced many graduates were still selecting this option, so next steps in the report highlighted that consideration would be made to reduce this further. Offering a ’don’t know’ response option can increase missing data; however, probes have been found to reduce missing data across different survey completion modes without negatively influencing respondent’s attitudes about a survey (DeLeeuw, 2018). In Cohort D of year three an additional validation pop-up was added when ‘don’t know’ was selected to try and encourage respondents to provide at least a partial postcode, which appeared to have a positive influence on response to the question in the online mode. Monitoring has continued this year to determine if there have been further improvements as a result of this additional validation.

Once graduates have answered the postcode question, they will then move on to the town/city question. Before year three of the survey the question was worded as follows, with the additional text in italics provided for context:

• “What was the town, city or area in which you worked? Please type in the town, city or area where your employment was/will be based and not the county. For example, if your employment was/will be based in London, please give the local area e.g. Holborn.”

From year three onwards the question wording was changed to:

• “What is the (nearest) city or town for your place of work?”

The change to the town/city question aimed to make it easier to answer and to provide continuity both in terms of question tensing and for future comparisons as a list of towns and cities was implemented for year four of the survey. It was also aimed at improving the usability of the data and to ensure that areas provided by graduates can be correctly identified for outputs.

As mentioned, a location list was added to the town/city question in the fourth year of the survey (C20072). HESA worked with devolved administrations and existing location information to develop the list, with the aim of improving the usability of the data and the graduate experience of answering the survey. The list has a search function to ensure it is usable. We committed to reviewing and improving the drop-down list, to ensure that it was fit for purpose. In depth assessment of the results from the list were ongoing through the year, to determine additions, removals and clarifications that may be required.

In the report last year, we also highlighted that the CATI completion mode may benefit from some further improvement for the postcode question. We suggested that discussion with the contact centre may aid in reducing ‘don’t know’ selection further, alongside further review of the validation text, as using these probes and messaging correctly can be very effective in reducing the selection of ‘don’t know’ in different ways across modes (DeLeeuw, 2018). As a result, an action plan was put in place for the CATI completion mode during cohort C of Year 4 and the information text was also reassessed to ensure it was clear for interviewers and respondents.

Results of the town/city list assessment will be discussed in this section, alongside the impact of the CATI action plan. Equally, the mapping of employment location will be assessed to determine if the addition of the town/city list has had an impact.

Methods and results

 
Impact of validation on postcode responses

Assessments of the postcode provided to both employment types are in Figure 10 and Figure 11, split by mode. Last year the assessment focussed on cohort D, as this is when the additional validation to encourage a postcode upon selection of ‘don’t know’ was introduced. Assessments this year instead look across the year to determine if these improvements have had a positive impact. Additionally, last year it was identified that further action may be required on CATI, as the validation was aimed more towards online respondents. As a result, an action plan was put in place to reduce ‘don’t know’ response on CATI, which the following assessments will aid in determining the outcomes of. Graduates included in these tables have answered the question before postcode (related to the country of their place of work) as England, Scotland, Wales or Northern Ireland.

Results are extremely positive this year, with the CATI mode seeing a reduction in the number of  ‘don’t know’ selections across both work types, but particularly for the ‘work for an employer’ group. Indeed, selections of long postcodes have increased the most, and there are also additional short postcodes provided.  This indicates that the CATI action plan has led to improvements in the data being collected for the survey. In terms of the online completion mode, it does seem that the additional validation has continued to improve response to the question. Year 4 had the validation for all four cohorts and the online mode has seen decreased selection of ‘don’t know’ for both employment types and increases in selection of long and short postcode provision. Overall, the response to the question has continued to improve.

Improvements to the town/city list so far

As mentioned, we committed to improving the town/city list which was introduced in Year 4. The list underwent a large amount of quality checking during the year, which aided in preparing for Year 5 and identified areas for further investigation. Indeed, since this first round of checking there have been other alterations made during the current ongoing survey (Year 5- C21072) and the review of the list will continue as the year progresses. We are also working with the contact centre on reducing the reliance on ‘other’ free-text use and are using this data to ensure that the list is fit for purpose.

Whilst a number of areas were reviewed and led to no further changes, there have been numerous alterations to the survey. It is worth noting that these changes were not introduced for the fourth year of the survey, however, an overview of some of the changes made for the Year 5 collection have been included for information. Initially, there were 17 new entries for Northern Ireland. Investigations indicated that coverage could be improved, with some areas included that were not used and other largely populated areas not in the list. Where additions created duplicates in the areas covered, entries were removed from the list. There were also changes to a number of the labels presented to graduates to make it clear when these options should be selected. For example, ‘Greater London’ was replaced with ‘London – borough unknown’, as investigations indicated confusion around which option to select due to the presence of boroughs in the list, and the option to select Greater London. Clarification was also added to ‘City of London’ following a review of the options, to ensure that graduates were aware this was not referring to London as a whole. Country was added to an entry and further clarification was added to a village named ‘Othery’ which seemed to have increased selection, likely as a result of confusion with the option ‘other’. ‘Greater Manchester’ and ‘Greater Glasgow’ have also been removed, as other areas at this level were not available to respondents, and this will allow more granular mapping. Research into current selection of these options indicated that graduates do have other options to select in the list that are more relevant to their area, and that removal would not have a detrimental impact on data collection.

Impact of the changes to the employment questions on location mapping

As mentioned previously, the town/city list was added to the survey in Year 4. To understand the impact that this may have had, the approach taken to the mapping of the location of employment this year is highlighted in Table 1.

Table 1: Mapping of graduate location in Year 4 following the addition of a town/city list for all graduates who provided location information

As can be seen from the table, over 97% of graduates this year were able to be mapped from a postcode or a drop-down list entry. This is a vast improvement on previous years, where only free-text responses were provided to town/city and has greatly reduced the amount of free-text mapping required. Only a small percentage of graduates were not able to be mapped. This highlights the benefits of introducing a drop-down list. Work on improving the list will continue and focus will also be on reducing the reliance of respondents on the ‘other’ free-text field that is still present in the survey.

Conclusions

In the online completion mode, the quality of postcode data has continued to improve, likely as a result of the additional validation, with a further reduction in the selection of ‘don’t know’  for both employment types.  On the CATI mode, the introduction of the action plan has also led to reductions in ‘don’t know’ selection, highlighting positive results due to the change. This will continue to be tracked. Both modes and work types saw increased selection of long postcodes predominantly, with an increase in short postcodes also present. Total item non-response remained fairly stable, with small changes between years. 

When considering the ability to map graduates to an area of the country, improvements are evident as a result of the introduction of the town/city list. Over 97% of graduates were able to be mapped on a postcode or list selection. We will continue to work on improvements to the list, to improve usability and hopefully attempt to reduce the use of ‘other’ online. We will also continue to work with the contact centre on reducing the use of ‘other' on the CATI completion mode.

Next: Processing error

References

DeLeeuw, E.D., 2018, August. Mixed-mode: Past, present, and future. In Survey Research Methods (Vol. 12, No. 2, pp. 75-89).

Tourangeau, R. and Yan, T., 2007. Sensitive questions in surveys. Psychological bulletin, 133(5), p.859.