Location data
Analysis of Employment location questions – Postcode and town/city
Introduction and context
Graduates in certain types of employment who state that their place of work is in England, Scotland, Wales, or Northern Ireland during the survey are asked to provide the postcode for their place of work. Whilst this is a mandatory question, there is a response option of ‘don’t know’ that is available for graduates to select. However, graduates who select ‘don’t know’ or provide a short postcode will be required to provide the nearest city or town to their place of work in the next question, whereas the town/city question is optional for graduates who provide a full postcode. Respondents to surveys can be reluctant to provide personal information, particularly if they feel that this information may not be kept confidential or if the questions are administered by an interviewer (Tourangeau and Yan, 2007). This may be exacerbated by more people working from home, potentially making this data feel more personal to them. Equally, respondents may not know certain information about their place of work, and this may be a particular problem for postcode in the CATI completion mode if respondents do not have access to this at the time of the call.
In year two, validation was added to the postcode question to check the first two digits of the postcode and to ensure the formatting and length were correct. An assessment of the year two data in the 2nd edition of the Graduate Outcomes Survey Quality Report highlighted very positive improvements in the quality of the data collected, likely as a result of the validation but also potentially other factors such as a change in working patterns. However, there was a slight increase in item non-response and whilst levels of ‘don’t know’ selection clearly reduced many graduates were still selecting this option, so next steps in the report highlighted that consideration would be made to reduce this further. Offering a ’don’t know’ response option can increase missing data; however, probes have been found to reduce missing data across different survey completion modes without negatively influencing respondent’s attitudes about a survey (DeLeeuw, 2018). In Cohort D of year three an additional validation pop-up was added when ‘don’t know’ was selected to try and encourage respondents to provide at least a partial postcode.
Once graduates have answered the postcode question, they will then move on to the town/city question. Before year three of the survey the question was worded as follows, with the additional text in italics provided for context:
- “What was the town, city or area in which you worked? Please type in the town, city or area where your employment was/will be based and not the county. For example, if your employment was/will be based in London, please give the local area e.g. Holborn.”
From year three onwards the question wording was changed to:
- “What is the (nearest) city or town for your place of work?”
The change to the town/city question aimed to make it easier to answer and to provide continuity both in terms of question tensing and for future comparisons as, although it was not present in year three, a list of towns and cities has been implemented for year four of the survey. It is also aimed to improve the usability of the data and to ensure that areas provided by graduates can be correctly identified for outputs.
Potential changes resulting from the additional validation for postcode will be investigated in this section, alongside a further assessment of the responses provided to town/city when postcode has been answered in some form, which was also highlighted as a requirement for continued assessment in the previous report. Equally, the mapping of employment location will be assessed to determine if the changes to the town/city question have had an impact.
Methods and results
Impact of validation on postcode responses
Assessments of the postcode provided to both employment types are in Figure 10 and Figure 11, split by mode. They compare the postcodes provided in cohort D of year two and three, when the additional validation to encourage a postcode upon selection of ‘don’t know’ was introduced. Graduates included in these tables have answered the question before postcode, related to the country of their place of work, with England, Scotland, Wales or Northern Ireland. Whilst mode is included for comparison and assessment purposes it is worth noting that the validation was aimed more towards online respondents.
Figure 8: Responses provided to postcode in cohort D of year two and year three, split by completion mode, when graduates are in paid work for an employer or voluntary/ unpaid work for an employer
Figure 9: Responses provided to postcode in cohort D of year two and year three, split by completion mode, when graduates are in self-employment/freelancing or running their own business
Whilst the CATI mode has seen a reduction in the number of long postcodes provided, there has not been an increase in the dropout rates for this question and the rates have reduced slightly in Figure 11. While levels of ‘don’t know’ selection have increased slightly in the CATI completion mode, most graduates are still providing a postcode, although it seems that the reduction in long postcode provision is replaced with a rise in short postcode provision. There could be several reasons for this change, including changes in the work location of graduates, or potentially some interviewer effects. Overall, however, the levels of postcode provision seem relatively stable on the CATI completion mode.
On the other hand, the online completion mode has seen a much larger change. This was the completion mode that the additional validation aimed to impact the most. For both employment sections, ‘don’t know’ responses have decreased by a fairly large amount, with a reduction of over 10% in the rate of selection in year three. Both employment types have seen a comparable increase in the provision of short postcodes, suggesting that the new validation is likely to be the reason for this increase and highlighting a very positive improvement resulting from the introduction of the additional validation to encourage short postcode provision when ‘don’t know’ is selected. Whilst it is difficult to check that responses are genuine, the existing validation, which checks the first two digits of postcode to ensure they are correct, should help to ensure that postcodes provided by graduates are valid entries. Equally, text analysis of the most common postcodes for both employment types did not highlight any large concerns e.g. excessive selection of the example postcode. It is interesting to note that the online mode also saw a slight decrease in long postcode provision, perhaps suggesting that this reduction in both modes may be a result of external factors. As the validation is only shown to graduates who have already selected ‘don’t know’ as their response it should hopefully avoid discouraging graduates from providing a longer postcode, as they would not see the additional guidance unless they had already made this selection.
Another interesting difference is noticeable between the two employment types. In both years graduates in paid work or voluntary/unpaid work for an employer selected ‘don’t know’ less and provided a long postcode more often in the online mode compared to graduates in self-employment/ freelancing or running their own business.
Interaction between postcode and the town/city question
To contribute towards the understanding of the data quality of location and the relationship between responses to town/city and postcode, as well as the assessment of the requirement of both location questions in the survey, Table 15 and Table 16 highlight whether responses were received to the town/city question in cohort D for each employment type. These results are split by graduates for whom the question was optional and those for whom it was mandatory. The question is optional when postcode has been answered fully and mandatory for graduates with a short postcode or response of ‘don’t know’. The assessment also aims to determine differences in response levels following the change to the town/city questions in year three.
Table 25: Responses provided to the town/city questions where the question was either optional or mandatory, including graduates in paid work for an employer or voluntary/unpaid work for an employer or contracted to start work in cohort D, split by completion mode.
|
|
Mode |
Year 2 |
Year 3 |
Difference |
|---|---|---|---|---|
|
Mandatory and Answered |
CATI |
99.63% |
99.49% |
-0.13% |
|
Optional and Answered |
CATI |
99.83% |
99.47% |
-0.36% |
|
Mandatory and Answered |
Online |
99.32% |
98.86% |
-0.45% |
|
Optional and Answered |
Online |
98.64% |
99.51% |
0.87% |
Table 26: Responses provided to the town/city questions where the question was either optional or mandatory, including graduates in self-employment/freelancing or running their own business in cohort D, split by completion mode.
|
|
Mode |
Year 2 |
Year 3 |
Difference |
|---|---|---|---|---|
|
Mandatory and Answered |
CATI |
99.61% |
99.54% |
-0.07% |
|
Optional and Answered |
CATI |
99.72% |
99.45% |
-0.28% |
|
Mandatory and Answered |
Online |
98.81% |
98.85% |
0.04% |
|
Optional and Answered |
Online |
98.51% |
99.07% |
0.56% |
Impact of the changes to the employment questions on location mapping
As mentioned previously, the town/city question was altered in year three and asks graduates for the nearest town/city to their place of work, rather than the area/ town/ city of their place of work. This change had a number of purposes but in part aimed to improve the ability to correctly identify the location of the graduate for output purposes. Table 17 and Table 18 highlight the ability to assign graduates to a location at the government office region level across the three years of the Graduate Outcomes survey so far, split by employment type. It is important to note that there are several factors that may influence this, and these differences will not all result from changes to the employment questions. The figures shown include graduates who selected that the country of their place of work is in England, Wales, Scotland, or Northern Ireland, as with previous assessments in this section, but also includes some graduates who selected Channel Islands or Isle of Man at this section. This is as some graduates who selected another country occasionally map to these areas, and as the option was added later it allows a more accurate comparison. There may be graduates in this question who did not answer further than the country of their place of work.
Table 27: Levels of mapping for the location of employment of graduates at the government office region level across years, when graduates have answered their location of employment as England, Scotland, Wales, Northern Ireland or Channel Islands/Isle of Man and they are in paid work for an employer or voluntary/unpaid work for an employer or contracted to start work
|
Region |
Year 1 |
Year 2 |
Year 3 |
Y3-Y1 |
Y3-Y2 |
|---|---|---|---|---|---|
|
North East |
3.35% |
3.29% |
3.25% |
-0.10% |
-0.04% |
|
North West |
10.17% |
10.06% |
9.87% |
-0.30% |
-0.19% |
|
Yorkshire and The Humber |
7.15% |
7.08% |
6.93% |
-0.22% |
-0.15% |
|
East Midlands |
5.47% |
5.56% |
5.47% |
0.00% |
-0.09% |
|
West Midlands |
7.34% |
7.42% |
7.31% |
-0.03% |
-0.12% |
|
East of England |
6.57% |
6.73% |
6.74% |
0.17% |
0.01% |
|
London |
24.67% |
24.04% |
24.49% |
-0.18% |
0.45% |
|
South East |
11.49% |
11.57% |
11.36% |
-0.13% |
-0.20% |
|
South West |
6.96% |
7.08% |
7.13% |
0.17% |
0.05% |
|
England |
2.32% |
2.12% |
2.26% |
-0.06% |
0.14% |
|
Northern Ireland |
3.00% |
3.10% |
3.16% |
0.16% |
0.06% |
|
Scotland |
7.57% |
7.89% |
7.77% |
0.21% |
-0.12% |
|
Wales |
3.92% |
3.88% |
4.06% |
0.13% |
0.17% |
|
Guernsey, Jersey and the Isle of Man |
0.02% |
0.17% |
0.19% |
0.17% |
0.02% |
|
Non-UK* |
0.01% |
0.01% |
0.02% |
0.01% |
0.01% |
*Where Dublin has been returned in free text or the country provided contradicts the selection made for location of employment
Table 28: Levels of mapping for the location of employment of graduates at the government office region level across years, when graduates have answered their location of employment as England, Scotland, Wales, Northern Ireland or Channel Islands/Isle of Man and are in self-employment/freelancing or running their own business
|
Region |
Year 1 |
Year 2 |
Year 3 |
Y3-Y1 |
Y3-Y2 |
|---|---|---|---|---|---|
|
North East |
2.98% |
2.95% |
2.64% |
-0.35% |
-0.31% |
|
North West |
9.43% |
9.52% |
8.60% |
-0.83% |
-0.92% |
|
Yorkshire and The Humber |
6.10% |
6.34% |
5.96% |
-0.13% |
-0.38% |
|
East Midlands |
4.83% |
4.92% |
5.01% |
0.18% |
0.09% |
|
West Midlands |
6.61% |
7.15% |
6.90% |
0.28% |
-0.26% |
|
East of England |
6.56% |
6.76% |
6.84% |
0.28% |
0.08% |
|
London |
29.04% |
27.71% |
29.25% |
0.22% |
1.54% |
|
South East |
10.82% |
11.45% |
11.30% |
0.48% |
-0.15% |
|
South West |
7.09% |
6.87% |
6.92% |
-0.17% |
0.05% |
|
England |
3.26% |
3.14% |
3.32% |
0.06% |
0.19% |
|
Northern Ireland |
2.60% |
2.53% |
2.61% |
0.01% |
0.08% |
|
Scotland |
6.79% |
6.58% |
6.48% |
-0.32% |
-0.11% |
|
Wales |
3.84% |
3.88% |
3.96% |
0.12% |
0.08% |
|
Guernsey, Jersey and the Isle of Man |
0.03% |
0.17% |
0.19% |
0.16% |
0.01% |
|
Non-UK* |
0.01% |
0.01% |
0.02% |
0.01% |
0.01% |
*Where Dublin has been returned in free text or the country provided contradicts the selection made for location of employment
Overall, levels of mapping seem fairly similar, and it is interesting to see the differences between the three years of the survey, perhaps indicating changes in the employment locations of graduates over the last few years. Whilst many countries have seen a slight increase in mapping, perhaps highlighting a decrease in the ability to map at a more granular level, others have decreased slightly depending on the year comparisons. Changes do not seem concerning for either employment type, but it will be important to continue to assess this over time. London is one area that appears to have seen an increase in mapping, which may be a result of more graduates entering London as their nearest city instead of a small area name that previously may have mapped elsewhere. However, when compared to year one this increase is not as large and decreases in Table 17. Equally, the decrease in mapping to the East of England and the South East does not seem comparable. It therefore seems likely that although changes to the question may have contributed slightly to the change, it is unlikely to be the sole cause. When assessing the mapping at a lower level it seems that some areas that were previously mapping to smaller locations within a city are now mapping to the city itself, which is as expected. Indeed, whilst it is difficult to ascertain the full impact of the changes, it is also possible that some areas are now mapping more reliably to the correct area if graduates were previously providing names of small areas in the free-text field.
Conclusions
In the online completion mode, the quality of postcode data appears to have improved further as a result of the additional validation, with a large reduction in the selection of ‘don’t know’ in cohort D of year three for both employment types. Whilst the same cannot be seen in the CATI completion mode levels seem fairly stable, although there is a slight increase in ‘don’t know’ selection for both employment types. Additionally, both completion modes saw a reduction in long postcode provision and an increase in short postcode provision, suggesting that these differences could be due to changes in the working patterns of the population perhaps leading to increased inability or reluctance of graduates to provide this information. The addition of the validation should not be related to the reduction in long postcode provision as the validation is only shown to graduates who have already selected ‘don’t know’ as their response. Total item non-response remained fairly stable, with small changes between years.
Assessing the relationship between postcode and the town/city question following it highlights similar levels of response between the two years. Whilst CATI has seen very small decreases in the question being answered across both mandatory and optional graduates for both employment types, online has seen slight increases for all but one of these groups. Overall, response rates for this question remain high where postcode has received some form of answer, and this does not seem to have been impacted much by the additional validation or by changes to the town/city question.
When considering the ability to map graduates to an area of the country from the two employment questions, the quality of the data appears to remain good. Mapping has continued at similar levels, with many graduates able to be mapped at the government office region level.
In terms of next steps, work has already begun on improving the location questions further, with the addition of a drop-down list for the town/city question already in place in the year four survey. Evaluation of the requirements for these questions will continue in the survey review process and as data quality assessment continues, to determine whether both postcode and town/city are required once analysis determines if the data quality or provision of responses has improved sufficiently. As highlighted in this assessment, the CATI completion mode may benefit from some further improvement and assessment for the postcode question. Discussion with the contact centre may aid in reducing ‘don’t know’ selection further, alongside further review of the validation text, as using these probes and messaging correctly can be very effective in reducing the selection of ‘don’t know’ in different ways across modes (DeLeeuw, 2018). Continued monitoring of the questions and assessment of potential improvements will aid in mitigating issues and to ensure there is not a further reduction in the granularity of the data, with fewer graduates entering a long postcode, particularly following the success of the additional validation in reducing ‘don’t know’ as a response online. Equally, assessment of the new drop-down list for town/city is continuously ongoing to ensure that the list is fit for purpose, and a free-text option still exists for graduates to use if they cannot find their location in the list. This free-text field will aid in assessing the appropriateness of the list for continued use in the future.
References
DeLeeuw, E.D., 2018, August. Mixed-mode: Past, present, and future. In Survey Research Methods (Vol. 12, No. 2, pp. 75-89).
Tourangeau, R. and Yan, T., 2007. Sensitive questions in surveys. Psychological bulletin, 133(5), p.859.