Skip to main content

Reliability of sensitive data

The Graduate Outcomes survey collects data on a number of topics which might be considered sensitive. In the case of both salary data and subjective wellbeing (SWB) data, there is a risk of social desirability bias, inasmuch as respondents might be expected to assume that some responses (such as higher salaries and generally positive SWB scores) are more favourable than others. Given that data on both of these topics is reported back to HE providers – although SWB data is only returned to providers in aggregate – there is some possibility that respondents will feel an incentive to answer questions about salary and SWB in such a way as to create a more favourable impression of how they are doing 15 months after course completion. In order to ascertain whether these potentially sensitive areas are subject to an elevated risk of misreporting, HESA has therefore undertaken a range of quality checks.[1] Respondents can be wary of offering precise location information because of privacy concerns. Location is therefore another potentially sensitive data item that we have added to our priorities for analysis. Feedback from users during the first year of operation highlighted the importance of location data, and we have committed to improving quality here. One element of this has been to try to understand what impact mode effects and related error might be having on the quality of these sensitive data items, during fieldwork.

In this section we first offer a discussion of our overall conclusions from the analysis, before covering the analysis itself in more detail in the following three subsections.

Assessment of a dataset, particularly in response to survey changes, forms an important part of the quality analysis process. Indeed, each of the three areas analysed in this report had alterations made to them in year two which contributed towards an improvement in the quality of the data provided in response to the questions. This highlights the value that regular review of survey questions and data can provide. In terms of subjective well-being and salary, the analysis indicates that improving the clarity around either question wording or scale can improve the reliability of the responses provided to the question. Equally, postcode analysis provides interesting results relating to the value of adding validation to free-text fields, as it seems that it aids in encouraging higher quality responses without increasing non-response too significantly.

Additionally, the analysis provides insight into the mode effects that may be influencing the data. In the analysis of subjective well-being data, it appeared that whilst mode effects had decreased for the anxiety question, they had increased slightly for the three positively worded questions. However, mode effects were still smaller overall for the three positively worded questions than the anxiety question. This highlights the complexities involved with analysing mode, but also indicates that steps can be taken to mitigate against mode effects to a certain extent. For instance, overall, higher quality data is received for postcode on the online mode than for telephone interviews, but telephone interviews also saw the biggest improvements in year two. The use of telephone interviews is invaluable in increasing response rates, so taking steps like this to aid in reducing any measurement error that may occur as a result of utilising this mode is important.[2] Participants were generally more likely to provide information to a question online, with higher levels of response for salary and postcode in this mode. This is to be expected, as self-administration of a survey increases the likelihood of a person disclosing information.[3] However, the quality of the data is not always higher, for example there were higher levels of misinterpretation and straight-lining online for the subjective well-being questions and, even with larger improvements in data quality for salary, online still had higher levels of graduates selecting one or two figure salaries. Whilst improvements are clear, there is arguably more that could be done in future to improve the dataset and reduce mode effects further.

In terms of the subjective well-being questions, although positive effects have been seen due to the changes, it is important to consider further areas for improvement. In relation to the questions, there may still be further possibilities for improvement of the layout. Although they are not in a grid format, the fact that they follow on from each other could still be contributing to an assumption of scale.[4] Scale labels could also be changed, as at present the happiness and satisfaction questions still utilise the word ‘extremely’. This could be brought more in line with other surveys utilizing the four ONS subjective well-being questions and may help to improve the interpretation and understanding of the scale further.

For salary, it is important to ensure that graduates are aware of the relevance of the salary question and that interviewers remain trained to improve the responses received to the question through the telephone mode. Equally, review will continue to determine if there are further steps that can be taken to improve validation or clarity in the question.

Next steps for improving postcode quality will consider methods for reducing the numbers of ‘Don’t know’ responses further and attempting to reduce the percentage of graduates who do not answer the question. Equally, consideration may need to be made of responses where the town/city question is optional now that postcode validation is in place and seems to be improving the data quality. Indeed, as validation requires a correct postcode, it may be the case that there is no longer a requirement to collect city if postcode has been collected, especially whilst city is a free-text field. In that case it could be removed to reduce the burden on participants and decrease the risk of drop-out. Therefore, detailed analysis comparing the responses received for the postcode and the city question could be a useful next step for improvement. This will also help to confirm the quality of the survey data. Changes were also made to the town/city question in year 3 to simplify the question wording. Previously, the question asked the graduate to provide the town, city or area in which they worked. It has now been simplified to ask the graduate for the nearest city or town to their place of work. Assessment of improvements in responses because of these changes will also contribute towards the quality assurance of location data.

More broadly, data quality could benefit from some further completion mode analysis considering primacy and recency effects and the influence of the mode of completion.[5] Equally, mode analysis could benefit from the inclusion of characteristic data, to check whether effects are influenced by the characteristics of the graduates completing on a certain mode. This will be particularly relevant if, for example, significant methodological changes are made to the way different modes of data collection are used in the survey. These steps will form part of the continual monitoring and improvement of the survey data in future.

We now turn to explain the details of our analysis and what we discovered for salary, SWB, and location data in turn, in each of the following three multi-part subsections.

Salary data

As part of the quality assurance process for the first year of Graduate Outcomes salary data, HESA conducted a series of comparisons between Graduate Outcomes and Longitudinal Educational Outcomes (LEO) salary data in order to check that the Graduate Outcomes salary data was not showing any unexpected patterns.[6] Although there are some key differences between the two datasets, including the fact that LEO does not distinguish between full- and part-time earnings, the comparisons still allowed us to see that the Graduate Outcomes salary data exhibited the trends which we would expect. Looking at the 2017/18 tax year, LEO data showed that median graduate earnings a year after graduation were £19,700 for females and £21,000 for males. Median Graduate Outcomes salaries for holders of first degrees were slightly higher, with females earning £22,000 and males earning £24,000, but the difference in earnings between male and female graduate in the two surveys is roughly equivalent. A comparison between Graduate Outcomes salary data by JACS (Joint Academic Coding System) subject and equivalent LEO data likewise showed broad similarities, with both datasets showing graduates with degrees in medicine and dentistry earning the highest salaries and graduates with degrees in creative arts and design earning the lowest salaries. A further comparison between these data sources cross-tabulated by SOC major groups again showed similar patterns.

An assessment of the salary data collected in year one of Graduate Outcomes lead to some changes to the salary question in an attempt to improve the quality of the data. One concern was that graduates were misinterpreting the phrase ‘to the nearest thousand’, for example, by seeing it a request for a one or two figure response. This led to the removal of this wording from the question. Previously, the question also had two separate validation pop ups which could exacerbate effects such as respondent fatigue. One pop up was removed entirely and the other was edited to clarify the amount entered by the graduate with a pound sign in front of it. This aimed to improve the overall quality of the salary data, both by ensuring one or two figure responses were lessened but also by reducing the provision of salaries that were too low or too high due to an unnoticed error by a graduate.

Mode of data collection can impact the responses provided to a question, especially where the information may be viewed as sensitive in nature to the graduate. Salary data is often seen as sensitive. Therefore, split by mode is included in the following analysis. Online can be seen as more confidential mode of data collection and may receive more responses.[7] Indeed, other factors such as the identification of a graduate, for example in a personalized salutation in an email, can increase response rates to a survey overall, but has also been found to increase bias and decrease response rates in questions such as salary.[8]

Additionally, the rapid impact of the pandemic on the economy, lifestyles, and employment[9] increases the importance for quality analysis on the salary responses to the survey. Checks will aim to ensure that graduates were still entering salary correctly wherever possible and that the quality of responses received to the question was not negatively impacted.

Assessment of non-response to salary

Question non-response is an important consideration in survey data quality and can be closely intertwined with other selection and mode effects. Item non-response can be caused by a number of factors, but graduates are less likely to answer difficult or sensitive questions.[10] This may also lead to higher levels of drop out overall. To begin to assess item non-response for salary, the percentages of blank salaries are below for paid work or self-employed full-time graduates who have answered currency as United Kingdom £ in cohort D, split by mode. This aims to give an indication of non-response to the salary question and to determine if there are mode effects at play.

Table 7 Highlighting differences in non-response to the salary question in Cohort D, by mode

Completion Mode   Cohort D Y1 % Cohort D Y2 % Difference
CATI Salary Blank 6.04% 9.37% 3.33%
CATI Salary Answered 93.96% 90.63% -3.33%
Online Salary Blank 4.62% 4.10% -0.52%
Online Salary Answered 95.38% 95.90% 0.52%

Although the online mode has seen a reduction in blank salaries, telephone interviews have seen a larger increase in blank salaries comparably. Further analysis in this report assesses if the extra data collected online is likely to be of a high quality. For telephone interviews, the numbers not responding are still relatively low for such a large survey, however non-response has clearly increased.

Boxplots can help to identify outliers and to highlight the layout of the data further and are therefore useful in visualising salary responses. They can also be useful in ensuring that data is consistent between modes and can provide extra visual reassurance that potential mode effects and increases in non-response have not negatively impacted the data. The boxplots below indicate salary split by mode, using data from Cohort D of year one and year two and outliers are not shown.

Boxplots show that salary responses are broadly consistent across online and telphone survey modes. Salaries have risen slightly across both modes between year one and year two. 

Figure 1 Boxplots of Cohort D salary responses, split by mode and year

The boxplots highlight some slight differences, but in general salary is consistent across modes and salaries appear to have risen slightly in both modes between year one and year two.

Additional analysis has been performed on blank salaries below. In the survey, the text preceding the currency question may warn the participant that they are about to be asked about their salary. The following analysis attempts to determine if the trends between telephone and online are a result of graduates feeling less inclined to provide their salary on the telephone mode, or if it is a result of graduates on the online mode being able to pre-empt the question and therefore not answering their currency either. As a result, this analysis utilizes employment basis, the question before the salary and currency question, to determine response rates where graduates have not yet seen the following question. As employment basis is only asked of graduates who are contracted to start work or in paid work for an employer, this analysis is only completed on graduates in paid work for an employer. For a fair comparison, this will be compared to only those graduates who are in paid work for an employer who have answered currency as UK £, to determine whether there are any issues.

Table 8 Highlighting differences in non-response to the salary question in Cohort D for graduates in paid work for an employer, where employment basis has been answered and with no filter by currency

Completion Mode   Cohort D Y1 % Cohort D Y2 % Difference
CATI Blank 10.34% 12.51% 2.17%
CATI Salary Blank 89.66% 87.49% -2.17%
Online Blank 7.22% 7.07% -0.15%
Online Salary Blank 92.78% 92.93% 0.15%

Table 9 Highlighting differences in non-response to the salary question in Cohort D for graduates who are in paid work for an employer, filtered by currency in UK £

Completion Mode   Cohort D Y1 % Cohort D Y2 % Difference
CATI Blank 5.68% 8.76% 3.07%
CATI Salary Blank 94.32% 91.24% -3.07%
Online Blank 4.48% 3.97% -0.51%
Online Salary Blank 95.52% 96.03% 0.51%

The two tables above highlight that when considering graduates who have answered employment basis, the online percentage difference is slightly closer to the telephone interview percentage difference for non-response. However, online still has far fewer blank salaries than the CATI mode. This highlights that it is unlikely that the difference is a result of graduates on the online mode being prewarned that the next question will be about their salary and therefore feel disinclined to answer currency.

Assessment of one and two digit salaries

The changes that were made to the salary question aimed to help reduce the number of one or two figure salaries provided by graduates in full-time employment due to the previous wording of the question. It was considered possible that graduates were interpreting the request for salary ‘to the nearest thousand’ as a request for graduates to provide one or two figures for their salary (for example, if a graduate had a salary of £20,000, they may enter ‘20’ as their response). The following table compares graduates who responded with figures under £100 in year one against year two, split by mode. To ensure that reductions were not just a result of fewer graduates selecting a salary of £0 these were separated from the one and two figure responses in the table.

Table 10 Highlighting the percentage of one or two figure responses in cohort D of year one and year two

Completion Mode   Year 1 % Year 2 % Difference
CATI £0 0.98% 0.76% -0.22%
CATI £100 or Over 98.77% 99.11% 0.34%
CATI Under £100 0.25% 0.13% -0.12%
Online £0 0.28% 0.38% 0.10%
Online £100 or Over 96.53% 99.03% 2.50%
Online Under £100 3.18% 0.58% -2.60%

There has been a decrease in one and two figure responses in both modes, but particularly online. Interestingly, the online mode has seen a slight increase in responses of £0, although the numbers are very small. It is also worth nothing that, for various reasons, it is expected that there will be a certain number of graduates who respond with figures of under £100.

Assessment of low salaries

For graduates who have said that they are working full-time, it is expected that they should be earning above a certain amount based on minimum wage. The changes made in year two aimed to reduce confusion and to encourage graduates to think about their response to the question, in order to improve the quality of answers. It was also considered possible that the wording was also making people think that they needed to provide salary in some other format, and as a result these graduates may therefore not be pulled into the previous assessment. An arbitrary selection of £15,000, which is slightly less than the expected minimum annual wage for a full-time employee in the UK, is used in the following table to indicate salaries that may be deemed as ‘too low’, and therefore may have quality issues.

Table 11 Responses to salary either under or above £15,000 in Cohort D, split by mode

Completion Mode   Cohort D Y1 % Cohort D Y2 % Difference
CATI £15,000 or Over 91.48% 92.41% 0.93%
CATI Under £15,000 8.52% 7.59% -0.93%
Online £15,000 or Over 90.57% 93.18% 2.61%
Online Under £15,000 9.43% 6.82% -2.61%

Both modes have seen a reduction in the percentages of graduates stating that their salary is under £15,000, which is a positive indication that quality has improved. Improvement is greater in the online mode.

Assessment of high salaries

There are also graduates who enter very large numbers in response to salary, perhaps to avoid answering the survey or due to confusion causing them to add extra digits to their salary. It is hoped that these numbers will have reduced, although they are relatively small regardless.

Table 12 Responses to salary either under or above £100,000 in Cohort D, split by mode

Completion Mode   Cohort D Y1 % Cohort D Y2 % Difference
CATI £100,000 or Under 99.73% 99.74% 0.01%
CATI Over £100,000 0.27% 0.26% -0.01%
Online £100,000 or Under 99.58% 99.59% 0.01%
Online Over £100,000 0.42% 0.41% -0.01%

Changes between years are minimal, as may be expected due to the small figures already included in this group. However, there has still been a slight reduction in both groups.

Our salary distribution results in a high positive skewness and high kurtosis. Although this will generate a relatively large proportion of extremely high values, high kurtosis values for positive skewness "appear to be important in the modelling of income."[11] However, the previous approach to trimming the top 1.5% of salaries, while at a comparatively high level, excluded a significant absolute number of plainly credible graduate salaries, for example for graduates from professional postgraduate courses with considerable pre-existing work history. It is generally accepted that analysing extreme earnings is difficult through survey based data[12] but given the other quality characteristics of the data in (in terms of overall numbers and rates or responses, low observable bias and representativeness) we expect this will not be a problem encountered with Graduate Outcomes data. HESA has therefore taken the decision to trim only 0.1% of extremely high values in recognition of user preferences for more comprehensive data on higher salaries. We are also amending our salary banding approach to make the kurtosis and skewness of the data more apparent to users. Users of the Graduate Outcomes microdata will need to consider their treatment of extreme values, and especially whether they choose to trim salary data further at the highest end of the distribution.

Conclusions on salary data quality

Positively, non-response reduced in the online mode for salary, with an increase in responses to the salary question. This is a positive outcome and could be a result of many factors, possibly including the question being clearer, increased publicity and knowledge of Graduate Outcomes leading to increased trust, or an increased feeling of privacy when answering online.[13] However, there was an increase in non-response to salary for telephone interviews of 3.33%. This may be a result of the pandemic, as there may be graduates who are not sure of their salaries or who are more wary of sharing their full income with an interviewer, especially given the circumstances that were surrounding employment, with the implementation of the furlough scheme and loss of business for many. With increased sensitivity, it seems likely that graduates would be less inclined to disclose this information to an interviewer.[14] We will keep this under review to observe any other patterns of respondent behaviour that may explain the differences, including any signs of an increase in item non-response. Due to the layout of the salary and currency questions, and the analysis being based on currency being answered, the possibility that the difference between telephone and online was due to graduates dropping out before currency in the online mode was considered. A separate analysis based on employment basis being answered was completed and it was determined that the difference was not due to graduates dropping out earlier and it is more likely that this is a mode effect. Indeed, this is to be expected, as a reluctance to disclose sensitive information such as salary over the phone is common and the use of an online mode can increase a participant's likeliness to disclose sensitive information.[15] Therefore, it may be expected to see increased disclosure of salary in the online mode, and therefore a more complete dataset, so these results are not surprising.

Changes that were made aimed to improve the quality of the salary data. Both modes saw a decrease in one and two digit responses, but online saw the biggest decrease. This is a positive improvement, which could be a result of the changes that were implemented to the salary question with this aim, particularly as the biggest effect was seen for the self-completion mode. Interestingly, online also saw a slight increase in responses of £0. The change was very small but could perhaps be an impact of the pandemic or graduates not feeling as comfortable sharing their information. In terms of low salaries, reductions were seen for both modes. Again, there was a greater impact on the online self-completion mode which is a positive indication of an improvement in data quality because of the changes. For salaries of over £100,000 the changes were minimal, although there was a very slight reduction for both, and numbers were previously low regardless. Overall, the data has seen improvements and results point to a reduction in responses that may have been a result of confusion caused by the wording of the question.

Next: Subjective wellbeing data

[1] For details of the various kinds of respondent error that we consider salient to Graduate Outcomes, see the Respondent error section.

[2] (Chang and Krosnick, 2010)

[3] (Brown et al., 2008)

[4] (DeLeeuw, 2018)

[5] (Chang and Krosnick, 2010; Kocar and Biddle, 2020)

[6] For the LEO data used in these comparisons, see Department for Education (2020), Graduate Outcomes (LEO): employment and earnings outcomes for higher education graduates by subject studied and graduate characteristics in 2017/18.

[7] (Brown et al., 2008)

[8] (Joinson et al., 2007)

[9] (Sułkowski, 2020)

[10] (Loosveldt and Billiet, 2002)

[11] (McDonald et al, 2013)

[12] (Office for National Statistics, 2015)

[13] (Lensvelt-Mulders and Boeije, 2007)

[14] (Kocar and Biddle, 2020)

[15] (Brown et al, 2008)