Reliability of sensitive data

Introduction and context

Within the Graduate Outcomes survey there are some questions that could be perceived by a respondent as sensitive in nature, and this kind of question in particular can be at risk of reduced data quality in responses as a result, for example through increased item non-response or the misreporting of answers. (Previous editions of this report contain details of our investigations into subjective wellbeing data. The latest update can be found in the 2^nd edition of the Graduate Outcomes Survey Quality Report). Many factors can influence the responses provided to potentially sensitive questions including the mode of completion, question wording, presence of third parties whilst completing a survey and assurances about privacy, confidentiality, or use of the data (Tourangeau and Yan, 2007; Ong and Weiss, 2000). For mode of completion, self-administration modes are generally found to increase respondents reporting potentially undesirable behaviours (Tourangeau and Smith, 1996; DeLeeuw, 2018). Confidentiality and privacy assurances have also been found to improve responses to sensitive questions. However, in some cases, these assurances can have the opposite effect, potentially as they bring data usage or privacy concerns to the forefront of a respondent’s mind who previously may not have considered it (Acquisti, Brandimarte and Lowenstein, 2015).

In Cohort D of year 3, additional hover text was added to certain employment questions that may be viewed as more sensitive by respondents. The text aims to reassure graduates and add context around the use of the survey data for their job title, salary and employer name and is worded as follows (with words in brackets used depending on relevancy to the question):

• “This information is used to produce national statistics on (occupations/industries) and individuals cannot be identified from published material”

To assess whether this question has encouraged graduates to respond, there are a few areas to assess. As mentioned, both item non-response and the quality of survey data will need to be considered to determine if the addition has led to any changes in the levels of item non-response or potential misreporting by respondents to protect their privacy. 

Methods and results

 
Impact of hover text on job title and employer name

It is important to consider that questions asked in a survey have the potential to be perceived as sensitive or personal to the respondent. It is difficult to determine which questions may be perceived as sensitive, but dropout rates and question responses can aid in identifying them. Job title and employer name were identified as questions that graduates seemed reluctant to respond to and a hover text feature was added to reassure graduates of the use of their data in cohort D of year three. These questions both have the potential to be asked at two points in the survey, depending on the activities selected and the route taken by the graduate. They are mandatory questions that require graduates to provide their job title or employer name during the relevant census week for their cohort. Responses are entered in a free-text field, so it is possible for graduates to provide a response that is not accurate, for example by editing their job title or entering random characters to bypass the question due to influences such as privacy concerns or social desirability bias.

Item non-response to job title or employer name

Initially, item non-response has been assessed for job title in cohort D of year one, year two and year three and results are shown in Table 1 and Table 2. The assessment has been carried out on both employment types and intensities separately, where the prior mandatory question of employment intensity has been answered. It includes splits by completion mode and does not include information that would have been copied over from the other employment type if the activities were the same.

Table 6: Item non-response to job title for graduates in route B (paid work, unpaid/ voluntary work for an employer, or contracted to start work) who answered the relevant employment intensity question, split by completion mode and employment intensity.

	Full time					Part time
	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2
CATI	0.69%	0.71%	0.77%	0.08%	0.05%	0.69%	0.80%	0.86%	0.17%	0.05%
Online	4.26%	5.66%	6.34%	2.08%	0.68%	6.36%	8.23%	8.32%	1.96%	0.09%

Table 7: Item non-response to job title for graduates in route C (self-employment/ freelancing, running their own business or developing a portfolio) who answered the relevant employment intensity question, split by completion mode and employment intensity.

	Full time					Part time
	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2
CATI	1.11%	1.00%	1.31%	0.20%	0.31%	0.98%	0.88%	1.00%	0.02%	0.12%
Online	11.99%	12.64%	12.22%	0.23%	-0.42%	9.52%	9.31%	9.77%	0.25%	0.46%

Across most of the groups item non-response has increased slightly since year two, although differences are small. However, non-response did decrease for full time graduates who went down route C in the online completion mode. The biggest increase was in route B between year one and year three. Comparisons across years indicate that in most groups non-response was rising regardless, with increases visible between year one and year two when no changes occurred. Overall, it doesn’t appear that the hover text has had a positive impact on level of response provision to job title, although it cannot be certain that it has had a negative impact either and results may be impacted by other factors, such as external influences or changes to other questions in the survey.

One of the other questions that was selected for the addition of hover text was the question on employer name, which has the potential to be asked at two points in the survey depending on the activities selected by the graduate. It is asked of graduates who answer section B (in paid work for an employer, voluntary/ unpaid work for an employer or contracted to start work) or, alternatively, graduates in section C (but only those in self-employment/ freelancing or running their own business). In route B it can be preceded by either employment basis, which is mandatory, or the optional questions of salary and currency, if they are relevant to the graduate. In route C it is always preceded by employment duties, which is mandatory.

Analysis below highlights item non-response to employer name split by the survey route taken by the graduate. It does not include information that would have been copied over from the other employment type if the activities were the same. Results for route B are shown in Table 3 for graduates who answered employment basis, and salary and currency if relevant, and are shown in Table 4 for graduates in route C who answered the job duties question.

Table 8: Item non-response to employer name in route B. Results are shown for cohort D across years, when employment basis was answered and either both salary and currency were answered or were not relevant to the graduate, split by employment intensity

	Full time					Part time
	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2
CATI	0.64%	0.66%	0.63%	-0.01%	-0.03%	0.64%	0.70%	0.66%	0.02%	-0.04%
Online	2.28%	2.94%	2.82%	0.54%	-0.12%	3.14%	3.54%	3.45%	0.31%	-0.09%

Table 9: Item non-response to employer name in route C when graduates are in self-employment/freelancing or running their own business. Results are shown for cohort D across years, when job duties were answered, split by employment intensity

	Full time					Part time
	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2
CATI	1.00%	1.09%	1.00%	-0.01%	-0.10%	0.89%	0.85%	1.02%	0.14%	0.17%
Online	4.11%	7.71%	4.98%	0.87%	-2.73%	4.00%	6.43%	4.99%	0.99%	-1.44%

There has been a slight reduction in item non-response to employer name between year two and three in the online mode, with a more noticeable reduction in route C, especially when considering that item non-response rose on CATI for route C but dropped for route B. This reduction may be a result of the hover text addition; however, the difference is not extreme, and it is important to consider other factors in this reduction. Indeed, by this point in the survey graduates will have seen the privacy hover text at least one other time. If strong privacy concerns were raised as a result of the hover text that caused graduates to drop out it is likely that this would have already occurred, and these graduates would no longer remain in the survey to see the text at the point of employer name. Equally, non-response has actually risen since year one, highlighting that there may be other reasons for the difference.

Free-text analysis of job title or employer name

As both job title and employer name are mandatory questions with free-text fields, respondents must enter text to move on in the survey. Graduates who are unable or do not want to provide the relevant information, but do not want to drop out of the survey, sometimes enter other text to achieve this. This can include direct refusal to answer, descriptions of non-disclosure agreements, entry of random characters or the inclusion of other information about the role. As a result, it can be difficult to determine the levels of misreporting or reluctance to respond due to the variety of answers used, but some analysis of free text can be useful in understanding more about respondent’s interactions with the questions.

Assessment of free text was carried out in a number of ways. Results in Table 5 to Table 8 highlight the presence of five free text responses against all other responses to provide a brief summary of job title or employer name entries that indicate an inability or reluctance to respond. Manual checks were carried out to try and ensure genuine responses were not pulled in (e.g. refuse collector). It is important to note that a variety of responses can be included, including instances where respondents have refused to provide exact names etc but have provided other details of their employer (e.g. not giving the name of a specific school but providing the name of the relevant local council). Whilst the first three options searched for any inclusion of the part of the word included in the table, the final two options (N/A or NA) only include entries that directly matched and are included to highlight another type of response that may be used. It may be used in refusal but is also used by graduates who cannot answer the question for various reasons and may indicate inability, rather than reluctance, to respond in some cases. 

Table 10: Free text responses to job title in route B between cohort D of year two and year three of the survey, with same activity responses removed

	CATI			Online
	Year 2	Year 3	Difference	Year 2	Year 3	Difference
Refus	0.05%	0.04%	-0.01%	0.00%	0.00%	0.00%
Confidential	0.03%	0.02%	0.00%	0.00%	0.00%	0.00%
Disclos	0.02%	0.03%	0.00%	0.01%	0.01%	0.00%
N/A	0.05%	0.06%	0.02%	0.06%	0.06%	0.00%
NA	0.01%	0.01%	0.00%	0.02%	0.03%	0.00%
All other	99.84%	99.83%	-0.01%	99.91%	99.90%	-0.01%

Table 11: Free text responses to job title in route C between cohort D of year two and year three of the survey, with same activity responses removed

	CATI			Online
	Year 2	Year 3	Difference	Year 2	Year 3	Difference
Refus	0.11%	0.12%	0.01%	0.00%	0.00%	0.00%
Confidential	0.02%	0.04%	0.02%	0.00%	0.00%	0.00%
Disclos	0.02%	0.06%	0.04%	0.00%	0.00%	0.00%
N/A	0.72%	0.91%	0.19%	1.96%	1.57%	-0.39%
NA	0.10%	0.17%	0.07%	0.37%	0.20%	-0.17%
All other	99.03%	98.70%	-0.33%	97.67%	98.23%	0.56%

Table 12: Free text responses to employer name in route B between cohort D of year two and year three of the survey, with same activity responses removed

	CATI			Online
	Year 2	Year 3	Difference	Year 2	Year 3	Difference
Refus	1.22%	0.88%	-0.34%	0.01%	0.01%	0.00%
Confidential	0.42%	0.48%	0.07%	0.03%	0.02%	0.00%
Disclos	0.49%	0.52%	0.03%	0.11%	0.10%	-0.01%
N/A	0.48%	0.50%	0.02%	1.24%	1.71%	0.47%
NA	0.10%	0.09%	-0.01%	0.39%	0.63%	0.25%
All other	97.30%	97.53%	0.23%	98.23%	97.53%	-0.70%

Table 13: Free text responses to employer name in route C between cohort D of year two and year three of the survey, with same activity responses removed

	CATI			Online
	Year 2	Year 3	Difference	Year 2	Year 3	Difference
Refus	1.13%	0.79%	-0.34%	0.00%	0.03%	0.03%
Confidential	0.41%	0.38%	-0.03%	0.11%	0.07%	-0.04%
Disclos	0.43%	0.44%	0.02%	0.22%	0.15%	-0.07%
N/A	2.83%	2.73%	-0.10%	4.92%	5.27%	0.36%
NA	0.50%	0.51%	0.01%	1.11%	1.14%	0.03%
All other	94.70%	95.14%	0.43%	93.64%	93.34%	-0.30%

Job title has generally seen few changes, particularly in route B, highlighting that the hover text did not appear to have an overly negative impact on the responses received to the question. It also saw a reduction in the selection of NA or N/A in the online completion mode in route C, compared to CATI which saw a very slight increase. This may indicate that hover text was effective online at reducing reluctancy to respond to job title, although it must be considered that other factors may impact this, such as changes in the employment patterns of the cohort. Employer name saw some slight decreases in refusal in the CATI mode. Both online and CATI saw increases and decreases across the selected groups. Online, NA or N/A selections saw a slight increase across both routes, although this could be impacted by a number of factors and is interesting as employer name was the group that saw a decrease in item non-response after the introduction of hover text. Overall, changes have not been large and do not indicate any areas of major concern. More in-depth assessments of the free text fields have taken place and the responses will continue to be monitored.

Impact of hover text on SOC coding

Graduate Outcomes employment data is coded using the Office for National Statistics’ (ONS) Standard Occupational Classification (SOC). Both the job title and the employer’s name provided by the graduate are considered in the coding of records. In some cases, a code cannot be assigned to a graduate, for example if there are no appropriate codes present in the indexes or if information is missing in the survey data returned by the graduate, and in these cases the records are assigned a code of 0001 to indicate that they are uncodable. Although there are a number of factors that can influence coding it could be useful to assess the prevalence of these uncodables in the dataset to ensure that the hover text in job title and employer name did not have a negative impact on the ability to code records.

The percentage of uncodable records, split by activity type, is shown in Table 9 for cohort D across years. Hover text was only introduced in cohort D of year three, so results are also included for cohorts A, B and C across years in Table 10 to help identify further whether any changes may be a result of the addition of hover text.

Table 14: Percentage of uncodable records returned by the coding supplier in cohort D of the survey across years

	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2
Route B	0.60%	0.49%	0.73%	0.13%	0.24%
Route C	3.61%	3.61%	3.81%	0.20%	0.20%

Table 15: Percentage of uncodable records returned by the coding supplier in cohorts A-C of the survey across years

	Year 1	Year 2	Year 3	Y3-Y1	Y3-Y2
Route B	0.64%	0.55%	0.69%	0.05%	0.13%
Route C	6.10%	2.46%	2.97%	-3.13%	0.51%

As indicated in Table 9, levels of uncodables have risen in year three cohort D, however, they have also risen in Table 10 between year two and year three in cohorts A-C, which did not have the hover text addition. Differences are not large and cannot be attributed to specific changes, but do not seem to indicate any areas of particular concern for the coding of the data as a result of the hover text addition. 

Changes to salary and currency questions

Income is commonly considered to be a sensitive topic for a survey question, and it often has higher levels of item non-response associated with it due to the intrusive nature of the question and concerns of disclosure (Tourangeau and Yan, 2007). As a result, it was selected as one of the questions to receive the additional hover text to offer reassurance to graduates in cohort D of year three. Alongside this change, the order of currency and salary were switched in cohort A of year three, which will need to be considered when comparing the difference in the results between cohort D in year two and year three. In year three, currency was also edited to offer only ‘United Kingdom, Pounds, £’ or ‘Other’, with salary only being asked when ‘United Kingdom, Pounds, £’ is selected as the response. This reduces the collection of unnecessary data. From cohort B of year two onwards currency was made compulsory once a salary value was entered. Prior to this change graduates could provide a salary but skip currency. This also led to the collection of unusable data. The new question order removes the need for a compulsory response in this block, which it was hoped would reduce drop-out rates. Changes made to the salary question prior to year three were assessed previously, and outcomes can be found in, the 2^nd edition of the Graduate Outcomes Survey Quality Report.

Salary is asked at a different point in the survey depending on the route taken by a graduate. As a result, graduates have been assigned into a work type depending on the point at which they will have answered salary, rather than their main activity, for the purposes of quality analysis. For example, a graduate with a main activity of self-employment may have been asked salary within the paid work section if they also selected paid work as an activity in the first question. It is important to note that to determine drop out points for item non-response rates they would therefore be assigned to the paid work group. 

Item non-response to salary

Due to the removal of salary as a question for graduates earning in international currencies in year three, the majority of non-response analysis will focus on graduates with a currency of UK pounds. However, initial checks were completed on item non-response to either salary or currency, by mode. Due to currency and salary being switched around at the beginning of the year three collection the comparison was based on the set of the two questions as opposed to each one individually. This also allowed for the fact that currency and salary are generally optional questions which a respondent can skip without answering.  Comparisons highlighted a much lower percentage of graduates who did not respond to either question in cohort D of year 3 of the survey, across both modes. This is unlikely to be solely as a result of the addition of hover text and will also be related to the other changes in the question, such as the change in order. It does indicate that the change in order has encouraged a response to be provided to at least one of the questions, although this does not necessarily highlight an improvement in survey coverage as graduates may be more likely to respond to currency when it appears first as it is not as sensitive, and they may have been discouraged from responding to either question previously when salary preceded currency.

Due to the removal of salary as a question for currencies other than United Kingdom Pounds in year three, further analysis will focus on this group only. Alongside the removal of the salary question for some graduates in year three, currency was also made compulsory when salary was answered in cohort B of year two, which adds further complications to analysis. Table 11 and Table 12 indicate the level of item non-response to salary when currency was selected as United Kingdom Pounds for cohort D of year two and three.

Table 16: Item non-response to salary in cohort D when currency is answered as 'United Kingdom, Pounds' in the CATI mode

	Full time			Part time
	Year 2	Year 3	Difference	Year 2	Year 3	Difference
Contracted to start work	12.21%	24.08%	11.87%	35.79%	49.66%	13.87%
Paid work	8.64%	17.45%	8.81%	21.26%	37.73%	16.48%
Self-employed	26.96%	40.88%	13.92%	33.53%	47.88%	14.35%

	Full time			Part time
	Year 2	Year 3	Difference	Year 2	Year 3	Difference
Contracted to start work	4.92%	8.31%	3.40%	13.57%	21.80%	8.23%
Paid work	3.95%	7.81%	3.86%	9.37%	16.66%	7.28%
Self-employed	10.22%	21.42%	11.21%	13.81%	23.21%	9.40%

Overall, item non-response appears higher. However, it is important to note that rates of response to either of the questions has increased, as highlighted previously, and currency and salary order has changed. The difference is not as pronounced in the online completion mode. This may be a result of the financial uncertainty caused by the pandemic which is making respondents either reluctant to or unable to provide accurate details of their salary and indeed, questions about income often see much higher levels of item non-response (Tourangeau and Yan, 2007). Also worth noting is that graduates who are self-employed and contracted to start work have higher levels of item non-response, and part time employment also sees more non-response.

To check whether item non-response to salary is also leading to increased drop out, graduates who answered currency as UK £ in the paid work route but did not answer salary were assessed to see if they had answered the employer name question that comes after salary. It is worth noting that this question can also be considered sensitive and can have higher dropout rates than other questions, but it may indicate whether non-responding graduates are just skipping the salary question or if they are dropping out of the survey completely.

Table 18: Levels of item non-response to employer name for graduates who answered currency as UK £ but did not answer salary in the paid work route, split by employment intensity and completion mode

	Year 1		Year 2		Year 3
	Full time	Part time	Full time	Part time	Full time	Part time
CATI	2.41%	1.14%	2.13%	1.28%	2.19%	1.40%
Online	7.73%	7.17%	9.54%	8.43%	10.33%	10.58%

As mentioned, employer name is itself a more sensitive survey question and does cause some dropout of graduates. However, it is also clear that there is some dropout of graduates at the salary question, which appears to have worsened slightly across years. As with previous assessments, changes in the questions make it difficult to compare years, but it appears that drop out is more likely online than on the CATI completion mode.  It may be that graduates online do not realise that the salary question is optional and therefore drop out of the survey instead of skipping the question, whereas in the CATI completion mode interviewers will be able to encourage graduates to move on and continue with other questions. These potential dropout rates will continue to be assessed.

As the order of currency was changed at the start of year three, but hover text was not added, a useful comparison for hover text impact may be the comparison between item non-response online to salary and currency between cohorts A-C and cohort D. These cohorts can have very different populations, and numbers of graduates in different groups can be much smaller in cohorts A-C which must be considered in conclusions, but it may provide a more accurate view of the hover text impact.

Table 14 and Table 15 have been added to compare item non-response between cohorts A, B and C and cohort D in year three, with a table for each completion mode to provide a comparison that may aid in understanding changes in the online mode better. As hover text was only added in cohort D, but changes to the questions were made at the start of year three, it may aid in assessing the impact of the hover text addition.

Table 19: Item non-response to salary in cohorts A, B and C versus cohort D of year three, when currency is answered as 'United Kingdom, Pounds' in the CATI completion mode

	Full time			Part time
	A, B, C	D	Difference	A, B, C	D	Difference
Contracted to start work	17.90%	24.08%	6.18%	49.15%	49.66%	0.50%
Paid work	14.38%	17.45%	3.07%	28.03%	37.73%	9.70%
Self-employed	38.50%	40.88%	2.38%	46.15%	47.88%	1.73%

	Full time			Part time
	A, B, C	D	Difference	A, B, C	D	Difference
Contracted to start work	13.67%	8.31%	-5.36%	28.18%	21.80%	-6.38%
Paid work	9.28%	7.81%	-1.47%	16.32%	16.66%	0.34%
Self-employed	18.14%	21.42%	3.29%	19.78%	23.21%	3.44%

Graduates who are contracted to start work see the biggest reduction in non-response between the first three and last cohort across both employment intensities online, however, as mentioned the number of graduates in these sections can be very small, particularly for cohorts A-C. Full-time, paid work respondents also seem to have seen a reduction in item non-response to salary in cohort D, whereas levels have increased for self-employment across both employment intensities. Whilst it is difficult to draw direct comparisons, it may be that the hover text has had some impact on the levels of non-response. This is backed further when considering the CATI completion mode, which saw an increase in non-response across all employment types and intensities. Interestingly, confidentiality reassurances can have different impacts on respondents and reassurances can either increase divulgence or reduce it if its inclusion raises privacy concerns that weren’t present previously (Acquisti, Brandimarte and Loewenstein, 2015).

Feedback from interviewers suggests that the extent to which this text helps reassure varies by respondent and their level of suspicion or reticence to provide personal information. While the text may help convince some to share the information, it has little impact on others, which in their experience, is fairly typical of reassurances. They have not encountered cases where the reassurance text has led to additional concern or reservations, so while the positive impact the hover text can have varies, they are confident it is not having a negative impact. This is supported by the finding that item non-response at salary is not causing an increase in unit non-response. Nevertheless, high item non-response on CATI is concerning and HESA is exploring other possible explanations with the contact centre to understand the causes and implement mitigation measures, where possible.

As hover texts were only introduced at the end of cohort D and they appear to have some positive impact on item non-response they have been retained during year 4 and this evaluation will be revisited once 4 cohorts worth of data has been collected (i.e. at the end of cohort C, year 4).

Distribution of responses received to salary

Another indication of data quality improvements in relation to salary may be reductions in salaries outside the ‘expected’ range. Previous changes to the question aimed to reduce confusion that may have been causing some graduates to provide one-digit or two-digit salaries, but whilst this seemed effective some particularly low or high salaries remained. Though this cannot be avoided fully, and some may be genuine responses, it is likely that some of these responses are a result of graduates feeling reluctant to provide a genuine response to this question due to the sensitive nature of the question, perhaps leading to measurement error (Tourangeau and Yan, 2007). To aid in determining improvements in the online salary provision as a result of the addition of hover text, distributions of salaries provided in United Kingdom Pounds are split into broad salary groupings for quality analysis purposes and are shown in Figure 1. 

As a broad overview, an increase in provision of salaries of more than £15,000 to £100,000 may be useful in determining a potential improvement in quality for the full-time salary provision in particular. This increase can be seen across all employment types, not only for full-time, but also part-time graduates. Provision of salaries of £0 has decreased across most, with the exception of graduates in full-time paid work, which have a very small increase which does not seem concerning. This is a positive sign that some graduates may have felt more comfortable providing a salary in year three. Furthermore, for full-time graduates across all employment types there was also a reduction in graduates providing salaries for both the under £100 and £100-£15,000 groups. Whilst it must be noted that there are a number of factors that will influence these salary groupings it does seem to be a positive indication that provision of salaries may have improved. Some of these employment groups, particularly the contracted to start work group, can have smaller numbers of graduates associated with it which should be kept in mind, particularly when considering groups with smaller percentages.  This is the case for the over £100,000 group, which generally makes up a small percentage of selection. Whilst it appears to have increased for many of these groups, numbers are small and the levels of provision in this group do not seem to have increased at a concerning rate. 

Conclusions

The introduction of hover text across the job title, salary and employer name questions does not appear to have had a large impact, although some smaller changes are visible in responses to the relevant questions. When comparing impacts on non-response, for example, some groups have seen increases whilst others have dropped. Job title predominantly saw an increase, whereas employer name groups tended to see a decrease in item non-response. As the questions hover text was added to allow graduates to enter their answers as either any numeric response for salary or any text response for job title or employer name the layout or quality of responses is also an important consideration. Salary item non-response comparisons were difficult due to changes in the questions, but the layout of the responses provided did seem to improve in many cases. Equally, the percentage of SOC records which were uncodable did not seem to increase a concerning amount, which is a positive indicator that job title and employer name response quality did not decrease significantly. It is also important to consider the fact that hover text may be seen multiple times in the survey, and from comparisons of year one and two when nothing had changed it is obvious that there are other influences on item non-response. As changes are often small, it is therefore difficult to attribute them to the hover text. Effects were also different depending on the mode of completion. The CATI completion mode was a useful comparison in considering whether changes online may have been solely due to other factors, but it was also useful to consider whether CATI responses had been impacted by hover text. However, as CATI relied on interviewers choosing to read out the text, compared to the online mode in which all graduates had the opportunity to see it if they wanted, it was expected that effects may be smaller. The hover text will continue to be assessed in year four of the survey to determine whether it should remain in the survey. 

Next: Graduate voice data

References 

Acquisti, A., Brandimarte, L. and Loewenstein, G., 2015. Privacy and human behavior in the age of information. Science, 347(6221), pp.509-514.

DeLeeuw, E.D., 2018, August. Mixed-mode: Past, present, and future. In Survey Research Methods (Vol. 12, No. 2, pp. 75-89).

Ong, A.D. and Weiss, D.J., 2000. The impact of anonymity on responses to sensitive questions 1. Journal of Applied Social Psychology, 30(8), pp.1691-1708.

Tourangeau, R., & Smith, T. W., 1996, Asking Sensitive Questions: The Impact of Data Collection Mode, Question Format, and Question Context. The Public Opinion Quarterly, 60(2), 275–304. http://www.jstor.org/stable/2749691

Tourangeau, R. and Yan, T., 2007. Sensitive questions in surveys. Psychological bulletin, 133(5), p.859.