Activity and employment assurance

Introduction

The quality of survey data is a high priority for Graduate Outcomes. At present, there is an ongoing data quality assurance plan being undertaken, with the aim of assessing quality across all survey questions. In the following section, we will present some of the preliminary research that was undertaken on two of the survey sections. This includes the first section in the survey, which includes the questions regarding the activities undertaken by the graduate during census week, and will also cover some of the second section, which is answered by respondents in paid or voluntary/unpaid work for an employer.  This analysis was designed to highlight some of the lines of investigation we may want to take to assess data quality across the Graduate Outcomes survey.

Section A- Activity data

All graduates are asked which activities they were in during the census week, as this is the first question in the survey. The answers provided in this section will determine the route the graduate takes later in the survey, and whether they answer any of the other questions in Section A. This analysis aims to highlight issues such as high item non-response or incorrect entries in the data and aims to identify further areas of investigation.

Individual response rates to the questions in Section A are highlighted in Table 1. Please note that as the ‘all activities’ question is the first in the survey, this will always have a 100% response rate when considering item non-response and drop-out levels of those respondents who started the survey.

Table 1: Response rates to the Section A questions, split by completion mode and year

As can be seen, the response levels to questions in Section A are generally good and all are above 95% response to the question. When looking at separate completion modes, all are performing on a similar level with only Home Country having lower response rates in online completion modes when compared to CATI. Comparison to the previous year shows an overall slight increase or similar levels of response.

All activities

Proportions of graduates selecting each activity option were split by completion mode. The proportions show the percentages of graduates that have selected each activity option and as such could include the same graduate if they selected more than one option (Table 2/Figure 1). For example, a graduate selecting ‘Paid work for an employer’ and ‘Engaged in a course of study, training or Research’ would be counted towards both. Levels are based on data that has not yet been processed for output purposes.

Table 2: ‘All activity’ proportions sorted in descending order. The proportions can add up to more than 100% as graduates can select more than one activity.

Trends show differences between CATI and online survey modes with almost all activity options having higher proportions in the CATI mode when compared to the online mode. Top selections for both the CATI and online modes are “Paid work for an employer” and “Engaged in a course of study, training or research” with the third option differing between the modes (“Developing a creative, artistic or professional portfolio” for CATI and “Unemployed and looking for work” for online).

To investigate the higher proportions of activity selections seen in CATI, the proportions of the number of activities selected by graduates were assessed (Figure 2/Table 2).

Table 3: Proportions of number of activities selected by graduates by completion mode.

By splitting the data by the selected number of activities it was revealed that graduates answering the survey using CATI were more likely to select more than one activity when compared to the online mode. The data provides an important insight into the differences in graduate behaviour between the two completion modes. It suggests that having an interviewer read through all activity options can lead to the graduate providing more information when compared to online mode where graduates tend to select just one option. This effect has also been observed in the previous year with almost identical proportions. Such mode effect has been observed in a number of studies such as Molenberghs et al. (2010). To further investigate the activity selections and the differences between the completion modes, we looked at the most common activity combinations for each completion mode.

Table 4: Most commonly selected activity combinations split by completion mode. For full details of activities, visit the coding manual.

As can be seen from the activity combinations (Table 4), when more than one activity was selected, the most common combinations are almost identical across the two completion modes. Combined with the previous activity data, this leads to a conclusion that the disparity between the CATI and online completion modes is mainly related to the graduates in the online mode having a tendency to only select one option. Unfortunately, it is rather challenging to overcome this effect. Graduates using the online mode are given all the options on one screen and (according to the above data) tend to ignore other options after selecting their main one. Some solutions have been suggested such as randomising the order of the options presented on the screen which would force the graduates to go through the whole list. However, this introduces additional issues. The list of activity options follows a rational order starting with “Paid work for an employer” and ending with “Unemployed and looking for work” and “Doing something else”. Randomising this order might lead to graduates getting confused and dropping out of the survey. Another possibility is to present each activity on a separate screen. However, this lengthens the survey and can lead to survey fatigue (Weitzer et al., 2004). Nevertheless, we are continuing to work on this question. This is an area that has been highlighted to investigate further and it is also explored further in the next section, which may indicate that on CATI graduates are more likely to be selecting more than one activity which relates to the same role.

Main activity

To investigate the activity selections further, and differences between the completion modes, the most important activity selection were analysed (see Figure 3).

Whilst proportions of initial activity selections are similar but do differ by completion mode, analysis of most important activity highlights very similar proportions across the activities, regardless of completion mode. This reassures us that although graduates may be more likely to select multiple activities, there is not a significant issue with under recording the main activity of the graduate. To investigate this further, activity check is analysed below, which indicates whether the activity was the same or different.

If graduates select “Paid work for an employer” or “Voluntary/unpaid work for an employer” and “Self-employment/freelancing” or “Running my own business” or “Developing a creative, artistic or professional portfolio” they are asked the same activity question. It asks them to clarify whether these activities were the same, or whether they were two separate activities. 

Splitting the data by completion mode revealed an interesting insight (see Figure 4). Graduates in the Online mode were selecting the "Yes these activities are different" option significantly more often than in CATI.  The main explanation for this may relate to the previously discussed data showing that graduates are more likely to select more than one activity option on CATI. It may be that these additional activities are not separate, rather graduates on CATI are more likely to select multiple options that relate to the same activity. It may also be related to the wording of the question and (similarly to the activity question) the possible input from the interviewer. In this case, graduates on the CATI mode may be able to ask for a clarification regarding the question. As it can be seen in the chart, the difference in proportions between "Yes these activities are different" and "No these activities were the same" answers in CATI is significantly lower when compared to the Online mode. While this may indicate that the questions need to be looked at in more detail to ensure the meaning is clear, it also indicates there may not be a loss of granularity in the online mode. 

Further investigation was done by looking at the most common activity combinations split by the activity check answer and completion mode (Table 5), to investigate whether this revealed anything further about the differences between the completion modes.

Table 5

This investigation has revealed a couple of important points. First of all, graduates were selecting the same activity pairs regardless of the activity check option (“Paid work for an employer” and “Developing a creative, artistic or professional portfolio” in CATI and “Paid work for an employer” and “Self-employment/freelancing” in the online mode). This may support the earlier mentioned explanation that graduates in the online mode might have a difficulty understanding the question and are not able to get it clarified as in CATI.  Secondly, the most common same activity combinations differ between the two modes. While the “Paid work for an employer” is present in both modes, the second option is “Developing a creative, artistic or professional portfolio” in CATI and “Self-employment/freelancing” in online completion modes highlighting the difference between the selections by mode. Lastly, Portfolio activity is being selected more commonly in CATI compared to online completion mode which can be seen in Table 2. This, combined with the activity check data might suggest that graduates are more likely to feel they should select more than one activity on CATI, even if this is the same. Equally, it may be linked to the fact that they listen to all options rather than rushing through the question on the CATI mode. They may also be able to get clarification regarding the portfolio option and therefore feel that they are able to select that this also relates to the other activity they are undertaking. Research has shown that receiving clarification regarding survey questions increases response accuracy (e.g. Conrad & Schober, 2000; Schober & Conrad, 1997). In the same activity, or the portfolio case, it is possible that the description is not clear enough and further clarity may benefit the questions which will be investigated further.

Section B- Employment data

This following section presents some of the analysis that has been undertaken in relation to the second section of the survey, known as ‘Section B’, which graduates may be routed to if they answer that they are in paid work for an employer, or voluntary/unpaid work for an employer. It is one of the most common routes taken by respondents, and this assessment aims to provide an initial view of data quality and to identify certain issues such as high item non-response and missing data, or incorrect data entries. Additionally, the research also aims to identify further areas of investigation, as well as analysis that may benefit from linking to other sections in the survey.

Initially, response levels for five of the questions from Section B are presented in Table 6,split by completion modes and completion year. These response rates will aid in identifying questions with high item non-response and may aid in identifying further lines of investigation. The data presented here only includes graduates who were in paid or unpaid/voluntary work for an employer.

Table 6: Response rates for Year 3 and Year 4 for the first five questions in Section B split by completion modes. The response rates only include graduates who were in paid or unpaid/voluntary work for an employer.

Generally, response levels to questions in the Graduate Outcomes survey are good and as you can see here, all are above 90% response to the question. Sensitive questions can be more susceptible to completion mode effects and reduced level of response, but some further research that has been done on this topic this year can be found in the section on Reliability of Sensitive Data. The levels of response will be discussed within the relevant sections for each question, including the difference in response across completion modes to job title and job duties.

Multiple Jobs (same activity)

Graduates who were only in paid or unpaid/voluntary work for an employer and answered main activity (not including information copied over from same activity) were asked if they were working in one or more than one job during the census week. Looking at the response rates (see Table 6) it can be seen that they were high (>99%) in all the completion modes. When comparing to the previous year, response rates have increased slightly in CATI and mobile completion modes, with a slight decrease in the desktop mode. 

Table 7: Proportions for the multiple jobs question.

When looking at the proportions of jobs held (one versus more than one), the data is similar between the two years with majority of students (>92%) indicating that they have one job with a slight increase in Year 4.

Employment Intensity

After being asked how many jobs graduates were working in during the census week, the next question was regarding the intensity of the job (full-time or part-time). As in the multiple job question, response rates were high (>98%) in all of the completion modes. As previously, highest response rates were in CATI, followed by desktop, with the lowest response rates being in the mobile completion modes. When looking at the differences between the years, the patterns follow the previous multiple job question, with rates slightly increasing in CATI and mobile with a slight decrease in desktop completion modes.

Table 8: Full time and Part time proportions split by completion mode.

Looking at proportions (see Table 8), there have been some fluctuations between years when split by mode. It would be useful to investigate this further split by work type to understand if these differences are likely to be a result of genuine selection or other possible effects.  

Job title and job duties

Job title and job duties have the potential to be asked at two points in the survey, depending on the activities selected and the route taken by the graduate. They are mandatory questions that require graduates to provide their job title during the relevant census week for their cohort. Responses are entered in a free-text field, so it is possible for graduates to provide a response that is not accurate, for example by editing their job title or entering random characters to bypass the question due to influences such as privacy concerns or social desirability bias. Job title has previously been identified as a question that some respondents may view as sensitive, and as a result hover text was added to reassure graduates in cohort D of year 3. Previous research was completed on these questions in the 3^rd edition of the data quality report, which considered these in the Reliability of Sensitive Data section. It is therefore useful to reassess the item non-response levels to these questions to see if this has improved during the fourth year of the survey. 

As can be seen in Table 6, the item non-response levels for both of these questions reduced slightly in Year 4 compared to Year 3 in the CATI and mobile completion modes. However, there was a slight increase in item non-response rates in the desktop mode, although rates of response were still higher than on the mobile completion mode. The only change to the questions was the newly introduced hover text explaining that the job title and job duties information will not be used to identify individuals and no attempts will be made to contact their employers. However, it is difficult to say if this addition was what affected the drop in response on the desktop code. First of all, the hover text was added in cohort D (last cohort of the year) of Year 3. Secondly, this effect was not observed in the mobile completion mode and in fact the non-response rates decreased. It is important to note that the overall non-response rates are higher in online completion modes when compared to CATI. This could be due to the sensitivity of these questions and the effect of an interviewer in CATI (Conrad & Schober, 2000; Schober & Conrad, 1997) making graduates more likely to provide information. As such, more time and investigation is needed to explore this effect and its causes.  

Impact of hover text on SOC coding 

Graduate Outcomes employment data is coded using the Office for National Statistics’ (ONS) Standard Occupational Classification (SOC). Both the job title and the employer’s name provided by the graduate are considered in the coding of records. In some cases, a code cannot be assigned to a graduate, for example if there are no appropriate codes present in the indexes or if information is missing in the survey data returned by the graduate, and in these cases the records are assigned a code of 0001 to indicate that they are uncodable. Although there are a number of factors that can influence coding it could be useful to assess the prevalence of these uncodables in the dataset to ensure that the hover text in job title and employer name did not have a negative impact on the ability to code records. Levels of uncodables are stable this year.

Discussion

The preliminary analysis shows that the activity section (Section A) generally receives good response levels, with all of them being above 95% and with separate completion modes all performing on a similar level. When comparing to the previous year (Year 3), Year 4 shows a slight overall increase or similar levels of response rates to the questions.

The analysis of activity data provides an important insight into graduate behaviour with graduates in online completion modes leaning towards selecting just one activity option when compared to CATI. This effect has also been observed in Year 3. However further analyses when more than one activity was selected indicated that the most common combinations are almost identical across the two completion modes, leading to a conclusion that the disparity between the CATI and online completion modes is mainly related to the graduates in the online mode having a tendency to only select one option. Solutions have been discussed such as randomising the order of the options presented on the screen or presenting the options on separate screens, however that would introduce further problems such as confusion or survey fatigue. As such this area been highlighted for further investigation.

A similar issue was also found in the same/different activity question with graduates in the online mode selecting the "Yes these activities are different" option significantly more often than in CATI.  The main explanation for this may relate to the previously discussed effect of graduates in CATI more often selecting more than one activity option, which may in fact relate to the same role. Additionally, it was discussed that this effect might be related to the wording of the question and the possible input from the interviewer. In this case, graduates on the CATI mode may be able to ask for a clarification regarding the question. While the overall data showed that there might not be any loss of granularity in the online mode, further research is needed into the question.

Investigation into activity pairs revealed that graduates were selecting the same similar activity pairs regardless of the activity check in both completion modes with “Paid work for an employer” being in both but “Developing a creative, artistic or professional portfolio” being more common in CATI. It has been discussed this might be due to graduates in CATI being able to get clarification regarding the portfolio option and therefore feeling more informed. It is possible that the description is not clear enough and further clarity may benefit the questions. However, it may also be related to the fact that graduates select more than one option more frequently on the CATI mode, even if this is the same activity, and that portfolio is the more common selection for this.

With regards to the employment investigation (Section B) so far, response rates are generally looking very positive. For multiple jobs, employment intensity and employment basis, response levels across all completion modes were above 98%. Initial investigations into these questions indicated no major concerns but highlighted other areas to investigate. For example, further analysis split by work type would be useful for some of the questions, such as employment intensity.

Job title and job duties have been assessed in previous years as sensitive questions, with the addition of hover text to reassure graduates in cohort D of Year 3. The online completion mode tends to see lower levels of response for these questions. However, the mobile completion mode and the CATI completion mode have both seen increased response this year. On the other hand, the desktop mode has seen a slight reduction in response, although response rates are still higher than on mobile. An assessment of SOC coding indicated that levels of uncodable records are stable, which is a reassuring indication that the quality of responses being provided has not deteriorated greatly. Continued assessment would be useful here in order to identify if there are any further changes that may aid in reducing item non-response and drop-out. It may be useful to repeat some of the free-text analysis which has been performed on the question in previous years to support the uncodable analysis.

The assessment is ongoing and will continue to investigate the quality of the survey questions. While there are avenues of research that need to be pursued to improve some of the questions, overall Section A provides good initial data on graduate activities with improving response rates and Section B response levels look good with some further work required on the online mode for job title and job duties.

Next: Location data

References

Conrad, F. G., & Schober, M. F. (2000). Clarifying question meaning in a household telephone survey. Public opinion quarterly, 64(1), 1-28.

Porter, S. R., Whitcomb, M. E., & Weitzer, W. H. (2004). Multiple surveys of students and survey fatigue. New directions for institutional research, 2004(121), 63-73.

Schober, M. F., & Conrad, F. G. (1997). Does conversational interviewing reduce survey measurement error?. Public opinion quarterly, 576-602.

Vannieuwenhuyze, J., Loosveldt, G., & Molenberghs, G. (2010). A method for evaluating mode effects in mixed-mode surveys. Public opinion quarterly, 74(5), 1027-1045.