PIs: Changes to postcode indicator and overview of changes 2002/03
- Summary of changes (2002/03 onwards)
- Changes to the postcode indicator (2005/06)
- Introduction of tariff (2002/03)
- Overview of changes between 2001/02 and 2002/03 (2002/03)
Changes to the postcode indicator
From 2005/06, there was quite a major change in the way in which the low participation data has been produced in the Widening Participation Indicators.
Introduction
Following the recommendations of the PI review, the Performance Indicators Steering Group (PISG) has agreed that the postcode indicator should be replaced from this year. The new indicator is based on the revised POLAR definitions of low participation areas, using the lowest quintile of wards as low participation. Further explanation and details of the quintiles used, and their relationship to postcodes, can be found at http://www.hefce.ac.uk/widen/polar/polar2/ (see the CSV file at the bottom of that page for the lookup table).
A POLAR2 low participation location adjusted benchmark has been provided for all institutions. For Scottish institutions, it is recommended that the location adjusted benchmark is used in preference over the raw benchmark, as location adjustment takes account of variation in participation rates between the different Home Nations, and in particular the higher participation rate in Scotland. See discussion below.
Discussion
The PI review noted that the existing postcode indicator was based on an old geodemographic classifier, and proposed two replacements. One was to be based on the Index of Multiple Deprivation (IMD), and the other on the POLAR2 definition of low participation areas.
There was general agreement that a replacement was needed, and support for both of the proposals was expressed. However, it was clear that further work on the detail was needed. One of the difficulties with the IMD was the difficulty of comparison across the countries of the UK, with definitions used for the index in one country being slightly different from those in another. In the end, it was accepted that these difficulties could not be overcome in the short term, although PISG are looking for an alternative measure that would be acceptable.
It has therefore been agreed that initially there should be just one replacement indicator, based on the POLAR2 definition. For this indicator, which is applied to young and mature, full-time and part-time entrants, a 2001 Census Area Statistics ward is defined as low participation if its participation rate places it in the bottom 20 per cent of wards ranked by this measure.
The value of this new indicator (POLAR2) for young full-time first degree entrants across the sector is 9.0 percent, rather less than the value for the old indicator (Super Profiles) of 14.7 percent (14.0 percent in 2005/06). The benchmarks are calculated as before, and take account of this reduced average value.
Differences
For the majority of institutions, the change does not adversely affect the indicator. As with the other indicators, there will be fluctuations at institutions year on year, but it is not anticipated that these will be any greater than previously.
Scottish institutions
The one group which is affected is the set of Scottish institutions. The revised participation rate is higher in Scotland than elsewhere in the UK, and so the number of students from low participation areas in Scotland is lower than in the rest of the UK. The percentage of students from low participation areas at Scottish institutions has fallen substantially (from 18.5 percent in 2006/07 using Super Profiles to 3.2 percent in 2006/07 using POLAR2). The benchmark shows that most of the institutions in Scotland are significantly below the UK average, however the location-adjusted benchmark allows for this between-country difference.
There are two main reasons why this participation-based classification of areas in Scotland differs substantially from the classification used previously for the performance indicators. Firstly, rather than use a geodemographic classifier, the new method uses Census Area Statistics wards. Previously, the classifier clustered together very small areas (enumeration districts or data zones) that were judged to be similar in terms of the values of a range of census 1991 variables. The resulting 160 GB clusters each then contained a large enough population for which to calculate participation rates, and it was on the basis of these calculated rates that the designation as low participation or not was made. The disadvantage of this was that areas within a cluster were not necessarily close together, so many of the clusters formed using Scottish data zones also included numbers of English enumeration districts. This meant that the high participation rates in many of the poorer areas of Scotland (high relative to those in England) were ‘diluted’ by those English areas. Now that the more numerous and smaller wards are being used as the units for calculating participation rates, this dilution no longer occurs, and the high participation rates in Scotland are correctly recorded.
The second important reason for the changes is that the measure of participation is much improved from what we had available when the Performance Indicators were first published in 1999. In particular the new participation measure includes HE that takes place in FE colleges and is returned on non-HESA records. Previous work by HEFCE (2005/03) showed that this type of participation in FECs is much more prevalent in Scotland than in England and Wales, particular so for poorer areas (HEFCE 2005/03, page 42).
Both these effects act to substantially reduce the proportion of areas in Scotland that are classified as having low participation relative to the rest of the UK. In turn this has acted to lower the low participation area performance indicator for any institution that has substantial recruitment from Scotland.
More information on the localised effects
Under certain conditions the location of an institution can have an impact on the low participation neighbourhood indicator, making it appear different from the other widening participation indicators. In particular, there are several characteristics which have an impact on institutions in London and Scotland.
In London, institutions tend to recruit a high proportion of students from this area. The participation rate overall is higher in London than for most other parts of the country. These factors taken together mean that areas in London may be less likely than similar areas elsewhere to be classed as low participation. As a result, institutions in London tend to have a lower proportion of students from low participation neighbourhoods relative to their benchmarks.
In Scotland, again institutions tend to recruit a high proportion of students from the local area. The old low participation indicator clustered together very small areas (enumeration districts or data zones) that were judged to be similar in terms of the values of a range of census 1991 variables. The resulting 160 GB clusters each then contained a large enough population for which to calculate participation rates, and it was on the basis of these calculated rates that the designation as low participation or not was made. The disadvantage of this was that areas within a cluster were not necessarily close together, so many of the clusters formed using Scottish data zones also included numbers of English enumeration districts. This meant that the high participation rates in many of the poorer areas of Scotland (high relative to those in England) were ‘diluted’ by those English areas. The POLAR2 low particiaption method uses the more numerous and smaller Census Area Statistics wards as the units for calculating participation rates so this dilution no longer occurs and the high participation rates in Scotland are correctly recorded. The participation measure also includes HE that takes place in FE colleges and is returned on non-HESA records. Previous work by HEFCE (2005/03) showed that this type of participation in FECs is much more prevalent in Scotland than in England and Wales, particular so for poorer areas (HEFCE 2005/03, page 42). Both these effects act to substantially reduce the proportion of areas in Scotland that are classified as having low participation relative to the rest of the UK. In turn this has acted to lower the low participation area performance indicator for any institution that has substantial recruitment from Scotland. For this reason, low participation data for Scottish Institutions have been excluded from the Performance Indicators.
Measuring effects of locality
Supplementary Table SP1 shows the percentages of young entrants from each of the regions of the UK who come from low participation neighbourhoods (POLAR2); NS-SEC Classes 4, 5, 6 and 7; and state schools. The scale of the differences between regions means that institutions which recruit most of their students locally may find they have characteristics quite different from the national average.
Because of these differences, we have looked at ways in which a student’s domicile could be incorporated into the existing benchmarks of the widening participation indicators. Using the same methodology as is used for the current benchmarks, and taking the student’s region of origin as another factor, we have produced a value that will give an indication of how important the location factor is. This is the location-adjusted benchmark.
For institutions which recruit from across the UK, there is very little difference between the standard benchmark and the location-adjusted benchmark. Institutions which recruit more locally will have larger differences, possibly 3 or 4%, between the original and the location-adjusted benchmark. These larger differences show that the indicator is affected by the characteristic of the area the institution recruits from. In general, the greatest differences occur for the low participation indicator, and the smallest for the NS-SEC indicator.
Questions
In considering how best to measure locality effects, a major concern was raised. By allowing for the effects of locality, there is a danger that what we are trying to measure could be partly obscured. Differences between geographical areas may be caused by disparities between institutions, or these disparities may be the result of geographical differences. Until we have resolved this circularity we need to be careful in making allowances for geographical effects.
There is a further difficulty with the method used. In theory, if an institution situated in a region of low participation were to recruit predominantly from another region of high participation, that institution’s benchmark would not reflect its locality. Rather, it would reflect the locality from which its students were recruited. In practice that is unlikely to happen, partly because we have used region rather than some smaller geographical area as the basis.
The location-adjusted benchmark has only been used with the participation indicators, because of the known differences in the way these groups are spread across the country. They have not been used with the indicators of retention or non-continuation, nor is there any plan to do so, for two reasons. The major reason is that to include location as a factor in non-continuation would imply that people from different regions could have different continuation rates, even taking into account their subject of study and their entry qualifications. This would not be acceptable. A further reason is that the differences between the non-continuation rates for students from different regions is small. A location-adjusted benchmark for these indicators would therefore not provide any extra information.
Introduction of tariff
When the tariff score was first introduced, the Performance Indicators Steering Group (PISG) agreed that the groupings to be used in calculating the PI benchmarks should be reviewed after two or three years, once sufficient data were available. However, the effect on the benchmarks for the state school indicators in particular was such that an earlier review was called for.
The review was carried out during this year, and the following paragraphs report the results of this review.
Review of tariff groupings for benchmark calculations
In calculating the benchmarks, students are divided into categories according to what subject they are studying and what qualifications they had on entry to their course. As most full-time students enter with A-levels or Scottish Highers, these are further sub-divided into groups according to the overall scores they obtained. Prior to 2002, the method of scoring was based on the best three qualifications obtained, producing a points score whose maximum value was 30. From 2002, the new tariff system was introduced, superseding the old system which was then no longer available. The tariff counts all qualifications obtained, and is calculated in a different way. It is not possible to obtain the old points score from the new tariff score, and so the tariff system has had to replace the points score in the calculations of the benchmark. This tariff system is not under the control of either HESA or HEFCE.There are two main differences between the new tariff scores and the A-level points previously used. First, the relativities between different A-level grades and between A-levels and Highers are not the same for the tariff system as for the former points scores system. Secondly, and probably more importantly for this review, the cap on the number of examinations that could be included was lifted for the tariff. Under the old scheme, the top score was obtained by anyone who had at least three A grades at A-level. Under the new scheme there is no maximum score, so someone with exactly three A grades at A-level will have a lower score than someone with four B grades. This is further complicated by the fact that AS-level grades for those subjects that are not taken on to A-level are also included in the total tariff score.
If individual examination scores were available, then it would be relatively straightforward to either replace the cap, or produce a set of groups that took the individual values into account. However, the scores are recorded as a total for each qualification type – the total score for A-levels and the number of A-levels, the total score for AS-levels and the number of AS-levels, etc. In addition, to obtain the total tariff score it is not possible to sum the totals for the different groups, as there may be duplicate subjects between the groups.
A number of different options were examined. Firstly, the possibility of omitting the tariff score from the benchmark altogether was suggested. Entry qualifications were identified in the early stages of developing the PIs as an important factor contributing to differences between institutions, therefore it was agreed that this suggestion should not be followed.
Secondly, it was suggested that only the tariff score for A-levels and Highers were used, instead of the total tariff. Using only the A-level scores would be technically feasible, but would change the benchmark of those institutions that are not so selective, and would not necessarily change things significantly for the most selective institutions. In addition, trying to incorporate the Highers score as well is difficult, as (a) there are two categories of Highers to be taken into account, reported separately and whose score cannot necessarily be combined; and (b) there are a number of students with both A-levels and Highers, again with scores that cannot necessarily be combined.
Thirdly, it was suggested that a group be defined to contain only those students who have at least three A grades at A-level, or five A grades in Scottish Highers. This is feasible, but has the perverse effect that while the most selective institutions showed a drop in their benchmarks, as expected, those institutions which generally selected from groups just below the three A grades, e.g. asking for two A grades and one B, showed an increase in benchmark.
Conclusions
In view of this, it was agreed by PISG that the current grouping of tariff scores would continue until a further detailed review could be carried out.Further to this it was noted that, while the tariff score is the only way at present in which entry qualifications can be included in the benchmarks, PISG should look at what more information it should request HESA to collect to be more useful in this regard. The outcome of any review by PISG of how the benchmarks are calculated may require alterations to the specification of the dataset collected by HESA. Such changes to the data collection are subject to a record view and consultation process which takes a number of years to implement. It is accepted that the tariff does not provide the full range of information which could allow better differentiation between certain types of institution, particularly those that are highly selective, and PISG will work with all institutions to see how this information can be improved.
Notes
- As institutions become more different in nature, making comparisons between them within the Performance Indicators becomes less valid. Therefore only comparisons between institutions of a similar nature should be made.
- View full information on how the benchmarks are calculated.
Changes between 2001/02 and 2002/03
In 2002/03, there were a number of changes to the data collected, to some of the definitions and to a number of the calculations used. They are brought together here for convenience.
Data changes
The HESA record was modified for the 2002/03 academic year and some of these changes have affected the performance indicators (PIs). In addition, some of the data formerly taken from UCAS records is now being taken from the HESA record.
Two variables, the previous school of the student and the social categorisation of their parents, have been obtained from UCAS records in the past. For 2002/03, UCAS supplied both these fields to institutions, who checked this information and sent it on to HESA. In some cases, institutions added extra information to that supplied by UCAS. The main effect for the performance indicators has been to change the proportion of known data available.
The other fields on the HESA record which changed this year and that are having the most effect on the indicators, are those providing information on entry qualifications and the subject fields. The entry qualification scoring system used in 2002/03 is the tariff system rather than the points scores used previously. Also the subject codes now being used are the JACS codes, agreed as a common coding system for use from 2002 entrants in higher education. Details of the effects of these changes are provided below. Note that these changes do not affect the benchmarks for the indicators of non-continuation or progression, which this year are based on 2001/02 entrants. The UCAS tariff page provides overview of its use within the benchmark calculations.
Definitional changes
Social Class
For the 2001 census, a new classification, National Statistics - Socio-Economic Classification (NS-SEC), was developed to replace Social Class. It took into account new work patterns in the UK and the changes in education levels required for and the status of, large numbers of occupations. This new classification was used for the social class PI this year and as a result this will now be called the SEC indicator.
In previous years, the six categories of Social Class were combined by taking classes I, II and IIIn as 'high' social class and classes IIIm, IV and V as 'low' social class. The new classification has seven analytic classes and groups 1 to 3 are used as 'high' class and 4 to 7 as 'low'. This has increased the overall percentage from low social class by over 2.5%.
In order to see how much of this change is due to the different definition, the mapping published by ONS has been used to obtain the NS-SEC categories from the occupation codes used in the 2001 HESA data collection. The data available on the HESA record were not complete for all institutions in 2001, but provide some indication of the sort of value that this indicator would take. The results of this analysis show that the value of the indicator based on 2001 data but using the new definition would have been 27.8%. The following table compares this with the UK values published last year and this year.
| Percent from low social class in 2001, based on old definition (published) | 25.8 |
| Percent from low social class in 2001, based on new definition | 27.8 |
| Percent from low social class in 2002, based on new definition (in this document) | 28.4 |
This suggests that the increase since last year is mainly due to the change in definition.
Subject codes
The new subject categories are very similar to the old ones and this year the same groupings of categories have been used for the benchmarks as in previous years. There appears to be little effect on the benchmarks.
Entry qualifications
The introduction of the tariff score in the sector has led to changes to the benchmark groupings. Most of the change is to the groupings of scores for A-levels and Scottish Highers, but in addition there is a new category for Baccalaureate (formerly included with up to 4 A-level points) and a category for students with both Vocational A-levels (VCE) and A-levels or Highers. The GNVQ level 3 category now contains students who have tariff scores just for VCE qualifications.
The tariff score categories used have been chosen so that as far as possible they are of equal size. These categories will be used for two years and then a decision will be taken as to whether these are the most sensible categories to use. The main problem this year is that there are much larger numbers of students than usual who have A-level or Higher qualifications for whom there is no tariff score – over 25,000 this year among young entrants compared with under 3,000 last year. This may be a temporary problem, to do with deferred entry students, but it is necessary to obtain at least one more year's figures before the pattern can be clarified.
The tariff score, unlike the previous points score, does not put a cap on the number of qualifications that can be included, nor does it set a maximum value for the score. The effect for benchmark purposes appears to be that the high tariff score categories contain a greater proportion of students from the 'under-represented groups' than the 30 A-level point category did last year. This is particularly apparent for the state school indicator. For example, last year 68% of students with 30 A-level points came from state schools, while this year 77% of those with over 480 tariff points came from state schools. It is possible that this is a temporary blip for the same reason that the proportion of A-levels with unknown tariff scores has increased, but again until there are at least one further year's figures we cannot be sure. However, the effect this year has been to increase the benchmarks, particularly the state school benchmark, for institutions which admit high proportions of students with high A-level scores.
Changes to calculations
Allocation of students to subjects
In previous years, a student with more than one subject of qualification aim was allocated to just one subject group, often 'Combined Studies'. This year, students have been allocated pro-rata to subject groups according to how many subjects they are studying and whether they are doing a balanced combination of subjects or a major/minor split. So, for example a student doing a balanced combination of subject X and subject Y would be allocated half to X and half to Y. This has affected some subject groups more than others and in particular has reduced the number of students allocated to the 'Combined Studies' group from about 25,000 to just under 2,000.
It is not straightforward to separate the effects of this change from those of other changes mentioned above. For example, the 'Combined Studies' group shows much higher percentages from all three under-represented groups than last year, but this could be due either to the institutions where this subject is defined as such, or to the effects of the tariff scores, as well as to the reduction in size of the group, or indeed some other effects.
Linking files for progression
Until last year, the linking method used for tables T3, T4 and T5 took no account of the student instance (HIN) link introduced in 1998. Last year, table T3 used a revised linking method that used the HIN as a basis and this year the revised method has been extended for tables T4 and T5 as well.
A further change has been made to the way links, once found, are categorised; by treating all records with a zero value for student FTE as inactive, unless the academic year type is not standard.
The effect of these changes on the national figures is quite small. Some institutions show changes that appear relatively large, but these are mainly the small institutions where relatively large changes are common. Nevertheless, it should not be assumed that changes between the years are simply due to an improvement or otherwise in the performance of the institution.
Calculation of projected outcomes
In addition to the change in linking methods, the states used in calculating the projected outcomes have been re-defined this year. The new states used in the transition matrix are defined in Annex C.
The calculations using the transition matrix are the same as last year, although the matrix itself is slightly smaller. Again, the effects at the national level are small and for most institutions there is no discernible effect.
Efficiency
Last year, it was suggested that the efficiency figure should be dropped from table T5, but a number of institutions felt that it was a useful addition to the other statistics available, particularly as it could be compared across the years. However, in view of the other changes being introduced this year it was decided that the efficiency calculations could no longer provide any continuity with previous years and so it was agreed to drop them.
Employment indicator
From 2002/03, the employment indicator was based on the Destination of Leavers from Higher Education (DLHE) survey which replaced the First Destinations Supplement. Prior to 2002/03, the First Destinations indicators were published by HEFCE in table E1.
The DLHE indicator follows the standard categories for publication and is defined as the number of respondents working or studying (or both) divided by the number of respondents working or studying or seeking work. All other categories are excluded from this indicator. It was agreed by the Performance Indicators Steering Group (PISG) that there should be no second indicator from 2002/03.
The benchmarks prior to 2002/03 were calculated using a model-based approach. The model was a multi-level model containing eight student level factors and three institution level factors. It was decided that this approach is no longer necessary and that a simpler method would be more suitable. In part this is because many of the factors included in the previous model were not statistically significant – all three institution level factors, for example, had little effect on the model and the social class and low participation neighbourhood variables also were not significant. With the number of factors reduced by removing these, the original method used for the other indicators becomes feasible and this is what has been done.
For most institutions, the effect of this change on the benchmark was small. For institutions with at least 200 responding students in the base population, the difference was no more than 1% when rounded to the nearest percent. For smaller institutions and particularly those with the highest/lowest values for the indicators, the differences may be larger.
Enquiries
Press: Call 01242 211120 or email pressoffice@hesa.ac.uk.
General enquiries should be sent to piteam@hesa.ac.uk.
Enquiries regarding the Performance Indicators Steering Group (PISG)
should be directed to the HEFCE Press Office on 0117 931 7307
