PIs 2005/06: Adjusted sector benchmarks – technical notes and detailed information
Adjusted sector benchmarks – technical notes and detailed information
(applicable to tables T1 to T3, T7 and E1)
This page contains the technical details and assumptions made in producing the adjusted sector benchmarks for tables T1 to T3 and T7. It also covers the location-adjusted benchmarks, and the calculations for the standard deviations. Details of the subject and entry qualifications breakdown used to obtain the benchmarks, and tables showing the numbers of students in each category and the proportion of students in each category with different characteristics, are given at the end of this document.
Most of the indicators included in these tables have benchmarks attached. The benchmarks are not targets. They are average values which will change from one year to the next if the overall value of the characteristic changes. They are provided to give information about the sort of values that might be expected for an institution’s indicator if no factors other than those allowed for were important. The corollary of this is that where differences do exist, this may be due to the institution’s performance, or it may be due to some other factor which is not included in the benchmark.
What should be included in the benchmark?
The factors to be included in the benchmarks need to have a number of characteristics. In particular they should:
- be associated with what is being measured
- vary significantly from one institution to another
- not be in the institutions’ control, and so not be part of their performance.
The first two characteristics were easy to identify. It was obvious from analysis already done that non-continuation rates, for example, varied between subjects, so subject as a factor had the first characteristic. It also had the second characteristic, as the proportion of students in each subject area varied between institutions.
It was not so easy to identify factors with the third characteristic. For example, the subjects offered at an institution could be considered to form part of that institution’s performance, in that they could theoretically be changed, but in practice changing an institution’s subject mix substantially is very rare. After much discussion it was agreed that both subject of study and entry qualifications should be counted as outside an institution’s control.
The benchmarks were therefore set up to take account of the entry qualifications of an institution’s students, the subjects they studied, and their age. It needs to be stressed that because a difference between institutions may be accounted for by differences in the subject or entry qualification profiles of the institutions this does not imply a justification of that difference. The purpose of the benchmarks is to allow any discussion of the reasons for the differences to be carried out on an informed basis.
Using the benchmarks
The tables of indicators, by including all institutions in one table, allow direct comparisons to be made both between institutions, and between an institution and the sector. However, if the benchmarks were ignored such comparisons would not take account of the effects of different subject profiles or the different entry qualifications of the students. In general, indicators from two institutions should only be compared if the institutions are similar. If the benchmarks are not similar, then this suggests that the subject / entry qualification profiles of the institutions are not the same, and so differences between the indicators could be due to these different profiles rather than to different performances by the two institutions.
To compare an institution’s indicators to the sector, the benchmark should be used in preference to the overall sector average, again because it takes account of the subject and entry qualifications profile. We have provided a symbol beside the benchmark to show whether the difference between the indicator and the benchmark is significant.
Two symbols are used to show significance. A plus sign, ‘+’, indicates that the institution’s indicator is significantly better than its benchmark and a minus sign, ‘-’, indicates that the indicator is significantly worse than its benchmark. If there is a blank, the institution can say that its indicator is similar to the sector average allowing for subject and entry qualifications. Institutions whose indicator is significantly worse than the benchmark should look carefully at their figures to determine why the difference is occurring, bearing in mind that there may be some explanation based on factors that have not been taken into account.
Location-adjusted benchmarks
For institutions in England location-adjusted benchmarks are included in tables T1 and T2, in addition to the original benchmarks. These benchmarks take account of where an institution’s students come from, as well as their subject and entry qualifications. They are the result of work done by HEFCE to try and measure the effect of location on the access indicators in these tables.
The difference between the two benchmarks will show how much effect the region of origin of an institution’s students has on the indicator. Small differences, say no more than 1 or 2 per cent, suggest there is little effect. Either the institution recruits nationally, or it recruits locally from a region which is similar to the average of the UK as a whole. Larger differences mean that the geographical effect seems to be important.
Which benchmark is used will depend on the context. Both benchmarks provide information about the institution, and together they can shed light on why an indicator takes certain values. Note that in deciding whether two institutions are similar, it is the original benchmark that is most informative – the fact that the location-adjusted benchmarks of two institutions are different may only indicate that the institutions are in different parts of the country. Institutions which do better against the location-adjusted benchmark than against the original one can point out that their location, in the sense of where their students come from, is affecting their results. An institution that does better against its original benchmark than against the location-adjusted benchmark may note that, although much of its success in recruiting students from low participation neighbourhoods, for example, is because of its location, nevertheless it is still taking in large numbers from such areas. In both cases institutions should examine their results critically.
The location-adjusted benchmarks have not been included for institutions in Wales, Scotland or Northern Ireland. The funding bodies for these institutions have decided that such benchmarks could be confusing when applied to institutions in these areas.
Technical notes
The factors allow the population to be broken down into well-defined categories, which are used in the calculation of the adjusted sector benchmark. In addition, the ‘sector population’ needs to be defined, as it is not the same in all cases. Each indicator relates to a specific sub-set of the institution’s students, for example, young full-time first degree students, or mature part-time undergraduates, and the adjusted sector benchmark is based on the equivalent sub-set of the sector population.
The sub-set of the population used will only contain students for whom information to calculate the indicator is available. The institution’s profile is also based only on those of its students with that information available. So, for example, if the information about school type is available for only 80 per cent of an institution’s students, the institutional profile used to obtain the benchmark for the indicator will be based on that 80 per cent.
The number of categories used in the calculation of the benchmarks will depend on which factors are included. As there are 18 subject groups and 22 entry qualification groups, the original adjusted sector benchmark for the access indicators is based on 18×22=396 categories. For the non-continuation indicator for all ages, where age is also taken into account, the number of categories will double to 792 and for the location-adjusted benchmark for the access indicators, where region is also a factor, there will be 396×13=5148 categories.
Assume there are C categories, numbered from 1 to C, and U institutions, numbered from 1 to U. Let the number of students in institution j in category k be njk. Then the total number of students at institution j is 
, the number from the sector in category k is 
, and the total number of students in the sector is 
.
Let p.k be the proportion of students in the sector from category k who have the characteristic of interest, for example, are from state schools, or have left HE after a year, and the equivalent proportion for institution j be pjk . The proportion of students in institution j with the characteristic of interest can be found as
This is the value of the indicator. If the proportion of students with the characteristic at the institution in each subject/entry qualification category was the same as in each category in the sector, then the overall proportion with the characteristic would be
This is what we have called the ‘adjusted sector benchmark’.
Another way of interpreting this is to say it is the value that the sector average would have if the sector students were split across the C categories in the same proportions as at the institution.
Standard deviations
In general, small differences between an indicator and its benchmark are not important. However, it is not always obvious what constitutes a small difference. A standard deviation measures the amount by which one would expect a statistic to change, based solely on random sampling, and can therefore be used to say that a particular difference is significant or not. We have calculated the standard deviations of the differences between the indicators and their benchmarks, using a method developed by Professor David Draper and Mark Gittoes, formerly at the University of Bath. (Note that, because these are standard deviations of a statistic, they are more usually called standard errors.)
The mechanics of the calculations are explained below. More details of the statistical model used can be found in ‘Statistical analysis of performance indicators in UK higher education" by D. Draper and M. Gittoes, in JRSS Series A, volume 167, part 3, 2004.
Assume that there are C categories for the factors used in the benchmarks, and U institutions. The complete set of C × U cells will be called the basic grid. The actual indicator at institution j, pj., is a weighted average of the form
The proportion of students in the sector in category k, p.k, is
and the benchmark for institution j, Ej, is
The difference between the indicator for institution j and its benchmark, Dj = pj. - Ej, can then be written as a weighted sum of all C × U cells in the basic grid:
where
and 
Assuming that the njk students at institution j in category k are like a random sample (with replacement) from the population of all such future students, the values pik and Dj can be estimated as 
and 
respectively. The variance of is given by
We then have to estimate the variance of .
Draper and Gittoes show that a reasonable estimate of this variance is obtained by using a shrinkage estimation procedure. The value used here is
where
, and 
is the estimated percentage with the characteristic of interest in the sector as a whole.
The square root of the estimated variance, which is the standard deviation of the indicator, can then be used to test whether the difference between the indicator and its benchmark is small or not. A difference that is less than twice the size of the standard deviation can certainly be said to be small, but we have been more conservative. In the tables, we have marked as ‘large’ those institutions where the difference is both greater than three times the standard deviation and greater than three percentage points. This is to draw attention to areas where the difference is large in both statistical and practical terms.
If an institution is marked in this way, it should be taken as an invitation to investigate possible causes for the differences that have been identified, whether they arise from an indicator that is better than the benchmark (marked +), or worse than the benchmark (marked -). Where the difference is not marked, the indicator is either within the range that would be expected given random fluctuations, or is less than three percentage points away from the benchmark.
Projected outcomes
The adjusted sector benchmarks for the projected outcomes indicators in table T5 are obtained by adjusting the transition matrix rather than the actual indicators. The standard deviations have therefore been obtained by assuming students have been selected at random from the outcome categories. These are simplifications, but appear to give realistic results in most cases. Further details of the methods used can be found in the projected outcomes definitions.
Subject and entry qualifications breakdown for the sector
In 2002/03 a new subject classification was introduced called the Joint Academic Coding System (JACS). This subject classification looks similar to that previously published but has been devised in a different way. Therefore subject data is not comparable to that previously published.
Additionally, from 2002/03, a new procedure of apportionment has been introduced. Under apportionment, each headcount is, where necessary, divided in a way that in broad-brush terms reflects the pattern of a split programme. This is analogous to the use of FTE calculations, but should not be confused with them, since the splits used for apportionment are conventional rather than data-based.
For split programmes not involving an initial teacher training (ITT) component, the apportionment algorithm is as follows:
50%:50% for a balanced two-way split;
66.667%:33.333% for a major/minor two-way split;
33.333%:33.333%:33.333% for a balanced three-way split.
ITT students at undergraduate level who also have a specialism subject recorded (typically, secondary ITT students) are apportioned 50% to the ‘Education’ subject area and the remaining 50% is further apportioned according to the algorithm for non-ITT students. Where no subject other than education is recorded, or where the student is on a PGCE course, apportionment is 100% to the ‘Education’ subject area.
Subject categories
The subject categories have been obtained from the subjects of qualification aim, HESA fields SBJQA1, SBJQA2, and SBJQA3. The table below shows the codes used for the subject categories, and the subjects they represent. These differ from the standard HESA categories in that medicine & dentistry has been grouped with veterinary science.
| Subject of study | JACS subject area codes |
| Medicine & dentistry and veterinary science | 1, 4 |
| Subjects allied to medicine | 2 |
| Biological sciences | 3 |
| Agriculture & related subjects | 5 |
| Physical sciences | 6 |
| Mathematical sciences | 7 |
| Computer sciences | 8 |
| Engineering & technology | 9 |
| Architecture, building & planning | A |
| Social studies | B |
| Law | C |
| Business & administrative studies | D |
| Mass communications & documentation | E |
| Languages | F |
| Historical & philosophical studies | G |
| Creative arts & design | H |
| Education | I |
| Combined subjects | J |
Entry qualifications
The majority of students in the UK still enter higher education with either A-levels or Scottish Highers, and this is recognised in the groupings of entry qualifications used. As with subjects, the groupings have been chosen so that as far as possible the students within each group are relatively homogeneous.
For entrants from 2002/03, grades achieved in A-levels, Scottish Highers, Vocational A-levels and some other examinations have been converted to tariff scores instead of the old points scores. Full details of the tariff are provided on the UCAS website. Tariff scores are held on the HESA record in fields GCEASTS, GCEATS, SAHTS, SHTS, VCEASTS, VCEATS, KSQTS, UKSATS, SCSTS, SI2TS, SSGCTS and TOTALTS.
The table below shows the categories used for 2005/06 entrants, their descriptions, and the values of the QUALENT2 field used in the definition. In addition, if QUALENT2 has a value of 39 or 40 (A-levels, AS levels or Scottish Highers) then the values in the fields GCEASTS, GCEATS, SAHTS, SHTS, VCEASTS, VCEATS, KSQTS, UKSATS, SCSTS, SI2TS, SSGCTS and TOTALTS have been used to determine whether the student has A-levels, AS levels or Scottish Highers with a tariff score, VCE or GNVQ only, or some combination of A-levels, AS-levels, Highers, VCEs or GNVQs.
To determine the number of tariff points, the scores in the HESA fields KSQTS, UKSATS and SCSTS have been subtracted from the total score in the TOTALTS field.
| Entry qualification | QUALENT2 code |
| A-levels / Scottish Highers, with no VCE or GNVQ, zero tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 001 to 100 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 101 to 160 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 161 to 200 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 201 to 230 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 231 to 260 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 261 to 290 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 291 to 320 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 321 to 350 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 351 to 380 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 381 to 420 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 421 to 480 tariff points | 39, 40 |
| A-levels / AS levels / Scottish Highers, with no VCE or GNVQ, 481 to 999 tariff points | 39, 40 |
| VCE or GNVQ and no A-levels / AS levels / Scottish Highers | 39, 40 |
| VCE or GNVQ and A-levels / AS levels / Scottish Highers | 39, 40 |
| Baccalaureate | 47 |
| Foundation or Access course | 29, 43, 44, 45, 48 |
| BTEC, ONC, SCOTVEC or equivalent | 41 |
| Higher education qualification | 01, 02, 03, 04, 05, 10, 11, 12, 13, 14, 15, 16, 21, 22, 23, 24, 25, 26, 27, 28, 30, 31 |
| No previous qualification | 92, 93, 98 |
| Other qualifications not given elsewhere | 55, 56, 94, 97 |
| Unknown qualification | 99 |
Region of domicile
The domicile regions used in the location-adjusted benchmark are the nine Government Office regions in England, plus the three other countries of the United Kingdom: Wales, Scotland and Northern Ireland. Note that the Performance Indicators tables only include students whose normal residence is in the United Kingdom, excluding the Channel Isles and the Isle of Man.
| Regions | |
| North East | |
| North West* | |
| Yorkshire and the Humber | |
| East Midlands | |
| West Midlands | |
| East | |
| London | |
| South East | |
| South West | |
| England region unknown | |
| Wales | |
| Scotland | |
| Northern Ireland | |
*includes Merseyside | |
Enquiries
Press: Call 01242 211120 or email pressoffice@hesa.ac.uk.
General enquiries should be sent to piteam@hesa.ac.uk.
Enquiries regarding the Performance Indicators Steering Group (PISG)
should be directed to the HEFCE Press Office on 0117 931 7307
