PIs 2005/06: Research indicators - Technical notes and more detailed information

Research indicators - Technical notes and more detailed information

The research indicators used here are based on two measures of input and two measures of output. Three of these measures are available through the HESA collections, while the fourth is obtained from the funding councils. Note that summaries of the data collected by HESA are published by them, for example ‘Students in Higher Education Institutions’ for student data and ‘Resources of Higher Education Institutions’ for finance data.

The indicators used are standardised by cost centre, as explained below. If an institution uses the same proportion of input as it produces of output, then the indicator will take a value of 1.

Measuring input to research

There are a number of problems in creating a measure of research input. These arise partly because data on expenditure are not split between research and teaching. There is therefore no one ideal input measure, so it is proposed to use two complementary measures. These are:

Academic staff costs. This measure is available for all institutions by cost centre from the HESA finance record. It is therefore more complete than any based on the staff returns, although no separation of expenditure on research from that on teaching is possible.

Research income from funding councils. The measure based on academic staff costs takes no account of the varying relative levels of resources spent on research and teaching. The measure proposed here, the research income from funding councils allocated for quality (QR funding), aims to provide an indicator that does take account of the varying level of resources available for research. It is based on the funding allocation model used to allocate the recurrent research funds. Such funds form part of the block grant, which institutions are free to distribute internally as they see fit. This measure assumes that this internal distribution will follow the funding allocation model used by the funding council. This, in general, will not be the case, so the measure will provide only a rough indication of what funds go where. The research funding is allocated by 69 RAE units of assessment which have been mapped to the 33 cost centres.

Measuring research output

Similarly, two measures of research output based on HESA records are proposed:

PhDs awarded. The number of doctoral degree completions provides a measure of the vitality of the institution in educating new researchers. The number of PhDs has been taken from the current HESA student record, summing records with a qualification obtained of ‘PhD mainly by research’. The cost centre has been identified through the same record, wherever possible, or from records for the student in previous years. In some cases, we have had to map the subject of study to cost centre.

Research grants and contracts. Although this could be thought of as an input, it also provides a measure of the success of researchers in attracting funds over and above those allocated by the funding councils. The value of research grants and contracts comes from the current HESA finance record.

Standardising for subject variation

There are often differences in the characteristics of research output between subjects. These can be adjusted for by making the measure specific to cost centre. If the output is, say, research contract income, then the indicator treats a pound of income as a different ‘currency’ of research output for each cost centre. This is done by looking not at the actual input and output, but at what proportions they form of the sector input and output. This means that institutions whose research is primarily in areas where the unit costs are low are not at a disadvantage compared with institutions whose research is mainly in areas with high unit costs.

The computation of the indicators is carried out as follows. Let:
r_k = research output of institution
R_k = research output of sector
s_k = input to institution
S_k = input to sector
all in cost centre k.

Then, is the total input to the institution.

For cost centre k, the relative performance of an institution, p_k, is given by:

The overall performance of the institution, p, is then calculated by summing these cost centre ratios using the weighting (s_k/t):

Measures of coverage

To put these indicators into context, a number of measures of coverage were considered. The ones included in the table are amount of research funds from the funding councils, the percentage of funding from the funding councils allocated for research, and the total number of cost centres to which there is some input.

In addition, the table available under 'Extended' contains a measure of how different the institution’s pattern of provision is from the sector’s pattern. This is called a measure of specialisation, and it can be used in conjunction with the number of cost centres to give an indication of how specialist an institution is.

Context statistics

There is one context statistic for the research indicators, which has been called a ‘measure of specialisation’. It is designed to be used, in conjunction with the number of active cost centres, to show how comparable the institution is to others in the sector with regards to the spread of research quantity across cost centres.

If an institution has only one active cost centre, then the measure of specialisation gives a value of zero. In such cases, the institution is clearly very specialised, in the sense that it is concentrating all its research effort into one cost centre, so the measure of specialisation is unnecessary.

If an institution has more than one cost centre, the measure of specialisation gives an indication of how research activity (as measured by the input, that is either academic staff costs, or funding council allocation for research) is spread across these cost centres compared with how it is spread across the same centres in the sector as a whole. If the institution is similar to the sector in the way its research activity is spread across its cost centres, the measure of specialisation will be close to zero. If the institution is very different from the sector in the way its research activity is spread across its cost centres, then the measure of specialisation will be closer to one. In practice, it is rare for a value greater than 0.2 to occur, and any institution with such a value can be regarded as very different from the rest of the sector. Also in practice, institutions with high numbers of active cost centres have very low values for the measure of specialisation, which is not unexpected as such institutions are generally not very specialised.

The measure of specialisation depends on the inputs to the institution in each cost centre, and to the sector as a whole, again for each cost centre. Let s_k be the input to the institution in cost centre k, and S_k be the input to the sector in cost centre k. Then t = ∑(s_k) is the total input to the institution for all subjects.

In addition, for the institution of interest let

Then the number of active cost centres at the institution, n, is given by n = ∑(n_k)

The measure of specialisation, d, for the institution can be derived as follows:

where is the total input to the sector across all cost centres that are active at the institution (so the sum is over all values of k for which s_k > 0). Then the measure of specialisation is d = ∑(d_k/n)

Rounding strategy

Due to the provisions of the Data Protection Act 1998 and the Human Rights Act 1998, HESA implements a strategy in published and released tabulations designed to prevent the disclosure of personal information about any individual. This strategy involves rounding all numbers to the nearest 5. A summary of this strategy is as follows:

• 0, 1, 2 are rounded to 0
• All other numbers are rounded to the nearest 5.

So for example 3 is represented as 5, 22 is represented as 20, 3286 is represented as 3285, while 0, 20, 55, 3510 remain unchanged.

This rounding strategy is also applied to total figures; the consequence of which is that the sum of numbers in each row or column will rarely precisely match the total shown.

Average values, proportions and FTE values prepared by HESA will not be affected by the above strategy, and will be calculated on precise raw numbers. However, percentages and indicators calculated on populations which contain less than 20 individuals will be suppressed and represented as a blank value.

Enquiries

Press: Call 01242 211120 or email pressoffice@hesa.ac.uk.
General enquiries should be sent to piteam@hesa.ac.uk.
Enquiries regarding the Performance Indicators Steering Group (PISG) should be directed to the HEFCE Press Office on 0117 931 7307