Measure for measures
Our evaluation of user needs for HESA performance metrics in UK higher education
Many of HESA’s data users will be aware of the UK Performance Indicators publication, as one of our longest-running series of statistics about higher education. The UK Performance Indicators (UKPIs) are official statistics which help users compare the performance of universities and colleges against benchmarks. Current UKPIs include measures for widening participation in HE, and for student non-continuation. In the past, the UKPIs have covered other aspects of HE sector performance, such as measures associated with research and graduate destinations.
The UKPIs were conceived in the early 1990’s, prior to devolution and the transfer of education policy as a devolved competence. Historically, the purpose of the indicators has been to provide reliable and stable information on the nature and performance of the UK higher education sector; to support comparison between individual HE providers of a similar nature, where appropriate; and to enable providers to benchmark their own performance. They were also intended to inform policy developments and contribute to the public accountability of higher education.
Performance Indicators were first developed and published for the 1996/97 academic year. From 1998 to the present, the UKPIs were produced according to specifications set by the UK Performance Indicators Steering Group (UKPISG). Until 2017, the UKPIs were managed in the interests of stakeholders in the UK-wide HE sector, and on behalf of the administrations in England, Wales Scotland and Northern Ireland, by the former Higher Education Funding Council for England (HEFCE). Working under these arrangements, the UKPIs have been produced and published by HESA since 2002/03. My own involvement with the indicators spans much of the period since HESA took on their publication, serving as Chair of the technical sub-group to UKPISG for many years.
Of course reliable and stable sets of statistics have many benefits, especially when they describe a stable and unchanging environment. However, higher education in the UK has seen significant changes over the time period covered by the UKPIs, as the policy and regulatory environments across the UK home nations have diverged. In the latter years of the UKPISG, development of the indicators faced some significant challenges. Although development work on new indicators was undertaken, there were significant difficulties in reaching consensus on UK-wide definitions and deployment. This resulted in proposed new indicators never achieving approval for launch. Whilst ‘stable’ may often be interpreted as a positive descriptor, ‘static’ is usually rather less favourable. Measures such as those on area-based participation (the ‘POLAR’ methodology) have become less relevant in some UK home nations, being replaced within other published statistics with indices of deprivation as indicators of disadvantage. The state school indicator is hard to interpret given differences in composition of the state and independent school sectors across the UK. The indicator measuring proportions of students in receipt of Disabled Students Allowance (DSA) is arguably less valuable as a proxy for disability, as changes in eligibility criteria and purpose have led to overall numbers remaining static whilst numbers of students self-reporting disabilities has grown substantially. These are just a few examples of where the current indicators would need fundamental revision.
Whilst we have seen development of the UKPIs stagnate, we have witnessed increasing pace in the development of formal regulatory and policy measures across the UK. There are now measures published by HE sector bodies which appear to address the same concepts but present different values due to different definitional choices. This has the potential to cause confusion amongst users of HE statistics and is a threat to coherence of data. Coherence between statistical outputs is an important dimension of quality as expressed within the Code of Practice for Statistics, with which HESA is obliged to comply as a designated producer of Official Statistics. Lack of coherence can undermine public confidence in published statistics and has the potential to undermine effective regulation.
HESA has continued to produce and publish the existing indicators since the dissolution of the previous UKPIs governance structure, but we have been acutely aware that all data and statistics must evolve to remain relevant. In our view, fundamental changes in HE policy and regulation make it vital to undertake a similarly fundamental review of the role and purpose of a statistical product such as the UKPIs.
Over the past twelve months HESA has worked on developing a new strategic vision for the UKPIs, and we have sought views on this from HE provider bodies, regulators, funders and government departments across the UK. A key aim in seeking feedback on the strategic vision has been to ascertain the extent to which it is capable of resolving the lack of coherence in statistics and addressing alignment with regulation, whilst delivering new and demonstrable value to users across the UK.
“The next edition of the UKPIs in 2022 will be the last in its current form”
Having now completed this engagement process we have reflected extensively on the outcomes, both within HESA and with the HESA Board.
We have reached the following conclusions:
- No clear consensus has emerged on a new strategic vision for the UKPIs.
- There is a clear desire to see the lack of coherence between the UKPIs and formal policy and regulatory metrics resolved.
- The target audience, range of purposes, expectations of, and constraints on, the UKPIs are very broad – perhaps too broad for a single product in the current operating environment. More value may be delivered at lower cost by a more differentiated set of products and services.
- The policy and regulatory landscapes across the UK continue to evolve. HESA’s statistical products must adapt to remain relevant and useful.
- There is a need to more effectively align statistics and measures to the policy and regulatory contexts in the UK home nations – balancing this requirement with UK-wide comparability where achievable. UK-wide comparability and consistency in published statistics remains at the heart of HESA’s role.
- Diversity of HE provider types has increased markedly since the UKPIs were first conceived as a product. Our approach to statistical benchmarking based upon weighted sector averages has not been evaluated in the light of this changed environment.
- The UKPIs are perceived as holding some regulatory significance within the UK HE Sector. This was never their purpose and they were not developed as regulatory tools. HESA’s role is to collect and deliver high quality data and statistics to support policy development and regulation (among many other purposes), not to assess the ‘performance’ of HE providers.
- Establishment of a new governance structure, development strategy and structure for the indicators would be resource-intensive and therefore costly. Such costs could only be met through HE provider subscriptions meaning we need a compelling business case to put to the HE sector. This business case is not apparent.
In view of these conclusions we have decided that the UKPIs require fundamental reform. This will include a number of actions.
Firstly, the UKPIs ‘brand’ and labelling as performance measures will be discontinued. This will help to resolve misunderstandings about the regulatory status of HESA statistics.
We will undertake a review of the existing indicators with a view to migrating some, or their underlying data, into our core official statistics and open data products. This may also involve updating methodologies and definitions to ensure the statistics are robust and relevant. Some of the existing indicators will be discontinued where they have lost relevance and cannot be readily adapted to serve current needs.
The current approach to statistical benchmarking used in the UKPIs will not be applied to indicators that are selected for migration into our core official statistics. HESA may identify alternative applications for this benchmarking approach in future.
We will undertake research and development activities to identify new products and services to meet some of the user requirements that will no longer be readily serviced following the changes described above. Most notably there remains a desire within HE providers to compare and benchmark their characteristics and activities with other similar providers, and to set their activities in a wider ‘HE sector’ context. HESA will work with our partner Jisc to consider how best this requirement can be met. Existing provider services such as the data analytics tool Heidi Plus may provide an effective platform for new benchmarking services.
We will focus our efforts on developing useful and innovative approaches to data analysis, in order to deliver greater insight for our users. This will be informed by research activity and grounded in a sound understanding of the evolving HE policy environments across the UK.
“Measures selected for migration into core official statistics will be published in 2023 with no gap in the sequence of academic years”
As a result of this process of reform, we are announcing that the next edition of the UKPIs in 2022 will be the last in its current form. We expect to deliver the outcomes of the review of existing indicators through 2022 and early 2023, so that those measures which are selected for migration into core official statistics and open data will be published in 2023 with no gap in the sequence of academic years covered.
Another implication of this process is that development work on a new Graduate Outcomes (GO) UKPI to replace the former employment outcomes indicator will now cease. Whilst HESA will continue to enhance and expand our publication of statistics from the GO survey, and this will involve the creation of new analyses and statistics, HESA will not publish any new provider performance metrics derived from this survey.
I believe these changes will enable HESA to distil the greatest value from the existing UKPIs suite and continue to deliver this through our core statistical publications. At the same time it will enable us to focus efforts on innovation and enhancement – creating new statistics and services that can be rapidly adapted to address the changing needs of our users. We are proud of our role in the production of a very long-lived and influential set of performance measures, but we are also very excited by the opportunities we see to substantially enhance our data and statistics in future.