Relevance

When considered as a dimension of statistical quality, relevance refers to the extent to which statistical outputs meet the current and potential needs of users. In order to assess relevance, it is necessary first to identify likely users of the data and their needs. The data sources and statistical concepts used in the production of a statistical output are also a factor in determining relevance; depending on user needs, different data sources and classification schemes will be appropriate. Finally, it is important to identify any gaps between the statistical output and known user needs and to assess how those gaps may be filled in future.

Users and user needs

As a producer of official statistics, HESA is obligated to consider a number of principles in assessing and reporting on the quality of statistical outputs. One of these principles, originating from the European Statistical System Code of Practice and endorsed by the UK Government Statistical Service, is the assessment of user needs and perceptions[0]

A wide variety of users, in the HE sector and beyond, work with HESA data on graduates. HESA has obligations to a range of statutory customers in all four UK nations, including the funding and regulatory bodies for higher education in each nation; our statutory obligations to these customers require us to provide them with the data which they need to carry out their public functions[1]. As Designated Data Body for England, HESA is further required by law to publish ‘appropriate information relating to registered HE providers and the higher education courses they provide’. According to the Higher Education and Research Act 2017 (HERA), the category of ‘appropriate information’ includes information which may be helpful to students in higher education, potential higher education students, and HE providers; HERA also specifies that the designated data body must provide appropriate information to the Office for Students (OfS), UK Research and Innovation (UKRI), and the Secretary of State for Education[2]. In addition to those users whose needs we are required by law to consider, we also wish to consider the needs of others for whom high quality data on graduates will be useful, including HE funding and regulatory bodies, local and national governmental agencies, graduate employers, and academic researchers[3].

Different users have different needs for the Graduate Outcomes data. Prospective HE students may look to Graduate Outcomes in order to make informed choices about providers and courses, while HE providers may use the data for strategic planning purposes. Funding and regulatory bodies may use Graduate Outcomes data to assess the performance of providers and courses, while government agencies—both local and national—and graduate employers may look to the data to provide information both about the regional supply of graduates with different skills and about the roles played by graduates in society more generally. Since the publication of the first statistical outputs based on the survey, HESA has tracked citations of Graduate Outcomes data, and seen it used a variety of publications, from stories on higher education in the national media, to publications designed to support student choice, to analyses conducted by funding and regulatory bodies[4].

Throughout the design and implementation phases of the Graduate Outcomes survey, HESA has been engaging with the various potential users of the survey data, actively. In the early stages of the NewDLHE review, a Strategic Group and a Working Group were convened; these groups, which were comprised of representatives from a wide variety of HE providers and other sector bodies, were responsible for setting the remit for the review and developing a workplan to pursue this remit[5]. Later in the review, HESA carried out two consultations, the first to determine user needs for the successor to DLHE, and the second to solicit feedback on the draft model for the new survey[6].

Responses to the second consultation suggested a high level of stakeholder approval for the proposed model, giving HESA a mandate from potential survey users to proceed with the implementation of the new survey. The model proposed in the second consultation called for the establishment of the Graduate Outcomes Steering Group; this group, like the earlier Strategic and Working Groups, is designed to reflect the diversity of stakeholders for the Graduate Outcomes survey and is comprised of representatives from HE providers and HESA statutory customers from across the UK. The Steering Group met quarterly during the development and implementation of the survey to advise HESA on all aspects the Graduate Outcomes survey. HESA values the expertise and input which has so far been contributed by the Graduate Outcomes Steering Group, and it is envisaged that the group will continue to operate in an oversight capacity and to help guide further improvements to the survey[7].

In addition to the regular meetings of the Graduate Outcomes Steering Group, HESA continues to solicit feedback from the sector on particular issues. While the charts and tables to be included in the first Graduate Outcomes Statistical Bulletin and open data release were being developed from conceptual designs into logical wireframes, HESA convened a group of sector representatives to ascertain whether the planned outputs met with user needs. This engagement with stakeholders informed the initial publication of Graduate Outcomes data and has continued to guide our decision-making process as we have prepared for subsequent years of outputs[8]. Over the the second and third years of surveying, HESA has invited key users to participate in a review of the survey questionnaire, and has also continued to consider user feedback about the survey and its associated outputs submitted to the Agency via other channels[9].

A second phase of the survey review was launched in April 2022 and is currently ongoing. In addition to a review of user needs surrounding the existing survey questions this second phase of the review will also include a mechanism for identifying potential requirements for new survey topics. Through an ongoing programme of horizon scanning and engagement with relevant policy experts, we aim to ensure that the Graduate Outcomes survey continues to meet the evolving needs of its users.