Conclusion

The Code of Practice for Statistics is based on three pillars: trustworthiness, quality, and value.[1] In order to comply fully with the Code of Practice, producers of statistics must ensure that the statistics they produce reflect these three attributes. While this report has been primarily concerned with assessing the Graduate Outcomes survey in terms of quality, the mutually supportive nature of the three pillars means that any assessment of statistical quality will also, of necessity, have implications for the trustworthiness and value of the statistics in question.

Statistical trustworthiness depends on the conditions of statistical production. If statistics are to be trustworthy, there must be a high degree of public confidence in the people and organisations responsible for producing them. This confidence must extend to the honesty and integrity of statistical producers, to their independence, to their commitment to the orderly release of statistics, to the transparency of their operating processes, to their professional capability, and to their standards of data governance.

In producing this report on the Graduate Outcomes survey and the statistical outputs derived from it, we hope to have shed some additional light on the processes underlying the design and implementation of the survey, the processing of survey data, and the production of statistical outputs. In so doing, we have contributed to the transparency of HESA’s operations, as required in section T4 of the Code of Practice; we hope that increased transparency will give users the information they need to have confidence also in the other elements which contribute to statistical trustworthiness. By explaining the processes by which we assess the accuracy and reliability of our data, for example, we hope to give users of the Graduate Outcomes survey confidence in the professional capability of HESA, Jisc and the partner organisations involved in survey administration and data processing; similarly, by discussing the efforts we have taken to protect personal information, we hope to give users confidence in our data governance practices.

Statistical quality is a characteristic of the statistical products themselves. It is not sufficient for statistical products to be produced in a trustworthy fashion; instead, the Code of Practice for Statistics stipulates that ‘the statistics must be the best available estimate of what they aim to measure’.[2] Producing high quality statistical outputs depends on collecting data from suitable sources, on employing sound methodology in the collection, processing, and analysis of data, and on being able to provide users with clear information about how the quality of data and statistics has been assured.

Over the course of this quality report, we have guided users of the Graduate Outcomes survey through the processes used by HESA and Jisc to assess the quality of the survey and the resulting statistical outputs. At each stage in the development and implementation of Graduate Outcomes HESA considered how best to ensure that Graduate Outcomes would be a high quality survey, leading to high quality official statistics outputs. The survey was designed both to capture relevant data about the experiences of graduates after course completion and to reach as many members of our target population as possible. Rigorous quality assurance processes were built into our data collection and processing systems, and we have continued to take user feedback onboard and refine our methodology at each stage of the process.

Since the completion of first full cycle of collection, processing, and publication, HESA and Jisc have continued to work to improve the quality of our data. In addition to our routine quality assurance work, we have also embarked on a complete post-implementation review of the Graduate Outcomes survey. For the last six months, a number of work streams have been involved in developing recommendations for the Graduate Outcomes Steering Group with the goal of ensuring that future iterations of the survey yield data which remains as relevant, reliable, accessible, timely, and coherent as possible.

The final pillar of the Code of Practice for Statistics is value. While trustworthiness and quality refer to how statistics are produced and the nature of the statistics themselves, statistical value depends on whether statistical products are fit for purpose. As is stated in the opening sentence of the introduction to the Code of Practice, ‘official statistics are an essential public asset.’[3] Official statistics thus exist for the benefit of their users, and neither the quality of outputs nor the trustworthiness of their production can make up for a failure to consider user needs for statistics that contribute usefully to issues of public concern.

HE providers have collected information on the destinations of their graduates since at least the 19th century; as participation in higher education has expanded and debates about the value of higher education have grown increasingly prominent, the appetite for data on graduates has increased. In designing and implementing the new Graduate Outcomes survey, HESA and Jisc have worked to iterate from and improve upon previous graduate destination surveys; we have retained questions which were deemed to have value, but we have also refined old questions and added new questions to provide additional insight. Having worked with key users to design a survey that collects data on the most relevant questions about the outcomes of graduates, HESA aims to produce statistical outputs which present that data as clearly and accessibly as possible. The Graduate Outcomes data releases and supporting materials affirm our commitment to the principles of open data, and, even more importantly, they also ensure that all of our users have access to statistical outputs designed to meet their needs.

Next: References