03/04 - October

Dear Colleagues

CONSULTATION DOCUMENT: TEACHING QUALITY INFORMATION - QUANTITATIVE DATA

Introduction/context

The final report of the Task Group looking at ‘Information on quality and standards in higher education’ published as HEFCE Report 02/15 in March 2002 set out, amongst other things, a range of topics on which quantitative teaching quality information should be published. It was always intended that HESA should provide the data to support publication of this information, thus removing any additional burden of collation and submission from institutions and ensuring consistent and high quality information. Institutions will therefore not be required to provide any data to HESA specifically for this purpose. Since publication of HEFCE 02/15 considerable work has been done to refine and elaborate the exact information to be published; format, level of disaggregation and the publication arrangements. Where decisions have been made, these, and the reasons for them, are presented in 'Information on quality and standards in HE: Final guidance’ (HEFCE 2003/51).

This consultation document presents current thinking in outstanding areas and asks for institutional views, particularly on the proposed detailed content and format of the quantitative information to be published. This information is to be published through the HERO web portal, at present, for institutions in England and Northern Ireland. The relevant bodies in Scotland and Wales are currently considering adopting some of this scheme, but adapting it as appropriate for their own circumstances and constituency.

Once the definitions and content are confirmed, it is intended that the necessary quantitative information will be generated centrally from the HESA Student and Destinations of Leavers from Higher Education (DLHE) Records. Institutions will therefore not be required to calculate these statistics locally for provision to HERO.

Proposed publication content

Tables

The tables to be presented will cover the topics set out in paragraph 17 of HEFCE 2003/51. A mock-up of the tables is provided at Annex A.

The first section will cover qualifications on entry. It is proposed that there be two tables, the first giving a breakdown of the highest qualification on entry for different student groups and the second summarising tariff scores for entrants to full-time undergraduate programmes. This second table will show tariff score information for those entrants with A/AS-levels only or Highers/Advanced Highers only as well as that for all entrants with ‘tariffable’ qualifications.

The second section will focus on progression and completion. In HEFCE 02/15 it had been proposed that the performance indicators and benchmarks already published annually be used in this context. However, investigation by analysts at HEFCE concluded that this approach would not be feasible below the level of an institution. It is therefore proposed that at a subject level simple summaries of student progression from one year to the next are given, with no benchmarks.

The third section will set out degree classifications for first-degree graduates.

The final section will show two different presentations of information on leavers’ destinations. The first of these will be a summary of graduate destinations (employment, further study etc.), and the second will list the most common job types obtained by leavers in this subject from an institution. Exact detail of the data definitions, particularly relating to those who studied part-time, will be considered following receipt by HESA of the first DLHE data.

Details of the data definitions to be used to prepare the student record-based tables are provided in Annex B and align with the standard HESA publication definitions, unless there is a specific reason to diverge from these. As part of the presentation of the tables it is proposed that the column/row headings will act as links to short explanations to some of the terms used. Some examples of these short notes are given at Annex C.

Charts

It is proposed that, where appropriate, users are offered the choice of seeing the information presented as charts rather than tables. A variety of stacked bar/column and pie charts are proposed. Where numbers of students are small it may not be appropriate to display information graphically as this might give a false impression. It is proposed that where the total student population to be included in a pie chart, or in a single bar/column or a stacked bar/column chart is less than 50 the chart or bar/column will not be shown, and users will see a message indicating that the number of students is too small to provide meaningful charts. This is consistent with HESA’s rounding strategy used for all its standard publications.

Coverage

The data to be presented will cover students registered at the HEI on both undergraduate and taught postgraduate programmes, including those franchised to other providers, also including NHS-funded and TTA-funded provision and overseas and privately-funded students registered at the HEI (ref. para. 19 of HEFCE 2003/51). In most cases the same information will be presented for all categories of students. However, tariff-based data for entry qualifications will be available only for full-time undergraduate entrants and class-of-degree information is relevant only for first degree students.

Subject representation

In HEFCE 02/15 it had been proposed that the quantitative information should be published for 19 subject areas based on JACS subject coding (see index.php?option=com_content&task=view&id=531&Itemid=233 for details). Following the pilot that HEFCE organised with prospective students and others, it has since been decided that this breakdown will not provide sufficient detail for many users, and that a finer level of resolution should also be provided.

It is therefore now planned that data be additionally available to users at the level of JACS Principal Subjects, of which there are 141. (In a very small number of cases the subject area definitions required a principal subject code to be further subdivided; see Annex B). It has also been suggested that within a given subject area or principal subject, it may be appropriate to show students studying single subjects separately from those for whom the subject forms only a part of their programme of study. While this latter proposal is feasible in theory, in practice it is thought likely that small numbers of students may mean that many reports cannot be generated because of data protection limitations. It is, however, also intended that users should be able to generate their own custom-made ‘baskets’ of principal subjects, which should allow the data at below subject-area level to be seen even when individual principal subjects cannot be shown because of small numbers.

Why not PIs?

In HEFCE 02/15 it had been proposed that performance indicators be published at subject level for progression, successful completion and destinations. Following publication of 02/15 HEFCE undertook further analysis of the current performance indicators. The existing method of projecting outcomes is unsuited to disaggregation below institutional level, due to transfers between subjects and the small numbers that would be used in forming the projections. In addition, this analysis found that where cohorts were small, e.g. less than 200, the performance relative to benchmark was liable to vary significantly year on year. At subject level a significant number of cohorts were below this limit, therefore it was decided not to publish PIs, but instead to publish information relating to progression. The same approach has been adopted for destinations information for similar reasons.

More explanation of this issue is given in HEFCE 2003/51, Annex D.

Time series

It is proposed that for the first year of publication of information (2004) only one year’s data will be made available – data for 2002/03. Following this, it is proposed that the information builds up to show three year rolling time series. So, in 2005 two years’ data will be available, and in 2006 and onwards three years’ data will be shown. It is not considered practicable to publish data for the years prior to 2002/03 because of major changes made to the records in that year (e.g. introduction of JACS and tariff on the student record and the replacement of FDS with DLHE).

Data protection

In order to adhere to appropriate legislation all of the teaching quality information figures will be presented using the standard HESA rounding strategy, which rounds all numbers to the nearest five, with 0, 1 and 2 shown as 0. Percentages will not be calculated where the denominator is less than 50.

Proposed format/timescales for making information available

Timetable

It is proposed that the teaching quality information generated from the HESA Student Record will be updated on the HERO site at the end of February each year with data about the previous academic year. Data about destinations will be updated at the end of July each year with data about leavers during the academic year completed one year earlier. For the first year (2002/03 data) all data will be published by the summer of 2004.

Data sign-off

Once the content and definitions for this information have been finalised, it is proposed that institutions be asked to sign off the data for this purpose as part of the data collection process. To assist institutions with this it would perhaps be possible to add some limited additional tables to the check documentation from 2003/04 onwards. Institutions are invited to suggest the sort of tabulations that would be most useful for this purpose. It will not be possible to give institutions an opportunity to make changes to the data after the relevant HESA data collection has closed, and so it will be most important that attention is given to the accuracy of the data during the collection process.

For the first year (2002/03) it was not possible to use this mechanism as the definitions of the teaching quality information to be published have yet to be finalised and the relevant data collection is almost complete. Institutions will therefore be given an opportunity to preview the first data for publication at a later stage in the process. HESA will use its Data Dissemination System (DDS) for this purpose – a system with which institutions are already familiar. It is not proposed that institutions should have the opportunity to make changes to the data at this stage, although it may be possible to add notes to the publication if any obvious inaccuracies are discovered.

Format

A mock-up of how the data is to be presented in table form can be found at Annex A. HERO will design the layout and general presentation to fit in with the general look and feel of the site, and in the light of views expressed by stakeholders. This annex therefore illustrates only detail of the proposed table structure and content, not the final format.

The data will be available via the HERO website; each subject/institution combination will constitute a web page generated to meet a specific user request.

As mentioned earlier, it is proposed that titles/headings be set up as links in HTML to provide users with relevant definitions/notes etc.. It will not, however, be easy to write these in such a way that they will meet the needs of the entire range of stakeholders – who are likely to have varying levels of knowledge/experience of the sector – therefore they will be written for the primary target audience, that is intending students and their advisors.

Action required

As stated previously, in parallel with this HESA consultation, HEFCE are issuing guidance to institutions on many other aspects of Teaching Quality Information (HEFCE 2003/51). The HEFCE document also contains some more detailed explanation of some of the decisions taken post-HEFCE 02/15 that are reflected in this consultation document.

Institutions are invited to comment on all of the proposals set out in this document and the Annexes. In particular, institutions are asked to focus on the following questions:

1) Are you content with the proposed technical definitions for the teaching quality tables and charts?

2) Are the proposals for ensuring data protection appropriate, or are there outstanding issues?

3) Are the example definitions and messages in Annex C appropriate?

4) Do you agree with the proposal to show three years’ rolling time series for the quantitative information?

5) Would the option to present data separately for students studying single and combined subjects significantly improve or otherwise affect the information provided about your institution?

Please send your comments on this consultation document to catherine.benfield@hesa.ac.uk by 5 December 2003.

Yours sincerely

C. Jane Wild
Director of Operations