CDBU’s response to the TEF consultation

Response ID ANON-TWMW-TN1X-T

Submitted to Independent Review of TEF: Call for views. Submitted on 2019-02-26 10:24:26

Who are you?

1 What is your name?

Name:

Professor Dorothy Bishop, FRS, FBA, FMedSci

2 What is your role/position (if relevant)?

What is your role/position (if relevant)? :

Professor of Developmental Neuropsychology, University of Oxford

3 What is your email address?

Email:

dorothy.bishop@psy.ox.ac.uk

4 In what capacity are you responding to this consultation?

Other (please state)

If other, please specify below:

Member of executive committee: Council for the Defence of British Universities

5 Are you responding on behalf of an organisation (eg. higher education provider, student union or representative group)?

Yes

a. If yes, what is the name of your organisation?:

Council for the Defence of British Universities (http://cdbu.org.uk/)

Yes

6 Have you been involved preparing for or writing a TEF or subject TEF submission?

No

7 Have you been involved as a TEF assessor or panel member (for provider TEF or in the subject pilots)?

No

8 Would you like us to keep your responses confidential?

No

If yes, what is the reason for confidentiality?:

9 Please indicate which UK country/other country you are responding from.

Please indicate which UK country/other country you are responding for. :

England

If you are responding from a country outside of the UK, please write this in below.:

Why have TEF?

10 Do you support the aim of assessing the quality of teaching excellence and student outcomes across providers of higher education?

No

Please explain why:

We do not think that it is necessary or desirable for the whole sector to undergo a formal evaluation and ranking. We think that alternative approaches should be considered (see in particular response to 18b).

It is clearly desirable to maintain high quality in teaching, and provide students with information about courses. However, the TEF does not achieve either goal.

The interests of students would be best served by providing them with the kind of data that TEF draws on, supplemented by information about who will be teaching the course. Maintenance of standards can be achieved be monitoring of the kind traditionally undertaken by the Quality Assurance Agency (QAA).

Why have TEF?

11 These purposes fall into two main areas: providing information, and enhancing the provision of higher education.

Providing information

b. Please outline below the reasons for your answers :

There is evidence that students often embark on courses with little knowledge of the nature of the institution and the requirements and content of the course. We think that UniStats needs to be developed and publicised as a way of providing such information, but that providing students with a rating of institutions on a 3-point scale is counterproductive.
So, yes, it is important to provide information, but TEF does not do so in a way that is useful.

12 Should there be any other purposes for TEF?

Should there be any other purposes for TEF? :

No.
Since we don’t think TEF is needed, we don’t think its mission should be expanded.

What is TEF?

How does TEF work?

13 Are the criteria used in TEF (see Figure 1 for a list of the criteria) appropriate?

No

If not, what criteria would be more appropriate? :

We do not think that TEF has benefits that justify its costs.
UK higher education is currently held in high regard worldwide without having needed sector-wide institutional rankings. TEF threatens this. We conclude TEF should be abandoned, so we would not suggest alternative criteria.

14 There is no direct measurement of teaching quality currently available. As a result, the TEF uses existing data as indirect measures of teaching quality. These measures are known as “proxies”.

No

b. If you answered no, what metrics would be more suitable proxies? :

None. Use of proxies is a terrible idea. It encourages gaming. The pursuit of outcomes favourable to the institution’s scores has potential to distort institutional priorities, such as student recruitment to prefer applicants who are predicted to provide positive results on the proxy metrics.

Student survey data are problematic on four counts:
1. They do not measure teaching quality.
2. Indeed, there is evidence that students tend to give lower ratings to courses that are intellectually challenging (Wallisch, P., & Cachia, J. Y. A., 2019, Determinants of perceived teaching quality: the role of divergent interpretations of expectations. PsyArXiv Preprints. doi:10.31234/osf.io/dsvgq).
3. There is evidence that ratings are affected by irrelevant characteristics of the lecturer: this literature is more inconsistent, but includes the finding that better ratings are given to instructors who award higher average grades (Blackhart, G. C, et al., 2006), Factors influencing teaching evaluations in higher education. Teaching of Psychology, 33, 37-39), and those that are rated as attractive (https://slate.com/technology/2018/04/hotness-affects-student-evaluations-more-than-gender.html).
4. Students’ confidence in the NSS has been damaged by previous attempts to link TEF to fee-raising powers, and response rates are now low. We gather the data will be used if there is a response rate of 50% or more, this raises concerns that respondents will not be representative of their cohort.

Continuation data are important. Courses with high drop-out rates should be flagged and investigated by QAA. You do not need TEF to do this.

The Destinations of Leavers survey can only ever provide an indicator of past performance. Available information is bound to be short-term and may not reflect typical longer-term outcomes. But if long-term outcomes were used, then they will not reflect the current course. So the relevance of the information is questionable.
In addition, employment outcomes of the graduates of an institution will depend on various factors, including their geographic location, course of study, socio-economic backgrounds, as well as changing economic circumstances. The relevant report from the Institute for Fiscal Studies is at https://www.ifs.org.uk/publications/8235. Furthermore, for some students, prospects for employment will be a key factor in deciding on a course, but others may be more concerned with what they will learn, rather than how much they will earn. To treat employment outcomes as a proxy for teaching quality is offensive to students and to academics.

15 The TEF metrics are benchmarked to account for factors such as the subject of study, prior attainment, ethnicity and educational disadvantage of the provider’s student intake (see that ‘What is TEF?’ section for detail).

No

b. Does TEF benchmark for the right factors?:

There are problems with the benchmarkings process in terms of (a) intelligibility; (b) coverage and (c) logic.
Intelligibility: The process was developed by Draper and Gittoes (2005) Statistical analysis of performance indicators in UK higher education. Journal of the Royal Statistical Society, Series A, 167, 449. You need an advanced degree in statistics to make sense of it. The resulting rankings have the potential to confuse students who are likely to assume that Gold/Silver/Bronze is a direct reflection of teaching quality. They would gain more meaningful information about courses from the raw data in the format provided on Unistats.

Coverage: The ability to detect whether the difference from benchmarked and obtained score is meaningful, rather than part of random variation, is heavily dependent on the sample size. This means that it is considerably easier for large institutions (with thousands of students) to obtain bronze or gold rankings than for medium-sized or small institution. Indeed, the same score that is flagged as average in a small institution could be flagged as seriously deficient in a large institution.
Logic: Potentially, benchmarking runs the risk of entrenching disadvantage, by creating the impression that a high drop-out rate is acceptable if students come from disadvantaged backgrounds. Or consider the use of subject of study as a benchmark: if we find that students of engineering have a 80% satisfaction rate on one of the NSS indicators, whereas students of medicine have a 90% satisfaction rate, we then adjust for this, which means that we’d regard 80% satisfaction as acceptable for an engineering student but evidence of ‘poor teaching quality’ for a medical student. But shouldn’t students be satisfied with their teaching regardless of the course of study?

HESA’s own documents note some of these problems:
‘It needs to be stressed that because a difference between HE providers may be accounted for by differences in the subject or entry qualification profiles of the HE providers this does not imply a justification of that difference. The purpose of the benchmarks is to allow any discussion of the reasons for the differences to be carried out on an informed basis.'(e.g. https://www.hesa.ac.uk/data-and-analysis/performance-indicators/benchmarks)

16 The TEF process uses both quantitative evidence (for example, the core metrics) and qualitative evidence (for example, the written submission).

a. What are your views about the balance of quantitative and qualitative evidence considered in arriving at ratings?:

As noted above, the quantitative information is misleading, because it is based on invalid and unreliable measures. Supplementing it with qualitative evidence doesn’t remove that problem. As soon as you move to using qualitative information, you introduce an element of subjectivity in evaluation, and raise the likelihood that powerful institutions with resources to employ people to write these statements will push up their award by telling convincing stories.

b. Are there any other aspects of the process that you wish to comment on?:

Difficulties with the statistical analysis used in TEF have been noted by the Royal Statistical Society and others, but so far ignored. (http://www.rss.org.uk/Images/PDF/influencing-change/2018/RSS-Evidence-Dept-Education-Teaching-Excellence-Framework_final-21May-2018.pdf). We trust that your statistical review will be done by competent statisticians who can evaluate these properly, otherwise there may be legal redress from universities that are disadvantaged by an invalid process.

The process is not transparent: although data are made available on the OfS website, this does not provide any scripts that allow one to trace the process of transforming the numbers from one stage to the next. Although student-level data are confidential, it should be possible to provide an analysis script for use with institutional level data, and/or provide a dummy dataset showing the workings from start to finish. It is possible for institutions to correct their own basic statistics, but it is hard to see how any errors or distortions in the process of computing benchmarked scores could be detected and corrected. This is inadequate by modern standards of data management.

Are the ratings right?

17 Are the purpose(s) of TEF met by:

No

Please explain your answer:

This kind of scale not only fails to provide students with useful information, but is positively misleading. The Times Higher Education reported on 31 January 2019, (p. 9) that a survey conducted on behalf of the DfE following the publication of the first set of TEF results in the summer of 2017 showed that only 43% of respondents had heard of the TEF at the time of application and only 15% referred to it when making their decisions about where to apply; two-thirds of those who had heard of the TEF thought that TEF scores were based on Ofsted-style inspections. (Their expectation of what happened at university level was evidently coloured by the regime that was familiar to them at school.) The report is at https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/771385/TEF_Year_2_Evaluation_Report__6_.pdf, and the relevant portion begins at p. 80.

No

Please explain your answer :

We need a diverse sector that has a range of institutions that cater for different needs in different ways. As well as conventional Universities that take mostly school-leavers, we need institutions that cater for the needs of mature and part-time students, and others focusing on specialist training, with a clear difference

between more academic and vocational subjects, There is no point of engaging in a vast bureaucratic exercise that maps individual strengths and weaknesses to a crude three-point scale – which cannot be transparently related back to the data used to generate it.
Including ‘provisional’ awards for those unable to be assessed risks the reputation of UK Higher Education, given that these awards have been given to some institutions with remarkably niche offerings, such as one, with a history of investigation by the UK Border Agency, which had only 12 of its 18 students complete a qualification in DJing and sound engineering.

No

Please explain your answer :

This kind of ranking has the potential to damage the reputation of our institutions overseas, by implying that ‘Bronze’ indicates a bad university. Given that all participating universities achieve at least bronze, nobody will believe that “bronze” in the TEF is the equivalent of bronze in the Olympics.

18 If you answered no, what alternatives you would suggest.

a. For provider-level TEF?:

None. We need to seriously question what purpose is served by having this this crude ranking applied at institutional level. Those reviewing TEF should be asking what would be lost by abandoning it.

b. For subject-level TEF?:

Although subject-level information is of far more interest to students than institutional-level information, TEF is not the instrument to provide this. The validity of the proxy indicators is already in question; poor reliability of the results will become overwhelmingly problematic when the method is extended to much smaller sample sizes. We’ll be in a situation where the responses of one or two students could influence a course’s award.

Alternatives to TEF
First, we agree that students need good information before embarking on a course, and they should be encouraged by their schools to seek this out. They can find specific indicators for the course they are considering on Unistats, presented in a way that encourages students to compare courses. This provides a personalised student-first way of communicating strengths and weaknesses of potential universities.

Second, the relative undervaluing of teaching relative to research in our universities is reflected in the rise of the ‘academic precariat’, which includes a swathe of teaching staff on insecure contracts. This is likely to damage the student experience. Rather than engaging in tortuous and logically dubious manipulations of results from proxy metrics, we should be providing information on who is doing the teaching, and what proportion of teaching is done by staff on long-term contracts. If this information were added to Unistats, then TEF would become obsolete, and universities would be incentivised to create more security for teaching staff.

c. If your previous response(s) reflected on the impact of the TEF on the international reputation of institutions and/or the UK as a whole, we would welcome any evidence or information you can provide that might support your view or help inform the independent review.:
Virtually every speech by every minister for Higher Education begins by praising the high esteem in which UK Higher Education is held internationally.
We risk damaging that by implying that some Bronze-rated HEIs are ‘falling short’ of expectations, while at the same time having a 4-point scale of evaluation that includes everything from ‘Point Blank Ltd’ (https://en.wikipedia.org/wiki/Point_Blank_Music_School) through to Cambridge University.

Has TEF changed anything?

19 Has the introduction of TEF positively changed the educational experience of students (e.g. teaching and learning)?

No

If yes, how?:

We asked for contributions to this question from our members.
None gave a positive response to this question – the likely changes in educational experiences were not ones of value.

Here is a typical response:
‘It would be an unwise vice-chancellor that invested money to improving the quality of teaching, given the data that this investment would have a small to negligible impact on the student assessment metrics. On the other hand, the business case for gaming the metrics is overwhelming. The data suggests that the cheapest and most effective ways to improve student assessment scores will be grade inflation, making courses easier, and providing pleasant snacks before the students fill in the NSS (Hessler et al, 2018). These have large impacts on student assessment scores. We have seen all these measures in action in our universities.’
Hessler, M., et al. (2018). Availability of cookies during an academic course session affects evaluation of teaching. Medical Education, 52, 1064-1072. doi:10.1111/medu.13627

20 Has the introduction of TEF negatively changed the educational experience of students (e.g. teaching and learning)?

Yes

If yes, how?:

Here is a selection of points made by our members:
‘The emphasis TEF places on metrics means that reductionistic focus/bureaucracy creeps into staff-student interaction’
‘Diversion of resources away from teaching towards “gaming” metrics. Grade inflation to improve “value added” degrades quality of education.
‘By responding to consumer demand (more contact time, more lectures, more classes, more summative assessment, less reliance on final exam) we are unwittingly encouraging a passive educational model, in which the student/consumer is the recipient of a service called teaching. The independent active learner

model which has prevailed in the universities hitherto is now under threat.’
‘Teaching has become even more bureaucratic, with peripheral so-called “quality assurance'”processes further increased, which take time and energy away from actual teaching.’
‘It encourages students to see themselves as consumers rather than learners. It has placed additional stresses and administration on teaching staff.’
‘”Customer satisfaction” is now the name of the game, with anything recognisably educational almost entirely lost. ‘

21 Has the introduction of TEF impacted positively on research and/or knowledge transfer?

No

If yes, how?:

No.
One of our members summed up the views of many as follows:

‘Research and teaching at university level are integral parts of academic life and are not mutually exclusive nor can sensibly be divided in the way TEF and REF require. To give an illustration – a large research project in collaboration with industry will have theoretical and applied aspects that feed directly into scientific publications and teaching (syllabus development, teaching modules, learning outcomes that reflect both theoretical and applied requirements for student’s acquired knowledge and skills). The feedback from industrial partners, researchers who read our papers and from students, help us to develop new research programmes, apply for innovative research projects and generate research outcomes, which in turn result in scientific publications, teaching, new forms of academic and non-academic collaboration.

In contrast, we now have REF and TEF – and the difference between them is very damaging as it ignores the integrity of real scholarly practice.

In addition, the current arbitrary distinctions between research, teaching and other scholarly activities are in direct conflict with both academic contracts of employment and academic practice which have created one of the best (if not THE best) system of higher education and scientific research in the world.’

22 Has the introduction of TEF impacted negatively on research and/or knowledge transfer?

Yes

If yes, how? :

Again, a selection of points were made by our members
‘Insofar as TEF forces us to concentrate on teaching/ gaming metrics, without the provision of additional resources, it can only reduce the resources available to conduct research/knowledge transfer.’
‘Teaching and research now seen as in competition with one another, rather than as an integrated part of academic experience’
‘To measure and improve ‘performance’ we are now auditing more and more activities, reducing the time available for research and scholarship.’
‘It eats into teaching, and thus into research, time. More importantly, it reinforces the ideology of REF by imposing that ideology into teaching (or rather, customer satisfaction maintenance).’

Is TEF worth it?

23 Does TEF help you as a student/provider/employer?

No

Please explain the reasons for your answer.:

We find it telling that your question does not include teaching staff in the list of potential respondents, although change in teaching experience depends crucially on us. We do feel treated as pawns in this exercise.
The TEF exacerbates tensions between academics and their employers, by encouraging a mistaken managerial view in which teaching staff need to be endlessly monitored because they will avoid doing their jobs properly unless there are external incentives to drive them.

Given that most of us regard our academic jobs as a vocation, value our students, and hold ourselves to account for teaching quality, there is widespread distrust of the whole process.

24 Explaining your reasoning, what are the most significant costs of:

a. Provider-level TEF?:

A shift in the teacher-student relationship to one of consumer-provider
Damage to the international reputation of Universities
Further emphasis on competition rather than collaboration between institutions
Further emphasis on a single dimension of ‘excellence’, which ignores the need for a wide range of different types of institution performing different roles in the sector
Adding to the bureaucratic and managerial pressures on academic staff. The adverse effects on staff morale are explicitly noted on p. 11 of the Year 2 Evaluation at https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/771385/TEF_Year_2_Evaluation_Report__6_.pdf

b. Subject-level TEF?:

A waste of time and energy to generate statistics that are bound, given the small numbers in each course, to be meaningless.

25 Explaining your reasoning, what are the most significant benefits of:

a. Provider-level TEF?:

For the reasons given in response to questions 10, 14, 15, 16, 20 and 22 above, we see no benefits.

b. Subject-level TEF?:

None

Is TEF fair?

26 Are there particular types of students, provision or providers that are disadvantaged by the current design of TEF, in a disproportionate way?

No

If so, what changes could be made to address this?:

As noted above (q 15), there is a risk of entrenching disadvantage, by creating the impression that a high drop-out rate or low level of satisfaction is acceptable if students come from disadvantaged backgrounds, or pursue particular types of course.

Students could be discouraged from pursuing careers that are low-paying but socially valuable, as this would impact negatively on the institution’s TEF score. Removing outcomes data from TEF ratings would address this.

27 Are there particular types of students, provision or providers that are advantaged by the current design of TEF, in a disproportionate way?

No

If so, what changes could be made to address this?:

We can see no benefits of TEF for anyone involved in higher education.
It uses invalid metrics to generate misleading information via a non-transparent process. It does not provide the information students need. It diverts attention and resources from actual teaching and creates further unnecessary and counterproductive managerialism in the sector.

Changes: encourage teachers and careers’ advisors to direct sixth-formers to Unistats to seek out comparative information on courses.