Opinion piece by Dorothy Bishop

The Higher Education and Research Bill (HERB) has had a rocky passage through parliament. As explained here, a Bill goes through several stages before it becomes law, with debate in both the House of Commons and the House of Lords, as well as Committee work that involves detailed line-by-line scrutiny of the text.  The Lords are primarily the revising chamber and they have passed a series of major amendments to HERB. The government has incorporated some of these amendments, but on its return to the Commons, the Bill may lose the Lords’ changes, and, as discussed in the Times Higher, there could be some to-and-fro before the Commons votes through a final version.

We have featured a series of posts on this blog discussing the amendments that have been proposed over the past two months:

• The Haldane Principle: remembering Fisher and getting that definition right
• Rival sanctions? The Office for Students and the Competition and Markets Authority
• New DAPs anyone? Is the proposed multiplication of types of degree-awarding powers a good idea?
• A ‘cohesive and self-critical academic community': an ideal under threat
• Update on amendments to the Higher Education and Research Bill

One topic that particularly exercised the Lords was the new Teaching Excellence Framework (TEF). Introduction of the TEF did not require legislation: it was a manifesto commitment. Specifically, the Conservative manifesto stated: ” “We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality; encourage universities to offer more two-year courses; and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”. Accordingly, the TEF is already underway in a preliminary form. Nevertheless, numerous objections have been raised to it. Many of these were stated in the response to the initial Green Paper introducing the idea, including our own response. Yet, as documented in an earlier post, these objections were largely ignored. However, the Lords, who include many people with considerable experience in Higher Education, continued to have concerns.

At last, it seems, the Government are taking the concerns seriously. Jo Johnson, the Minister responsible for HERB, and Viscount Younger of Leckie, a Conservative Peer who is the Lords Spokesperson for Higher Education, have written a letter on 3^rd March to offer clarification of questions that had been raised about the TEF in the House of Lords.

As someone who has voiced repeated concerns about the TEF, I offer here an analysis of the points made in that letter. For detailed sources please see these slides from a lecture on this topic.

Page 1. Para 2a states ‘The TEF is essential to driving up standards of teaching. All those who spoke during the Committee stages of the Bill agreed that teaching quality is of paramount importance.’

We can all agree that teaching quality is of paramount importance in our Universities – this is motherhood and apple pie. However, the statement that the TEF is essential to drive up teaching standards contains two shaky premises: that standards need ‘driving up’, and that the TEF would achieve this. We only need to introduce sector-wide measures to address teaching if there is evidence it is inadequate. I have argued elsewhere that this case has never been made. In a speech in 9^th September, Johnson talked of ‘patchiness in the student experience’ but presented only anecdote to support his case. In the Green Paper, it was claimed that both students and employers were dissatisfied with teaching, yet the evidence was cherry-picked and mis-represented the sources from which it came.

The Government is fond of treating Higher Education as a market, yet their ideas for improving the market are ones that few businesses would adopt. Anyone operating a widget factory knows that to maintain quality control, you don’t give every widget a detailed inspection: you adopt a process of sampling a small proportion, so you can be ready to check the production process if you find an unacceptable level of problems. Even Jo Johnson accepts that our Universities are world-leading, yet he wants to impose a further burden of evaluation to complement the Research Excellence Framework (REF). This will take money and resources from research and teaching and so be counterproductive. I would suggest that instead of a TEF we need:

• A system of quality control that will scrutinise any institutions that show signs of failing (e.g. because of high level of complaints). We already have this with the Quality Assurance Agency (QAA).
• Incorporation of some information about teaching quality in the REF Impact section – see this proposal by Roger Brown.
• Disincentives for Universities to use zero-hours contract staff for teaching; this could include a section on staff well-being and moral in the Environment section of the REF.
• Incentives for sharing good and innovative teaching practices

Page 1 para 1b. ‘The TEF will provide the financial and reputational incentives for providers to prioritise teaching excellence and student outcomes. Genuine and clear differentiation between providers and a link with fees are essential to achieving these outcomes.‘

The downside of yet another ranking system are simply ignored here. What is needed is a diverse higher education system, with different institutions developing their own specialised approaches to suit the very varied needs of our population. A crude ranking, further subdivided into Gold, Silver and Bronze, treats higher education as some kind of horse race, where quality can be measured on one dimension.

Page 1 para 2-3 notes that ‘It is important that we treat fairly those who have already decided to take part….The assessment process is already well underway and to change the “rules of the game”… would be extremely unfair on those who have invested their time and effort to participate this year’

I’m tempted to respond, ‘Well, whose fault is that, Minister’. It is Johnson who has chosen to implement an ill-thought-through evaluation system, with threats to participating organisations that they will lose their ability to raise fees unless they take part. And he also promised that TEF would not be onerous. But now we are told that those institutions who have taken part would have invested time and effort and so it would be unfair to stop. One is reminded of someone who orders a meal that makes them sick but persists in eating it, because the money has been spent.

What is really unfair is to adopt a system of evaluation that is based on unsuitable metrics, is incapable of making meaningful distinctions between institutions, yet which has the potential to damage the standing of a University.

Page 2, para 2. ‘We recognise the genuine and considered concerns raised by noble Lords about the TEF, in particular around the speed at which it is being implemented and on the use of metrics and ratings….a genuine lessons-learned exercise will take place after this trial year.’

Well, at last it seems there is recognition that the TEF in its current form is unworkable. Criticism has come from individuals who might be expected to support it; see e.g. this account of an interview with Chris Husbands, Chair of the TEF Panel on Jan 25^th in the Times Higher:

‘“I do not think student satisfaction is an accurate proxy for teaching quality,” said Professor Husbands, a former director of the UCL Institute of Education who was named chair of the TEF panel by the Department for Business, Energy and Industrial Strategy in August 2016.

Professor Husbands drew attention to US studies that show that student satisfaction scores are heavily influenced by non-academic factors, including the sex or ethnicity of a lecturer.

“Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” he said of the US research findings.’

And Stuart Croft, VC at Warwick on Jan 31^st wrote in the Times Higher ’…the metrics are flawed. This is not renegade opinion, but rather the overwhelming view of those actually involved in higher education….Yet the government has us over a barrel. It has linked the TEF to tuition fees and, potentially, our ability to recruit international students.’

Page 3. Para 1 states that ‘…the TEF is about much more than metrics. Providers submit additional evidence alongside their metrics and this evidence will be given significant weight by the panel.’

But what is the reliability and validity of the ‘additional evidence’? This seems like encouragement to Universities to tell plausible stories about their teaching and try to anticipate which buttons they need to hit to convince a panel that they are Gold rather than Silver and Bronze. There is huge scope here for subjectivity at best and corruption at worst.

Page 3. Para 2 state ‘All of the metrics used in the TEF are trusted, widely used and well established in the sector….. The NSS is just one of three principal sources of metrics data being used, and we have explicitly said that the NSS metrics are the least important. The TEF does not use in any way the overall ‘satisfaction’ rating, about which the House has rightly expressed concern, but instead uses specific questions from the NSS, related to teaching and learning.’

It’s hard to know where to start with this. There are numerous issues with using the NSS for assessing teaching quality, and changing the items that are used does not solve them. There are problems with validity – does the measure assess what it is supposed to measure, bias – are ratings affected by the teacher’s gender, race or disability, and sensitivity – is the spread of scores adequate for differences between institutions to be meaningful? Statistical criticisms have been made by both the Royal Statistical Society and the Office of National Statistics. We are told in para 2 that ‘the Government has already taken a number of steps to ensure that the statistics being used are robust, including commissioning a report by the Office of National Statistics.’ They seem to think that the ONS will somehow be able to magically transform an unreliable, insensitive scale into a meaningful indicator of teaching quality. Statisticians are good, but not that good. Furthermore, all the statistical problems that afflict the TEF are magnified with small samples, and small samples are going to be inevitable if we move, as we have been told we must, to subject-specific assessments.

It is depressing, though not surprising, to find that the letter ends by telling us the university sector must be ‘shaken out of its complacency’. It’s an easy but lazy point to say that opposition to the TEF is just the response of a complacent group of academics who are set in their ways, because it means you don’t actually have to consider the arguments. I suspect that Johnson would be surprised to find just how passionately many of us care about our Higher Education system. If he could grasp that point, he might start to take note of us when we tell him his superficial and rushed approach to evaluating teaching is resisted because of the considerable potential it has for harm.