Government’s letter to the Lords re TEF: A reply

Opinion piece by Dorothy Bishop

The Higher Education and Research Bill (HERB) has had a rocky passage through parliament. As explained here, a Bill goes through several stages before it becomes law, with debate in both the House of Commons and the House of Lords, as well as Committee work that involves detailed line-by-line scrutiny of the text.  The Lords are primarily the revising chamber and they have passed a series of major amendments to HERB. The government has incorporated some of these amendments, but on its return to the Commons, the Bill may lose the Lords’ changes, and, as discussed in the Times Higher, there could be some to-and-fro before the Commons votes through a final version.

We have featured a series of posts on this blog discussing the amendments that have been proposed over the past two months:

One topic that particularly exercised the Lords was the new Teaching Excellence Framework (TEF). Introduction of the TEF did not require legislation: it was a manifesto commitment. Specifically, the Conservative manifesto stated: ” “We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality; encourage universities to offer more two-year courses; and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”. Accordingly, the TEF is already underway in a preliminary form. Nevertheless, numerous objections have been raised to it. Many of these were stated in the response to the initial Green Paper introducing the idea, including our own response. Yet, as documented in an earlier post, these objections were largely ignored. However, the Lords, who include many people with considerable experience in Higher Education, continued to have concerns.

At last, it seems, the Government are taking the concerns seriously. Jo Johnson, the Minister responsible for HERB, and Viscount Younger of Leckie, a Conservative Peer who is the Lords Spokesperson for Higher Education, have written a letter on 3rd March to offer clarification of questions that had been raised about the TEF in the House of Lords.

As someone who has voiced repeated concerns about the TEF, I offer here an analysis of the points made in that letter. For detailed sources please see these slides from a lecture on this topic.

Page 1. Para 2a states ‘The TEF is essential to driving up standards of teaching. All those who spoke during the Committee stages of the Bill agreed that teaching quality is of paramount importance.’

We can all agree that teaching quality is of paramount importance in our Universities – this is motherhood and apple pie. However, the statement that the TEF is essential to drive up teaching standards contains two shaky premises: that standards need ‘driving up’, and that the TEF would achieve this. We only need to introduce sector-wide measures to address teaching if there is evidence it is inadequate. I have argued elsewhere that this case has never been made. In a speech in 9th September, Johnson talked of ‘patchiness in the student experience’ but presented only anecdote to support his case. In the Green Paper, it was claimed that both students and employers were dissatisfied with teaching, yet the evidence was cherry-picked and mis-represented the sources from which it came.

The Government is fond of treating Higher Education as a market, yet their ideas for improving the market are ones that few businesses would adopt. Anyone operating a widget factory knows that to maintain quality control, you don’t give every widget a detailed inspection: you adopt a process of sampling a small proportion, so you can be ready to check the production process if you find an unacceptable level of problems. Even Jo Johnson accepts that our Universities are world-leading, yet he wants to impose a further burden of evaluation to complement the Research Excellence Framework (REF). This will take money and resources from research and teaching and so be counterproductive. I would suggest that instead of a TEF we need:

  • A system of quality control that will scrutinise any institutions that show signs of failing (e.g. because of high level of complaints). We already have this with the Quality Assurance Agency (QAA).
  • Incorporation of some information about teaching quality in the REF Impact section – see this proposal by Roger Brown.
  • Disincentives for Universities to use zero-hours contract staff for teaching; this could include a section on staff well-being and moral in the Environment section of the REF.
  • Incentives for sharing good and innovative teaching practices

Page 1 para 1b. ‘The TEF will provide the financial and reputational incentives for providers to prioritise teaching excellence and student outcomes. Genuine and clear differentiation between providers and a link with fees are essential to achieving these outcomes.

The downside of yet another ranking system are simply ignored here. What is needed is a diverse higher education system, with different institutions developing their own specialised approaches to suit the very varied needs of our population. A crude ranking, further subdivided into Gold, Silver and Bronze, treats higher education as some kind of horse race, where quality can be measured on one dimension.

Page 1 para 2-3 notes that ‘It is important that we treat fairly those who have already decided to take part….The assessment process is already well underway and to change the “rules of the game”… would be extremely unfair on those who have invested their time and effort to participate this year’

I’m tempted to respond, ‘Well, whose fault is that, Minister’. It is Johnson who has chosen to implement an ill-thought-through evaluation system, with threats to participating organisations that they will lose their ability to raise fees unless they take part. And he also promised that TEF would not be onerous. But now we are told that those institutions who have taken part would have invested time and effort and so it would be unfair to stop. One is reminded of someone who orders a meal that makes them sick but persists in eating it, because the money has been spent.

What is really unfair is to adopt a system of evaluation that is based on unsuitable metrics, is incapable of making meaningful distinctions between institutions, yet which has the potential to damage the standing of a University.

Page 2, para 2. ‘We recognise the genuine and considered concerns raised by noble Lords about the TEF, in particular around the speed at which it is being implemented and on the use of metrics and ratings….a genuine lessons-learned exercise will take place after this trial year.’

Well, at last it seems there is recognition that the TEF in its current form is unworkable. Criticism has come from individuals who might be expected to support it; see e.g. this account of an interview with Chris Husbands, Chair of the TEF Panel on Jan 25th in the Times Higher:

‘“I do not think student satisfaction is an accurate proxy for teaching quality,” said Professor Husbands, a former director of the UCL Institute of Education who was named chair of the TEF panel by the Department for Business, Energy and Industrial Strategy in August 2016.

Professor Husbands drew attention to US studies that show that student satisfaction scores are heavily influenced by non-academic factors, including the sex or ethnicity of a lecturer.

“Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” he said of the US research findings.’

And Stuart Croft, VC at Warwick on Jan 31st wrote in the Times Higher ’…the metrics are flawed. This is not renegade opinion, but rather the overwhelming view of those actually involved in higher education….Yet the government has us over a barrel. It has linked the TEF to tuition fees and, potentially, our ability to recruit international students.

Page 3. Para 1 states that ‘…the TEF is about much more than metrics. Providers submit additional evidence alongside their metrics and this evidence will be given significant weight by the panel.’

But what is the reliability and validity of the ‘additional evidence’? This seems like encouragement to Universities to tell plausible stories about their teaching and try to anticipate which buttons they need to hit to convince a panel that they are Gold rather than Silver and Bronze. There is huge scope here for subjectivity at best and corruption at worst.

Page 3. Para 2 state ‘All of the metrics used in the TEF are trusted, widely used and well established in the sector….. The NSS is just one of three principal sources of metrics data being used, and we have explicitly said that the NSS metrics are the least important. The TEF does not use in any way the overall ‘satisfaction’ rating, about which the House has rightly expressed concern, but instead uses specific questions from the NSS, related to teaching and learning.’

It’s hard to know where to start with this. There are numerous issues with using the NSS for assessing teaching quality, and changing the items that are used does not solve them. There are problems with validity – does the measure assess what it is supposed to measure, bias – are ratings affected by the teacher’s gender, race or disability, and sensitivity – is the spread of scores adequate for differences between institutions to be meaningful? Statistical criticisms have been made by both the Royal Statistical Society and the Office of National Statistics. We are told in para 2 that ‘the Government has already taken a number of steps to ensure that the statistics being used are robust, including commissioning a report by the Office of National Statistics.’ They seem to think that the ONS will somehow be able to magically transform an unreliable, insensitive scale into a meaningful indicator of teaching quality. Statisticians are good, but not that good. Furthermore, all the statistical problems that afflict the TEF are magnified with small samples, and small samples are going to be inevitable if we move, as we have been told we must, to subject-specific assessments.

It is depressing, though not surprising, to find that the letter ends by telling us the university sector must be ‘shaken out of its complacency’. It’s an easy but lazy point to say that opposition to the TEF is just the response of a complacent group of academics who are set in their ways, because it means you don’t actually have to consider the arguments. I suspect that Johnson would be surprised to find just how passionately many of us care about our Higher Education system. If he could grasp that point, he might start to take note of us when we tell him his superficial and rushed approach to evaluating teaching is resisted because of the considerable potential it has for harm.

 

 

 

 

 

 

 

Universities as cartels?

Opinion piece

by James Ladyman

It is said that trust in institutions is in short supply these days. Sadly the government doesn’t trust universities. According to the Minister, the Higher Education and Research Bill is necessary to “break open a closed shop that for too long has set the rules of the game in its own interests”. He does not believe that academics who act as external examiners and universities that oversee the provision of teaching make and apply rules in the interests of academic standards and education, nor that their deliberations are informed by the wider political and social good. According to the minister, and one must presume the entire government and many MPs and peers, universities have been acting in their own interests all this time. This makes sense to them because they also insist that universities are businesses and encourage them to behave as such. Businesses’ primary interest is in maximizing their revenues. Since, universities are businesses their primary interest must be in maximising their revenues.

The new providers that are allegedly needed to drive up teaching standards – the minister is disparaging about the latter – do indeed include businesses interested only in making a profit. They contrast with universities that hitherto have hosted academics who collaborate across institutions for the good of their disciplines. Yet the minister thinks of the sector as ‘cartel-like’.

We all know how effective cartels allow prices to be kept artificially high from the way that energy market has functioned. The measures in the Bill are supposed to protect students from a similar cartel-like scam in HE. It will supposedly do this by allowing fees to be raised. Yes, that’s right folks, the market will save students from being ripped off by the existing university system, by allowing universities to raise prices.

Despite the disastrous effects of marketisation on the health service and the penal system, and with no regard to the outstanding international reputation of British higher education, the HE Bill proposes to treat the healthy patient with bad medicine.

On the subject of the closed shop, one might as well ask why we let the medical profession control who becomes a doctor rather than the market. We don’t expect patient choice to set standards in clinical care, and we should no more expect student choice to set standards in higher education.

We have not been given any explanation for why the idea of the self-critical academic community enshrined in established thinking about academic and educational standards has been set aside so completely.

Students are entitled to know that their fees are paying for a decent standard of education, but this government is determined to sacrifice them in the interests of the profits that the new providers will make out of their tax-payer funded loans.

 

Perils of ignoring consultation on the Higher Education and Research Bill

Opinion piece by Dorothy Bishop

Earlier this week, Lord Patten, former Conservative chairman, published a searing attack on the government’s Higher Education and Research Bill in the Observer, stating:

“It seems ham-fisted to turn the academic world upside down when universities face so much turbulence and uncertainty after the Brexit vote and the rhetoric surrounding immigration.”

He was supported by a cross-party group of peers, including Labour’s shadow higher education minister Lord Stevenson, who stated:

“This bill is an attempt from the Government to run a market experiment through the bloodstream of our university system, and a classic case of understanding the cost of everything and the value of nothing.”

These comments are closely aligned with the CDBU’s evaluation of the Bill, and we are pleased to see a list of 516 Amendments to the Bill has now been proposed by the Lords.

We have to ask how on earth the situation has arisen whereby a Bill is not only being comprehensively revised at this late stage in its progress, but is also receiving criticism from heavyweight Conservatives who might have been expected to support the Government.

The problem appears to be the intransigence of the Minister, Jo Johnson, who has  ignored serious arguments against his Bill, which have been flagged up many times over the last year.

Let’s start by looking at the responses to the Green Paper which outlined the main proposals in the Bill. The volume of responses to the consultation was quite remarkable: there were 618 respondees in all, including 136 Higher Education Institutions, 26 alternative providers, 22 further education colleges, 78 student unions, and a large number of individuals and organisations with a stake in the sector.

These responses were summarised in a report by BIS. In addition, many institutions posted their response on the web, including the Universities of Cambridge and Oxford: I’ve singled these out because they are our most ancient universities, and might be expected to represent an establishment view. Critical comments in their feedback anticipated the points now being raised in the House of Lords, and might have been seen as a canary in the mine by a Minister less self-assured than Jo Johnson. Consider this comment from the preamble to the response by the University of Cambridge:

“The Green Paper fails to demonstrate an understanding of the purpose of our universities and the reasons for the sector’s international standing. Universities exist to pursue knowledge for the benefit of society, through education (teaching and learning), scholarship and research.”

Are such objections restricted to the Russell Group? It would appear not. The summary of the consultation findings by BIS does not break down responses according to their origins, but it does provide quantitative data for several items where respondents were asked to state Agree/Not Sure/Disagree. These had a worrying high level of Not Sure responses, suggesting that the proposals in the Green Paper were too vague or unclear to evaluate; in addition, the rates of Disagree were higher than Agree for many items – only 6 of 24 items attracted more than 50% agreement. And where Agree outnumbered Disagree, it tended to be for questions concerning general ideals, such as the desirability of improving access for disadvantaged groups; questions about specifics of implementation of proposals received a higher rate of negative responses.  The two items marked with * concerned the link between TEF results and fees, and were among those with lowest levels of endorsement.

screen-shot-2017-01-07-at-09-11-40Figure 1: Proportions of Agree/Not Sure/Disagree to items included in the Green Paper consultation (See report for details of items; number in bracket refers to consultation question number).

There was also a technical consultation on the TEF, which fared no better. The Office of National Statistics expressed concern about the suitability of the metrics to be used to evaluate teaching quality, and the Royal Statistical Society witheringly noted:

“It is concerning that Figure 4 equates student satisfaction as measured by the National Student Survey (NSS) with Teaching Quality. We are not aware that there is any evidence of a statistical association between the two concepts.”

Given this level of concern, one might have expected some radical revisions between the Green and White Papers. In fact, there were few changes, except that the TEF, which had been a major (and much-criticised) feature of the Green Paper was removed from the White Paper. Because it does not require legislation to introduce the TEF, it was possible to keep its implementation separate from the rest of the Bill, so it could be steamrollered through ready for introduction in 2017.

There were further opportunities for the Bill to be amended as it proceeded through the Committee stage in Parliament. And Jo Johnson described himself as listening to feedback, saying:

During September and October, a cross-party committee of MPs scrutinised the Bill, along with over 300 additional tabled amendments. We heard evidence from a wide range of witnesses from university vice-chancellors and the National Union of Students to the head of Research Councils UK and consumer groups such as Which? And now we’re at Report Stage, you will see that we have reflected on these views.

Except that none of the 300 tabled amendments were included! A handful of modifications were made to some statements to increase clarity and avoid ambiguity, but the core plans remained unchanged.

The House of Lords includes peers of all political stripes, but a key point is that many of them have extensive experience of the HE sector. They have run universities, sat on Councils, and they understand about all kinds of higher and further education, not just Oxbridge. Similarly, the CDBU has members covering a wide political spectrum, who are not always in agreement about issues affecting universities. The Higher Education and Research Bill has, however, created a united front: left- and right-leaning academics agree that the Bill shows little understanding of the nature and purpose of Higher Education and risks damaging the reputation and autonomy of our universities.

In future, if a Minister invites comments on a consultation document, he might be well advised to take notice of the responses, to avoid the kind of embarrassment that Johnson is now confronted with.

Placing teaching and research in different Departments of State: the funding implications

No. 5 in a series of guest posts by G. R. Evans

One of the most significant regulatory changes introduced by the Bill is the decisive separation of research done by a research-active higher education provider in England from its ‘educational’ or teaching work. This has now been reinforced by the decision to move teaching into the Department for Education. Matters affecting the research activities of those providers involved in research (a proportion of the publicly-funded universities) are to remain in a revised Department for Business, the new Department of Business, Energy and Industrial Strategy (BEIS).

Jo Johnson as Minister of State for Universities and Science will hold his post jointly in the two Departments.

dfe-and-bis_a

So teaching will be under the control of the Office for Students (OfS) which will also have the power to register or de-register a provider, to grant or remove degree-awarding powers and to grant or remove the right to use ‘university title’. The OfS will be overseen by the Department of Education (DfE).

Research funding will come under the control of UK Research and Innovation (UKRI), overseen by the new Department of Business, Energy and Industrial Strategy (BEIS).

UKRI

The Higher Education and Research Bill ss.83-103 will set up a UK Research and Innovation body incorporating the Research Councils currently responsible for project funding. The infrastructure element remaining as block grant direct public funding, will be distributed through a new body ‘Research England’. This will take over the part of the infrastructure function of HEFCE which currently funds research, but it will no longer be a separate entity. It will disburse infrastructure funding within UKRI.

Continue reading