Government’s letter to the Lords re TEF: A reply

Opinion piece by Dorothy Bishop

The Higher Education and Research Bill (HERB) has had a rocky passage through parliament. As explained here, a Bill goes through several stages before it becomes law, with debate in both the House of Commons and the House of Lords, as well as Committee work that involves detailed line-by-line scrutiny of the text.  The Lords are primarily the revising chamber and they have passed a series of major amendments to HERB. The government has incorporated some of these amendments, but on its return to the Commons, the Bill may lose the Lords’ changes, and, as discussed in the Times Higher, there could be some to-and-fro before the Commons votes through a final version.

We have featured a series of posts on this blog discussing the amendments that have been proposed over the past two months:

One topic that particularly exercised the Lords was the new Teaching Excellence Framework (TEF). Introduction of the TEF did not require legislation: it was a manifesto commitment. Specifically, the Conservative manifesto stated: ” “We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality; encourage universities to offer more two-year courses; and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”. Accordingly, the TEF is already underway in a preliminary form. Nevertheless, numerous objections have been raised to it. Many of these were stated in the response to the initial Green Paper introducing the idea, including our own response. Yet, as documented in an earlier post, these objections were largely ignored. However, the Lords, who include many people with considerable experience in Higher Education, continued to have concerns.

At last, it seems, the Government are taking the concerns seriously. Jo Johnson, the Minister responsible for HERB, and Viscount Younger of Leckie, a Conservative Peer who is the Lords Spokesperson for Higher Education, have written a letter on 3rd March to offer clarification of questions that had been raised about the TEF in the House of Lords.

As someone who has voiced repeated concerns about the TEF, I offer here an analysis of the points made in that letter. For detailed sources please see these slides from a lecture on this topic.

Page 1. Para 2a states ‘The TEF is essential to driving up standards of teaching. All those who spoke during the Committee stages of the Bill agreed that teaching quality is of paramount importance.’

We can all agree that teaching quality is of paramount importance in our Universities – this is motherhood and apple pie. However, the statement that the TEF is essential to drive up teaching standards contains two shaky premises: that standards need ‘driving up’, and that the TEF would achieve this. We only need to introduce sector-wide measures to address teaching if there is evidence it is inadequate. I have argued elsewhere that this case has never been made. In a speech in 9th September, Johnson talked of ‘patchiness in the student experience’ but presented only anecdote to support his case. In the Green Paper, it was claimed that both students and employers were dissatisfied with teaching, yet the evidence was cherry-picked and mis-represented the sources from which it came.

The Government is fond of treating Higher Education as a market, yet their ideas for improving the market are ones that few businesses would adopt. Anyone operating a widget factory knows that to maintain quality control, you don’t give every widget a detailed inspection: you adopt a process of sampling a small proportion, so you can be ready to check the production process if you find an unacceptable level of problems. Even Jo Johnson accepts that our Universities are world-leading, yet he wants to impose a further burden of evaluation to complement the Research Excellence Framework (REF). This will take money and resources from research and teaching and so be counterproductive. I would suggest that instead of a TEF we need:

  • A system of quality control that will scrutinise any institutions that show signs of failing (e.g. because of high level of complaints). We already have this with the Quality Assurance Agency (QAA).
  • Incorporation of some information about teaching quality in the REF Impact section – see this proposal by Roger Brown.
  • Disincentives for Universities to use zero-hours contract staff for teaching; this could include a section on staff well-being and moral in the Environment section of the REF.
  • Incentives for sharing good and innovative teaching practices

Page 1 para 1b. ‘The TEF will provide the financial and reputational incentives for providers to prioritise teaching excellence and student outcomes. Genuine and clear differentiation between providers and a link with fees are essential to achieving these outcomes.

The downside of yet another ranking system are simply ignored here. What is needed is a diverse higher education system, with different institutions developing their own specialised approaches to suit the very varied needs of our population. A crude ranking, further subdivided into Gold, Silver and Bronze, treats higher education as some kind of horse race, where quality can be measured on one dimension.

Page 1 para 2-3 notes that ‘It is important that we treat fairly those who have already decided to take part….The assessment process is already well underway and to change the “rules of the game”… would be extremely unfair on those who have invested their time and effort to participate this year’

I’m tempted to respond, ‘Well, whose fault is that, Minister’. It is Johnson who has chosen to implement an ill-thought-through evaluation system, with threats to participating organisations that they will lose their ability to raise fees unless they take part. And he also promised that TEF would not be onerous. But now we are told that those institutions who have taken part would have invested time and effort and so it would be unfair to stop. One is reminded of someone who orders a meal that makes them sick but persists in eating it, because the money has been spent.

What is really unfair is to adopt a system of evaluation that is based on unsuitable metrics, is incapable of making meaningful distinctions between institutions, yet which has the potential to damage the standing of a University.

Page 2, para 2. ‘We recognise the genuine and considered concerns raised by noble Lords about the TEF, in particular around the speed at which it is being implemented and on the use of metrics and ratings….a genuine lessons-learned exercise will take place after this trial year.’

Well, at last it seems there is recognition that the TEF in its current form is unworkable. Criticism has come from individuals who might be expected to support it; see e.g. this account of an interview with Chris Husbands, Chair of the TEF Panel on Jan 25th in the Times Higher:

‘“I do not think student satisfaction is an accurate proxy for teaching quality,” said Professor Husbands, a former director of the UCL Institute of Education who was named chair of the TEF panel by the Department for Business, Energy and Industrial Strategy in August 2016.

Professor Husbands drew attention to US studies that show that student satisfaction scores are heavily influenced by non-academic factors, including the sex or ethnicity of a lecturer.

“Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” he said of the US research findings.’

And Stuart Croft, VC at Warwick on Jan 31st wrote in the Times Higher ’…the metrics are flawed. This is not renegade opinion, but rather the overwhelming view of those actually involved in higher education….Yet the government has us over a barrel. It has linked the TEF to tuition fees and, potentially, our ability to recruit international students.

Page 3. Para 1 states that ‘…the TEF is about much more than metrics. Providers submit additional evidence alongside their metrics and this evidence will be given significant weight by the panel.’

But what is the reliability and validity of the ‘additional evidence’? This seems like encouragement to Universities to tell plausible stories about their teaching and try to anticipate which buttons they need to hit to convince a panel that they are Gold rather than Silver and Bronze. There is huge scope here for subjectivity at best and corruption at worst.

Page 3. Para 2 state ‘All of the metrics used in the TEF are trusted, widely used and well established in the sector….. The NSS is just one of three principal sources of metrics data being used, and we have explicitly said that the NSS metrics are the least important. The TEF does not use in any way the overall ‘satisfaction’ rating, about which the House has rightly expressed concern, but instead uses specific questions from the NSS, related to teaching and learning.’

It’s hard to know where to start with this. There are numerous issues with using the NSS for assessing teaching quality, and changing the items that are used does not solve them. There are problems with validity – does the measure assess what it is supposed to measure, bias – are ratings affected by the teacher’s gender, race or disability, and sensitivity – is the spread of scores adequate for differences between institutions to be meaningful? Statistical criticisms have been made by both the Royal Statistical Society and the Office of National Statistics. We are told in para 2 that ‘the Government has already taken a number of steps to ensure that the statistics being used are robust, including commissioning a report by the Office of National Statistics.’ They seem to think that the ONS will somehow be able to magically transform an unreliable, insensitive scale into a meaningful indicator of teaching quality. Statisticians are good, but not that good. Furthermore, all the statistical problems that afflict the TEF are magnified with small samples, and small samples are going to be inevitable if we move, as we have been told we must, to subject-specific assessments.

It is depressing, though not surprising, to find that the letter ends by telling us the university sector must be ‘shaken out of its complacency’. It’s an easy but lazy point to say that opposition to the TEF is just the response of a complacent group of academics who are set in their ways, because it means you don’t actually have to consider the arguments. I suspect that Johnson would be surprised to find just how passionately many of us care about our Higher Education system. If he could grasp that point, he might start to take note of us when we tell him his superficial and rushed approach to evaluating teaching is resisted because of the considerable potential it has for harm.

 

 

 

 

 

 

 

Update on Amendments to the Higher Education and Research Bill

David Midgley has kindly provided this useful summary of recent developments:

The text of the Bill as amended by the Lords in report stage is now available at https://www.publications.parliament.uk/pa/bills/lbill/2016-2017/0112/17112.pdf.

The most significant changes with regard to the regime proposed for higher education are these:

  • Lord Kerslake’s amendment of 6 March to the effect that the TEF “must not be used to rank English higher education providers as to the regulated course fees they charge”, carried by 263 votes to 211, which is now Clause 12;
  • Lord Blunkett’s amendment of 8 March, carried by 280 votes to 186, which requires any “scheme to assess and provide consistent and reliable information about the quality of education and teaching” to be subject to rigorous independent scrutiny, and precludes it from being “used to create a single composite ranking of English higher education providers” – this has become Clause 27;
  • Baroness Wolf’s amendment of 8 March, carried by 201 votes to 186, which makes explicit the role of the Quality Assessment Committee in relation to the authorisation of any new provider and requires such a provider to have been “established for a minimum of four years with satisfactory validation arrangements in place” before authorisation may be issued – this has been incorporated into Clause 47 (as sub-sections 12 and 13); and
  • Lord Judge’s amendment of 8 March, carried by 185 votes to 151, which has the effect of strengthening the power of appeal against a revocation of degree-awarding powers and/or university title – the relevant clause is now Clause 49.
  • Also of considerable importance for the future of universities is Lord Hannay’s amendment of 13 March, carried by a noteworthy 313 votes to 219, which seeks to protect the position of international students in British higher education – this appears as Clause 90 in the latest version of the Bill.
  • Also carried on 6 March, by 200 votes to 189, was Baroness Royall’s amendment, which requires that all eligible students should have “the opportunity to opt in to be added to the electoral register”, which has now been incorporated into Clause 16.

 

A ‘cohesive and self-critical academic community': an ideal under threat

Opinion piece

G.R.Evans

An article in Times Higher Education on 2 March collected suggestions from a ‘panel of university administration experts’ on the best way to frame a job advertisement. Wondering where suitably qualified and ‘excellent’ teachers are going to be found to staff an ever-expanding English higher education system as the TEF evolves, I have been exploring such advertisements in recent months.

If institutional autonomy is to remain a fundamental of higher education provision, with new providers trusted to set and maintain standards, what does that imply for the recruitment of a provider’s academic staff? Among the Government’s background papers published in connection with the Higher Education and Research Bill is the Factsheet on Degree Awarding Powers and University Title (January 2017). It explains that the proposed:

reforms are aimed at removing unnecessary barriers that may currently stand in the way of providers that can demonstrate that they have:

  • the ability to design and deliver high quality HE degree courses,
  • the ability to set and maintain academic standards, and
  • their teaching is informed by scholarship and research.

So it is recognised that having academic staff of sufficient quality remains fundamental to justifying the institutional autonomy of new providers as much as to satisfactory ‘teaching performance’ in the TEF.

A teacher at higher education level has traditionally been expected to have academic qualifications over and above those needed to teach to A-Level GCE (Level 3 of the Qualifications and Credit Framework (QCF) ). These are defined by the Quality Assurance Agency (QAA) in line with Government guidance for the grant of degree-awarding powers and university title. The guiding principle is that lecturers in higher education should be qualified to teach at the level of the award offered, with research degrees therefore requiring research-active staff. The Government guidance was revised in September 2015 and further revision is expected to be needed under the provisions of the Higher Education and Research Bill s.40 and following.

The range of subjects in which it is possible to gain a degree has expanded hugely in recent years and so have the modes of study. ‘Workplace learning’, apprenticeship degrees, the need to meet the accreditation requirements of professional and regulatory bodies, vocational courses, frequently call for non-traditional teaching staff. ‘Vacancies’ advertisements for academic staff now often include terms such as ‘professional qualification’ or ‘experience’ as alternatives to the possession of even an undergraduate degree.

There is a further longstanding expectation, going beyond recruiting suitably-qualified academic staff piecemeal. This is that an institution’s academic staff will form a ‘self-critical academic community’ in setting and maintaining standards. In 1985 the Lindop Report (2.5) said that:

Traditionally the university degree was an acknowledgement, by the self-governing community of scholars which constituted the university in its wider sense, of academic attainment such as to warrant full membership of the community.

Sir Keith Joseph, then Secretary of State for Education and Science, added in a House of Commons Written Answer 17 March 1986:

The Government share the [Lindop] committee’s view that the most effective safeguard of an institution’s academic standards is the existence within it of a strong, cohesive and self-critical academic community. In future, the main purpose of external validating arrangements must be to foster the development of such communities.

Kenneth Clarke, as Secretary of State for Education and Science, maintained the principle in the House of Commons, 16 December 1991, speaking of

Criteria for Degree Awarding Powers

The principal criterion would be that any institution seeking degree awarding power for taught courses would need to be a self-critical, cohesive academic community with a proven commitment to quality assurance supported by effective assurance and enhancement systems.

Jo Johnson, in the House of Commons Higher Education and Research Bill Committee, 15 September 2016, reiterated the expectation that a university is: expected to be an institution that brings together a body of scholars to form a cohesive and self-critical academic community that provides excellent learning opportunities for people, the majority of whom are studying to degree level or above. We expect teaching at such an institution to be informed by a combination of research, scholarship and professional practice.

This expectation that providers with degree-awarding powers or university title will be entitled to autonomy because they employ a critical mass of appropriately qualified academic staff therefore remains fundamental.

The Factsheet gives three Case Studies, describing scenarios in which a new provider might apply successfully for degree awarding powers, all making that assumption. In all three the key element is the presence of a group of ‘experienced academics’. One is a company wishing to set up a new specialist provider. It ‘employs a group of experienced academics‘. Another is an existing ‘world-renowned’ US provider wanting to set up in England and offer English degrees. It sends across the Atlantic ‘an experienced team of academics’. A third is a ‘spin-off’ from a ‘top-ten University’ initiated by ‘a group of leading academics’.

But in the real world much has changed in the patterns of recruitment of academic staff. Advertisements mentioning research as an ‘essential’ requirement now cluster in the older universities, though a number of the newer providers say the possession of a doctorate or a willingness to seek one is ‘desirable’. There is a visible trend towards increasing the number of teaching-only contracts even in ‘research-intensive’ universities. Where the degree is in a vocational subject the job advertisement may be adjusted accordingly. Bought-in and externally assessed courses such as Pearson-Edexcel HNC and HND courses at levels 4 and 5 may plausibly be delivered by teachers without the traditional qualifications of a member of the academic staff envisaged by Lindop and the policy-makers who continue to quote him.

The range of types of employment contract on offer in higher education has been broadening in the direction of the ‘insecure’, though in what proportions it is hard to quantify exactly. Data provided by the Higher Education Statistics Agency includes information on fixed-term contract numbers, but does not make visible in detail the numbers or proportions of staff on hourly-paid or zero-hours contracts. (Some individuals may of course seek a zero-hour contract because of its flexibility, but the wording of job advertisements suggests it is rarely a matter of choice):

UCU has always been clear that those people returned by universities as atypical academics only represent a part of the real hourly-paid workforce. The rest are concealed within fixed-term contract data.

This pattern of recruitment undermines the expectation that providers will maintain an established ‘community’ of academic staff.

There seems to be a need for some joining-up of ideal and reality to ensure that before the OfS grants degree-awarding powers or university title, the new providers warrant the institutional autonomy they will gain if they succeed by showing that in their lecturers they have that ‘critical community’ of academic staff.

 

 

 

 

 

 

 

 

 

 

 

 

 

New DAPs anyone? Is the proposed multiplication of types of degree-awarding powers a good idea?

Preamble: The Higher Education and Research Bill will reach undergo line-by-line examination in the House of Lords on 6th March, when it reaches the Report Stage. This post is the third in a series of opinion pieces by G. R. Evans, reflecting on the Amendments tabled prior to this stage.

Opinion piece

by G. R. Evans

Concerned comment has focussed on the plan set out in the Higher Education and Research Bill to allow ‘high quality’ new providers to grant degrees on a probationary basis for three years rather than having to learn the ropes first under the supervision of an experienced ‘validating’ provider.

Even that existing common-sense provision has led to problems. The QAA’s Quality Code (B10) covers the whole field of collaborative and validating provision under the heading Managing higher education provision with others. This has been designed in the light of a history of disasters, from the ‘franchising scandals’ of the late 1990s to the collapse of the University of Wales when it had overreached itself in entering into validating arrangements. Years of active QAA consultation on all this can be found in a moment’s internet search.

To allow a brand new provider to plunge straight into the granting of degrees is clearly a high-risk strategy, as the Government seems to be realising. In the general discussion of the Amendments to the Bill published in late February is a reference to a paper published in January entitled Market Entry Reforms. That in turn refers the enquirer to another paper, where, it says, ‘we set out more detail on this and other changes to Degree Awarding Powers, as well as linked reforms to University Title’.

This further Factsheet paper explains that:

Our reforms are aimed at removing unnecessary barriers that may currently stand in the way of providers that can demonstrate that they have:

  • the ability to design and deliver high quality HE degree courses,
  • the ability to set and maintain academic standards, and
  • their teaching is informed by scholarship and research

Whereas the existing process for gaining degree-awarding powers requires a provider to demonstrate that it has actually been doing these things in a satisfactory manner for a period of time (thus satisfying Government Evidence Criterion B), the new proposals recognise that something different will be needed for a complete beginner. It will merely need to provide details of its plans:

Policies, procedures and guidance associated with programme development, approval and review; Policies, procedures and guidance associated with assessment of students and external examining; List of external examiners (if available); Terms of reference, constitution, reporting line and minutes for last two years of any advisory body; any reports from Professional Statutory and Regulatory Bodies

The PDAP (Probationary Degree Awarding Powers) candidate is envisaged as quite possibly offering only a single subject or seeking powers only to offer undergraduate (bachelors) degrees. These, let us call them SSDAPs and T(B-only)DAPS), would be something quite new in the list of DAPS.

It is easily forgotten that until 2004 – and still in Scotland and Northern Ireland – the degree-awarding powers held by a provider which wished to call itself a ‘university’ had to be teaching-and-research powers. Since 2004 in England and Wales a provider may acquire only taught degree-awarding powers (TDAP) and apply to become a university. However, the TDAP has included powers to grant taught postgraduate (usually Masters’) degrees. The T(B-only)DAPs will set a lower requirement.

A holder of SSDAPs at T(B-only) level, like any other new holder of degree-awarding powers, will in future have the powers on a time-limited basis. After three years they can expect to be granted unlimited DAPs.

A holder of indefinite DAPS may then apply for University Title. This will not include holders of only Foundation Degree-awarding powers (FDAPs) at Level 5. These were introduced in 2010, restricted to Further Education Colleges, though automatically included in the DAPS of any publicly-funded provider holding TDAPs. There has been a limited take-up of these so far and they seem to be rather a dead-end at present. HEFCE’s Register lists only half a dozen.

However it seems that the SSDAPs and T(B-only)DAPs holder will be free to apply to become universities.

Companies seeking to use the sensitive word ‘university’ in their titles may still do so:

This does not replace any requirements for consent under the Companies Act 2006, however we do expect all English higher education providers to obtain OfS consent. Where a provider requires a non-objection letter to register a new company or business name that includes the word ‘University’ with Companies House, the intention is that the OfS will issue such a non-objection letter at the same time as granting University Title.

The Factsheet containing these details is reassuring that:

we expect the OfS to set out the approach it will use to identify the ‘institution’ and legal entity that is seeking DAPs.

It is surely important for more of this to be clarified on the face of the Bill, for the multiplication of types of lower-level or restricted DAP constitutes a bigger change than the Factsheet seems to realise

Rival sanctions? The Office for Students and the Competition and Markets Authority

Preamble: The Higher Education and Research Bill will reach undergo line-by-line examination in the House of Lords on 6th March, when it reaches the Report Stage. This post is the second in a series of opinion pieces by G. R. Evans, reflecting on the Amendments tabled prior to this stage.

Opinion piece

by G. R. Evans

The Competition and Markets Authority (CMA) is a non-ministerial government department which works ‘to promote competition for the benefit of consumers, both within and outside the UK.’ Its stated aim is ‘to make markets work well for consumers, businesses and the economy.’

It has undertaken consultations on the fairness of student contracts with higher education providers. CMA advice for higher education providers on how to comply with consumer protection law  was published in March 2015, with guidance on consumer rights for students.

In a Press Release on 8 February, the Competition and Markets Authority drew attention to a compliance agreement reached with the University of East Anglia, stating:

‘Students will receive a fairer deal at the University of East Anglia (UEA) after the CMA secured changes to the university’s contract terms.’

The Press release explained that changes to modules at UEA had been found to have:

made significant changes to the content of a course – by introducing compulsory modules thus limiting students’ choice of optional modules – and not adequately informing prospective students who had received course offers about the changes.

The University has cooperated fully in amending its practice.

The CMA gives details on its case page of six agreements with universities reached since November. Not all were in England (one was with Glasgow).

There are, however, unanswered questions about how the CMA will work with the new Office for Students. Under the Higher Education and Research Bill the Office for Students will be able to suspend or de-register providers which are found to be in breach of their general or specific ongoing registration conditions, conditions which may be imposed by the OfS itself. It is far from clear whether these conditions will or may include compliance with CMA guidance. Perhaps it should be.

When he was questioned by the Education Select Committee at a Pre-Appointment Hearing on February 21 Sir Michael Barber was asked how he envisaged the OfS working with the CMA when he became the OfS Chair. He admitted he had not yet given the matter much thought.

Universities UK have begun some work on ‘how to ensure that fair terms and conditions are sufficiently flexible to allow courses to evolve’:

In order to understand this issue from the student perspective we are doing some work with the polling company ComRes.

This is not, however, the only area in which the CMA Guidance expresses concerns. Another became the subject of a finding and an adjustment in 2016. An agreement had been reached with UCL :

UCL will ensure that students are not prevented from graduating or re-enrolling because of non-academic debts, following action by the CMA.

A further area of concern to the CMA is the now very common claiming by higher education providers of the intellectual property of students, that is if the terms of the student contract (or the also common) inclusion in its statutes and regulations to which the contract binds the student) of a provision to ‘assign all intellectual property rights (IP) for any of [the student’s] work to the university, regardless of the circumstances’(3.6) .

Might or should the OfS impose sanctions on a provider the CMA found to be non-compliant? It may be wise to think this through before the Higher Education and Research Bill has reached the end of its journey towards royal assent.