TEF and the reputation of UK Higher Education

Posted on by
Reply

Opinion piece by Dorothy Bishop

The publication of the Teaching Excellence Framework (TEF) results this week was greeted with predictable glee by sections of the media. The Times was delighted to report that “The LSE, Southampton and Liverpool, all members of the elite Russell Group, were handed the lowest bronze award in the first Teaching Excellence Framework (TEF). They shared the ranking with the likes of Accrington and Rossendale College and Grimsby Institute of Further and Higher Education.” The scent of Schadenfreude was thick in the air: Oh, how are the mighty fallen: those snobby ivory towers mingling with the proletarian Northern oiks. And was it deserved? Of course it was: a Bronze award meant that they had been “short-changing students with poor lectures, aloof tutors or second-rate facilities.” Our poor students, who are now paying £9,000 per annum in fees, must be warned off these snobbish institutions, who ignore their needs while pursuing their dilettante interests in research.

This account of TEF results was, of course, rather at odds with the criteria for a Bronze award provided by HEFCE, which state it is: “for delivering teaching, learning and outcomes for its students that meet rigorous national quality requirements for UK higher education.” This is in contrast with a Silver award, for teaching that “consistently exceeds” those requirements, and Gold, which is “consistently outstanding“.

Much has been, and no doubt will be said about problems with the methods used by the TEF. Indeed, the chair of the TEF panel, Chris Husbands, admits it is not a measure of teaching excellence, but rather is “a measure based on some of the outcomes of teaching.” But the general response of those involved was that nothing’s perfect, the TEF was here to stay, and we’d better make the best of it.

This is strangely reminiscent of Brexit, which is widely seen to be a risky process likely to play havoc with the nation’s economic prosperity and general wellbeing, yet is regarded as inevitable as “the will of the people” and therefore cannot be questioned, but must be embraced and treated as an opportunity.

I beg to differ. I see it as the height of irresponsibility to go along with a process that exposes our higher education system to potential for harm without considering whether those risks outweigh the benefits. The potential for reputational damage is all too evident  in the reactions by the media. Whatever HEFCE or Chris Husbands may say, it is clear that an institution in receipt of a Bronze award will be regarded as third rate. Since both the reliability and the validity of the rankings are questionable, this means that, at a time when Brexit is already posing major challenges to the sector, we are throwing in spurious denigration of a subset of institutions for good measure. Of course, one can say, it’s the fault of the media. It is clear that the Times would not fare well if newspapers were evaluated on a Reporting Excellence Framework. But the reaction of the media was entirely predictable, and anyone who doubted that they would make a meal of this story is naïve.

In other sections of the media, and in government, those who raise objections to TEF are accused of underhand motives. We don’t value teaching, or we are arrogant, complacent, and unable to take criticism. That may be true for some, but the majority of academics worth their salt will reject TEF because it is everything good academic research should not be: simplistic, arbitrary and inadequately tested. As Helen Czerski noted on Twitter: “It is the tombstone of irony in higher education that ability of universities to teach nuance, subtle judgement and critical thinking is branded gold, silver, or bronze.” And Neuroneurotic wrote in a blogpost: “The one lesson I would take from this for UK Universities, is that we are clearly failing to educate politicians and policy makers to think carefully about evidence based policy.“

Another argument that keeps popping up is a version of put up or shut up: if academics can’t think of better metrics for TEF, then they can’t argue against it. Well, here’s a suggestion. We are told that we desperately need TEF because students want to have information that is reflected in the metrics. Well, why not provide the raw information? In fact, most of it, such as the National Student Survey results, is already publicly available – and indeed in a more relevant subject-specific form. It’s already established that higher education institutions should make available online information about details such as their course content, entry requirements, and drop-out rates. They could also be invited to include on their websites the kind of detailed narrative account of their teaching practices that was submitted to the TEF. All of this could be done without any need to convene a committee to sit down and ponder how to condense all this rich multifactorial information into three categories – applied not to the teaching of a specific subject, but to the entire institution.

I have written previously about the fiction that the TEF was developed in response to demand by students.  Unfortunately, the true reason for reducing teaching evaluation to this drastically clumsy and gross 3-item scale is to have a means of exerting control by using it to determine fee levels. We have to ask ourselves whether the vice-chancellors of our universities are guilty of neglect for taking that bait and going along with a scheme that poses such risks to the reputation of our higher education system.

 

 

 

A Motion of Regret: a constitutional avenue to ensure that fee rises are decoupled from TEF?

Posted on by
Reply

Opinion piece by G. R. Evans

On April 4 the Higher Education and Research Bill left the Lords for the Commons after its Third Reading and on the same day the Technical and Further Education Bill also went back to the Commons with some Lords’ amendments.  The following day the Lords debated a Motion of Regret.

This is a rarely used but powerful constitutional device. Because the Parliament Acts of 1911 and 1949 apply only to primary legislation, the House of Lords retains powers of veto (though not powers of amendment) over secondary legislation. It rarely uses those powers, not least because to do so might easily tempt any Government to amend the law to remove them.  But when used they may be ‘fatal’ not only to the specific piece of secondary legislation being vetoed, but with more far-reaching policy consequences.

The Higher Education and Research Bill now faces a period of ping-pong, for it seems unlikely that without some hard nudging, the House of Commons will simply approve the Bill in the radically amended form in which it will now reach the elected House. One of the most contentious of the Lords’ successful Amendments is like to be Lord Kerslake’s Amendment 19 to Clause 11 passed on Division on 6 March,  263 to 211, decoupling the right to charge higher inflation-linked tuition fees to TEF performance.

So Lord Stevenson’s Motion of Regret could prove important as a nudge. It runs as follows:

That this House regrets that the Higher Education (Basic Amount) (England) Regulations 2016 and the Higher Education (Higher Amount) (England) Regulations 2016 together with retrospectively changed loan conditions for existing students are further incremental burdens on students that risk worsening the opportunities for young people from low-income backgrounds, mature students and those undertaking part-time courses; and calls on Her Majesty’s Government to report annually to Parliament on the impact on the economy of the increasing quantum of graduate debt, estimates of payback rates, and the estimate of the annual cost to the Exchequer of the present system.

The Motion refers to the  21st Report from the Secondary Legislation Scrutiny Committee (January 2017), which sets out the various areas of concern in paras.7-14, to which Lord Stevenson spoke in the debate.  As he noted, the Regulations were ‘negative instruments’ – that is legislation which passed unless challenged – and ‘the time for praying against them has long passed’. However, the Secondary Legislation Scrutiny Committee had drawn these Instruments to the special attention of the House:

on the ground that they give rise to issues of public policy likely to be of interest to the House.

Lord Stevenson summarised his argument at the beginning of a lengthy speech in which he was able to rehearse, with close reference to the Committee’s detailed concerns, many of the themes which had occupied members of the House of Lords in the debates of the preceding four months while the Higher Education and Research Bill had been before them:

I am going to argue that the neoliberal marketisation of our higher education system is wrong in principle, because higher education is not a market; that it loads students with personal debt; that it will not improve opportunities to study for young people from disadvantaged and low-income backgrounds, mature students and those who wish to undertake part-time courses; and that linking fee rises, thereby increasing the personal debt of students, to only one of the attributes of a good university is a mistake. I will end by arguing that the cost of these polices to the public purse is now so complex and uncertain that it is virtually impossible to challenge what the Government are doing: we need more and more regular information and I call on the Government to provide it.

Baroness Garden spoke next, to endorse what he had said, emphasising ( in line with one of the Lords’ amendments to the Bill) that:

We on these Benches totally reject the idea of linking fees to teaching excellence framework gradings, as the noble Lord, Lord Stevenson, set out. They are an untried and untested form of assessment which should not be used to determine fees. There appears to be no correlation between increased fees and improved teaching quality.

Lord Bew added his support a little later.

Lord Willetts spoke at some length in defence of his own policy-changes and their consequences.  Lord Younger reminded the House that:

the parliamentary process is still ongoing, and I look forward to Peers’ further engagement on this matter. Our policy intention remains to link maximum fees to the quality of provision via the teaching excellence framework as part of our wider reform package, as we are doing through these regulations. It is counter to government policy to see fee caps rise under any other circumstances.

Lord Stevenson, wriggled uncomfortably under the flow of personal compliments he received because he was, as Baroness Garden noted,  with ‘much regret’, ‘stepping down from his Front-Bench role’. Bringing peers back to the matter in hand, he concluded by emphasising the significance of his choice of an unusual constitutional device:

I have now realised, after nearly seven years here, that the way to tackle these issues is by tabling this sort of Motion because in the normal cut and thrust of debate and in the discussion of legislation and questions, one can never get down to a serious debate about serious issues.

He gave notice that it might need to be used again on this topic when the missing supplementary information had been published:

I just want something simple. If we cannot interpret this system on the basis of the DfE’s published accounts, perhaps tabling another Motion at an appropriate time agreed with the Minister, because he is a friend as well, would be the way forward. However, in the interim, we should get things started by testing the opinion of the House on whether it would like to see more information on this interesting area.

The Motion of Regret was carried on Division by 174:162.

Government’s letter to the Lords re TEF: A reply

Posted on by
2

Opinion piece by Dorothy Bishop

The Higher Education and Research Bill (HERB) has had a rocky passage through parliament. As explained here, a Bill goes through several stages before it becomes law, with debate in both the House of Commons and the House of Lords, as well as Committee work that involves detailed line-by-line scrutiny of the text.  The Lords are primarily the revising chamber and they have passed a series of major amendments to HERB. The government has incorporated some of these amendments, but on its return to the Commons, the Bill may lose the Lords’ changes, and, as discussed in the Times Higher, there could be some to-and-fro before the Commons votes through a final version.

We have featured a series of posts on this blog discussing the amendments that have been proposed over the past two months:

One topic that particularly exercised the Lords was the new Teaching Excellence Framework (TEF). Introduction of the TEF did not require legislation: it was a manifesto commitment. Specifically, the Conservative manifesto stated: ” “We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality; encourage universities to offer more two-year courses; and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”. Accordingly, the TEF is already underway in a preliminary form. Nevertheless, numerous objections have been raised to it. Many of these were stated in the response to the initial Green Paper introducing the idea, including our own response. Yet, as documented in an earlier post, these objections were largely ignored. However, the Lords, who include many people with considerable experience in Higher Education, continued to have concerns.

At last, it seems, the Government are taking the concerns seriously. Jo Johnson, the Minister responsible for HERB, and Viscount Younger of Leckie, a Conservative Peer who is the Lords Spokesperson for Higher Education, have written a letter on 3rd March to offer clarification of questions that had been raised about the TEF in the House of Lords.

As someone who has voiced repeated concerns about the TEF, I offer here an analysis of the points made in that letter. For detailed sources please see these slides from a lecture on this topic.

Page 1. Para 2a states ‘The TEF is essential to driving up standards of teaching. All those who spoke during the Committee stages of the Bill agreed that teaching quality is of paramount importance.’

We can all agree that teaching quality is of paramount importance in our Universities – this is motherhood and apple pie. However, the statement that the TEF is essential to drive up teaching standards contains two shaky premises: that standards need ‘driving up’, and that the TEF would achieve this. We only need to introduce sector-wide measures to address teaching if there is evidence it is inadequate. I have argued elsewhere that this case has never been made. In a speech in 9th September, Johnson talked of ‘patchiness in the student experience’ but presented only anecdote to support his case. In the Green Paper, it was claimed that both students and employers were dissatisfied with teaching, yet the evidence was cherry-picked and mis-represented the sources from which it came.

The Government is fond of treating Higher Education as a market, yet their ideas for improving the market are ones that few businesses would adopt. Anyone operating a widget factory knows that to maintain quality control, you don’t give every widget a detailed inspection: you adopt a process of sampling a small proportion, so you can be ready to check the production process if you find an unacceptable level of problems. Even Jo Johnson accepts that our Universities are world-leading, yet he wants to impose a further burden of evaluation to complement the Research Excellence Framework (REF). This will take money and resources from research and teaching and so be counterproductive. I would suggest that instead of a TEF we need:

  • A system of quality control that will scrutinise any institutions that show signs of failing (e.g. because of high level of complaints). We already have this with the Quality Assurance Agency (QAA).
  • Incorporation of some information about teaching quality in the REF Impact section – see this proposal by Roger Brown.
  • Disincentives for Universities to use zero-hours contract staff for teaching; this could include a section on staff well-being and moral in the Environment section of the REF.
  • Incentives for sharing good and innovative teaching practices

Page 1 para 1b. ‘The TEF will provide the financial and reputational incentives for providers to prioritise teaching excellence and student outcomes. Genuine and clear differentiation between providers and a link with fees are essential to achieving these outcomes.‘

The downside of yet another ranking system are simply ignored here. What is needed is a diverse higher education system, with different institutions developing their own specialised approaches to suit the very varied needs of our population. A crude ranking, further subdivided into Gold, Silver and Bronze, treats higher education as some kind of horse race, where quality can be measured on one dimension.

Page 1 para 2-3 notes that ‘It is important that we treat fairly those who have already decided to take part….The assessment process is already well underway and to change the “rules of the game”… would be extremely unfair on those who have invested their time and effort to participate this year’

I’m tempted to respond, ‘Well, whose fault is that, Minister’. It is Johnson who has chosen to implement an ill-thought-through evaluation system, with threats to participating organisations that they will lose their ability to raise fees unless they take part. And he also promised that TEF would not be onerous. But now we are told that those institutions who have taken part would have invested time and effort and so it would be unfair to stop. One is reminded of someone who orders a meal that makes them sick but persists in eating it, because the money has been spent.

What is really unfair is to adopt a system of evaluation that is based on unsuitable metrics, is incapable of making meaningful distinctions between institutions, yet which has the potential to damage the standing of a University.

Page 2, para 2. ‘We recognise the genuine and considered concerns raised by noble Lords about the TEF, in particular around the speed at which it is being implemented and on the use of metrics and ratings….a genuine lessons-learned exercise will take place after this trial year.’

Well, at last it seems there is recognition that the TEF in its current form is unworkable. Criticism has come from individuals who might be expected to support it; see e.g. this account of an interview with Chris Husbands, Chair of the TEF Panel on Jan 25th in the Times Higher:

‘“I do not think student satisfaction is an accurate proxy for teaching quality,” said Professor Husbands, a former director of the UCL Institute of Education who was named chair of the TEF panel by the Department for Business, Energy and Industrial Strategy in August 2016.

Professor Husbands drew attention to US studies that show that student satisfaction scores are heavily influenced by non-academic factors, including the sex or ethnicity of a lecturer.

“Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” he said of the US research findings.’

And Stuart Croft, VC at Warwick on Jan 31st wrote in the Times Higher ’…the metrics are flawed. This is not renegade opinion, but rather the overwhelming view of those actually involved in higher education….Yet the government has us over a barrel. It has linked the TEF to tuition fees and, potentially, our ability to recruit international students.’

Page 3. Para 1 states that ‘…the TEF is about much more than metrics. Providers submit additional evidence alongside their metrics and this evidence will be given significant weight by the panel.’

But what is the reliability and validity of the ‘additional evidence’? This seems like encouragement to Universities to tell plausible stories about their teaching and try to anticipate which buttons they need to hit to convince a panel that they are Gold rather than Silver and Bronze. There is huge scope here for subjectivity at best and corruption at worst.

Page 3. Para 2 state ‘All of the metrics used in the TEF are trusted, widely used and well established in the sector….. The NSS is just one of three principal sources of metrics data being used, and we have explicitly said that the NSS metrics are the least important. The TEF does not use in any way the overall ‘satisfaction’ rating, about which the House has rightly expressed concern, but instead uses specific questions from the NSS, related to teaching and learning.’

It’s hard to know where to start with this. There are numerous issues with using the NSS for assessing teaching quality, and changing the items that are used does not solve them. There are problems with validity – does the measure assess what it is supposed to measure, bias – are ratings affected by the teacher’s gender, race or disability, and sensitivity – is the spread of scores adequate for differences between institutions to be meaningful? Statistical criticisms have been made by both the Royal Statistical Society and the Office of National Statistics. We are told in para 2 that ‘the Government has already taken a number of steps to ensure that the statistics being used are robust, including commissioning a report by the Office of National Statistics.’ They seem to think that the ONS will somehow be able to magically transform an unreliable, insensitive scale into a meaningful indicator of teaching quality. Statisticians are good, but not that good. Furthermore, all the statistical problems that afflict the TEF are magnified with small samples, and small samples are going to be inevitable if we move, as we have been told we must, to subject-specific assessments.

It is depressing, though not surprising, to find that the letter ends by telling us the university sector must be ‘shaken out of its complacency’. It’s an easy but lazy point to say that opposition to the TEF is just the response of a complacent group of academics who are set in their ways, because it means you don’t actually have to consider the arguments. I suspect that Johnson would be surprised to find just how passionately many of us care about our Higher Education system. If he could grasp that point, he might start to take note of us when we tell him his superficial and rushed approach to evaluating teaching is resisted because of the considerable potential it has for harm.

 

 

 

 

 

 

 

Update on Amendments to the Higher Education and Research Bill

Posted on by
Reply

David Midgley has kindly provided this useful summary of recent developments:

The text of the Bill as amended by the Lords in report stage is now available at https://www.publications.parliament.uk/pa/bills/lbill/2016-2017/0112/17112.pdf.

The most significant changes with regard to the regime proposed for higher education are these:

  • Lord Kerslake’s amendment of 6 March to the effect that the TEF “must not be used to rank English higher education providers as to the regulated course fees they charge”, carried by 263 votes to 211, which is now Clause 12;
  • Lord Blunkett’s amendment of 8 March, carried by 280 votes to 186, which requires any “scheme to assess and provide consistent and reliable information about the quality of education and teaching” to be subject to rigorous independent scrutiny, and precludes it from being “used to create a single composite ranking of English higher education providers” – this has become Clause 27;
  • Baroness Wolf’s amendment of 8 March, carried by 201 votes to 186, which makes explicit the role of the Quality Assessment Committee in relation to the authorisation of any new provider and requires such a provider to have been “established for a minimum of four years with satisfactory validation arrangements in place” before authorisation may be issued – this has been incorporated into Clause 47 (as sub-sections 12 and 13); and
  • Lord Judge’s amendment of 8 March, carried by 185 votes to 151, which has the effect of strengthening the power of appeal against a revocation of degree-awarding powers and/or university title – the relevant clause is now Clause 49.
  • Also of considerable importance for the future of universities is Lord Hannay’s amendment of 13 March, carried by a noteworthy 313 votes to 219, which seeks to protect the position of international students in British higher education – this appears as Clause 90 in the latest version of the Bill.
  • Also carried on 6 March, by 200 votes to 189, was Baroness Royall’s amendment, which requires that all eligible students should have “the opportunity to opt in to be added to the electoral register”, which has now been incorporated into Clause 16.

 

A ‘cohesive and self-critical academic community': an ideal under threat

Posted on by
Reply

Opinion piece

G.R.Evans

An article in Times Higher Education on 2 March collected suggestions from a ‘panel of university administration experts’ on the best way to frame a job advertisement. Wondering where suitably qualified and ‘excellent’ teachers are going to be found to staff an ever-expanding English higher education system as the TEF evolves, I have been exploring such advertisements in recent months.

If institutional autonomy is to remain a fundamental of higher education provision, with new providers trusted to set and maintain standards, what does that imply for the recruitment of a provider’s academic staff? Among the Government’s background papers published in connection with the Higher Education and Research Bill is the Factsheet on Degree Awarding Powers and University Title (January 2017). It explains that the proposed:

reforms are aimed at removing unnecessary barriers that may currently stand in the way of providers that can demonstrate that they have:

  • the ability to design and deliver high quality HE degree courses,
  • the ability to set and maintain academic standards, and
  • their teaching is informed by scholarship and research.

So it is recognised that having academic staff of sufficient quality remains fundamental to justifying the institutional autonomy of new providers as much as to satisfactory ‘teaching performance’ in the TEF.

A teacher at higher education level has traditionally been expected to have academic qualifications over and above those needed to teach to A-Level GCE (Level 3 of the Qualifications and Credit Framework (QCF) ). These are defined by the Quality Assurance Agency (QAA) in line with Government guidance for the grant of degree-awarding powers and university title. The guiding principle is that lecturers in higher education should be qualified to teach at the level of the award offered, with research degrees therefore requiring research-active staff. The Government guidance was revised in September 2015 and further revision is expected to be needed under the provisions of the Higher Education and Research Bill s.40 and following.

The range of subjects in which it is possible to gain a degree has expanded hugely in recent years and so have the modes of study. ‘Workplace learning’, apprenticeship degrees, the need to meet the accreditation requirements of professional and regulatory bodies, vocational courses, frequently call for non-traditional teaching staff. ‘Vacancies’ advertisements for academic staff now often include terms such as ‘professional qualification’ or ‘experience’ as alternatives to the possession of even an undergraduate degree.

There is a further longstanding expectation, going beyond recruiting suitably-qualified academic staff piecemeal. This is that an institution’s academic staff will form a ‘self-critical academic community’ in setting and maintaining standards. In 1985 the Lindop Report (2.5) said that:

Traditionally the university degree was an acknowledgement, by the self-governing community of scholars which constituted the university in its wider sense, of academic attainment such as to warrant full membership of the community.

Sir Keith Joseph, then Secretary of State for Education and Science, added in a House of Commons Written Answer 17 March 1986:

The Government share the [Lindop] committee’s view that the most effective safeguard of an institution’s academic standards is the existence within it of a strong, cohesive and self-critical academic community. In future, the main purpose of external validating arrangements must be to foster the development of such communities.

Kenneth Clarke, as Secretary of State for Education and Science, maintained the principle in the House of Commons, 16 December 1991, speaking of

Criteria for Degree Awarding Powers

The principal criterion would be that any institution seeking degree awarding power for taught courses would need to be a self-critical, cohesive academic community with a proven commitment to quality assurance supported by effective assurance and enhancement systems.

Jo Johnson, in the House of Commons Higher Education and Research Bill Committee, 15 September 2016, reiterated the expectation that a university is: expected to be an institution that brings together a body of scholars to form a cohesive and self-critical academic community that provides excellent learning opportunities for people, the majority of whom are studying to degree level or above. We expect teaching at such an institution to be informed by a combination of research, scholarship and professional practice.

This expectation that providers with degree-awarding powers or university title will be entitled to autonomy because they employ a critical mass of appropriately qualified academic staff therefore remains fundamental.

The Factsheet gives three Case Studies, describing scenarios in which a new provider might apply successfully for degree awarding powers, all making that assumption. In all three the key element is the presence of a group of ‘experienced academics’. One is a company wishing to set up a new specialist provider. It ‘employs a group of experienced academics‘. Another is an existing ‘world-renowned’ US provider wanting to set up in England and offer English degrees. It sends across the Atlantic ‘an experienced team of academics’. A third is a ‘spin-off’ from a ‘top-ten University’ initiated by ‘a group of leading academics’.

But in the real world much has changed in the patterns of recruitment of academic staff. Advertisements mentioning research as an ‘essential’ requirement now cluster in the older universities, though a number of the newer providers say the possession of a doctorate or a willingness to seek one is ‘desirable’. There is a visible trend towards increasing the number of teaching-only contracts even in ‘research-intensive’ universities. Where the degree is in a vocational subject the job advertisement may be adjusted accordingly. Bought-in and externally assessed courses such as Pearson-Edexcel HNC and HND courses at levels 4 and 5 may plausibly be delivered by teachers without the traditional qualifications of a member of the academic staff envisaged by Lindop and the policy-makers who continue to quote him.

The range of types of employment contract on offer in higher education has been broadening in the direction of the ‘insecure’, though in what proportions it is hard to quantify exactly. Data provided by the Higher Education Statistics Agency includes information on fixed-term contract numbers, but does not make visible in detail the numbers or proportions of staff on hourly-paid or zero-hours contracts. (Some individuals may of course seek a zero-hour contract because of its flexibility, but the wording of job advertisements suggests it is rarely a matter of choice):

UCU has always been clear that those people returned by universities as atypical academics only represent a part of the real hourly-paid workforce. The rest are concealed within fixed-term contract data.

This pattern of recruitment undermines the expectation that providers will maintain an established ‘community’ of academic staff.

There seems to be a need for some joining-up of ideal and reality to ensure that before the OfS grants degree-awarding powers or university title, the new providers warrant the institutional autonomy they will gain if they succeed by showing that in their lecturers they have that ‘critical community’ of academic staff.

 

 

 

 

 

 

 

 

 

 

 

 

 

New DAPs anyone? Is the proposed multiplication of types of degree-awarding powers a good idea?

Posted on by
Reply

Preamble: The Higher Education and Research Bill will reach undergo line-by-line examination in the House of Lords on 6th March, when it reaches the Report Stage. This post is the third in a series of opinion pieces by G. R. Evans, reflecting on the Amendments tabled prior to this stage.

Opinion piece

by G. R. Evans

Concerned comment has focussed on the plan set out in the Higher Education and Research Bill to allow ‘high quality’ new providers to grant degrees on a probationary basis for three years rather than having to learn the ropes first under the supervision of an experienced ‘validating’ provider.

Even that existing common-sense provision has led to problems. The QAA’s Quality Code (B10) covers the whole field of collaborative and validating provision under the heading Managing higher education provision with others. This has been designed in the light of a history of disasters, from the ‘franchising scandals’ of the late 1990s to the collapse of the University of Wales when it had overreached itself in entering into validating arrangements. Years of active QAA consultation on all this can be found in a moment’s internet search.

To allow a brand new provider to plunge straight into the granting of degrees is clearly a high-risk strategy, as the Government seems to be realising. In the general discussion of the Amendments to the Bill published in late February is a reference to a paper published in January entitled Market Entry Reforms. That in turn refers the enquirer to another paper, where, it says, ‘we set out more detail on this and other changes to Degree Awarding Powers, as well as linked reforms to University Title’.

This further Factsheet paper explains that:

Our reforms are aimed at removing unnecessary barriers that may currently stand in the way of providers that can demonstrate that they have:

  • the ability to design and deliver high quality HE degree courses,
  • the ability to set and maintain academic standards, and
  • their teaching is informed by scholarship and research

Whereas the existing process for gaining degree-awarding powers requires a provider to demonstrate that it has actually been doing these things in a satisfactory manner for a period of time (thus satisfying Government Evidence Criterion B), the new proposals recognise that something different will be needed for a complete beginner. It will merely need to provide details of its plans:

Policies, procedures and guidance associated with programme development, approval and review; Policies, procedures and guidance associated with assessment of students and external examining; List of external examiners (if available); Terms of reference, constitution, reporting line and minutes for last two years of any advisory body; any reports from Professional Statutory and Regulatory Bodies

The PDAP (Probationary Degree Awarding Powers) candidate is envisaged as quite possibly offering only a single subject or seeking powers only to offer undergraduate (bachelors) degrees. These, let us call them SSDAPs and T(B-only)DAPS), would be something quite new in the list of DAPS.

It is easily forgotten that until 2004 – and still in Scotland and Northern Ireland – the degree-awarding powers held by a provider which wished to call itself a ‘university’ had to be teaching-and-research powers. Since 2004 in England and Wales a provider may acquire only taught degree-awarding powers (TDAP) and apply to become a university. However, the TDAP has included powers to grant taught postgraduate (usually Masters’) degrees. The T(B-only)DAPs will set a lower requirement.

A holder of SSDAPs at T(B-only) level, like any other new holder of degree-awarding powers, will in future have the powers on a time-limited basis. After three years they can expect to be granted unlimited DAPs.

A holder of indefinite DAPS may then apply for University Title. This will not include holders of only Foundation Degree-awarding powers (FDAPs) at Level 5. These were introduced in 2010, restricted to Further Education Colleges, though automatically included in the DAPS of any publicly-funded provider holding TDAPs. There has been a limited take-up of these so far and they seem to be rather a dead-end at present. HEFCE’s Register lists only half a dozen.

However it seems that the SSDAPs and T(B-only)DAPs holder will be free to apply to become universities.

Companies seeking to use the sensitive word ‘university’ in their titles may still do so:

This does not replace any requirements for consent under the Companies Act 2006, however we do expect all English higher education providers to obtain OfS consent. Where a provider requires a non-objection letter to register a new company or business name that includes the word ‘University’ with Companies House, the intention is that the OfS will issue such a non-objection letter at the same time as granting University Title.

The Factsheet containing these details is reassuring that:

we expect the OfS to set out the approach it will use to identify the ‘institution’ and legal entity that is seeking DAPs.

It is surely important for more of this to be clarified on the face of the Bill, for the multiplication of types of lower-level or restricted DAP constitutes a bigger change than the Factsheet seems to realise

Rival sanctions? The Office for Students and the Competition and Markets Authority

Posted on by
Reply

Preamble: The Higher Education and Research Bill will reach undergo line-by-line examination in the House of Lords on 6th March, when it reaches the Report Stage. This post is the second in a series of opinion pieces by G. R. Evans, reflecting on the Amendments tabled prior to this stage.

Opinion piece

by G. R. Evans

The Competition and Markets Authority (CMA) is a non-ministerial government department which works ‘to promote competition for the benefit of consumers, both within and outside the UK.’ Its stated aim is ‘to make markets work well for consumers, businesses and the economy.’

It has undertaken consultations on the fairness of student contracts with higher education providers. CMA advice for higher education providers on how to comply with consumer protection law  was published in March 2015, with guidance on consumer rights for students.

In a Press Release on 8 February, the Competition and Markets Authority drew attention to a compliance agreement reached with the University of East Anglia, stating:

‘Students will receive a fairer deal at the University of East Anglia (UEA) after the CMA secured changes to the university’s contract terms.’

The Press release explained that changes to modules at UEA had been found to have:

made significant changes to the content of a course – by introducing compulsory modules thus limiting students’ choice of optional modules – and not adequately informing prospective students who had received course offers about the changes.

The University has cooperated fully in amending its practice.

The CMA gives details on its case page of six agreements with universities reached since November. Not all were in England (one was with Glasgow).

There are, however, unanswered questions about how the CMA will work with the new Office for Students. Under the Higher Education and Research Bill the Office for Students will be able to suspend or de-register providers which are found to be in breach of their general or specific ongoing registration conditions, conditions which may be imposed by the OfS itself. It is far from clear whether these conditions will or may include compliance with CMA guidance. Perhaps it should be.

When he was questioned by the Education Select Committee at a Pre-Appointment Hearing on February 21 Sir Michael Barber was asked how he envisaged the OfS working with the CMA when he became the OfS Chair. He admitted he had not yet given the matter much thought.

Universities UK have begun some work on ‘how to ensure that fair terms and conditions are sufficiently flexible to allow courses to evolve’:

In order to understand this issue from the student perspective we are doing some work with the polling company ComRes.

This is not, however, the only area in which the CMA Guidance expresses concerns. Another became the subject of a finding and an adjustment in 2016. An agreement had been reached with UCL :

UCL will ensure that students are not prevented from graduating or re-enrolling because of non-academic debts, following action by the CMA.

A further area of concern to the CMA is the now very common claiming by higher education providers of the intellectual property of students, that is if the terms of the student contract (or the also common) inclusion in its statutes and regulations to which the contract binds the student) of a provision to ‘assign all intellectual property rights (IP) for any of [the student’s] work to the university, regardless of the circumstances’(3.6) .

Might or should the OfS impose sanctions on a provider the CMA found to be non-compliant? It may be wise to think this through before the Higher Education and Research Bill has reached the end of its journey towards royal assent.

The Haldane principle: remembering Fisher and getting that definition right

Posted on by
Reply

Opinion piece by G. R. Evans

It is very welcome news that the Government has decided to include a definition of the Haldane Principle on the face of the Bill. Jo Johnson made a special point of this in his speech to Universities UK on 24 February.  An accompanying document was published jointly by both the Departments of State that will in future be responsible for higher education. It proudly states that:

the amendment that we have tabled will, for the first time in history, enshrine the Haldane Principle in law.

This document did not, however, give more details. The actual Amendment of Clause 99 proposing and containing the definition is to be found in yet another document:

Page 64, line 10, at end insert -

“The ‘Haldane principle’ is the principle that decisions on individual research proposals are best taken following an evaluation of the quality and likely impact of the proposals (such as a peer review process).”

Note that this definition does not stipulate an exercise of academic judgement, merely an ‘evaluation’ including ‘likely impact’ of research to be funded. Furthermore, the definition does not mention that infrastucture funding will come from Research England. Rather, an earlier statement merely stipulates the Councils (a sub-set of UKRI) will be responsible for the disbursement of project funding:

Page 64, line 7, at end insert -

“the Haldane principle, where the grant or direction mentioned in subsection (1) is in respect of functions exercisable by one or more of the Councils mentioned in section 91(1) pursuant to arrangements under that section,”

This took me back to the question what Haldane actually called for and the context in which he did so. His thoughts on higher education matters are chiefly to be found in some collected writings put together in a period when he was actively involved in fostering the development of the new ‘redbrick’ universities. He developed a special enthusiasm for technical education but essentially he was interested in the work of a university as a whole, not merely its research.

He recognised that if higher education was going to expand successfully something would have to be done about the funding that would be needed:

‘the truth is that work of this kind must be more largely assisted and fostered by the State than is the tradition of today if it is to succeed’

(Education and Empire: Addresses on certain topics of the day (London, 1902), p.38).

The new universities began to accept state funding but it was not at first expected that Oxford or Cambridge would need to apply. The First World War upset many expectations.

A decisive correspondence followed between November 1918 and May 1919, between the then President of the Board of Education, H. A. L. Fisher, and the Vice-Chancellor of the University of Oxford. This was published in full in May in the Oxford University Gazette, under the heading Applications for Government Grants (Oxford University Gazette, 49 (1918-9), p.471-8).

A deputation from the universities ‘asking for larger subsidies from the State’ met Fisher on 23 November. Oxford and Cambridge consulted one another and agreed that it would be wise to join in, but without committing themselves. Oxford was understandably nervous about accepting state funding because of the likelihood that it would bring State control.

But the Oxford scientists, scenting money, put in their own bids for specific sums for particular purposes. The heads of departments of the University Museum wrote on 3 March, 1919 with a list of such ‘needs’, identifying sums for capital outlay and salaries and pensions for Heads of Department and scholarships for what would now be called STEM subjects.

It was in this context that Fisher seems to have made his far-reaching policy decision and stated the ‘Fisher Principle’, that the state would not interfere in the allocation of funds within universities. It would not decide directly whether to fund, say, science at Oxford, or History at Manchester. It would give funding in the form of ‘Block Grants’ and allow the universities themselves to decide how to use the money.

He wrote to the Oxford Vice-Chancellor on 16 April:

‘Henceforth…each University which receives aid from the State will receive it in the form of a single inclusive grant, for the expenditure of which the University, as distinguished from any particular Department, will be responsible. Both the Government and, I think, the great majority of the Universities are convinced that such an arrangement is favourable not only to the preservation of University autonomy but also to the efficient administration of the University funds.’

The University’s Council (then the Hebdomadal Council, meeting weekly in term-time) requested an interview with Fisher and on May 15 a deputation of five, led by the Vice-Chancellor, had a meeting with him. The Memorandum of the Interview ‘kindly furnished by Mr. Fisher’s Secretary’ is also published in the Gazette. It repeated the policy principle arrived at in November, that ‘the English Universities in receipt of State-aid favoured …a general Block Grant’. It was explained that a Standing Committee was in process of ‘formation’ and that ‘henceforward, practically all the money for University Education would be borne on the Treasury Vote and would be allocated in annual Block Grants’ as the Standing Committee recommended.

This Standing Committee developed into the University Grants Committee, which was replaced a quarter of a century ago by first one then four Funding Councils. One of those, HEFCE, is now to be replaced as distributor of the remnant of that Block Grant mainly by Research England within UKRI, with only a vestige of the element previously used to fund teaching still remaining.

So there seem to be features of the Government Amendment to Clause 99 which would bear further thought if a definition of the ‘Haldane Principle’ is to enter statute.

The Haldane Principle arguably needs to be understood as it was developed in the ‘Fisher Principle’ and has been maintained for a century since. That placed a ‘buffer’ body between State and university and protected the freedom of the university to choose how to use its block grant on academic not political principles. That is not quite the thrust of the definition as it stands at present.

Nor did the ‘Fisher-Haldane Principle’ apply to the buffer body itself. The buffer stood between academic freedom and state control. It was not itself subject to that principle. It merely ensured that it was respected.

It is to be hoped that the legal draftsmen working on the Bill will try again. The version in the current Amendment, if it passes into law, will fail to protect the autonomy of the providers receiving funding from UKRI. Nor will it require funding decisions to be taken by academics or by autonomous institutions. The ‘peer review process’ is given as a mere example. There seems nothing to prevent a Minister or Secretary of State conducting ‘an evaluation of the quality and likely impact of the proposals’. Haldane and Fisher could both be turning in their graves.

 

 

 

Why the NSS is garbage

Posted on by
Reply

A note by Lord Lipsey    

The National Students Survey results matter. First, they are used by students to evaluate institutions by comparison with rival institutions. Secondly, they are one of the metrics to be used in the TEF, in awarding gold, silver or bronze markings to institutions which apply to take part. These ratings will decide if an institution can or cannot raise its fees beyond £9K.

The idea that student satisfaction should play a major role in the rating of universities is controversial. Research shows that there is no correlation between student satisfaction and student results in terms of degree grade . However the government has opted to increase the importance of student choice, competition and satisfaction in the higher education landscape; and this short note does not seek to address the rights and wrongs of that.

Rather it focusses on a narrow point: whether the National Student Survey (NSS), the chosen instrument to measure student satisfaction, is fit for purpose or not.  Here the evidence is unequivocal, The NSS is statistical garbage, The reasons are widely understood by the statistical community and were set out inter alia by the Royal Statistical Society in its response to the government’s technical consultation about the TEF http://www.rss.org.uk/Images/PDF/influencing-change/2016/RSS-response-to-BIS-Technical Consultation-on-Teaching-Excellence-Framework-year-2.pdf.

  1. The NSS is not based on a random or representative sample of students. It is more akin to a census, which includes everyone who chooses to complete the form but not those who don’t. Some groups eg ethnic minorities are seriously underrepresented. As the final report of the ONS Review of Data Sources for the TEF 2016 said: “under-reporting of certain groups and over-coverage of others …could lead to bias in use of the data.” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/556352/Teaching-Excellence-Framework-review-of-data-sources.pdf

I chair Trinity Laban Conservatoire of Music and Dance, a leading London Conservatoire. This year Trinity Laban’s response rate to the NSS rose from about 60% to about 80%. This makes any comparison between last year’s results and this year’s results invalid. We know nothing about the 20%; so a 50% satisfaction rate amongst respondents could perfectly well be a 70% satisfaction rate amongst students as whole.

  1. Even if the responses are treated as a random or representative sample, making calculations of statistical significance possible, the margins of error in the NSS figures are large. The ONS concluded that “differences between institutions at the overall level are small and are not significant” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/556352/Teaching-Excellence-Framework-review-of-data-sources.pdf Yet on that are being based student decisions on where to go, and the level of fees institutions can charge.

So for example Trinity Laban’s  music results – and music is our biggest group of students -are based on 112 students. To take an example only 49% of these students say that marking on their course has been fair. Statistically however and treating the returns as if they were a sample, this means that there is a 95% chance that between 34% and 64% of students think it is fair. This perhaps exaggerates the unreliability of the NSS insofar as. if response rates are high, then the results may be less unreliable that they would be for a sample. But the point still applies. And.in cases of smaller sub-samples the low response renders the results vacuous. For example for Trinity Laban Musical Theatre the poor results were based on response from just 23 students.

The small sample size is a particular problem for small institutions. It is also a grave problem in that the most valid comparisons are not between institutions, but between particular departments in institutions teaching the same or similar courses. For individual courses, there will tend to be only a fraction of the responses obtained for institutions as a whole.

  1. The results can be greatly affected by happenstance. Results in the TL musical theatre department in the past have been good which suggests that there was some peculiar chemistry about this year (and also there is some suggestion that students coordinated their responses to make a point). A survey of dance satisfaction for Trinity Laban was completed the day after a one-off cock-up about room bookings; had it been done two days earlier the results might have been much better.
  1. There is scope for “gaming” the results and encouraging students to give positive results,. Trinity Laban does not do this. Anecdotal evidence, as cited by the RSS paper above suggests that less scrupulous institutions do, and the incentive to do so will be greatly increased now that the NSS has new significance as a metric for the TEF.
  1. Less concrete but not less important, an emphasis on the student experience may lead to undesirable effects on what they are taught. Institutions focussed on a high NSS score will tend to dumb down degrees or go for safe options on content. ]

Here is an example from Trinity Laban. TL exists due to an amalgamation of Trinity College of Music and Laban dance. A few years ago, the Principal introduced a compulsory two-week course called CoLab which was a creative programme involving dance and music working together. At first students were furious at being deprived of these two weeks off from the focus of their studies – and no doubt this would have been reflected in their satisfaction ratings. However time has passed; familiarity has grown; and CoLab is now one of the most popular things we offer with our students.

In addition, during the ONS review of the NSS,  respondents expressed reservations about wider issues related to the use of information from the NSS and the DLHE. Concerns included:

  • limited variation between institutions of the raw scores from the student responses
  • difficulty in trying to compare widely differing institutions
  • difficulty in capturing the wider benefits beyond academic results of attending a higher education institution

The government has already downgraded the importance of the NSS in TEF – the so-called LSE amendment made when it was pointed out that NSS suggested that LSE and a number of other prestigious institutions were rated low by students. The metric should be subject to intense scrutiny when the House of Lords debates the Higher Education and Research Bill in committee; and I am tabling amendments to make sure it is.

The spurious precision of the NSS has the capacity to damage staff morale, to put students off certain institutions and affect the validity of the TEF. I know of at least once case where a head of department at a major institution came close to resigning because of a disappointing NSS result. I know of another where a Principal was dismissed inter alia because of the failure of that institution’s NSS results to improve. This is terrifying.

Lord Lipsey is joint Chair of the All Party Statistics Group, a Fellow of the Academy of Social Science, a Board member of Full Fact the fact checking charity, a former member of the advisory council of the National Centre for Social Research,  founder of Straight Statistics and a former adviser on opinion polling to James Callaghan as prime minister. In other words, though he may talk nonsense on many subjects, this is not one of them!