Government’s letter to the Lords re TEF: A reply

Opinion piece by Dorothy Bishop

The Higher Education and Research Bill (HERB) has had a rocky passage through parliament. As explained here, a Bill goes through several stages before it becomes law, with debate in both the House of Commons and the House of Lords, as well as Committee work that involves detailed line-by-line scrutiny of the text.  The Lords are primarily the revising chamber and they have passed a series of major amendments to HERB. The government has incorporated some of these amendments, but on its return to the Commons, the Bill may lose the Lords’ changes, and, as discussed in the Times Higher, there could be some to-and-fro before the Commons votes through a final version.

We have featured a series of posts on this blog discussing the amendments that have been proposed over the past two months:

One topic that particularly exercised the Lords was the new Teaching Excellence Framework (TEF). Introduction of the TEF did not require legislation: it was a manifesto commitment. Specifically, the Conservative manifesto stated: ” “We will ensure that universities deliver the best possible value for money to students: we will introduce a framework to recognise universities offering the highest teaching quality; encourage universities to offer more two-year courses; and require more data to be openly available to potential students so that they can make decisions informed by the career paths of past graduates”. Accordingly, the TEF is already underway in a preliminary form. Nevertheless, numerous objections have been raised to it. Many of these were stated in the response to the initial Green Paper introducing the idea, including our own response. Yet, as documented in an earlier post, these objections were largely ignored. However, the Lords, who include many people with considerable experience in Higher Education, continued to have concerns.

At last, it seems, the Government are taking the concerns seriously. Jo Johnson, the Minister responsible for HERB, and Viscount Younger of Leckie, a Conservative Peer who is the Lords Spokesperson for Higher Education, have written a letter on 3rd March to offer clarification of questions that had been raised about the TEF in the House of Lords.

As someone who has voiced repeated concerns about the TEF, I offer here an analysis of the points made in that letter. For detailed sources please see these slides from a lecture on this topic.

Page 1. Para 2a states ‘The TEF is essential to driving up standards of teaching. All those who spoke during the Committee stages of the Bill agreed that teaching quality is of paramount importance.’

We can all agree that teaching quality is of paramount importance in our Universities – this is motherhood and apple pie. However, the statement that the TEF is essential to drive up teaching standards contains two shaky premises: that standards need ‘driving up’, and that the TEF would achieve this. We only need to introduce sector-wide measures to address teaching if there is evidence it is inadequate. I have argued elsewhere that this case has never been made. In a speech in 9th September, Johnson talked of ‘patchiness in the student experience’ but presented only anecdote to support his case. In the Green Paper, it was claimed that both students and employers were dissatisfied with teaching, yet the evidence was cherry-picked and mis-represented the sources from which it came.

The Government is fond of treating Higher Education as a market, yet their ideas for improving the market are ones that few businesses would adopt. Anyone operating a widget factory knows that to maintain quality control, you don’t give every widget a detailed inspection: you adopt a process of sampling a small proportion, so you can be ready to check the production process if you find an unacceptable level of problems. Even Jo Johnson accepts that our Universities are world-leading, yet he wants to impose a further burden of evaluation to complement the Research Excellence Framework (REF). This will take money and resources from research and teaching and so be counterproductive. I would suggest that instead of a TEF we need:

  • A system of quality control that will scrutinise any institutions that show signs of failing (e.g. because of high level of complaints). We already have this with the Quality Assurance Agency (QAA).
  • Incorporation of some information about teaching quality in the REF Impact section – see this proposal by Roger Brown.
  • Disincentives for Universities to use zero-hours contract staff for teaching; this could include a section on staff well-being and moral in the Environment section of the REF.
  • Incentives for sharing good and innovative teaching practices

Page 1 para 1b. ‘The TEF will provide the financial and reputational incentives for providers to prioritise teaching excellence and student outcomes. Genuine and clear differentiation between providers and a link with fees are essential to achieving these outcomes.

The downside of yet another ranking system are simply ignored here. What is needed is a diverse higher education system, with different institutions developing their own specialised approaches to suit the very varied needs of our population. A crude ranking, further subdivided into Gold, Silver and Bronze, treats higher education as some kind of horse race, where quality can be measured on one dimension.

Page 1 para 2-3 notes that ‘It is important that we treat fairly those who have already decided to take part….The assessment process is already well underway and to change the “rules of the game”… would be extremely unfair on those who have invested their time and effort to participate this year’

I’m tempted to respond, ‘Well, whose fault is that, Minister’. It is Johnson who has chosen to implement an ill-thought-through evaluation system, with threats to participating organisations that they will lose their ability to raise fees unless they take part. And he also promised that TEF would not be onerous. But now we are told that those institutions who have taken part would have invested time and effort and so it would be unfair to stop. One is reminded of someone who orders a meal that makes them sick but persists in eating it, because the money has been spent.

What is really unfair is to adopt a system of evaluation that is based on unsuitable metrics, is incapable of making meaningful distinctions between institutions, yet which has the potential to damage the standing of a University.

Page 2, para 2. ‘We recognise the genuine and considered concerns raised by noble Lords about the TEF, in particular around the speed at which it is being implemented and on the use of metrics and ratings….a genuine lessons-learned exercise will take place after this trial year.’

Well, at last it seems there is recognition that the TEF in its current form is unworkable. Criticism has come from individuals who might be expected to support it; see e.g. this account of an interview with Chris Husbands, Chair of the TEF Panel on Jan 25th in the Times Higher:

‘“I do not think student satisfaction is an accurate proxy for teaching quality,” said Professor Husbands, a former director of the UCL Institute of Education who was named chair of the TEF panel by the Department for Business, Energy and Industrial Strategy in August 2016.

Professor Husbands drew attention to US studies that show that student satisfaction scores are heavily influenced by non-academic factors, including the sex or ethnicity of a lecturer.

“Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” he said of the US research findings.’

And Stuart Croft, VC at Warwick on Jan 31st wrote in the Times Higher ’…the metrics are flawed. This is not renegade opinion, but rather the overwhelming view of those actually involved in higher education….Yet the government has us over a barrel. It has linked the TEF to tuition fees and, potentially, our ability to recruit international students.

Page 3. Para 1 states that ‘…the TEF is about much more than metrics. Providers submit additional evidence alongside their metrics and this evidence will be given significant weight by the panel.’

But what is the reliability and validity of the ‘additional evidence’? This seems like encouragement to Universities to tell plausible stories about their teaching and try to anticipate which buttons they need to hit to convince a panel that they are Gold rather than Silver and Bronze. There is huge scope here for subjectivity at best and corruption at worst.

Page 3. Para 2 state ‘All of the metrics used in the TEF are trusted, widely used and well established in the sector….. The NSS is just one of three principal sources of metrics data being used, and we have explicitly said that the NSS metrics are the least important. The TEF does not use in any way the overall ‘satisfaction’ rating, about which the House has rightly expressed concern, but instead uses specific questions from the NSS, related to teaching and learning.’

It’s hard to know where to start with this. There are numerous issues with using the NSS for assessing teaching quality, and changing the items that are used does not solve them. There are problems with validity – does the measure assess what it is supposed to measure, bias – are ratings affected by the teacher’s gender, race or disability, and sensitivity – is the spread of scores adequate for differences between institutions to be meaningful? Statistical criticisms have been made by both the Royal Statistical Society and the Office of National Statistics. We are told in para 2 that ‘the Government has already taken a number of steps to ensure that the statistics being used are robust, including commissioning a report by the Office of National Statistics.’ They seem to think that the ONS will somehow be able to magically transform an unreliable, insensitive scale into a meaningful indicator of teaching quality. Statisticians are good, but not that good. Furthermore, all the statistical problems that afflict the TEF are magnified with small samples, and small samples are going to be inevitable if we move, as we have been told we must, to subject-specific assessments.

It is depressing, though not surprising, to find that the letter ends by telling us the university sector must be ‘shaken out of its complacency’. It’s an easy but lazy point to say that opposition to the TEF is just the response of a complacent group of academics who are set in their ways, because it means you don’t actually have to consider the arguments. I suspect that Johnson would be surprised to find just how passionately many of us care about our Higher Education system. If he could grasp that point, he might start to take note of us when we tell him his superficial and rushed approach to evaluating teaching is resisted because of the considerable potential it has for harm.

 

 

 

 

 

 

 

Update on Amendments to the Higher Education and Research Bill

David Midgley has kindly provided this useful summary of recent developments:

The text of the Bill as amended by the Lords in report stage is now available at https://www.publications.parliament.uk/pa/bills/lbill/2016-2017/0112/17112.pdf.

The most significant changes with regard to the regime proposed for higher education are these:

  • Lord Kerslake’s amendment of 6 March to the effect that the TEF “must not be used to rank English higher education providers as to the regulated course fees they charge”, carried by 263 votes to 211, which is now Clause 12;
  • Lord Blunkett’s amendment of 8 March, carried by 280 votes to 186, which requires any “scheme to assess and provide consistent and reliable information about the quality of education and teaching” to be subject to rigorous independent scrutiny, and precludes it from being “used to create a single composite ranking of English higher education providers” – this has become Clause 27;
  • Baroness Wolf’s amendment of 8 March, carried by 201 votes to 186, which makes explicit the role of the Quality Assessment Committee in relation to the authorisation of any new provider and requires such a provider to have been “established for a minimum of four years with satisfactory validation arrangements in place” before authorisation may be issued – this has been incorporated into Clause 47 (as sub-sections 12 and 13); and
  • Lord Judge’s amendment of 8 March, carried by 185 votes to 151, which has the effect of strengthening the power of appeal against a revocation of degree-awarding powers and/or university title – the relevant clause is now Clause 49.
  • Also of considerable importance for the future of universities is Lord Hannay’s amendment of 13 March, carried by a noteworthy 313 votes to 219, which seeks to protect the position of international students in British higher education – this appears as Clause 90 in the latest version of the Bill.
  • Also carried on 6 March, by 200 votes to 189, was Baroness Royall’s amendment, which requires that all eligible students should have “the opportunity to opt in to be added to the electoral register”, which has now been incorporated into Clause 16.

 

A ‘cohesive and self-critical academic community': an ideal under threat

Opinion piece

G.R.Evans

An article in Times Higher Education on 2 March collected suggestions from a ‘panel of university administration experts’ on the best way to frame a job advertisement. Wondering where suitably qualified and ‘excellent’ teachers are going to be found to staff an ever-expanding English higher education system as the TEF evolves, I have been exploring such advertisements in recent months.

If institutional autonomy is to remain a fundamental of higher education provision, with new providers trusted to set and maintain standards, what does that imply for the recruitment of a provider’s academic staff? Among the Government’s background papers published in connection with the Higher Education and Research Bill is the Factsheet on Degree Awarding Powers and University Title (January 2017). It explains that the proposed:

reforms are aimed at removing unnecessary barriers that may currently stand in the way of providers that can demonstrate that they have:

  • the ability to design and deliver high quality HE degree courses,
  • the ability to set and maintain academic standards, and
  • their teaching is informed by scholarship and research.

So it is recognised that having academic staff of sufficient quality remains fundamental to justifying the institutional autonomy of new providers as much as to satisfactory ‘teaching performance’ in the TEF.

A teacher at higher education level has traditionally been expected to have academic qualifications over and above those needed to teach to A-Level GCE (Level 3 of the Qualifications and Credit Framework (QCF) ). These are defined by the Quality Assurance Agency (QAA) in line with Government guidance for the grant of degree-awarding powers and university title. The guiding principle is that lecturers in higher education should be qualified to teach at the level of the award offered, with research degrees therefore requiring research-active staff. The Government guidance was revised in September 2015 and further revision is expected to be needed under the provisions of the Higher Education and Research Bill s.40 and following.

The range of subjects in which it is possible to gain a degree has expanded hugely in recent years and so have the modes of study. ‘Workplace learning’, apprenticeship degrees, the need to meet the accreditation requirements of professional and regulatory bodies, vocational courses, frequently call for non-traditional teaching staff. ‘Vacancies’ advertisements for academic staff now often include terms such as ‘professional qualification’ or ‘experience’ as alternatives to the possession of even an undergraduate degree.

There is a further longstanding expectation, going beyond recruiting suitably-qualified academic staff piecemeal. This is that an institution’s academic staff will form a ‘self-critical academic community’ in setting and maintaining standards. In 1985 the Lindop Report (2.5) said that:

Traditionally the university degree was an acknowledgement, by the self-governing community of scholars which constituted the university in its wider sense, of academic attainment such as to warrant full membership of the community.

Sir Keith Joseph, then Secretary of State for Education and Science, added in a House of Commons Written Answer 17 March 1986:

The Government share the [Lindop] committee’s view that the most effective safeguard of an institution’s academic standards is the existence within it of a strong, cohesive and self-critical academic community. In future, the main purpose of external validating arrangements must be to foster the development of such communities.

Kenneth Clarke, as Secretary of State for Education and Science, maintained the principle in the House of Commons, 16 December 1991, speaking of

Criteria for Degree Awarding Powers

The principal criterion would be that any institution seeking degree awarding power for taught courses would need to be a self-critical, cohesive academic community with a proven commitment to quality assurance supported by effective assurance and enhancement systems.

Jo Johnson, in the House of Commons Higher Education and Research Bill Committee, 15 September 2016, reiterated the expectation that a university is: expected to be an institution that brings together a body of scholars to form a cohesive and self-critical academic community that provides excellent learning opportunities for people, the majority of whom are studying to degree level or above. We expect teaching at such an institution to be informed by a combination of research, scholarship and professional practice.

This expectation that providers with degree-awarding powers or university title will be entitled to autonomy because they employ a critical mass of appropriately qualified academic staff therefore remains fundamental.

The Factsheet gives three Case Studies, describing scenarios in which a new provider might apply successfully for degree awarding powers, all making that assumption. In all three the key element is the presence of a group of ‘experienced academics’. One is a company wishing to set up a new specialist provider. It ‘employs a group of experienced academics‘. Another is an existing ‘world-renowned’ US provider wanting to set up in England and offer English degrees. It sends across the Atlantic ‘an experienced team of academics’. A third is a ‘spin-off’ from a ‘top-ten University’ initiated by ‘a group of leading academics’.

But in the real world much has changed in the patterns of recruitment of academic staff. Advertisements mentioning research as an ‘essential’ requirement now cluster in the older universities, though a number of the newer providers say the possession of a doctorate or a willingness to seek one is ‘desirable’. There is a visible trend towards increasing the number of teaching-only contracts even in ‘research-intensive’ universities. Where the degree is in a vocational subject the job advertisement may be adjusted accordingly. Bought-in and externally assessed courses such as Pearson-Edexcel HNC and HND courses at levels 4 and 5 may plausibly be delivered by teachers without the traditional qualifications of a member of the academic staff envisaged by Lindop and the policy-makers who continue to quote him.

The range of types of employment contract on offer in higher education has been broadening in the direction of the ‘insecure’, though in what proportions it is hard to quantify exactly. Data provided by the Higher Education Statistics Agency includes information on fixed-term contract numbers, but does not make visible in detail the numbers or proportions of staff on hourly-paid or zero-hours contracts. (Some individuals may of course seek a zero-hour contract because of its flexibility, but the wording of job advertisements suggests it is rarely a matter of choice):

UCU has always been clear that those people returned by universities as atypical academics only represent a part of the real hourly-paid workforce. The rest are concealed within fixed-term contract data.

This pattern of recruitment undermines the expectation that providers will maintain an established ‘community’ of academic staff.

There seems to be a need for some joining-up of ideal and reality to ensure that before the OfS grants degree-awarding powers or university title, the new providers warrant the institutional autonomy they will gain if they succeed by showing that in their lecturers they have that ‘critical community’ of academic staff.

 

 

 

 

 

 

 

 

 

 

 

 

 

New DAPs anyone? Is the proposed multiplication of types of degree-awarding powers a good idea?

Preamble: The Higher Education and Research Bill will reach undergo line-by-line examination in the House of Lords on 6th March, when it reaches the Report Stage. This post is the third in a series of opinion pieces by G. R. Evans, reflecting on the Amendments tabled prior to this stage.

Opinion piece

by G. R. Evans

Concerned comment has focussed on the plan set out in the Higher Education and Research Bill to allow ‘high quality’ new providers to grant degrees on a probationary basis for three years rather than having to learn the ropes first under the supervision of an experienced ‘validating’ provider.

Even that existing common-sense provision has led to problems. The QAA’s Quality Code (B10) covers the whole field of collaborative and validating provision under the heading Managing higher education provision with others. This has been designed in the light of a history of disasters, from the ‘franchising scandals’ of the late 1990s to the collapse of the University of Wales when it had overreached itself in entering into validating arrangements. Years of active QAA consultation on all this can be found in a moment’s internet search.

To allow a brand new provider to plunge straight into the granting of degrees is clearly a high-risk strategy, as the Government seems to be realising. In the general discussion of the Amendments to the Bill published in late February is a reference to a paper published in January entitled Market Entry Reforms. That in turn refers the enquirer to another paper, where, it says, ‘we set out more detail on this and other changes to Degree Awarding Powers, as well as linked reforms to University Title’.

This further Factsheet paper explains that:

Our reforms are aimed at removing unnecessary barriers that may currently stand in the way of providers that can demonstrate that they have:

  • the ability to design and deliver high quality HE degree courses,
  • the ability to set and maintain academic standards, and
  • their teaching is informed by scholarship and research

Whereas the existing process for gaining degree-awarding powers requires a provider to demonstrate that it has actually been doing these things in a satisfactory manner for a period of time (thus satisfying Government Evidence Criterion B), the new proposals recognise that something different will be needed for a complete beginner. It will merely need to provide details of its plans:

Policies, procedures and guidance associated with programme development, approval and review; Policies, procedures and guidance associated with assessment of students and external examining; List of external examiners (if available); Terms of reference, constitution, reporting line and minutes for last two years of any advisory body; any reports from Professional Statutory and Regulatory Bodies

The PDAP (Probationary Degree Awarding Powers) candidate is envisaged as quite possibly offering only a single subject or seeking powers only to offer undergraduate (bachelors) degrees. These, let us call them SSDAPs and T(B-only)DAPS), would be something quite new in the list of DAPS.

It is easily forgotten that until 2004 – and still in Scotland and Northern Ireland – the degree-awarding powers held by a provider which wished to call itself a ‘university’ had to be teaching-and-research powers. Since 2004 in England and Wales a provider may acquire only taught degree-awarding powers (TDAP) and apply to become a university. However, the TDAP has included powers to grant taught postgraduate (usually Masters’) degrees. The T(B-only)DAPs will set a lower requirement.

A holder of SSDAPs at T(B-only) level, like any other new holder of degree-awarding powers, will in future have the powers on a time-limited basis. After three years they can expect to be granted unlimited DAPs.

A holder of indefinite DAPS may then apply for University Title. This will not include holders of only Foundation Degree-awarding powers (FDAPs) at Level 5. These were introduced in 2010, restricted to Further Education Colleges, though automatically included in the DAPS of any publicly-funded provider holding TDAPs. There has been a limited take-up of these so far and they seem to be rather a dead-end at present. HEFCE’s Register lists only half a dozen.

However it seems that the SSDAPs and T(B-only)DAPs holder will be free to apply to become universities.

Companies seeking to use the sensitive word ‘university’ in their titles may still do so:

This does not replace any requirements for consent under the Companies Act 2006, however we do expect all English higher education providers to obtain OfS consent. Where a provider requires a non-objection letter to register a new company or business name that includes the word ‘University’ with Companies House, the intention is that the OfS will issue such a non-objection letter at the same time as granting University Title.

The Factsheet containing these details is reassuring that:

we expect the OfS to set out the approach it will use to identify the ‘institution’ and legal entity that is seeking DAPs.

It is surely important for more of this to be clarified on the face of the Bill, for the multiplication of types of lower-level or restricted DAP constitutes a bigger change than the Factsheet seems to realise

Rival sanctions? The Office for Students and the Competition and Markets Authority

Preamble: The Higher Education and Research Bill will reach undergo line-by-line examination in the House of Lords on 6th March, when it reaches the Report Stage. This post is the second in a series of opinion pieces by G. R. Evans, reflecting on the Amendments tabled prior to this stage.

Opinion piece

by G. R. Evans

The Competition and Markets Authority (CMA) is a non-ministerial government department which works ‘to promote competition for the benefit of consumers, both within and outside the UK.’ Its stated aim is ‘to make markets work well for consumers, businesses and the economy.’

It has undertaken consultations on the fairness of student contracts with higher education providers. CMA advice for higher education providers on how to comply with consumer protection law  was published in March 2015, with guidance on consumer rights for students.

In a Press Release on 8 February, the Competition and Markets Authority drew attention to a compliance agreement reached with the University of East Anglia, stating:

‘Students will receive a fairer deal at the University of East Anglia (UEA) after the CMA secured changes to the university’s contract terms.’

The Press release explained that changes to modules at UEA had been found to have:

made significant changes to the content of a course – by introducing compulsory modules thus limiting students’ choice of optional modules – and not adequately informing prospective students who had received course offers about the changes.

The University has cooperated fully in amending its practice.

The CMA gives details on its case page of six agreements with universities reached since November. Not all were in England (one was with Glasgow).

There are, however, unanswered questions about how the CMA will work with the new Office for Students. Under the Higher Education and Research Bill the Office for Students will be able to suspend or de-register providers which are found to be in breach of their general or specific ongoing registration conditions, conditions which may be imposed by the OfS itself. It is far from clear whether these conditions will or may include compliance with CMA guidance. Perhaps it should be.

When he was questioned by the Education Select Committee at a Pre-Appointment Hearing on February 21 Sir Michael Barber was asked how he envisaged the OfS working with the CMA when he became the OfS Chair. He admitted he had not yet given the matter much thought.

Universities UK have begun some work on ‘how to ensure that fair terms and conditions are sufficiently flexible to allow courses to evolve’:

In order to understand this issue from the student perspective we are doing some work with the polling company ComRes.

This is not, however, the only area in which the CMA Guidance expresses concerns. Another became the subject of a finding and an adjustment in 2016. An agreement had been reached with UCL :

UCL will ensure that students are not prevented from graduating or re-enrolling because of non-academic debts, following action by the CMA.

A further area of concern to the CMA is the now very common claiming by higher education providers of the intellectual property of students, that is if the terms of the student contract (or the also common) inclusion in its statutes and regulations to which the contract binds the student) of a provision to ‘assign all intellectual property rights (IP) for any of [the student’s] work to the university, regardless of the circumstances’(3.6) .

Might or should the OfS impose sanctions on a provider the CMA found to be non-compliant? It may be wise to think this through before the Higher Education and Research Bill has reached the end of its journey towards royal assent.

Why the NSS is garbage

A note by Lord Lipsey    

The National Students Survey results matter. First, they are used by students to evaluate institutions by comparison with rival institutions. Secondly, they are one of the metrics to be used in the TEF, in awarding gold, silver or bronze markings to institutions which apply to take part. These ratings will decide if an institution can or cannot raise its fees beyond £9K.

The idea that student satisfaction should play a major role in the rating of universities is controversial. Research shows that there is no correlation between student satisfaction and student results in terms of degree grade . However the government has opted to increase the importance of student choice, competition and satisfaction in the higher education landscape; and this short note does not seek to address the rights and wrongs of that.

Rather it focusses on a narrow point: whether the National Student Survey (NSS), the chosen instrument to measure student satisfaction, is fit for purpose or not.  Here the evidence is unequivocal, The NSS is statistical garbage, The reasons are widely understood by the statistical community and were set out inter alia by the Royal Statistical Society in its response to the government’s technical consultation about the TEF http://www.rss.org.uk/Images/PDF/influencing-change/2016/RSS-response-to-BIS-Technical Consultation-on-Teaching-Excellence-Framework-year-2.pdf.

  1. The NSS is not based on a random or representative sample of students. It is more akin to a census, which includes everyone who chooses to complete the form but not those who don’t. Some groups eg ethnic minorities are seriously underrepresented. As the final report of the ONS Review of Data Sources for the TEF 2016 said: “under-reporting of certain groups and over-coverage of others …could lead to bias in use of the data.” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/556352/Teaching-Excellence-Framework-review-of-data-sources.pdf

I chair Trinity Laban Conservatoire of Music and Dance, a leading London Conservatoire. This year Trinity Laban’s response rate to the NSS rose from about 60% to about 80%. This makes any comparison between last year’s results and this year’s results invalid. We know nothing about the 20%; so a 50% satisfaction rate amongst respondents could perfectly well be a 70% satisfaction rate amongst students as whole.

  1. Even if the responses are treated as a random or representative sample, making calculations of statistical significance possible, the margins of error in the NSS figures are large. The ONS concluded that “differences between institutions at the overall level are small and are not significant” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/556352/Teaching-Excellence-Framework-review-of-data-sources.pdf Yet on that are being based student decisions on where to go, and the level of fees institutions can charge.

So for example Trinity Laban’s  music results – and music is our biggest group of students -are based on 112 students. To take an example only 49% of these students say that marking on their course has been fair. Statistically however and treating the returns as if they were a sample, this means that there is a 95% chance that between 34% and 64% of students think it is fair. This perhaps exaggerates the unreliability of the NSS insofar as. if response rates are high, then the results may be less unreliable that they would be for a sample. But the point still applies. And.in cases of smaller sub-samples the low response renders the results vacuous. For example for Trinity Laban Musical Theatre the poor results were based on response from just 23 students.

The small sample size is a particular problem for small institutions. It is also a grave problem in that the most valid comparisons are not between institutions, but between particular departments in institutions teaching the same or similar courses. For individual courses, there will tend to be only a fraction of the responses obtained for institutions as a whole.

  1. The results can be greatly affected by happenstance. Results in the TL musical theatre department in the past have been good which suggests that there was some peculiar chemistry about this year (and also there is some suggestion that students coordinated their responses to make a point). A survey of dance satisfaction for Trinity Laban was completed the day after a one-off cock-up about room bookings; had it been done two days earlier the results might have been much better.
  1. There is scope for “gaming” the results and encouraging students to give positive results,. Trinity Laban does not do this. Anecdotal evidence, as cited by the RSS paper above suggests that less scrupulous institutions do, and the incentive to do so will be greatly increased now that the NSS has new significance as a metric for the TEF.
  1. Less concrete but not less important, an emphasis on the student experience may lead to undesirable effects on what they are taught. Institutions focussed on a high NSS score will tend to dumb down degrees or go for safe options on content. ]

Here is an example from Trinity Laban. TL exists due to an amalgamation of Trinity College of Music and Laban dance. A few years ago, the Principal introduced a compulsory two-week course called CoLab which was a creative programme involving dance and music working together. At first students were furious at being deprived of these two weeks off from the focus of their studies – and no doubt this would have been reflected in their satisfaction ratings. However time has passed; familiarity has grown; and CoLab is now one of the most popular things we offer with our students.

In addition, during the ONS review of the NSS,  respondents expressed reservations about wider issues related to the use of information from the NSS and the DLHE. Concerns included:

  • limited variation between institutions of the raw scores from the student responses
  • difficulty in trying to compare widely differing institutions
  • difficulty in capturing the wider benefits beyond academic results of attending a higher education institution

The government has already downgraded the importance of the NSS in TEF – the so-called LSE amendment made when it was pointed out that NSS suggested that LSE and a number of other prestigious institutions were rated low by students. The metric should be subject to intense scrutiny when the House of Lords debates the Higher Education and Research Bill in committee; and I am tabling amendments to make sure it is.

The spurious precision of the NSS has the capacity to damage staff morale, to put students off certain institutions and affect the validity of the TEF. I know of at least once case where a head of department at a major institution came close to resigning because of a disappointing NSS result. I know of another where a Principal was dismissed inter alia because of the failure of that institution’s NSS results to improve. This is terrifying.

Lord Lipsey is joint Chair of the All Party Statistics Group, a Fellow of the Academy of Social Science, a Board member of Full Fact the fact checking charity, a former member of the advisory council of the National Centre for Social Research,  founder of Straight Statistics and a former adviser on opinion polling to James Callaghan as prime minister. In other words, though he may talk nonsense on many subjects, this is not one of them!

The mysterious letters in the library

Opinion Piece

by G. R. Evans

The Higher Education and Research Bill has now had seven sessions before the Lords in Committee, ending on 30 January. As Lord Willetts acknowledged, it has become ‘famous’ for the sheer number of ‘letters’ promised to various Peers at the Committee stage, by those tasked with undertaking the role of the Minister in the Commons: Lord Younger (Spokesperson on Higher Education in the Lords), Baroness Goldie (who stood in for him from time to time) or Lord Prior. Lord Prior was appointed Parliamentary Under Secretary of State at the Department for Business, Energy and Industrial Strategy only on 21 December 2016 and he too found himself unable to provide answers on 30 January when the Lords were discussing UKRI, which is to be the responsibility of his own Department of State. (‘Rather than ad lib on this, I had better consult officials and write to the noble Lord’).

These ‘letters’ occupy an anomalous place in the legislative process. The steps by which a Bill passes into an Act of Parliament ready for Royal Assent are fully transparent in every respect but this. First the Commons then the Lords have their First and Second Readings and their Committee stages, the debates are now broadcast live and a verbatim account may be read in Hansard the next morning. But when a Minister cannot give an adequate response to a question raised in debate he may simply offer to write one of these clarificatory ‘letters’ which, unlike Written Answers to Questions put by MPs or peers, do not appear in Hansard.

The assiduous reader of Hansard may want to know what the promised letter says. Even Lord MacKay of Clashfern, who has been a member of the House of Lords since 1979, declared himself not sure in the debate on 30 January:

My noble friend mentioned a letter. I was at a meeting last week with a number of people interested in the Bill and its progress [the CDBU Annual Lecture given by Martin Wolf]. They mentioned the letters referred to in Hansard. They asked where they could see them. I was not certain, but I assume they are in the Library.

It is possible to find out. Such letters may indeed be deposited in the Library of the House of Commons or the House of Lords by a Minister or by the Speaker. However, the online search for a deposited letter is a slow business because one may search under only limited fixed headings. There seems no systematic linkage of a given letter with the exact passage in the debate to which it refers, though it would surely be easy enough to provide that link.

It all looks rather a muddle. In the case of the letters so far written to Lords during the Committee stage some appear in duplicate with more than one number; some with the same text are written separately to different Lords by name. Appendices said in a letter to be attached to it are often not there.

There seems no easy way for Lords to discover what has been deposited, no routine notification of deposit of a particular letter, no listing of letters outstanding and not yet written (as there is for Questions requiring Written Answers). Deposited papers are not ‘presented or laid formally before Parliament’ though the public may access them and members of either House may now ask to have a particular letter by email. So there is some degree of transparency and orderliness. But at the crucial final stages of the Bill it will be hard work for Commons, Lords or general public to be confident that all the Government’s arguments including those in the letters are marshalled together for detailed review. The ‘all Bill documents’ link on the Bill’s website does not include them. It should.

These letters on points which baffled Government spokesmen in debate are surely too important to be treated so casually in the legislative process. If as appears to be acknowledged, the number promised in the committee stage of the Higher Education and Research Bill is quite exceptional, the need to examine the role of such letters in an otherwise transparent legislative process seems clear.

It is especially important in the case of the Higher Education and Research Bill, of which Baroness Deech said at the end of the final Committee session that it ‘is not much more than a framework, albeit a very heavy framework, on which later policy is to be hung’:

Now that we have reached the end of the Committee with so many gaps in the Bill, can the Minister assure us that there will be some process of post-legislative scrutiny to ensure that this experiment is working?

Those gaps might indeed be filled by the promised letters, of which comparatively few appear yet to be available, but where they are so numerous and extensive they should surely be provided, online, in a consolidated set. Lord Watson put his fear frankly:

the Minister sought to reassure noble Lords that he will reflect on all amendments. That will be of very limited value if, at a later date, he simply comes back on Report or in letters to say that, having reflected, he is not minded to accept the amendments’ (Lord Watson, 11 January).

Those letters surely ought to set out the process of ‘reflection’ in giving their promised answers or Government will have failed shamefully to explain itself.

 

Uses and Abuses of Economics in the Debate on Universities

Report on CDBU Annual Lecture 2017

by Dorothy Bishop

Last night, Martin Wolf, chief economics commentator at the Financial Times, delivered a splendid lecture with the title “Uses and Abuses of Economics in the Debate on Universities”. It is not possible to do this justice in a brief report, but I aim here to give a synopsis of the main arguments, which are highly relevant to our concerns over the Higher Education and Research Bill, currently under discussion in the House of Lords.

Wolf drew a distinction between three aspects of economics relevant to higher education:

  • The ends to be served by higher education
  • The means used to serve these ends
  • The resources that the government should invest in higher education

The principal point he made about the ends concerned the government’s conceptualisation of what higher education is for. A stated goal of the Higher Education and Research Bill is to achieve a successful ‘knowledge-driven economy’. This, said Wolf, is entirely reasonable, insofar as we need highly-skilled citizens to maintain our prosperity. The problem, however, is that the government’s plans appear to see this as the predominant goal of Higher Education. A point that was clearly understood by the CDBU members in the audience, but by few in government, was that higher education also serves the goal of creating enlightened citizens, capable of rational argument and evaluation of evidence, who will influence society not only in the UK but in the wider world. Yes, government as a funder has a right to ensure its money is properly spent, but our universities are threatened if this is taken to mean that the sole criterion for success is an economic one.

The means proposed to achieve the government’s ends is market competition. Wolf noted that the word ‘competition’ occurred in the White Paper introducing the Bill fifty times. Yet he pointed out that the notion of a market for higher education was deeply flawed on several counts: one simply cannot treat universities like businesses selling goods, because what they provide cannot be evaluated in advance. Failure of universities has dire consequences for students, for example, as would encouraging for-profit providers to enter the sector without adequate scrutiny or a substantial track record. Perhaps even more important, a profit-seeking institution cannot be a university in the full meaning of the term: if incentives for providers are solely financial, then we risk losing the very quality that makes our universities so well-respected internationally: the focus on innovative research and intellectual inquiry for its own sake.

Of course, the government is aware of some of these difficulties, but their attempt to deal with them, by creating the Office for Students, creates more problems than it solves. Wolf expressed concern about the sweeping powers proposed for the OfS, in what he termed ‘a fully-fledged government takeover of the UK’s university sector’. He added that ‘Anybody who thinks this will end with more diverse, more innovative, more courageous and more independent institutions is surely a fool.’

Wolf then turned to consider the financial resources available for Higher Education, noting that although the UK spends a slightly higher proportion of GDP on tertiary education than other European countries, a relatively high proportion of this now comes from private sources.  However, UK universities have been remarkably successful producers of high-quality research, and Wolf linked this to the way in which government has funded research. Without this public funding, we would not be able to continue as a global superpower in higher education. Wolf did not discuss student loans in any detail, but he agreed with the view that it was more equitable to rely partly on loans than entirely on tax-funding, although it is certain that a part of this debt, possibly a substantial part, will ultimately be picked up by the taxpayer because not all fees will be repaid.

Given his criticisms of the Higher Education and Research Bill, it is interesting to consider what alternative approach Wolf would like to adopt. Here, he argued, the problem was that the government’s proposals were not radical enough. They persist with the traditional approach of equating higher education with the university sector and ignoring the rest of tertiary education. What about the high proportion of the population for whom university is not a desirable goal? They have long been neglected, both in terms of funding and in terms of post-school educational options. Wolf argued that we should extend the loans system, so that people could access different types of education throughout adult life, and that a much wider range of tertiary education options should be made available.

Wolf started his lecture with a quote from H. L. Mencken “For every complex problem there is an answer that is clear, simple, and wrong”. He made a compelling case that the funding and organisation of our system of higher education system is a complex problem that requires a complex solution. By seeking to frame the issue in terms of simple-minded market economics, the government had got it badly wrong. They need, he concluded, to think again.

 

 

 

 

 

The definition of a university: how many new providers fit the bill?

Rapid reaction to the House of Lords debate on the Higher Education and Research Bill

by

G.R. Evans

The headline-grabbing result of the first session of the House of Lords in Committee on 9 January was a vote which defeated the Government. By a majority of 248-221 the Lords approved Amendment 1, which was to provide a first Clause defining ‘university’. The Amendment assumes that higher education teaching must have the support of scholarship and research. This is high stakes: many of those wanting to enter the market would not meet these criteria.

The Amendment by the House of Lords: functions of a University

Screen Shot 2017-01-17 at 18.00.16Were the Lords wise to seek a definition? In a blogpost in Times Higher Education, Nick Hillman thought not. Agreeing with some of those in the debate, he argued that there never had been a definition of university in English law, and that to have one would tend to prevent improvement and innovation.

However, Lord Krebs was voicing the majority opinion when he called on the Government two days later when debate resumed, to ‘define your terms’. More defining of terms was already in prospect when on Wednesday, the Lords tackled Clause 2.

What is a regulator?

There was a call to define ‘regulator’. Lord Stevenson of Balcamara said ‘it would be useful and comforting if the Minister could write to us explaining exactly what the term regulator implies. That would give reassurance to some of us who have been worrying about this issue’. Baroness Wolf noted a particular difficulty:

we are not quite clear whether [OfS] is a regulator or not, and we also have the Competition and Markets Authority. One question that I have is whether there are incipient conflicts between these two important and powerful bodies.

What is the meaning of higher education?

In framing Amendment 72, Lords called for another definition. This was to be a new Clause headed ‘Meaning of higher education’. It would come at the end of Clause 2, setting out the ‘general duties’ of the OFS, and say:

For the purposes of this Act, the provision of higher education by English higher education providers comprises higher education provision by—

(a) universities,

(b) colleges of further education, and

(c ) other higher education providers, both registered and unregistered.

It was in the list of Amendments (34-59 )‘not moved’ that evening.

The importance of research for universities

Where does this leave the definition of a university as approved by the Lords on 9 January? On 11 January, discussing the importance of ensuring that the new Office for Students (OfS) and the new UK Research and Innovation (UKRI) bodies would work together, speaker after speaker stressed the importance of the research element in higher education teaching, the stimulus it offered to students, the edge it gave to have the content of what was taught visibly touching the boundaries where knowledge was being expanded.

But two days earlier, Lord Younger, responding to the debate so far, took a different view:

while I agree that teaching should be informed and supported by scholarship and research, I have to agree with the changes made under the Labour Government in 2004. As my noble friend Lord Willetts explained, those changes to the criteria for university title removed the requirement for universities to need to award research degrees …. The amendment would be a regressive step. The changes were rightly made … and recognised that teaching is a legitimate primary activity for a university. If we place barriers in the way of new and innovative universities, we risk diminishing the relevance and value of our higher education sector to changing student and employer needs—becoming a relic of the 20th century while the rest of the world moves on.

This is a significant speech, whose implications should not be lost sight of. It contrasts ‘must’ and ‘may’, a distinction which was made much of on 11 January in the attempts of peers to persuade the Government to make OfS’s duties expressly into ‘musts’. While the Lords’ Amendment speaks of UK universities, Lord Younger defended a change which applies only to those in England and Wales. There have been calls for more attention to be paid to ‘national differences’. This was the decision of 2004 to allow providers with only taught-degree-awarding powers to gain university title. This, he argued on 9 January, meant that teaching was recognised as ‘a legitimate primary activity for a university’. In reality it amounted to something more, a recognition that teaching might be the only defining activity of a university.

Neither Scotland nor Northern Ireland has so far seen advantage in moving in the same direction and removing the requirement for a provider to obtain powers to award research degrees before becoming eligible to call itself a university.

Teaching-only ‘universities’

In England university or ‘university college’ title has been bestowed on seven alternative providers, all – except the longstanding University of Buckingham which had gained its title decades earlier and before this change – with only taught degree-awarding powers. Arden University; BPP University Ltd.; University College of Estate Management; the University of Law Ltd; the London Institute of Banking and Finance’; Regent’s University, London.

Several of these fulfil another desideratum expressed by Lord Younger in the same speech, that ‘specialist’ universities should be encouraged. The Bill envisages even single-subject universities. Yet none of them can be said to be strikingly ‘innovative’. Apart from the long established College of Estate Management, which does what its name suggests, they all concentrate mainly on business and management, law, accounting and finance, sometimes with tourism, health and social care. This is exactly the range of subjects mainly offered at Level 4 and 5 in the hundreds of alternative providers which run Pearson-Edexcel courses.

Where do we draw the line?

The Minister’s claim that that the Lords’ Amendment represents merely the vested interest of a ‘cartel’ of publicly-funded research-active universities should not be taken seriously until the Government can provide a coherent and evidence-based account of the present reality and the acceptable range of future inclusion of novelties.

Where does Jo Johnson think the boundaries of higher education diversity should be set? Hamburger University is real enough, and McDonalds offers a Foundation Degree in the UK, ‘certified by Manchester Metropolitan University’, but that presumably counts as one of the ‘cartel’ keeping innovative specialist providers from moving straight to their own probationary degree-awarding powers and then to university title. The Faculty of Astrological Studies published its examination results at the recent Winter Solstice when ‘Saturn its lord and god of the harvest, is elevated in the sky together with the Sun in the ingress chart for London’. If this provider would not be eligible to seek degree-awarding powers and university title should the Government be able to explain why not?

Screen Shot 2017-01-17 at 17.52.03

Universities as cartels?

Opinion piece

by James Ladyman

It is said that trust in institutions is in short supply these days. Sadly the government doesn’t trust universities. According to the Minister, the Higher Education and Research Bill is necessary to “break open a closed shop that for too long has set the rules of the game in its own interests”. He does not believe that academics who act as external examiners and universities that oversee the provision of teaching make and apply rules in the interests of academic standards and education, nor that their deliberations are informed by the wider political and social good. According to the minister, and one must presume the entire government and many MPs and peers, universities have been acting in their own interests all this time. This makes sense to them because they also insist that universities are businesses and encourage them to behave as such. Businesses’ primary interest is in maximizing their revenues. Since, universities are businesses their primary interest must be in maximising their revenues.

The new providers that are allegedly needed to drive up teaching standards – the minister is disparaging about the latter – do indeed include businesses interested only in making a profit. They contrast with universities that hitherto have hosted academics who collaborate across institutions for the good of their disciplines. Yet the minister thinks of the sector as ‘cartel-like’.

We all know how effective cartels allow prices to be kept artificially high from the way that energy market has functioned. The measures in the Bill are supposed to protect students from a similar cartel-like scam in HE. It will supposedly do this by allowing fees to be raised. Yes, that’s right folks, the market will save students from being ripped off by the existing university system, by allowing universities to raise prices.

Despite the disastrous effects of marketisation on the health service and the penal system, and with no regard to the outstanding international reputation of British higher education, the HE Bill proposes to treat the healthy patient with bad medicine.

On the subject of the closed shop, one might as well ask why we let the medical profession control who becomes a doctor rather than the market. We don’t expect patient choice to set standards in clinical care, and we should no more expect student choice to set standards in higher education.

We have not been given any explanation for why the idea of the self-critical academic community enshrined in established thinking about academic and educational standards has been set aside so completely.

Students are entitled to know that their fees are paying for a decent standard of education, but this government is determined to sacrifice them in the interests of the profits that the new providers will make out of their tax-payer funded loans.