The main points can be summarised as follows:

The committee needs to take a close look at the purpose of the REF; there has been considerable mission creep and it is trying to do too many different things. Its cost-effectiveness has never been properly evaluated.

Specific suggestions are:

• Reward institutions that have high levels of staff satisfaction.
• Reward institutions that foster early-career researchers.
• Consider a system where funding reflects the number of research-active staff who have contracts extending into the future.
• Do not use incentives that treat accumulation of research funding as an end in itself.
• Do not award a higher proportion of funding on the basis of impact.
• Do not rely on competition to drive up standards: create incentives for more co-operation both within and between institutions.
• Take steps to encourage a diverse research landscape, rather than creating further concentration of research in a few institutions.
• Be vigilant about the dangers of introducing criteria that might work well in one discipline but be unsuitable for others

We thank those CDBU members who contributed to our submission, and look forward to covering future developments on our website.

I spent Sunday reading the Green Paper “Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice“, a consultation document that outlines radical plans to change how universities are evaluated and funded. The CDBU is preparing a response, but here’s the problem. BIS is not seeking views on whether the new structures they plan to introduce are a good idea. They are telling us that they are a good idea, a necessary idea, and an idea that they will implement. The consultation is to ask for views on details of that implementation.

The government will no doubt be braced for howls of protest from the usual suspects. Academics are notorious for resisting change, so there is an expectation that there will be opposition from many of the rank and file who work in universities, especially from those whose political allegiances are left of centre. CDBU is, however, a broad church, and disquiet with the Green Paper comes from academics covering a wide range of political views.

The idea behind the TEF is that teaching has not been taken seriously enough in our Universities, because they have been fixated on research. As a consequence, students are getting a raw deal and employers are dissatisfied that graduates are not adequately prepared for the workplace. However, the evidence for these assertions is pretty shaky. If you’re going to introduce a whole new administrative machinery, then you have to demonstrate that it will fix a problem. A number of commentators have warned that TEF is a solution to a problem that does not exist.

Let’s look at the three premises behind the justification for TEF.

Premise 1: A focus on research has led to neglect of teaching in our universities. No evidence is presented for this claim. If chasing a high REF score led to neglect of teaching, we might expect to see poorer student satisfaction ratings in institutions who did well in the REF. We don’t. This is a complex issue, not readily reduced to simple indicators, but insofar as there is a problem, we could fix it by adopting Roger Brown’s suggestion of including evaluation of teaching in the next REF, rather than setting up a whole new exercise.

Premise 2: Students are demanding ‘higher quality, transparency and value for money’. No source is given for this claim. According to HEFCE’s report on the 2015 National Student Survey, “Overall satisfaction levels among students have increased steadily”. We should, of course, never be complacent, and the overall positive picture may hide some weaker performers. But are the problems really so severe that they merit an expensive bureaucratic exercise in evaluation that takes in all HEIs in the country?  Most of the system is working to a high standard, and much of it is outstanding. This is the reason why the UK is so popular with overseas students.

Students need to make well-informed choices, but so do those introducing major administrative changes to our Universities. Let’s go for evidence-based decisions, instead of making up policy on the hoof. There are plenty of good social scientists in our Universities who could tender for conducting a survey to discover what information students used when selecting a course, and how their expectations matched up to reality.

Premise 3: The Green Paper states that “While employers report strong demand for graduate talent, they continue to raise concerns about the skills and job readiness of too many in the graduate labour pool.” Here there is at least a source: a report by BIS. However, this explicitly stated that it was based on qualitative rather than quantitative research and: “Although a large and diverse range of employers were included in this research, the interview data cannot be interpreted as being statistically representative of all graduate recruiters in England or be used to describe the numbers and proportions of organisations displaying particular characteristics or behaviours.” (p. 11).

The BIS report noted that some employers mentioned problems: skill shortages in some areas, and poor quality applicants in others. Nevertheless, it was found that “In general employers are satisfied with the graduates they recruit, with more than four out of five feeling graduates are well prepared for work and similarly the employers in this study were also mostly satisfied with the graduates they had hired” (p. 13).

When the stated rationale for change does not bear scrutiny, one has to ask whether there is a hidden agenda. In the case of TEF, it is barely concealed: the outcomes from TEF are to be used to determine whether or not Universities can raise their fees, to facilitate the entry of new providers, and to take further the programme of transforming the Higher Education sector into a free market.

I suspect the government anticipates that it will be easy to sell this new vision to Vice Chancellors, many of whom seem influenced more by money than ideology. They have gone along with other major changes, notably introduction of and escalation of student fees, and implementation of the Research Excellence Framework (REF). But will they support Jo Johnson’s Teaching Excellence Framework (TEF)? I suspect not, because it doesn’t represent a good deal for them. Students now bear the cost of their higher education, and the benefits of a good TEF will be very different from those of a good REF. A high REF score affected the amount of money the University received from central funds, as well as reputation. A good TEF will allow the University to increase their fees, but, as UCL Provost Michael Arthur pointed out, it does not look as if the additional income would compensate for the cost of the exercise: “it is fairly clear that this doesn’t represent the way forward to the sunny uplands of financial sustainability, and at this level we would be unlikely to submit to TEF for purely financial reasons!”

I could say much more about further ideological and logical objections to TEF, but these points have been better made by others, such as Sheffield VC Keith Burnett, and those giving oral evidence to BIS on this topic.

If you think universities should challenge the premises of the Green Paper, rather than merely consulting on its implementation, please lobby your Vice Chancellor on this point – and join CDBU. We need members who share our concerns about the future of British Universities.

Introduction

In Everything for Sale? with Helen Carasso (Routledge, 2013) the writer argued that the main changes in higher education policy over the past thirty or so years could be explained in terms of the progressive marketisation of the system by governments of all political persuasions, a process that began with the Thatcher Government’s abolition of the subsidy for overseas students from 1980. The Green Paper Fulfilling Our Potential: Teaching Excellence, Social Mobility and Student Choice (BIS, 2015) published on 6^th November represents the latest stage in this process. This short paper offers an initial assessment of the main proposal: the introduction of a Teaching Excellence Framework.

The Teaching Excellence Framework

The Green Paper implies that both quality of and participation in higher education have increased since the full fee regime came into effect in 2012. However:

More needs to be done to ensure that providers offering the highest quality courses are recognised and that teaching is valued as much as research. Students expect better value for money; employers need access to a pipeline of graduates with the skills they need; and the taxpayer needs to see a broad range of economic and social benefits generated by the public investment in our higher education system (page 18).

The main proposal for achieving these is the Teaching Excellence Framework (TEF). We are told that:

The TEF should change providers’ behaviour. Those providers that do well within the TEF will attract more student applications and will be able to raise fees in line with inflation. The additional income can be reinvested in the quality of teaching and allow providers to expand so that they can teach more students. We hope providers receiving a lower TEF assessment will choose to raise their teaching standards in order to maintain student numbers. Eventually, we anticipate some lower quality providers withdrawing from the sector, leaving space for new entrants, and raising quality overall. (page 19)  

Will the TEF raise teaching standards?

The Green Paper states (page 11):

To be able to make the best choices about where and what to study, individuals need access to robust, timely and objective information regarding the quality of teaching they are likely to experience and what this is likely to mean for their future employment.  

It claims (page 31) that the aims of the TEF can be achieved through ‘a simple, robust system’. But to link student choices or funding judgements to assessments of quality at institutional level requires the following conditions to be met:

• it must be possible to produce valid and reliable information about the relative quality of programmes and awards in a subject or field of practice at different institutions (and aggregate them to institutional level);
• this information must be available in a timely, accessible, economical and equitable fashion;
• the information must be related to the individual student’s wants, needs, resources and circumstances;
• it must be interpreted in a rational, or at least semi-rational, manner by the student or those advising them;
• it must lead to action by providers to adjust price and/or quality.

To begin with, to be able to make robust judgments about the comparative quality of programs and awards at different institutions would require:

• the programmes to be reasonably comparable in aims, design, structure, content, learning outcomes, delivery, support, learning environment, resourcing, mission, ethos, etc.;
• the associated awards to involve comparable assessment approaches, methods, criteria, procedures, outcomes;
• the assessment judgments to be valid, reliable, consistent and fair;
• the students pursuing the programme, or interested in doing so, to have comparable starting attainments, aspirations, motivations, learning objectives, etc.

One has only to reflect on these points for a moment to see how unlikely it is that these conditions can be fulfilled in the complex, mass system that now exists in England, much of it in direct response to student preferences. In effect, it would require a common curriculum for every subject or combination of subjects on offer, with tests administered by system examiners, something that is virtually unimaginable even by the present Government.¹

But even if these problems could be overcome, the other difficulties would remain:

• the information needs to be provided in advance, yet student education is a ‘post-experience good’(Weimer and Vining, 1992), the quality of which cannot be established until after, often well after, the course concerned;
• in a mass system with a huge diversity of students, it is hard to see how the information can take proper account of individual students’ backgrounds;²
• the information needs to be balanced, accurate and fair, but what incentives do institutions have to ensure this, against the need to attract students by whatever means they can (Watson, 2008)?;
• in accord with what we know from behavioural economics, students are no more ‘rational’ in the choices they make than other categories of consumer. Yet if they don’t act rationally, it is not easy to see how institutions can make the appropriate responses;
• finally, all of this assumes that the student is merely a passive recipient of their education, yet it is a commonplace of the pedagogical literature that the student is not only a powerful constituent of their own learning but also of that of others (‘co-production’). Even if all the other requirements could be met, it is very hard to see how any system of information can accommodate this dimension.

It follows that, rather than improving teaching quality by linking it to funding, the TEF is much more likely to absorb and divert institutional resources that should be being used for quality assurance into gaming the system. This would be a great pity, because there is plenty of evidence that the preference that some academics have for research does affect the attention that they give their teaching, and, more generally, that institutions are not exploiting the potential synergies between staff research and scholarship and student learning. However, there is an obvious, and much more straightforward, solution, namely, to abandon research selectivity across most disciplines.This is something that is highly desirable in its own right in enabling us to make better use of the increasingly constrained resources being devoted to research.³ An alternative, if this is felt to be too radical, would be to include the effect on student learning in the impact section of the next REF. This would correct the extraordinary anomaly – at least to those unfamiliar with the quality of British higher education policymaking –  that the recent REF looked at every kind of impact except for the one – on student learning – that provides the strongest – some would say, the only – justification for conducting research in universities in the first place!

Impact on graduate employability

The Green Paper argues (at one and the same time) that there is a shortage of highly skilled job applicants in (especially) STEM subjects, and that too many graduates are in non-graduate jobs:

Higher education providers need to provide degrees with lasting value to their recipients. This will mean providers being open to involving employers and learned societies representing professions in curriculum design. It will also mean teaching students the transferrable [sic] work readiness skills that businesses need, including collaborative teamwork and the development of a positive work ethic, so that they can contribute more effectively to our efforts to boost the productivity of the UK economy. (page 11).  

The TEF should be a ‘good deal’ for employers and the taxpayer:

The aim is to improve the teaching that students receive, which in turn should improve their productivity and help them secure better jobs and careers. It should enable employers to make more informed choices about the graduates they recruit, providing better understanding of the range of skills and knowledge they bring from their course, and deliver graduates who are more work ready following an active engagement in their studies. With higher returns, more graduates will be able to pay back more of their loans, reducing the amount that needs to be subsidised by the taxpayer in the longer term. This is on top of the benefits to taxpayers from having a stronger economy powered by a higher skilled workforce. (page 21).  

However the same difficulties that affect information for students apply equally to information for employers. In any case, the implication of the statements in the Green Paper is that the responsibility for any failure to match the supply and quality of graduates to the needs of the economy lies with the universities. But how far should universities be attempting to take account of the needs of the economy, always assuming that anyone knows what these are? Is it really the job of the universities to produce ‘work ready’ graduates, whatever that means? And whose responsibility is it to ensure that graduates’ skills are fully utilised? Isn’t the real problem the fact that the deflationary, supply-side oriented macroeconomic policies followed by governments of all parties since the 1980s have provided too few incentives for companies to invest sufficiently in upgrading their employees’ skills, as well as improving their efficiency (see Brown, Unpublished, for the full argument).

Widening participation

The same attempt to place the blame for historic failures onto the universities – as a means of diverting attention away from the Government’s role? – can be seen in the Green Paper’s discussion of widening participation and social mobility.  There is indeed ample evidence of declining social mobility (for example, Bukodi et al., 2014). But it is clear from the evidence we have that whilst universities undoubtedly have a part to play in raising aspirations, the main obstacle to improving participation by under-represented groups is the relative scarcity of suitably qualified applicants; that the main cause of this is the very variable performance of the schools; that this reflects the segregation of British, and especially English, schools on socioeconomic lines; and that this in turn contributes to, and indeed reinforces, rising inequality (Brown, Submitted for review).

For all the rhetoric about being a One Nation Government, there is a whole series of current policies that will actually increase inequality:

• the planned further massive reductions in public expenditure will hit the public services on which the poorer and less advantaged members of society especially rely, the scale and quality of which have already been reduced in the last Parliament;
• the freezing of the top marginal rate of Income Tax will do nothing to reduce inequality whilst the freezing of the VAT rate means that the poorest 10 per cent of households will continue to pay a considerably higher share of their income in taxes than the wealthiest;
• the increase in the Inheritance Tax Threshold will increase still further the value of inherited wealth, as well as reducing the incentive to ‘trade down’ to help housing supply (cuts in the annual tax allowance for pension contributions for people with earnings of £150,000 or more could have been used to reduce the welfare cuts);
• the social security reforms announced, not least the cutbacks in tax credits, are bound to increase child poverty, as a recent Government assessment leaked to The Guardian (Butler and Malik, 2015) made clear;⁴
• the industrial relations reforms will further weaken the ability of the trades unions to act as a moderating force on inequality, including the lowering of wages that is almost certainly one of the main causes of our poor recent productivity record (there are no parallel constraints on the movement or withdrawal of capital by companies);
• the fragmentation of the school system will increase still further with additional academies and free schools, and there are no proposals to put the state and independent schools on the same footing as regards resourcing or esteem (Brown, submitted for review).

In Anarchical Fallacies: Being an Examination of the Declarations of Rights Issued During the French Revolution Jeremy Bentham (1843) coined the phrase ‘nonsense on stilts’ to describe the idea of natural rights. Surely, the Green Paper’s attempt to blame the universities for differential participation in British higher education can be described as ‘hypocrisy on stilts’?

Notes

1. Brazil had a common exam taken by all students for a while but eventually dropped it on grounds of effectiveness and value for money.
2. The first independent review of the NSS emphasised:

the need to take into account student profiles when making any comparisons using the … data, as ‘raw’ figures do not take into account the characteristics of students, their courses and the institutions in which they study may produce at best misleading and at worst invalid measures of teaching (Surridge, 2006: 132).  

The same point has appeared in subsequent evaluations but, needless to say, this ‘health warning’ does not appear with NSS outcomes.

3. It has long been clear that the costs and detriments of the RAE/REF outweigh any remaining benefits (see the summary in Chapter 8 of Everything for Sale?). Sector-wide selectivity should be confined to subjects or areas of perceived weakness and/or where it is desired to expand research and/or where there is a serious risk of expensive duplication of resources. Where there is no such need, the quality of staff research can be monitored through routine quality assurance, as it is anyway in preparation for sector-wide exercises.
4. According to the Guardian report, the memorandum sent to the Work and Pensions Secretary two weeks before the election stated:

Around 40,000 more…children might, in the absence of any behaviour change, find themselves in poverty as a result of reducing the cap to £23,000. If these families respond to the cap by making behaviour change, for example moving into work, they are likely to see themselves and their children move out of relative poverty.

The same report quotes the Department for Work and Pensions as saying:

As the document itself makes clear, these figures do not take into account a key impact of the cap: that it incentivises people to move into work and improve their lives.

References

BIS (2015) Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice.(Cm 9141). London: BIS.

Brown, R. (Ed.) (2011) Higher Education and the Market. New York and London: Routledge.

Brown, R. (Submitted for review) Education and inequality.

Brown, R. (Unpublished) Macroeconomic Policy

Brown, R. and Bekhradnia, B. (2013) The Future Regulation of Higher Education in England. Oxford: HEPI.

Bukodi, E., Goldthorpe, J.H., Waller, L. and Kuha, J. (2014) The mobility problem in Britain: new findings from the analysis of birth cohort data, British Journal of Sociology, 65, 3, 1-25.

Butler, P. and Malik, S. (2015) Thousands plunged into poverty by benefit cap, The Guardian, 30 May, 1-2.

Friedman, M. (1962) Capitalism and Freedom. Chicago: University Press.

Goddard, A. (2015) Financial risks ahead: Funding council chief raises concerns, Higher Education from Research Fortnight, 13^th November. https://www.researchprofessional.com/0/rr/he/agencies/hefce/2015/11/Financial-warning.html. (accessed 15th November 2015).

Hirsch, F. (1976) Social Limits to Growth Cambridge Ma. Harvard University Press.

King, R. (2011) The Risks of Risk-Based Regulation: The Regulatory Challenges of the Higher Education White Paper for England. Oxford: HEPI.

Surridge, P. (2006) The National Student Survey 2005: Findings – Main Report. Bristol: HEFCE.

Watson, D. (2008) Universities behaving badly? Higher Education Review, 40, 3, 3-14.

Weimer, D.L. and Vining, A.R. (1992) Policy Analysis: Concepts and Practice. Englewood Cliffs, New Jersey: Prentice-Hall.

Current policies are imposing business practices on education, and the consequences are blighting the profession, and will continue to inflict ever deeper blight on the people engaged in it – at all levels. The relinquishing of financial support by the state is not accompanied by diminution of authority: indeed the huge expansion of management follows from direct state interference in education as well as other essential elements of a thriving society.

Last September I wrote an article for the London Review of Books about my departure from the University of Essex, followed by another piece in March reflecting on the perversion of UK Higher Education. The responses I had to these articles came from people at every stage of the profession. I had feared that I was a nostalgic humanist, but if I am, the ideals of my generation have not died. Access to education to high standards fits very ill with business models – as the strong drift towards removing the cap on fees shows. The result of the market will be an ever-deepening divide between elite universities at one end and ‘sink’ institutions at the other.

I am going to focus on those who fulfil the prime purpose of the whole endeavour; that is those who pass on their knowledge and foster the spirit of inquiry and understanding in their students: the teachers.

First, the policies that are now being discussed, changing the rules regarding Further Education in particular, will need more and more teachers. Yet throughout the profession there is a shortage, and the toll taken on those who do teach in higher education is heavy and growing heavier – economically, psychologically, socially.

Student debt, already heavy after a first degree, is preventing individuals from carrying on, sometimes to an MA, and very frequently to a PhD. The numbers of graduate students from abroad conceals a yawning gap in educating the next generation here, and the one after that. I would like to see a graph of the rise in EU, US and graduates from Asia studying here: they are making it possible for our places of higher education to flourish, but their numbers – most welcome numbers – mask a significant and widening hole in the country’s intellectual ozone layer. Current anti-European feeling as well as rising xenophobia deepen this danger.

The intense promotion of mass lectures in digital classrooms belongs to the business model for education, and such methods might work for passing on basic data, but won’t work to develop independent thinking or test skills of reading and interpretation. Besides, personal contact hours are already a live issue with students.

Secondly, the current form of the REF – and the promised TEF – exacerbates the stress on young academics – who are indented into the precariat – the new underclass. One effect among many has been the flight of many in the profession from teaching into administration: this lies at the root of the current disastrous disequilibrium in the universities. As an administrator, you get to give orders and you get paid twice, three, four times as much. The invasiveness of government and its shifting and endless demands for measurements – many of them, including the REF itself, entirely ill-suited to represent the objects of its inquiry – places power in the hands of people many of whom have realized they are never going to be REFable themselves and don’t want to teach. It also clears a space for bullies, as surely as the playground of a failing school. Several of the letters and messages I received recalled appalling scenes of this kind, which will be directly familiar to many of us, unfortunately: the historic autonomy of departments and their elected heads has come to an end.

One highly distinguished correspondent – who has to remain anonymous – wrote to me to say the mistake academics made was to allow our salaries to fall so far behind business and civil service standards. This happened because the teachers were doing it for love…we thought it was a vocation, not a business.

Thirdly, a particular business model has been chosen. It does not follow the example of Germany, one of the most successful capital industrial powerhouses in the world, and where, by the way, fees have been abolished altogether. Does Germany, which as we know is hardly a friend to wild utopianism, see something we don’t see about the importance of higher education for all? By contrast, the economic model dominating many if not most universities in England is also intensely hierarchical, and ideologically committed to competition as the spur to productivity. Colleagues in the humanities are under ukases, sometimes written into contracts, to win grants. Several individuals in the same dept. will be ordered to pursue the same funding at the same time … The harvest is meagre, the winnowing ruthless. But quite apart from the losses we are suffering as a result – the gifted people who are driven down and out, the atmosphere of the workplace is poisoned by the rivalry and the success of the few.

I am not saying everything was hunky-dory before between colleagues. Of course not, but the profession is not a happy one today – except for the very few stars, who mostly now no longer teach at all, as these elusive and desirable grants are intended to buy out that time.

So what can be done? A few things are critical and must be looked at immediately:

• Graduate support – this applies to the FE sector as well as to HE if those students get new degrees and would flourish if they continued to study

• The ratio of managers to teachers

• The differential between salaries across an institution

• Regarding the Humanities, the REF should be set aside altogether, in my opinion, but, as that is probably not going to happen, then any department should be represented in its totality, while dates of appointment in relation to publication need to be taken into account, and there should be no advantage in opportune transfers and sales of star players.

• Finally, academic contracts need to be secure and our pay higher in order to ensure status and influence – unfortunately.

There is a huge amount of good will as well as talent and knowledge out there among the actual and potential teaching professionals, and it needs to be cared for, fostered, recognized as vital, valuable, and essential, the main catalyst of future national well-being.

Note: This blogpost is based on a talk given at a meeting on Perspectives on Education at the British Academy, on Friday 24^th July 2015.

Editorial Note: In our previous blogpost, we noted that while there was agreement that REF2014 was problematic, there was less agreement about alternatives. To make progress, we need more debate. We hope that this piece by Derek Sayer will stimulate this, and we welcome comments. Please note that comments are moderated and will not be published immediately.

1.

The rankings produced by Times Higher Education and others on the basis of the UK’s Research Assessment Exercises (RAEs) have always been contentious, but accusations of universities’ gaming submissions and spinning results have been more widespread in REF2014 than any earlier RAE. Laurie Taylor’s jibe in The Poppletonian that ‘a grand total of 32 vice-chancellors have reportedly boasted in internal emails that their university has become a top 10 UK university based on the recent results of the REF’[i] rings true in a world in which Cardiff University can truthfully[ii] claim that it ‘has leapt to 5th in the Research Excellence Framework (REF) based on the quality of our research, a meteoric rise’ from 22nd in RAE2008. Cardiff ranks 5th among universities in the REF2014 ‘Table of Excellence,’ which is based on the GPA of the scores assigned by the REF’s ‘expert panels’ to the three elements in each university’s submission (outputs 65%, impact 20%, environment 15%)—just behind Imperial, LSE, Oxford and Cambridge. Whether this ‘confirms [Cardiff’s] place as a world-leading university,’ as its website claims, is more questionable.[iii]  These figures are a minefield.

Although HEFCE encouraged universities to be ‘inclusive’ in entering their staff in REF2014, they were not obliged to return all eligible staff and there were good reasons for those with aspirations to climb the league tables to be more ‘strategic’ in staff selection than in previous RAEs. Prominent among these were (1) HEFCE’s defunding of 2* outputs from 2011, which meant outputs scoring below 3* would now negatively affect a university’s rank order without any compensating gain in QR income, and (2) HEFCE’s pegging the number of impact case studies required to the number of staff members entered per unit of assessment, which created a perverse incentive to exclude research-active staff if this would avoid having to submit a weak impact case study.[iv] Though the wholesale exclusions feared by some did not materialize across the sector, it is clear that some institutions were far more selective in REF2014 than in RAE2008.

Unfortunately data that would have permitted direct comparisons with numbers of staff entered by individual universities in RAE2008 were never published, but Higher Education Statistical Authority (HESA) figures for FTE staff eligible to be submitted allow broad comparisons across universities in REF2014. It is evident from these that selectivity, rather than an improvement in research quality per se, played a large part in Cardiff’s ‘meteoric rise’ in the rankings. The same may be true for some other schools that significantly improved their positions, among them Kings (up to 7th in 2014 from 22= in 2008), Bath (14= from 20=), Swansea (22= from 56=), Cranfield (31= from 49), Heriot-Watt (33 from 45), and Aston (35= from 52=).  All of these universities except Kings entered fewer than 75% of their eligible staff members, and Kings has the lowest percentage (80%) of any university in the REF top 10 other than Cardiff itself.

Cardiff achieved its improbable rank of 5th on the basis of a submission that included only 62% of eligible staff. This is the second-lowest percentage of any of the 28 British universities that are listed in the top 200 in the 2014-15 Times Higher Education World University Rankings (of these schools only Aberdeen entered fewer staff, submitting 52%). No other university in this cohort submitted less than 70% of eligible staff, and half (14 universities) submitted over 80%. Among the top schools, Cambridge entered 95% of eligible staff, Imperial 92%, UCL 91% and Oxford 87%.

Many have suggested that ‘research power’ (which is calculated by multiplying the institution’s overall rounded GPA by the total number of full-time equivalent staff it submitted to the REF) gives a fairer indication of a university’s place in the national research hierarchy than GPA rankings alone. By this measure, Cardiff falls to a more credible but still respectable 18th. But when measured by ‘research intensity’ (that is, GPA multiplied by the percentage of eligible staff entered), its rank plummets from 5th to 50th. To say this provides a more accurate indication of its true standing might be overstating the case, but it certainly underlines why Cardiff does not belong among ‘world-leading’ universities. Cardiff doubtless produces some excellent research, but its overall (and per capita) performance does not remotely justify comparisons with Oxford, Cambridge, or Imperial—let alone Caltech, Harvard, Stanford, Princeton, MIT, UC-Berkeley and Yale (the other universities in the THE World University Rankings top 10). In this sense the GPA Table of Excellence can be profoundly misleading.

‘To their critics,’ writes Paul Jump in Times Higher Education, ‘such institutions are in essence cheating because in reality their quality score reflects the work produced by only a small proportion of their staff.’[v] I am not sure the accusation of cheating is warranted, because nobody is doing anything here that is outside HEFCE’s rules. The problem is rather that the current REF system rewards—and thereby encourages—bad behavior, while doing nothing to penalize the most egregious offenders like Cardiff.

The VCs at Bristol (11= in the REF2014 GPA table) and Southampton (18=, down from 14= in 2008) might be forgiven for ruefully reflecting that they, too, might now be boasting that they are ‘a top ten research university’ had they not chosen to submit 91% and 90% of their eligible faculty respectively—a submission rate that on any reasonable criteria (as distinct from HEFCE’s rules) might itself be seen as an indicator of research excellence.  Measured by research intensity Bristol comes in at 5= (jointly with Oxford) and Southampton at 8= (jointly with Queen’s University Belfast, which submitted 95% of its staff and is ranked 42= on GPA).  Meantime the VCs at St Andrews (down from 14= to 21=, 82% of eligible staff submitted), Essex (11th to 35=, 82% submitted), Loughborough (28= to 49=, 88% submitted) and Kent (31= to 49=, 85% submitted) may by now have concluded that—assuming they hold onto their jobs—they will have no alternative other than to be much more ruthless in culling staff for any future REF.

2.

The latest Times Higher Education World University Rankings puts Cardiff just outside the top 200, in the 201-225 group—which places it 29= among UK universities, along with Dundee, Newcastle, and Reading. Taking GPA, research power and research intensity into account—as we surely should, in recognition that not only the quality of research outputs but the number and proportion of academic staff who are producing them are also necessary elements in evaluating any university’s overall contribution to the UK’s research landscape—such a ranking feels intuitively to be just about right.

I have shown elsewhere[vi] that there was, in fact, a striking degree of overall agreement between the RAE2008 rankings and the Times Higher Education World University Rankings. Repeating the comparison for UK universities ranked in the top 200 in the THE World University Rankings for 2014-15 and the REF2014 GPA-based ‘Table of Excellence’ yields similar findings. These data are summarized in Table 1.

Table 1:  REF2014 performance of universities ranked in the top 200 in Times Higher Education World University Rankings 2014-15   

Seven UK universities make the top 50 in the 2014-15 THE World University Rankings: Oxford, Cambridge, Imperial, UCL, LSE, Edinburgh, and Kings.  Six of these are also in the REF2014 top 10, while the other (Edinburgh) is only just outside it at 11=.  Four of the leading five institutions are same in both rankings (the exception being UCL, which is 8= in REF 2014), though not in the same rank order. Of 11 UK universities in THE top 100, only one (Glasgow, at 24th) is outside the REF top 20.  Of 22 UK universities in THE top 150, only two are outside REF top 30 (Birmingham, 31 in REF, and Sussex, 40 in REF).  Of 28 UK universities in THE top 200, only two are outside the REF top 40 (Aberdeen at 46= and Leicester at 53).

Conversely, only two universities in the REF2014 top 20, Cardiff at 6 and Bath at 14=, do not make it into the THE top 200 (their respective ranks are 201-225 and 301-350). Other universities that are ranked in the top 40 in REF2014 but remain outside the THE top 200 are Newcastle (26=), Swansea (26=), Cranfield (31), Herriot-Watt (33), Essex (35=), Aston (35=), Strathclyde (37), Dundee (38=) and Reading (38=).

Table 2 provides data on the performance of selected UK universities that submitted to REF2014 but are currently ranked outside the THE world top 200.

Table 2.  REF2014 performance of selected UK universities outside top 200 in Times Higher Education World University Rankings 2014-15

Dundee, Newcastle and Reading only just miss the THE cut (they are all in the 201-225 bracket). While all three outscored Aberdeen and Leicester, who are above them in the THE rankings (in Leicester’s case, at 199, very marginally so) in the REF, only Newcastle does substantially worse in the THE rankings than in the REF.  It is ranked 26= in the REF with Nottingham and Royal Holloway, ahead of Leicester (53), Aberdeen (46), Sussex (40), Liverpool (33), Birmingham (31) and Exeter (30)—all of which are in the top 200 in the THE World Rankings. While there was a yawning gulf between Essex’s RAE2008 ranking of 11th and its THE ranking in the 301-350 group, the latter does seem to have presaged its precipitous REF2014 fall from grace to 35=.  Conversely, the THE inclusion of Plymouth in the 276-300 group of universities places it considerably higher than its RAE rank of 66= would lead us to expect. This is not the case with most of the UK universities listed in the lower half of the THE top 400. Birkbeck, Bangor, Aberystwyth and Portsmouth also all found themselves outside the top 40 in REF 2014.

The greatest discrepancies between REF2014 and THE World Rankings come with Cardiff (6 in REF, 201-225 in REF), Bath (14= in REF, 301-350 in THE), Swansea (26= in REF, not among THE top 400), Aston (35= in REF, 350-400 in THE), Cranfield, Heriot-Watt and Strathclyde (31=, 33 and 37 respectively in REF, yet not among THE top 400). On the face of it, these cases flatly contradict any claim that THE (or other similar) rankings are remotely accurate predictors of REF performance. I would argue, on the contrary, that these are the exceptions that prove the rule. With the exception of Strathclyde (18 in research intensity with 84% of eligible staff submitted and the worst-performing member of this group in REF GPA), all these schools were prominent among universities who inflated their GPA by submitting smaller percentages of their eligible staff in REF2014.  Were we to adjust raw GPA figures by research intensity, we would get a much closer match, as Table 3 shows.

Table 3.  Comparison of selected universities performance in THE World University Rankings 2014-15 and REF2014 by GPA and research intensity.

The most important general conclusion to emerge from this discussion is that despite some outliers there is a remarkable degree of agreement between the top 40 in REF2014 and top 200 in the THE 2014-15 World University Rankings, and the correlation increases the higher we go in the tables.  Where there are major discrepancies, these are usually explained by selective staff submission policies.

One other correlation is worth noting at this point.  All 11 of the British universities in the THE top 100 are members of the Russell Group, as are 10 of the 17 British universities ranked between 100-200. The other six universities in this latter cohort (St Andrews, Sussex, Royal Holloway, Lancaster, UEA, Leicester) were all members of the now-defunct 1994 Group. Only one British university in the THE top 200 (Aberdeen) belonged to neither the Russell Group nor the 1994 Group. Conversely, only two Russell Group universities, Newcastle and Queen’s University Belfast, did not make the top 200 in the THE rankings.[vii]  In 2013-14 Russell Group and former 1994 Group universities between them received almost 85% of QR funding.  Here, too, an enormous amount of money, time, and acrimony seems to have been expended on a laborious REF exercise that merely confirms what THE rankings have already shown.

3.

The most interesting thing about this comparative exercise is that the Times Higher Education World University Rankings not only make no use of RAE/REF data, but rely on quantitative methodologies that have repeatedly been rejected by the British academic establishment in favor of the ‘expert peer review’ that is supposedly offered by REF panels. THE gives 30% of the overall score for the learning environment, 7.5% for international outlook, and 2.5% for industry income. The remaining 60% is based entirely on research-related measures, of which ‘the single most influential of the 13 indicators,’ counting for 30% of the overall THE score, is ‘the number of times a university’s published work is cited by scholars globally’ as measured by the Web of Science. The rest of the research score is derived from research income (6%), ‘research output scaled against staff numbers’ (6%, also established through the Web of Science), and ‘a university’s reputation for research excellence among its peers, based on the 10,000-plus responses to our annual academic reputation survey’ (18%).

The comparisons undertaken here strongly suggest that at such metrics-based measures have proved highly reliable predictors of performance in REF2014—just as they did in previous RAEs. To be sure, there are differences in the details of the order of ranking of institutions between the THE and REF, but in such cases can we be confident that it is the REF panels’ highly subjective judgments of quality that are the more accurate? To suggest there is no margin for error in tables where the difference in GPA between 11th (Edinburgh, 3.18) and 30th (Exeter, 3.08) is a mere 0.1 points would be ridiculous.  I have elsewhere suggested that there are in fact many reasons why such confidence would be totally misplaced, including lack of specialist expertise among panel members and lack of time for reading outputs in the depth required.[viii]  But my main point is this.

If metrics-based measures can produce similar results to those arrived at through the REF’s infinitely more costly, laborious and time-consuming process of ‘expert review’ of individual outputs, there is a compelling reason to go with the metrics; not because it is necessarily a valid measure of anything but because it as reliable as the alternative (whose validity is no less dubious for different reasons) and a good deal more cost-efficient. The benefits for collegiality and staff morale of universities not having to decide whom to enter or exclude from the REF might be seen as an additional reason for favoring metrics. I am sure that if HEFCE put their minds to it they could come up with a more sophisticated basket of metrics than Times Higher Education, which would be capable of meeting many of the standard objections to quantification.  I hope James Wilsdon’s committee might come up with some useful suggestions for ways forward.

[i] Laurie Taylor, ‘We have bragging rights!’ in The Poppletonian, Times Higher Education, 8 January 2015.

[ii] Well, not quite. Cardiff is actually ranked 6th in the REF2014 ‘Table of Excellence,’ which is constructed by Times Higher Education on the basis of the grade point average (GPA) of the marks awarded by REF panels, but the #1 spot is held not by a university but the Institute of Cancer Research (which submitted only two UoAs).  This table and others drawn upon here for ‘research power’ and ‘research intensity’ can be downloaded here.

[iii] ‘REF 2014,’ Cardiff University website.

[iv] Paul Jump, ‘Careers at risk after case studies ‘game playing’, REF study suggests.’  Times Higher Education, 22 January 2015, at

[v] Paul Jump, ‘REF 2014 rerun: who are the ‘game players’?’  Times Higher Education, 1 January 2015.

[vi] See Derek Sayer, Rank Hypocrisies: The Insult of the REF.  London: Sage, 2014.

[vii] I have discussed Newcastle already.  Queen’s came in just outside the REF top 40 (42=) but with an excellent intensity rating (8=, 95% of eligible staff submitted).

[viii] See, apart from Rank Hypocrisies, ‘One scholar’s crusade against the REF,’ Times Higher Education, 11 December, 34-6; ‘Time to abandon the gold standard?  Peer Review for the REF Falls Far Short of Internationally Acceptable Standards,’ LSE Impact of Social Sciences blog, 19 November (reprinted as ‘Problems with peer review for the REF,’ CDBU blog, 21 November).

*This is a revised version of an article first posted on Sayer’s blog coastsofbohemia on 27 January 2015.

Note: This article gives the views of the author, and not the position of the Council for Defence of British Universities. This work is licensed under a Creative Commons Attribution 3.0 Unported License unless otherwise stated.

Results from the research excellence framework (REF) were publicly announced on 18th December, followed by a spate of triumphalist messages from University PR departments. Deeper analysis followed, both in the pages of the Times Higher Education, and in the media and on blogs.

CDBU has from the outset expressed concern about the REF, much of it consistent with the criticism that has been expressed elsewhere. In particular, we note:

Inefficiency: As Derek Sayer has noted, the REF has absorbed a great deal of time and money that might have been spent better elsewhere. The precise cost has yet to be reported, but it is likely to be greater than the £60m official figure, and that is not taking into account the cost in terms of the time of academic staff. Universities have taken on new staff to do the laborious work of compiling data and writing impact statements, but this has diverted funds from front-line academia and increased administrative bloat.

Questionable validity: Derek Sayer has cogently argued the case that the peer review element of REF is open to bias from subjective, idiosyncratic and inexpert opinions. It is also unaccountable in the sense that ratings made of individual outputs are destroyed. One can see why this is done: otherwise HEFCE could be inundated with requests for information and appeals. But if the raw data is not available, then this does not inspire confidence in the process, especially when there are widespread accusations of games-playing and grade inflation.

Concentration of funding in a few institutions: We are told that the goal is to award quality-related funding, but as currently implemented, this leads inevitably to a process whereby the rich get richer and the poor get poorer, with the bulk of funds concentrated in a few institutions. We suspect that the intention of including ‘impact’ in the REF was to reduce the disparity between the Golden Triangle (Oxford, Cambridge and London) and other institutions which might be doing excellent applied work, but if anything the opposite has happened. We do not yet know what the funding formula will be, but if it is, as widely predicted, heavily biased in favour of 4* research, we could move to a situation where only the large institutions will survive to be research-active. There has been no discussion of whether such an outcome is desirable.

Shifting the balance of funding across disciplines: A recent article in the Times Higher Education noted another issue: the tendency for those in the Sciences to obtain higher scores on the REF than those in the Humanities. Quotes from HEFCE officials in the article offered no reassurance to those who were concerned this could mean a cut in funding for humanities. Such a move, if accompanied by changes to student funding to advantage those in STEM subjects, could dramatically reduce the strength of Humanities in the UK.

Unaccountable flexibility in the funding formula: There are many different ways of achieving ratings: For instance, whether or not the ratings include ‘intensity’ (number of returnable staff who were entered), can dramatically alter rank orderings. Or we could look at a suggestion by Graeme Wise that a ‘bang for your buck’ metric that assessed outputs in relation to grant income would be most appropriate. Even more radical is a suggestion by Dermot Lynott, that we should be giving the most rewards to those whose outputs were impressive in relation to their scores on environment. Needless to say, a very different profile of winners and losers emerged from such an analysis.  It will ultimately be a political decision as to how to translate the REF scores into funding. We have to ask whether it worth going through this entire long-winded exercise if, by simply changing the funding formula, one can make a dramatic difference to an institution’s funding to achieve a politically expedient outcome.

Damage inflicted on careers and morale: The criteria for entering staff for the REF could appear quite cavalier; for instance, the requirement for a numerical ratio between number of staff entered and number of case studies meant that some departments with few case studies were unable to enter all plausible staff. Derek Sayer has described instances of decisions to enter staff being made on what appeared to be flimsy evidence based on ad hoc internal evaluations. Yet being identified as ‘non-REFable’ is not only damaging to morale, but could have real impacts on prospects for promotion and job security.

Focus on competition rather than collaboration:  The REF exercise creates rank orderings, and everyone is desperately trying to nudge ahead of the others. In fact, there are so many different ways of doing the ranking, that almost everyone can be satisfied that they are ‘among the top’ on some index or other. Those who crowed loudest about their success tried to temper this by arguing that they were celebrating a broader ‘British’ success, but this seems perverse. Why should concentrating ever more of the excellent research in an ever small number of institutions be regarded as a national success story? It is, of course, widely believed that competition is a force for good, stimulating people to do better than they otherwise might. However, many in academia take the view that they don’t need to be incentivised by competition to work hard: they are in the job for the love of it, and would like their efforts to be appreciated for what they are, not because they help push the institution up a league table. Competition can also damage relationships between different departments within a University, if there are disparities in REF performance that lead to bickering about who is more valuable.

Perverse incentives that damage research: These may play out differently in different subject areas, but overall, many academics feel that they are not able to do the research they want in the way they would like. In science, there are intense pressures to publish in high-impact journals and bring in grant income. Some institutions are notorious for threatening redundancy to scientific staff who do not meet some agreed quota of research income, creating incentives to do ever more expensive research (see for instance cases at Kings College London, Imperial College, and Warwick University Medical School).  In humanities, the pressure to produce a steady stream of research articles and monographs has led to the sense of an enforced move to over-specialisation, with academics increasingly incapable of explaining or demonstrating the broader significance of their work. Younger generations of academics find that their direction of research is being wholly driven by ‘REF-ability’, and that journal publications automatically trump those in volumes of collected essays, even when the latter may be more important for the field.

Perverse incentives on hiring practices: This is another consequence of the intensely competitive culture that is induced by the REF. We have, particularly around the time of the REF, a market in research ‘super-stars’, who can attract impressive transfer fees. People from other institutions who are employed at only 20 per cent part-time suddenly appear on the books, boosting the institution’s return on funding and outputs.

Devaluation of non-research activity: Academics whose positions require them to teach and do research have felt pressured to focus principally on research, and teaching has consequently been devalued. It is sometimes suggested that the Impact agenda of the REF also encourages academics to spend time on public engagement, but in fact it has the opposite effect. Public engagement does not count as ‘impact’ for REF  purposes: to demonstrate REF impact, one must provide concrete documentation of how a specific piece of research has influenced non-academic users, such as policy-makers, health professionals, museums, etc.

How have we come to this?

Given that many of these points were made in the run-up to the announcement of REF results, we have to ask how it is that we find ourselves trapped in such an undesirable system. It is noteworthy that the REF is popular with many vice-chancellors and administrative teams. It makes it easier to manage staff, with objective criteria for hiring and firing, and provides league tables to measure progress by. For those already attached to the vision of a university as big business, it seems the natural next step to have objective rules for defining winners and losers so that one can directly measure an individual’s likely ability to bring money into the university without having to make difficult judgements about the intrinsic quality of their work.

It is a moot point whether the collapse of the circle of winners to Oxbridge and London was the result of deliberate planning, or an unintended consequence of how the system operates. Be this as it may, one concern is that this could provide further pressure for the British university system to reconfigure itself so that it can compete with the American private elite. There would be increasing reluctance to use general taxation to concentrate even more educational resources within close proximity to London, and instead we could see a shift to private funding, increasing the extent to which access to the best institutions is distributed according to wealth rather than ability.  In a few short years we could see the destruction of a genuinely national system of higher education, publicly funded because it is designed to serve everyone, to a privately funded system that is world-class for the few who can afford to access it, but a disaster for the country as a whole. As in the US, educational opportunity would be concentrated overwhelmingly in places where it can be accessed primarily by the wealthy, privileged and well-connected.

At the time of the announcement of REF results, there was a sense that anyone who criticised the celebrations was either a bad loser, or – if they came from an institution who did well – a traitor for not celebrating British success. At CDBU we are proud of UK Universities and their research reputation, but our loyalty is to our discipline, our profession, our vocation and our sense of their place in the wider scheme of things: we fear that the assessment process embodied in REF will in the longer term damage these.

Where next?

It is, of course, all very well to criticise and paint visions of a dystopian future. If we wish to replace the current system, we must look at alternative ways forward. At a debate about the REF, organised by Sage Publishers on 8^th December, David Willetts MP, who until July 2014 was Minister of State for Universities and Science, took the view that some of those who disapproved of REF were just dinosaurs who wanted to go back to the 1970s, when funds were allocated to institutions by a group of the great and the good making judgements over dinner at the Athenaeum. We would dispute that this is the only alternative to the current system. But Willetts was right on another count: he pointed out that the current REF system was not imposed by government, nor by HEFCE. Indeed, they had been actively pursuing the idea of using a simpler metrics-based system for the REF, but it was resoundingly rejected by the academic community. Government, according to Willetts, would listen to any reasonable proposal for a new system. Clearly, it is now up to the academics themselves to propose a viable alternative.

At the same meeting, David Sweeney, Director of HEFCE responsible for Education and Knowledge Exchange, implied that critics of REF just wanted to be handed public money without any accountability. He emphasised that the government and taxpayer put money into university research and had a right to know about the outcomes from the investment they had made. We totally agree. But we take issue with those who, like Mark Leach, Director and Editor-in-Chief of Wonke, think that the REF, for all its limitations, provides a good solution. Our position is that, for all the reasons given above, the REF is a seriously flawed system for deciding on disbursement of research funds, which in the long run will do the UK University system more harm than good.

What alternatives are there?

1. One possibility that has been discussed is to remove the QR component of funding altogether, and give all funds to the research councils. The problem with this solution is that it would mean the research councils would have to grow in size enormously, the load on reviewers, already seen as unsustainable by many would increase yet further, and pressures on academics to bring in research grants would be even more intense. It has also been argued that it would further increase disparities between the Golden Triangle (Oxbridge and London) and the rest, and would disfavour non-STEM disciplines.

2. HEFCE is has been looking at various publication-based metrics that might substitute for the REF, and plans to do some empirical studies comparing metric-based evaluation with REF results. However, metrics have been vigorously opposed by many in the academic community, especially in the humanities, where there is much worse agreement with expert opinion than in the sciences. We do at least now have hard data from the REF that can be considered when evaluating how metrics would perform, but there’s a real risk that introduction of any metric will further distort incentives, so that the measure becomes the goal.

3. At the Sage meeting, Derek Sayer put forward another interesting alternative, which was that funding should be based just on the ‘research environment’ component of the REF, which focuses more on inputs than outputs.

4. An even simpler option that would retain the dual support system but remove quality-related funding would involve disbursing funds purely on the basis of the number of active researchers in a department. This could be criticised for leading to a ‘prairie farming’ model whereby departments would band together to create enormous conglomerates that would benefit from economies of scale. One could, however, put a limit on the size of unit entered. At the Sage meeting, David Sweeney expressed himself as strongly opposed to this solution, even though in the last round it gave a funding result that was highly correlated with actual funding outcomes from RAE, in both science and humanities. He is right in noting that, despite the high correlation, there would nevertheless undoubtedly be winners and losers who would suffer substantial gains and losses in real terms, relative to the RAE result, but the question is whether this would involve unfairness. It’s hard to say, given that we have no gold standard. You could of course further argue that you have to have some measure of quality to incentivise people to do better, as well as a need to guard against freeloaders, like Laurie Taylor’s Dr Piercemuller. Yet, as we have noted, for most academics, exhortations from managers to ‘do better’ don’t achieve much and may indeed be counterproductive. We need to be accountable for the public money spent on university research, but subjecting every apple in the barrel to an exhaustive x-ray examination may not be the best way to identify the rotten ones.

We do not have a single solution, but we think that academics must take control of this process and not leave it in the hands of HEFCE and the government. This article about Dutch Universities Netherlands has strong parallels with the UK situation: the author argues academics have been too passive in accepting an emphasis on competition, and the use of evaluation systems as means of control. There is unlikely to be an ideal solution and we may have to live with the ‘least bad’ option. But let us consider all options in terms of how far they are likely to exacerbate or resolve the problems outlined above, or we may find ourselves saddled with something even worse than REF2014.

We hope this article opens up the discussion on this topic. Please do add your comments. We are moderating comments to exclude spam, but non-anonymous, on-topic comments will be published unless they contravene the usual rules.

Finally, if you agree with the broad concerns expressed here, please do consider joining the CDBU to help us campaign more effectively for change.

At the behest of universities minister David Willetts, HEFCE established an Independent review of the Role of Metrics in Research Assessment in April 2014 chaired by James Wilsden. This followed consultations in 2008-9 that played a decisive role in persuading the government to back down on previous plans to replace the RAE with a metrics-based system of research assessment. Wilsden’s call for evidence, which was open from 1 May to 30 June 2014, received 153 responses ‘reflecting a high level of interest and engagement from across the sector’ (Letter to Rt. Hon. Greg Clark MP). Sixty-seven of these were from HEIs, 27 from learned societies and three from mission groups. As in 2008-9, the British academic establishment (including the Russell Group, RCUK, the Royal Society, the British Academy, and the Wellcome Trust) made its voice heard. Predictably, ’57 per cent of the responses expressed overall scepticism about the further introduction of metrics into research assessment,’ while ‘a common theme that emerged was that peer review should be retained as the primary mechanism for evaluating research quality. Both sceptical and supportive responses argued that metrics must not be seen as a substitute for peer review … which should continue to be the “gold standard” for research assessment’ (Wilsden review, Summary of responses to submitted to the call for evidence).

The stock arguments against the use of metrics in research assessment were widely reiterated: journal impact factors cannot be a proxy for quality because ‘high-quality’ journals may still publish poor-quality articles; using citations as a metric ignores negative citation and self-citation; in some humanities and social science disciplines it is more common to produce books than articles, which will significantly reduce their citation counts, and so on. Much of this criticism, I would argue, is a red herring. Most of these points could easily be addressed by anybody who seriously wished to consider how bibliometrics might sensibly inform a research assessment exercise rather than kill any such suggestion at birth (don’t use JIFs, exclude self-citations, use indices like Publish or Perish that include monographs as well as articles and control for disciplinary variations). What is remarkable, however, is that while these faults are often presented as sufficient reason to reject the use of metrics in research assessment out of hand, the virtues of ‘peer review’ are simply assumed by most contributors to this discussion rather than scrutinized or evidenced. This matters because whatever the merits of peer review in the abstract—and there is room for debate on what is by its very nature a subjective process—the evaluation procedures used in REF 2014 (and previous RAEs) not only fail to meet HEFCE’s own claims to provide ‘expert review of the outputs’ but fall far short of internationally accepted norms of peer review.

In sharp contrast with the evaluative procedures used in a range of other academic contexts in the UK and internationally, the REF relies entirely on in-house assessment by panels of (so-called) experts. On some panels, like history, just one assessor may read each output, something unheard of in peer review for journal and book submissions, research grant competitions, and tenure and promotion proceedings. Additionally, almost all REF panelists are drawn from British universities alone. David Eastwood admitted back in 2007 that ‘international benchmarking of quality’ was ‘one thing that the RAE has not been able to do’—which is cute, considering that REF panels award their stars on the basis of whether outputs are ‘world leading’, ‘internationally excellent,’ or merely ‘recognized internationally.’ Eastwood then still hoped to solve the problem with ‘bibliometrics, used with sensitivity and sophistication’ (‘Goodbye to the RAE … and hello to the REF’, Times Higher Education, 30 November 2007). HEFCE’s prohibitions on using journal impact factors, rankings, or the perceived standing of publishers (and humanities and most social science panels’ refusal to use any bibliometric data, including citations) reinforce the total dependency of REF evaluations on these panelists’ subjective judgments. Meantime the abandonment of RAE 2008’s use of external advisors where a panel felt it lacked specialist expertise and an overall cut in the number of panels from 67 in RAE 2008 to 36 in REF 2014 further reduced the pool of expertise available to panels, which were now also only ‘exceptionally’ allowed to cross-refer outputs to other panels. If we could be confident that REF panels nevertheless ‘provide sufficient breadth and depth of expertise to undertake the assessment across the subpanel’s remit (including as appropriate expertise in interdisciplinary research and expertise in the wider use or benefits of research)’ (HEFCE, REF 2014: Units of Assessment and Recruitment of Expert Panels, 2010, para 55) this might not be a problem. But we cannot. Nobody on the REF history panel, for example, has specialist knowledge of China, Japan, the Middle East, Latin America, or many countries in Europe (including once-great powers like Austria-Hungary, Spain and Turkey), though work on all these areas has likely been submitted to the REF. Whatever their general eminence in the historical profession, these ‘experts’ do not know the relevant languages, archives, or literatures. How, then, can they possibly judge the ‘originality’ of an output or its ‘significance’ in any of these fields? And on what conceivable basis can they be entrusted to determine whether it is ‘internationally excellent’ or merely ‘internationally recognized’—the critical borderline between 3* research that will attract QR funding and 2* research that will not?

REF panelists are unlikely to have the time to do a proper assessment anyway. In all, around 1000 evaluators will have graded all 191,232 outputs for REF 2014 in under a year—the same number in total as the US National Endowment for the Humanities uses to evaluate 5700 applications for its 40 grant programs! Peter Coles calculates that each member of the physics panel must read 640 research papers, i.e., about two a day. ‘It is … blindingly obvious,’ he concludes, ‘that whatever the panel does do will not be a thorough peer review of each paper, equivalent to refereeing it for publication in a journal’ (‘The apparatus of research assessment’, LSE Impact Blogs, 14 May 2014). One RAE 2008 panelist told Times Higher Education that it would require ‘two years’ full-time work, while doing nothing else’ to read properly the 1200 journal articles he had been allocated (‘Burning questions for the RAE panels’, 24 April 2008). Another admitted: ‘You read them sufficiently to form a judgment, to get a feeling … you don’t have to read to the last full stop’ (‘Assessors face “drowning” as they endeavour to read 2,363 submissions’, 17 April 2008).

For major academic journals the process of review is often double blind. Though university presses and other academic book publishers divulge authors’ identities to reviewers, they will often also first consult with the author on appropriate reviewers. Reviewers are required to provide substantial comments in either case. The REF, by contrast, makes no attempt to protect authors’ anonymity—something we might think especially important when judgments may lie in the hands of a single assessor. And far from providing comments justifying their grades, RAE 2008 subpanels shredded all documents showing how they reached their conclusions and ordered members ordered to destroy personal notes in order to avoid having to reveal them under Freedom of Information Act requests. It is difficult to think of a procedure that would make it easier for evaluators to further ideological agendas or settle personal scores, should they be so inclined.

Metrics may have problems. But a process that willfully ignores whether an output has gone through any peer review before publication, where it has been published, and how often it has been cited in favor of the subjective opinions of evaluators who may have no specialist expertise in the field and then systematically erases all records does not strike me as a very defensible alternative. It also gives extraordinary gatekeeping power to the individuals who sit on REF panels. This is worrying because the mechanisms through which panelists are recruited are tailor-made for the sponsored replication of disciplinary elites. All applicants for panel chairs have to be endorsed by learned societies, chairs in turn ‘advise’ on the appointment of panel members, and at least one third of panelists have to have served in a previous RAE. Apart from being disproportionately older, white, and male compared with the UK academic profession in general, these may not always be the scholars best placed to identify cutting-edge research, especially where such research crosses or challenges disciplinary boundaries. NEH, we might note, prohibits its evaluators from serving in successive competitions to reduce this risk.

Were Wilsden’s committee to assess what might be achieved employing appropriately sophisticated metrics against what is actually done in the REF rather than comparing a crude caricature of metrics with an idealized chimera of peer review, I think it would have to take a different view of the merits of the two systems than that put forward in most responses to the call for evidence. For its REF process to be comparable to what is understood elsewhere as peer review, HEFCE would have to use subject matter-specific experts from an international pool, commission a minimum of two reviews of each output, and not overload reviewers with too many outputs for them to read them properly in the timeframe available. This would be even costlier in public money and academics’ time than the present REF. To replace the REF with metrics, on the other hand, would yield a process that is (in Dorothy Bishop’s words) ‘transparent and objective, it would not require departments to decide who they do and don’t enter for the assessment, and most importantly, it wins hands down on cost-effectiveness’ (‘An alternative to REF2014?’ Bishopblog, 26 January 2013).

Metrics have in fact proven to be highly reliable predictors of RAE performance, irrespective of whether or not they provide valid measures of research quality. It is not without irony that there is considerable overlap between RAE scores and major commercial university rankings, even though the research component in the latter relies primarily on bibliometrics. The Times Higher Education bases 30% of its World University Rankings on citations. Eleven British universities made the top 100 in its 2013–2014 rankings. Of these, eight were also in the top 10 in RAE 2008 and the other three in the top 20. Knowing this, given the choice between the intellectual charade of REF ‘expert peer review’ and appropriate metrics (the Web of Science is of little use in history) I would unhesitatingly choose the latter. It is infinitely cheaper, much less time consuming, and does not have the negative consequences for collegiality and staff morale of the present system. My own department must be one of many in which some colleagues are now no longer talking to one another because of a breakdown in trust over staff selection for REF 2014—hardly a framework for research excellence.

The conclusion I would rather draw, however, is that peer review vs. metrics is in many ways not the issue. Neither is capable of measuring research quality as such—whatever that may be. Peer review measures conformity to disciplinary expectations and bibliometrics measure how much a given output has registered on other academics’ horizons, either of which might be an indicator of quality but neither of which has to be. It seems rather silly to base 65% of the REF ranking on something that we cannot measure and that may be inherently unmeasurable because it is a subjective judgment. Perhaps we should instead be asking which features of the research environment (which currently counts for a mere 15% of the REF assessment) are most conducive to a vibrant research culture and focus funding accordingly. Library and laboratory resources, research income, faculty members’ involvement in conferences, journal or series editing, and professional associations, PhD student numbers and the intellectual life of a department as reflected in research seminars and public lectures are all good indicators of research vitality. They are also measurable.

Derek Sayer is Professor of Cultural History at Lancaster University and Professor Emeritus (Canada Research Chair in Social Theory and Cultural Studies) at the University of Alberta.  His new book Rank Hypocrisies: The Insult of the REF is to be published by Sage on December 3.

*This piece originally appeared on the LSE’s Impact of Social Sciences blog, under the title ‘Time to abandon the gold standard? Peer review for the REF falls far short of internationally accepted standards’, and is reposted with permission.

Note: This article gives the views of the author, and not the position of the Council for Defence of British Universities. This work is licensed under a Creative Commons Attribution 3.0 Unported License unless otherwise stated.