It is very welcome news that the Government has decided to include a definition of the Haldane Principle on the face of the Bill. Jo Johnson made a special point of this in his speech to Universities UK on 24 February.  An accompanying document was published jointly by both the Departments of State that will in future be responsible for higher education. It proudly states that:

the amendment that we have tabled will, for the first time in history, enshrine the Haldane Principle in law.

This document did not, however, give more details. The actual Amendment of Clause 99 proposing and containing the definition is to be found in yet another document:

Page 64, line 10, at end insert -

“The ‘Haldane principle’ is the principle that decisions on individual research proposals are best taken following an evaluation of the quality and likely impact of the proposals (such as a peer review process).”

Note that this definition does not stipulate an exercise of academic judgement, merely an ‘evaluation’ including ‘likely impact’ of research to be funded. Furthermore, the definition does not mention that infrastucture funding will come from Research England. Rather, an earlier statement merely stipulates the Councils (a sub-set of UKRI) will be responsible for the disbursement of project funding:

Page 64, line 7, at end insert -

“the Haldane principle, where the grant or direction mentioned in subsection (1) is in respect of functions exercisable by one or more of the Councils mentioned in section 91(1) pursuant to arrangements under that section,”

This took me back to the question what Haldane actually called for and the context in which he did so. His thoughts on higher education matters are chiefly to be found in some collected writings put together in a period when he was actively involved in fostering the development of the new ‘redbrick’ universities. He developed a special enthusiasm for technical education but essentially he was interested in the work of a university as a whole, not merely its research.

He recognised that if higher education was going to expand successfully something would have to be done about the funding that would be needed:

‘the truth is that work of this kind must be more largely assisted and fostered by the State than is the tradition of today if it is to succeed’

(Education and Empire: Addresses on certain topics of the day (London, 1902), p.38).

The new universities began to accept state funding but it was not at first expected that Oxford or Cambridge would need to apply. The First World War upset many expectations.

A decisive correspondence followed between November 1918 and May 1919, between the then President of the Board of Education, H. A. L. Fisher, and the Vice-Chancellor of the University of Oxford. This was published in full in May in the Oxford University Gazette, under the heading Applications for Government Grants (Oxford University Gazette, 49 (1918-9), p.471-8).

A deputation from the universities ‘asking for larger subsidies from the State’ met Fisher on 23 November. Oxford and Cambridge consulted one another and agreed that it would be wise to join in, but without committing themselves. Oxford was understandably nervous about accepting state funding because of the likelihood that it would bring State control.

But the Oxford scientists, scenting money, put in their own bids for specific sums for particular purposes. The heads of departments of the University Museum wrote on 3 March, 1919 with a list of such ‘needs’, identifying sums for capital outlay and salaries and pensions for Heads of Department and scholarships for what would now be called STEM subjects.

It was in this context that Fisher seems to have made his far-reaching policy decision and stated the ‘Fisher Principle’, that the state would not interfere in the allocation of funds within universities. It would not decide directly whether to fund, say, science at Oxford, or History at Manchester. It would give funding in the form of ‘Block Grants’ and allow the universities themselves to decide how to use the money.

He wrote to the Oxford Vice-Chancellor on 16 April:

‘Henceforth…each University which receives aid from the State will receive it in the form of a single inclusive grant, for the expenditure of which the University, as distinguished from any particular Department, will be responsible. Both the Government and, I think, the great majority of the Universities are convinced that such an arrangement is favourable not only to the preservation of University autonomy but also to the efficient administration of the University funds.’

The University’s Council (then the Hebdomadal Council, meeting weekly in term-time) requested an interview with Fisher and on May 15 a deputation of five, led by the Vice-Chancellor, had a meeting with him. The Memorandum of the Interview ‘kindly furnished by Mr. Fisher’s Secretary’ is also published in the Gazette. It repeated the policy principle arrived at in November, that ‘the English Universities in receipt of State-aid favoured …a general Block Grant’. It was explained that a Standing Committee was in process of ‘formation’ and that ‘henceforward, practically all the money for University Education would be borne on the Treasury Vote and would be allocated in annual Block Grants’ as the Standing Committee recommended.

This Standing Committee developed into the University Grants Committee, which was replaced a quarter of a century ago by first one then four Funding Councils. One of those, HEFCE, is now to be replaced as distributor of the remnant of that Block Grant mainly by Research England within UKRI, with only a vestige of the element previously used to fund teaching still remaining.

So there seem to be features of the Government Amendment to Clause 99 which would bear further thought if a definition of the ‘Haldane Principle’ is to enter statute.

The Haldane Principle arguably needs to be understood as it was developed in the ‘Fisher Principle’ and has been maintained for a century since. That placed a ‘buffer’ body between State and university and protected the freedom of the university to choose how to use its block grant on academic not political principles. That is not quite the thrust of the definition as it stands at present.

Nor did the ‘Fisher-Haldane Principle’ apply to the buffer body itself. The buffer stood between academic freedom and state control. It was not itself subject to that principle. It merely ensured that it was respected.

It is to be hoped that the legal draftsmen working on the Bill will try again. The version in the current Amendment, if it passes into law, will fail to protect the autonomy of the providers receiving funding from UKRI. Nor will it require funding decisions to be taken by academics or by autonomous institutions. The ‘peer review process’ is given as a mere example. There seems nothing to prevent a Minister or Secretary of State conducting ‘an evaluation of the quality and likely impact of the proposals’. Haldane and Fisher could both be turning in their graves.

Results from the research excellence framework (REF) were publicly announced on 18th December, followed by a spate of triumphalist messages from University PR departments. Deeper analysis followed, both in the pages of the Times Higher Education, and in the media and on blogs.

CDBU has from the outset expressed concern about the REF, much of it consistent with the criticism that has been expressed elsewhere. In particular, we note:

Inefficiency: As Derek Sayer has noted, the REF has absorbed a great deal of time and money that might have been spent better elsewhere. The precise cost has yet to be reported, but it is likely to be greater than the £60m official figure, and that is not taking into account the cost in terms of the time of academic staff. Universities have taken on new staff to do the laborious work of compiling data and writing impact statements, but this has diverted funds from front-line academia and increased administrative bloat.

Questionable validity: Derek Sayer has cogently argued the case that the peer review element of REF is open to bias from subjective, idiosyncratic and inexpert opinions. It is also unaccountable in the sense that ratings made of individual outputs are destroyed. One can see why this is done: otherwise HEFCE could be inundated with requests for information and appeals. But if the raw data is not available, then this does not inspire confidence in the process, especially when there are widespread accusations of games-playing and grade inflation.

Concentration of funding in a few institutions: We are told that the goal is to award quality-related funding, but as currently implemented, this leads inevitably to a process whereby the rich get richer and the poor get poorer, with the bulk of funds concentrated in a few institutions. We suspect that the intention of including ‘impact’ in the REF was to reduce the disparity between the Golden Triangle (Oxford, Cambridge and London) and other institutions which might be doing excellent applied work, but if anything the opposite has happened. We do not yet know what the funding formula will be, but if it is, as widely predicted, heavily biased in favour of 4* research, we could move to a situation where only the large institutions will survive to be research-active. There has been no discussion of whether such an outcome is desirable.

Shifting the balance of funding across disciplines: A recent article in the Times Higher Education noted another issue: the tendency for those in the Sciences to obtain higher scores on the REF than those in the Humanities. Quotes from HEFCE officials in the article offered no reassurance to those who were concerned this could mean a cut in funding for humanities. Such a move, if accompanied by changes to student funding to advantage those in STEM subjects, could dramatically reduce the strength of Humanities in the UK.

Unaccountable flexibility in the funding formula: There are many different ways of achieving ratings: For instance, whether or not the ratings include ‘intensity’ (number of returnable staff who were entered), can dramatically alter rank orderings. Or we could look at a suggestion by Graeme Wise that a ‘bang for your buck’ metric that assessed outputs in relation to grant income would be most appropriate. Even more radical is a suggestion by Dermot Lynott, that we should be giving the most rewards to those whose outputs were impressive in relation to their scores on environment. Needless to say, a very different profile of winners and losers emerged from such an analysis.  It will ultimately be a political decision as to how to translate the REF scores into funding. We have to ask whether it worth going through this entire long-winded exercise if, by simply changing the funding formula, one can make a dramatic difference to an institution’s funding to achieve a politically expedient outcome.

Damage inflicted on careers and morale: The criteria for entering staff for the REF could appear quite cavalier; for instance, the requirement for a numerical ratio between number of staff entered and number of case studies meant that some departments with few case studies were unable to enter all plausible staff. Derek Sayer has described instances of decisions to enter staff being made on what appeared to be flimsy evidence based on ad hoc internal evaluations. Yet being identified as ‘non-REFable’ is not only damaging to morale, but could have real impacts on prospects for promotion and job security.

Focus on competition rather than collaboration:  The REF exercise creates rank orderings, and everyone is desperately trying to nudge ahead of the others. In fact, there are so many different ways of doing the ranking, that almost everyone can be satisfied that they are ‘among the top’ on some index or other. Those who crowed loudest about their success tried to temper this by arguing that they were celebrating a broader ‘British’ success, but this seems perverse. Why should concentrating ever more of the excellent research in an ever small number of institutions be regarded as a national success story? It is, of course, widely believed that competition is a force for good, stimulating people to do better than they otherwise might. However, many in academia take the view that they don’t need to be incentivised by competition to work hard: they are in the job for the love of it, and would like their efforts to be appreciated for what they are, not because they help push the institution up a league table. Competition can also damage relationships between different departments within a University, if there are disparities in REF performance that lead to bickering about who is more valuable.

Perverse incentives that damage research: These may play out differently in different subject areas, but overall, many academics feel that they are not able to do the research they want in the way they would like. In science, there are intense pressures to publish in high-impact journals and bring in grant income. Some institutions are notorious for threatening redundancy to scientific staff who do not meet some agreed quota of research income, creating incentives to do ever more expensive research (see for instance cases at Kings College London, Imperial College, and Warwick University Medical School).  In humanities, the pressure to produce a steady stream of research articles and monographs has led to the sense of an enforced move to over-specialisation, with academics increasingly incapable of explaining or demonstrating the broader significance of their work. Younger generations of academics find that their direction of research is being wholly driven by ‘REF-ability’, and that journal publications automatically trump those in volumes of collected essays, even when the latter may be more important for the field.

Perverse incentives on hiring practices: This is another consequence of the intensely competitive culture that is induced by the REF. We have, particularly around the time of the REF, a market in research ‘super-stars’, who can attract impressive transfer fees. People from other institutions who are employed at only 20 per cent part-time suddenly appear on the books, boosting the institution’s return on funding and outputs.

Devaluation of non-research activity: Academics whose positions require them to teach and do research have felt pressured to focus principally on research, and teaching has consequently been devalued. It is sometimes suggested that the Impact agenda of the REF also encourages academics to spend time on public engagement, but in fact it has the opposite effect. Public engagement does not count as ‘impact’ for REF  purposes: to demonstrate REF impact, one must provide concrete documentation of how a specific piece of research has influenced non-academic users, such as policy-makers, health professionals, museums, etc.

How have we come to this?

Given that many of these points were made in the run-up to the announcement of REF results, we have to ask how it is that we find ourselves trapped in such an undesirable system. It is noteworthy that the REF is popular with many vice-chancellors and administrative teams. It makes it easier to manage staff, with objective criteria for hiring and firing, and provides league tables to measure progress by. For those already attached to the vision of a university as big business, it seems the natural next step to have objective rules for defining winners and losers so that one can directly measure an individual’s likely ability to bring money into the university without having to make difficult judgements about the intrinsic quality of their work.

It is a moot point whether the collapse of the circle of winners to Oxbridge and London was the result of deliberate planning, or an unintended consequence of how the system operates. Be this as it may, one concern is that this could provide further pressure for the British university system to reconfigure itself so that it can compete with the American private elite. There would be increasing reluctance to use general taxation to concentrate even more educational resources within close proximity to London, and instead we could see a shift to private funding, increasing the extent to which access to the best institutions is distributed according to wealth rather than ability.  In a few short years we could see the destruction of a genuinely national system of higher education, publicly funded because it is designed to serve everyone, to a privately funded system that is world-class for the few who can afford to access it, but a disaster for the country as a whole. As in the US, educational opportunity would be concentrated overwhelmingly in places where it can be accessed primarily by the wealthy, privileged and well-connected.

At the time of the announcement of REF results, there was a sense that anyone who criticised the celebrations was either a bad loser, or – if they came from an institution who did well – a traitor for not celebrating British success. At CDBU we are proud of UK Universities and their research reputation, but our loyalty is to our discipline, our profession, our vocation and our sense of their place in the wider scheme of things: we fear that the assessment process embodied in REF will in the longer term damage these.

Where next?

It is, of course, all very well to criticise and paint visions of a dystopian future. If we wish to replace the current system, we must look at alternative ways forward. At a debate about the REF, organised by Sage Publishers on 8^th December, David Willetts MP, who until July 2014 was Minister of State for Universities and Science, took the view that some of those who disapproved of REF were just dinosaurs who wanted to go back to the 1970s, when funds were allocated to institutions by a group of the great and the good making judgements over dinner at the Athenaeum. We would dispute that this is the only alternative to the current system. But Willetts was right on another count: he pointed out that the current REF system was not imposed by government, nor by HEFCE. Indeed, they had been actively pursuing the idea of using a simpler metrics-based system for the REF, but it was resoundingly rejected by the academic community. Government, according to Willetts, would listen to any reasonable proposal for a new system. Clearly, it is now up to the academics themselves to propose a viable alternative.

At the same meeting, David Sweeney, Director of HEFCE responsible for Education and Knowledge Exchange, implied that critics of REF just wanted to be handed public money without any accountability. He emphasised that the government and taxpayer put money into university research and had a right to know about the outcomes from the investment they had made. We totally agree. But we take issue with those who, like Mark Leach, Director and Editor-in-Chief of Wonke, think that the REF, for all its limitations, provides a good solution. Our position is that, for all the reasons given above, the REF is a seriously flawed system for deciding on disbursement of research funds, which in the long run will do the UK University system more harm than good.

What alternatives are there?

1. One possibility that has been discussed is to remove the QR component of funding altogether, and give all funds to the research councils. The problem with this solution is that it would mean the research councils would have to grow in size enormously, the load on reviewers, already seen as unsustainable by many would increase yet further, and pressures on academics to bring in research grants would be even more intense. It has also been argued that it would further increase disparities between the Golden Triangle (Oxbridge and London) and the rest, and would disfavour non-STEM disciplines.

2. HEFCE is has been looking at various publication-based metrics that might substitute for the REF, and plans to do some empirical studies comparing metric-based evaluation with REF results. However, metrics have been vigorously opposed by many in the academic community, especially in the humanities, where there is much worse agreement with expert opinion than in the sciences. We do at least now have hard data from the REF that can be considered when evaluating how metrics would perform, but there’s a real risk that introduction of any metric will further distort incentives, so that the measure becomes the goal.

3. At the Sage meeting, Derek Sayer put forward another interesting alternative, which was that funding should be based just on the ‘research environment’ component of the REF, which focuses more on inputs than outputs.

4. An even simpler option that would retain the dual support system but remove quality-related funding would involve disbursing funds purely on the basis of the number of active researchers in a department. This could be criticised for leading to a ‘prairie farming’ model whereby departments would band together to create enormous conglomerates that would benefit from economies of scale. One could, however, put a limit on the size of unit entered. At the Sage meeting, David Sweeney expressed himself as strongly opposed to this solution, even though in the last round it gave a funding result that was highly correlated with actual funding outcomes from RAE, in both science and humanities. He is right in noting that, despite the high correlation, there would nevertheless undoubtedly be winners and losers who would suffer substantial gains and losses in real terms, relative to the RAE result, but the question is whether this would involve unfairness. It’s hard to say, given that we have no gold standard. You could of course further argue that you have to have some measure of quality to incentivise people to do better, as well as a need to guard against freeloaders, like Laurie Taylor’s Dr Piercemuller. Yet, as we have noted, for most academics, exhortations from managers to ‘do better’ don’t achieve much and may indeed be counterproductive. We need to be accountable for the public money spent on university research, but subjecting every apple in the barrel to an exhaustive x-ray examination may not be the best way to identify the rotten ones.

We do not have a single solution, but we think that academics must take control of this process and not leave it in the hands of HEFCE and the government. This article about Dutch Universities Netherlands has strong parallels with the UK situation: the author argues academics have been too passive in accepting an emphasis on competition, and the use of evaluation systems as means of control. There is unlikely to be an ideal solution and we may have to live with the ‘least bad’ option. But let us consider all options in terms of how far they are likely to exacerbate or resolve the problems outlined above, or we may find ourselves saddled with something even worse than REF2014.

We hope this article opens up the discussion on this topic. Please do add your comments. We are moderating comments to exclude spam, but non-anonymous, on-topic comments will be published unless they contravene the usual rules.

Finally, if you agree with the broad concerns expressed here, please do consider joining the CDBU to help us campaign more effectively for change.

At the behest of universities minister David Willetts, HEFCE established an Independent review of the Role of Metrics in Research Assessment in April 2014 chaired by James Wilsden. This followed consultations in 2008-9 that played a decisive role in persuading the government to back down on previous plans to replace the RAE with a metrics-based system of research assessment. Wilsden’s call for evidence, which was open from 1 May to 30 June 2014, received 153 responses ‘reflecting a high level of interest and engagement from across the sector’ (Letter to Rt. Hon. Greg Clark MP). Sixty-seven of these were from HEIs, 27 from learned societies and three from mission groups. As in 2008-9, the British academic establishment (including the Russell Group, RCUK, the Royal Society, the British Academy, and the Wellcome Trust) made its voice heard. Predictably, ’57 per cent of the responses expressed overall scepticism about the further introduction of metrics into research assessment,’ while ‘a common theme that emerged was that peer review should be retained as the primary mechanism for evaluating research quality. Both sceptical and supportive responses argued that metrics must not be seen as a substitute for peer review … which should continue to be the “gold standard” for research assessment’ (Wilsden review, Summary of responses to submitted to the call for evidence).

The stock arguments against the use of metrics in research assessment were widely reiterated: journal impact factors cannot be a proxy for quality because ‘high-quality’ journals may still publish poor-quality articles; using citations as a metric ignores negative citation and self-citation; in some humanities and social science disciplines it is more common to produce books than articles, which will significantly reduce their citation counts, and so on. Much of this criticism, I would argue, is a red herring. Most of these points could easily be addressed by anybody who seriously wished to consider how bibliometrics might sensibly inform a research assessment exercise rather than kill any such suggestion at birth (don’t use JIFs, exclude self-citations, use indices like Publish or Perish that include monographs as well as articles and control for disciplinary variations). What is remarkable, however, is that while these faults are often presented as sufficient reason to reject the use of metrics in research assessment out of hand, the virtues of ‘peer review’ are simply assumed by most contributors to this discussion rather than scrutinized or evidenced. This matters because whatever the merits of peer review in the abstract—and there is room for debate on what is by its very nature a subjective process—the evaluation procedures used in REF 2014 (and previous RAEs) not only fail to meet HEFCE’s own claims to provide ‘expert review of the outputs’ but fall far short of internationally accepted norms of peer review.

In sharp contrast with the evaluative procedures used in a range of other academic contexts in the UK and internationally, the REF relies entirely on in-house assessment by panels of (so-called) experts. On some panels, like history, just one assessor may read each output, something unheard of in peer review for journal and book submissions, research grant competitions, and tenure and promotion proceedings. Additionally, almost all REF panelists are drawn from British universities alone. David Eastwood admitted back in 2007 that ‘international benchmarking of quality’ was ‘one thing that the RAE has not been able to do’—which is cute, considering that REF panels award their stars on the basis of whether outputs are ‘world leading’, ‘internationally excellent,’ or merely ‘recognized internationally.’ Eastwood then still hoped to solve the problem with ‘bibliometrics, used with sensitivity and sophistication’ (‘Goodbye to the RAE … and hello to the REF’, Times Higher Education, 30 November 2007). HEFCE’s prohibitions on using journal impact factors, rankings, or the perceived standing of publishers (and humanities and most social science panels’ refusal to use any bibliometric data, including citations) reinforce the total dependency of REF evaluations on these panelists’ subjective judgments. Meantime the abandonment of RAE 2008’s use of external advisors where a panel felt it lacked specialist expertise and an overall cut in the number of panels from 67 in RAE 2008 to 36 in REF 2014 further reduced the pool of expertise available to panels, which were now also only ‘exceptionally’ allowed to cross-refer outputs to other panels. If we could be confident that REF panels nevertheless ‘provide sufficient breadth and depth of expertise to undertake the assessment across the subpanel’s remit (including as appropriate expertise in interdisciplinary research and expertise in the wider use or benefits of research)’ (HEFCE, REF 2014: Units of Assessment and Recruitment of Expert Panels, 2010, para 55) this might not be a problem. But we cannot. Nobody on the REF history panel, for example, has specialist knowledge of China, Japan, the Middle East, Latin America, or many countries in Europe (including once-great powers like Austria-Hungary, Spain and Turkey), though work on all these areas has likely been submitted to the REF. Whatever their general eminence in the historical profession, these ‘experts’ do not know the relevant languages, archives, or literatures. How, then, can they possibly judge the ‘originality’ of an output or its ‘significance’ in any of these fields? And on what conceivable basis can they be entrusted to determine whether it is ‘internationally excellent’ or merely ‘internationally recognized’—the critical borderline between 3* research that will attract QR funding and 2* research that will not?

REF panelists are unlikely to have the time to do a proper assessment anyway. In all, around 1000 evaluators will have graded all 191,232 outputs for REF 2014 in under a year—the same number in total as the US National Endowment for the Humanities uses to evaluate 5700 applications for its 40 grant programs! Peter Coles calculates that each member of the physics panel must read 640 research papers, i.e., about two a day. ‘It is … blindingly obvious,’ he concludes, ‘that whatever the panel does do will not be a thorough peer review of each paper, equivalent to refereeing it for publication in a journal’ (‘The apparatus of research assessment’, LSE Impact Blogs, 14 May 2014). One RAE 2008 panelist told Times Higher Education that it would require ‘two years’ full-time work, while doing nothing else’ to read properly the 1200 journal articles he had been allocated (‘Burning questions for the RAE panels’, 24 April 2008). Another admitted: ‘You read them sufficiently to form a judgment, to get a feeling … you don’t have to read to the last full stop’ (‘Assessors face “drowning” as they endeavour to read 2,363 submissions’, 17 April 2008).

For major academic journals the process of review is often double blind. Though university presses and other academic book publishers divulge authors’ identities to reviewers, they will often also first consult with the author on appropriate reviewers. Reviewers are required to provide substantial comments in either case. The REF, by contrast, makes no attempt to protect authors’ anonymity—something we might think especially important when judgments may lie in the hands of a single assessor. And far from providing comments justifying their grades, RAE 2008 subpanels shredded all documents showing how they reached their conclusions and ordered members ordered to destroy personal notes in order to avoid having to reveal them under Freedom of Information Act requests. It is difficult to think of a procedure that would make it easier for evaluators to further ideological agendas or settle personal scores, should they be so inclined.

Metrics may have problems. But a process that willfully ignores whether an output has gone through any peer review before publication, where it has been published, and how often it has been cited in favor of the subjective opinions of evaluators who may have no specialist expertise in the field and then systematically erases all records does not strike me as a very defensible alternative. It also gives extraordinary gatekeeping power to the individuals who sit on REF panels. This is worrying because the mechanisms through which panelists are recruited are tailor-made for the sponsored replication of disciplinary elites. All applicants for panel chairs have to be endorsed by learned societies, chairs in turn ‘advise’ on the appointment of panel members, and at least one third of panelists have to have served in a previous RAE. Apart from being disproportionately older, white, and male compared with the UK academic profession in general, these may not always be the scholars best placed to identify cutting-edge research, especially where such research crosses or challenges disciplinary boundaries. NEH, we might note, prohibits its evaluators from serving in successive competitions to reduce this risk.

Were Wilsden’s committee to assess what might be achieved employing appropriately sophisticated metrics against what is actually done in the REF rather than comparing a crude caricature of metrics with an idealized chimera of peer review, I think it would have to take a different view of the merits of the two systems than that put forward in most responses to the call for evidence. For its REF process to be comparable to what is understood elsewhere as peer review, HEFCE would have to use subject matter-specific experts from an international pool, commission a minimum of two reviews of each output, and not overload reviewers with too many outputs for them to read them properly in the timeframe available. This would be even costlier in public money and academics’ time than the present REF. To replace the REF with metrics, on the other hand, would yield a process that is (in Dorothy Bishop’s words) ‘transparent and objective, it would not require departments to decide who they do and don’t enter for the assessment, and most importantly, it wins hands down on cost-effectiveness’ (‘An alternative to REF2014?’ Bishopblog, 26 January 2013).

Metrics have in fact proven to be highly reliable predictors of RAE performance, irrespective of whether or not they provide valid measures of research quality. It is not without irony that there is considerable overlap between RAE scores and major commercial university rankings, even though the research component in the latter relies primarily on bibliometrics. The Times Higher Education bases 30% of its World University Rankings on citations. Eleven British universities made the top 100 in its 2013–2014 rankings. Of these, eight were also in the top 10 in RAE 2008 and the other three in the top 20. Knowing this, given the choice between the intellectual charade of REF ‘expert peer review’ and appropriate metrics (the Web of Science is of little use in history) I would unhesitatingly choose the latter. It is infinitely cheaper, much less time consuming, and does not have the negative consequences for collegiality and staff morale of the present system. My own department must be one of many in which some colleagues are now no longer talking to one another because of a breakdown in trust over staff selection for REF 2014—hardly a framework for research excellence.

The conclusion I would rather draw, however, is that peer review vs. metrics is in many ways not the issue. Neither is capable of measuring research quality as such—whatever that may be. Peer review measures conformity to disciplinary expectations and bibliometrics measure how much a given output has registered on other academics’ horizons, either of which might be an indicator of quality but neither of which has to be. It seems rather silly to base 65% of the REF ranking on something that we cannot measure and that may be inherently unmeasurable because it is a subjective judgment. Perhaps we should instead be asking which features of the research environment (which currently counts for a mere 15% of the REF assessment) are most conducive to a vibrant research culture and focus funding accordingly. Library and laboratory resources, research income, faculty members’ involvement in conferences, journal or series editing, and professional associations, PhD student numbers and the intellectual life of a department as reflected in research seminars and public lectures are all good indicators of research vitality. They are also measurable.

Derek Sayer is Professor of Cultural History at Lancaster University and Professor Emeritus (Canada Research Chair in Social Theory and Cultural Studies) at the University of Alberta.  His new book Rank Hypocrisies: The Insult of the REF is to be published by Sage on December 3.

*This piece originally appeared on the LSE’s Impact of Social Sciences blog, under the title ‘Time to abandon the gold standard? Peer review for the REF falls far short of internationally accepted standards’, and is reposted with permission.

Note: This article gives the views of the author, and not the position of the Council for Defence of British Universities. This work is licensed under a Creative Commons Attribution 3.0 Unported License unless otherwise stated.