Editorial Note: In our previous blogpost, we noted that while there was agreement that REF2014 was problematic, there was less agreement about alternatives. To make progress, we need more debate. We hope that this piece by Derek Sayer will stimulate this, and we welcome comments. Please note that comments are moderated and will not be published immediately.

1.

The rankings produced by Times Higher Education and others on the basis of the UK’s Research Assessment Exercises (RAEs) have always been contentious, but accusations of universities’ gaming submissions and spinning results have been more widespread in REF2014 than any earlier RAE. Laurie Taylor’s jibe in The Poppletonian that ‘a grand total of 32 vice-chancellors have reportedly boasted in internal emails that their university has become a top 10 UK university based on the recent results of the REF’[i] rings true in a world in which Cardiff University can truthfully[ii] claim that it ‘has leapt to 5th in the Research Excellence Framework (REF) based on the quality of our research, a meteoric rise’ from 22nd in RAE2008. Cardiff ranks 5th among universities in the REF2014 ‘Table of Excellence,’ which is based on the GPA of the scores assigned by the REF’s ‘expert panels’ to the three elements in each university’s submission (outputs 65%, impact 20%, environment 15%)—just behind Imperial, LSE, Oxford and Cambridge. Whether this ‘confirms [Cardiff’s] place as a world-leading university,’ as its website claims, is more questionable.[iii]  These figures are a minefield.

Although HEFCE encouraged universities to be ‘inclusive’ in entering their staff in REF2014, they were not obliged to return all eligible staff and there were good reasons for those with aspirations to climb the league tables to be more ‘strategic’ in staff selection than in previous RAEs. Prominent among these were (1) HEFCE’s defunding of 2* outputs from 2011, which meant outputs scoring below 3* would now negatively affect a university’s rank order without any compensating gain in QR income, and (2) HEFCE’s pegging the number of impact case studies required to the number of staff members entered per unit of assessment, which created a perverse incentive to exclude research-active staff if this would avoid having to submit a weak impact case study.[iv] Though the wholesale exclusions feared by some did not materialize across the sector, it is clear that some institutions were far more selective in REF2014 than in RAE2008.

Unfortunately data that would have permitted direct comparisons with numbers of staff entered by individual universities in RAE2008 were never published, but Higher Education Statistical Authority (HESA) figures for FTE staff eligible to be submitted allow broad comparisons across universities in REF2014. It is evident from these that selectivity, rather than an improvement in research quality per se, played a large part in Cardiff’s ‘meteoric rise’ in the rankings. The same may be true for some other schools that significantly improved their positions, among them Kings (up to 7th in 2014 from 22= in 2008), Bath (14= from 20=), Swansea (22= from 56=), Cranfield (31= from 49), Heriot-Watt (33 from 45), and Aston (35= from 52=).  All of these universities except Kings entered fewer than 75% of their eligible staff members, and Kings has the lowest percentage (80%) of any university in the REF top 10 other than Cardiff itself.

Cardiff achieved its improbable rank of 5th on the basis of a submission that included only 62% of eligible staff. This is the second-lowest percentage of any of the 28 British universities that are listed in the top 200 in the 2014-15 Times Higher Education World University Rankings (of these schools only Aberdeen entered fewer staff, submitting 52%). No other university in this cohort submitted less than 70% of eligible staff, and half (14 universities) submitted over 80%. Among the top schools, Cambridge entered 95% of eligible staff, Imperial 92%, UCL 91% and Oxford 87%.

Many have suggested that ‘research power’ (which is calculated by multiplying the institution’s overall rounded GPA by the total number of full-time equivalent staff it submitted to the REF) gives a fairer indication of a university’s place in the national research hierarchy than GPA rankings alone. By this measure, Cardiff falls to a more credible but still respectable 18th. But when measured by ‘research intensity’ (that is, GPA multiplied by the percentage of eligible staff entered), its rank plummets from 5th to 50th. To say this provides a more accurate indication of its true standing might be overstating the case, but it certainly underlines why Cardiff does not belong among ‘world-leading’ universities. Cardiff doubtless produces some excellent research, but its overall (and per capita) performance does not remotely justify comparisons with Oxford, Cambridge, or Imperial—let alone Caltech, Harvard, Stanford, Princeton, MIT, UC-Berkeley and Yale (the other universities in the THE World University Rankings top 10). In this sense the GPA Table of Excellence can be profoundly misleading.

‘To their critics,’ writes Paul Jump in Times Higher Education, ‘such institutions are in essence cheating because in reality their quality score reflects the work produced by only a small proportion of their staff.’[v] I am not sure the accusation of cheating is warranted, because nobody is doing anything here that is outside HEFCE’s rules. The problem is rather that the current REF system rewards—and thereby encourages—bad behavior, while doing nothing to penalize the most egregious offenders like Cardiff.

The VCs at Bristol (11= in the REF2014 GPA table) and Southampton (18=, down from 14= in 2008) might be forgiven for ruefully reflecting that they, too, might now be boasting that they are ‘a top ten research university’ had they not chosen to submit 91% and 90% of their eligible faculty respectively—a submission rate that on any reasonable criteria (as distinct from HEFCE’s rules) might itself be seen as an indicator of research excellence.  Measured by research intensity Bristol comes in at 5= (jointly with Oxford) and Southampton at 8= (jointly with Queen’s University Belfast, which submitted 95% of its staff and is ranked 42= on GPA).  Meantime the VCs at St Andrews (down from 14= to 21=, 82% of eligible staff submitted), Essex (11th to 35=, 82% submitted), Loughborough (28= to 49=, 88% submitted) and Kent (31= to 49=, 85% submitted) may by now have concluded that—assuming they hold onto their jobs—they will have no alternative other than to be much more ruthless in culling staff for any future REF.

2.

The latest Times Higher Education World University Rankings puts Cardiff just outside the top 200, in the 201-225 group—which places it 29= among UK universities, along with Dundee, Newcastle, and Reading. Taking GPA, research power and research intensity into account—as we surely should, in recognition that not only the quality of research outputs but the number and proportion of academic staff who are producing them are also necessary elements in evaluating any university’s overall contribution to the UK’s research landscape—such a ranking feels intuitively to be just about right.

I have shown elsewhere[vi] that there was, in fact, a striking degree of overall agreement between the RAE2008 rankings and the Times Higher Education World University Rankings. Repeating the comparison for UK universities ranked in the top 200 in the THE World University Rankings for 2014-15 and the REF2014 GPA-based ‘Table of Excellence’ yields similar findings. These data are summarized in Table 1.

Table 1:  REF2014 performance of universities ranked in the top 200 in Times Higher Education World University Rankings 2014-15   

Seven UK universities make the top 50 in the 2014-15 THE World University Rankings: Oxford, Cambridge, Imperial, UCL, LSE, Edinburgh, and Kings.  Six of these are also in the REF2014 top 10, while the other (Edinburgh) is only just outside it at 11=.  Four of the leading five institutions are same in both rankings (the exception being UCL, which is 8= in REF 2014), though not in the same rank order. Of 11 UK universities in THE top 100, only one (Glasgow, at 24th) is outside the REF top 20.  Of 22 UK universities in THE top 150, only two are outside REF top 30 (Birmingham, 31 in REF, and Sussex, 40 in REF).  Of 28 UK universities in THE top 200, only two are outside the REF top 40 (Aberdeen at 46= and Leicester at 53).

Conversely, only two universities in the REF2014 top 20, Cardiff at 6 and Bath at 14=, do not make it into the THE top 200 (their respective ranks are 201-225 and 301-350). Other universities that are ranked in the top 40 in REF2014 but remain outside the THE top 200 are Newcastle (26=), Swansea (26=), Cranfield (31), Herriot-Watt (33), Essex (35=), Aston (35=), Strathclyde (37), Dundee (38=) and Reading (38=).

Table 2 provides data on the performance of selected UK universities that submitted to REF2014 but are currently ranked outside the THE world top 200.

Table 2.  REF2014 performance of selected UK universities outside top 200 in Times Higher Education World University Rankings 2014-15

Dundee, Newcastle and Reading only just miss the THE cut (they are all in the 201-225 bracket). While all three outscored Aberdeen and Leicester, who are above them in the THE rankings (in Leicester’s case, at 199, very marginally so) in the REF, only Newcastle does substantially worse in the THE rankings than in the REF.  It is ranked 26= in the REF with Nottingham and Royal Holloway, ahead of Leicester (53), Aberdeen (46), Sussex (40), Liverpool (33), Birmingham (31) and Exeter (30)—all of which are in the top 200 in the THE World Rankings. While there was a yawning gulf between Essex’s RAE2008 ranking of 11th and its THE ranking in the 301-350 group, the latter does seem to have presaged its precipitous REF2014 fall from grace to 35=.  Conversely, the THE inclusion of Plymouth in the 276-300 group of universities places it considerably higher than its RAE rank of 66= would lead us to expect. This is not the case with most of the UK universities listed in the lower half of the THE top 400. Birkbeck, Bangor, Aberystwyth and Portsmouth also all found themselves outside the top 40 in REF 2014.

The greatest discrepancies between REF2014 and THE World Rankings come with Cardiff (6 in REF, 201-225 in REF), Bath (14= in REF, 301-350 in THE), Swansea (26= in REF, not among THE top 400), Aston (35= in REF, 350-400 in THE), Cranfield, Heriot-Watt and Strathclyde (31=, 33 and 37 respectively in REF, yet not among THE top 400). On the face of it, these cases flatly contradict any claim that THE (or other similar) rankings are remotely accurate predictors of REF performance. I would argue, on the contrary, that these are the exceptions that prove the rule. With the exception of Strathclyde (18 in research intensity with 84% of eligible staff submitted and the worst-performing member of this group in REF GPA), all these schools were prominent among universities who inflated their GPA by submitting smaller percentages of their eligible staff in REF2014.  Were we to adjust raw GPA figures by research intensity, we would get a much closer match, as Table 3 shows.

Table 3.  Comparison of selected universities performance in THE World University Rankings 2014-15 and REF2014 by GPA and research intensity.

The most important general conclusion to emerge from this discussion is that despite some outliers there is a remarkable degree of agreement between the top 40 in REF2014 and top 200 in the THE 2014-15 World University Rankings, and the correlation increases the higher we go in the tables.  Where there are major discrepancies, these are usually explained by selective staff submission policies.

One other correlation is worth noting at this point.  All 11 of the British universities in the THE top 100 are members of the Russell Group, as are 10 of the 17 British universities ranked between 100-200. The other six universities in this latter cohort (St Andrews, Sussex, Royal Holloway, Lancaster, UEA, Leicester) were all members of the now-defunct 1994 Group. Only one British university in the THE top 200 (Aberdeen) belonged to neither the Russell Group nor the 1994 Group. Conversely, only two Russell Group universities, Newcastle and Queen’s University Belfast, did not make the top 200 in the THE rankings.[vii]  In 2013-14 Russell Group and former 1994 Group universities between them received almost 85% of QR funding.  Here, too, an enormous amount of money, time, and acrimony seems to have been expended on a laborious REF exercise that merely confirms what THE rankings have already shown.

3.

The most interesting thing about this comparative exercise is that the Times Higher Education World University Rankings not only make no use of RAE/REF data, but rely on quantitative methodologies that have repeatedly been rejected by the British academic establishment in favor of the ‘expert peer review’ that is supposedly offered by REF panels. THE gives 30% of the overall score for the learning environment, 7.5% for international outlook, and 2.5% for industry income. The remaining 60% is based entirely on research-related measures, of which ‘the single most influential of the 13 indicators,’ counting for 30% of the overall THE score, is ‘the number of times a university’s published work is cited by scholars globally’ as measured by the Web of Science. The rest of the research score is derived from research income (6%), ‘research output scaled against staff numbers’ (6%, also established through the Web of Science), and ‘a university’s reputation for research excellence among its peers, based on the 10,000-plus responses to our annual academic reputation survey’ (18%).

The comparisons undertaken here strongly suggest that at such metrics-based measures have proved highly reliable predictors of performance in REF2014—just as they did in previous RAEs. To be sure, there are differences in the details of the order of ranking of institutions between the THE and REF, but in such cases can we be confident that it is the REF panels’ highly subjective judgments of quality that are the more accurate? To suggest there is no margin for error in tables where the difference in GPA between 11th (Edinburgh, 3.18) and 30th (Exeter, 3.08) is a mere 0.1 points would be ridiculous.  I have elsewhere suggested that there are in fact many reasons why such confidence would be totally misplaced, including lack of specialist expertise among panel members and lack of time for reading outputs in the depth required.[viii]  But my main point is this.

If metrics-based measures can produce similar results to those arrived at through the REF’s infinitely more costly, laborious and time-consuming process of ‘expert review’ of individual outputs, there is a compelling reason to go with the metrics; not because it is necessarily a valid measure of anything but because it as reliable as the alternative (whose validity is no less dubious for different reasons) and a good deal more cost-efficient. The benefits for collegiality and staff morale of universities not having to decide whom to enter or exclude from the REF might be seen as an additional reason for favoring metrics. I am sure that if HEFCE put their minds to it they could come up with a more sophisticated basket of metrics than Times Higher Education, which would be capable of meeting many of the standard objections to quantification.  I hope James Wilsdon’s committee might come up with some useful suggestions for ways forward.

[i] Laurie Taylor, ‘We have bragging rights!’ in The Poppletonian, Times Higher Education, 8 January 2015.

[ii] Well, not quite. Cardiff is actually ranked 6th in the REF2014 ‘Table of Excellence,’ which is constructed by Times Higher Education on the basis of the grade point average (GPA) of the marks awarded by REF panels, but the #1 spot is held not by a university but the Institute of Cancer Research (which submitted only two UoAs).  This table and others drawn upon here for ‘research power’ and ‘research intensity’ can be downloaded here.

[iii] ‘REF 2014,’ Cardiff University website.

[iv] Paul Jump, ‘Careers at risk after case studies ‘game playing’, REF study suggests.’  Times Higher Education, 22 January 2015, at

[v] Paul Jump, ‘REF 2014 rerun: who are the ‘game players’?’  Times Higher Education, 1 January 2015.

[vi] See Derek Sayer, Rank Hypocrisies: The Insult of the REF.  London: Sage, 2014.

[vii] I have discussed Newcastle already.  Queen’s came in just outside the REF top 40 (42=) but with an excellent intensity rating (8=, 95% of eligible staff submitted).

[viii] See, apart from Rank Hypocrisies, ‘One scholar’s crusade against the REF,’ Times Higher Education, 11 December, 34-6; ‘Time to abandon the gold standard?  Peer Review for the REF Falls Far Short of Internationally Acceptable Standards,’ LSE Impact of Social Sciences blog, 19 November (reprinted as ‘Problems with peer review for the REF,’ CDBU blog, 21 November).

*This is a revised version of an article first posted on Sayer’s blog coastsofbohemia on 27 January 2015.

Note: This article gives the views of the author, and not the position of the Council for Defence of British Universities. This work is licensed under a Creative Commons Attribution 3.0 Unported License unless otherwise stated.

Results from the research excellence framework (REF) were publicly announced on 18th December, followed by a spate of triumphalist messages from University PR departments. Deeper analysis followed, both in the pages of the Times Higher Education, and in the media and on blogs.

CDBU has from the outset expressed concern about the REF, much of it consistent with the criticism that has been expressed elsewhere. In particular, we note:

Inefficiency: As Derek Sayer has noted, the REF has absorbed a great deal of time and money that might have been spent better elsewhere. The precise cost has yet to be reported, but it is likely to be greater than the £60m official figure, and that is not taking into account the cost in terms of the time of academic staff. Universities have taken on new staff to do the laborious work of compiling data and writing impact statements, but this has diverted funds from front-line academia and increased administrative bloat.

Questionable validity: Derek Sayer has cogently argued the case that the peer review element of REF is open to bias from subjective, idiosyncratic and inexpert opinions. It is also unaccountable in the sense that ratings made of individual outputs are destroyed. One can see why this is done: otherwise HEFCE could be inundated with requests for information and appeals. But if the raw data is not available, then this does not inspire confidence in the process, especially when there are widespread accusations of games-playing and grade inflation.

Concentration of funding in a few institutions: We are told that the goal is to award quality-related funding, but as currently implemented, this leads inevitably to a process whereby the rich get richer and the poor get poorer, with the bulk of funds concentrated in a few institutions. We suspect that the intention of including ‘impact’ in the REF was to reduce the disparity between the Golden Triangle (Oxford, Cambridge and London) and other institutions which might be doing excellent applied work, but if anything the opposite has happened. We do not yet know what the funding formula will be, but if it is, as widely predicted, heavily biased in favour of 4* research, we could move to a situation where only the large institutions will survive to be research-active. There has been no discussion of whether such an outcome is desirable.

Shifting the balance of funding across disciplines: A recent article in the Times Higher Education noted another issue: the tendency for those in the Sciences to obtain higher scores on the REF than those in the Humanities. Quotes from HEFCE officials in the article offered no reassurance to those who were concerned this could mean a cut in funding for humanities. Such a move, if accompanied by changes to student funding to advantage those in STEM subjects, could dramatically reduce the strength of Humanities in the UK.

Unaccountable flexibility in the funding formula: There are many different ways of achieving ratings: For instance, whether or not the ratings include ‘intensity’ (number of returnable staff who were entered), can dramatically alter rank orderings. Or we could look at a suggestion by Graeme Wise that a ‘bang for your buck’ metric that assessed outputs in relation to grant income would be most appropriate. Even more radical is a suggestion by Dermot Lynott, that we should be giving the most rewards to those whose outputs were impressive in relation to their scores on environment. Needless to say, a very different profile of winners and losers emerged from such an analysis.  It will ultimately be a political decision as to how to translate the REF scores into funding. We have to ask whether it worth going through this entire long-winded exercise if, by simply changing the funding formula, one can make a dramatic difference to an institution’s funding to achieve a politically expedient outcome.

Damage inflicted on careers and morale: The criteria for entering staff for the REF could appear quite cavalier; for instance, the requirement for a numerical ratio between number of staff entered and number of case studies meant that some departments with few case studies were unable to enter all plausible staff. Derek Sayer has described instances of decisions to enter staff being made on what appeared to be flimsy evidence based on ad hoc internal evaluations. Yet being identified as ‘non-REFable’ is not only damaging to morale, but could have real impacts on prospects for promotion and job security.

Focus on competition rather than collaboration:  The REF exercise creates rank orderings, and everyone is desperately trying to nudge ahead of the others. In fact, there are so many different ways of doing the ranking, that almost everyone can be satisfied that they are ‘among the top’ on some index or other. Those who crowed loudest about their success tried to temper this by arguing that they were celebrating a broader ‘British’ success, but this seems perverse. Why should concentrating ever more of the excellent research in an ever small number of institutions be regarded as a national success story? It is, of course, widely believed that competition is a force for good, stimulating people to do better than they otherwise might. However, many in academia take the view that they don’t need to be incentivised by competition to work hard: they are in the job for the love of it, and would like their efforts to be appreciated for what they are, not because they help push the institution up a league table. Competition can also damage relationships between different departments within a University, if there are disparities in REF performance that lead to bickering about who is more valuable.

Perverse incentives that damage research: These may play out differently in different subject areas, but overall, many academics feel that they are not able to do the research they want in the way they would like. In science, there are intense pressures to publish in high-impact journals and bring in grant income. Some institutions are notorious for threatening redundancy to scientific staff who do not meet some agreed quota of research income, creating incentives to do ever more expensive research (see for instance cases at Kings College London, Imperial College, and Warwick University Medical School).  In humanities, the pressure to produce a steady stream of research articles and monographs has led to the sense of an enforced move to over-specialisation, with academics increasingly incapable of explaining or demonstrating the broader significance of their work. Younger generations of academics find that their direction of research is being wholly driven by ‘REF-ability’, and that journal publications automatically trump those in volumes of collected essays, even when the latter may be more important for the field.

Perverse incentives on hiring practices: This is another consequence of the intensely competitive culture that is induced by the REF. We have, particularly around the time of the REF, a market in research ‘super-stars’, who can attract impressive transfer fees. People from other institutions who are employed at only 20 per cent part-time suddenly appear on the books, boosting the institution’s return on funding and outputs.

Devaluation of non-research activity: Academics whose positions require them to teach and do research have felt pressured to focus principally on research, and teaching has consequently been devalued. It is sometimes suggested that the Impact agenda of the REF also encourages academics to spend time on public engagement, but in fact it has the opposite effect. Public engagement does not count as ‘impact’ for REF  purposes: to demonstrate REF impact, one must provide concrete documentation of how a specific piece of research has influenced non-academic users, such as policy-makers, health professionals, museums, etc.

How have we come to this?

Given that many of these points were made in the run-up to the announcement of REF results, we have to ask how it is that we find ourselves trapped in such an undesirable system. It is noteworthy that the REF is popular with many vice-chancellors and administrative teams. It makes it easier to manage staff, with objective criteria for hiring and firing, and provides league tables to measure progress by. For those already attached to the vision of a university as big business, it seems the natural next step to have objective rules for defining winners and losers so that one can directly measure an individual’s likely ability to bring money into the university without having to make difficult judgements about the intrinsic quality of their work.

It is a moot point whether the collapse of the circle of winners to Oxbridge and London was the result of deliberate planning, or an unintended consequence of how the system operates. Be this as it may, one concern is that this could provide further pressure for the British university system to reconfigure itself so that it can compete with the American private elite. There would be increasing reluctance to use general taxation to concentrate even more educational resources within close proximity to London, and instead we could see a shift to private funding, increasing the extent to which access to the best institutions is distributed according to wealth rather than ability.  In a few short years we could see the destruction of a genuinely national system of higher education, publicly funded because it is designed to serve everyone, to a privately funded system that is world-class for the few who can afford to access it, but a disaster for the country as a whole. As in the US, educational opportunity would be concentrated overwhelmingly in places where it can be accessed primarily by the wealthy, privileged and well-connected.

At the time of the announcement of REF results, there was a sense that anyone who criticised the celebrations was either a bad loser, or – if they came from an institution who did well – a traitor for not celebrating British success. At CDBU we are proud of UK Universities and their research reputation, but our loyalty is to our discipline, our profession, our vocation and our sense of their place in the wider scheme of things: we fear that the assessment process embodied in REF will in the longer term damage these.

Where next?

It is, of course, all very well to criticise and paint visions of a dystopian future. If we wish to replace the current system, we must look at alternative ways forward. At a debate about the REF, organised by Sage Publishers on 8^th December, David Willetts MP, who until July 2014 was Minister of State for Universities and Science, took the view that some of those who disapproved of REF were just dinosaurs who wanted to go back to the 1970s, when funds were allocated to institutions by a group of the great and the good making judgements over dinner at the Athenaeum. We would dispute that this is the only alternative to the current system. But Willetts was right on another count: he pointed out that the current REF system was not imposed by government, nor by HEFCE. Indeed, they had been actively pursuing the idea of using a simpler metrics-based system for the REF, but it was resoundingly rejected by the academic community. Government, according to Willetts, would listen to any reasonable proposal for a new system. Clearly, it is now up to the academics themselves to propose a viable alternative.

At the same meeting, David Sweeney, Director of HEFCE responsible for Education and Knowledge Exchange, implied that critics of REF just wanted to be handed public money without any accountability. He emphasised that the government and taxpayer put money into university research and had a right to know about the outcomes from the investment they had made. We totally agree. But we take issue with those who, like Mark Leach, Director and Editor-in-Chief of Wonke, think that the REF, for all its limitations, provides a good solution. Our position is that, for all the reasons given above, the REF is a seriously flawed system for deciding on disbursement of research funds, which in the long run will do the UK University system more harm than good.

What alternatives are there?

1. One possibility that has been discussed is to remove the QR component of funding altogether, and give all funds to the research councils. The problem with this solution is that it would mean the research councils would have to grow in size enormously, the load on reviewers, already seen as unsustainable by many would increase yet further, and pressures on academics to bring in research grants would be even more intense. It has also been argued that it would further increase disparities between the Golden Triangle (Oxbridge and London) and the rest, and would disfavour non-STEM disciplines.

2. HEFCE is has been looking at various publication-based metrics that might substitute for the REF, and plans to do some empirical studies comparing metric-based evaluation with REF results. However, metrics have been vigorously opposed by many in the academic community, especially in the humanities, where there is much worse agreement with expert opinion than in the sciences. We do at least now have hard data from the REF that can be considered when evaluating how metrics would perform, but there’s a real risk that introduction of any metric will further distort incentives, so that the measure becomes the goal.

3. At the Sage meeting, Derek Sayer put forward another interesting alternative, which was that funding should be based just on the ‘research environment’ component of the REF, which focuses more on inputs than outputs.

4. An even simpler option that would retain the dual support system but remove quality-related funding would involve disbursing funds purely on the basis of the number of active researchers in a department. This could be criticised for leading to a ‘prairie farming’ model whereby departments would band together to create enormous conglomerates that would benefit from economies of scale. One could, however, put a limit on the size of unit entered. At the Sage meeting, David Sweeney expressed himself as strongly opposed to this solution, even though in the last round it gave a funding result that was highly correlated with actual funding outcomes from RAE, in both science and humanities. He is right in noting that, despite the high correlation, there would nevertheless undoubtedly be winners and losers who would suffer substantial gains and losses in real terms, relative to the RAE result, but the question is whether this would involve unfairness. It’s hard to say, given that we have no gold standard. You could of course further argue that you have to have some measure of quality to incentivise people to do better, as well as a need to guard against freeloaders, like Laurie Taylor’s Dr Piercemuller. Yet, as we have noted, for most academics, exhortations from managers to ‘do better’ don’t achieve much and may indeed be counterproductive. We need to be accountable for the public money spent on university research, but subjecting every apple in the barrel to an exhaustive x-ray examination may not be the best way to identify the rotten ones.

We do not have a single solution, but we think that academics must take control of this process and not leave it in the hands of HEFCE and the government. This article about Dutch Universities Netherlands has strong parallels with the UK situation: the author argues academics have been too passive in accepting an emphasis on competition, and the use of evaluation systems as means of control. There is unlikely to be an ideal solution and we may have to live with the ‘least bad’ option. But let us consider all options in terms of how far they are likely to exacerbate or resolve the problems outlined above, or we may find ourselves saddled with something even worse than REF2014.

We hope this article opens up the discussion on this topic. Please do add your comments. We are moderating comments to exclude spam, but non-anonymous, on-topic comments will be published unless they contravene the usual rules.

Finally, if you agree with the broad concerns expressed here, please do consider joining the CDBU to help us campaign more effectively for change.

Universities have become obsessed with competition: it is no longer enough to do well; you have to demonstrate you are better than the rest. And to do that, you need some kind of metric. Organisations have grown up to meet this need, and to produce league tables that compare institutions on a range of characteristics, including research excellence, reputation and teaching.

The National Student Survey has become established as a major component of this process. It has run annually across all publicly funded Higher Education Institutions (HEIs) in the UK. It features prominently in student guides to the best universities, such as this one by the Guardian. There is no doubt that the survey has made universities more responsive to student views, and it is to be welcomed that reported student satisfaction levels have increased since the survey was introduced. Nevertheless, some, like Arti Agrawal have expressed concerns about universities introducing quick fixes that may produce higher ratings in the short term, but lower academic quality overall: ‘With increased tuition fees, students are seen as customers who must be kept happy, and the NSS is now a customer satisfaction survey’. We even have evidence that within some universities, student satisfaction is used as an index of the quality of the teaching staff.

It is perhaps not surprising then, that as the same time as we are told that students are getting happier and happier, academic staff seem to be growing ever more miserable. Now this could, of course, just be down to the fact that everyone likes a good moan¹. But the impression one gets from reading the Times Higher Education and looking at stories anonymously contributed to CDBU’s Record the Rot archive is that there is more to it than that. The very same pressures that lead managers to treat students as consumers have led them to treat academic staff as dispensible ‘human resources’. The view of universities as institutions in constant competition with one another and the rest of the world has trickled down to the departmental level, destroying any sense of collegiality. In the long run, if teaching is done by a body of demoralised and ever-changing academics, this can only be bad for staff and students alike.

But this is only anecdote, and it would be good to have some data. The Times Higher Education started a Best Workplace Survey last year, which has the potential to provide just that. However, the sample was relatively small and self-selected. Findings such as 39 per cent of academics felt their health was negatively affected by their work, and one third felt their job was not secure are hard to interpret given the vagaries of sampling. Is this typical, or was it the most disaffected who replied? Concerns about the low response rate and potential for bias meant that the THE decided not to report results by institution. My guess is that if we had proper survey data, and if staff satisfaction were incorporated into ‘best university’ rankings, then rank orderings might change quite dramatically. Furthermore, institutions sacked staff to improve rankings might find their strategy backfiring.

The THE’s workplace survey for 2015 is now live. I would encourage everyone working in higher education to take part, whether or not you have something you want to moan about. We need to get an adequate database on this topic so that we can have a solid basis for identifying those institutions that are genuinely at the top of the league, in terms of their treatment of staff, versus those who achieve a high status on other indicators while presiding over an anxious and demoralised staff.

¹ Especially the English. I can thoroughly recommend this book for an amusing and informative account: Fox, K. (2005). Watching the English: The Hidden Rules of English Behaviour. London: Hodder & Stoughton.