More Misrepresentation in the Green Paper Damages its Credibility

Opinion piece by Dorothy Bishop

In my previous post, I queried the justification for the proposed new Teaching Excellence Framework. According to the Green Paper, research-intensive universities undervalue teaching, and students and employers are dissatisfied with the quality of teaching in UK Higher Education. I argued that evidence for these claims was lacking. I have now scrutinised in detail the case made in the Green Paper. I thought that perhaps there was better evidence buried in there that I had missed. What I discovered was alarming. I found numerous instances where evidence was cited but in a misleading way.

Here are some examples:

Introduction: The productivity challenge, point 9:

Higher education providers need to provide degrees with lasting value to their recipients. This will mean providers being open to involving employers and learned societies representing professions in curriculum design. It will also mean teaching students the transferrable work readiness skills that businesses need, including collaborative teamwork and the development of a positive work ethic, so that they can contribute more effectively to our efforts to boost the productivity of the UK economy.”

A report by the UK Commission for Employment and Skills is cited in the Green Paper. There is just one mention of graduates in the report, on page 3. It states: “Over eight in ten employers found university graduates to be well prepared for work.”

Introduction: The transparency challenge, point 15:

Students are also concerned about value for money, with one third of undergraduates paying higher fees in England believing their course represents very poor or poor value for money.”

This refers to findings from the 2015 HEPI-HEA Student Academic Experience Survey, which states:

“Perceptions of value for money have diverged as a result of the increase in the full-time undergraduate fee cap to £9,000 in 2012 for students from England (and for students from Scotland and Northern Ireland who travel elsewhere in the UK to study). Only 7% of students from England on the higher fees feel they receive ‘very good’ value for money – the figure for students from Scotland who remain there to study is five times higher (35%).”

This puts into perspective the Green Paper proposals to fix the problem by enabling Universities with good ratings for teaching excellence to increase their fees.

Introduction: Driving up teaching standards, point 19:

There are many examples of excellent teaching within the higher education system but, as NSS data suggests, teaching quality is variable. In 2015, more than half of providers performed significantly below expected levels in at least one element of the NSS.”

I covered this claim in this blogpost. It seems that BIS have presented the data to portray University teaching in the worst possible light. I contacted HEFCE to ask for the source of these statistics and they explained the analysis did not come from them, and they referred me on to BIS. I will report on my personal blog if I get any enlightenment from them as to how the figures were arrived at.

Part A; Chapter 1; point 8:

Information about the quality of teaching is also vital to UK productivity. In an increasingly globalised world, the highest returns go to the individuals and economies with the highest skills. However, the absence of information about the quality of courses, subjects covered and skills gained makes it difficult for employers to identify and recruit graduates with the right level of skills and harder for providers to know how to develop and improve their courses. For example, the Association of Graduate Recruiters (2015) found that almost a quarter of employers had open vacancies because they couldn’t find the right skills in the most recent graduate cohort14.”

The reference (14) is to this report by the National Centre for Universities and Business. As summarised in this Telegraph piece, the report does indeed refer to a skills shortage. But this has nothing to do with teaching quality, and everything to do with a lack of students taking degrees in science and engineering. For instance, the report states: “We have a booming IT sector, estimated to require 750,000 skilled digital workers by 2017, but in 2011 there were only 56,025 computer science graduates, a drop of 23.3% over 10 years.” There is a problem to be fixed, but it is not going to be fixed by a Teaching Excellence Framework.

Part A, Chapter 1, point 10:

There is evidence to suggest ‘strong orientations towards research often reveal a weak emphasis on teaching, and vice versa’15. At its most extreme, because some universities see their reputation, their standing in prestigious international league tables and their marginal funding as being principally determined by scholarly output, this can result in teaching becoming something of a poor cousin to research in parts of our system.”

Reference 15 is to this report by Graham Gibbs. This is a fascinating and scholarly examination of factors affecting teaching quality, but the quote from the Green Paper is lifted from a section (5.4) that gives a much more nuanced analysis of the relationship between research focus and teaching quality. Gibbs states: “The best research departments may or may not be the best teaching departments: there is no correlation between measures of a department’s research and measures of its teaching”, supporting this statement with a raft of references. The quote used in the Green Paper refers to a comment he makes on the negative impact on students when an institution is focused overwhelmingly on research: there is one supporting reference for this, from a study conducted in the USA and published in 1993.

Anecdotally, most UK academics will have stories of downgrading of teaching because of perverse incentives induced by the Research Excellence Framework (REF), but, as noted in my previous post, evidence from the National Student Survey (NSS) does not support the idea that this is a general problem. Many would argue that the solution is to get rid of the REF, rather than to create an equally perverse system to evaluate teaching excellence. Insofar as institutions need an external incentive to prioritise teaching, the NSS itself has provided this, as institutions recognise that a poor showing on NSS league tables will dent student applications.

Part A, Chapter 1, point 12:

The annual HEPI Survey (undertaken with HEA in 2015) found that clear priorities of students while at university included; ‘having more hours of teaching’, ‘reducing the size of teaching groups’ and ‘better training for lecturers’, but there is little information for prospective students on this in advance. However, most league tables do not seek to measure teaching excellence, for which there have only been imperfect proxy measures to date in any case. This can ultimately lead to disappointment in what they receive, poor value for the student and a poorer return for the economy as a whole.”

Here are the first three conclusions from the summary report:

  • The UK higher education system is in good health: an overwhelming majority of undergraduates (87%) are ‘fairly’ or ‘very’ satisfied with the quality of their course.
  • Students were asked to think back to when they applied for their current university and to compare their original expectations with the reality of their academic experience. More students (28%) said their expectations had been exceeded than said their experience was worse than expected (12%), but around half of respondents (49%) said their experience was better in some ways and worse in others.
  • Those who felt their experience had not matched up to their expectations or had been better in some ways and worse in others were asked why. The most common option was that they had not put in enough effort themselves.

What about those student priorities? The way this is presented implies that students are dissatisfied with the number of teaching hours, the size of teaching groups and the level of training of their lecturers. I searched the report for this evidence, but it seemed to refer to a set of questions that were framed the opposite way, i.e. “In which areas would you most prefer your University to save money?” The least preferred options were “reducing spending on learning facilities (5%), lowering the number of contact hours (6%) and reducing the support available for academics to improve their teaching (8%).” Framed this way, it seems students regard these aspects of their academic experience as important, which is not the same as saying they are currently dissatisfied.

I would not dismiss the need for better information available to students, but I see no evidence here of widespread student disappointment in teaching quality.

Overall, I see the Green Paper as illustrating an urgent need for those who drafted it to spend some time visiting UK Universities, where they might learn the skills of logical thinking and balanced appraisal of evidence.