A note by Lord Lipsey    

The National Students Survey results matter. First, they are used by students to evaluate institutions by comparison with rival institutions. Secondly, they are one of the metrics to be used in the TEF, in awarding gold, silver or bronze markings to institutions which apply to take part. These ratings will decide if an institution can or cannot raise its fees beyond £9K.

The idea that student satisfaction should play a major role in the rating of universities is controversial. Research shows that there is no correlation between student satisfaction and student results in terms of degree grade . However the government has opted to increase the importance of student choice, competition and satisfaction in the higher education landscape; and this short note does not seek to address the rights and wrongs of that.

Rather it focusses on a narrow point: whether the National Student Survey (NSS), the chosen instrument to measure student satisfaction, is fit for purpose or not.  Here the evidence is unequivocal, The NSS is statistical garbage, The reasons are widely understood by the statistical community and were set out inter alia by the Royal Statistical Society in its response to the government’s technical consultation about the TEF http://www.rss.org.uk/Images/PDF/influencing-change/2016/RSS-response-to-BIS-Technical Consultation-on-Teaching-Excellence-Framework-year-2.pdf.

1. The NSS is not based on a random or representative sample of students. It is more akin to a census, which includes everyone who chooses to complete the form but not those who don’t. Some groups eg ethnic minorities are seriously underrepresented. As the final report of the ONS Review of Data Sources for the TEF 2016 said: “under-reporting of certain groups and over-coverage of others …could lead to bias in use of the data.” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/556352/Teaching-Excellence-Framework-review-of-data-sources.pdf

I chair Trinity Laban Conservatoire of Music and Dance, a leading London Conservatoire. This year Trinity Laban’s response rate to the NSS rose from about 60% to about 80%. This makes any comparison between last year’s results and this year’s results invalid. We know nothing about the 20%; so a 50% satisfaction rate amongst respondents could perfectly well be a 70% satisfaction rate amongst students as whole.

2. Even if the responses are treated as a random or representative sample, making calculations of statistical significance possible, the margins of error in the NSS figures are large. The ONS concluded that “differences between institutions at the overall level are small and are not significant” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/556352/Teaching-Excellence-Framework-review-of-data-sources.pdf Yet on that are being based student decisions on where to go, and the level of fees institutions can charge.

So for example Trinity Laban’s  music results – and music is our biggest group of students -are based on 112 students. To take an example only 49% of these students say that marking on their course has been fair. Statistically however and treating the returns as if they were a sample, this means that there is a 95% chance that between 34% and 64% of students think it is fair. This perhaps exaggerates the unreliability of the NSS insofar as. if response rates are high, then the results may be less unreliable that they would be for a sample. But the point still applies. And.in cases of smaller sub-samples the low response renders the results vacuous. For example for Trinity Laban Musical Theatre the poor results were based on response from just 23 students.

The small sample size is a particular problem for small institutions. It is also a grave problem in that the most valid comparisons are not between institutions, but between particular departments in institutions teaching the same or similar courses. For individual courses, there will tend to be only a fraction of the responses obtained for institutions as a whole.

3. The results can be greatly affected by happenstance. Results in the TL musical theatre department in the past have been good which suggests that there was some peculiar chemistry about this year (and also there is some suggestion that students coordinated their responses to make a point). A survey of dance satisfaction for Trinity Laban was completed the day after a one-off cock-up about room bookings; had it been done two days earlier the results might have been much better.

4. There is scope for “gaming” the results and encouraging students to give positive results,. Trinity Laban does not do this. Anecdotal evidence, as cited by the RSS paper above suggests that less scrupulous institutions do, and the incentive to do so will be greatly increased now that the NSS has new significance as a metric for the TEF.

5. Less concrete but not less important, an emphasis on the student experience may lead to undesirable effects on what they are taught. Institutions focussed on a high NSS score will tend to dumb down degrees or go for safe options on content. ]

Here is an example from Trinity Laban. TL exists due to an amalgamation of Trinity College of Music and Laban dance. A few years ago, the Principal introduced a compulsory two-week course called CoLab which was a creative programme involving dance and music working together. At first students were furious at being deprived of these two weeks off from the focus of their studies – and no doubt this would have been reflected in their satisfaction ratings. However time has passed; familiarity has grown; and CoLab is now one of the most popular things we offer with our students.

In addition, during the ONS review of the NSS,  respondents expressed reservations about wider issues related to the use of information from the NSS and the DLHE. Concerns included:

• limited variation between institutions of the raw scores from the student responses
• difficulty in trying to compare widely differing institutions
• difficulty in capturing the wider benefits beyond academic results of attending a higher education institution

The government has already downgraded the importance of the NSS in TEF – the so-called LSE amendment made when it was pointed out that NSS suggested that LSE and a number of other prestigious institutions were rated low by students. The metric should be subject to intense scrutiny when the House of Lords debates the Higher Education and Research Bill in committee; and I am tabling amendments to make sure it is.

The spurious precision of the NSS has the capacity to damage staff morale, to put students off certain institutions and affect the validity of the TEF. I know of at least once case where a head of department at a major institution came close to resigning because of a disappointing NSS result. I know of another where a Principal was dismissed inter alia because of the failure of that institution’s NSS results to improve. This is terrifying.

Lord Lipsey is joint Chair of the All Party Statistics Group, a Fellow of the Academy of Social Science, a Board member of Full Fact the fact checking charity, a former member of the advisory council of the National Centre for Social Research,  founder of Straight Statistics and a former adviser on opinion polling to James Callaghan as prime minister. In other words, though he may talk nonsense on many subjects, this is not one of them!