Picture the scenes last month in the offices of senior university administrators up and down the country. Tense minutes of refreshing pages and gazing into screens, staring back at their reflections and seeing their own faces set with anticipation. “What will it be? Gold, silver, or bronze?”
The results of the Teaching Excellence Framework—ostensibly a means of evaluating the quality of teaching in higher education—were being keenly awaited.And then, as the emails came through and mobiles vibrated, the reality was with us. That morning, students at my university—Durham—all received the same dismayed email about the apparently inadequate “Silver” ranking. “We plan to appeal,” it read.
The government had legitimate reasons for introducing the TEF back in 2015, when plans for it were included in a higher education green paper. Research-intensive institutions—including most Russell Group universities—are notorious for their tunnel-vision approach to research and this is alleged to have led to a neglect of teaching. The TEF should, in theory, have addressed this. There is another rationale behind an evaluation of teaching standards. Students today are set to leave university with debts of over £50,000 according to the Institute for Fiscal Studies. This has encouraged students to ask whether they are getting value for money from their courses, and so a measure designed to assess that makes sense—or so the theory runs.
The TEF has had some positive effects. For starters, it has given due credit to many post-1992 universities: former polytechnics that have over the years focused attention on teaching and consequently achieved the vaunted Gold status. Coventry and De Montfort, for example. Simultaneously, it has signalled to prestigious institutions, such as the London School of Economics—which achieved Bronze—that more focus on teaching is paramount. So in one sense, it is subverting higher education’s outdated hierarchy.
But speak to the experts and its shortcomings quickly become apparent. After chatting to academics opposed to the TEF, as well as some in favour of it, I discovered just how flawed this tool really is.
Measuring the wrong things
For starters, the metrics it rests on are flawed. One example is the controversial National Student Survey, which attempts to evaluate “student experience.” Even university leaders who have come out in support of the TEF as a whole have criticised the survey as a means of evaluating student satisfaction.Studies (such as this one) have demonstrated that student satisfaction scores are influenced by non-academic factors. Troubling as it may be, the harsh truth is that factors like the physical appearance of a lecturer influence the scores given by students. “Student satisfaction seems to be driven by the physical attractiveness of academics rather than anything else,” Chris Husbands, chair of the TEF no less, has claimed, citing US research. Student satisfaction clearly does matter, but it should not be seen as simply reflecting teaching quality when so many outside factors affect the scores.
Rates of student retention are also synthesised into the TEF; the underlying supposition being that large numbers of students dropping out reflects poor teaching. However, rates of retention have been shown to be influenced by underlying factors like students’ economic backgrounds and mental health. Similarly, an Office for Fair Access report has suggested that black students are comparatively more likely to dropout than their white and Asian peers. Again, it is difficult to isolate teaching itself as a factor.
In addition to this, the TEF takes into account employment rates for students six months after they have graduated from a given institution. This is particularly problematic. The unfortunate reality is that employment is in part determined by class, race and gender. This remains the case despite the false promises made during the era of rapid university expansion under Tony Blair, when the young were sold the promise that attendance of university was the key to success. Many students still graduate only to find themselves over-qualified for the jobs they’re working.
But the problems run deeper than all this. For the TEF’s problems lie not only in dodgy metrics. No, it is not just that the TEF measures the wrong things, but that we can’t even be sure it is measuring anything accurately at all. It is possible for universities to game the system. Consultancy firms now advertise to universities with slogans such as “Are you TEF ready?"
Joshua Forstenzer is an outspoken critic of the TEF, Vice-Chancellor's Fellow for the Public Benefit of Higher Education at the University of Sheffield and has received awards for his inspirational teaching. He told me that the negative effects of the TEF won’t be felt for years to come—the new measure being in its nascent stages. But eventually it will, in Forstenzer’s mind, push institutions to invest funds into performing well in the metrics, “but not necessarily in areas that will obviously improve the quality of teaching and ultimately learning.”
"Consultancy firms now advertise to universities with slogans such as 'Are you TEF ready?'"Graham Galbraith is a pro-Teffer and Vice Chancellor of Portsmouth University, which received a Gold ranking in the TEF2 published last month. He conceded to me that there are risks of system-gaming, citing Goodhart’s Law, which says that “when a measure becomes a target, it ceases to be a good measure.” Galbraith also argued that the aim should be to improve the metrics, by for example bringing in “benchmarking” to mitigate the non-academic factors outlined above.
Galbraith also had opinions on the narrative statement which institutions are allowed to submit to the TEF panel: having ranked the university according to measures like student satisfaction, the panel then accepts a statement from each institution outlining some context about the university.
Galbraith suggested that this should be able to act as a counterbalance to the pitfalls of the metrics. However, analysis from higher education think tank Wonkhe, as well as from the Times Higher Education, suggests that in their current form these statements are highly controversial indicators of quality. Some research-intensive institutions have been able to boost their TEF scores through these submissions, with institutions like Imperial carried over the line to Gold, while “other institutions rated silver or bronze may wonder why their appeals for leniency in problem areas were not always acknowledged to the same degree.” The issue here is clearly immensely complicated, but there is no use in factoring in contextual statements if they mislead rather than inform.
A final problem is worth exploring. The TEF is essentially a measure designed by government. What are the wider dangers of increased bureaucracy in universities, driven by Whitehall? Forstenzer hit on this note, telling me that this may in turn cause “top academics… [to] become more attracted to working in other countries.” This will surely only exacerbate recruitment and retention problems caused by Brexit, with the Guardian reporting more than a thousand academics leaving British universities in the past year.
The hunt for an alternative
But while the TEF poses serious problems for our universities—problems which could compound those caused by political issues elsewhere—what are the alternatives for ensuring that academics invest their efforts in teaching?Sally Hunt, general secretary of the University and College Union (UCU), told me that there needs to be nothing short of a shift in how universities are funded and organised, along with “staff entitlement to high-quality training, support and professional development in teaching, a promotion system that genuinely recognises and rewards good teaching as well as research” and a “push for small class sizes.”
Forstenzer floated the idea of more student involvement in the decision-making process when hiring academic staff. These thoughts were echoed by a professor at Sussex who wished to remain anonymous, who told me he “would like to see a much more inclusive consultation involving students, professional services staff and faculty about how to improve teaching.”
But more generally, both Forstenzer and Hunt outlined the need to ensure lecturers are employed on secure contracts to ensure they have the time, freedom and motivation to simultaneously pursue research and teaching.
53 per cent of universities which responded to UCU’s FOI requests have teaching staff on zero-hours contracts, while a separate survey showed that 30 per cent of these staff earn less than £1,000 a month. “For all staff except those at the top, there has been a continued erosion in the value of higher education take-home pay,” said Hunt.
Galbraith noted that the prevalence of casual contracts is in roles with a large degree of teaching, because up until the present, “university success was only measured in terms of research.” Thus, resources are shifted in this direction.
He went on to outline his view that the TEF will push university leaders to “re-direct resources to achieve good TEF outcomes,” incentivising the retention of high quality teachers. But it remains to be seen whether this will improve teaching quality per se, or just encourage institutions to game the system as outlined above.
And if a consequence of the TEF is that universities set out to please students as though they are customers, rather than actually working on improving teaching itself, they will increasingly resemble corporations.
Universities, not corporations
I recently attended a lecturer by Stefan Collini, Professor of Intellectual History at Cambridge and author of recent book Speaking of Universities. He spoke of how government papers over the years addressing higher education have gradually become rife with the language of business administration. Although Collini did not specifically criticise the TEF, the White Paper which officiated it includes the words “innovation” ninety-six times, “market” fifty times, sixty-nine occurrences of “competition,” and additionally over twenty appearances of the word “risk-based.” Language matters. And Collini argued that the language which we use to describe a service then begins to shape the lens through which we view it.If we are to reinvigorate teaching in higher education, and ensure universities are centres of critique, not extensions of the business world, then we must change the path we have set ourselves on. We must stop deluding ourselves that teaching—at its most basic, simply a discussion between two people—is something which can be quantified.
Without the TEF there can still be oversight over teaching. To me, the suggestions made by Forstenzer and Hunt offer the way forward. We must empower the voices of the academics on the frontline, rewarding them with better wages and more secure contracts, trusting them to carry out their jobs sufficiently when those conditions are met.
Maybe then we will truly have excellence across the board.
