It’s been almost a week since TEF results were made public, and some of the predictable coverage, posturing, and agonising have occurred. Here are a few of my thoughts.

The importance of the written submission

In advance, we were all told that the written submission mattered, but at the same time, that the initial hypothesis that would be based purely on metrics was felt to be the factor that would determine classification. Looking at the result then plenty of universities have been awarded a TEF rating higher than their initial metrics would suggest. (This is personally pleasing since I wrote a significant amount of my previous employer’s submisison). The commentary provided by the TEF panel on each submission makes it clear that a written submission that demonstrated that an institution understood why it it missed benchmarks, could explain this in terms of contextual data, and show that activity was taking place to remediate the situation, then the higher award was possible.

The press didn’t understand what was being measured.

In advance of publication I was asked on Twitter whether anyone outside the sector was going to be interested in the results. Inevitably those papers who have a vested interest (by publishing their own university guides) or who have a reputation for being a TEF booster ( I’m looking at you here, The Times), were always going to publish something.

We inevitably saw articles reminding us that Southampton, LSE and Liverpool of the Russell Group had not performed as expected, and this this showed the shake up in the sector. Equally, there was criticism that the expected ranking or established order was not being replicated.

Any paper that publishes its own league table is going to be concerned if another form of ranking does not tally with their figures. But this is to misunderstand what TEF is – it’s about measures against benchmarks, not absolute performance, hence the difficulty for some unis in scoring above already high benchmarks, and for the press to create a simple story from a more complex narrative.

Universities love to celebrate

There was plenty of gold across those who felt they’d done well! This despite the rumblings and complaints in advance that the idea of three levels of ranking, like medals, was reductive and couldn’t possibly communicate the complexity of what a university does

How much does it matter to the sector?

TEF clearly matters to those in the sector, and will have implications for behaviours in the future. Universities already work hard to make sure that they optimise their data returns to HESA, that they get good scores in NSS by promoting and managing survey completion, and getting good scores on DLHE by managing those returns.

In future, these activities might drive performance management behaviours in universities even more than at present, with possible unforeseen consequences – courses and subject areas that perform poorly on a key metric may not longer be considered as viable, especially while TEF continues to be at institutional level.

For planning departments, then we can expect to see ever more sophisticated models of academic portfolio performance, and increased scrutiny of data returns.

((From the Modern Toss Work postcard set: http://ow.ly/hFV530cT60U )

The impact on fees has been temporarily removed, and with possible changes to funding in future (let’s face it, HE funding is back on the agenda after the recent General Election), then TEF as an instrument of marketisation through differential fees loses its power.

How much does it matter to the press?

For those in the press, TEF might just be a way to get easy headlines about perceived poor performance of established universities, while expressing shock at the performance of some FE colleges.

For the specialist press, commentariat and twitterati, TEF is a gift – something for the wonks to pore over and luxuriate in, in that quiet period at the end of an academic year.

How much does it matter to the punters?

For parents and potential students, TEF is just one more set of information to use, and has to be added to existing marketing collateral, multiple league tables, and guidance from schools and colleges. Without a clear explanation of what i being measured (particularly the issue of relative performance rather than absolute) then it’s not a straightforward measure, but just one more to add to the mix. Coupled with the Guardian University Guide concept of “value added” then it’s hardly surprising that potential students aren’t always clear about what might be on offer.

Finally, TEF may just be ignored if it does not provide the confirmation bias that people often use on making these kind of decisions. For example, I have a son who wants to study History in a year’s time. Both Staffordshire and Durham scored Silver. But I’m only going to recommend one of those.

You can bet though, that universities will shout about their TEF outcome (provided it was good) at this summer’s open days.

My previous blog has now been migrated to this site (actually this has been a mirror site for quite a long time, but never used beyond that).

It’s time to start writing new content, and building up stats again – but just to show how much the old blog was read, here’s a peek at the final usage stats:

And that “best ever” – that was on the day I wrote about the Guardian University Guide in 2014. In fact most of the high traffic posts have been about league tables, although one of the latest posts on “Does UK HE have a Retention Problem” has been pretty popular.

After a quiet time on the wonk front, last week saw the publication of the White Paper and two new reports on employability of STEM graduates, announcement of a Higher education bill in the Queen’s speech, and the launch of the technical consultation on the Teaching Excellence Framework, not forgetting the previous week’s plans for consulting on the future of DLHE. Anyone would think that HE wonks had been twiddling their thumbs for a while, with nothing to critique or criticise. For a really good set of resources on this, it is worth looking at WonkHE.

The White Paper contained few real surprises – changes to quality arrangements, making it easier for new entrants to the market, the introduction of a teaching excellence framework, changes to the landscape and research support – all were previously consulted on in the previous Green Paper. Overall, the sector has not been unreservedly supportive, but even with a small parliamentary majority, the bill is likely to become law, and so we need to learn how we can work as well as possible within this revised landscape.

Overall, the changes are to drive further the marketisation of higher education – no matter how we might suggest that HE does not operate as a fully open market, the government is wedded to the idea that increasing competition will drive up quality. Hence, the idea that new entrants  – “challenger” institutions will be able to provide competition to existing incumbents. Similarly, the teaching excellence framework is touted as providing more information to prospective students, hence helping them to make more informed decisions. There is, of course, little evidence that students make decisions purely on data, and for many students, there may not be a free choice of where they study, based on financial circumstances, and family or work commitments.

Nonetheless, we will have a TEF, and so it’s important to understand what will drive success in this, so that we can get the best possible outcome which reflects our performance. One piece of good news is that the government did listen to the sector in terms of timing of implementation, even if concerns about the metrics to be used fell upon stony ground.

From the technical consultation, we know that the following principles should underpin TEF:

• keep bureaucracy and burden to a minimum
• be voluntary, allowing providers to consider the benefits and costs of applying before deciding whether or not they wish to
• allow for diverse forms of excellence to be identified and recognised
• support rather than constrain creativity and innovation
• respect institutional autonomy
• be based on peer assessment
• be robust and transparent
• result in clear judgements about excellence for students, employers and other stakeholders
• avoid driving perverse or unintended behaviours in the pursuit of demonstrating excellence
• be sufficiently flexible to allow for further development as the TEF evolves.

From year 2 of TEF, institutions who choose to be assessed can be judged to meet one of three outcomes: Meets Expectations, Excellent or Outstanding. To get to this, we would be assessed on: teaching quality, learning environment, student outcomes and learning gain.

And the part we need to be mindful of is how this will be assessed.

Teaching quality will be based on questions 1- 9 of the National Student Survey (teaching and assessment and feedback). Learning environment will be judged on questions 10-12 of the NSS (academic support) and non-continuation data from HESA, while outcomes will be assessed by the results of DLHE.

This does look remarkably like a league table, and so institutions will work harder than ever to make sure that their NSS results and DLHE figures show outcomes in the best possible light.

In addition to the data, providers will provide a written submission of no more than 15 pages. This is where we will be able to provide more context to what we do – examples cited in the document discuss: use of student surveys, collecting and responding to module feedback, staff development activities, timeliness of feedback, use of employers on validation panels, levels of contact time and independent study.

This is going to be a lot to cover in 15 pages, so it will be key for institutions to have their policies really clearly defined in terms of how their various mechanisms work, and how they can be shown to improve student experience and outcomes.

Our recent work on changing module evaluation processes and observation of teaching, and our review of quality processes will put us in a good position to explain how we manage our academic delivery to provide the best experience for students. We will clearly need to focus more on some of our student survey scores, and get to the bottom of why we have such a wide variety of reported experiences.

Next steps for us will be:  how we review our student survey outcomes; how we deliver our new employability strategy; how we ensure that we use the information from module evaluations and teaching observations to optimise student success, and how we review the performances of all of our courses.

There will no doubt be an ongoing resistance to TEF – the metrics chosen are still not ideal, and when we move to looking at subject level analysis, then there will be concerns regarding reliability of data – but this is a system we are going to have to work with. It would make sense to make sure we are best prepared as we can be.

The first of the major University league tables, the Complete University Guide, is published today.

This table uses metrics  on ten measures: Student Satisfaction, Research Quality, Research Intensity, Entry Standards, Student: Staff Ratio; Spending on Academic Services; Spending on Student Facilities; Good Honours degrees achieved; Graduate Prospects and Completion.

From the CUG press release:

Dr Bernard Kingston, principal author of TheCompleteUniversityGuide.co.uk, said: “There is a considerable degree of stability at the upper end of the league table this year. While dramatic changes may be newsworthy, this stability indicates that the rankings are robust and credible for young people seeking a university place – our primary purpose.”
This year’s release sees TheCompleteUniversityGuide.co.uk publish a number of new rankings. Alongside new subject tables for Creative Writing, Forensic Science and Occupational Therapy, there is now a Creative and Performance Arts table, containing 14 institutions that do not feature in the Main Table.
Dr Kingston said: “We have simultaneously released a survey of universities’ relative success in resolving student complaints. This shows significant variations between universities and is an important source of information for prospective students who what to know that their complaints will be effectively resolved.” (See attached Press Release and Table).
“Last year’s Higher Education Green Paper, Higher education: teaching excellence, social mobility and student choice, stated that applicants need access to robust, timely and objective information, based on criteria that are straightforward and easily understood.

So the top ten are:

Not really any surprises there. Staffordshire falls 6 places to 109th.

What is always of more interest are the big movers, both up and down, and the identification through reading the individual subject tables to see why these changes have happened.

So this year’s big winners are:

• Manchester Met – up 16
• Harper Adams – up 14
• Buckingham – up 14
• Liverpool Hope – up 14
• Sunderland – up 14
• Falmouth – up 12
• Winchester – up 12
• Edge Hill – up 11
• Middlesex – up 11

At the other end we have

• Oxford Brookes – down 11
• St Mark and St John – down 12
• Brighton – down 14
• Queen Margaret – down 15
• Royal Agricultural University – down 17
• Arts University Bournemouth – down 19

The section on complaints and their resolution will be of interest to academic registrars. Over a 3 year period, the number of completion of procedure letters issued after exhausting internal complaints process per 1000 students is ranked. The ranking here does not follow any meaningful pattern – it might be assumed that students at one type of university are more likely to use a complaints procedure than others, or that part of the sector would be better at dealing with complaints but this is clearly not the case. Pleasingly, the figure quoted for Staffordshire is considerably better than for some. Whether this data is of any meaningful us to prospective students is debatable.