The finish line is now in sight, across the country policy wonks and planners are finessing their submissions for the Teaching Excellence Framework.

I’ve previously written for MediaFHE on the decision to rank providers as gold silver or bronze, and how this system could be seen to be flawed.

More recently an interesting article was published this week by Gordon McKenzie, CEO of GuildHE, who questioned the amount of predestination vs fee will in TEF.

“..this may just be the logical consequence of metrics that may be the best we have but are not a perfect proxy for teaching excellence; if the measure is inherently vulnerable then the narrative has to concentrate on shoring it up. But it is also a bit of a shame. While the specification does touch on examples of the rich activity that makes for an excellent learning environment and the highest quality teaching, I fear this richness will get squeezed out of the 15 pages to which submissions are limited and will fall victim to the need to feed the metrics. The structure of any performance assessment framework tends to shape the responses and behaviour of those being assessed. As teachers teach to the test, so providers will submit to the metrics.”

Looking at the assessment process, then the implication is that the metrics being used  – National Student Survey, DLHE and non-continuation rates, with evidence of how these are split based on student demographics – are going to be the primary determinant of a provider’s TEF outcome. The updated guidance from HEFCE (originally published in September and updated this week reinforces this:

Looking into the scoring process (section 7.10 and 7.11), then we learn that:

“A provider with three or more positive flags (either + or ++) and no negative flags (either – or – – ) should be considered initially as Gold.

A provider with two or more negative flags should be considered initially as Bronze, regardless of the number of positive flags. Given the focus of the TEF on excellence above the baseline, it would not be reasonable to assign an initial rating above Bronze to a provider that is below benchmark in two or more areas.

All other providers, including those with no flags at all, should be considered initially as Silver.

In all cases, the initial hypothesis will be subject to greater scrutiny and in the next steps, and may change in the light of additional evidence. This is particularly so for providers that have a mix of positive and negative flags.”

All providers received their illustrative metrics back in July 2016, with the note that the final versions would not vary significantly. Indeed, looking at the actual data provided this week, we can see that there has been minimal change.

So it’s like a great game of poker – no-one is revealing their hand, or saying yet how they will approach the written submission, but knowing how the metrics will heavily influence the initial assessment of gold, silver or bronze, most providers already have a pretty good idea of their likely outcome

For those providers who have the most clear-cut metrics – the gold and the bronze award winners, the results would seem to be predestined. With seemingly little opportunity for contextual explanations to change the decision of the TEF assessors, then those providers will be able to say now what they expect to score in TEF. They’ll also know in which areas they would need to improve in future, or which groups of students they might need to focus on. Those who have a mixture of good metrics and no significance flags, and perhaps only one poor score will be able to create a narrative for a silver award.

One thing we should welcome is the emphasis on different groups of students in the split metrics – use of these figures and the possible impact on the TEF rating that a university might achieve based on poor experience or outcomes for students from WP backgrounds or non-white ethnicities might act as a nudge to push the social mobility agenda that universities can influence.

It’s also worth noting in the guidance a comment on NSS scores in 7.21b

“Assessors should be careful not to overweight information coming from the NSS, which provides three separate metrics in two out of three aspects, and ensure that positive performance on these metrics is triangulated against performance against the other metrics and additional evidence. They should also bear in mind that it has been suggested that, in some cases, stretching and rigorous course design, standards and assessment (features of criterion TQ326), could adversely affect NSS scores.”

Heaven forfend that one of our “top” universities fails to do well because of a poor score for student experience.

And finally on outcomes (section 7.32):

“Should a provider include very little additional evidence in its submission, proportionately more weight will be placed on the core and split metrics in making decisions. In the extreme case where a provider submission contains no substantive additional evidence, assessors will be required to make a judgement based on the core and split metrics alone, according to the following rules:

Five or six positive flags in the core metrics for the mode of delivery in which it teaches the most students and no negative flags in either mode of delivery or split metrics confers a rating of Gold.

No flags, one, two, three or four positive flags in the core metrics for the mode of delivery in which it teaches the most students and no negative flags in either mode of delivery or split metrics confers a rating of Silver.

Any negative flags in either mode of delivery for any core or split metric confers a rating of Bronze.

If your own assessment of your score is that you pass the threshold for satisfactory quality, but that you have too many poor scores, then why would you put too much effort into the written submission – you’re going to get a bronze award anyway.

And I still don’t think that’s how medals are awarded.