Category Archives: TEF

Learning Lessons from School

Posted on by

Often at this time of year, I write a “review of the year” blog piece, summarising my writing over the year, and what HE and my own university in particular have gone through. That all changes this year – my university and I parted company in March (I think I’m allowed to say that) – so there is little to review from that perspective.

However, it was still a big year in HE – a new Higher Education and Research Act, the first gold silver and bronze TEF awards, and now a consultation on the role of the new OfS.

In all of these, the importance of student experience, and particularly use of metrics to demonstrate how well a university performs is paramount, ostensibly to allow prospective students to make choices, but most likely to allow league table compilers, journalists and others to make (specious?) comparisons.

So, if I can’t write much about my experiences of HE this year, what can I share? Since September I’ve spent most of my time both as a postgraduate student, and learning to teach in secondary school classrooms, and there are three lessons I can take from there that, if I were a subject lead in HE, I’d be considering.

I fully accept that university lecturers and school teachers do different jobs. But if you’re in a teaching intensive post-92 university, chances are that some of what you need to do is not that different from a teacher in a local academy or school.

Lesson One – Assessment and Feedback

Pretty much every university and course gains poor scores in the National Student Survey for this. Learning and Teaching committees agonise over it, develop complicated feedback procedures and guidelines, set minimum times to return marks (15 to 20 days), but wonder why students are still unhappy about feedback.

Maybe it’s because they compare with their school experience. Assessment happens constantly. Formative assessment (peer or self assessed maybe) in nearly every lesson. Personalised verbal and written feedback every couple of weeks. If students take a test, marks and feedback are returned within the week. This is the expectation of students – universities could think about how to develop a transition to enable them to adjust to university approaches, but equally support them in those early, vulnerable weeks.

Lesson two – Learning from Teaching Observation

Peer (or heaven forfend, management) observation of teaching in universities is, at its best, a collaborative experience, sharing good practices and relying on professional approaches to self development. However for many this is based on being observed for possibly one hour a year and maybe observing others for maybe two or three hours.

New lecturers may be observed a little more in their first year as part of any post graduate certificate they are taking.

Contrast this with the school experience. As a trainee teacher I am currently observed teaching for 10 hours a week. Every session that I deliver. And at the end of each session, there is a four page feedback form where I scored against 8 standards, with 3 levels of competence. Advice is provided on what worked well, and what I can do to improve. Targets are set for the next week’s season, and I have to provide my own reflection. Guess what – my teaching improves each week, and I develop new techniques and new ideas from the advice I get. I also go and observe a range of other people whenever I want to.

If we want to make big changes in teaching practice, and expose people to a greater number of great teachers and different ideas, then the annual observation round leaves much to be desired, as does the way in which new lecturers are suppported.

Lesson 3 – Know your Students

It’s well now that a sense of belonging aids student retention and attainment, as evidenced in projects such as the Paul Hamlyn/HEA What Works project, as well as universities’ focus on developing course identity.

I talked to a university undergraduate course leader the other week, with maybe 30-40 students in each year. They said that they did not know the names of all their final year students, let alone the others. In school, I’m expected to know the names – and use them – of all my pupils. So in a class of over 30 11 year olds, I can talk to each of them individually, and know a little about them and their abilities. And importantly, I should be able to do this within a couple of weeks of meeting them.

If this is what pupils are used to in school, it’s maybe not surprising that they don’t feel a sense of belonging when they first come to university.

Conclusion

So what’s to be done?

Firstly, I don’t think universities need to replicate schools: the two sectors have different functions, cultures and behaviours. However, a really hard look at transition to university, again particularly in teaching focussed universities whose students may lack some of the cultural capital to thrive instantly, could provide a way of maximising student engagement and attainment.

Seriously, why not send your teaching staff to spend two weeks in a secondary school, shadowing a teacher, and seeing what the school experience is these days – and then reflect on this to see if you could develop your first semester, in terms of formative assessment, teaching practice and sense of belonging to really help your students.

TEF – the finish line is in sight

Posted on by

The finish line is now in sight, across the country policy wonks and planners are finessing their submissions for the Teaching Excellence Framework.

I’ve previously written for MediaFHE on the decision to rank providers as gold silver or bronze, and how this system could be seen to be flawed.

rankings-gold-silver-and-bronze

More recently an interesting article was published this week by Gordon McKenzie, CEO of GuildHE, who questioned the amount of predestination vs fee will in TEF.

“..this may just be the logical consequence of metrics that may be the best we have but are not a perfect proxy for teaching excellence; if the measure is inherently vulnerable then the narrative has to concentrate on shoring it up. But it is also a bit of a shame. While the specification does touch on examples of the rich activity that makes for an excellent learning environment and the highest quality teaching, I fear this richness will get squeezed out of the 15 pages to which submissions are limited and will fall victim to the need to feed the metrics. The structure of any performance assessment framework tends to shape the responses and behaviour of those being assessed. As teachers teach to the test, so providers will submit to the metrics.”

Looking at the assessment process, then the implication is that the metrics being used  – National Student Survey, DLHE and non-continuation rates, with evidence of how these are split based on student demographics – are going to be the primary determinant of a provider’s TEF outcome. The updated guidance from HEFCE (originally published in September and updated this week reinforces this:

Looking into the scoring process (section 7.10 and 7.11), then we learn that:

“A provider with three or more positive flags (either + or ++) and no negative flags (either – or – – ) should be considered initially as Gold.

A provider with two or more negative flags should be considered initially as Bronze, regardless of the number of positive flags. Given the focus of the TEF on excellence above the baseline, it would not be reasonable to assign an initial rating above Bronze to a provider that is below benchmark in two or more areas.

All other providers, including those with no flags at all, should be considered initially as Silver.

In all cases, the initial hypothesis will be subject to greater scrutiny and in the next steps, and may change in the light of additional evidence. This is particularly so for providers that have a mix of positive and negative flags.”

All providers received their illustrative metrics back in July 2016, with the note that the final versions would not vary significantly. Indeed, looking at the actual data provided this week, we can see that there has been minimal change.

So it’s like a great game of poker – no-one is revealing their hand, or saying yet how they will approach the written submission, but knowing how the metrics will heavily influence the initial assessment of gold, silver or bronze, most providers already have a pretty good idea of their likely outcome

For those providers who have the most clear-cut metrics – the gold and the bronze award winners, the results would seem to be predestined. With seemingly little opportunity for contextual explanations to change the decision of the TEF assessors, then those providers will be able to say now what they expect to score in TEF. They’ll also know in which areas they would need to improve in future, or which groups of students they might need to focus on. Those who have a mixture of good metrics and no significance flags, and perhaps only one poor score will be able to create a narrative for a silver award.

One thing we should welcome is the emphasis on different groups of students in the split metrics – use of these figures and the possible impact on the TEF rating that a university might achieve based on poor experience or outcomes for students from WP backgrounds or non-white ethnicities might act as a nudge to push the social mobility agenda that universities can influence.

It’s also worth noting in the guidance a comment on NSS scores in 7.21b

“Assessors should be careful not to overweight information coming from the NSS, which provides three separate metrics in two out of three aspects, and ensure that positive performance on these metrics is triangulated against performance against the other metrics and additional evidence. They should also bear in mind that it has been suggested that, in some cases, stretching and rigorous course design, standards and assessment (features of criterion TQ326), could adversely affect NSS scores.”

Heaven forfend that one of our “top” universities fails to do well because of a poor score for student experience.

And finally on outcomes (section 7.32):

“Should a provider include very little additional evidence in its submission, proportionately more weight will be placed on the core and split metrics in making decisions. In the extreme case where a provider submission contains no substantive additional evidence, assessors will be required to make a judgement based on the core and split metrics alone, according to the following rules:

Five or six positive flags in the core metrics for the mode of delivery in which it teaches the most students and no negative flags in either mode of delivery or split metrics confers a rating of Gold.

No flags, one, two, three or four positive flags in the core metrics for the mode of delivery in which it teaches the most students and no negative flags in either mode of delivery or split metrics confers a rating of Silver.

Any negative flags in either mode of delivery for any core or split metric confers a rating of Bronze.

If your own assessment of your score is that you pass the threshold for satisfactory quality, but that you have too many poor scores, then why would you put too much effort into the written submission – you’re going to get a bronze award anyway.

And I still don’t think that’s how medals are awarded.