Don’t trust your gut: a little bit more on the problem with grading lessons

This evening, there will be debate on the role lesson observation in England’s schools with such educational luminaries as Professor Robert Coe, David Weston (the man behind the Teacher Development Trust), Lead Ofsted inspector Mary Myatt, Sam Freedman (Director of Teach First and ex-special advisor to Gove),, Dame Alison Peacock (Headteacher of The Wroxham School) and, er… me. Quite what qualifies me to participate beyond having a big gob and a stubborn streak a mile wide I’m not sure. However, I’m pretty damn excited to have been asked and, despite suffering with an appallingly debilitating cold, am sure it will be an excellent event. You can, I’m assured, watch it live on the net even if you can’t be there in person.

Now some background. In his post, Classroom observation: it’s harder than you think, Professor Coe lays out 5 very sensible reasons why we are so prone to misjudging lessons:

  1. Observation produces a strong emotional response – if we like a thing it must be good, if we don’t it’s obviously rubbish
  2. Learning is invisible – we rely on (often very poor) proxies to make judgements about whether pupils are learning. And, worryingly, good performance is not necessarily a sound indicator of learning.
  3. Accepted ‘good practice’ may be more fashionable than effective – we are often victims of the ‘cult of the new’. Just because a teacher is doing something pedagogically interesting doesn’t mean it’s effective.
  4. We assume that if you can do it you can spot it – classroom observation is a very different skill teaching (although there are of course overlaps.) Grading lessons based on pupils’ progress is problematic because it takes an almost Herculean effort of will to concentrate on the pupils and not be distracted by the teacher.
  5. We don’t believe observation can miss so much – if we focus on one area, it is almost a guarantee that we will miss other stuff. We’re tremendously poor at predicting what we will be able to do and are often victims of our intuition. Very often the facts are counter intuitive. A useful rule of thumb, in all areas of life, might be to assume that what ‘feels right’ is probably wrong.

Now all this is, I hope, so clear and persuasive as to make you at least doubt your ability to accurately judge a lesson using something so vague and poorly designed as Ofsted’s criteria for assessing teaching and learning. Frankly, if you’re not doubting your ability at this point then I can only assume that you’re dangerously incompetent!

But let’s assume that you’re only beginning to doubt and, as yet, have no plans to abandon judgmental lesson observations. Well, consider this: there is a pretty compelling body of research that suggests that good performance might actually be preventing learning! In a scarily comprehensive review of research into learning and performance over the past century, Bjork and Soderstrom have amassed enough material to at least cast serious doubt on the idea that students’ current performance will in any way correlate with the likelihood that learning will be retained and/or transferred to other domains.

For instance:

[E]arly research demonstrated that learning could occur without changes in performance. More recently, the converse has also been shown—specifically, that improvements in performance can fail to yield significant learning. In fact, numerous experiments in the domains of perceptual-motor learning and verbal-conceptual learning have shown certain manipulations—including distributing practice, varying the conditions of practice, reducing feedback, and testing/generation—to have opposite effects on learning and performance: Conditions that induce the most errors during acquisition are often the very conditions that lead to the most learning! Furthermore, that performance is often fleeting and, consequently, a highly imperfect index of learning does not appear to be appreciated by learners or instructors who frequently misinterpret short-term performance as a guide to long-term learning. (p 2)

And

The early experiments on latent learning and overlearning suggested that learning can occur with no discernible changes in performance. More recently, the converse has also been
demonstrated—namely, that performance gains during training can impede post-training learning compared to those conditions that induce more errors during performance. This dissociation has been demonstrated by manipulating the practice or study schedules of to-be-learned skills or information. (p 13 my emphasis.)

And

Mannes and Kintsch 1987 finds that providing outlines organized differently from a studied text passage promoted a deeper understanding of that passage compared to when the outline was organized similarly to the passage. Furthermore, increasing the variation of problems during an acquisition phase enhances analogical reasoning(Gick and Holyoak 1983), geometrical problem-solving (Paas and Van Merrienboer 1994), and complex troubleshooting (Van Merrienboer and de Croock 1997) of novel problems, as well as one’s ability to solve previously encountered anagrams (Goode, et al. 2008). (p 21)

This research also cast doubts on some of our darlings. The role of feedback has been much documented and there is universal consensus that more is better. Or is there?

Numerous studies—some of them dating back decades—have shown that frequent and immediate feedback can, contrary to intuition, degrade learning. (p 23)

The problem is this:

Although the learning-performance distinction is overwhelming supported by empirical
evidence, there appears to be a lack of understanding on the part of instructors and learners alike
that current performance is a highly imperfect index of long-term learning; consequently, how
we learn is often vastly misaligned with our metacognitive assessments of how we think we learn. (p 33)

I know I’m not the only teacher to have taught an ‘outstanding’ lesson where pupils performance has been exhilarating only to discover that they’ve forgotten it all next lesson. And I also know I’m not alone in having presided over watch seems a car crash only for student to come back having apparently learned what I wanted them to learn.

Despite the fact that we’re often mistaken in our beliefs about what is or will be effective, it’s much more comfortable and much easier to carry on doing what ‘feels right’. In lots of areas this doesn’t really matter but when it comes to judging teachers’ careers based on ignorance and misinformation it’s quite another matter. How would, for instance, Tristram Hunt’s pie-eyed proposals on teacher MOTs fit into this? Would those given the right to ‘strike off’ teachers be trained in the nuances of classroom observation and have read and understood the staggering weight of evidence documenting the differences between learning and performance? Would they?

For now, let me leave you with these suggestions:

  1. Understand the distinction between learning and performance and, for God’s sake, tell others about it!
  2. Assume that you don’t know, especially when judging others. Don’t trust your gut!

Related posts

Has lesson observation become the new Brain Gym?
Where lesson observations go wrong
Get ahead of the curve: stop grading lessons

17 Responses to Don’t trust your gut: a little bit more on the problem with grading lessons

    • Amy says:

      I’m so glad I discovered this website this evening. All of the issues raised are matters that have been racing through my mind recently; it’s so reassuring to know that other like minded professionals are having the same thoughts as me about the state of education in our secondary schools.
      I particular like the above article as classroom observation practice is an area that baffles – and infuriates me – immensely. The idea of grading lessons using a ‘tick box’ approach is completely ludicrous. I just don’t see how this actually measures learning. My colleagues and I often joke about the way lessons are judged in our schools…every school? For example, if the students are made to ‘get into groups’, or hold a white board up in the air on a regular basis, then you are more likely to be judged as ‘outstanding’ purely because that’s what Ofsted criteria classifies as ‘independent learning’. Whether learning is taking place doesn’t seem to matter as long as it ‘looks’ good. However, if they are sat at their desks actually doing quiet, focused work on their own, then that can never be more than ‘satisfactory’ because Ofsted said so! Don’t get me wrong, I do have time for using various learning methods in the classroom – providing they actual assist learning and aren’t just used as a gimmick to bag Ofsted points!
      When will people begin to see that teaching and learning is much more than individual, compartmentalised episodes of learning – it’s much more holistic than that, isn’t it? We’re not robots, the students aren’t robots! However, maybe the person who invented the lesson observation criteria is a robot???

  1. Ian Lynch says:

    Questioning the validity of lesson observation and other judgements that might affect a teacher’s career is no bad thing, but are we not also guilty of going with a gut feeling in this? Aren’t we making the same sort of judgements about children all the time without worrying too much about the effect on their psychological well being never mind their careers?

  2. […] @LearningSpy probably provided the sensationalist quote of the evening. He presents some of his thoughts in a pre-seminar blog here […]

  3. nancy says:

    Thank you for this. I often wonder if children learn in spite of us…and that the most effective learning happens when we can respond to them, and find out what they actually want to know!

  4. Ian Lynch says:

    Motivation is a key part of learning. Good teachers that know their stuff clearly make a significant difference but with sufficient motivation it is possible to compensate for poor teaching. And of course people learn a lot of useful things outside school too.

    • David Didau says:

      Of course. I completely agree. Recently read this little post on motivation which neatly confirms my bias http://www.tes.co.uk/article.aspx?storycode=6382668

      • Ian Lynch says:

        I think it is one of the things that gets confused with preferred “learning styles”. Probably what is observed is preferred learning context which drives motivation more than a less preferred context. This would explain why people choose particular fields of study and reject others. The preferred field generates pleasant feelings and emotions and therefore motivates us to study harder, remember more and learn more. This is more relates to the emotional side of the brain than the cognitive. If you believe you are not good at a subject or method or that it is irrelevant to you, you simply aren’t motivated to learn it. A teacher can make the difference by giving confidence or whipping up enthusiasm. Incentives like getting a good job might work too. But demonstrating to someone at an early stage that they really are no good at something and reinforcing it time after time is probably going to result in avoidance unless something else compensates. Probably explains the polarisation in maths between the good mathematicians and the “mathphobics” because it is so easy to define performance with apparently absolute precision. Plus society often tolerates poor maths as a badge of honour, whereas poor literacy is the badge of poor class.

  5. This is excellent stuff and I couldn’t agree more. Observers also have agendas, personal friends, political consideration, quotas to meet and a host of other non pedagogic considerations when observing teaching.

    In terms of motivation I don’t think we have progressed beyond Vygotsky’s constructivism. Other will disagree no doubt

    One problem of an over prescriptive curriculum is that it limits the opportunity to find out what motivates a particular learner and go with it. On the other hand, if the curriculum become subject to teacher whim then it can mean that knowledge becomes too diverse and it impedes conceptual development.

    Anyway waffle waffle, this is good stuff I will use it in my ongoing negotiations with management about next years observations.

    Thanks

  6. […] David Didau talks about this in his latest blog: Don’t trust your gut. […]

  7. […] Don’t trust your gut: a little bit more o the problem with grading lessons Deliberately difficult: why it might be better to make learning harder […]

  8. […] Why can’t we tell a good teacher through lesson observations? Where lesson observations go wrong Don’t trust your gut: a little bit more on the problem with grading lessons How can we make classroom observation more effective? The Cult of Outstanding™: the problem with […]

  9. […] more often than we’d care to admit. (David Didau has written eloquently about it here and here, and I’d strongly advise everyone to read those […]

  10. […] Don’t trust your gut: a little bit more on the problem with grading lessons […]

Constructive feedback is always appreciated

%d bloggers like this: