“Works for me!” The problem with teachers’ judgement

nas-01

It is with our judgments as with our watches: no two go just alike, yet each believes his own.

Alexander Pope

One of the difficulties inherent in challenging teachers’ judgments is that when those judgements appear to be contradicted teachers sometimes say, “Well, it works for me and my students.” This is hard to challenge.

Anthony Radice made a similar point in a recent blog post about the debilitating nature of complacent certainty:

A clear example of this kind of complacency is contained in the words, ‘I know my pupils’. It’s the killer punch to an argument, because it is not falsifiable. There is no definitive evidence that can be presented to refute this statement. One can debate endlessly over the evidence for this approach or that, only to have the whole discussion closed down with these four short words. The implication tends to be that ‘you can pontificate all you like about cognitive science; you can use any logical argument you like; but I possess knowledge which supports my approach, knowledge to which you evidently have no access, you cold-hearted intellectual, you!’

Falsifiability is important because it’s the backbone of progress in science. A scientist states a theory, describes the evidence which supports that theory and then other scientists are able to test the theory and replicate the experiment looking for possible flaws. This is not how, say, religion works. Religious belief relies on faith. When faith is challenged by evidence, it’s possible to say things like, “God works in mysterious ways.” It’s not possible to disprove such a statement because it’s not falsifiable. How could you begin to attempt to show that God doesn’t work in mysterious ways? if a statement or a theory is true in all possible circumstances then it’s unfalsifiable: you can’t disprove it. If we can’t disprove something then no amount of evidence or reason could possibly sway us from our entrenched positions because those positions are not based on evidence or reason but are instead based on faith.

If, in the face of contradictory evidence, we make the claim that a particular practice ‘works for me and my students’, then we are in danger of adopting an unfalsifiable position. We are free to define ‘works’ however we please. If we’re told that students’ exam results might improve if we changed our practice we can say things like, “There’s more to education than exam results” and claim that our students are happier, better rounded, or have an excess of some other vague, unmeasurable trait. We can laugh at the idea of measurement and say, “Just because you can’t measure it, doesn’t mean it isn’t important.” We can insulate ourselves from logic and reason and instead trust to faith that we know what’s best for our students and who can prove us wrong?

The thing is, this sort of faith results in stagnation. Science has made tremendous leaps and bounds over the past thousand or so years because scientists learn from their mistakes. Religions have stayed pretty much the same (much to the satisfaction of their adherents as their beliefs represent eternal Truth.) If we as teachers adopt a position of faith, we cannot learn or improve. We cannot receive meaningful feedback about our mistakes if our response is to shift the parameters and respond with, “Yeah, but”. By testing our ideas and accepting where we have made mistakes, we learn.

As Daniel Kahneman and Gary Klein point out in Conditions for Intuitive Expertise: A Failure to Disagree, there are some professions – psychotherapy, clinical psychology, recruitment, radiology, stock broking, the judiciary – where practice and experience doesn’t seem to improve judgement. What all these professions have in common is that it’s very hard to learn from mistakes. If you work in recruitment it’s unlikely that you’ll track the long-term success of the personnel you recruit so you never know when your judgement was successful and when it was not. A psychotherapist takes feedback from the apparent progress of patients in the clinic but has no mechanism for knowing how they behave in the real world. All these professions rely on their subjective judgement without getting meaningful feedback on how successful their judgements might be. Teaching is, perhaps, a little like this. We get instant and meaningful feedback on some aspects of the job; if our behaviour management isn’t up to snuff, students are unambiguous in their feedback. But how do we know if our ability to get students to retain and transfer new skills and knowledge is good enough? Most of the time we rely on our subjective judgement and certain visible proxies such as whether children are working hard, doing what they’re told and performing well in the classroom. It’s easy to look with satisfaction at a sea of happy, contented faces and conclude, “Well, it works for me.” As Kahneman and Klein put it, “human judgments are noisy to an extent that substantially impairs their validity.”

There are two main barriers to teacher improvement. One is that we often fail to notice whether our ability to teach is any good. The other is the way we are held to account. We are asked to justify and explain why students failed to make the grade; we are under pressure is to make excuses and conceal mistakes to avoid being blamed. Instead of admitting that what we’re doing doesn’t appear to be effective we shrug and say, “These things happen” and “What can you expect with kids like these?” Sometimes we blame specification changes or marking protocols: It must be the fault of the exam board, Ofqual, the DfE. I’ve heard failure blamed on timetabling, sports fixtures, room temperature and litany of equally plausible excuses. My favourite excuse is one I wrote about in my recent book:

Once in an exam analysis meeting, a school leader who taught in a particular department said that the reasons the exam results of that department were so poor was because of their outstanding teaching. They concentrated on independent learning and refused to ‘spoon feed’. This obviously meant kids did less well in the test.

If we want to improve as teachers we have to acknowledge our errors. If school leaders want teachers to acknowledge their errors they must create a culture where this is safe and normal. One positive change to school culture would be to change the norm from using evidence to confirm our prejudices to using it explore how we might think and act differently. It still startles me how many people read this post and wanted to argue their judgements were correct, no matter the evidence.

If you want to object to what I’ve written, fine. But please, is it too much to ask that you think first?

47 Responses to “Works for me!” The problem with teachers’ judgement

  1. Love the idea of a school culture where error acknowledgement is safe and normal.

    In the EYFS for example, how do we tell the difference between ‘using evidence to confirm our prejudices’ and using day to day observational evidence to inform judgements about a child? Apart from testing them.

    • David Didau says:

      Not being an expert in the area of EYFS I couldn’t advise you in terms of specifics but my suggestion would be to look at what the research says about the best ways to go about doing what you do. Best of luck

  2. julietgreen says:

    Which is why the culture of leaving it to individual teachers and schools to make up their own systems is inherently flawed. There must be some systems that have been shown to work better than others. I’m not able to get this across in my current twitter discussion with @HarfordSean who suggests that I’m being ‘reductionist’ to suggest that we should not all be inventing our own wheels, that we have now got the autonomy we asked for and that professional teachers will grasp this opportunity. It is problematic asking schools to invent an assessment system that ‘works for them’. It wastes massive amounts of time and it penalises the pupils who are in the systems that don’t particularly work.

    • David Didau says:

      Interesting conundrum. I think sean is right to say that Ofsted should not impose methodologies on schools – that’s not and shouldn’t be their role. That’s what they were doing up until Wilshaw’s tenure and it was disastrous. That said, schools should be held to account based on what the evidence tells us about what works. If they’re just claiming what they do ‘works for them’ with no validation, they should be penalised.

      • julietgreen says:

        And I agree that it is not Ofsted’s role. But they should also be more definitive, given that many school leaders are now tying themselves up in knots trying to work out what this should be. I acknowledge that the guidance proscribes this, but that guidance is ignored.

  3. I wonder whether Pope’s point might have been a jab at arrogant modern man with his overweening confidence in his ability to make objective independent judgements, and his consequent rejection of tradition. One of the key results of understanding just how shaky our own judgement is, should be that we seek to learn from those who are wiser, which might just include the great thinkers of the past.

  4. […] It is with our judgments as with our watches: no two go just alike, yet each believes his own. Alexander Pope One of the difficulties inherent in challenging teachers’ judgments is that when those judgements appear to be contradicted teachers sometimes say, “Well, it works for me and my students.” This is hard to challenge.  […]

  5. David Williams says:

    Hugely thought provoking blog. Really, really good. I wonder if the first problem is in fact the test culture in the UK. One thing I think UK teachers are experts at is getting around the test. Take writing narrative in exams. I don’t know of a UK teacher who doesn’t teach the pupils to write and learn a story they can adjust to fit the title. Another example from me. I had a French teacher who was certainly excellent at teaching to the test. I have an A level in French thanks a lot to him and I could even write you an essay on “La Toxicomanie”…but I can’t speak French.
    Certainly if we are truly brave enough to allow pupils to take real responsibility for their exams, then results would go down, but would pupils learn more from this? Right now in many schools pupils expect revision sessions that will take them through without taking real responsibility. One pupil last year said he would judge me as a teacher on results day as if I were the one taking his exams.
    I think you make really important points about trying to interrogate your teaching and I agree we should all do this. But I also think the high stakes nature of external exams for teachers leads to exactly the opposite. Spoon feeding can go disastrously wrong if the exam format changes, because some pupils cannot or perhaps will not deviate from the path that has been drilled into them.
    Perhaps intelligent testing is the place to start, before we can have intelligent accountability. I don’t know when that will ever happen.

    • julietgreen says:

      “I don’t know of a UK teacher who doesn’t teach the pupils to write and learn a story they can adjust to fit the title.” This used to be the practice when writing was still externally assessed. It’s less likely now, although it’s still an impossible task to teach children to write 40 min stories. I have stopped doing that. Stories are weeks’ worth now. I very much agree that we need intelligent testing. I think this will need intelligent application of technology.

  6. Tim Delaney says:

    Most scrutiny of teachers has very little to do with professional development. School managers construct performance management structures of Byzantine complexity in order to demonstrate that they are doing their job and to deflect any blame for inadequate pupil performance onto teachers. Now performance management is linked to pay progression there is a greater incentive for managers to find justifications for deny teachers their annual pay increments. (These were previously awarded for a further years’ satisfactory experience.) Every teacher denied their increment leaves more cash in the pot for management salaries.

    Your blogpost offers some excellent advice on tactics for redeflecting criticism. I will be recommending it to younger colleagues.

  7. Another thought: you seem to have taken as inexorable the Kantian dichotomy of faith vs reason, and science vs philosophy. Have you considered the views of earlier thinkers on this distinction?

    • David Didau says:

      I don’t take anything as ‘inexorable’ and am always open to new ideas. Where would you point me?

      • Thomas Aquinas would give you the most contrasting view to Kant.

      • You could go back to the roots of Western philosophy of course, and take another look at Plato and Aristotle. Anyway, something thoroughly pre modern is needed as an antidote to Kant. Allan Bloom gives an invigorating alternative view of Western cultural history in ‘The Closing of the American Mind’, although I don’t agree with his dismissal of scholasticism. Even he has to admit that Aquinas understood Aristotle better than most modern scholars do, however.

  8. Josie says:

    I too think the idea about a culture of non judgemental support in schools is excellent. When teachers don’t feel they have to excel and meet every criteria of success in every lesson, ideas and advice seem less like personal critiques and more like an opportunity to collectively learn how to do a better job for the students.
    I do, however, think the issue of ‘evidence’ and ‘excuses’ is a thorny one. Teachers who have greater value to their work than test scores know that many of the outcomes of their work is immeasurable, the confidence building, the peer / social skills, the organisational skills etc. Seeing students develop in these areas and having annecdotal feedback from students, their families and other teachers is a worthwhile form of evidence that isn’t necessarily quantifiable. Rather than causing stagnation, this faith in efficacy beyond what the academic results reflect may fuel commitment to the role and prevent stagnation.
    Lastly, context is crucial- no theory or practice is effective in every context. The application of ideas does require ‘knowing the students’ and ‘what works’, so it’s a healthy compromise between good theoretical grounding and practical, contextual application.

    • David Didau says:

      Hi Josie – I’ve written extensively on this blog (and in my recent book) on the troubling nature of evidence in education. This one is perhaps a useful starter: http://www.learningspy.co.uk/research/research-vs-evidence/ Of course, no theory or practice is effective in every context. But research does allow us to look at the ‘best bets’ and make informed judgements on what has worked in other contexts. Ignoring this in favour of ‘gut instinct’ or some other equally flawed subjective judgement is unacceptable. Human cognition is very predictably and profoundly flawed.

      I’m more than happy to agree that test scores or only a part of the picture, but they do matter. When teachers excuse poor results by saying at least they’re confident, have great social skills or whatever, they’re being unprofessional.

      • Yes- agree- but what constitutes poor test results is highly questionable too – the nature of the test, who decides a target and subsequent success or failure etc.
        I’m not advocating gut instinct over evidence or confidence over grades- just indicating their value.
        A professional teacher is one that sees (as you say) test results as a part of a bigger picture, doesn’t make excuses and is willing to learn from others, and often the most effective teachers have something of a gut instinct about what works that is reflected in the research but has developed through practice

        • David Didau says:

          Your ideal of a professional teacher is probably spot on. Sadly, we have a system which mitigates against such professionalism.

          And complaining about the test is amongst the very worst expressions of denialism.

          • Fully agree that the system mitigates against professionalism, but that includes the way students and teachers are assessed in some cases.
            Do you think complaining about the test is always denial or ever valid?

          • David Didau says:

            That depends whether your complaint is made to excuse or explain your results

          • The problem with teacher accountability is that justifying or explaining results rakes precedence over critical discourse about how and why we assess and the content of assessments

  9. […] It is with our judgments as with our watches: no two go just alike, yet each believes his own. Alexander Pope One of the difficulties inherent in challenging teachers’ judgments is that when those judgements appear to be contradicted teachers sometimes say, “Well, it works for me and my students.” This is hard to challenge.  […]

  10. […] be worth reading as background before getting stuck into this one. Firstly, there’s this: “Works for me!” The problem with teachers’ judgement in which I hold up falsifiability as an antidote for the argument that personal experience […]

  11. […] This is clear, testable and, most importantly of all, falsifiable. […]

  12. […] October “Works for me!” The problem with teachers’ judgement The problem with relying on teachers’ judgement is that, despite what we intuitively […]

  13. […] of the best blogs I read this year was “Works for me!” The problem with teachers’ judgement by the always superb David Didau. As he correctly points […]

  14. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  15. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  16. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  17. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  18. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  19. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  20. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  21. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  22. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  23. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  24. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  25. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  26. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  27. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  28. […] Regarding my claim of traditional math working for me and others, I am mindful of the advice given by David Didau (author of “What if Everything you Know About Education is Wrong?”) who points out the following with respect to educational debates: […]

  29. […] is their intuition that they know ‘what works’ for them and their students. As I argued here, this is the flimsiest, most tenuous of […]

  30. […] But if you’re already a true believer in the benefits of edtech – or anything else for that matter – you will tend not to be very critical of evidence which says edtech is great and completely scornful of any suggestion that it’s not all that. All too often, technology enthusiasts get excited by the product and then start thinking about how to use it in the classroom. The most common, and most worrying, effect of confirmation bias is when enthusiastic teachers draw the conclusion, “Well, it works for me and my students!” […]

Constructive feedback is always appreciated

%d bloggers like this: