What might be a good proxy for learning?

Professor Rob Coe’s speech, From Evidence to Great Teaching, at the ASCL conference last Friday seemed to generate quite a bit of energy on Twitter, as did Carl Hendrick’s post on engagement. Coe has been referring to the idea that we confuse learning with various ‘poor proxies’ since the publication of Improving Education.

These are the proxies of which he speaks:

Screen Shot 2015-03-22 at 17.02.34

It’s small wonder, perhaps, that so many get so upset by being told that the certainties on which they’ve based their careers may not actually be true. The cognitive dissonance produced leads us to either agree with Prof Coe and abandon our previous beliefs, or – more likely – dismiss him as ‘just an academic’; after all, what do academics know about the real world?

As I’ve mentioned before (at tedious length) learning is invisible. Or at least, other people’s is. It happens inside our minds and, as such, we tend to believe either that everyone else behaves pretty much as we do, or that certain visible signifiers that we associate with learning must, in fact be learning. The proxies Coe has identified do not preclude learning – at some level we are probably always learning something – just these conditions may be present without students learning anything a teacher intends them to learn.

In Learning vs Performance, Soderstrom & Bjork sift through a mass of evidence that indicates that learning and performance are quite distinct phenomena. Performance is what someone can do in the here and now; learning involves retaining new knowledge – whether it’s procedural or declarative – over time and being to transfer it to new contexts. Performance is all we are ever able to see or measure; learning can only be inferred. Everything we can see will only ever be a proxy for learning, but some proxies might be better than others.

It seems sensible to suggest that the greater the distance, both in terms of time and context, between the conditions of instruction and the conditions of performance, the more secure we can be that we are seeing reliable evidence of learning. So, a test taken a week after a lesson is a better proxy for learning than what a student can do during the lesson. And a test taken a year later is better still. But we’re still just talking about performance and proxies – we should never be fooled into thinking we can see something as mysterious and unpredictable as learning.

Coe asks,

If it is true that teaching is sometimes not focussed on learning, how can we make them better aligned? One answer is that it may help to clarify exactly what we think learning is and how it happens, so that we can move beyond the proxies. I have come up with a simple formulation:

Learning happens when people have to think hard

Obviously, this is over-simplistic, vague and not original. But if it helps teachers to ask questions like, ‘Where in this lesson will students have to think hard?’ it may be useful.

I find the following set of propositions useful to think about how and where learning is most likely:

1.”If nothing has changed in long-term memory, nothing has been learned.”

As Kirschner, Sweller & Clark suggest, if you can’t remember a thing it seems foolish to think it has been learned. Learning may well be more than merely remembering, but any definition that doesn’t include memory is sadly lacking. This shouldn’t be seen as an argument for memorising or rote learning, just an acknowledgement that learning depends on memory.

2. “Memory is the residue of thought.”

Daniel Willingham makes the point in Why Don’t Students Like School?  that we remember what we think about. In order to ensure that students are remembering the things we want them to remember, we need to make sure lessons are designed to make them think about these things because we tend to think about what we do. While it may very well be the case that engagement and motivation increase the likelihood that students will want to expend energy thinking complex thoughts, all too often an emphasis on engagement is a distraction that makes students think about the wrong things. Students often remember the context of a lesson whilst forgetting the content. This can lead to the illusion of learning: we remember the memory of having known a thing. This prevents us from noticing that the underlying substance has leaked away leaving us only with the husk of certainty.

3. “Anything that occupies your working memory reduces your ability to think.”

In Thinking, Fast and Slow, Daniel Kahneman points out that any burden placed on our attention gets in the way of thinking: we turn down the radio in order to concentrate on reverse parking. It therefore follows that the less distractions we have to contend with, the more we are able to focus on thinking. And the more we are able to think hard the more likely it is that those looked for changes will take place in long-term memory.

Thinking is hard to assess or measure. How do you know if students are thinking hard about the right things? And ultimately it’s just another proxy for learning, although probably a better one; if students can’t remember what they were supposed to be thinking about whatever you thought was happening becomes irrelevant. Maybe all this tells us it that lessons should be an exercise in providing focused opportunities to think about the richest subject content, and assessments should provide opportunities for students to use what they’ve remembered in contexts as different as possible from the one in which they were instructed.

All this is messy and asks us to suspend judgement and embrace uncertainty. But uncertainty is deeply uncomfortable and so, the fact that Clipboards find it much easier to tick off Coe’s ‘poor proxies’ means their popularity is unlikely to wane.

24 Responses to What might be a good proxy for learning?

  1. chemistrypoet says:

    Clearly, if we can’t measure learning, we can’t measure an individual teacher’s contribution to learning either. So, definitely time to stop trying, until we understand what is going on (don’t hold your breath). Let’s focus on something else, and take the pressure of teachers and the whole system. (Let’s argue about what should be in the curriculum instead).

    • David Didau says:

      Well, exactly. Attempts to measure the unmeasurable become increasingly fatuous.

    • Learning is not completely immeasurable. It’s true that measurement is indirect and relies on proxies. It’s also true that some poor proxies are (too often) used and, at other times, there is over-reliance on improved assessment results as a proxy.

      I completely agree that the result of this is pressure on teachers which is ridiculous and unjustifiable. However, pupil learning should be each teacher’s primary concern and, as such, they should endeavour to gain as clear a picture of what is occurring as they can using any and all proxies in an intelligent way.

      For me, the problem exists mainly with the external interpretation and judgement. Here, the observer has so many potential ‘measures’ available to them that they can basically support whatever conclusion they want. We’ve reached the point where this is (usually unwittingly) regularly abused.

      • David Didau says:

        Learning may not be ‘completely’ unmeasurable but any attempt to measure it will almost always result in error. The subsequent points you make are spot on – we’d be better off with a situation where all attempts to measure were actively preevented rather than continuing with the abusive status quo.

  2. An excellent summary of previous thinking done on this issue – reviewing our appraisal system at school at the moment, and these ideas are certainly proving influential. I suppose that teachers who have a great track record of exam success (particularly A-Level/IB), one could argue that “clearly what I think is great learning is great learning, because they seem to retain the knowledge and be able to apply it to novel contexts in an unseen exam.”… and then back-track from there. Then again, there is plenty of evidence gathered now from universities showing that even A Grade students arrive without knowing the basics from their previous course – so perhaps “learning” in schools needs to be assessed from a longer term perspective. If that was ever the case I think that the direction would have to be mastery of a smaller amount of knowledge, rather than the kind of breadth we see in A-Levels at the moment.

    • David Didau says:

      I think attempting to assess learning in any meaningful sense is probably unrealistic. We’re probably as well off assessing test taking – at least it provides a reasonably valid and reliable metric from which to compare school performance. This also suggests that we need to think differently about teacher appraisal.

  3. Mark says:

    1980’s schooling had none of the clipboard/tick box stuff that now populates my school’s lesson observation form. In fact I never saw anybody observing my teachers. Although you knew instinctively that some teachers were more effective than others, the idea then was that YOU were responsible for how well you did at school and not the teachers. I was academically average but I did my homework (mostly!) made sure I revised for my exams and in an internet free world I went and found stuff out in the library.

    I now spend my time chasing my students to do their homework, I put revision sessions on that most of them don’t attend because they have busy social lives and if I ask them to ‘research’ a topic they cut and paste from the net without reading the whole article. Hand it in – job done.

    Some balance has to be restored.

    • David Didau says:

      You’re right – making teachers so completely accountable for students’ results is helpful to neither teacher nor students. Maybe it’s reasonable for schools to mandate certain ‘non-negotiable’ behaviours they wish to see from both groups and then accept that que sera sera?

  4. heatherfblog says:

    I’m becoming increasingly convinced KS2 SATS should be sat at the start of yr7. It would ensure focus will be more on learning long term. You might need to change the tests so there were fewer occasions when kids lose marks just because they dodn’t remember the technique though (which would be a very good thing!)

    • Tom Burkard says:

      A much better reason for sitting KS2 SATs at the beginning of Yr7 is to stop primary school teachers from gaming the system. It never ceases to amaze me that the Yr7 ‘slump’ is attributed to the stress of leaving a nice cuddliy and caring enviroment for an unfamiliar school with big kids and lots of different teachers who hardly know your name. I know of one comp where they did an assessment of their intake and found that KS2 SATs overstated their progress by an entire level–or in other words, about two full years.

      Peter Tymms at Durham’s CEM has long argued that asking teachers to assess their own pupils is, in effect, both corrupt and corrupting. Teachers should not be put in a position where they either have to game the system or put their career at risk.

  5. In quantum mechanics the term “observer effect” refers to changes that the act of observation will make on a phenomenon being observed.

    No wonder graded observations don’t work – physics says so !

    • David Didau says:

      The quantum world famously doesn’t play by the same rules and I’m not sure that the double slit test would ever work at a macro level, but psychology produces its own observer effects and biases: the Hawthorne Effect and the the Placebo Effect are probably the most well known

      • Interestingly the Hawthorne Effect looked at the effect of lighting levels on worker productivity. The impact of environmental factors in the workplace and education has gained new impetus partly driven by the growth in low cost sensors (the Internet of Things phenomenon).

  6. […] vs. students – A commenter on my blog points out how this relationship seems to have changed since he was a […]

  7. […] What might be a good proxy for learning? – David Didau: The Learning Spy. […]

  8. […] What might be a good proxy for learning? | David Didau: The Learning Spy ‘As I’ve mentioned before (at tedious length) learning is invisible. Or at least, other people’s is. It happens inside our minds and, as such, we tend to believe either that everyone else behaves pretty much as we do, or that certain visible signifiers that we associate with learning must, in fact be learning. The proxies Coe has identified do not preclude learning – at some level we are probably always learning something – just these conditions may be present without students learning anything a teacher intends them to learn.’ (via The Echo Chamber, also the proximate source of several other links) […]

  9. […] actually involve much in the way of learning. Hard work often isn’t fun. If Rob Coe is right (and I think he is) that “learning happens when people have to think hard” then it’s small wonder […]

  10. mark House says:

    Kahneman’s book should be essential reading for anyone involved in making judgements on others. To think that in Education we are any less likely to use System 1 (Fast, automatic, frequent, emotional, stereotypic, subconscious) rather than System 2 ( Slow, effortful, infrequent, logical, calculating, conscious) is ridiculous. However, I am still told that observation ‘evidence’ of learning is in someway reliable and valid by people who seem to believe what they are telling me.

  11. […] 4. Narrow measures of success We’ve become really good at measuring some things such as exam results, but not the hidden effects of what happens in classrooms. In the Vietnam war McNamara thought the success of war could be measured by body counts, leading to… the result of that conflict. In education we measure what we can easily measure, such as Robert Coe’s ‘poor proxies for learning‘. […]

  12. […] were initially covered six weeks ago with those discussed today then we’re forcing them to think harder. However, we also need to build that memory through repetition and constantly revisiting […]

  13. […] Professor Rob Coe’s speech, From Evidence to Great Teaching, at the ASCL conference last Friday seemed to generate quite a bit of energy on Twitter, as did Carl Hendrick’s post on engagement. Coe has been referring to the idea that we confuse learning with various ‘poor proxies’ since the publication of Improving Education. These are the  […]

  14. […] Didau (@LearningSpy) has written a nice blog post looking at the idea of proxies here, which is worth a few minutes of your […]

  15. […] sensible and I’ve written about this formulation on a number of occasions, most recently here. I saw Rob speak at a conference on Friday and tweeted the […]

Constructive feedback is always appreciated

%d bloggers like this: