Can we improve school interviews? Part 2: Intuition vs. statistical prediction

In Part 1 I reviewed some of the research around the best way to recruit and how this might apply to school recruitment. One of the suggestion I made was that schools should “design an interview format around no more than six qualities or attributes and come up with a short list of questions for each attribute. Then score each interview on a scale of 1-5 for each of the metrics you’ve come up with.” In this post I will go into more detail about exactly what that might look like.

I’m basing these suggestions on the ideas of Daniel Kahneman and Paul Meehl, neither of whom were interested or involved in the recruitment of teachers. Meehl was a clinical psychologist, probably best know for his work on the Minnesota Multiphasic Personality Inventory (MMPI), who in 1954 wrote a short book called Clinical versus Statistical Prediction. In it he set out some of the problems with relying on the intuitions gained through interviews. Essentially, his finding was that an algorithm that had only students’ high school grades and the results of an aptitude test provided better than a trained interviewer at predicting the performance of college freshmen. Although the interviewers had access to far more data, most of it was irrelevant and led to the interviewers basing their predictions on emotion and bias. This is a finding that has stood up to further research in the intervening decades and, years later Meehl said in Causes and Effects of My Disturbing Little Book, “There is no controversy in social science which shows such a large body of qualitatively diverse studies coming out so uniformly in the same direction as this one.”

Kahneman, the Nobel prize pinning economist and behavioural psychologist, applied Meehl’s findings to real situations and designed an interview process that harnessed the power of statistical rules without completely sidelining the role of intuition. Kahneman was tasked with improving the interview process for determining whether a prospective soldier would perform well in the military. Previous attempts to use psychometric tests and probing interviews has shown no statistical validity and so he determined to apply Meehl’s insights.

What he did was to select six characteristics that seemed relevant to a soldier’s performance and, for each trait, he came up with a series of purely factual questions about aspects of the candidate’s life: what different jobs he had done, how punctual and reliable he’d been, what sort of relationships he had, and what sorts of interests he had pursued. The focus on factual questions was an attempt to combat the halo effect where favourable first impressions influence later judgements. As a further precaution, the questions were to be asked in a strictly standardised sequence with each trait scored from 1-5 before moving on to the next. After these six sets of scores had been compiled, Kahneman asked his interviews to ‘close your eyes’ and assign another holistic score of 1-5. He found that totting up the scores to give a total from 6 – 30 provided a much more reliable prediction than the previous regime of testing and unstructured interviewing, but, surprisingly, so did the ‘close your eyes’ exercise. The lesson he learned was, “intuition adds value even in the justly derided selection interview, but only after a disciplined collection of objective information and disciplined scoring of separate traits.” (Thinking, Fast and Slow p. 232)

So, how to apply this to a teaching interview? The first job is to select a set of no more than six desirable traits. This could result in hours of fierce debate as to what makes an effective teacher, but for simple expediency I’m going to go with the traits suggested in the Sutton Trust’s What Makes Great Teaching:

  • Pedagogical content knowledge: “a strong understanding of the material being taught” and “the ways students think about the content” Effect teachers are “able to evaluate the thinking behind students’ own methods, and identify students’ common misconceptions.” (p. 2)
  • Quality of instruction: “effective questioning and use of assessment”; “reviewing previous learning, providing model responses for students, giving adequate time for practice to embed skills securely and progressively introducing new learning (scaffolding)” (p. 2-3)
  • Classroom climate: “quality of interactions between teachers and students, and teacher expectations” and “attributing student success to effort rather than ability and valuing resilience to failure”. (p. 3)
  • Classroom management: the ability “to make efficient use of lesson time, to coordinate classroom resources and space, and to manage students’ behaviour with clear rules that are consistently enforced”. (p. 3)
  • Beliefs: “Why teachers adopt particular practices, the purposes they aim to achieve, their theories about what learning is and how it happens and their conceptual models of the nature and role of teaching in the learning process”. (p. 3)
  • Professional behaviour: such as “reflecting on and developing professional practice, participation in professional development, supporting colleagues, and liaising and communicating with parents”. (p. 3)

The next, and much trickier step is to come up with questions from which each of these traits can be indexed. I’ve had a go at this and, while I’m sure my attempts won’t satisfy everyone, I hope they might at least provide a useful starting place for developing your own questions.

Pedagogical content knowledge

  1. What do you consider to be strengths in your subject knowledge?
  2. What areas of your subject knowledge need development?
  3. How do you respond when a student misunderstands a concept?
  4. How would you go about assessing why the misconception occurred?
  5. What are the common mistakes you see students making in your subject area?
  6. How do you anticipate and avoid these mistakes being made?

Quality of instruction

  1. How do you assess whether all students have understood a new topic?
  2. Give an example of how you would question students to ensure they have understood what you have taught.
  3. How would you go about reviewing what has been learned in previous lessons?
  4. Explain how you would model how students should [insert a task relevant to subject].
  5. How much time would you give students to practice applying a new skill or demonstrating their understanding of a new concept?
  6. How would you scaffold a task to make sure all students were able to achieve a high standard?

Classroom climate

  1. How do you think teachers should interact with students?
  2. How do you demonstrate high expectations for all students?
  3. How important is it to recognise students’ sense of self-worth?
  4. If a student has performed well in a task, how could you get them to aim higher next time?
  5. If a student has performed poorly in a task, how could you ensure they try harder next time?
  6. If a student decides they are incapable of achieving a task, how could you stop them from giving up?

Classroom management

  1. How do you ensure lesson time is used effectively?
  2. How would you organise your classroom to make the best use of the space?
  3. How would you go about making sure resources are used effectively?
  4. What do you see as the teacher’s role in enforcing school rules?
  5. How would you respond if a student was disrupting your lesson?
  6. How would you use the structures and support in place at your current school?

Beliefs

  1. What do you see as the most important aspects of being a teacher?
  2. What do you see as the purpose of education?
  3. How do you go about achieving this purpose?
  4. What is your view on current theories of learning?
  5. What are the main barriers to students learning?
  6. What do teachers need to do to ensure learning is taking place?

Professional behaviour

  1. Considering the lesson you taught earlier today, is there anything you would change if you were to teach it again?
  2. How do you go about seeking feedback on your development as a teacher?
  3. How do you go about finding out about new developments in your subject?
  4. What steps have you taken over the last year to develop your practice as a teacher?
  5. Explain how you would go about offering support to a struggling colleague.
  6. How do you go about communicating with parents?

As you can see, I’ve come up with six questions for each of the six traits. This happened fairly naturally but I don’t think an equal numbers is especially important. It is important to decide on what you think a good or bad response will look like before asking the questions. This will vary greatly between individuals – especially for the section on beliefs – so you probably need to discuss this with other members of the interview panel in advance. I think the advantage of scoring each trait individually and giving each equal weight is that it should help to reduce the effect of our inevitable biases.

In order to avoid group think and false consensus, your panel will be composed of people who don’t all believe and expect exactly the same things. I would recommend giving everyone’s scores equal weighting; for this to work it’s vital to avoid giving the headteacher a casting vote. Ideally, the final decision maker would not be part of interview panel but just given the totalled scores.

The final thing to reiterate is that the candidate with the highest scores (and by all means feel free to assign scores for other aspects of the interview such as the observed lesson) should be the candidate you appoint. Kahneman cautions us against second guessing ourselves and deciding that a candidate who we preferred but who scored less well should be given the job:

A vast amount of research offers a promise: you are much more likely to find the best candidate if you use this procedure than if you do what people normally do in such situations, which is to go to the interview unprepared and make choices by an overall intuitive judgement such as, “I looked him in the eye and I liked what I saw.” Thinking, Fast and Slow, p. 233

Please feel free to add to, adapt or improve upon my suggestions in the comments below.

35 Responses to Can we improve school interviews? Part 2: Intuition vs. statistical prediction

  1. I’ve really enjoyed these two posts, David. As governors we are supposed to be involved in only SLT and head appointments. It would be very interesting to chat over coffee with you about these interviews.

  2. Mig says:

    I would add in why would you want to work in this school. I would like to know that a candidate has carried out some research into the school, the council or local authority and has a clear view of why they want that job.

    • David Didau says:

      This is typically just flannel. I’d avoid opportunities for interviewers to feel they had to flatter the school.

      • Mig says:

        It’s not about flattering the school, it’s about a candidate doing their research about a school or council. Up here teachers tend to get interviewed by the council to go onto a general pool and I would want to know that they have looked into the socioeconomic factors within the council and what the council’s priorities are. What the council might be doing for educational research etc. If the interview was specific to a school particularly for a senior management post then I would want some evidence of prior research into the school and how the candidate sees what skills and knowledge they can bring to the post.

        • David Didau says:

          I don’t really care about that. It’s enough to know that you wanted the job enough to apply for it. If you particularly value those things then it would make sense to tell potential applicants that this is your expectation so that they don’t have to guess.

          • Rick Barnes says:

            I agree, if a school asks you what you think their goal/ethos stands for or means to you then maybe it is not clear enough. One recent interview springs to mind where the school ethos does not mean what it says on the tin. Or tell us about yourself when what they mean is what can you do for us. Or a practical task, here is our marking policy, mark this work. I’m hoping staff get a little more training than that or is that how they select staff, because they all marked in exactly the same way?

          • Mig says:

            It’s probably quite a common expectation in our council interviews but you are right for a maingrade post in a specific school the expectation should be made clear.

  3. […] my next post I’m going to give particular attention to point 4 and provide a template for effective […]

  4. Michael Pye says:

    Interesting. I see how these questions would tease out a applicants beliefs and practices. I like how even with these questions different schools would still hire different people depending on their beliefs. You mentioned Kanehman saying that you should have a good idea of a good and bad answer to each of these questions. This would mitigate the calibration issues that first struck me when reading them.

  5. Michael Pye says:

    In the question “If a student has performed poorly in a task, how could you ensure they try harder next time?” was the word harder a bit of a trick/or a mistake? Did you mean what would they try differently? Am I just over reading it. Thinking about earlier posts around learning requiring thinking hard and how your position has shifted.

  6. Tom MS says:

    I’ve been waiting for this kind of discussion for a long time, thank you. A similar conversation is happening in the tech/software world, where candidates often do a ‘whiteboard interview’ where they solve a coding problem on a whiteboard. Food for thought : https://techcrunch.com/2013/06/22/the-technical-interview-is-dead/

  7. Here’s my list of 6 (I’m recruiting right now)

    • is humble and receptive to feedback
    • has an excellent academic background
    • has a record of success in raising attainment
    • is really good at telling stories and explaining ideas
    • can demonstrate a continued love of scholarship
    • is strict and kind

    • Mig says:

      I like number 4 a lot.

    • David Didau says:

      Interesting list. I think these are all covered by the Sutton Trust’s six traits: #1 is part of professional behaviour, #2 would come under PCK, #3 is tricky – I avoid this as it would clearly mitigate against NQTs also, it’s not really a trait. Some of it will be accounted for by the six traits I’ve suggested, some of it will be down to luck. #4 Is part of Quality of Instruction, #5 is covered by professional behaviour, and #6 is covered by Classroom Management/Climate.

      The advantage of using what research says matters rather than just going with our guts is that we reduce our tendency to privilege what we think is important over what’s actually important.

      Thanks.

      • Thanks – agree with your thoughts that the suggestions are covered by the 6 traits (though I can’t work out what PCK is?).

        I liked my wording because, being written in plain English rather than jargon (I hoped, will accept it if I’m wrong), it would focus the candidate’s thoughts and make it easier for them to write their statement.

        But really useful set of blogs – I found the academic ranking the most interesting idea. I’m going to work out the questions for the six traits. I’m going to have a third person, not involved in the interview to look over the scoring and I’m going to really try to pick the person who has the highest score, not the person I like.

        Off to read blog 3.

        • David Didau says:

          The problem with ‘plain English’ is that sometimes we reduce complexity too far to be actually useful. Meanings blur and clarity ends up being lost. I think it’s reasonable to assume candidates for a teaching job are capable of working out what is meant form the the questions which come under my six traits. There’s no particular need to share the headings with them.

          PCK stands for pedagogical content knowledge – broadly it means not just knowing your subject, but knowing how to teach it.

          Thanks for you comments, David

    • Michael pye says:

      What happens if they question some of the feedback. It would make it hard to appear humble?

      How they react to feedback offers more flexibility. You could then model different responses.

      • Ummm – I have noticed that there are some people who seek feedback, some who accept it and some who reject it. Unfortunately I’ve also noticed that your reaction to feedback tends to differ based on whether you think the person giving it to you is a genius or a fool. So, I’d find it useful in an interview to give feedback that I think the cadidate will disagree with/struggle to accept. If they reject it they either they think I’m a fool or they are unable to take feedback. Either way, it’s probably going to be difficult for them to work with me?

        • Also, questioning feedback to understand it is fine. Questioning because you disagree with it doesn’t work.

          • Michael Pye says:

            Nope your assuming you can understand someone’s personality from flawed data as there are other reasons then someone thinking you a genius or a fool.. You have missed out that the reaction may simply be because you were wrong. (They could be as well).

            Why will questioning because you disagree not work accept in the sense that they shouldn’t annoy you simply because of the one sided dynamics of a interview.

            Simply scrap the unhelpful humble bit and create some simple models of how people could agree/disagree/clarify feedback and yet remain a professional and a desirable employee.

            As a more concrete example imagine how you would respond if someone fed-back that you should have considered students learning styles. (You can swap LS out for many other concepts).

            I may have misunderstood but I thought the idea was to mitigate the interviews bias’s, this is the premise behind my point.

          • I agree – very helpful to be clear on how to disagree. What I’m really looking for though, is the humble person. The person who when you say, ‘that’s wrong’ they are really interested in why. Some people (maybe because they don’t know any better) just argue.

  8. Matt Perks says:

    Is a question like “What do you consider to be strengths/areas for development in your subject knowledge?” purely factual in the sense Kahneman intended? I’m thinking that you can say what you want the interviewers to hear i.e. that your subject knowledge is excellent across the board apart from one fairly small area that might be expected and that you are already aware of and addressing…
    Might a test of teachers’ understanding of misconceptions like that paper Nick Rose has used a lot give a more objective score?

    • David Didau says:

      Yes, fair point. No more ‘factual’ than Kahneman asking if candidates were punctual, I’d say. Of course, people will try to make themselves look good, but that’s why I balanced it with asking about strengths as well as areas of development. I PCK test would be an interesting addition to an interview day. I think many people might prefer it to my suggestion of an IQ test.

  9. Shaun says:

    These are pretty good from a Selection point of view. To make it even better is to focus less on the questions which ask: what would you do, or how would you go about ect as often what we think we do is far less accurate than what we have previously done.

    • David Didau says:

      Actually, that was something I looked into in my previous post and there’s no secure evidence to support that view. Hypotheticals are no different to past experience it would appear.

  10. […] 11 Reasons Teachers Aren't Using Technology #edchat #edtech. Listening strategies: Active listening. Can we improve school interviews Part 2. […]

  11. Alex Jones says:

    Ethical issues should be addressed I think so I like some of your questions that approach these areas.

  12. mrvieito says:

    This is supported by what I’ve been reading in Dylan Wiliam’s new book. The standard recruitment process is successful just over 50% of the time. The more we can add in terms of quality to the picture we build up about a person the more we can shoot that % towards 65-70%.
    There was reference to a survey in the states (I’ll have to look it up) where situational questions were asked (multiple choice with no right answer) which could indicate the degree of empathy a person might have and whether that would be a attribute you would want in a candidate.

    I would suggest adding a classroom session as a ‘form tutor’ and run a ‘circle-time’ activity along with an exercise where an unseen lesson was planned. I think that would reveal whether a candidate was putting on a ‘show’ or whether they had a good grasp of what makes great teaching.

  13. […] 1 of this series I reviewed some of the evidence on what makes for effective interviews, and in Part 2 I looked specifically at creating a less biased, more structured formal interview. In this post […]

  14. sampullan says:

    A bit late on this – though I’ve had them earmarked to read for a while – but I note there’s nothing in your suggested questions about safeguarding. I’ve been asked about that in the past and took it as an indication of how seriously a school looks upon it. The questions were hypothetical: “What would you do in a situation where…”

    Do you have views (I imagine you might!) on the relevance and utility of asking questions in this area? Admittedly it has no bearing on teaching ability, but it’s an important part of our broader responsibilities, so aren’t we in some way bound to satisfy ourselves in this area?

    • David Didau says:

      Honestly, I think asking questions on safeguarding is a box-ticking waste of time. It is the school’s responsibility to train staff on this issue, regardless of current knowledge.

  15. sampullan says:

    Thanks. That’s interesting.

Constructive feedback is always appreciated

%d bloggers like this: