Surrogate outcomes in education

I have recommended several times Ben Goldacre's book, Bad Science. It's a particular favourite of mine, dealing, as it does, with the ways that people misrepresent, misuse and generally misunderstand science. A section I was re-reading only yesterday deals with the tricks that are used when publishing the results of drugs trials. It's interesting in itself but one of them struck a chord with me as having a wider application, specifically an application in my own area - education.

I'll tie the threads up later but first I need to make a couple of detours, one to explain what he was talking about in medical terms and one to deal with the rather more vexed question of what education is actually for.

The specific "trick" that I was reading about was measuring surrogate outcomes. When you are testing a drug you have, one presumes, a reason for doing so. There is something that you expect that drug to do. It might be to prevent heart attacks, to alleviate pain, to treat cancer. You therefore design a test, hopefully a properly thought out randomised double-blind trial, and see if patients on the new drug show the required result without too many undesired side effects. But not all drugs trials do this. Some measure surrogate outcomes. For example a drug to prevent strokes might use reduction in blood pressure as a surrogate outcome. It isn't actually checking how many patients have a reduced stroke risk it's checking how many patients have reduced blood pressure and extrapolating that this equates to a reduced stroke risk. It's measuring something that is easily and quickly measurable and then using that to say that the drug has achieved it's desired result. Whether this is a good or a bad thing you can find out by reading Ben's book. The relevence it has to teaching I'll come to later.

OK. Detour number two. What is education for? I'd argue that education is to equip people with skills or knowledge that they don't already have so that they can lead a happier, more fulfilled and possibly (though not necessarilly) more productive life. This isn't actually what the Government would like education to be for. The Government would like education to have one aim - to get people out of education, off benefits and into work. The "productive" bit above is the only one they are really interested in. The "happier" and "more fulfilled" bits don't really concern them. Because it's, ultimately, the Government who pay for it they call the shots and schools and colleges toe the line and make education about preparation for work.

Now, as I've ranted before, the Government is also obsessed with targets and measurability. This might be viewed as a good thing. Evidence based education surely is as valid a concept as evidence based medicine. After all if we are trying to educate people, for whichever set of reasons, it would be useful to have some evidence that the way we are doing it works. Targets and measurability can be viewed as tools to achieve this aim.

So, how do we measure whether the way we are doing it works? We use exams, and a finer example of a surrogate outcome would be hard to find. An exam doesn't test whether the person taking it has acquired the skills or knowledge they set out to get, it tests whether they have acquired the skill of passing an exam. This is especially true in the "soft" subjects, such as languages - subjects where what is being learned isn't simply a list of facts or procedures. I have only this morning been devising some materials for my class to prepare them for their forthcoming reading exam. The materials are very carefully tailored to include practice at exactly the kinds of question that I know will be on the paper and these aren't language questions at all. They are things that we would never ask in real life unless we were, perhaps, doing a course in semantics or semiotics. What is the purpose of this text? Who is the intended audience for this text? How would you describe the language in this text?

I will dilligently teach them how to answer all these and more, but I know and they know that what I am teaching them has little if anything to do with the basic skills of speaking, reading and writing that they need. I am teaching them how to pass the exam. Passing the exam is the measure of my success and theirs, but it is a surrogate outcome.

Of course that's specific to the design of the exam that we use in our college and different exams may correlate more - or less - closely with the actual language skill in question but whether they do or don't correlate is largely irrelevent. A surrogate outcome is NOT measuring the actual desired objective, it's measuring something else and then claiming that because this is positive then it follows that the real outcome must also be positive.

It doesn't follow. Exams test exam passing ability. They test how well the students have been coached to produce the answers that the exam board expects.

And we need to accept that this is the case and, if we can, find a better way of doing things, or at least a better exam design, or any decisions and policy we make based on our results will be fundamentally flawed.