So what were we testing?

Time for another rant about education, I think.

This week I was invigilating another exam, this time an ESOL writing exam, and I was thinking about how we test our students and whether the exams effectively (or even ineffectively) test what they claim to be testing.

Because I've been teaching an intermediate class who have been learning English for a couple of years that's what was particularly on my mind.

I was considering specifically the reading, rather than the writing exam. It's divided into three tasks.

The first of these is a form filling task. The more astute will notice straight away that this is more of a writing task, and indeed it is duplicated on the writing exam. It is meant to show that they can understand the instructions on a form. Now, there are any number of criticisms that I could level at the design of the task and the marking scheme, but I'd rather focus on whether such a task can ever achieve its aim, because I don't think it can.

Lets consider a typical form as used in these exams. First of all it doesn't correspond to the real world because the forms are designed for the exam and isn't complicated enough to be a real form. Nevertheless it is a form and it does have questions.

Typically the first few questions are Name; Address; Postcode; Telephone; e-mail. If there are any instructions accompanying them, they will be "Complete the form in capital letters" or "Complete the form in BLUE or BLACK ink". As often as not there are no instructions.

Now the second of these is going to be automatic. The students have already been told to complete the whole exam in blue or black ink. They can't fail that one. What about the other instruction? Well, I have had students whose sole knowledge of English was how to write their name, address and telephone number. They couldn't recognize them or read them but the could nonetheless write them. It's just about the first thing anyone learning English learns to do. The know that if they see "NAME" then they have to write a particular word. It isn't knowledge, it's more of a Pavlovian response.

As an aside, I once had a student who would take ages to write her name. Upon investigation I found that she didn't actually know the letters, she had learned to do it as a kind of drawing. It could have been anything. She wasn't writing, she was making a picture from memory of something she had seen.

Nevertheless, they do this so often that it ceases to be a meaningful test of anything. Incidentally the marking schemes – regardless of the instruction – usually says "Capital letters are used appropriately, for proper nouns etc." As they know that forms are usually filled in in capitals they can't get this wrong.

What about the rest of the form? Well, that depends on the specific exam being used, but it will typically include tick boxes (no understanding required – see a set of boxes, tick one), circle or delete questions (fifty-fifty whether they get that right), and a few boxes where they have to fill in other details (and that does indeed require them to understand three or four words – but usually very common ones).

And that's it. Not an adequate test of anything related to reading, in my view.
What about the other questions then?

These are based around reading texts. So far so good. They frequently have unfamiliar language in odd contexts but it's certainly arguable that the skill of reading includes coping with that. My main concern is the kinds of questions that are asked. While there are a few that are content based, (What time does the centre open on Saturday? List two of the dangers mentioned in the text.) most of the marks come from things that are analytical. They don't test understanding of the text, they test your ability to interpret points of view and write opinions based on your interpretation. Typically they include things like

How would you describe the language in the text?

What is the purpose of the text?

Do you think this text is friendly or unfriendly? Give examples from the text.

Why did Mr Smith write this letter to you?

One of my favourites, and a I'm paraphrasing a little to avoid mentioning the specific board, is something along the lines of "Give reasons why you think the loyalty card is a good scheme."

For that one, I once had a student say that he didn't think it was a good scheme. The mark scheme listed the answers that were acceptable – all "good" things taken from the leaflet. I marked it correct when he said it wasn't because it was to make him buy things he didn't want – in my view a far better answer than ANY of the ones I was supposed to accept. I even wrote on the paper why I had accepted it in case anyone queried it.

These questions are entirely subjective but it's worse than that. The answers are proscribed. You could write whole essays on them but that's not what's required.

Take the first one.

How would you describe the language in the text?

Think of a text and decide how you would answer it. Sorry you got that wrong. The correct answer should be friendly but formal, or perhaps friendly but businesslike.

Those are the answers on the mark scheme. Answering that question has nothing at all to do with reading and everything to do with having been taught what the exam requires. I could teach that you always write friendly but formal and they would be right 99% of the time without reading the text.

And that's the problem. This type of question doesn't encourage reading. It discourages it. Teachers can't just teach the students to read they have to teach students how to pass the exam. They have to be taught a list of possible answers (usually only two or three) and a strategy for picking one of them. I'm sorry but I don't consider that to be teaching English at all. At best, it's teaching exam technique. At worst, it's teaching a cheat's technique of how to maximise your chances of a pass without being able to read.

The student's aren't stupid and they pick up on it. I've frequently had students ask me why we have to keep on doing exam practice when they would rather be learning English. It seems that about a third of the year is spent on nothing else. The students resent it. I resent it and nobody benefits.

Give me a couple of days and I could produce a far superior set of exams on my own that would test everything they were supposed to be testing. Of course I'd never be able to get them recognised by the educational establishment and the Government but that just reveals what the purpose of exams really is, doesn't it.

(And later I may rant for a while about writing exams.)