The Hunting of the Mark - An Agony in Multiply Chosen Fits

I make my living teaching people about the mysteries of computers. I show them how to sacrifice goats on the network servers so as to get the best performance. I demonstrate that if you frighten a computer into obedience it will tell all its friends on the network about how big and bad you are, and those other computers will all behave themselves as well. Sometimes my students believe me when I tell them these things…

Microsoft have recently changed the rules of teaching. As of January 2001, all Microsoft trainers must have what they call a premier certification in order to be able to continue teaching the Microsoft courses. A premier certification means you have followed an approved course of study and passed exams in the major and minor arcana. Few if any Microsoft trainers run courses in all the subjects that make up a single premier certification. We each have our own smaller areas of expertise – the field is too large and nobody can be an expert on everything. Nevertheless, we must still venture in to those areas. Rules are rules.

So, gritting my teeth, I embarked on a course of study designed to award me a Microsoft Certified Solution Developer certificate, or MCSD for short. Recently I passed the last exam and got the certificate.

Two days after passing the exam I received an email from Microsoft congratulating me on becoming an MCSD. The day after that I received another email from Microsoft congratulating me on becoming an MCSD. The following day I received a third email from Microsoft congratulating me on becoming an MCSD. I began to feel quite overwhelmed by all the attention.

Then the rot set in. The following day I received a Microsoft email that warned me (in no uncertain terms) that if I didn’t get my premier certification by the end of the year I was no longer eligible to teach their courses. It urged me (in the politest possible way) to get my finger out.

One department within the monolith was obviously not communicating with the other and their knickers were right royally twisted as a result.

I’ve been taking exams, on and off, since I was eleven years old. My generation of English children was one that had to take the dreaded 11-plus; an exam that marked you for life, for if you didn’t pass it you were considered to be one of life’s failures. You were sent to a secondary modern school and you studied woodwork and metalwork until you left to get a dead end job. If you passed the exam you went to a grammar school where you studied latin and science and maths and eventually you went to university and became a captain of industry. It wasn’t a very fair system and the exam has long since been abolished. But that’s the way it worked back then.

The exam itself was largely an IQ test, though there were papers in English comprehension and grammar and papers in arithmetic as well. For a whole year before the exam proper we did nothing but practice IQ tests in the classroom. The experts will tell you that practising for an IQ test does no good. They would have you believe that intelligence is a fixed quantity and you cannot affect the absolute value at all by practising for the test. This is bullshit of the smelliest, most diarrhoeal variety. Over the course of the academic year, every single person in our class managed to raise their IQ by measurable amounts as we got used to the way the tests worked and as we got our heads around the sometimes rather twisted thinking that the questions required of us. If we’d managed one more year of study, I think we might all have reached genius level. As it was, we were merely very, very bright.

Many years later, when I was considerably out of practice, I applied to join the intellectually elite world of Mensa. The entry requirement is to have an IQ of 140 or greater. I took their test and my IQ turned out to be 138. If I’d studied for it in the same way I’d studied for the 11-plus, I think I’d have turned the scale around 150. But sour grapes set in and I never bothered trying again.

Grammar school was fun. But it was hard work, geared all the time towards the General Certificate of Education Ordinary level exams (GCE O-Levels) that we took at age sixteen and the advanced level exams (A-Levels) that were taken at age eighteen by those who stayed on into the sixth form. The O-levels covered an enormous range of subjects to a quite extraordinary depth. One day might find me writing an erudite essay on Dickens, followed by page after page of squiggles as I explored the oddities of the differential calculus. Perhaps I would be required to translate a Latin text or write an essay in French, balance a chemical equation or dissect a grasshopper. I studied nine subjects to O-level (the average was seven) and I passed them all.

After the generalisations of the O-levels, we were required to specialise for the A-Levels and the subjects were studied in much greater depth than before. I joined the science stream and studied maths, physics and chemistry. However to keep my hand in, I also took something called "General Studies" which was a hodge podge of art, science, history, philosophy, politics; you name it. I’d leave the rarefied world of thermodynamics or organic synthesis or celestial mechanics and read Camus and Sartre, discuss renaissance art and trade union history. I also learned to sew and to cook (the girls were taught woodwork, metalwork and technical drawing – in many ways my school was a very enlightened one for its time).

My A-level exams opened up new worlds of discourse. Prior to this, exams had merely required me to regurgitate accepted wisdom on topics we had studied in class and the questions were straight down the middle of the road. However my A-levels began to push the envelope. I had to apply my knowledge in new situations and demonstrate understanding rather than rote learning. I had to develop opinions and justify them. I had to explore implications. There was a certain exhilaration in that and while the exams were tough, they were also fun and enormously satisfying intellectually.

At university I specialised in chemistry. For three years I did nothing but study chemistry (with side tracks into physics and maths and a brief flirtation with German. Virtually every chemistry research paper of any merit from the first quarter of the twentieth century originated in Germany). The pressure was extreme.

Intellectually the subject was incredibly stimulating. When I first came across quantum physics I spent an entire year on an unbelievable intellectual high. Who needs drink or drugs or sex when you can turn yourself on like that? So many puzzling things suddenly fell into place. So many mysteries revealed and so many more subtle ones introduced. Quantum physics is, shall we say, less than complete. I never reached such a peak again, but I never forgot the excitement. I was in love with ideas, with the pursuit of intellectual challenges for their own sake. Ideas were important simply because, like Everest, they were there.

One of our maths lecturers was a visiting American. We found him hilarious. He called the subject "math" and he called an exam a "quiz". He gave us "scores". He "graded" us for our work during the "semester". None of us had a clue what he was talking about.

He gave us the first multi-choice exam that any of us had ever seen (at least since the 11-plus. IQ tests, by their very nature, are always multi-choice). We hated it and almost all of us failed. Our lecturer told us we had "flunked" it. Again, we had absolutely no idea what he meant. Out of a group of about 50 students, I think only ten passed. I was not one of them. Our lecturer was quite shocked and surprised; he’d had high hopes for us.

I think it was lack of familiarity with the format that defeated us. We were used to wider areas of discourse, broader strokes of the pen, greater intellectual freedom. We needed the right to explore an idea and examine it under ranges of conditions. The concentration of minutiae required for this exam was foreign to us. We couldn’t cope with the rigidity of thinking, the walls around the ideas, the lack of an opportunity to argue our case. Who says there is only one right answer? And why on earth do we have to agree with our teachers?

The level of detail floored us. Detail had never mattered before. In his Life of Johnson, Boswell quoted Samuel Johnson as saying:

                        Knowledge is of two kinds. We know a subject ourselves, or we
                        know where we can find information upon it.

To us that seemed self evident. You can find the detail whenever you need it, so it isn’t in the least bit important. What mattered to us were ideas, principles, the broad picture, the structure. It was hard to descend to the mundane level that these "quizzes" demanded. (It was also annoying that no credit was given for partial answers. It was all black and white; right and wrong. Yin and yang. Excessive attention to detail does that to you).

At the end of the year, our lecturer went back to America and multi-choice exams vanished from our lives, much to our relief. We returned to the more open ended, discursive questions that we were used to. I started to pass my maths exams again and in the fullness of time I obtained an honours degree in chemistry.

Twenty five years after I left university, multi-choice questions re-entered my life when I began to struggle with the Microsoft exams.

There’s nothing intrinsically wrong with multi-choice questions. The Open University in England uses them almost exclusively. However their exams are carefully thought out, and carefully phrased. The "correct" answer cannot be chosen from the list until a huge amount of work has been done researching the question. Depth of understanding is required (and must be demonstrated). However this is the exception – most multi-choice exams are badly thought out, and ambiguously phrased. As a teacher, I am constantly amazed at the number of different ways my students find to misinterpret my questions. No matter carefully I phrase them, no matter how clear I think I have been, there always seems to be room for manoeuvre, for misunderstanding. For a multi-choice exam to be fair, crystal clarity of expression is required but is all too seldom achieved. Their sole merit is that they are easy to mark – but that is a benefit to the examiner, not the examinee. Surely that is the wrong way round?

Once, in conversation with a Microsoftie, I learned that in many countries (particularly Asian ones) candidates are given an extra hour in the exam because English is not their native language.

"Wait a minute," I said. "The exams are actually written in American, not English. And American is not my native language. Can I have an extra hour too, please?"

It went down like a lead balloon. You can’t make jokes about Microsoft to a Microsoftie. When the Microsofties have the operation that implants the chips in their skulls, the surgeons always take care to remove the sense of humour gland as well.