Meditations of Lambchop

What do aptitude tests measure?

This excellent essay, A Mathematician’s Lament, by Paul Lockhart has convinced me of something I suspected since learning actual mathematics at university (and not even, at first, within official lectures): our schooling ruins mathematics for children. The same kind of thinking that has created the education system appears to have infected human resource departments in most major companies. I refer, of course, to the ubiquitous use of aptitude tests (a subset of psychometric assessments).  It seems to me that our schools are satisfied with teaching arithmetic rather than mathematics and HR is satisfied with testing “skills” that no candidate will ever need. As with schooling it seems hardly anyone questions the current system.

Companies want as much information as they can get about potential candidate (particularly, it seems, about graduates as they have no relevant work experience).  Aptitude tests were created, it seems, as a kind of one-size-fits-all solution to this problem. Want to know if your candidate is smart? Have him take these tests.

Advantages of psychometric tests:

1. They are objective
2. They are easy to perform on multiple candidates
3. Results are available instantly
4. They make HR people feel like they are contributing
5. They make money for testing companies and suppliers of practice aptitude tests

The question, however, is: do these tests measure something worth measuring? I suspect that they do not. Numerical and verbal reasoning tests purport to give an idea of the candidate’s strength in these areas. However, just as mindless arithmetic is not mathematics, standardised numerical and verbal problems have little to do with mathematical or linguistic ability.

Aptitude testing companies (who have a vested interest in the perpetual use of these tests) will tell you that there is a statistical correlation between the scores on these tests and job performance. This is probably true: but this may stem mainly from the fact that the kinds of people who perform well are the kinds of people most willing to put in the amount of practice needed for the tests. In other words:  the test may as well be “write down the first thousand numbers in the phone book from memory”. These people would memorise the phone book and we would convince ourselves that this was somehow a meaningful skill (Rain man is awesome, right?).

I recently wrote two kinds of aptitude tests.

Numerical

These ask you to do some simple calculations, usually involving reading numbers from tables and graphs. These are things like: calculate the amount of coal produced in France in 2008; in which year did the electricity consumption increase the most?; what is the profit…; what is the cheapest way to…

You have to be fast and accurate. The time pressure is very high. In other words: to do well you need to punch in numbers on your calculator automatically, almost without thinking. After practicing the standard variations of these questions, there is nothing to do but make it second nature. There is no more thinking involved. It is not far above the intellectual level of memorising the phonebook (actually the latter is probably a much larger intellectual challenge).

If your job involves performing repetitive calculations at lightning speed, then this test is perfect. If it involves anything even a modicum more complicated (unpredictable situations, the need to be thorough and careful (in other words slow) rather than blindingly fast, abstract thinking, creativity, etc. ) then it is probably close to irrelevant. It is not even relevant to jobs that are thought to involve lots of numbers (accounting, actuarial science, engineering) – accuracy is important, of course, but through methodical checking and by being aware of what you are trying to do (as opposed to blindly plugging in numbers).

Verbal

Verbal tests asks you to read a passage and then decide whether some statements are true, false, or unknown.  For instance, consider this passage (which I have contrived purely for illustration) and the statements that follow.

Birds are animals with wings. Crocodiles have teeth. Many endangered species will become extinct in the next century.

• Birds have wings: True.
• Crocodiles lack teeth: False
• Birds are endangered species: unknown.

Consider though, the statement: Crocodiles are a type of bird. This statement is clearly false. But it does not follow from the statements in the above passage. The truth of this statement relies on your background knowledge about crocodiles and birds.

On the face of it this does not seem to be a problem. If given the statement “Crocodiles are a type of bird” you answer unknown. You ignore any information not in the passage. You discount your background knowledge, which is what is expected. There are (at least) two problems with this:

1. It is impossible to understand the passage without using your background knowledge. For instance for the statement “many kinds of animals will die out in the foreseeable future” the most reasonable answer is True. But this relies on your knowing that “species” is the same as “kinds of animals” and “extinct” means “die out.” It is hardly possible, I think, to only use words in the passage to avoid this problem and I do not trust people setting up the tests to think this deeply even if it were possible. This means that to answer the questions you must discount some  of your background knowledge but not all  of it. How much of it do you throw away? Who knows.
2. Never ever in your life will you be expected to make a decision or evaluate something without your background knowledge. Most people are, in fact, hired because  of the their background knowledge. It makes their decisions better. Training people to ignore their background knowledge seems extremely silly in this light.

Admittedly the ability quickly to make a decision based on limited information is valuable (for instance to traders), but this is the more valuable because of good intuition, I would say. These tests do not allow for intuition at all. Intuition is all about background knowledge. In most situations it is in any case undesirable to act so quickly. Facts must be considered carefully. Additional information needs to be obtained.

There is a third type of test, referred to as diagrammatic reasoning and I have written an indictment of the very foundation of these tests here(http://blog.johandp.com/2013/09/dont-spot-pattern.html). Basically they expect you to extrapolate from a finite set of data and forget that there are in fact infinitely many (valid) ways of doing so.

I should, of course, back all of my arguments above up with data. If you know of any studies that have been done (particularly ones that oppose my reasoning above, please let me know).  I also realise that companies use far more than just aptitude tests in their decision-making. Interviews, role-play, personality assessments (and I have doubts about every one of these methods) all play a role and none are perfect. As long as these aptitude tests do predict job performance, they have some legitimate use and I am not saying they do not measure reasoning ability at all. However, there is good reason to be suspicious of what they actually do measure. Do we really want to judge candidates based on their ability to do tasks that have virtually no relationship with what we actually expect them to do?