For a bit of fun I did the “IQ Tickle test” a year ago and just received the “detailed results” for free – of course I wasn’t going to pay for them when I did the test. While psychometricians may claim that a test is valid in some sense, the tests often have certain problems, and this IQ test demonstrates some of these limitations. I’ve been working as a test developer for a year, and so I’m more aware than previously of the pitfalls of tests (eg, a test can only capture very limited aspects of anything).
When I was a kid, I loved IQ tests, but of course, as you get older you become aware that things are much more complex than you thought they were as a kid, and in any event, there are all the problems associated with giving any credence whatsoever to test results, eg, what does a certain test result actually mean?
One question on this test in particular irked me:
John likes 400 but not 300; he likes 100 but not 99; he likes 3600 but not 3700. Which does he like:
(The arrow was my response and the tick is the “correct answer”.) The “answer” is supposedly 900. Why? Well, apparently all the numbers that John likes are squares of integers, those he doesn’t like are not squares of integers. This sounds fine at first glance, so ok, but this is only one possible answer.
The other three responses could equally well be “correct”, and this can be seen by turning the question into the following one which, to me, seems almost logically equivalent:
Given three points on a sheet of graph paper with x and y axes, which of the four following points also lies on a polynomial passing through the three given points and NOT passing through three other specified points.
(A difference between this problem and the Tickle problem is that points on an x-y plane need two pieces of data to describe them, while only one piece of data is given in the Tickle problem. Nevertheless, the principle is not dissimilar.)
Of course, any mathematician will immediately say “using Lagrange interpolating polynomials, I can, in principle, construct a polynomial passing through all three given points and also any one of the four potential responses and not passing through any of the points so specified”.
The “answer” of 900 appears to be somewhat arbitrary.
In addition to this logical problem, there is also a problem in the way the Tickle question is constructed. Is John’s dislike of 300 related to his liking of 400, or does he just dislike the number 300 anyway, ie, are the numbers 400 and 300 specially connected in some way? This is unclear in the question. If I were writing the question I would say something like:
“John likes some numbers and dislikes others. Some of the numbers he likes are 400, 100, 3600. Some of the numbers he dislikes are 300, 99, 3700. Which of the following numbers does John also like?”
A problem with this wording is that the “pattern” of squares is not broken by the non-squares, so I would guess that more people would choose the “correct” answer.
I couldn’t work out a relation between the numbers he liked and the numbers he disliked, and so I guessed the answer (very usual behaviour for test takers). I was very surprised to learn that the “answer” was connected to squares of integers. Perhaps the “simplest” answer is what is required in these things. But what does “simplest” mean, and why is the “simplest” answer the correct one?
These are the sorts of questions that need to be raised in the construction of any test, and often they are, but can they be resolved in a satisfactory way?