A fruitful exchange with the State Examination Commission

While looking through some past and sample papers to help with some volunteer maths tutoring I do, I came across what I was fairly convinced were errors in a question in a sample paper written by the State Examination Commission.

What followed was a very positive exchange of correspondence with the examiner, and the end result is that they are going to review the question. It was refreshing to have such an exchange with a public body, in particular with somebody who directly and properly engaged with my query. If only all interactions with public bodies were as satisfactory.

The question

The question at issue is Question 1 on paper 2 of their 2012 sample. It reads, in relevant part,

The errors

One complaint was that part (d) is flat wrong. The intended answer is surely ‘set A’. However, one can construct a data set with the properties in the table but which contains no negative numbers. Looking ahead, a pleasingly clean example was in fact provided by the SEC in their response:

My other complaint, which is more abstract, was with part (e), which I claimed was ill-posed because no information on the source of the data-sets was given. The question doesn’t say that they were drawn from any particular distributions, so it doesn’t make sense to ask questions about the probability that certain properties are true of them. Another complaint was that the combined data-set has an even number of elements, so the median will (for continuous distributions) be between two values, so almost surely will not be in any set.

I bundled this up into a letter, which I duly sent off to the SEC. In retrospect, this initial letter of mine probably took too snotty a tone in places, but nevertheless I did get a proper response from the SEC, from somebody who obviously knew what they were talking about, and had taken my enquiry seriously. I was impressed by this, given that this was a very busy time of year for them. The key points of their response [PDF is formatted by me from their email] I paraphrase as follows:

• They were seeking to test whether students had gained some intuitive grasp of the concepts of mean and standard deviation. [This is all for the good, if it can be achieved without giving up rigour.]
• Part (d) wasn’t checked carefully enough; they agree that set A needn’t contain negative numbers, and provided the example above. [It was especially pleasing that they’d thought about this enough to come up with a cleaner example than the one I’d given.]
• They stood over part (e), arguing that students should assume that, unless told otherwise, data sets are similar to those encountered in their course. The SEC agreed that the absence of a probability space makes the question unsatisfactory within an axiomatic approach to probability, but argued that this isn’t how the course treats the subject. [They didn’t directly address the point about the even-sized data-set, but this is a reasonable argument on the ‘no distribution’ point.]
• They will review the question in due course, fixing part (d) and thinking about whether part (e) can be adjusted. [Excellent!]

I wrote back, suggesting, in light of their remarks, a couple of concrete ways in which I thought the question could be clarified, and got a final reply

Thank you for these further observations and suggestions. Your suggested amendments have merit and will be considered when finalising the amended version of the question.

Their reply also gave me permission to publish the correspondence here, for which I’m grateful.