The value of a second opinion

Posted on by

Suppose you feel terrible and go to the doctor. She takes some blood, runs a test and then says to you: ‘That blood test was positive I’m afraid. With that in mind, there’s a 70% chance you have the Dreaded Lurgy’.

Seeking more information, you go to a different doctor, who takes a urine sample. She runs a test and then says to you: ‘That urine test was positive I’m afraid. With that in mind, there’s a 70% chance you have the Dreaded Lurgy’.

What now are the chances that you have the Dreaded Lurgy?

Simplifying and modelling

To think about this, we need to make some modelling assumptions about the Dreaded Lurgy and the tests for it. We will do quite a lot of ‘supposing’.

Prevalence of the Lurgy

We model the Lurgy as a disease with a base rate of b among the population. E.g., if b = 0.1 then 10% of the population have it. Suppose that it’s completely random whether you have the Lurgy, and suppose the doctors test everyone for the Lurgy, so you learn nothing from the fact that they tested you.

Suppose we have no idea what the base rate for the Lurgy is: b is completely unknown to us.

Accuracy of the blood test

Suppose the blood test done by the first doctor has an accuracy of a. To be more precise, it has a probability of a of giving the correct diagnosis:

  • If you have the Lurgy, it returns positive with probability a and negative with probability (1 – a).
  • If you do not have the Lurgy, it returns positive with probability (1 – a) and negative with probability a.

To avoid silliness, suppose a ≥ 0.5. (Otherwise they would just swap the labels on the results.) Suppose also that we do not know a.

First doctor’s analysis

Suppose the first doctor is completely rational and knows the base rate, the test accuracy, and Bayes’ theorem. The blood test was positive, so she reasons that the posterior probability of you having the Lurgy is

\[ P(\textrm{have-Lurgy}\,|\,\textrm{test-positive}) = \frac{P(\textrm{have-Lurgy AND test-positive})}{P(\textrm{test-positive})} \]

Looking at the denominator, you can test positive either by having the Lurgy and the test being correct, or by not having the Lurgy and the test being wrong. The fraction therefore becomes

\[ p = \frac{ba}{ba+(1-b)(1-a)}.\]

She told you that you have a 70% chance of having the Lurgy, so we know p = 0.7. From the above equation, there is a family of values of base rate (b) and test accuracy (a) which could give rise to this conclusion. We can solve for a, and get

\[ a = \frac{p(1-b)}{p+b-2bp} \]

which looks like this (the vertical axis starts at a = 0.5 as discussed above):

In qualitative terms, this graph fits our intuition. The two extreme situations which could give rise to a posterior of 70% are:

  • the base rate is 70% and the test gives no information: b = 0.70 and a = 0.5, the solid circle at the bottom-right end of the curve;
  • the base rate is very low (but non-zero) and the test is almost perfectly accurate: b ≈ 0 and a ≈ 1, the empty circle at the top-left end of the curve,

with a continuous relationship in between.

(On a modern browser you should be able to use the slider to see the analysis if instead the first doctor had told you a posterior probability other than ‘70%’.)

Analysis of the second opinion

When you see the second doctor, she uses a different test. Suppose that, given a patient with the Lurgy, the urine test result and the blood test result are independent, and likewise given a patient without the Lurgy. In other words, the Lurgy and only the Lurgy is responsible for any correlation between the results of the two tests. Also suppose that the second doctor doesn’t know about your trip to the first doctor or about the blood test result.

Then, for a given base rate b, the urine test must have the same accuracy as the blood test, because both led to the doctor declaring a posterior probability of 70%.

Given that both doctors received a ‘positive’ test result, we can now find the final posterior probability, f, that you have the Lurgy. It will depend on the base rate b and the test accuracy a (which in turn is dependent on b, and is equal for the two tests):

\[ f = \frac{ba^2}{ba^2 + (1-b)(1-a)^2}. \]

This looks like:

Again, this qualitatively fits our intuition:

Near the bottom-right end of the curve, the base rate is almost 70%, and the tests have very limited accuracy. The fact that two independent very weak tests gave a positive result doesn’t really serve to increase our confidence much.

On the other hand, near the top-left end of the curve, the Lurgy is a rare disease, and the tests must be highly accurate for each doctor to (independently) arrive at a 70% probability. If two accurate tests independently give positive results, we are now almost sure we have the Lurgy.

Back to the original question

The original question was: how likely are you to have the Lurgy given that both tests were positive?

We see that we don’t have enough information to answer this. The best we can do is ‘somewhere between 70% and 100%’. The answer depends on the base rate of the Lurgy.

Edit

Thanks to an esteemed colleague for pointing out a woolliness / inaccuracy in the independence assumption at the start of Analysis of the second opinion, and suggesting some improved wording. [20150126]