99% Ain’t Good Enough (Sometimes)

Let’s run through this thought experiment.

A cocaine test is 99% sensitive. In other words, it will accurately show positive when someone did use cocaine 99% of the time.

That cocaine test is also 99% specific. In other words, it will accurately show negative when someone did not use cocaine 99% of the time.

In reality, let’s say we know that 0.5% of people use cocaine.

This test sounds pretty good. But is it? Let’s break it down.

1. Imagine you tested 1000 people.
2. Out of that sample of 1000 people, you would expect 5 users (1000 * 0.5%).
3. The test would accurately detect 4.95 users (5 users * 99% sensitivity).
4. Out of the sample of 1000 people, you would expect 995 to be clean (1000 * 99.5%).
5. The test would inaccurately say 9.95 of the clean people are users (995 people * (100%–99% specificity).
6. In total, the test would show 14.9 users (4.95 + 9.95).
7. 4.95 real users / 14.9 total “users” = 33% accuracy

At first glance, a test that predicts 99% of true positives and 99% of true negatives sounds great. But it’s not.