Kaplan Test Prep - Lucky They're Not Teaching You Causal Inference

Kaplan is a service that runs a test preparation service - you sometimes see their flyers on public transport (on the very small number of dire occasions I've been forced to ride the bus or train), and other places. They help people prepare for standardised tests like the LSAT, MCAT, GMAT, and all those other scary acronyms ending in a capital T.

 I have no idea whether they do a good job or not. But I do know that their "High Score Guarantee" on their website is stupid, and designed to fool only stupid people. Which, as it happens, are probably the people who need help on these tests.

"Higher score guaranteed or your money back
We have the most comprehensive guarantee in the industry. Get a
higher LSAT score guaranteed or your money back."

So what's the problem with this? Well, it's an old one familiar to economists - identification. To work out whether Kaplan is actually doing anything, we'd like Kaplan courses to be randomly assigned to students. And they're not. So what else could be going on?

The most obvious problem is that I'd expect everyone to do better on average on their second attempt, regardless of whether they took a test prep course. Why? Because the fact that you took it a second time probably means that the first time you took the test you had been drunk the night before, assumed it would be easy, and screwed it up. And that's why you're doing it again. Had you gotten an unusually high score the first time you took it, it's very unlikely you'd take it again. Reversion to the mean alone will get you there, let alone the fact that on the second time people are probably doing the study that they didn't do the first time.

So a better comparison would be:

"Honest Kaplan Score Guarantee - We guarantee that the improvement in your test score will be higher than the average improvement for all second-time test takers"

This is better, but it still isn't perfect. Specifically,  people who want to pay money for a test prep service are likely more intrinsically motivated to study than the average second-time test taker. So what would be the ideal guarantee?:

"Holmes Testing Service Guarantee - We guarantee that if we take a  sample of 200 second time test takers and randomly assign half of them a Holmes Test Prep Course, the group with the test prep will have a larger average score increase, or you get your money back. We guarantee this because we did the experiment, and it works." 

You're looking at that thinking, "Wait, what the hell? How is this guaranteeing me anything? How could I get my money back?"

Well, it depends what you're after.

If you're after useful information that the course you're about to take will actually help you, the Holmes Guarantee is far more useful than any of the others.

If you're the type who buys the extended warranty on your dishwasher, likes the idea of something for nothing, and doesn't understand causal inference, go with the Kaplan guarantee. Your score will definitely go up!

Kaplan has decided that the market for the latter is far larger than the market for the former. (Assuming they themselves understand inference, which is far from clear). Depressingly, they're almost certainly correct in their assessment. Which is why I'm not in the test preparation business.