Psychologists Getting Statistics Wrong

Ace of Spades links to a study that claims to show that people view atheists as being less trustworthy. This was also covered in the National Post. The headline claim is attention-grabbing:

Atheists cannot be trusted: Religious people rank non-believers alongside rapists, study

Controversial stuff. As in all this stuff, you should always read the original study before rubbishing it. The author, Will Gervais, kindly has a version on his webpage, which you can read here. And I'm sorry to say that nearly the whole study appears to be done wrong.

So how exactly does Mr Gervais establish that atheists are as untrustworthy as rapists? Let the study tell the story - this is Study 2 of 6, but 5 out of the 6 studies have the same problem:

One hundred five UBC undergraduates (age range 18 –25 years, M 19.95; 71% female) participated for extra credit. Participants read the following description of an untrustworthy man who is willing to behave selfishly (and criminally) when other people will not find out:

Richard is 31 years old. On his way to work one day, he accidentally backed his car into a parked van. Because pedestrians were watching, he got out of his car. He pretended to write down his insurance information. He then tucked the blank note into the van’s window before getting back into his car and driving away. Later the same day, Richard found a wallet on the sidewalk. Nobody was looking, so he took all of the money out of the wallet. He then threw the wallet in a trash can.

Next, participants chose whether they thought it more probable that Richard was either (a) a teacher or (b) a teacher and XXXX. We manipulated XXXX between subjects. XXXX was either “a Christian” (n 26), “a Muslim” (n 26), “a rapist” (n 26), or “an atheist (someone who does not believe in God)” (n 27).

So the authors are relying on the conjunction fallacy of Tversky and Kahnemann (1983) - logically, the probability of being a teacher and [Y] is less than or equal to the unconditional probability of being a teacher, for all values of [Y]. People sometimes get this the wrong way around if the behaviour is associated with the trait. That is what the authors are trying to test (I think). They report that the proportion of people who answered (wrongly) that the person was more likely to be a teacher and an atheist was higher than the proportion who answered (wrongly) that the person was more likely to be a teacher and a Christian.

The first thing that should make alarm bells start ringing in your head is the way the question is phrased. To say 'are atheists untrustworthy?' is to ask the probability of being untrustworthy given you're an atheist. But the question implicitly being asked in the survey is something different, namely the probability of being an atheist given you're untrustworthy. These are not the same thing!!!! And this is really going to screw up the inferences.

If statistics bore you, let me skip to the punchline - the authors screw it up because they're not taking into account that there's tons of atheists and very few rapists. This means that the probability of being an atheist given you're untrustworthy is always going to be much higher than the probability of being a rapist given you're untrustworthy. But this says nothing at all about trustworthiness, and everything about how rare it is that a person is a rapist! And this makes the whole study flawed.

For stats people, what is actually being asked is whether people erroneously believe that:
P(teacher | Untrustworthy actions)  <  P(teacher AND atheist | Untrustworthy actions).

This answer is then compared to answers to the question as to whether:
P(teacher | Untrustworthy actions)  <  P(teacher AND rapist | Untrustworthy actions).

Since the left hand side is the same in each inequality, let's think about what could drive differences in the right hand side (even if people are screwing it up via the conjunction fallacy, this is still the implicit comparison). Using Bayes Rule:



(where A1 = Teacher and Atheist, A2 = Teacher and Rapist, and B = Untrustworthy).

Let's ignore the teacher bit for simplicity (it doesn't change the logic). What the author really wants to know is the second ratio - are people viewed as more likely to be untrustworthy given they're an atheist, relative to being untrustworthy given they're a rapist.

What they're actually measuring is the first ratio: the probability of being an atheist given you're untrustworthy versus the probability of being a rapist given you're untrustworthy.

But the difference between the two ratios is also driven by the third ratio -  the overall probability of being a rapist versus an atheist, regardless of whether you're untrustworthy.

And this ratio is huge! The study was done at the University of British Columbia. According to Wikipedia, 42.2% of Vancouver is atheist. What's the probability of being a rapist?  The overall rate of rape crimes in Canada is 0.016 per 1000 people. As long as each rape is only committed by one rapist, this will overstate the probability of being a rapist (i.e. if a rapist has multiple victims, the probability of being a rapist will be lower. If a victim is raped by multiple people in a single rape, the number will be higher, however)

So the third term is equal to 42.2/0.0016 = 26,375! In other words, suppose that people thought that you were 1000 times more likely to be untrustworthy if you were a rapist than an atheist (i.e. the second ratio equals 1/1000). The left hand side will be equal to 26375/1000 = 26.375. In other words, P(atheist | untrustworthy) will always be much higher than P(rapist | untrustworthy), even if rapists are considered far less trustworthy than atheists.

The authors only report the proportion of respondents who made the conjunction error - in other words, they report the number who state that P(teacher | Untrustworthy actions)  <  P(teacher AND Y | Untrustworthy actions), which is clearly wrong, and compare this for different values of Y. Sadly, this doesn't allow us to say anything about the real ratio, which is P(Untrustworthy | Atheist) versus P(Untrustworthy | Rapist).

In other words, the study is unsalvageable if you're trying to answer the question you're hoping to ask. Which is a shame, because it's actually an interesting question.