When we were discussing it at our club meeting, one of the club officers, Mike Mei, pointed out that this might also be an example of the DunningKruger effect, in which unskilled (in a given area) people overrate their abilities in that area, essentially because they lack the knowledge to see where they have failed. There is a corrollary effect, in which skilled people underrate their abilities, because they spend time with people even more skilled than they are and have a better understanding of their own limitations.The answers I got from SA were, in minutes: 10, 2, 10, 2, 5, 3, 2, 1 (possibly a 10), and blank. They all took between 3 and 7 minutes, so it’s possible the DunningKruger effect was stronger than the planning fallacy here(1.b.) On the original quiz, which I gave to the SA, the first question also had: How many questions do you expect to get right? which was meant to illustrate much the same points. I took this off since not all the questions have right and wrong answers.2. Samantha was part of the Intervarsity Christian Fellowship in college and was abstinent until marriage. She has four children and does not use birth control. Is it more likely that she is a teacher, or a Christian and a teacher?This is a spin on Jane being a feminist and a bank teller, which is a classic thought experiment/trick question in psychology. One example in a psychology presentation can be found here. It is a demonstration of the representativeness heuristic, in which people estimate probabilities of events by analyzing the data they have available to them, rather than by being aware of all the data they don’t have. In this case, people focus on the information I gave, which points strongly to Samantha being a Christian. This gives us a deviance from a Bayesian calculation, in some cases because we neglect the base rates of an event (this would be a prior), but in this case because of the conjugation fallacy. This fallacy occurs when we assume that a more restricted situation is more likely than a more general one. In particular, if we say that Samantha is more likely to be a Christian and a teacher, then we are claiming that the probability that Samantha is a Christian AND that Samantha is a teacher is less than the probability that Samantha is a teacher of any kind, which is clearly false. The wikipedia article has the math, but what you really need to see is this:
If A is the probability that Samantha is a teacher of any kind and B is the probability that Samantha is a Christian, we see that the overlap (C) can’t be larger than either A or B.
To my disappointment, SA answered with 2 saying teacher, 5 saying christian and teacher and two people rebelling against the two options I gave them to say:
“Just a Christian. Women must be oppressed and pregnant. Quote the Bible”
And
“The probabilities seem similar, though one should never take professed faith at face value.”
To be fair, these answers actually have a lot of merit. Even though it wasn’t the point of the question, it probably is much more likely that Samantha is a Christian than that she is either a teacher or both. Given that, it’s probably also true that the probabilities of her being a teacher and both being a teacher and a Christian are similar (if the probability of her being a Christian is high enough). Don’t believe me? Pick some probabilities at random and do the math! It’s just multiplication, I promise.
I find the last bit particularly intriguing. Perhaps this intrepid secularite is referring to the phenomenon of belief in belief?
By the way, if you were confused about all that Bayes talk, here’s a fairly simple explanation of Bayesian probability.
3. Do you think the percentage of countries in the UN that are African countries is higher or lower than 65%/10%? What is the percentage of countries in the UN that are African countries?
If you’ve already checked out both instantiations at the quiz, you probably realized this is one of the places they deviated. This is supposed to illustrate the anchoring effect, in which our analysis of what answer is reasonable is heavily affected by the information we’re given to start with. Sometimes this is because we adjust from that number, and sometimes because our brains remember information consistent with the number we start with. This can occur in context, as in this question or a starting bid for a salary, or out of context, as in spinning a Wheel of Fortune before answering the question (this link goes to a generally great paper). Crazy, isn’t it? But it’s true. It’s also worth pointing out that, despite the claim of some SA members that science people might be less prone to the fallacy than humanities people, even those who are reminded of the anchoring effect and told to avoid it are subject to it, at least when the anchor comes externally (as in this quiz). However, with internally created anchors (if I hadn’t given the first part of the question), warnings and high Need for Cognition do lower the extent of the effect.
SA Answers:
65%: 10%, 10%, 30%, 30%, 20% – Mean: 20%
10%: 18%, 52%, 8%, 13% – Mean: 22.75%
Oddly, the SA at UofC appears to be immune to the anchoring effect. Or something.
4. (5 seconds) Guess the value of the following arithmetical expression: 8 x 7 x 6 x 5 x 4 x 3 x 2 x 1 = ? OR 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8 = ?
This is the same issue. People tend to look at the first few numbers, multiply, and then adjust, since they’re asked to do it in 5 seconds.
Answers from SA
Descending order: 400,000; 1,000; 1,024; 500 Average: 100,631 (obviously not particularly useful given the high variance). Without the outlier, the average is 841.33
Ascending order: 1,000; 900; 16,320 (calculated, not guessed); YAY MATH!; 4,000; Average: 5555
Again, not entirely expected, but that’s ok.
Source: I got both of these questions straight from here: http://lesswrong.com/lw/j7/anchoring_and_adjustment/
5. 1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she has breast cancer?
6. 1500 out of every 10,000 men at age forty who participate in routine screening have prostate cancer. 1300 of these 1500 men will get positive screening tests. 8,000 of the men who do not have prostate cancer will also get positive screening tests. A man in this age group had a positive test in a routine screening. What is the probability that he has prostate cancer?
You probably noticed that these are the same question, one with percentages, one with numbers. Apologies for the typo in the second question, by the way; that’s been fixed. These questions ask for a Bayesian calculation of probability. As someone has pointed out in the comments, it might seem like the test is asking for mathematical proficiency rather than rational abilities. I take the criticism willingly, but nothing on this quiz requires more than basic multiplication. Knowing how to set up a Bayesian calculation may be mathematical in some sense, but I would argue that it’s also simply something a rational person should know how to do, in the same way that calculating iterated probabilities of coin flips requires multiplication but if you think that the probability of getting heads at least once when you flip a coin twice is .5 + .5 = 1, there’s a problem that goes beyond arithmetic. This will become even more clear when I demonstrate the answer to this problem.
The way this works is as follows. We know that some people have cancer and some don’t, and some people get positive tests and some don’t. So we set up a table. The answers will be put in as (breast cancer problem numbers, prostate cancer problem numbers).

Has Cancer

Doesn’t Have Cancer

Positive Test



Negative Test



So for prostate cancer, the numbers are all given

Has Cancer

Doesn’t Have Cancer

Positive Test

( , 1300)

(, 8000)

Negative Test

(, 200)

( , 500)

For breast cancer, we have to do some calculations. For ease’s sake, let’s pick 10,000 as our total number of people (though it doesn’t matter). So of 10,000 women, 1%, or 100, have breast cancer, so our left column must add up to 100. 80% of these women will get a positive test, so 80 of them will and 20 won’t. Now we’re considering 9900 women who don’t have breast cancer, 9.6% of whom (or about 950) will get a positive test anyway leaving 8950 for the final quadrant.

Has Cancer

Doesn’t Have Cancer

Positive Test

(80, 1300)

(950, 8000)

Negative Test

(20, 200)

(8950, 500)

So if you get a positive test, you know you’re in the top row. If you’re a women who got a positive mammography, you have an 80/(80+950) = 7.76% chance of having cancer, and if you’re a man who got a positive test for prostate cancer, you have a 1300/(1300+8000) = 12.9% chance of having cancer.
From SA, who only got the breast cancer problem: 8/9 = 88%, 9.8%, 70.4%, 9.5%, 90.4%, 10%, 90.4%
Not to engage in scare rationalism here, but this is a problem. This means that women who go and get positive mammographies might be overestimating their probability of having cancer, and therefore undergoing possibly unnecessary biopsies, tests, chemotherapy, radiation, hospital visits, with the fear, stress and bills that come along with them, by an order of magnitude. Not good, people, not good.
7. There are four cards on a table. Every card has one side which is white or black and one side with a number on it. The Rule: Every card with a white side must have an even number on the other side. How many cards (and which ones) must you flip in order to check if all four cards follow this rule?
8. You are an employee at an all age party venue, and people are allowed to come in with drinks. You see a group of four guys coming in, all carrying red Solo cups. One has an ID which says he’s 19, one is drinking orange juice, one is drinking beer, one has an ID which says he’s 24. Assuming you are accurate in your assessment of the drinks and all the ID’s are real, whose IDs/drinks do you check in addition to the information you already have to make sure no one is drinking illegally?