Concept 05 / 08Statistics and Probability
Total Probability and Bayes' Theorem
Partitions of the sample space, the theorem of total probability as a weighted average over causes, Bayes' theorem for reversing conditional probabilities, tree diagrams, prior and posterior probabilities, and classic applications such as testing and identifying a source.
Many problems describe an experiment in two stages: first one of several 'causes' or 'sources' occurs, then an observed event happens with a probability depending on which cause occurred. The events must form a partition: mutually exclusive and exhaustive, with .
Theorem of total probability.
.
Proof: , a disjoint union, so , and each term equals by the multiplication rule. In the tree diagram in the figure, is the sum over all paths that end in , each path's probability being the product along its branches. It is a weighted average of the conditional probabilities , weighted by how likely each cause is.
Bayes' theorem. Given that has occurred, the probability that it came from cause is
.
In words: the probability of the path through to , divided by the total probability of reaching . is the prior probability (before observing ) and the posterior (after). Bayes' theorem reverses the direction of conditioning: from , which is usually given, to , which is usually asked.
Why intuition fails: base rates. Consider a disease affecting of people and a test that is accurate on both the sick and the healthy. For a positive result: . Only half the positives are truly sick, because the healthy group is so much larger that its of false positives equals the sick group's true positives. Ignoring the prior (the base rate) is the most common reasoning error in probability, and Bayes' theorem corrects it.
Method for problems.
1. Identify the partition (the possible causes) and their prior probabilities; check they add to .
2. Write the conditional probability of the observed event under each cause.
3. Multiply along branches and add for .
4. Divide the relevant branch by the total for the posterior.
The posteriors over all causes add up to , which is a useful check.
Key Takeaways:
- Partition: mutually exclusive, exhaustive .
- Total probability: (sum of paths to ).
- Bayes: (one path over all paths).
- Priors become posteriors after observing evidence; posteriors sum to .
- Low base rates make even accurate tests give many false positives.
- Pitfall: confusing with .
Brain-Teaser 1: Bag 1 has red and black balls; bag 2 has red and black. A bag is chosen at random and a ball drawn is red. Find the probability it came from bag 1.
Solution: . Then . The figure shows the tree: the evidence 'red' makes bag 1, which has more red balls, much more likely than the prior .
Brain-Teaser 2: A student knows the answer to a multiple-choice question with probability ; otherwise the student guesses among options. Given that the answer is correct, what is the probability the student knew it?
Solution: . . Guessing contributes only of correct answers, so a correct answer is strong evidence of knowledge.
Brain-Teaser 3: Factories A, B, C produce , and of items, with defect rates , and . An item is defective. Which factory most likely made it?
Solution: The path probabilities are , and , totalling . Posteriors: , , . B and C are equally most likely, even though A makes the most items. The posterior balances a high prior against a low likelihood, and here they exactly offset between B and C.
Practice this concept
Free · no account required · instant solutions