A COVID test is 95% accurate.  Really?  How helpful is that information?

Imagine a test that was always positive no matter whether the patient had the infection or not.  This test, detecting no negatives, would have a low specificity.  Nevertheless, it would be correct at least some of the time.  So would a test that was always negative regardless of whether the patient was infected (poor sensitivity).  How often these tests were correct would depend in how many people were actually infected or not. 

The more people who are infected then the more times the ‘always positive’ test gets it right.  This is also true no matter how specific the test is.  Imagine a test which has a specificity and sensitivity of 95% where 5% of the population, that’s 1 in 20 people, are infected. Then, 5% of those people (who are actually positive) test negative.  Also 5% of those who are not infected test positive. 

So, of every 100,000 people tested, only five thousand people will be infected.  Now 5% of the 95,000 (4,750) who are not infected will test positive.  Of the 5,000 who are infected, only 4,750 will test positive.  So of the 9,500 people testing positive, only 4,750 people (50%) will actually be infected, the rest are ‘false alarms’; false positives.  If you are infected then the test is likely to be correct (95%), and if you are not, then it is also likely to be correct (95%).

But that’s only useful if you already know the answer which the test is supposed to provide – Are you infected?  If you only have a positive test result to go on, then it is 50/50 whether you are infected or not.

This demonstrates that to understand a positive result we need to know certain things.  Fundamentally, we need to know the probability of someone having the condition in the first place.  This is called the ‘base rate’ or ‘background rate’, and is a number that affects the entire calculation tree shown here.  Without it almost nothing useful can be determined either about the reliability of a test or the probability that any random person one meets may be infected (which I think is what most people would like to know).  The base rate is the probability that someone selected at random from the population would have the condition.

The base rate cannot be discovered by looking at high risk groups or any selected group; the sample must be randomly selected from the whole population.  Without an estimate of the base rate decisions regarding risk are impossible. 

This is a visual demonstration of what is known as Bayes’ Theorem.  This is a method for calculating the probability of an explanation or hypothesis as new data about the hypothesis emerges.  In the COVID example, the hypothesis is that a person has COVID and the new data is the result of their test.   Another way of describing data, such as a test result, is ‘evidence’; so, it is a way of assessing the probability of a hypothesis in the light of new evidence.

Another way of looking at the above discussion is to ask two questions;

1.       IF someone has COVID, what is the probability of a positive test? (95%)

2.       IF someone has a positive test, what is the probability that they have covid? (50%)

These are different questions, and produce different answers.  Note that in the first question, you must know that the person has the condition (COVID) but the test result is unknown.  In the second question, the test result is known, but the condition (COVID status) is not.

So what has this to do with forensic DNA?  It is which of these questions does a court seek to answer;

1.       IF it is the suspect’s DNA, what is the probability of a ‘match’ (positive results)?

2.       IF it’s a ‘match’, what’s the probability that it is the suspect’s DNA[1]?

In the first, we assume we know the source of the DNA is the suspect.  In the second, we use the result to assess the probability that it is the suspect’s DNA. The most common misunderstanding with DNA evidence today is that people translate the number into the answer to the second question; what’s the probability that it is the suspect’s DNA.  Here’s the surprise:  it is NOT.

In reality, current forensic practice does not answer either of these, but comes closer to the first approach in that it assumes that the crimestain did come from the suspect.  To understand what is actually being done it is necessary to understand the underlying logic which is based on the process described for COVID.

If we consider a DNA profile from one person (single source) then we can set up our chart as below.  There are billions and billions of possible profiles in the world, but we know we are looking for one which matches the crimestain.  The true contributor will match the crimestain.  For purposes of this exercise we assume the profile is unique so there is only one possible match.

Now the suspect may argue that the crimestain came from someone other than himself; in other words, this is a chance match and he has just been unlucky.  From the discussion of COVID, it can be seen that the probability of someone having the condition can be called the ‘base rate’; the frequency of the condition in the population.  There have been many studies measuring the frequency of DNA components in the population.  So, because we know the expected frequency of his profile in the population, the base rate, we can use standard probability to calculate the probability that the match was a random occurrence.  That is called the Random Match Probability (RMP), which is a ‘1 in X’ number such as 1 in a billion.

So far, so good.  However, many, if not most forensic DNA profiles obtained from items such as guns, drug wraps or other items are mixtures of DNA from more than one person.  This creates a new conundrum; when you mix DNA from two or more people the number of possible profiles increases dramatically as the number of people and the number of loci.

For example, take the words ‘one’ and ‘two’, mix the letters and now we can make more words than the originals; net, won, ten etc.  if one person has profile AB and the other has the profile CD, the mixture is ABCD.  Now the possible contributors are AB, CD, AC, AD, BC, and BD; 6 possibles but only two actual contributors.  The analysis is now more difficult as there are many more profiles that ‘match’ the crimestain than there are actual contributors to it. 

A false positive is when a profile is present in the mixture, but it is not the profile of an actual contributor.

It can be seen that there are now more false positive results than true positive results (false negatives are possible when dealing with Low Template DNA, but that is another story).  In other words, there are more profiles which could be from a contributor than there are actual contributors.  This situation becomes more pronounced as more alleles are added to a locus.  If there are 17 areas of DNA analysed (the ‘standard’ number in the UK) with this situation (6 possible genotypes at each locus) then there would be 617 (16,926,659,444,736) possible contributors but still only two actual contributors.

Of course it is more complicated when we try to calculate the weight of evidence against ny person because we need to take account of the frequencies of each genotype.  This article is simply a description of the problem of interpretation.

Many crimestain samples nowadays are mixtures of DNA from different people.  A DNA profile is a description of the different types of DNA in an individual.  We share some of these with our relatives but also can share at least a few with other people.  This creates a conundrum in understanding the possible contributors to a mixture.  All of the profiles which are possible from the combinations in the mixture, other than the two actual contributors, are in effect ‘false positives’.  Given these billions of possible contributor profiles, what is the probability that anyone selected at random from the population would ‘fit’ or have one of those possible profiles and therefore be considered a possible contributor?  This is the ‘base rate’ – the probability of a positive test regardless of whether the person is a contributor or not – a chance match.  This statistic is called the Combined Probability of Inclusion which is similar to the RMP but applied to mixtures.

It is obvious that there are more false outcomes than true outcomes (more false inclusions).  In fact, a false outcome is many times more probable than a true outcome.

Originally, this problem was dealt with by considering various forms of the question, “how likely is it that a randomly selected person would be one of the matching profiles?”  By implication, what is the probability that this is just an accidental ‘match’.  This statistic is termed Random Man Not Excluded (RMNE), or the Combined Probability of Inclusion (CPI), and is the equivalent of the RMP when applied to mixtures.

The LR approach is different.  It assumes that the crimestain came from the suspect and so the probability of observing that profile in the crimestain, if it came from the suspect, is 1 (100%).  i.e. if it came from him then it will certainly match.  So that is the probability of the EVIDENCE IF the DNA came from the suspect.

Going back to the single source example:

The LR now considers the probability of the evidence IF it came from someone OTHER THAN the suspect.  That is the RMP.

So the LR would be 1 over the RMP (1/(1/billion)), which is a billion.  Not 1 in a billion, just a billion.

It is easy to see that converting the LR to an RMP is easy: LR=1b, RMP=1in1b 

Current practice is many jurisdictions including the UK is to calculate a statistic called a Likelihood Ratio (LR).  The LR is a test of only the suspect’s profile, not the other possible contributor profiles. Although the LR calculation produces false positives there is no calculation of the ‘base rate’, no calculation of the probability of a suspect fitting the evidence by chance.  If you happen to have one of the possible profiles that might be in the mixture, the LR will provide compelling evidence that the DNA could have come from you (according to those who believe the LR), even though you are not a contributor.   The scope for false accusations is obvious.

Remember the two questions asked about the COVID test?  In DNA those become;

3.       IF it is the suspect’s DNA, what is the probability of a ‘match’ (positive results)?

4.       IF it’s a ‘match’, what’s the probability that it is the suspect’s DNA[2]?

Back to virus testing:  If the only known fact, the evidence, is the positive test result, then we have shown here that we need to know much more before we can properly interpret what it means.  A positive result may be very poor evidence of infection.  In DNA profiling, the only fact is the profile result.  Unless we know the base rate, or probability that this is a false positive because many profile from non-contributors will also produce a ‘positive’, it is impossible to properly evaluate the result.

There is more than one explanation for the evidence, i.e. many possible contributors to the mixture. Calculating the LR involves an answer only to the first question, with respect to just the one person.  It is calculating the probability of the evidence, not the hypothesis that it is the suspect’s DNA.   It is ASSUMING what most people believe we are trying to prove; that the suspect’s DNA is there.  This is like assuming that you already have covid before being tested.

Unfortunately, this ‘transposed conditional’ (assuming we are answering question 2) is made again and again in courts throughout the world and cause the evidence to be misunderstood.

[1] Strictly speaking, it is the probability that the DNA came from someone other than the suspect.
[2] Strictly speaking, it is the probability that the DNA came from someone other than the suspect.