Careful Formulation of a Likelihood Ratio Statement based on Anecdotal Evidence

The problem

Background

A DNA profile as evidence linking two things (i.e. suspect to crime scene) is a situation well suited to formulation of a likelihood ratio (LR) because the DNA profile is
1. well defined — we know what the profile is and hence what a "match" means, and
2. nicely quantifiable — most importantly we can compute with reasonable accuracy the probability of the random occurrence of a profile
Other kinds of evidence are not so nice. Especially anecdotal evidence – non-scientific evidence – is likely to be deficient with respect to both of the above criteria. Still, I think it can sometimes be usefully treated in a Bayesian way. Inevitably we will have to guess the strength of the evidence partly through intuition. But still there's scope for exacting analysis. Namely, it's important to formulate accurately the questions to be answered, so that at least we apply our subjective intuition to the correct questions.

ProBusqueda

In our ProBusqueda work, we have many cases of young adults who were lost to their families in the chaos of military fighting in El Salvador, lost their birth identities (BI), were adopted, and now as young adults (YA) wish to reconnect with their birth family. Sometimes DNA is sufficient to make a confident link between YA and BI, but sometimes it falls short. Always there is other connecting evidence of various kinds in which case we would like to do the impossible and quantify it in order to make a good decision as to whether the non-DNA evidence is sufficient to make up the shortfall between the DNA evidence and confident identification.

(anecdotal) example

Young adopted adult "Carter" bears scars.
Family of birth name/missing child "Nestor" mentions neck scars from vaccination & insect bites.

Analysis

What is the evidence?

Is the evidence that

Carter and Nestor both have scars?

Let's be more precise. The evidence is that

Carter has scars and relatives report that the infant Nestor had scars.

More carefully, let's describe the evidence as E_C & E_N, where

	EC =	Carter has scars of a certain description S.
	EN =	Nestor's family describes scars of a certain description S' (similar to but certainly not identical description to S)

What is the LR?

Then the LR = X/Y where

X = Pr(E_C & E_N | Carter is Nestor)
Y = Pr(E_C & E_N | Carter is not Nestor).

Do the math

... doesn't depend on the relationship

The way people describe Carter is not biased by his true history (especially if unknown).

Note: E_C doesn't depend on the relationship, so Pr(E_C | Carter is Nestor) = Pr(E_C | Carter is not Nestor) = Pr(E_C).

Consider X. There are two ways to apply the identity Pr(F & G) = Pr(F)Pr(G|F). In this case I choose an infant-centric formulation by letting E_C play the role of F:

X = Pr(EC & EN | Carter is Nestor) = Pr(EC | Carter is Nestor) Pr(EN | Carter is Nestor & EC) = Pr(EC) Pr(EN | Carter is Nestor & EC). And Y: Y = Pr(EC & EN | Carter is not Nestor) = Pr(EC | Carter is not Nestor) Pr(EN | Carter is not Nestor & EC) = Pr(EC) Pr(EN | Carter is not Nestor & EC). So LR=Pr(EN | N*) / Pr(EN | ~N*), where N* = Nestor is a childhood version of Carter, a person who bears scars S.

... apply the identity The mathematical rule for the probability of a conjunction in terms of the probabiltiies of the constituents. Example: The probability that a person is F=over fifty and G=unemployed can be computed if you know Pr(F) = proportion of people over fifty Pr(G|F) = proportion of unemployed among those over fifty. Now just multiply. Alternatively, it could be calculated from Pr(G) = proportion of unemployed Pr(F|G) = proportion of over-fifties among those unemployed. Now multiply.

Conclusion

Let's rephrase:

Evidence EN = Relatives remember and report scars S' for the child Nestor hypothesis H1 = the child is the same person as the adult who now has scars S hypothesis H0 = the child is random person LR = Pr(EN | H1) / Pr(EN | H0).

or in words: How many times more probable is EN when H1 is true, than when H0 is true?

Interpretation

The LR is thus a comparison of these two probabilities:

	Pr(EN | H1)	= The probability that relatives of a person with scars S would report childhood scars S'
versus
	Pr(EN | H0)	= The probability that relatives of some random infant would recall childhood scars S'.

No doubt the first probability is larger than the second one, hence LR>1. But unless the scars are fairly striking – obviously unusual – or the coincidence between the descriptions S and S' very strong, I would be reluctant to conclude that LR is very large.

An issue of bias

It's not clear, from the story I've presented, how the testimony from the relatives came about. It could be that the relatives spoke of the scars S' after being prompted in some way. That is, suppose that the actual evidence is a biased version E_N' of E_N:

Evidence EN' =	Relatives remember and report scars S' for the child Nestor after being given a suggestion in the form of information or a picture of Carter.

It may be that the bias makes the denominator much larger, i.e. that Pr(E_N' | H₀) >> Pr(E_N | H₀). (Maybe numerator too, but probably less so.) It may be that a very high percentage of children might be described as having scars if the survey is in the form of a leading question. The way the information is collected may have a big effect and consequently the actual LR from the anecdote may be not very big, not very helpful in concluding identity.

Exercises

Age coincidence

As an exercise, consider the following data:
• E_N: Nestor was born in 1977 and disappeared in 1982.
• E_C: Carter's belief that he was born about 1978 and was adopted in 1982
Hypothesis H₁ = the child is the same person as the adult who believes E_C
Hypothesis H₀ = the child is random person

We have some survey data based on 698 missing children:

• 19 children aged 5 disappeared in 1982. (That's 1 in 36 of the missing.)
• 58 children aged 4–6 disappeared in 1982. (1 in 12 of the missing.)
• 114 children aged 4–6 disappeared in 1981-1983. (1 in 6 of the missing.)

If Carter is not Nestor, we can assume that his possible birth identities are represented by some part of the sample of 698. Depending on how much vagueness about dates and ages we accept, there are from 19 to 114 out of those 698 that are consistent with his belief. Therefore the LR supporting identity based on this data is

6 ≤ LR ≤ 36.

LR shortfall or requisite prior

Let's consider the above analyses in this context:
• We estimate 30000 missing children altogether. Therefore we start with a "baseline prior probability" of 1/30000 that Carter=Nestor.
• We would like to maintain a standard of 99.9% (posterior) probability supporting an identity in order to assume it.
• Since 99.9% is 1000:1 odds and 1/30000 probability is 1:30000 odds and
  (posterior odds) = (prior odds) × LR
  the necessary total evidence LR_T that we require must satisfy
  LR_T = (posterior odds) ÷ (prior odds) ≥ 1000 ÷ (1/30000) = 30,000,000.

  Considering the total evidence LR_T to be composed of two factors, scientific (meaning DNA) and anecdotal (everything else), we have

  LRT	= LRDNA × LRother
  	= 11,700 × LRother
  	≥ 30,000,000, so we need
  LRother	≥ 30,000,000/11,700 = 2564.

• LR_other ≥ 2564 means that the DNA evidence leaves us with a LR shortfall of 2564.
Can we do it? Suppose we figure
• LR_other = LR_sex × LR_age × LR_scar, where
• LR_sex = 2 since Nestor was male as would be expected if he is Carter
• LR_age = 12 (say), from the age and date discussion above
• LR_scar represents the evidence from the coincidence about scarring.
Then to bring LR_other ≥ 2564, we need to believe that

LR_scar ≥ 2564 / (2 × 12) = 107. That's believable, but it's not obvious! Might depend on the details.