1. Introduction
    1. On validation
  2. Good and bad
    1. Mathematical paradigm
    2. Offensive style
    3. Mathematical deficiency
    4. Unprofessional rare haplotype literature
      1. SWGDAM on rare haplotypes
      2. BKW wrongly denies validity
      3. BKW wrongly asserts validity
  3. Discussion
    1. Statement of the problem
    2. State premises
    3. Derive consequences and results
    4. Validate
  4. References

Towards professionalism in forensic mathematics

CH Brenner, University of California at Berkeley School of Public Health, and DNA·VIEW

This essay is based on my September 3 talk, and proceedings contribution, for the 2011 ISFG conference in Vienna Austria.

Abstract

There are various mathematical problems in forensic genetics and they can most clearly and usefully be dealt with through a disciplined mathematical exposition which should be precise and logical — clear statement of the problem and of assumptions, deductive progression of ideas and justification of assumptions. Unfortunately our literature is not consistent in achieving or even aiming for such coherent mathematical standards; instead random scattershot discussions are common and even worse, recipe papers without any foundation at all. I mention some examples in two areas — the "exclusion" method for DNA mixture analysis, and Y-haplotype evidential value — and discuss more specifically some problems and mischief in the literature in the second area.

Keywords: forensic mathematics; forensic statistics; DNA mixture; rare haplotype; refereeing; misconduct


  1. Introduction
  2. DNA identification is interesting in offering, more than any other forensic area, scope for explicit mathematical treatment. Many specific problems have received mathematical attention (variously analysis, discussion, and debate) in the literature, beginning with straightforward linking suspect to unknown DNA profile, adding difficulties and complications such as rare haplotypes, database search or mixed sample, and identification via kinship with its many attendant intricacies. Mathematics as a tool has several potential attractions and advantages. Mathematical writing is explicit in definitions and assumptions, hence should provide clear unambiguous communication. The reader should know what is being claimed. Mathematical exposition is logical and deductive. Ideally the reader is led irresistibly along a linear deductive path. If not irresistibly, at least the exact point of resistance is manifest. Then the reader can say “I disagree with your premise” or can argue that step D doesn’t follow from step C; a productive discussion is then possible with a good chance for resolution of disagreement.

    1. On validation
    2. I believe that the main and possibly the only criterion for validation is that the method is fair to innocent suspects. The test is the innocent suspect – easy point to overlook since we don’t see a lot of them.

  3. Good and bad
    1. Mathematical paradigm
    2. The paradigm should be: State the problem, formulate it mathematically, state premises (inevitably including a model since this is applied mathematics), justify the premises (i.e. validate the model), derive the result.

      I tried to follow the mathematical tradition in my recent paper on rare haplotype evidence and (as I modestly define) the fundamental problem of forensic mathematics [FP]. The main problem is to find the evidentiary value of a previously unseen haplotype linking suspect to crime scene. Expressed mathematically it comes down to a conditional probability that a random innocent person will match. For the evidential likelihood ratio (the reciprocal of that probability) I derive the expression LR=n/(1-κ) where n=size of reference database after extending it with the crime scene type, and κ=proportion of that database that is singletons, which derivation is valid under a stated modelling condition. Then I show that the modelling condition holds, hence the formula is valid, for a wide range of theoretical populations which encompasses the plausible range of real populations.

      No doubt my paper isn’t a perfect example of the paradigm, nor is it a unique example. But disappointingly, contrary examples abound.

    3. Offensive style
    4. A paper which simply gives a recipe for calculation without any stated justification (let alone careful justification) is professionally deficient. We find mixture [B-mix], [S-mix] and rare haplotype papers [S-Yhap] of this sort. Also, there is a prevalent style of sophisticated writing that is entertaining but literally pointless: e.g. “an attractive point of view” (Is it attractive to the innocent suspect who is the victim of the entertainment?), or “An alternative framework is to suppose ... a prior distribution that is conveniently taken to be Beta” [BKW]. Is convenience of statisticians a valid criterion for deciding who goes to jail?

    5. Mathematical deficiency about the “exclusion” method
    6. In the ‘90s the exclusion method for mixtures was simple: If a suspect is “included” then report RMNE, calculated per-locus as the squared sum of the allele frequencies for alleles observed above 100rfu or so. No one actually wrote down the model but the formula is simple enough that it can be reverse-engineered to deduce what the model must be: The formula assumes that all alleles of a donor will be conspicuous (e.g. >100rfu) and “included” means all of ones alleles are conspicuous in the mixture. Obviously this is an absurd model. That it survived and was popular and accepted for years – perhaps still – proves the importance of explicitly writing down models and explicitly deriving and justifying the consequences. With nothing written down, nothing wrong is written down and errors are less obvious.

      The recent appearance of a refined “exclusion” method [B-mix] [S-mix] suggests that RMNE enthusiasts have woken up to the folly and unfairness of the original approach. They have not, though, woken up to the importance of models, let alone to justifying the shibboleths of the RMNE faith such as “easy to understand!”, “needs no assumption about number of contributors!”, “conservative!” These papers give no mathematical analysis at all; only recipes which apparently we are supposed to trust.

      And my point isn’t that the method fails. My point is that the adherents of a method have a positive responsibility to show why it works. They have not done a respectably professional scientific job if they don’t explain coherently. Otherwise the rest of us – reader, analyst in the laboratory, judge, and accused – ought to be suspicious of the validity of the method. And I am.

    7. Unprofessional rare haplotype literature
      1. SWGDAM on rare haplotypes
      2. SWGDAM instructions on rare haplotype matching [S-Yhap] don’t even state a problem but instead begin by proposing how to calculate a haplotype frequency. The reader who realizes that (population) frequency is not (matching) probability and that the evidential problem concerns probability, is left slack-jawed at the post with nothing with which to disagree while the paper gallops off into (irrelevant) mentions of formulas and ideas lacking not only foundation but lacking any chain of reasoning.

      3. BKW wrongly denies validity
      4. Much of the recent [BKW] is a sophisticated discussion of my [FP], resting largely on unsupported opinion (“we believe”, “it is difficult to perceive”), factual errors, incompetent mathematics and logic, and apparently scientific misbehaviour – a performance impossible for any of the able authors individually so I attribute it to the multiple author effect. I’ll review here only its discussion of validity. It concludes with the claim that “we have shown Brenner’s approach ... suffers from potential anti-conservativeness.” Mystified, I consulted the authors and learned that “shown” refers to a section where the paper tries a little math, a hopeful analytical approach for assessing my method, but the analysis runs aground – dead end. I sympathize. My own notebooks are littered with ideas that didn’t pan out. But I don’t try to publish them and I conclude nothing from them except my own limitations. Not so [BKW]. If their analysis goes nowhere it must be a shortcoming of the thing they are trying to analyze! In any case, it was an odd effort. They should instead have read the part of my paper in which I claim to establish validity, and if my argument is convincing it follows that any attempt to debunk validity is bound to fail.

      5. BKW wrongly asserts validity
      6. prima facie implausible — The frequency surveying approach is founded on the tempting intuition that the frequency of a haplotype is correllated with the frequency of its mutational neighbors. "Neighbors" are defined by assuming a single-step mutations, a good model. The evolutionary model though is tragically unstated. The evolutionary intuition may be that neighboring haplotypes replenish one another by repeated mutations. That is reasonable for a one or two locus haplotype, but when the number of loci is large — even seven loci, let alone 17 as is usual today — convergent mutation is very unusual and I do not see how the replenishing phenomenon can be a significant influence compared to the effect of genetic drift. But the unstated model implicitly assumes the opposite, that replenishment dwarfs drift.

        Compared to that the fact that the weighting formula doesn't follow from the mutational model (The W formula, summing N/distance, comes from thin air, not mathematics. Why not N/distance2 or N/edistance? Random guesswork.), or any imaginable model, is a secondary objection. Even if there were no drift, the model would be wrong in giving vastly too much weight to non-immediate neighbors.

        Finally and equally mysterious [BKW] falsely asserts that a simultaneous publication, [Ysurvey], validates an alternative approach, “frequency surveying.” There is nothing about validation in [Ysurvey]. An author of [Ysurvey] says that it includes no validation (M Anderson, pers comm). The method seems [FP] prima facie implausible. An author of [BKW] recalls that the assertion was added at the behest of the referee (B Weir, pers comm). If so the referee overlooked confused reasoning, an algebraic blunder and numerous factual errors in [BKW] but managed to join as anonymous co-author with a mistaken claim that could be important in a judicial setting. A judge might be falsely reassured with unjust effect. Or the defense, armed with the true and unflattering story, might argue convincingly that a breakdown in the peer-review process amounts to a shoddy lack of professionalism about statistical methods in forensic genetics as a whole. That’s harsh reasoning from one incident, but is the conclusion wrong?

    8. Discussion
    9. Important properties and components for a methodology paper in forensic mathematics include

      1. Statement of the problem
      2. Explain the problem first in words. This important step is difficult, but important in order to give reader and writer alike something to latch onto.

        It requires careful thought to find the right words to say exactly what you are trying to do. If those words are missing, that doesn't indicate mere carelessness. Rather take it as a sign of vague and fuzzy thinking, and a warning of more to come.

        If the words are carefully chosen it will be straightforward to give mathematical expression to the evidential value that is the goal.

      3. State premises
      4. In most cases the most important component of the premises will be the model. Writers who skip this step — i.e. most writers — deserve to be maligned because I believe they do so for a combination of bad reasons, of which the most innocuous is laziness — They vaguely imagine that the premises are known to all or are clear from their mathematics. But that's just asking the reader to do their work for them.

        I can imagine a reasonable forensic mathematics paper that gives a good argument for some method without giving a model, just based on statistical analysis of some experiments for example. But it would always be much better with a model. The model provides insight into why the suggested idea works, and it guides understanding into the range of circumstances under which the idea is valid.

      5. Derive consequences and results
      6. Emphasis here is on the idea of derivation, on a coherent deductive chain of mathematical reasoning that draws a conclusion from the premises. In the present context "conclusion" means a set of procedures and calculations for the evaluation of some kind of evidence.

      7. Validate
      8. The premises and model are adequate if the evidential value implied by them is in general fair to an innocent suspect.
          Conservative?

          By the way, the concern in particular for the innocent suspect isn't simply because we care more about them. It is because fairness to the innocent is almost certain, for a subtle but pervasive philosophical reason, to be more difficult than fair evaluation of evidence when the suspect is really guilty. A model is necessarily an idealization, an approximation to reality that omits some details. Omitting details means omitting evidence. For example, in the case of DNA mixture evidence, a simple model doesn't worry about signal intensity. When the suspect is really the donor to a mixture, the intensity information will in practice nearly always be further evidence so ignoring it is doing him a favor, being "conservative." But for an innocent suspect the tendency is exactly the opposite: the more we look at details the more we are likely to see things revealing the truth. That may be why it is very hard to find a simple method — a method based on a very much simplified model — that is fair to the innocent.

      The above points seem pretty obvious; it is hard to imagine anyone arguing that it is ok to omit stating the problem. To see the vital importance of stating premises and giving a model, look at any paper which does not do so and see immediately how chaotic and difficult to evaluate it is. Even for a quite simple paper I find that as I lay out the mathematics explicitly, I am forced to refine my results and conclusions. So I am convinced that slapdash papers — i.e. most papers — are riddled with unsoundness. Those criticized in this essay and many (most?) others are nothing more than guesswork, and the more closely the guesses are examined the more it is apparent that shoddy reasoning leads not just to poorly supported conclusions, but to poor conclusions.

    10. References
    11. [FP] Brenner CH,
      Fundamental problem of forensic mathematics – The evidential value of a rare haplotype, Forensic Sci. Int. Genet. 4 281-291

      [B-mix] Budowle B, Onorato AJ, Callaghan TF, Della Manna A, et al,
      Mixture interpretation: defining the relevant features for guidelines for the assessment of mixed DNA profiles in forensic casework, J Forensic Sci 2009; 54(4): 810–21

      [S-mix] SWGDAM Interpretation Guidelines for Autosomal STR Typing by Forensic DNA Testing Laboratories §3.5. Interpretation of DNA Typing Results for Mixed Samples at
      http://www.fbi.gov/about-us/lab/codis/swgdam-interpretation-guidelines

      [S-Yhap] SWGDAM Y-chromosome Short Tandem Repeat (Y-STR) Interpretation Guidelines. (§5.3. The basis for the haplotype frequency estimation is the counting method. at
      http://www.fbi.gov/about-us/lab/forensic-science-communications/fsc/oct2009/standards/2009_01_standards01.htm/

      [BKW] Buckleton JS, Krawczak M, Weir BS,
      The interpretation of lineage markers in forensic DNA testing, FSI Genetics 5 (2011) 78-83

      [Ysurvey] Willuweit S, Caliebe A, Anderson MM, et al,
      Y-STR frequency surveying method: a critical reappraisal, FSI Genetics 5 (2011) 84-90