The Error of Expected Loss Minimization

Transcription

1 Law, Probability and Risk (2003) 2, 1 7 The Error of Expected Loss Minimization RONALD J. ALLEN John Henry Wigmore Professor, Northwestern University School of Law. Northwestern University School of Law, 357 E. Chicago Ave., Chicago, IL 60611, USA [Received on 4 September 2002; revision received on 6 October 2002; accepted on 8 October 2002] Considerable effort has been expended by legal scholars attempting to apply formal approaches such as Bayes Theorem or decision theory to various aspects of the litigation process. These efforts consistently fail adequately to account for the reality of the phenomenon being investigated. Most recently, Professor Kaye attempted to give another decision-theoretic account of the standard of persuasion in civil cases. This attempt fails formally, as it requires that litigation involve single issues for decision, which is virtually never the case. It fails pragmatically because it ignores all the surrounding context of the decision rule that colours and gives meaning to the decision rule. Jury trials cannot plausibly be analysed as designed to maximize the expected return to jurors. Keywords: burden of persuasion; decision theory; expected utility; errors in litigation; equality; jury trials; algorithms; formalisms; formal reasoning. The inaugural issue of this journal contained an article by Professor David H. Kaye (Kaye, 2002) closely scrutinizing a few passages from a previous edition of my course book on Evidence (Allen et al., 1997). 1 Iampleased to have contributed even indirectly to the success of the journal s efforts, and even more pleased to have been the grain of sand stimulating yet another pearl of legal scholarship from Professor Kaye. 2 Icannot resist, though, the effort of a little polishing. Professor Kaye s article suggests that there is an opposition between three different conceptions of the policy underlying standards of persuasion: error equalization, error minimization, and expected loss minimization. I do not see the opposition this way, even though the passage cited by Professor Kaye plausibly could be read as he does. Rather, I see the opposition between error administration and expected error administration, i.e. between the first two lumped together and the third. Professor Kaye is correct that error rjallen@northwestern.edu 1 The current edition (Allen et al., 2002) contains some but not all of the scrutinized passages, and omits the ones technically troubling to Professor Kaye concerning the implications of his previous work on error rates and expected losses. 2 The ongoing discussion between Professor Kaye and myself about the relationship between formal reasoning and the legal process has been characterized as increasingly vituperative. Murphy, 2001, at 577, n. 24. Personally, I have found it increasingly invigorating, fruitful, and enlightening. External observations may differ, of course, but I want to be clear that I have the highest regard and admiration for Professor Kaye s work even if I disagree with some of the implications he draws from it. If anything I have written bears any other reasonable interpretation than my vigorous disagreement with some of those implications and my explanations for so disagreeing, I wish to disavow such remarks and express my apologies. c Oxford University Press 2003, all rights reserved

2 2 R. J. ALLEN equalization and error minimization are not the same, but one can see their consanguinity clearly in the graphs that we produce in the previous and present editions of the course book. If, to take but one example, the sets of deserving plaintiffs and defendants are of equal size, and probability assessments by jurors range in normal distributions over those sets as we have drawn them, the 0 5 standard will allocate errors equally over the parties and reduce the total number of errors. Equally plainly, however, modification of either assumption demonstrates that error equalization and error minimization can be at odds. Further, as we point out explicitly (Allen et al., 1997, p. 829), outcomes in the real world comprise an empirical, not analytical, question. In class, these are precisely the points I and my co-authors make by modifying the graphs appropriately, which facilitates examining the multi-faceted meaning of equality and the exploration of a difficult policy question. In the current version of the text (Allen et al., 2002, at 822), we reiterate many of these points and in addition note that one implication of the equivalency notion... is that there should be roughly the same number of errors made for plaintiffs as for defendants. That we are presenting one implication implies in turn that there are others. Perhaps this is too cursory for a set of teaching materials for a first course in evidence, although our experience is that it works fairly well. Nonetheless, we will consider expanding this discussion to clarify some of these matters when and if we do another edition. There is an important point lurking here beyond the architecture of teaching materials, though, which is how the legal system does or should conceive of its goals. Professor Kaye rejects error equalization, yet he previously defended error minimization (Kaye, 2000, at p. 262). This is at least superficially curious, because whether any legal scheme accomplishes either is purely an empirical, not an analytical question. For example, assume no factually deserving plaintiffs ever go to trial, and that fact finders at least once have found the probability of a plaintiff s case to be > 0 5. Plainly, the error minimization rule here would be to set the standard of persuasion higher than the highest probability that would ever be assigned to a plaintiff s case. And so on. To avoid problems like this, his earlier defense of error equalization rests upon the empirical assumption of accurate fact finder assessments of the probability of liability, an empirical assumption like all such assumptions which may be false. In short, his defence of error minimization is much like our exposition of error equalization in that it demonstrates clearly when that is likely to be the result and when it is not. Still, the important point is that it is not obvious how the legal system should handle conflicts between error minimization and equalization. At the extreme, a rigorous policy of error minimization may exacerbate social perceptions of inequality among classes of various kinds, while at the other extreme a rigorous policy of error equalization may vastly increase erroneous outcomes, with attendant social costs. We find this a critical question to examine with our students, and found our manner of discussing Professor Kaye s work a useful way to begin uncovering some of its implications. But, he has objected to our so using his work, an objection that resulted in the offending reference to it removed from the current edition (Allen et al., 2002). He objected, and reiterates the objection in his article, in part because he thinks that our discussion of his work confused expected loss minimization and error administration. He also finds our policy discussion to be opaque and unconvincing. I will first clarify the meaning of our remarks about the policy underlying the preponderance of the evidence

3 THE ERROR OF EXPECTED LOSS MINIMIZATION 3 standard, and thus at least remove the opacity if not the unpersuasiveness, and then briefly respond to the state of our confusion. Professor Kaye recites our sentence that The preponderance rule incorporates an underlying assumption concerning the participants in litigation that plaintiffs and defendants as a class generally ought to be treated equivalently. He goes through various critical constructions of this sentence that he finds implausible, and then at the end raises one final possibility that may have some promise to the effect that before learning anything about a civil action... the fact-finder should assume that plaintiff is as likely to be right as defendant. This is close. What we meant is that the preponderance standard reflects systemic goals to treat those coming before it with equal respect, and that the most intuitively compelling implementation of that goal is to structure things so that roughly the same number of errors are made against deserving plaintiffs and defendants. In class, but admittedly not in the book, we then go on to test the limits of that policy by introducing the potential conflict between error minimization and equalization as discussed above. Unfortunately, systemic goals do not come prepackaged, and rarely are explicitly articulated, but this one can be tested. We predict that if those who create and administer the legal system were asked, they would overwhelming subscribe to the view that the legal system should treat plaintiffs and defendants equally in the sense described above, and that at some point that policy should begin to yield to an error reduction policy. We predict further that, even when explained, they would overwhelmingly reject the expected loss approach that Professor Kaye defends. In fact, we predict that even those for whom the approach would be the most congenial law professors would also reject it. I even think Professor Kaye would reject the expected loss approach when push comes to shove. Suppose there were good reason to believe that fact finders applying the preponderance standard (and thus minimizing their expected loss ) systematically were making mistakes in some set of cases. Suppose further that there were good reason to believe that modifying the burden of persuasion would increase accuracy with no significant offsetting secondary consequences, whatever the effect on expected loss. I predict that virtually all observers would opt for modifying the burden of persuasion rule in an effort to increase accuracy, as the New York Court of Appeals did in Schechter vs. Klanfer, 269 N.E. 2d 812 (NY 1971), and as courts routinely do in res ipsa loquitur cases and through innumerable instructions on presumptions and inferences. If Professor Kaye agrees, he needs to explain how that is consistent with his strong emphasis on expected loss as the policy underlying burdens of persuasion. If he does not agree, I think we would find him to be in a very small minority within the legal profession. I cannot speak for my co-authors, but I would be pleased to work with Professor Kaye and the editors of this journal in conducting just such an empirical test. This takes us to Professor Kaye s point that perhaps my co-authors and I confused error equalization and expected lost minimization. We did not. We used Kaye s work to show one of the set of conditions under which error equalization would occur (and then from that point discuss in class error minimization and the empirical nature of the question). We did so because viewing the policy underlying the preponderance standard as expected loss minimization is implausible. It is inconsistent with the actual structure of the system, and its adoption would not improve matters. Indeed these are the reasons the legal system personified would reject his arguments, and I turn to them now.

4 4 R. J. ALLEN The structure of litigation in the United States is dominated by the implications of trial by jury. There is literally no evidence that trials are structured to permit juries to decide in order to minimize their expected loss apart perhaps from the preponderance of evidence standard itself. Quite the contrary, jurors are told to find the facts (expected loss is never mentioned), told to suppress their emotions (subjective expected utility is never mentioned), to abide by the law (no matter what they may think of it), not to import their own views of policy into cases (no matter how misguided the policy may be), shielded from substantial amounts of data (thus treated as potentially wayward children in need of strict guidance), yet bombarded with more data than any human being could digest and process algorithmically (thus deserving of greater confidence than our best supercomputers), forced into an extremely strange decision making process (make of this what you will), crippled with weird cognitive constraints (such as consider this evidence for impeachment but not on the merits), and so on. Out of this is supposed to come expected loss minimization because of a single instruction that the decision rule is a preponderance of the evidence? It doesn t seem plausible. What does is that the preponderance rule manifests the general social policy to be fair and evenhanded to all the parties, as jurors standardly are explicitly instructed, and equal treatment is incontrovertibly one critical component of fairness. In short, the reality of the trial process provides strong evidence contradicting the proposition that it is designed to achieve expected loss minimization from the juror s point of view, which in turn is evidence relevant to the understanding of the preponderance instruction. And if expected loss, not some form of error administration (equality of errors/reducing the total number or magnitude of errors) is at play, why are there so many evidentiary adjustments to free proof created by the rules of evidence? Why do we have such analytically misshapen creatures as the hearsay rule and character evidence rules, and why can evidence be excluded for unfair prejudice if what s at stake is minimization of expected loss instead of accurate adjudication? And why are there so many adjustments to the burden of persuasion in the form of affirmative defences, presumptions, and instructions on inferences of varying kinds? As Professor Kaye points out, accurate adjudication cannot be treated algorithmically, which implies that adjustments to accomplish it cannot be, either. That means in turn that one would expect to see analytically misshapen creatures as the system lurches along in the light of whatever information it (the people running it) thinks it has concerning the primary goal of accurate adjudication. None of this makes very much sense if the goal is to minimize expected loss. All that requires is a decision maker with a utility function. To be sure, as I have noted previously, maybe there are other relevant points of view (Allen, 1997, p. 346). Perhaps the preponderance standard exists because those who create and implement the legal system think it will minimize loss from their point of view, but Professor Kaye has never to my knowledge defended such a proposition, and so I put it aside. Moreover, it is logically possible, but it seems to me quite unlikely, that perhaps all of the rest of the system except the preponderance standard is designed with error administration in mind, but the preponderance standard shifts gears and is justified by expected loss minimization. Cases like Schecter, doctrines like res ipsa loquitur. and

5 THE ERROR OF EXPECTED LOSS MINIMIZATION 5 instructions on presumptions and inferences are to the contrary, but perhaps there is more to be said about the matter. 3 Perhaps the legal system is wrong, and should adopt expected loss minimization as a justification. I doubt it, and doubt it would. Professor Kaye s article criticizes the passages in our book for being neglectful of the empirical issues latent in error minimization, yet he neglects a critical empirical aspect of expected loss minimization. Expected loss minimization can bear any relationship whatsoever to error minimization; whatever the relationship is, it is empirical, not analytical. A system set up to minimize expected loss may make an error with respect to every single decision that it reaches (what if no deserving plaintiffs ever go to trial, yet jurors assess the evidence to favour them in each case?). The advantage of expected loss minimization is that it can be treated algorithmically, as Professor Kaye has demonstrated numerous times. The disadvantage of error administration is that it is empirics all the way down. But, so, too, is the relationship between expected loss minimization and errors. Thus, the system can embrace the algorithmic approach of expected loss minimization or it can attend to the empirics of actual decision making in an effort to increase accuracy, decrease error, treat parties as equal before the law, or pursue whatever other policies may commend themselves but it cannot do these latter algorithmically. As I said above, I suspect once fully explained the choice between these two would be easy for virtually all relevant actors comprising the legal system. In addition to neglecting an empirical point, the argument about expected loss minimization also neglects a significant analytical point as well. The various mathematical demonstrations that the preponderance standard will minimize expected loss only work on the assumption that there is a single issue to be decided. In Professor Kaye s most recent article (Kaye, 2002, p. 6), this is articulated as p be[ing] the probability that the true facts are such as to warrant recovery for plaintiff. But in no jurisdiction in the United States does the preponderance standard apply to the probability that the true facts are such as to warrant recovery for plaintiff. It applies instead to each element of the cause of action, and I know of no cause of action that has a single element. Indeed, many have a large number, as a cursory review of any book on standard jury instructions will attest. If the preponderance standard is applied to each element, and there is more than one, it will only be by chance that the probability of the conjunction will also exceed 0 5. This means that the expected loss minimization approach as it applies to every jurisdiction within the United States falls prey to the very same critique that Professor Kaye provides of error minimization: to wit, the consequences of the preponderance standard for expected loss minimization is purely an empirical, not an analytical, question. 4 3 Perhaps we are having a misdirected linguistic dispute that is not adequately distinguishing purpose from effect. I am asserting something about the intentionality of human beings; perhaps he is only asserting something about logical consequences of a decision rule. The effect of the decision rule may be what he asserts, although it may be employed to effectuate the purpose that I think animates the legal system. The question then might become what is instrumental to what. I don t deny any aspect of his formal proofs (although as I point out below, his proofs do not apply to the legal system in the United States in which all jurisdictions apply the preponderance standard to each element rather than to their conjunction); instead, I think the evidence indicates that the various procedural mechanisms are employed to advance numerous policies, chief among them error administration. I return to this point at the end of this response. 4 See Kaye (2002, at p. 4). This objection is clearer in other work by Professor Kaye, in particular Kaye (1999, at 14 16).

6 6 R. J. ALLEN Consequently, in addition to being implausible as a goal of the legal system, expected loss minimization as it would play out in real life does not provide any of the benefits of being algorithmic that seem to be its primary source of interest for legal theorists; Professor Kaye s rigorous analysis simply does not apply to the very phenomena under examination. 5 In some cases by chance someone s expected loss might be minimized, but normally no one s would be. If the system must immerse itself in the waters of empiricism regardless of whether it pursues expected loss minimization or error administration, it may as well pursue the latter that plainly has considerably stronger moral and political justification, which in turn may very well explain what we actually see in the system. 6 Frankly, I thought these points so convincing that for years I took the work dealing with expected loss minimization (not just Professor Kaye s his is just the best, which is why we cited it) as providing interesting heuristics giving an effective way to talk about and examine the implications of error administration. That was apparently wrong, which is why the present edition of our book removes the offending reference to Professor Kaye s work. I nonetheless think his work critically important to understanding all the matters discussed in our ongoing exchange, and more important for understanding the nature of the legal system. If there is a disagreement between us, it seems to be precisely as to what our separate observations of the legal process imply about its teleology. I am curious why the points I have made above are not adequate to establish that the legal system is not structured to reduce the expected loss of jurors, and the preponderance standard as presently administered would not do it standing alone, but instead is struggling to deal with errors. Perhaps the editors of this journal could ask for his comment on these points. Iwould find the answer interesting (even if discomfiting), and I suspect the community of evidence scholars would as well. But perhaps I misunderstand his basic point. Perhaps his basic point is simply to correct what he views as a technically incorrect discussion, and not to make any larger claims about the operation of the legal system. In that case, there is not much of a disagreement between us, except perhaps about how much detail on what evidence teachers would view as a 5 There has been one attempt to establish logically that the conjunction issue does not pose a problem for probability approaches akin to Professor Kaye s, and that is A. P. Dawid, The Difficulty About Conjunction (1987). Dawd s analysis simply makes a mistake about what the question is. He analyzes the implications of multiple witnesses testifying to the same event he analyzes the implications of corroboration of witnesses, in other words. The conjunction problem involves the implications of multiple elements to determine liability. The two are quite different, and thus what he demonstrates has virtually nothing to do with the conjunction problem that I and others have analyzed. There have been two other recent attempts to explain why the conjunction problem may not be a problem functionally. They are Stein (2001); and Levmore (2000). Both are interesting, but neither disposes of the conjunction issue adequately. I am presently preparing an analysis of both that explains why that is so. More to the point, neither yields support for Professor Kaye s argument. Even if Levmore or Stein is correct, it simply means that by the fortuitous confluence of unrelated events the conjunction issue does not impose costs on the system. Professor Kaye s argument is that the preponderance standard has certain formal characteristics with certain logical consequences. He is right that it does, but he neglects that the actual operation of the legal system is inconsistent with the necessary conditions for those formal characteristics to have those logical consequences. That they flow by happenstance is an entirely different matter, and would make rather plain that the purpose of the preponderance standard is not to minimize expected loss. Now the argument would have to be that someone had the purpose of responding to the actual structure of the legal system by requiring the preponderance standard because its idiosyncratic interaction with other peculiarities of the system will result in expected loss minimization. 6 The elemental structure of liability poses difficulties for all efforts to explain the proof rules. This is another reason why close parsing of a single aspect of the complex structure of litigation is not terribly useful.

7 THE ERROR OF EXPECTED LOSS MINIMIZATION 7 peripheral (to teaching) issue should be included in a course book (and perhaps on whether technical arguments should be used metaphorically). More globally, my concerns are with how the technical arguments map onto the actual operation of the legal system. When Professor Kaye turns his attention to the actual operation of the legal system, he writes things that could pass for my own. For example: I believe that the law s best strategy is to formulate rules of evidence that promote the accuracy of factual determinations and to encourage triers of fact to decide in accordance with their best-informed and best-justified assessments of the probabilities that the totality of the material facts is such as to warrant recovery. (Kaye, 2000, at p. 262). He goes on to defend this on the empirical ground that this is the most likely means of reducing error over the long haul. I largely agree with that, although I would be quite willing to make adjustments, including adjustments to the burden of persuasion áala Schecter as we proceed over the long haul if I thought circumstances warranted it (and thus that expected loss minimization is instrumental to error administration). If he agrees with that, then after much wrangling we have discovered ourselves at (and perhaps never left) the same position. REFERENCES ALLEN, R. J Reasoning and its foundations: some responses. International Journal of Evidence and Proof, 1, 343. ALLEN, R. J., KUHNS, R. B.& SWIFT, E Evidence: Text, Cases, and Problems, 2nd edn. New York: Aspen. ALLEN, R. J., KUHNS, R. B.& SWIFT, E Evidence: Text, Cases, and Problems, 3rd edn. New York: Aspen. DAWID, A.P.1987 The difficulty about conjunction. The Statistician, 36, 91. KAYE, D.H.2000 Bayes, burdens and base rates. Int. J. of Evidence and Proof, 4, 260. KAYE, D.H.2002 The error of equal error rates. Law Probability and Risk, I, 3. KAYE, D.H.1999 Clarifying the burden of persuasion: what Bayesian decision rules do and do not do. Int. J. Evidence Proof, 3, 1. LEVMORE, S.2001 Conjunction and aggregation. Mich. L. Rev., 99, 723. MURPHY, P. W Teaching evidence, proof, and facts: providing a background in factual analysis and case evaluation. J. of Legal Education, 51, 568. STEIN, A.2001 Of two wrongs that make a right; two paradoxes of the evidence law and their combined economic justification. Texas L. Rev., 79, 1199.

8