Academia.eduAcademia.edu
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 Blind expertise and the 2 3 problem of scientific 4 5 6 evidence 7 8 9 10 By John Danaher* 11 PhD Candidate, University College Cork 12 13 Abstract Scientific evidence presents a problem for the courts: the 14 subject-matter is often complex, the experts who present the evidence can be 15 cherry picked and biased; and judges and juries are frequently unsure about 16 how to weight the evidence once it has been presented. This article diagnoses 17 the problems associated with scientific evidence and then proceeds to consider two possible solutions to those problems: (1) the reliability test solution; and (2) 18 the blind expertise solution. The former is currently favoured by law reform 19 agencies in Ireland and England, but the primary focus of this article is on the 20 latter. It is concluded that the blind expertise solution has considerable 21 attractions and should be seriously considered as a reform option. 22 Keywords Scientific evidence; Expert witnesses; Reliability; Blind expertise 23 24 aw reform agencies in both Ireland and England have recently L 25 proposed reforms to the rules on the admissibility of scientific 26 evidence.1 They have done so partly because of confusions in the 27 28 29 *BCL (UCC), LLM (Dub). Email: johndanaher1984@gmail.com. The author would like to thank the Irish Research Council for the Humanities and Social Sciences for their funding of his research 30 from 2007–10. He would also like to thank Dr Mary Donnelly and Dr Shane Kilcommins, as well as 31 two anonymous referees for comments on earlier drafts. 32 1 For England, see Law Commission, The Admissibility of Expert Evidence in Criminal Proceedings in 33 England and Wales, Law Commission Consultation Paper No. 190 (April 2009) (hereinafter ‘Admissi- bility of Expert Evidence’), available at <http://www.justice.gov.uk/lawcommission/docs/cp190_ 34 Expert_Evidence_Consultation.pdf>, accessed 3 May 2011; and Expert Evidence in Criminal Proceedings 35 in England and Wales, Law Com. Report No. 325 (March 2011) (hereinafter ‘Expert Evidence in Criminal 36 Proceedings’), available at <http://www.justice.gov.uk/lawcommission/docs/lc325_Expert_Evidence_ 37 Report.pdf>, accessed 4 May 2011. For Ireland, see Law Reform Commission of Ireland, Consultation Paper on Expert Evidence (LRC CP 52–2008) (hereinafter ‘Consultation Paper on Expert Evidence’), in 38 particular ch. 2 proposing the introduction of a reliability test. The Law Reform Commission of 39 Ireland Consultation Paper is available at <http://www.lawreform.ie/_fileupload/consultation% 40 20papers/cpExpertEvidence.pdf>, accessed 3 May 2011. 41 doi:10.1350/ijep.2011.15.3.378 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF (2011) 15 E&P 207–231 207 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 existing legal positions in both jurisdictions,2 and partly because of the obvious 2 difficulties courts face when confronted with such evidence: the 3 subject-matter is often complex and esoteric; the experts can be cherry-picked 4 and biased; and courts and juries are consequently unsure of how to weigh the 5 evidence properly in their deliberations. 6 7 The suggestion in both jurisdictions is that a formal reliability test needs to be 8 introduced. This test would provide a judge with a checklist of (seven or eight)3 9 criteria that are thought to be indicative of reliable scientific evidence. If the 10 evidence presented to the court satisfies a sufficient number of these criteria,4 11 then the evidence is admissible and can be considered by triers of fact when they 12 make their decision. If the evidence does not satisfy a sufficient number of the 13 criteria, then it is inadmissible. 14 15 While such a proposal has considerable merits—some of which are addressed 16 below—it is not the only way to deal with the problem of scientific evidence. This 17 article considers an alternative proposal: the introduction of a system of blind 18 expertise.5 This is a proposal that tries to bring the blinding protocols that are 19 already used in scientific inquiry to bear on the acquisition and presentation of 20 scientific evidence to a court of law. 21 22 In order to address the strengths and weaknesses of this proposal, the article 23 proceeds in the following manner. The first part attempts to offer a reasonably 24 precise diagnosis of the problems arising from the presentation of scientific 25 26 27 2 This appears to have been a major motivation for the English proposal. The feeling seems to be that the current position, set down by King J in R v Boynthon (1984) 38 SASR 45 at 46–7, is unclear. The test 28 and its application in English law is analysed in Admissibility of Expert Evidence, above n. 1, Part 3. See 29 also A. Roberts, ‘Rejecting General Acceptance, Confounding the Gate-keeper: The Law 30 Commission and Expert Evidence’ [2009] Crim LR 551, who comments on the problems with the 31 Boynthon test. 3 See Consultation Paper on Expert Evidence, above n. 1 at 129 and Expert Evidence in Criminal Proceedings, 32 above n. 1 at para. 5.35. The Irish proposal has seven elements; the English has eight elements 33 (although there could be some dispute about how these are counted). 34 4 What counts as a ‘sufficient number’ is a judgment call for the court. See Expert Evidence in Criminal Proceedings, above n. 1, at paras 3.49–3.52, commenting on the need for flexibility in applying the 35 test. 36 5 Discussion will be based primarily on the proposal from C. T. Robertson, ‘Blind Expertise’ (2010) 85 37 New York University Law Review 174 and the work of Roger Koppl (and his colleagues), e.g. R. Koppl, R. 38 Kurzban and L. Kobilinsky, ‘Epistemics for Forensics’ (2008) 5 Episteme: A Journal of Social Epistemology 141; R. Koppl, ‘Epistemic Systems’ (2005) 2 Episteme: A Journal of Social Epistemology 91; and R. Koppl, 39 ‘Romancing the Law: Legal Failure in Forensic Science Administration’ in E. Lopez (ed.), Government 40 Failure in the Legal System (Palgrave MacMillan: New York, 2010). Koppl does not focus purely on the 41 idea of ‘blinding’, but his proposals do include a blinding element and are highly relevant to 42 addressing the problem of scientific evidence. 208 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE evidence. The second part, considering this diagnosis against the backdrop of the 1 rules of evidence as a whole, provides some parameters for measuring the success 2 of any proposed reforms to this area of the law. Two proposals are then outlined 3 for the introduction of blinding protocols in the acquisition and presentation of 4 scientific evidence. The fourth part of the article analyses and evaluates these 5 proposals in light of the parameters set down in the second part. The final part 6 concludes by suggesting that the blind expertise proposals have considerable 7 merits, but, as is the case with everything else, should not be considered a 8 panacea. 9 10 What is the problem of scientific evidence? 11 12 To diagnose the problems associated with scientific evidence, it will be useful to 13 think of the relationship between those who present evidence in court and those 14 who interpret the evidence as a signalling game.6 In other words, to conceive of it 15 as a strategic interaction between the senders and receivers of information. This 16 strategic interpretation will allow us to see the potential impediments to truth 17 that arise with evidence law in general and scientific evidence in particular. 18 19 A basic signalling game7 would have two players, a sender and a receiver, and two 20 sets of actions. For instance, the sender might have the option to send one of two 21 signals to the receiver (Sig 1, Sig 2). These signals would convey information about 22 the state of the world. This information would function in such a way as to raise or 23 lower the (subjective) probabilities that the receiver attaches to his or her beliefs 24 about the world. Since there are only two signals, we can assume that one of them 25 represents the world as being in State 1, while the other represents it as being in 26 State 2. The receiver must act in response to these signals. We can assume, for 27 simplicity’s sake, that there are only two acts, each of which is an appropriate 28 response for the different states (Act 1, Act 2). An appropriate response leads to a 29 positive payoff; an inappropriate response leads to a negative payoff. Obviously, 30 the goal for the receiver in any game of this sort is to work out which signal repre- 31 sents which state so that he can respond appropriately.8 32 33 34 35 6 Partly because Robertson puts it this way in his essay on blind expertise, as does Koppl in his work. 36 See materials cited above in n. 5. 37 7 Taken from B. Skyrms, Signals (Oxford University Press: Oxford, 2010) ch. 1. Skyrms works from the 38 basic model set down in D. Lewis, Convention (Oxford University Press: Oxford, 1969). 8 An interesting observation from evolutionary game theory is that provided the players can switch 39 roles, play the game repeatedly and have similar payoffs, a system of communication can emerge 40 that is purely conventional and yet can track the actual state of world with reasonable accuracy. 41 This issue is explored at some length throughout Skyrms, above n. 7. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 209 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 The presentation of evidence to a court of law can easily be conceptualised as a 2 signalling game. In this case, the court is the receiver, and the parties to the 3 litigation are the senders. They present signals (evidence) to the court, and the 4 court works out the relationship between those signals and the actual state of the 5 world (i.e. the facts of the case). The relevant signals will be those that raise or 6 lower the probability that the court attaches to the factum probandum (fact to be 7 proved); the irrelevant signals will be those that do not raise or lower the proba- 8 bility that the court attaches to the factum probandum. If the court works out the 9 relationship between the signals and the state of the world, and can attach appro- 10 priate weight to those signals, they can proceed to apply the principles to the facts 11 in the most legally sound manner. 12 13 The court faces two obstacles to successful performance in this game. The first 14 obstacle is that the signals can be complex and their relationship to the actual 15 state of the world can be difficult to work out.9 This gives rise to something we 16 shall call the problem of incomprehensibility. The second obstacle is that the interests 17 of the various participants in the game are not always aligned: while the court 18 might be interested in getting at the truth so as to serve the interests of justice, the 19 lawyers representing the different sides are primarily concerned with serving the 20 interests of their clients. This can lead to poor quality, biased signals being sent to 21 the court. These signals can have limited, if any, relevance. We shall call this the 22 problem of bias. Let us look at both of these problems in more detail. As we do so, it 23 should become clear why they are particularly acute when it comes to the presen- 24 tation and analysis of scientific evidence. 25 26 Consider first the problem of bias. We can take a classic biblical example as a 27 guide: the infamous judgment of Solomon in the child custody dispute between 28 the two prostitutes.10 The story is recounted in 1 Kings (3:24–28). Both women 29 claimed to be the mother of the same child and they asked Solomon to resolve 30 their dispute. Solomon had to base his judgment on the signals that he was 31 receiving from each of the women. But from his perspective the signals were 32 initially indistinguishable: they both claimed to be the mother and there was no 33 obvious reason to favour one claim over the other. He needed to do something to 34 break the deadlock, to differentiate the true signal from the false one. So he asked 35 for a sword to be brought so that he could divide the child in two and give half to 36 each woman. In response to this, one of the women pleaded with Solomon to spare 37 the child’s life and give it to the other woman. The other woman told him to go 38 39 9 This may be due to the fact that the world is itself complex. 40 10 I take this from A. Dixit and B. Nalebuff, The Art of Strategy (WW Norton: New York, 2008) 237–8. A 41 more complex game theoretic analysis of the story can be found in S. Brams, Biblical Games (MIT 42 Press: Cambridge, MA, 2003, originally published 1980) 118–23. 210 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE through with it. This allowed Solomon to see that the first woman was the true 1 mother (since she was protecting the life of the child) and so he could resolve the 2 dispute appropriately. 3 4 The obvious lesson to be derived from this story is that the adversarial system is 5 sometimes uniquely well suited to producing low quality, biased signals. That is, 6 signals that do not raise or lower the probability of the fact to be proved. This is 7 because only evidence that supports the argument of the sides is likely to be 8 presented and so it becomes impossible to tell whether the signal bears any 9 relation to the actual state of the world. In the case of scientific evidence, there are 10 several distinct forms that this bias can take. First, there is straightforward delib- 11 erate selection bias: the parties only select experts who support their argument. 12 Secondly, there are more subtle forms of unconscious bias. These arise when, even 13 though people might think they are being honest in giving evidence, they are 14 subconsciously biased in favour of conclusions that (1) support their prior beliefs, 15 (2) support the side with whom they are affiliated, (3) support the side by whom 16 they are remunerated.11 17 18 Bias by itself is not necessarily a big problem. Indeed, it is in many ways a distin- 19 guishing mark of the adversarial system. Parties are deliberately encouraged to 20 present as strong as possible an argument for their side of the dispute. Once they 21 do so, courts will then be in a position to compare the strengths and weaknesses of 22 the competing evidence, weigh it appropriately and determine what they think 23 actually happened. This is thought to be an ingenious ‘engine for getting at the 24 truth’.12 But the problem of bias is compounded when the evidence in question is 25 something about which the court is deeply uncertain. This is nearly always the 26 situation with expert evidence, which is, after all, introduced to assist the court 27 when it strays beyond its areas of competence. This is what gives rise to the 28 problem of incomprehensibility. 29 30 13 A classic example of this arose in the case of R v Clark. In that case, Sally Clark, a 31 solicitor, was convicted for the murder of two of her infant children. The evidence 32 against her was relatively thin and the defence tried to rebut the accusation by 33 34 11 These are covered in Robertson, above n. 5 at 184–9; see also Consultation Paper on Expert Evidence, 35 above n. 1, above n. 1 at ch. 4, for an extended discussion of this issue. This report identifies several 36 other forms of bias and notes that courts are alive to the problem of bias. Readable introductions to 37 the psychological experiments and theories underlying the problem of unconscious bias are T. 38 Gilovich, How We Know What Isn’t So (The Free Press: New York, 1991) and S. Sutherland, Irrationality, 2nd edn (Pinter and Martin: London, 2007). 39 12 H. Wigmore, Evidence, 3rd edn (Little, Brown: Boston, 1940) 29. 40 13 R v Clark [2000] EWCA Crim 54, [2000] All ER (D) 1219. A later case overturned the verdict although 41 this was because new evidence came to light: R v Clark [2003] EWCA Crim 1020. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 211 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 arguing that the children were victims of Sudden Infant Death Syndrome (SIDS). 2 At trial, the jury was told that there was only 1 in 73 million chance of double-SIDS 3 deaths in the same family. The implication, of course, being that the defence’s 4 argument carried, for all practical purposes, no weight whatsoever. 5 6 This, it turns out, was wrong on at least two counts. First, in arriving at the 1 in 73 7 million figure, the expert witness for the prosecution—a famous paediatrician 8 named Roy Meadow—failed to consider the possibility that having one SIDS death 9 in a family raises the probability of having a second one. This might be due to 10 shared genetic or environmental causative factors. In other words, he assumed 11 that the events were statistically independent when they were not. Ignoring this 12 made a big difference to the resulting figure.14 13 14 Secondly, the 1 in 73 million figure was presented out of context. In a trial, the 15 probabilities attaching to different events are meaningless if presented in 16 isolation. After all, what is at issue is not the probability of one event in isolation 17 but the contrasting probabilities of two events. Generically, these are (1) the proba- 18 bility that person is innocent; and (2) the probability that the person is guilty. In 19 the Clark case, the need for context meant that the probability of a double SIDS 20 needed to be contrasted with the probability of a double infanticide. The latter, it 21 turns out, was less probable than the former.15 22 23 The case illustrates just how inappropriate the weighting that is attached to some 24 kinds of expert evidence can be due to the problem of incomprehensibility. The 25 court, and the lawyers for the respective sides, did not understand the nature of 26 the (fairly elementary)16 probability assessments being made, and were conse- 27 quently unable to question the methodology that Meadow had employed when 28 arriving at his figure. This meant they were also unable to understand its signifi- 29 cance. When such difficulties in comprehension are combined with the problem 30 of bias, as they are in the case of scientific evidence, we have a potential recipe for 31 disaster: erroneous fact-finding, miscarriages of justice, and the undermining of 32 the credibility of the legal system. We need to do something to resolve these 33 problems. 34 35 36 37 38 14 On these issues see R. Hill, ‘Multiple Sudden Infant Deaths—Coincidence or Beyond Coincidence?’ (2004) 18 Pediatric and Perinatal Epidemiology 320 at 321–2. 39 15 Ibid. at 322–3. 40 16 Hill’s article, above n. 14, goes through the relevant calculations and does not, by my lights, 41 require anything more than second-level mathematics. The difficulty, as I see it, is finding good 42 data to use in those calculations. 212 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE How do we measure the success of proposed reforms? 1 2 Diagnosing the problems associated with scientific evidence can help point us in 3 the direction of a solution to those problems. To resolve the problem of bias, we 4 need to do something to change the incentives associated with the acquisition and 5 presentation of scientific evidence in such a way that biased evidence becomes less 6 feasible. To resolve the problem of incomprehensibility, we need to arm legal 7 decision-makers with the tools needed for understanding the evidence with which 8 they are presented. 9 10 In measuring the success of any proposed reforms to this area, it is worth stepping 11 back and considering the purpose of evidence and evidence law in general. This 12 process of stepping back will allow us to develop some parameters for measuring 13 the success of proposed reforms. As our first step backwards, notice that the two 14 problems outlined above, incomprehensibility and bias, affect the capacity of the 15 court to work out the facts of the case. Working out the facts of the case is 16 important because the primary function of the court is to apply legal principles to 17 facts.17 18 19 Once this principles-to-facts model has been accepted, the role of evidence and 20 evidence law becomes apparent. We need evidence in order to establish the facts. 21 But not just any kind of evidence will do. The evidence must be of a certain quality. 22 This is where the rules of evidence law come into play. They place limits on the 23 kinds of evidence that can be presented to the court, and thereby maintain the 24 requisite quality control. Three things need to be kept in mind when considering 25 the rules of evidence law. 26 27 First, the rules should ensure that only relevant evidence is admissible.18 This 28 is evidence that actually raises or lowers the probability of the fact to be 29 30 17 This is, of course, somewhat controversial since principles and facts are often opaque and indeter- 31 minate. See J. Coleman and B. Leiter, ‘Determinacy, Objectivity and Authority’ (1993) 142 University of Pennsylvania Law Review 549 for a detailed exploration of these issues. 32 18 This can be interpreted in a strictly probabilistic manner where relevance is determined by the 33 ratio of likelihoods of competing theories. Let P be the fact to be proved and E be the evidence 34 introduced. For the evidence to relevant, then the probability of P given E, Pr(P|E), must be higher 35 or lower than the probability of P on its own, Pr(P). To know whether the evidence is relevant we need to consider the ratio of likelihoods, i.e. Pr(E|P): Pr(E|~P). If this ratio is equal to one, then the 36 evidence is not relevant—it does nothing to raise or lower the probability of P. Otherwise, the 37 evidence is relevant. I take this from R. M. Goldman and A. I. Goldman, ‘Review of Truth, Error and 38 Criminal Law: An Essay in Legal Epistemology, by Larry Laudan’ (2009) 15 Legal Theory 55 at 61. For more 39 on the probabilistic framework see T. Anderson, D. A. Schum and W. L. Twining, Analysis of Evidence, 2nd edn (Cambridge University Press: Cambridge, 2005) ch. 9. On whether probability theory is the 40 best way to frame evidence law, see H. L. Ho, A Philosophy of Evidence Law (Oxford University Press: 41 Oxford, 2008) ch. 3 which offers a slightly different theory of legal epistemology. Ho does not 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 213 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 proved.19 If possible, the court should also be able to assign some appropriate 2 weighting to the evidence. The weighting will help the court determine, either 3 with precision or through approximations, by how much the evidence in 4 question raises or lowers the fact to be proved. 5 6 Secondly, a standard of proof needs to be agreed upon.20 In criminal law, the 7 relevant standard is beyond a reasonable doubt; in civil law, it is the balance of 8 probabilities. The standard of proof reflects the complex, and often imprecise 9 nature of proving facts. It shows that the law need not demand that there be 10 absolute, certain proof of anything: it need only demand that something be 11 proved to an acceptable level of probability. As Laudan notes in his essay on 12 evidence in the criminal law, when crafting a standard of proof we should try to 13 obtain a reasonable distribution of false acquittals to false convictions.21 In other 14 words, we should accept that we are likely to get things wrong on occasion, and we 15 should seek to minimise the damage associated with getting things wrong. In 16 criminal law, the damage is, presumably, felt to be higher, hence the standard of 17 proof is higher. 18 19 Finally, we need to ensure that the rules of evidence are consistent with other 20 values and policy goals to which we hold.22 For example, rules on the acquisition of 21 evidence should be consistent with the right to privacy, autonomy or bodily 22 integrity. Likewise, making sure the procedure is not too costly and makes good 23 use of public and private resources is a legitimate policy concern. 24 25 26 27 dispute the important role that probability assessments may play in legal fact-finding, but he does dispute the overall conception of legal fact-finding in these terms. Finally, on evidence and 28 probability theory in general (though specifically applied to the philosophy of science), see 29 E. Sober, Evidence and Evolution (Cambridge University Press: Cambridge, 2008) ch. 1 of which 30 provides an excellent introduction to and overview of the debate in the philosophy of evidence. 31 19 I take it that this statement is true no matter what interpretation of probability and proof one accepts. For example, if one accepts a strictly subjectivist interpretation of probability, relevant 32 evidence can help to raise or lowers one’s subjective probability measure for the event or fact that 33 needs to proved. Likewise, if one accepts an objective interpretation of probability, one assumes 34 that there either is, or is not, some measure that can determine the probability that the fact occurred. If there are such measures, they will assist in the fact-finding mission of the court; if 35 there are no such measures, that also assists the fact-finding mission (by essentially highlighting 36 the opaque nature of the facts in the particular case). Similarly, all proofs proceed from premises to 37 conclusions. In only a handful of cases (pure mathematics, say) are the premises known with any 38 certainty. In all other cases, probability assessments of some kind must be used to determine the overall soundness of the proof. 39 20 See Ho, above n. 18 at ch. 4 for more. Also L. Laudan, Truth, Error and Criminal Law (Cambridge 40 University Press: Cambridge, 2006) ch. 1. 41 21 Laudan, above n. 20. Part I of the book is dedicated to this issue. 42 22 Laudan, above n. 20 at 2. 214 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE The preceding characterisation can assist in formulating a set of parameters or 1 criteria for measuring the success of any proposed changes to the law of evidence. 2 They are as follows: 3 4 (1) Truth-directedness: Does the proposed change actually help the court to 5 get at the truth? Does it enable the court to determine what actually 6 happened? 7 (2) Practicality: Is the proposed change practicable? What are its cost impli- 8 cations? How easy would it be to implement? 9 10 Some elaboration and qualification might be in order if these parameters are to 11 become fully defensible. 12 13 First, with regards to truth-directedness, this parameter captures one of the core 14 normative virtues of the legal system in a liberal democratic state. Indeed, 15 according to the theories of epistemic democrats,23 democracies in general derive 16 their legitimacy from their ability to facilitate better, more truth-oriented 17 decision-making. Thus all reforms within a democracy must be assessed in terms 18 of their compatibility with the pursuit of truth. 19 20 Less abstractly, and more directly relevant to the operation of the legal system, 21 having truth-directed processes should ensure that the law is properly applied; 22 that the right to a fair trial is protected; that a reasonable distribution of false 23 acquittals to false convictions is achieved; and that miscarriages of justice are 24 avoided. Thus, truth-directedness is perhaps the central virtue of any proposed 25 reform to the law of evidence, and will consequently be awarded pride of place in 26 the assessment of the blind expertise proposal given below. When engaging in this 27 assessment, it is important to bear in mind that it is unlikely that any reform will 28 actually get us to the truth. Indeed, whether such a thing is ever obtainable is a 29 philosophically contentious matter.24 Still, philosophical worries of this sort 30 should not distract us from the possibility of avoiding the obvious impediments to 31 truth, such as those created by the problem of bias. 32 33 Although truth-directedness is important, its attainment needs to measured in 34 terms of its practicality and cost-effectiveness. As regards to both of these things, it 35 should be noted that any change in the status quo is likely to face, if only in the 36 37 38 23 See C. List and R. Goodin, ‘Epistemic Democracy: Generalizing the Condorcet Jury Theorem’ (2001) 39 9 Journal of Political Philosophy 277. 24 I am, of course, working with the presumption that there is some objective fact of the matter even 40 if I doubt the existence of a practical epistemic route to that matter. If one rejects the existence of 41 objective facts of the matter, one is unlikely to accept anything that is said in this article. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 215 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 short term, practical difficulties. This is because change can place burdens on 2 people who have grown accustomed to certain practices. The existence of such 3 burdens should not, however, lead us to ignore the possible long-term benefits 4 that can be derived from these short-term costs. 5 6 What is the blind expertise solution? 7 8 The blind expertise solution focuses primarily on the problem of bias outlined 9 above. Indeed a ‘blind’, as the concept is understood in scientific practice, is 10 simply something that shields an experimenter or investigator from potentially 11 biasing factors. In some ways, the perceptual metaphor is unfortunate because, as 12 is the case with the biasing factors themselves, blinding need not involve any 13 direct changes to conscious perception or awareness, but the term is in common 14 usage and so it will be adopted here. Two examples of blind expertise solutions 15 will be described below. The first comes from Christopher Tarver Robertson and 16 applies specifically to the civil context.25 Robertson’s analysis and defence of the 17 concept is probably the most comprehensive analysis, but since it is limited to the 18 civil context, a second example, from the work of Roger Koppl and his colleagues, 19 is described. This example covers the use of forensics in criminal trials.26 The 20 proposed solutions are considered below; critical commentary is postponed until 21 later in the article. 22 23 (a) Robertson’s proposal 24 Robertson envisages the implementation of a process involving a number of 25 stages, the net result of which is to eliminate selection, affiliation and compen- 26 sation biases.27 The process begins when the parties to the litigation discover they 27 have some problem question, relevant to their litigation, on which they will 28 require expert scientific opinion. At this stage, they will be faced with a choice (it is 29 crucial to Robertson’s proposal that they have this choice): they can opt to select 30 their own ‘hired-gun’ expert, or they can opt for a ‘blind’ expert. 31 32 If they opt for the blind expert, they must submit their problem question to an 33 intermediary body. This would be some sort of professional scientific association 34 or organisation. This intermediary body is itself divided into two groups. The first 35 group will take the litigant’s question and try to remove any indication of the legal 36 purpose behind it. Suppose, for illustrative purposes, that the question is epidemi- 37 ological in nature: the claimants in some toxic tort action want to know whether 38 there is a causal link between, say, the electromagnetic radiation from overhead 39 40 25 See materials cited in n. 5. 41 26 In particular Koppl et al., ‘The Epistemics of Forensics’, above n. 5. 27 The following description is taken from Robertson, above n. 5 at 206–13. 42 216 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE power lines and their leukaemia. To remove bias from this inquiry, the first group 1 will need to eliminate any hint of which side might be asking for this information 2 and avoid tilting the inquiry in one direction or the other. So they would need to 3 ask a non-leading question like: ‘Does exposure to EM-radiation raise lifetime risk 4 of leukaemia? If so, by how much?’ This, after all, is a question to which both sides 5 will need to know the answer. It may also be helpful to break this question down 6 into a set of more specific questions, each one of which is careful to avoid 7 prejudging any particular conclusion. 8 9 Once the litigant’s question has been stripped of any potentially biasing features, 10 it will be sent to the second group within the professional association who will 11 pick the expert whose opinion will be sought. The selection of the expert can 12 either be governed by a randomisation process (provided there is some predefined 13 pool of experts) or can be done deliberately (provided there is no hint of bias in 14 the selection). One may wonder whether deliberate selection would simply 15 reintroduce possibilities for bias. In response, Robertson suggests that there may 16 be no other choice in some cases and that the effect should be minimised if the 17 person doing the selecting is unaware of the litigious purpose that lurks behind 18 the request, which they should be if the first group has done their job properly.28 19 20 After that, the litigant’s question is sent on to the expert who prepares a report 21 and receives a fee for his or her troubles.29 The report is sent back to the interme- 22 diary who in turn sends it back to the litigant. The report will either be favourable 23 to the litigant’s case or not. Robertson argues that the litigant should have the 24 option to ignore the report or try the process again. He does so for a somewhat 25 technical reason discussed below. For now, I simply note that should the litigant 26 try the process again, the number of times he or she does this will be discoverable 27 by the opposing side. This fact alone should provide some disincentive against 28 multiple uses. 29 30 Once the report has been received and selected for use by the litigant, the blind is 31 lifted and the expert can appear in court to answer questions on the evidence 32 33 34 35 28 It is worth comparing this with Koppl’s comments ((b) below) on forensic labs being biased even if they are not told the purposes or preferences of those making the request for information. This is 36 one of things that leads Koppl to propose his system of competitive redundancy. However, there is 37 a significant disanalogy here since forensics are almost always used by those prosecuting or 38 investigating crime and so it would be difficult not to guess the purpose or preferences of those 39 making the request. Scientific information in toxic tort cases is often required by both sides to the dispute. I am indebted to an anonymous reviewer for identifying the need for this clarification. 40 29 The fee is not tied to or conditional upon any particular conclusion so the problem of compen- 41 sation bias is avoided. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 217 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 discussed in his report. The report would be available to both sides and rigorous 2 cross-examination would be permitted. 3 4 That is, roughly, how blind expertise might play out in the civil context. 5 6 (b) Koppl’s proposal 7 Roger Koppl and his colleagues suggest that blinding could also play a role in how 8 forensics labs supply information for use in assisting criminal prosecutions. In 9 most cases, the job of a forensic lab will be to find out whether a sample (e.g. DNA 10 or fingerprint) taken from a suspect matches a sample taken from a crime scene. 11 The problem, as Koppl sees it, is that the typical management and operation of 12 forensic labs—usually, one lab under police organisation—leads to potentially poor 13 quality signals being sent from the lab back to the investigators, prosecutors and, 14 ultimately, the court. The reason for this is partly to do with the error rates 15 associated with forensic techniques30 and partly due to biasing. As Koppl et al. note: 16 17 Under current institutions in common law countries, crime labs are 18 typically organized under the police. This mode of organization tends 19 to create a pro-police bias [footnote omitted]. The bias need not be 20 conscious. The police generally ask for a test when they believe a 21 match will identify their suspect. Thus, some forensic workers tend to 22 have a bias in favor of finding matches.31 23 24 Thus, removing bias may be somewhat more difficult in the case of forensic labs 25 than it is in the context of civil litigation. The hypothetical epidemiological 26 question discussed above could easily be asked by either party to a litigation, but 27 the norm in forensics is for the requests to come from the prosecutorial side. 28 Consequently, something more creative than merely carefully wording the 29 request must be done in order to ‘blind’ the experts. 30 31 32 Koppl et al. suggest that the requisite blinding could be achieved through a system 33 of rivalrous redundancy (this system has additional benefits to be discussed below). 34 Initially, this would require the existence of multiple independent forensic labs 35 performing the same tests. By itself, this redundancy would not eliminate the 36 37 30 The discussion of jury theorems at pp. XXX below provides the reason for thinking that a single 38 monopolistic lab performing tests with error rates is a problem. 31 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 145. The references cited in support of the 39 contention that there might be a pro-police bias in forensics labs are: D. M. Risinger, M. J. Saks, W. C. 40 Thompson and R. Rosenthal, ‘The Daubert/Kumho Implications of Observer Effects in Forensic 41 Science: Hidden Problems of Expectation and Suggestion’ (2002) 90 California Law Review 1 and P. C. 42 Gianelli, ‘Confirmation Bias’ (2007) 22 Criminal Justice 60. 218 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE pro-police bias, something extra is needed for that. This ‘extra’ is supplied by 1 making their payoffs (remuneration) dependent on both the actions of the other 2 labs and those requesting the information. Specifically, if each lab is made to 3 think that its results need to conform with the results sent by the majority of labs 4 in order to receive payment, it will be incentivised to send the most reliable, 5 truth-directed signal. Why is this? The reason comes from game theoretic consid- 6 erations: the best way to ensure that one is in the majority, when collusion is 7 rendered impossible, is to aim at the truth. Thus, the rivalrous redundancy 8 switches the lab’s attention away from typical biasing factors and gets it to focus 9 on providing the most accurate results.32 In two experiments using simple 10 signalling games, such as those outlined above, Koppl et al. tested to see whether 11 introducing a system of rivalrous redundancy really would improve the reliability 12 of the signals that are transferred between senders and receivers. The results 13 suggested that it would.33 14 15 In addition to a system of rivalrous redundancy, Koppl34 recommends that more 16 straightforward blinding practices be introduced into both the handling and 17 preparation of the samples being used in forensic science, as well as the requests 18 for information from forensic science labs. For example, Koppl suggests that, in 19 the case of fingerprint matching, one person should prepare the sample and 20 another should perform the analysis.35 Similarly, Risinger et al. argue that case 21 information not relevant to the particular forensic technique or test be masked or 22 hidden from forensic examiners.36 23 24 Would a system of blind expertise be any good? 25 26 Now that we have outlined two potential blind expertise-style solutions to the 27 problem of scientific evidence, we can proceed to assess them in light of the two 28 parameters set down earlier. 29 30 (a) The truth-directedness of blind expertise 31 The first question we need to ask is whether the introduction of a system of blind 32 expertise will actually help the court to receive less biased, and more reliable, 33 signals from expert witnesses. There are two reasons for thinking that it would. 34 The first stems from the fact that blinding practices are already widely accepted as 35 36 32 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 146–7. 37 33 Experiments are described in Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 148–54. 38 34 See Koppl, ‘Romancing Forensics: Legal Failure in Forensic Science Administration’, above n. 5 for a 39 discussion of these proposals both from himself and from others such as Risinger et al. and Gianelli, above n. 31. 40 35 Koppl, ‘Romancing Forensics: Legal Failure in Forensic Science Administration’, above n. 5. 41 36 Risinger et al., above n. 31. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 219 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 having that power in scientific practice; the second stems from the fact that the 2 proposals from Robertson and Koppl have the ability to exploit so-called jury 3 theorems. Both reasons will be discussed in some detail. It should be borne in 4 mind that, for the purposes of this analysis, although improving the reliability 5 and removing bias is assumed to get us closer to the truth, it is not assumed to get 6 us actually to the truth. As was mentioned earlier, the truth may often be 7 complex, and disagreement may well remain after the most obvious forms of bias 8 have been removed. Hence the success of blind expertise solutions must be 9 measured in terms of their improvement of the existing situation. 10 11 The first reason37 for thinking that ‘blinding’ can get us closer to the truth is that it 12 is already recognised as being an essential component in the scientific testing of 13 hypotheses, particularly in evidence-based medicine.38 For example, whenever a 14 new drug is being tested for its efficacy, a randomised double-blind controlled test 15 is performed. Very roughly, this means that a group of patients who are thought to 16 be fit subjects for taking the drug are recruited for the study; they are randomly 17 divided into two different groups; one group receives the drug and the other 18 receives a placebo (e.g. sugar pill); neither the patients nor the doctors who are 19 supplying them with their medication know whether they are receiving the 20 placebo or the drug being tested; and the improvements or disimprovements, as 21 the case may be, in the patient’s condition are recorded. 22 23 The double-blind (and randomised) method of testing allows the experimenter to 24 tell whether the active ingredient in the drug is having any effect on the patient’s 25 condition over and above the placebo effect and/or any other environmental or 26 genetic factors. If the protocols are broken and the blinding is lifted, then it 27 28 29 30 37 There is an additional reason for accepting blinding that seems, to me, to be of lesser significance, 31 namely that it is often thought to be an important part of the administration of justice (procedural fairness). Robertson, above n. 5 at 202–6 gives the example of John Rawls’s theory of justice in 32 which the members of a society are forced to negotiate a social contract from behind a ‘veil of 33 ignorance’, i.e. without knowing what position they will occupy in the society. Rawls felt that this 34 was the only way to satisfy the Kantian principles of impartiality and equal respect. Impartiality is 35 also a prominent feature of the principles of natural justice in administrative law (J. Rawls, A Theory of Justice (Belknap Press of Harvard University Press: Cambridge, MA, 1971). Robertson thinks that 36 these normative uses of blinding add to its acceptability. 37 38 Robertson, above n. 5 at 202–6. On evidence-based medicine and double-blind tests, see the 38 discussion T. Greenhalgh, How to Read a Paper: The Basics of Evidence-Based Medicine, 4th edn (Wiley-Blackwell: Chichester, 2010) ch. 4 which goes through the critical questions that should be 39 asked of the methodology underlying any medical study. A readable introduction to the basic 40 components of evidence-based medicine, including the randomised double-blind test, with a 41 critical application to alternative medicine is available in R. Barker Bausell, Snake Oil Science (Oxford 42 University Press: Oxford, 2007). 220 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE becomes impossible to tell whether the ‘signals’ (i.e. the experimental results) that 1 are being observed are due to the drug or these other confounding variables. 2 3 To the extent that both Robertson’s and Koppl’s proposals take advantage of 4 similar blinding and randomisation protocols, they should improve the quality of 5 the signals being received from expert witnesses. After all, as is the case in medical 6 trials, once obvious sources of bias have been removed, it becomes more likely that 7 the differences, if any remain, between the signals being received is due to 8 truth-relevant factors. 9 10 Despite these advantages, there is, perhaps, reason to think that these proposals 11 cannot remove one obvious form of bias, namely the self-selection bias that arises 12 when experts pick their own areas of expertise. To go back to the epidemiological 13 example given earlier, it might be argued that those epidemiologists with the 14 requisite knowledge to answer the litigant’s question have a vested interest in 15 finding a link. This would be because they would not bother specialising in that 16 area unless they thought there was the possibility of finding some sort of link. So if 17 the blind experts chosen through, say, Robertson’s selection process come from a 18 limited self-selecting pool, the net result may simply be to replicate the bias of the 19 given field of expertise. 20 21 This concern may be more apparent than real, there is some evidence, for example 22 in the history of research into links between EM-radiation from overhead power 23 lines and leukaemia, that suggests that while initial research may be driven by a 24 biased belief in the existence of a link, researchers with no biases in favour of 25 finding a link can get involved in research so that, over time, this initial effect is 26 removed.39 Furthermore, in an area like epidemiology, it is usually the method- 27 ological design of the studies that is crucial, not so much underlying specialist 28 knowledge about EM-radiation or the like.40 These methodological issues should 29 be familiar to all epidemiological researchers and so there should be no need to 30 limit the pool to just those specialising in research on the particular proposed 31 link. Indeed, methodological issues are probably the key to evaluating most forms 32 of scientific evidence, so the problem of self-selection should be limited. 33 34 This does, however, bring up another concern: is blind expertise capable of dealing 35 with evidence from emerging areas of scientific knowledge? A failure to do so 36 could be especially worrisome given that these areas are likely to be most prone to 37 38 39 39 G. Kabat, Hyping Health Risks (Columbia University Press: 2008) ch. 4. 40 40 Although note that Kabat, above n. 39, suggests that ignorance of the underlying physics of 41 EM-radiation was one reason why epidemiological research in this area was so misguided. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 221 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 self-selection bias by researchers. Given the need for a sufficiently diverse pool of 2 experts, there is indeed reason to think that the blind expertise solution is poorly 3 equipped to deal with such areas. This may tip the balance back in favour of a 4 reliability test solution, at least when it comes to the frontiers of scientific 5 knowledge. Indeed, one of the reasons often offered for adopting a Daubert-style 6 reliability test instead of a Frye-test (which focuses on general acceptance in the 7 relevant field) is that the former is more open to emerging areas of knowledge 8 That may indeed be a mark in favour of the formal reliability test, but it should be 9 noted that when assessing the reliability of the novel evidence, judges would still 10 need to be alert to the possibility of self-selecting bias and lack of blinding. In other 11 words, failure to blind would remain an indicator of unreliability.41 12 13 The second reason for thinking that a system of blind expertise might help courts 14 to get at the truth comes from its ability to exploit the power of jury theorems. 15 These are theorems in probability theory showing that having several people 16 ‘voting’ on a question or decision, and going along with the majority answer, can 17 make it more likely that we get the correct answer. The classic jury theorem is that 18 of the Marquis de Condorcet.42 Put most simply, this theorem states that if you 19 have a question for which there are two possible answers, only one of those is the 20 right answer, and the probability of any one person getting the right answer is 21 greater than 0.5, then you are more likely to get the right answer by getting two or 22 more people to ‘vote’ than you would be if you accepted the answer of just one 23 person.43 Indeed, the probability of getting the right answer converges on 1 as the 24 size of the voting pool tends to infinity. The reason for this is that the effects of 25 individual error rates can be minimised by pooling the votes of multiple 26 individuals. 27 28 This basic theorem has been expanded in a number of ways since, so that the 29 limitations imposed by Condorcet (i.e. greater than 0.5 probability of getting the 30 answer right, no more than two possible answers, etc.) are no longer needed. Of 31 32 33 41 Indeed a failure to appeal to a broader segment of the research community will continue to a 34 mark against any proffered evidence. One recent example of this, which happens to be of interest to this author, is the non-acceptance of fMRI-based lie detection evidence in US courts. See, e.g., 35 the case of United States v Semrau where, in a Daubert-hearing, the judge rejected the evidence from 36 fMRI lie detection at least partly on the basis of its non-acceptance by the relevant research 37 community. Full reference: United States v Lorne Allan Semrau, 31 May 2010, US District Court for 38 the Western District of Tennessee (Eastern Division) Case No. 07-10074 M1/P, available at <http://lawneuro.typepad.com/files/semrau.pdf>, accessed 3 May 2011. 39 42 Marquis de Condorcet, Essai sur l’application de l’analyse a` la probabilite´ des de´cisions rendues a` la 40 pluralite´ des voix (Paris, 1785). 41 43 When the number of jurors is even, some tie-breaking rule needs to be introduced to cover the 42 possibility of deadlock. 222 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE particular significance for our purposes is the fact that it still holds true even 1 when the probability of each individual getting the right answer dips below 0.5, 2 provided the mean probability of choosing the correct answer is greater than 0.5 3 across the jury as a whole.44 4 5 Both Robertson and Koppl make use of this basic jury theorem result in their 6 proposals. Robertson, for example, argues (with the requisite mathematical 7 demonstration) that getting both sides of a dispute to avail of his blind expertise 8 protocol will produce a more reliable signal than would be the case if only one side 9 availed of it.45 The reason being, as stated above, that the effect of error rates is 10 minimised. Likewise, Koppl’s call for having multiple labs perform the same test is 11 premised partly on the fact that this takes advantage of jury theorem effects. The 12 success of his proposal in this regard is demonstrated by the experiments he has 13 performed. In these experiments, it is possible to construct the ‘right’ answer and 14 see how well the proposed system of rivalrous redundancy does in reaching it.46 15 Obviously, outside of the lab it is not possible to know in advance exactly which 16 answer is the correct one, but the argument is that the knowledge gained from the 17 experimental setting (and from the probability theorems) should guide policy in 18 the real world. 19 20 There are two major concerns one can have about the application of jury theorems 21 to the proposed system of blind expertise. The first stems from the fact that most 22 extensions and applications of Condorcet’s result rely on the fact that there is a 23 dichotomous choice to be made by the voters (in our case, the experts). The 24 question is whether the types of answers that expert witnesses are asked to give 25 will fit with this two-option ideal. At first glance, it might seem like the answer 26 would be ‘no’. After all, the subject-matter addressed by expert evidence is often 27 complex and may not allow for simple yes/no, either/or responses. However, it may 28 be possible, through careful phrasing and restrictions of the questions asked, to 29 restrict the answers to the dichotomous form. This would certainly seem to be the 30 case in Koppl’s example of the forensic lab being asked whether the fingerprints 31 are a match or not. The answers to such a request, even though probabilistic in 32 form, would be binary in nature. 33 34 35 36 37 44 See B. Grofman, G. Owen and S. L. Feld, ‘Thirteen Theorems in Search of the Truth’ (1983) 15 Theory 38 and Decision 261 and P. J. Borland, ‘Majority Systems and the Condorcet Jury Theorem’ (1989) 38 Statistician 181. 39 45 Robertson, above n. 5 at 214–19. 40 46 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 144–7 and the experiments described at 41 148–54. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 223 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 In any event, jury theorems can be generalised to cover situations involving more 2 than two options. Christian List, for one, has shown that even when there are more 3 than two possible answers, as long as the probability of getting the right answer is 4 greater than the probability of getting the wrong answer, the majority vote is still 5 more likely to be correct.47 So, for example, suppose there are five possible answers 6 (P1 ... P5), and it is known that P1 is correct. Then, provided that the probability of 7 the voters picking P1 is greater than the probability of them picking P2, P3, P4, or 8 P5, the majority vote is still most likely to be P1. This is true even if the probability 9 of picking P1 is less than 0.5 (e.g. the probability of picking P1 could be 0.3, 10 compared with 0.2 for P2–P4 and 0.1 for P5). 11 12 Although this is certainly a useful result, there remain a couple of difficulties. 13 First, to get the truth-directed benefits of the theorem in such cases may require 14 large pools of voters (how large depends on the probability of getting the right 15 answer) not just the two or three envisaged by Robertson and Koppl. Furthermore, 16 in the multiple-options case, the straightforward majority vote system is not the 17 only procedure that is statistically more likely to arrive at the correct answer.48 18 Thus, some further refinements or even radical expansions of Robertson’s and 19 Koppl’s proposals may be in order. 20 21 The second major objection to the application of jury theorems to blind expertise 22 is that no matter what their scope (two-option or multiple-option), they rely on the 23 assumption that the voters (either individually or as a group) have a greater proba- 24 bility of picking the right answer than they would of picking the wrong answer(s). 25 Some may question whether that assumption holds up in the case of scientific 26 evidence. On the whole, I think the assumption is reasonable. One is entitled to 27 think that experts are more likely to get the right answer than the wrong one, but 28 that depends upon proper blinding protocols being in place. As Koppl notes, 29 having multiple labs performing the same tests will be of little utility if the 30 sources of bias are not removed.49 Thus, the benefits of the jury theorems should 31 be seen to arise only after blinding has been achieved. 32 33 Overall, then, in spite of the various concerns noted above, the blind expertise 34 solution, by utilising accepted protocols from scientific practice and imple- 35 menting the beneficial effects of jury theorems, scores well in terms of 36 truth-directedness. This implies that its introduction should help to protect the 37 38 47 See C. List and R. Goodin, ‘Epistemic Democracy: Generalizing the Condorcet Jury Theorem’ (2001) 39 9 Journal of Political Philosophy 277 (Appendix 1 for the formal proof). 40 48 Different possibilities such as Condorcet’s own multiple pairwise voting system (i.e. voting 41 repeatedly on two pairs of possibilities) are acknowledged in List and Goodin’s article, above n. 47. 42 49 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 146. 224 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE right to a fair trial, ensure that there is a reasonable distribution of false acquittals 1 to false convictions, and prevent potential miscarriages of justice. These are not 2 inconsiderable benefits. 3 4 (b) Practicality 5 The next question we need to ask is whether the blind expertise solution is 6 practical, whether it is something that could be readily implemented. As noted in 7 the second part of this article, we should be on our guard here: changes to the 8 status quo are often difficult to accept, and so there is a tendency, when consid- 9 ering the practicalities of those changes, to be quite negative. However, practical 10 hurdles also face other proposed changes to this area of law, such as the 11 programme of judicial education required for the reliability test solution. It is the 12 relative practicality of these proposals, measured over the long term, that counts. 13 14 The discussion here will focus initially on two major dimensions to the practi- 15 cality of the proposals under consideration. The first of these dimensions concerns 16 the infrastructural changes that would be required should these proposals go 17 through; the second concerns the financial impact of the proposals. Although these 18 two dimensions may overlap (and so are not perfectly orthogonal to each other, as 19 would be required if they were to fit the mathematical definition of a dimension), 20 they are sufficiently distinct to merit independent consideration. In addition to 21 these two major dimensions of practicality, a selection of further practical issues 22 are addressed below. 23 24 Let us look first to the infrastructural problems associated with Robertson’s 25 proposal. Crucial to that proposal is the existence of an administrative barrier 26 between the litigant and the expert. This would require the setting-up of an 27 agency that would strip the litigant’s questions of potentially biasing factors, and 28 select appropriate experts (possibly through a randomisation process) to answer 29 those questions. Robertson argues that it could be quite easy to create such an 30 administrative infrastructure if we take advantage of extant not-for-profit profes- 31 sional scientific organisations. The members of such organisations would benefit 32 from having such a system in place because they could still earn personal fees for 33 their services. Robertson argues that in the United States the American Associ- 34 ation for the Advancement of Science (AAAS) could act as the relevant 35 administrative agency.50 One could imagine something similar being done on, 36 perhaps, a pan-European basis through the European Academy of Sciences.51 If 37 38 50 Robertson, above n. 5 at 206–7. Partly this is because the AAAS already has an infrastructure that 39 judges can use for selecting court-appointed experts. Robertson notes that this infrastructure is 40 largely unused and could be retooled for his system. 41 51 See <http://www.eurasc.org>, accessed 3 May 2011. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 225 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 that proves to be infeasible (due perhaps to language barriers and jurisdictional 2 conflicts) a national organisation might be more appropriate. Failing that, one 3 could imagine the associations covering the major professions, such as medi- 4 cine and engineering,52 performing the relevant administrative function. More 5 narrowly focused groups of scientists would be difficult for lawyers to identify and 6 locate (although perhaps no more difficult than finding a ‘hired gun’ expert). Also, 7 narrower focus would make the self-selecting bias more of an issue. For instance, 8 seeking a blind expert from the (strictly hypothetical) Union of Epidemiologists 9 Concerned about the Effects of EM-Radiation from Overhead Power Lines is 10 unlikely to solve the problem of bias that the system is designed to solve. 11 12 If there is no readily available extant scientific organisation that could perform 13 the task, it might be necessary for a new, independent authority to be set up. This 14 would have to be organised on a not-for-profit basis so as to avoid any remuner- 15 ative biasing, and could carry with it some significant cost implications. However, 16 the impact might be lessened if the system could be self-financing. Robertson 17 suggests that it could be, and we will consider this possibility in a moment.53 18 19 Turning to Koppl’s proposal, the infrastructural problems there seem less 20 daunting. True, separate labs would need to be put in place, and it would be 21 preferable if they were not dependent on the police force or prosecutorial services 22 for direct funding. Nonetheless, there do not appear to be any serious impedi- 23 ments to their establishment. The major problem would be maintaining the 24 independence of the labs. This would be essential in order to preserve the 25 anti-biasing rivalry that needs to exist between the labs. To resolve this problem, 26 one could perhaps rely on the good faith of the scientists in the respective labs and 27 hope that they would not engage in any direct collusion. But this might be naïve 28 and some system of regulation and investigation could be both necessary and 29 wise. 30 31 32 33 34 52 Which are usually the most frequently consulted professions in legal cases. 53 An anonymous reviewer of an earlier draft of this article suggested an additional infrastructural 35 problem that might arise from Robertson’s proposal. This had to do with the effort to strip the 36 litigant’s question of all biasing factors. The reviewer wondered whether, once the expert’s answer 37 to this question had been received, a third-party panel would be required to reapply the answer to 38 the factual matrix of the case. I submit that this would be a bad idea because it would create an additional opportunity for biasing of the evidence. It would be much better simply to have the 39 expert’s report submitted as evidence. If the expert is questioned at trial, the questioning would be 40 based on the report, which could be scrutinised by all parties. The actual application of the report 41 to the facts of the case would be ultimately left to the court, although the parties could, obviously, 42 offer their interpretation. 226 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE Turning to the financial dimension to practicality, Robertson argues that in the 1 civil context the system can be self-financing and self-regulating.54 This is 2 because parties will be incentivised to make full use of it, and will be required to 3 pay the experts directly (not the intermediary organisation) for the privilege. 4 One might wonder why the parties would have an incentive to use a system in 5 which they have no control over whether the expert supports their interpre- 6 tation of the case. The reason, according to Robertson, is that they will be 7 trapped in a Prisoners’ Dilemma: it would probably be best for both parties to 8 agree that neither of them will use the system and to stick with hired-gun 9 experts, but if they did that one party could gain a strategic advantage by unilat- 10 erally breaking their promise and using the system to get an answer in their 11 favour. So, in the end, both parties end up using the system to their mutual disad- 12 vantage, but to the court’s advantage. 13 14 The success of Robertson’s argument depends on a couple of conditions. First, 15 judges and juries will need to be made aware of the epistemic advantages 16 bestowed by the blind protocol. This would require some sort of educational 17 programme, even if it does not rise to the level envisaged by the introduction of a 18 formal reliability test. Secondly, the parties will need to be given the option of 19 using the system multiple times, with the caveat that the number of times they 20 use the system will be discoverable by the opposing side. These conditions seem 21 reasonable and attainable, and if the natural antagonism between the two sides to 22 the litigation can be coerced into the strategic dynamics of a Prisoners’ Dilemma, 23 this would create a significant incentive to use the system and should, by proxy, 24 improve the reliability of the evidence being presented to the court. 25 26 Moving on to Koppl’s proposal, one might suppose that the creation of multiple 27 forensic labs or the performance of multiple tests within the one lab, along with 28 some system of regulation, would give rise to serious costs that would have to be 29 discharged from the public purse. However, Koppl offers some reasons for 30 thinking that the cost implications associated with his proposal are not as severe 31 as might first be thought. Indeed, he suggests that—at least in the case of finger- 32 print matching—the triplicate performance of the forensic analysis may lead to a 33 net reduction in costs. The precise details of his costing are somewhat intricate,55 34 but the basic logic is easily expressed. First, he notes that all forensic techniques 35 have certain rates of false positive and false negative errors associated with 36 them. False positive errors, i.e. errors that falsely link a suspect to a crime scene, 37 38 39 54 Robertson, above n. 5 at 230–41 sets down these arguments. 40 55 They can be found in Koppl, ‘Romancing Forensics: Legal Failure in Forensic Science Adminis- 41 tration’, above n. 5. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 227 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 will give rise to false convictions and these false convictions will give rise to costs 2 (associated with imprisonment and prosecution). Performing multiple forensic 3 tests also gives rise to costs, but because the performance of multiple tests can 4 reduce the number of false positives, it can also reduce the number of false 5 convictions. This could, in turn, lead to a net reduction in costs. Using US-based 6 statistics on the numbers and costs of convictions arising from the use of finger- 7 print evidence, and estimating a false positive rate of 0.8 per cent, Koppl shows 8 how triplicate fingerprint matching tests should result in significant annual 9 savings (upwards of $60 million per year).56 One can always quibble with the 10 assumptions used in costings of this sort, but with (for example) the annual 11 per-prisoner costs of incarceration standing at 77,222 in Ireland in 2009,57 it 12 would surely be prudent and possibly illuminating to perform detailed costings 13 to see whether or not significant savings could be made from implementing 14 Koppl’s proposal. 15 16 Moving now to other practical issues, one concern is that some varieties of scien- 17 tific evidence will be ill-suited to the blinding procedure, at least in the strict 18 form envisaged by Robertson. For example, fMRI-based lie detection tests have 19 recently been offered as evidence in criminal trials in the United States.58 The 20 questions that make up these tests need to be formulated through close consul- 21 tation with the person being tested59 and because of the interaction between the 22 tester and the subject of the test there is a real potential for biasing.60 Given the 23 personal nature of the test, it is difficult to see how this potential could be 24 prevented from being realised. Neverthelss, some blinding is usually possible. 25 For example, in the case of a lie detector test different people can administer and 26 interpret the results of the test (this can be aided by computer programs that 27 decode the relevant data and which are free of bias) thereby ensuring some level 28 29 30 31 56 Ibid. Koppl includes the possibility of false exonerations in his analysis, but the number is small. 57 Irish Prison Service, Annual Report 2009, 4, available at <http://www.irishprisons.ie/documents/ 32 AnnualReport2009PDF.pdf>, accessed 3 May 2011. 33 58 See United States v Lorne Allan Semrau, 31 May 2010, US District Court for the Western District of 34 Tennessee (Eastern Division) Case No. 07-10074 M1/P. 35 59 Nothing I say here should be taken to indicate a belief in the actual reliability or validity of lie detection tests. I am actually deeply sceptical about it. Nonetheless, the example is one with which 36 I am familiar and it seems appropriate to discuss it here. I should note that fMRI lie detection 37 evidence seems to be the perfect example of a fringe area of science with self-selecting experts who 38 have a vested interest in its success. 60 Indeed, this seems to have been an issue in the Semrau case. The person performing the test was 39 willing to discount one set of results that did not happen to be favourable to his client’s interests 40 on the (controversial) grounds that the client was too tired for the results to be accurate. There are 41 further problems with lie detection tests due to the way in which the questions that are asked 42 during the test are formulated after a consultation between the tester and the testee. 228 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE of impartiality.61 At a minimum, the possibilities for blinding should be explored 1 in these cases. 2 3 There are some further miscellaneous practical advantages to the blind expertise 4 solution. First, as Robertson argues, the system might be preferable to the intro- 5 duction of a system of court-appointed experts.62 His argument is premised on the 6 fact that a court-appointed system has had a negligible effect in the American 7 states that have introduced it. It seems that, in practice, few judges are willing to 8 employ their powers to appoint experts.63 This may be due to deeply engrained 9 norms of legal practice which favour the adversarial process. Robertson argues 10 that his proposal, insofar as it takes advantage of the competitive, adversarial 11 nature of the trial process, is more compatible with legal tradition. We can add to 12 this the fact that a single, monopolistic court-appointed expert would not realise 13 the potential of the jury theorems discussed earlier. One could, of course, imagine 14 multiple court-appointed witnesses being selected via some blinding procedure, 15 but this would seem to be redundant if the same effect can be achieved through 16 the adversarial process. It would also seem to rule out the self-financing advan- 17 tages of Robertson’s proposal. 18 19 In conclusion then, although there are practical hurdles facing the introduction 20 of a system of blind expertise, they are not insurmountable. Indeed, such a system 21 may be less difficult and less costly to implement than the reliability test solution 22 currently favoured by law reform agencies in Ireland and England since such 23 proposals would require the introduction of substantial judicial education 24 programmes. Furthermore, the solution may have certain financial advantages 25 associated with it, both in terms of its ability to reduce the costs associated with 26 erroneous incarcerations (in the criminal context) and its capacity to be 27 self-financing (in the civil context). 28 29 Should a system of blind expertise be introduced? 30 31 I think the answer to this question is a firm ‘yes’ with a ‘but’. The qualification is a 32 sop to the fact that there are really three questions to be considered here: (1) 33 Should a system of blind expertise be introduced on its own? (2) Should it be intro- 34 35 61 Or consider the proposal from Michael Bishop and J. D. Trout suggesting that, wherever possible, 36 statistical prediction rules should be used instead of human judgment. See M. Bishop and J. D. 37 Trout, ‘50 Years of Successful Predictive Modeling Should Be Enough: Lessons for Philosophy of 38 Science’ (2002) 69 Philosophy of Science: PSA 2000 Symposium Papers (supplement) S197–S208. And see also M. Bishop and J. D. Trout, Epistemology and the Psychology of Human Judgment (Oxford University 39 Press: Oxford, 2005). 40 62 Robertson, above n. 5 at 198–201 and Section V. 41 63 Ibid. at 198–201. 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 229 BLIND EXPERTISE AND SCIENTIFIC EVIDENCE 1 duced in addition to a reliability test? (3) Should it be introduced in preference to a 2 reliability test? 3 4 The first question is really asking: assuming no other changes to the existing legal 5 position with respect to scientific evidence are possible, should a system of blind 6 expertise be introduced? The answer to this is straightforwardly ‘yes’. The existing 7 legal position, in Ireland at any rate, is that the courts simply check the qualifica- 8 tions of the expert and the relevance of his or her evidence. This system does little 9 to resolve the problems of bias and incomprehensibility outlined in the second 10 part of this article. Introducing a system that does something to resolve at least 11 one of those problems is clearly going to be better than doing nothing. The 12 position in England is little better with judges asked to determine whether there is 13 a body of knowledge that is sufficiently well organised to be considered reliable. 14 15 The second question is really asking: given that the plan is to introduce a 16 reliability test, should a system of blind expertise also be introduced? I think the 17 answer to this is a straightforward ‘yes’. The reliability test proposal that was 18 outlined at the beginning of this article is clearly intended to resolve the problem 19 of incomprehensibility: it provides the court with a set of criteria that should 20 allow it to ascertain how reliable the evidence being presented really is. As we saw, 21 the problem of incomprehensibility is only one half of the problem of scientific 22 evidence. A system of blind expertise is focused on resolving the other half of the 23 problem. Thus, the reliability test and the system of blind expertise could be 24 almost perfect complements to one another and their joint introduction might 25 constitute an almost ‘ideal’ solution to our problems. That said, it is important to 26 emphasis that it would not actually constitute an ideal. Residual problems would 27 certainly remain. For example, a reliability test and an associated programme of 28 judicial education might help judges comprehend scientific evidence, but it does 29 nothing, by itself, to help the juries who will also have to understand the nature of 30 such evidence. 31 32 33 Finally, the third question is asking: if we had to choose between a system of blind 34 expertise and the introduction of a reliability test, which should we opt for? This 35 is, of course, the most difficult question to answer. It asks us to decide between 36 two options that are, as just pointed out, not direct alternatives but, rather, 37 complements. That said, accepting that having both might be preferable, the 38 correct answer is still the system of blind expertise. Why? Primarily, because the 39 truth-directedness and practicality of blind expertise seem to me to be superior to 40 those of the reliability test. Both of these have been discussed already. To reiterate, 41 a system of blind expertise makes use of protocols that are already widely accepted 42 in scientific practice to ensure that more reliable information is presented to the 230 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF BLIND EXPERTISE AND SCIENTIFIC EVIDENCE court. A reliability test merely gets judges to perform a similar check. 1 Furthermore, the proposals discussed above seem to have reasonable cost-savings 2 associated with them when compared to the introduction of a reliability test. The 3 only serious question mark is whether the necessary infrastructures could be 4 readily put in place. I have given some reasons for thinking that the obstacles 5 towards doing so are not great, but the devil, as always, will be in the details. Thus, 6 although it would not be ideal, I submit that the introduction of a system blind 7 expertise instead of a reliability test should be given serious consideration. 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 231