BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1
Blind expertise and the 2
3
problem of scientific 4
5
6
evidence 7
8
9
10
By John Danaher* 11
PhD Candidate, University College Cork 12
13
Abstract Scientific evidence presents a problem for the courts: the 14
subject-matter is often complex, the experts who present the evidence can be 15
cherry picked and biased; and judges and juries are frequently unsure about 16
how to weight the evidence once it has been presented. This article diagnoses
17
the problems associated with scientific evidence and then proceeds to consider
two possible solutions to those problems: (1) the reliability test solution; and (2) 18
the blind expertise solution. The former is currently favoured by law reform 19
agencies in Ireland and England, but the primary focus of this article is on the 20
latter. It is concluded that the blind expertise solution has considerable 21
attractions and should be seriously considered as a reform option. 22
Keywords Scientific evidence; Expert witnesses; Reliability; Blind expertise 23
24
aw reform agencies in both Ireland and England have recently
L
25
proposed reforms to the rules on the admissibility of scientific 26
evidence.1 They have done so partly because of confusions in the 27
28
29
*BCL (UCC), LLM (Dub). Email: johndanaher1984@gmail.com. The author would like to thank the
Irish Research Council for the Humanities and Social Sciences for their funding of his research
30
from 2007–10. He would also like to thank Dr Mary Donnelly and Dr Shane Kilcommins, as well as 31
two anonymous referees for comments on earlier drafts. 32
1 For England, see Law Commission, The Admissibility of Expert Evidence in Criminal Proceedings in 33
England and Wales, Law Commission Consultation Paper No. 190 (April 2009) (hereinafter ‘Admissi-
bility of Expert Evidence’), available at <http://www.justice.gov.uk/lawcommission/docs/cp190_
34
Expert_Evidence_Consultation.pdf>, accessed 3 May 2011; and Expert Evidence in Criminal Proceedings 35
in England and Wales, Law Com. Report No. 325 (March 2011) (hereinafter ‘Expert Evidence in Criminal 36
Proceedings’), available at <http://www.justice.gov.uk/lawcommission/docs/lc325_Expert_Evidence_
37
Report.pdf>, accessed 4 May 2011. For Ireland, see Law Reform Commission of Ireland, Consultation
Paper on Expert Evidence (LRC CP 52–2008) (hereinafter ‘Consultation Paper on Expert Evidence’), in 38
particular ch. 2 proposing the introduction of a reliability test. The Law Reform Commission of 39
Ireland Consultation Paper is available at <http://www.lawreform.ie/_fileupload/consultation% 40
20papers/cpExpertEvidence.pdf>, accessed 3 May 2011.
41
doi:10.1350/ijep.2011.15.3.378
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF (2011) 15 E&P 207–231 207
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 existing legal positions in both jurisdictions,2 and partly because of the obvious
2 difficulties courts face when confronted with such evidence: the
3 subject-matter is often complex and esoteric; the experts can be cherry-picked
4 and biased; and courts and juries are consequently unsure of how to weigh the
5 evidence properly in their deliberations.
6
7 The suggestion in both jurisdictions is that a formal reliability test needs to be
8 introduced. This test would provide a judge with a checklist of (seven or eight)3
9 criteria that are thought to be indicative of reliable scientific evidence. If the
10 evidence presented to the court satisfies a sufficient number of these criteria,4
11 then the evidence is admissible and can be considered by triers of fact when they
12 make their decision. If the evidence does not satisfy a sufficient number of the
13 criteria, then it is inadmissible.
14
15
While such a proposal has considerable merits—some of which are addressed
16
below—it is not the only way to deal with the problem of scientific evidence. This
17
article considers an alternative proposal: the introduction of a system of blind
18
expertise.5 This is a proposal that tries to bring the blinding protocols that are
19
already used in scientific inquiry to bear on the acquisition and presentation of
20
scientific evidence to a court of law.
21
22
In order to address the strengths and weaknesses of this proposal, the article
23
proceeds in the following manner. The first part attempts to offer a reasonably
24
precise diagnosis of the problems arising from the presentation of scientific
25
26
27 2 This appears to have been a major motivation for the English proposal. The feeling seems to be that
the current position, set down by King J in R v Boynthon (1984) 38 SASR 45 at 46–7, is unclear. The test
28
and its application in English law is analysed in Admissibility of Expert Evidence, above n. 1, Part 3. See
29 also A. Roberts, ‘Rejecting General Acceptance, Confounding the Gate-keeper: The Law
30 Commission and Expert Evidence’ [2009] Crim LR 551, who comments on the problems with the
31 Boynthon test.
3 See Consultation Paper on Expert Evidence, above n. 1 at 129 and Expert Evidence in Criminal Proceedings,
32 above n. 1 at para. 5.35. The Irish proposal has seven elements; the English has eight elements
33 (although there could be some dispute about how these are counted).
34 4 What counts as a ‘sufficient number’ is a judgment call for the court. See Expert Evidence in Criminal
Proceedings, above n. 1, at paras 3.49–3.52, commenting on the need for flexibility in applying the
35
test.
36 5 Discussion will be based primarily on the proposal from C. T. Robertson, ‘Blind Expertise’ (2010) 85
37 New York University Law Review 174 and the work of Roger Koppl (and his colleagues), e.g. R. Koppl, R.
38 Kurzban and L. Kobilinsky, ‘Epistemics for Forensics’ (2008) 5 Episteme: A Journal of Social Epistemology
141; R. Koppl, ‘Epistemic Systems’ (2005) 2 Episteme: A Journal of Social Epistemology 91; and R. Koppl,
39
‘Romancing the Law: Legal Failure in Forensic Science Administration’ in E. Lopez (ed.), Government
40 Failure in the Legal System (Palgrave MacMillan: New York, 2010). Koppl does not focus purely on the
41 idea of ‘blinding’, but his proposals do include a blinding element and are highly relevant to
42 addressing the problem of scientific evidence.
208 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
evidence. The second part, considering this diagnosis against the backdrop of the 1
rules of evidence as a whole, provides some parameters for measuring the success 2
of any proposed reforms to this area of the law. Two proposals are then outlined 3
for the introduction of blinding protocols in the acquisition and presentation of 4
scientific evidence. The fourth part of the article analyses and evaluates these 5
proposals in light of the parameters set down in the second part. The final part 6
concludes by suggesting that the blind expertise proposals have considerable 7
merits, but, as is the case with everything else, should not be considered a 8
panacea. 9
10
What is the problem of scientific evidence? 11
12
To diagnose the problems associated with scientific evidence, it will be useful to
13
think of the relationship between those who present evidence in court and those
14
who interpret the evidence as a signalling game.6 In other words, to conceive of it
15
as a strategic interaction between the senders and receivers of information. This
16
strategic interpretation will allow us to see the potential impediments to truth
17
that arise with evidence law in general and scientific evidence in particular.
18
19
A basic signalling game7 would have two players, a sender and a receiver, and two
20
sets of actions. For instance, the sender might have the option to send one of two
21
signals to the receiver (Sig 1, Sig 2). These signals would convey information about
22
the state of the world. This information would function in such a way as to raise or
23
lower the (subjective) probabilities that the receiver attaches to his or her beliefs
24
about the world. Since there are only two signals, we can assume that one of them
25
represents the world as being in State 1, while the other represents it as being in
26
State 2. The receiver must act in response to these signals. We can assume, for
27
simplicity’s sake, that there are only two acts, each of which is an appropriate
28
response for the different states (Act 1, Act 2). An appropriate response leads to a
29
positive payoff; an inappropriate response leads to a negative payoff. Obviously,
30
the goal for the receiver in any game of this sort is to work out which signal repre-
31
sents which state so that he can respond appropriately.8
32
33
34
35
6 Partly because Robertson puts it this way in his essay on blind expertise, as does Koppl in his work. 36
See materials cited above in n. 5. 37
7 Taken from B. Skyrms, Signals (Oxford University Press: Oxford, 2010) ch. 1. Skyrms works from the 38
basic model set down in D. Lewis, Convention (Oxford University Press: Oxford, 1969).
8 An interesting observation from evolutionary game theory is that provided the players can switch
39
roles, play the game repeatedly and have similar payoffs, a system of communication can emerge 40
that is purely conventional and yet can track the actual state of world with reasonable accuracy. 41
This issue is explored at some length throughout Skyrms, above n. 7.
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 209
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 The presentation of evidence to a court of law can easily be conceptualised as a
2 signalling game. In this case, the court is the receiver, and the parties to the
3 litigation are the senders. They present signals (evidence) to the court, and the
4 court works out the relationship between those signals and the actual state of the
5 world (i.e. the facts of the case). The relevant signals will be those that raise or
6 lower the probability that the court attaches to the factum probandum (fact to be
7 proved); the irrelevant signals will be those that do not raise or lower the proba-
8 bility that the court attaches to the factum probandum. If the court works out the
9 relationship between the signals and the state of the world, and can attach appro-
10 priate weight to those signals, they can proceed to apply the principles to the facts
11 in the most legally sound manner.
12
13 The court faces two obstacles to successful performance in this game. The first
14 obstacle is that the signals can be complex and their relationship to the actual
15 state of the world can be difficult to work out.9 This gives rise to something we
16 shall call the problem of incomprehensibility. The second obstacle is that the interests
17 of the various participants in the game are not always aligned: while the court
18 might be interested in getting at the truth so as to serve the interests of justice, the
19 lawyers representing the different sides are primarily concerned with serving the
20 interests of their clients. This can lead to poor quality, biased signals being sent to
21 the court. These signals can have limited, if any, relevance. We shall call this the
22 problem of bias. Let us look at both of these problems in more detail. As we do so, it
23 should become clear why they are particularly acute when it comes to the presen-
24 tation and analysis of scientific evidence.
25
26 Consider first the problem of bias. We can take a classic biblical example as a
27 guide: the infamous judgment of Solomon in the child custody dispute between
28 the two prostitutes.10 The story is recounted in 1 Kings (3:24–28). Both women
29 claimed to be the mother of the same child and they asked Solomon to resolve
30 their dispute. Solomon had to base his judgment on the signals that he was
31 receiving from each of the women. But from his perspective the signals were
32 initially indistinguishable: they both claimed to be the mother and there was no
33 obvious reason to favour one claim over the other. He needed to do something to
34 break the deadlock, to differentiate the true signal from the false one. So he asked
35 for a sword to be brought so that he could divide the child in two and give half to
36 each woman. In response to this, one of the women pleaded with Solomon to spare
37 the child’s life and give it to the other woman. The other woman told him to go
38
39
9 This may be due to the fact that the world is itself complex.
40 10 I take this from A. Dixit and B. Nalebuff, The Art of Strategy (WW Norton: New York, 2008) 237–8. A
41 more complex game theoretic analysis of the story can be found in S. Brams, Biblical Games (MIT
42 Press: Cambridge, MA, 2003, originally published 1980) 118–23.
210 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
through with it. This allowed Solomon to see that the first woman was the true 1
mother (since she was protecting the life of the child) and so he could resolve the 2
dispute appropriately. 3
4
The obvious lesson to be derived from this story is that the adversarial system is 5
sometimes uniquely well suited to producing low quality, biased signals. That is, 6
signals that do not raise or lower the probability of the fact to be proved. This is 7
because only evidence that supports the argument of the sides is likely to be 8
presented and so it becomes impossible to tell whether the signal bears any 9
relation to the actual state of the world. In the case of scientific evidence, there are 10
several distinct forms that this bias can take. First, there is straightforward delib- 11
erate selection bias: the parties only select experts who support their argument. 12
Secondly, there are more subtle forms of unconscious bias. These arise when, even 13
though people might think they are being honest in giving evidence, they are 14
subconsciously biased in favour of conclusions that (1) support their prior beliefs, 15
(2) support the side with whom they are affiliated, (3) support the side by whom 16
they are remunerated.11 17
18
Bias by itself is not necessarily a big problem. Indeed, it is in many ways a distin- 19
guishing mark of the adversarial system. Parties are deliberately encouraged to 20
present as strong as possible an argument for their side of the dispute. Once they 21
do so, courts will then be in a position to compare the strengths and weaknesses of 22
the competing evidence, weigh it appropriately and determine what they think 23
actually happened. This is thought to be an ingenious ‘engine for getting at the 24
truth’.12 But the problem of bias is compounded when the evidence in question is 25
something about which the court is deeply uncertain. This is nearly always the 26
situation with expert evidence, which is, after all, introduced to assist the court 27
when it strays beyond its areas of competence. This is what gives rise to the 28
problem of incomprehensibility. 29
30
13
A classic example of this arose in the case of R v Clark. In that case, Sally Clark, a 31
solicitor, was convicted for the murder of two of her infant children. The evidence 32
against her was relatively thin and the defence tried to rebut the accusation by 33
34
11 These are covered in Robertson, above n. 5 at 184–9; see also Consultation Paper on Expert Evidence,
35
above n. 1, above n. 1 at ch. 4, for an extended discussion of this issue. This report identifies several 36
other forms of bias and notes that courts are alive to the problem of bias. Readable introductions to 37
the psychological experiments and theories underlying the problem of unconscious bias are T.
38
Gilovich, How We Know What Isn’t So (The Free Press: New York, 1991) and S. Sutherland, Irrationality,
2nd edn (Pinter and Martin: London, 2007). 39
12 H. Wigmore, Evidence, 3rd edn (Little, Brown: Boston, 1940) 29. 40
13 R v Clark [2000] EWCA Crim 54, [2000] All ER (D) 1219. A later case overturned the verdict although 41
this was because new evidence came to light: R v Clark [2003] EWCA Crim 1020.
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 211
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 arguing that the children were victims of Sudden Infant Death Syndrome (SIDS).
2 At trial, the jury was told that there was only 1 in 73 million chance of double-SIDS
3 deaths in the same family. The implication, of course, being that the defence’s
4 argument carried, for all practical purposes, no weight whatsoever.
5
6 This, it turns out, was wrong on at least two counts. First, in arriving at the 1 in 73
7 million figure, the expert witness for the prosecution—a famous paediatrician
8 named Roy Meadow—failed to consider the possibility that having one SIDS death
9 in a family raises the probability of having a second one. This might be due to
10 shared genetic or environmental causative factors. In other words, he assumed
11 that the events were statistically independent when they were not. Ignoring this
12 made a big difference to the resulting figure.14
13
14 Secondly, the 1 in 73 million figure was presented out of context. In a trial, the
15 probabilities attaching to different events are meaningless if presented in
16 isolation. After all, what is at issue is not the probability of one event in isolation
17 but the contrasting probabilities of two events. Generically, these are (1) the proba-
18 bility that person is innocent; and (2) the probability that the person is guilty. In
19 the Clark case, the need for context meant that the probability of a double SIDS
20 needed to be contrasted with the probability of a double infanticide. The latter, it
21 turns out, was less probable than the former.15
22
23 The case illustrates just how inappropriate the weighting that is attached to some
24 kinds of expert evidence can be due to the problem of incomprehensibility. The
25 court, and the lawyers for the respective sides, did not understand the nature of
26 the (fairly elementary)16 probability assessments being made, and were conse-
27 quently unable to question the methodology that Meadow had employed when
28 arriving at his figure. This meant they were also unable to understand its signifi-
29 cance. When such difficulties in comprehension are combined with the problem
30 of bias, as they are in the case of scientific evidence, we have a potential recipe for
31 disaster: erroneous fact-finding, miscarriages of justice, and the undermining of
32 the credibility of the legal system. We need to do something to resolve these
33 problems.
34
35
36
37
38 14 On these issues see R. Hill, ‘Multiple Sudden Infant Deaths—Coincidence or Beyond Coincidence?’
(2004) 18 Pediatric and Perinatal Epidemiology 320 at 321–2.
39
15 Ibid. at 322–3.
40 16 Hill’s article, above n. 14, goes through the relevant calculations and does not, by my lights,
41 require anything more than second-level mathematics. The difficulty, as I see it, is finding good
42 data to use in those calculations.
212 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
How do we measure the success of proposed reforms? 1
2
Diagnosing the problems associated with scientific evidence can help point us in
3
the direction of a solution to those problems. To resolve the problem of bias, we
4
need to do something to change the incentives associated with the acquisition and
5
presentation of scientific evidence in such a way that biased evidence becomes less
6
feasible. To resolve the problem of incomprehensibility, we need to arm legal
7
decision-makers with the tools needed for understanding the evidence with which
8
they are presented.
9
10
In measuring the success of any proposed reforms to this area, it is worth stepping
11
back and considering the purpose of evidence and evidence law in general. This
12
process of stepping back will allow us to develop some parameters for measuring
13
the success of proposed reforms. As our first step backwards, notice that the two
14
problems outlined above, incomprehensibility and bias, affect the capacity of the
15
court to work out the facts of the case. Working out the facts of the case is
16
important because the primary function of the court is to apply legal principles to
17
facts.17
18
19
Once this principles-to-facts model has been accepted, the role of evidence and
20
evidence law becomes apparent. We need evidence in order to establish the facts.
21
But not just any kind of evidence will do. The evidence must be of a certain quality.
22
This is where the rules of evidence law come into play. They place limits on the
23
kinds of evidence that can be presented to the court, and thereby maintain the
24
requisite quality control. Three things need to be kept in mind when considering
25
the rules of evidence law.
26
27
First, the rules should ensure that only relevant evidence is admissible.18 This
28
is evidence that actually raises or lowers the probability of the fact to be
29
30
17 This is, of course, somewhat controversial since principles and facts are often opaque and indeter- 31
minate. See J. Coleman and B. Leiter, ‘Determinacy, Objectivity and Authority’ (1993) 142 University
of Pennsylvania Law Review 549 for a detailed exploration of these issues.
32
18 This can be interpreted in a strictly probabilistic manner where relevance is determined by the 33
ratio of likelihoods of competing theories. Let P be the fact to be proved and E be the evidence 34
introduced. For the evidence to relevant, then the probability of P given E, Pr(P|E), must be higher
35
or lower than the probability of P on its own, Pr(P). To know whether the evidence is relevant we
need to consider the ratio of likelihoods, i.e. Pr(E|P): Pr(E|~P). If this ratio is equal to one, then the 36
evidence is not relevant—it does nothing to raise or lower the probability of P. Otherwise, the 37
evidence is relevant. I take this from R. M. Goldman and A. I. Goldman, ‘Review of Truth, Error and 38
Criminal Law: An Essay in Legal Epistemology, by Larry Laudan’ (2009) 15 Legal Theory 55 at 61. For more
39
on the probabilistic framework see T. Anderson, D. A. Schum and W. L. Twining, Analysis of Evidence,
2nd edn (Cambridge University Press: Cambridge, 2005) ch. 9. On whether probability theory is the 40
best way to frame evidence law, see H. L. Ho, A Philosophy of Evidence Law (Oxford University Press: 41
Oxford, 2008) ch. 3 which offers a slightly different theory of legal epistemology. Ho does not 42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 213
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 proved.19 If possible, the court should also be able to assign some appropriate
2 weighting to the evidence. The weighting will help the court determine, either
3 with precision or through approximations, by how much the evidence in
4 question raises or lowers the fact to be proved.
5
6 Secondly, a standard of proof needs to be agreed upon.20 In criminal law, the
7 relevant standard is beyond a reasonable doubt; in civil law, it is the balance of
8 probabilities. The standard of proof reflects the complex, and often imprecise
9 nature of proving facts. It shows that the law need not demand that there be
10 absolute, certain proof of anything: it need only demand that something be
11 proved to an acceptable level of probability. As Laudan notes in his essay on
12 evidence in the criminal law, when crafting a standard of proof we should try to
13 obtain a reasonable distribution of false acquittals to false convictions.21 In other
14 words, we should accept that we are likely to get things wrong on occasion, and we
15 should seek to minimise the damage associated with getting things wrong. In
16 criminal law, the damage is, presumably, felt to be higher, hence the standard of
17 proof is higher.
18
19 Finally, we need to ensure that the rules of evidence are consistent with other
20 values and policy goals to which we hold.22 For example, rules on the acquisition of
21 evidence should be consistent with the right to privacy, autonomy or bodily
22 integrity. Likewise, making sure the procedure is not too costly and makes good
23 use of public and private resources is a legitimate policy concern.
24
25
26
27 dispute the important role that probability assessments may play in legal fact-finding, but he does
dispute the overall conception of legal fact-finding in these terms. Finally, on evidence and
28
probability theory in general (though specifically applied to the philosophy of science), see
29 E. Sober, Evidence and Evolution (Cambridge University Press: Cambridge, 2008) ch. 1 of which
30 provides an excellent introduction to and overview of the debate in the philosophy of evidence.
31 19 I take it that this statement is true no matter what interpretation of probability and proof one
accepts. For example, if one accepts a strictly subjectivist interpretation of probability, relevant
32 evidence can help to raise or lowers one’s subjective probability measure for the event or fact that
33 needs to proved. Likewise, if one accepts an objective interpretation of probability, one assumes
34 that there either is, or is not, some measure that can determine the probability that the fact
occurred. If there are such measures, they will assist in the fact-finding mission of the court; if
35
there are no such measures, that also assists the fact-finding mission (by essentially highlighting
36 the opaque nature of the facts in the particular case). Similarly, all proofs proceed from premises to
37 conclusions. In only a handful of cases (pure mathematics, say) are the premises known with any
38 certainty. In all other cases, probability assessments of some kind must be used to determine the
overall soundness of the proof.
39
20 See Ho, above n. 18 at ch. 4 for more. Also L. Laudan, Truth, Error and Criminal Law (Cambridge
40 University Press: Cambridge, 2006) ch. 1.
41 21 Laudan, above n. 20. Part I of the book is dedicated to this issue.
42 22 Laudan, above n. 20 at 2.
214 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
The preceding characterisation can assist in formulating a set of parameters or 1
criteria for measuring the success of any proposed changes to the law of evidence. 2
They are as follows: 3
4
(1) Truth-directedness: Does the proposed change actually help the court to 5
get at the truth? Does it enable the court to determine what actually 6
happened? 7
(2) Practicality: Is the proposed change practicable? What are its cost impli- 8
cations? How easy would it be to implement? 9
10
Some elaboration and qualification might be in order if these parameters are to 11
become fully defensible. 12
13
First, with regards to truth-directedness, this parameter captures one of the core 14
normative virtues of the legal system in a liberal democratic state. Indeed, 15
according to the theories of epistemic democrats,23 democracies in general derive 16
their legitimacy from their ability to facilitate better, more truth-oriented 17
decision-making. Thus all reforms within a democracy must be assessed in terms 18
of their compatibility with the pursuit of truth. 19
20
Less abstractly, and more directly relevant to the operation of the legal system, 21
having truth-directed processes should ensure that the law is properly applied; 22
that the right to a fair trial is protected; that a reasonable distribution of false 23
acquittals to false convictions is achieved; and that miscarriages of justice are 24
avoided. Thus, truth-directedness is perhaps the central virtue of any proposed 25
reform to the law of evidence, and will consequently be awarded pride of place in 26
the assessment of the blind expertise proposal given below. When engaging in this 27
assessment, it is important to bear in mind that it is unlikely that any reform will 28
actually get us to the truth. Indeed, whether such a thing is ever obtainable is a 29
philosophically contentious matter.24 Still, philosophical worries of this sort 30
should not distract us from the possibility of avoiding the obvious impediments to 31
truth, such as those created by the problem of bias. 32
33
Although truth-directedness is important, its attainment needs to measured in 34
terms of its practicality and cost-effectiveness. As regards to both of these things, it 35
should be noted that any change in the status quo is likely to face, if only in the 36
37
38
23 See C. List and R. Goodin, ‘Epistemic Democracy: Generalizing the Condorcet Jury Theorem’ (2001)
39
9 Journal of Political Philosophy 277.
24 I am, of course, working with the presumption that there is some objective fact of the matter even 40
if I doubt the existence of a practical epistemic route to that matter. If one rejects the existence of 41
objective facts of the matter, one is unlikely to accept anything that is said in this article. 42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 215
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 short term, practical difficulties. This is because change can place burdens on
2 people who have grown accustomed to certain practices. The existence of such
3 burdens should not, however, lead us to ignore the possible long-term benefits
4 that can be derived from these short-term costs.
5
6 What is the blind expertise solution?
7
8 The blind expertise solution focuses primarily on the problem of bias outlined
9 above. Indeed a ‘blind’, as the concept is understood in scientific practice, is
10 simply something that shields an experimenter or investigator from potentially
11 biasing factors. In some ways, the perceptual metaphor is unfortunate because, as
12 is the case with the biasing factors themselves, blinding need not involve any
13 direct changes to conscious perception or awareness, but the term is in common
14 usage and so it will be adopted here. Two examples of blind expertise solutions
15 will be described below. The first comes from Christopher Tarver Robertson and
16 applies specifically to the civil context.25 Robertson’s analysis and defence of the
17 concept is probably the most comprehensive analysis, but since it is limited to the
18 civil context, a second example, from the work of Roger Koppl and his colleagues,
19 is described. This example covers the use of forensics in criminal trials.26 The
20 proposed solutions are considered below; critical commentary is postponed until
21 later in the article.
22
23 (a) Robertson’s proposal
24 Robertson envisages the implementation of a process involving a number of
25 stages, the net result of which is to eliminate selection, affiliation and compen-
26 sation biases.27 The process begins when the parties to the litigation discover they
27 have some problem question, relevant to their litigation, on which they will
28 require expert scientific opinion. At this stage, they will be faced with a choice (it is
29 crucial to Robertson’s proposal that they have this choice): they can opt to select
30 their own ‘hired-gun’ expert, or they can opt for a ‘blind’ expert.
31
32 If they opt for the blind expert, they must submit their problem question to an
33 intermediary body. This would be some sort of professional scientific association
34 or organisation. This intermediary body is itself divided into two groups. The first
35 group will take the litigant’s question and try to remove any indication of the legal
36 purpose behind it. Suppose, for illustrative purposes, that the question is epidemi-
37 ological in nature: the claimants in some toxic tort action want to know whether
38 there is a causal link between, say, the electromagnetic radiation from overhead
39
40 25 See materials cited in n. 5.
41 26 In particular Koppl et al., ‘The Epistemics of Forensics’, above n. 5.
27 The following description is taken from Robertson, above n. 5 at 206–13.
42
216 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
power lines and their leukaemia. To remove bias from this inquiry, the first group 1
will need to eliminate any hint of which side might be asking for this information 2
and avoid tilting the inquiry in one direction or the other. So they would need to 3
ask a non-leading question like: ‘Does exposure to EM-radiation raise lifetime risk 4
of leukaemia? If so, by how much?’ This, after all, is a question to which both sides 5
will need to know the answer. It may also be helpful to break this question down 6
into a set of more specific questions, each one of which is careful to avoid 7
prejudging any particular conclusion. 8
9
Once the litigant’s question has been stripped of any potentially biasing features, 10
it will be sent to the second group within the professional association who will 11
pick the expert whose opinion will be sought. The selection of the expert can 12
either be governed by a randomisation process (provided there is some predefined 13
pool of experts) or can be done deliberately (provided there is no hint of bias in 14
the selection). One may wonder whether deliberate selection would simply 15
reintroduce possibilities for bias. In response, Robertson suggests that there may 16
be no other choice in some cases and that the effect should be minimised if the 17
person doing the selecting is unaware of the litigious purpose that lurks behind 18
the request, which they should be if the first group has done their job properly.28 19
20
After that, the litigant’s question is sent on to the expert who prepares a report 21
and receives a fee for his or her troubles.29 The report is sent back to the interme- 22
diary who in turn sends it back to the litigant. The report will either be favourable 23
to the litigant’s case or not. Robertson argues that the litigant should have the 24
option to ignore the report or try the process again. He does so for a somewhat 25
technical reason discussed below. For now, I simply note that should the litigant 26
try the process again, the number of times he or she does this will be discoverable 27
by the opposing side. This fact alone should provide some disincentive against 28
multiple uses. 29
30
Once the report has been received and selected for use by the litigant, the blind is 31
lifted and the expert can appear in court to answer questions on the evidence 32
33
34
35
28 It is worth comparing this with Koppl’s comments ((b) below) on forensic labs being biased even if
they are not told the purposes or preferences of those making the request for information. This is 36
one of things that leads Koppl to propose his system of competitive redundancy. However, there is 37
a significant disanalogy here since forensics are almost always used by those prosecuting or 38
investigating crime and so it would be difficult not to guess the purpose or preferences of those
39
making the request. Scientific information in toxic tort cases is often required by both sides to the
dispute. I am indebted to an anonymous reviewer for identifying the need for this clarification. 40
29 The fee is not tied to or conditional upon any particular conclusion so the problem of compen- 41
sation bias is avoided. 42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 217
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 discussed in his report. The report would be available to both sides and rigorous
2 cross-examination would be permitted.
3
4 That is, roughly, how blind expertise might play out in the civil context.
5
6 (b) Koppl’s proposal
7
Roger Koppl and his colleagues suggest that blinding could also play a role in how
8
forensics labs supply information for use in assisting criminal prosecutions. In
9
most cases, the job of a forensic lab will be to find out whether a sample (e.g. DNA
10
or fingerprint) taken from a suspect matches a sample taken from a crime scene.
11
The problem, as Koppl sees it, is that the typical management and operation of
12
forensic labs—usually, one lab under police organisation—leads to potentially poor
13
quality signals being sent from the lab back to the investigators, prosecutors and,
14
ultimately, the court. The reason for this is partly to do with the error rates
15
associated with forensic techniques30 and partly due to biasing. As Koppl et al. note:
16
17
Under current institutions in common law countries, crime labs are
18
typically organized under the police. This mode of organization tends
19
to create a pro-police bias [footnote omitted]. The bias need not be
20
conscious. The police generally ask for a test when they believe a
21
match will identify their suspect. Thus, some forensic workers tend to
22
have a bias in favor of finding matches.31
23
24
Thus, removing bias may be somewhat more difficult in the case of forensic labs
25
than it is in the context of civil litigation. The hypothetical epidemiological
26
question discussed above could easily be asked by either party to a litigation, but
27
the norm in forensics is for the requests to come from the prosecutorial side.
28
Consequently, something more creative than merely carefully wording the
29
request must be done in order to ‘blind’ the experts.
30
31
32 Koppl et al. suggest that the requisite blinding could be achieved through a system
33 of rivalrous redundancy (this system has additional benefits to be discussed below).
34 Initially, this would require the existence of multiple independent forensic labs
35 performing the same tests. By itself, this redundancy would not eliminate the
36
37 30 The discussion of jury theorems at pp. XXX below provides the reason for thinking that a single
38 monopolistic lab performing tests with error rates is a problem.
31 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 145. The references cited in support of the
39
contention that there might be a pro-police bias in forensics labs are: D. M. Risinger, M. J. Saks, W. C.
40 Thompson and R. Rosenthal, ‘The Daubert/Kumho Implications of Observer Effects in Forensic
41 Science: Hidden Problems of Expectation and Suggestion’ (2002) 90 California Law Review 1 and P. C.
42 Gianelli, ‘Confirmation Bias’ (2007) 22 Criminal Justice 60.
218 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
pro-police bias, something extra is needed for that. This ‘extra’ is supplied by 1
making their payoffs (remuneration) dependent on both the actions of the other 2
labs and those requesting the information. Specifically, if each lab is made to 3
think that its results need to conform with the results sent by the majority of labs 4
in order to receive payment, it will be incentivised to send the most reliable, 5
truth-directed signal. Why is this? The reason comes from game theoretic consid- 6
erations: the best way to ensure that one is in the majority, when collusion is 7
rendered impossible, is to aim at the truth. Thus, the rivalrous redundancy 8
switches the lab’s attention away from typical biasing factors and gets it to focus 9
on providing the most accurate results.32 In two experiments using simple 10
signalling games, such as those outlined above, Koppl et al. tested to see whether 11
introducing a system of rivalrous redundancy really would improve the reliability 12
of the signals that are transferred between senders and receivers. The results 13
suggested that it would.33 14
15
In addition to a system of rivalrous redundancy, Koppl34 recommends that more 16
straightforward blinding practices be introduced into both the handling and 17
preparation of the samples being used in forensic science, as well as the requests 18
for information from forensic science labs. For example, Koppl suggests that, in 19
the case of fingerprint matching, one person should prepare the sample and 20
another should perform the analysis.35 Similarly, Risinger et al. argue that case 21
information not relevant to the particular forensic technique or test be masked or 22
hidden from forensic examiners.36 23
24
Would a system of blind expertise be any good? 25
26
Now that we have outlined two potential blind expertise-style solutions to the
27
problem of scientific evidence, we can proceed to assess them in light of the two
28
parameters set down earlier.
29
30
(a) The truth-directedness of blind expertise
31
The first question we need to ask is whether the introduction of a system of blind
32
expertise will actually help the court to receive less biased, and more reliable,
33
signals from expert witnesses. There are two reasons for thinking that it would.
34
The first stems from the fact that blinding practices are already widely accepted as
35
36
32 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 146–7. 37
33 Experiments are described in Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 148–54. 38
34 See Koppl, ‘Romancing Forensics: Legal Failure in Forensic Science Administration’, above n. 5 for a
39
discussion of these proposals both from himself and from others such as Risinger et al. and
Gianelli, above n. 31. 40
35 Koppl, ‘Romancing Forensics: Legal Failure in Forensic Science Administration’, above n. 5. 41
36 Risinger et al., above n. 31. 42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 219
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 having that power in scientific practice; the second stems from the fact that the
2 proposals from Robertson and Koppl have the ability to exploit so-called jury
3 theorems. Both reasons will be discussed in some detail. It should be borne in
4 mind that, for the purposes of this analysis, although improving the reliability
5 and removing bias is assumed to get us closer to the truth, it is not assumed to get
6 us actually to the truth. As was mentioned earlier, the truth may often be
7 complex, and disagreement may well remain after the most obvious forms of bias
8 have been removed. Hence the success of blind expertise solutions must be
9 measured in terms of their improvement of the existing situation.
10
11 The first reason37 for thinking that ‘blinding’ can get us closer to the truth is that it
12 is already recognised as being an essential component in the scientific testing of
13 hypotheses, particularly in evidence-based medicine.38 For example, whenever a
14 new drug is being tested for its efficacy, a randomised double-blind controlled test
15 is performed. Very roughly, this means that a group of patients who are thought to
16 be fit subjects for taking the drug are recruited for the study; they are randomly
17 divided into two different groups; one group receives the drug and the other
18 receives a placebo (e.g. sugar pill); neither the patients nor the doctors who are
19 supplying them with their medication know whether they are receiving the
20 placebo or the drug being tested; and the improvements or disimprovements, as
21 the case may be, in the patient’s condition are recorded.
22
23 The double-blind (and randomised) method of testing allows the experimenter to
24 tell whether the active ingredient in the drug is having any effect on the patient’s
25 condition over and above the placebo effect and/or any other environmental or
26 genetic factors. If the protocols are broken and the blinding is lifted, then it
27
28
29
30 37 There is an additional reason for accepting blinding that seems, to me, to be of lesser significance,
31 namely that it is often thought to be an important part of the administration of justice (procedural
fairness). Robertson, above n. 5 at 202–6 gives the example of John Rawls’s theory of justice in
32
which the members of a society are forced to negotiate a social contract from behind a ‘veil of
33 ignorance’, i.e. without knowing what position they will occupy in the society. Rawls felt that this
34 was the only way to satisfy the Kantian principles of impartiality and equal respect. Impartiality is
35 also a prominent feature of the principles of natural justice in administrative law (J. Rawls, A Theory
of Justice (Belknap Press of Harvard University Press: Cambridge, MA, 1971). Robertson thinks that
36 these normative uses of blinding add to its acceptability.
37 38 Robertson, above n. 5 at 202–6. On evidence-based medicine and double-blind tests, see the
38 discussion T. Greenhalgh, How to Read a Paper: The Basics of Evidence-Based Medicine, 4th edn
(Wiley-Blackwell: Chichester, 2010) ch. 4 which goes through the critical questions that should be
39
asked of the methodology underlying any medical study. A readable introduction to the basic
40 components of evidence-based medicine, including the randomised double-blind test, with a
41 critical application to alternative medicine is available in R. Barker Bausell, Snake Oil Science (Oxford
42 University Press: Oxford, 2007).
220 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
becomes impossible to tell whether the ‘signals’ (i.e. the experimental results) that 1
are being observed are due to the drug or these other confounding variables. 2
3
To the extent that both Robertson’s and Koppl’s proposals take advantage of 4
similar blinding and randomisation protocols, they should improve the quality of 5
the signals being received from expert witnesses. After all, as is the case in medical 6
trials, once obvious sources of bias have been removed, it becomes more likely that 7
the differences, if any remain, between the signals being received is due to 8
truth-relevant factors. 9
10
Despite these advantages, there is, perhaps, reason to think that these proposals 11
cannot remove one obvious form of bias, namely the self-selection bias that arises 12
when experts pick their own areas of expertise. To go back to the epidemiological 13
example given earlier, it might be argued that those epidemiologists with the 14
requisite knowledge to answer the litigant’s question have a vested interest in 15
finding a link. This would be because they would not bother specialising in that 16
area unless they thought there was the possibility of finding some sort of link. So if 17
the blind experts chosen through, say, Robertson’s selection process come from a 18
limited self-selecting pool, the net result may simply be to replicate the bias of the 19
given field of expertise. 20
21
This concern may be more apparent than real, there is some evidence, for example 22
in the history of research into links between EM-radiation from overhead power 23
lines and leukaemia, that suggests that while initial research may be driven by a 24
biased belief in the existence of a link, researchers with no biases in favour of 25
finding a link can get involved in research so that, over time, this initial effect is 26
removed.39 Furthermore, in an area like epidemiology, it is usually the method- 27
ological design of the studies that is crucial, not so much underlying specialist 28
knowledge about EM-radiation or the like.40 These methodological issues should 29
be familiar to all epidemiological researchers and so there should be no need to 30
limit the pool to just those specialising in research on the particular proposed 31
link. Indeed, methodological issues are probably the key to evaluating most forms 32
of scientific evidence, so the problem of self-selection should be limited. 33
34
This does, however, bring up another concern: is blind expertise capable of dealing 35
with evidence from emerging areas of scientific knowledge? A failure to do so 36
could be especially worrisome given that these areas are likely to be most prone to 37
38
39
39 G. Kabat, Hyping Health Risks (Columbia University Press: 2008) ch. 4. 40
40 Although note that Kabat, above n. 39, suggests that ignorance of the underlying physics of 41
EM-radiation was one reason why epidemiological research in this area was so misguided.
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 221
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 self-selection bias by researchers. Given the need for a sufficiently diverse pool of
2 experts, there is indeed reason to think that the blind expertise solution is poorly
3 equipped to deal with such areas. This may tip the balance back in favour of a
4 reliability test solution, at least when it comes to the frontiers of scientific
5 knowledge. Indeed, one of the reasons often offered for adopting a Daubert-style
6 reliability test instead of a Frye-test (which focuses on general acceptance in the
7 relevant field) is that the former is more open to emerging areas of knowledge
8 That may indeed be a mark in favour of the formal reliability test, but it should be
9 noted that when assessing the reliability of the novel evidence, judges would still
10 need to be alert to the possibility of self-selecting bias and lack of blinding. In other
11 words, failure to blind would remain an indicator of unreliability.41
12
13 The second reason for thinking that a system of blind expertise might help courts
14 to get at the truth comes from its ability to exploit the power of jury theorems.
15 These are theorems in probability theory showing that having several people
16 ‘voting’ on a question or decision, and going along with the majority answer, can
17 make it more likely that we get the correct answer. The classic jury theorem is that
18 of the Marquis de Condorcet.42 Put most simply, this theorem states that if you
19 have a question for which there are two possible answers, only one of those is the
20 right answer, and the probability of any one person getting the right answer is
21 greater than 0.5, then you are more likely to get the right answer by getting two or
22 more people to ‘vote’ than you would be if you accepted the answer of just one
23 person.43 Indeed, the probability of getting the right answer converges on 1 as the
24 size of the voting pool tends to infinity. The reason for this is that the effects of
25 individual error rates can be minimised by pooling the votes of multiple
26 individuals.
27
28 This basic theorem has been expanded in a number of ways since, so that the
29 limitations imposed by Condorcet (i.e. greater than 0.5 probability of getting the
30 answer right, no more than two possible answers, etc.) are no longer needed. Of
31
32
33 41 Indeed a failure to appeal to a broader segment of the research community will continue to a
34 mark against any proffered evidence. One recent example of this, which happens to be of interest
to this author, is the non-acceptance of fMRI-based lie detection evidence in US courts. See, e.g.,
35
the case of United States v Semrau where, in a Daubert-hearing, the judge rejected the evidence from
36 fMRI lie detection at least partly on the basis of its non-acceptance by the relevant research
37 community. Full reference: United States v Lorne Allan Semrau, 31 May 2010, US District Court for
38 the Western District of Tennessee (Eastern Division) Case No. 07-10074 M1/P, available at
<http://lawneuro.typepad.com/files/semrau.pdf>, accessed 3 May 2011.
39
42 Marquis de Condorcet, Essai sur l’application de l’analyse a` la probabilite´ des de´cisions rendues a` la
40 pluralite´ des voix (Paris, 1785).
41 43 When the number of jurors is even, some tie-breaking rule needs to be introduced to cover the
42 possibility of deadlock.
222 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
particular significance for our purposes is the fact that it still holds true even 1
when the probability of each individual getting the right answer dips below 0.5, 2
provided the mean probability of choosing the correct answer is greater than 0.5 3
across the jury as a whole.44 4
5
Both Robertson and Koppl make use of this basic jury theorem result in their 6
proposals. Robertson, for example, argues (with the requisite mathematical 7
demonstration) that getting both sides of a dispute to avail of his blind expertise 8
protocol will produce a more reliable signal than would be the case if only one side 9
availed of it.45 The reason being, as stated above, that the effect of error rates is 10
minimised. Likewise, Koppl’s call for having multiple labs perform the same test is 11
premised partly on the fact that this takes advantage of jury theorem effects. The 12
success of his proposal in this regard is demonstrated by the experiments he has 13
performed. In these experiments, it is possible to construct the ‘right’ answer and 14
see how well the proposed system of rivalrous redundancy does in reaching it.46 15
Obviously, outside of the lab it is not possible to know in advance exactly which 16
answer is the correct one, but the argument is that the knowledge gained from the 17
experimental setting (and from the probability theorems) should guide policy in 18
the real world. 19
20
There are two major concerns one can have about the application of jury theorems 21
to the proposed system of blind expertise. The first stems from the fact that most 22
extensions and applications of Condorcet’s result rely on the fact that there is a 23
dichotomous choice to be made by the voters (in our case, the experts). The 24
question is whether the types of answers that expert witnesses are asked to give 25
will fit with this two-option ideal. At first glance, it might seem like the answer 26
would be ‘no’. After all, the subject-matter addressed by expert evidence is often 27
complex and may not allow for simple yes/no, either/or responses. However, it may 28
be possible, through careful phrasing and restrictions of the questions asked, to 29
restrict the answers to the dichotomous form. This would certainly seem to be the 30
case in Koppl’s example of the forensic lab being asked whether the fingerprints 31
are a match or not. The answers to such a request, even though probabilistic in 32
form, would be binary in nature. 33
34
35
36
37
44 See B. Grofman, G. Owen and S. L. Feld, ‘Thirteen Theorems in Search of the Truth’ (1983) 15 Theory 38
and Decision 261 and P. J. Borland, ‘Majority Systems and the Condorcet Jury Theorem’ (1989) 38
Statistician 181.
39
45 Robertson, above n. 5 at 214–19. 40
46 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 144–7 and the experiments described at 41
148–54.
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 223
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 In any event, jury theorems can be generalised to cover situations involving more
2 than two options. Christian List, for one, has shown that even when there are more
3 than two possible answers, as long as the probability of getting the right answer is
4 greater than the probability of getting the wrong answer, the majority vote is still
5 more likely to be correct.47 So, for example, suppose there are five possible answers
6 (P1 ... P5), and it is known that P1 is correct. Then, provided that the probability of
7 the voters picking P1 is greater than the probability of them picking P2, P3, P4, or
8 P5, the majority vote is still most likely to be P1. This is true even if the probability
9 of picking P1 is less than 0.5 (e.g. the probability of picking P1 could be 0.3,
10 compared with 0.2 for P2–P4 and 0.1 for P5).
11
12 Although this is certainly a useful result, there remain a couple of difficulties.
13 First, to get the truth-directed benefits of the theorem in such cases may require
14 large pools of voters (how large depends on the probability of getting the right
15 answer) not just the two or three envisaged by Robertson and Koppl. Furthermore,
16 in the multiple-options case, the straightforward majority vote system is not the
17 only procedure that is statistically more likely to arrive at the correct answer.48
18 Thus, some further refinements or even radical expansions of Robertson’s and
19 Koppl’s proposals may be in order.
20
21 The second major objection to the application of jury theorems to blind expertise
22 is that no matter what their scope (two-option or multiple-option), they rely on the
23 assumption that the voters (either individually or as a group) have a greater proba-
24 bility of picking the right answer than they would of picking the wrong answer(s).
25 Some may question whether that assumption holds up in the case of scientific
26 evidence. On the whole, I think the assumption is reasonable. One is entitled to
27 think that experts are more likely to get the right answer than the wrong one, but
28 that depends upon proper blinding protocols being in place. As Koppl notes,
29 having multiple labs performing the same tests will be of little utility if the
30 sources of bias are not removed.49 Thus, the benefits of the jury theorems should
31 be seen to arise only after blinding has been achieved.
32
33
Overall, then, in spite of the various concerns noted above, the blind expertise
34
solution, by utilising accepted protocols from scientific practice and imple-
35
menting the beneficial effects of jury theorems, scores well in terms of
36
truth-directedness. This implies that its introduction should help to protect the
37
38
47 See C. List and R. Goodin, ‘Epistemic Democracy: Generalizing the Condorcet Jury Theorem’ (2001)
39
9 Journal of Political Philosophy 277 (Appendix 1 for the formal proof).
40 48 Different possibilities such as Condorcet’s own multiple pairwise voting system (i.e. voting
41 repeatedly on two pairs of possibilities) are acknowledged in List and Goodin’s article, above n. 47.
42 49 Koppl et al., ‘The Epistemics of Forensics’, above n. 5 at 146.
224 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
right to a fair trial, ensure that there is a reasonable distribution of false acquittals 1
to false convictions, and prevent potential miscarriages of justice. These are not 2
inconsiderable benefits. 3
4
(b) Practicality 5
The next question we need to ask is whether the blind expertise solution is 6
practical, whether it is something that could be readily implemented. As noted in 7
the second part of this article, we should be on our guard here: changes to the 8
status quo are often difficult to accept, and so there is a tendency, when consid- 9
ering the practicalities of those changes, to be quite negative. However, practical 10
hurdles also face other proposed changes to this area of law, such as the 11
programme of judicial education required for the reliability test solution. It is the 12
relative practicality of these proposals, measured over the long term, that counts. 13
14
The discussion here will focus initially on two major dimensions to the practi- 15
cality of the proposals under consideration. The first of these dimensions concerns 16
the infrastructural changes that would be required should these proposals go 17
through; the second concerns the financial impact of the proposals. Although these 18
two dimensions may overlap (and so are not perfectly orthogonal to each other, as 19
would be required if they were to fit the mathematical definition of a dimension), 20
they are sufficiently distinct to merit independent consideration. In addition to 21
these two major dimensions of practicality, a selection of further practical issues 22
are addressed below. 23
24
Let us look first to the infrastructural problems associated with Robertson’s 25
proposal. Crucial to that proposal is the existence of an administrative barrier 26
between the litigant and the expert. This would require the setting-up of an 27
agency that would strip the litigant’s questions of potentially biasing factors, and 28
select appropriate experts (possibly through a randomisation process) to answer 29
those questions. Robertson argues that it could be quite easy to create such an 30
administrative infrastructure if we take advantage of extant not-for-profit profes- 31
sional scientific organisations. The members of such organisations would benefit 32
from having such a system in place because they could still earn personal fees for 33
their services. Robertson argues that in the United States the American Associ- 34
ation for the Advancement of Science (AAAS) could act as the relevant 35
administrative agency.50 One could imagine something similar being done on, 36
perhaps, a pan-European basis through the European Academy of Sciences.51 If 37
38
50 Robertson, above n. 5 at 206–7. Partly this is because the AAAS already has an infrastructure that
39
judges can use for selecting court-appointed experts. Robertson notes that this infrastructure is 40
largely unused and could be retooled for his system. 41
51 See <http://www.eurasc.org>, accessed 3 May 2011.
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 225
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 that proves to be infeasible (due perhaps to language barriers and jurisdictional
2 conflicts) a national organisation might be more appropriate. Failing that, one
3 could imagine the associations covering the major professions, such as medi-
4 cine and engineering,52 performing the relevant administrative function. More
5 narrowly focused groups of scientists would be difficult for lawyers to identify and
6 locate (although perhaps no more difficult than finding a ‘hired gun’ expert). Also,
7 narrower focus would make the self-selecting bias more of an issue. For instance,
8 seeking a blind expert from the (strictly hypothetical) Union of Epidemiologists
9 Concerned about the Effects of EM-Radiation from Overhead Power Lines is
10 unlikely to solve the problem of bias that the system is designed to solve.
11
12 If there is no readily available extant scientific organisation that could perform
13 the task, it might be necessary for a new, independent authority to be set up. This
14 would have to be organised on a not-for-profit basis so as to avoid any remuner-
15 ative biasing, and could carry with it some significant cost implications. However,
16 the impact might be lessened if the system could be self-financing. Robertson
17 suggests that it could be, and we will consider this possibility in a moment.53
18
19 Turning to Koppl’s proposal, the infrastructural problems there seem less
20 daunting. True, separate labs would need to be put in place, and it would be
21 preferable if they were not dependent on the police force or prosecutorial services
22 for direct funding. Nonetheless, there do not appear to be any serious impedi-
23 ments to their establishment. The major problem would be maintaining the
24 independence of the labs. This would be essential in order to preserve the
25 anti-biasing rivalry that needs to exist between the labs. To resolve this problem,
26 one could perhaps rely on the good faith of the scientists in the respective labs and
27 hope that they would not engage in any direct collusion. But this might be naïve
28 and some system of regulation and investigation could be both necessary and
29 wise.
30
31
32
33
34 52 Which are usually the most frequently consulted professions in legal cases.
53 An anonymous reviewer of an earlier draft of this article suggested an additional infrastructural
35
problem that might arise from Robertson’s proposal. This had to do with the effort to strip the
36 litigant’s question of all biasing factors. The reviewer wondered whether, once the expert’s answer
37 to this question had been received, a third-party panel would be required to reapply the answer to
38 the factual matrix of the case. I submit that this would be a bad idea because it would create an
additional opportunity for biasing of the evidence. It would be much better simply to have the
39
expert’s report submitted as evidence. If the expert is questioned at trial, the questioning would be
40 based on the report, which could be scrutinised by all parties. The actual application of the report
41 to the facts of the case would be ultimately left to the court, although the parties could, obviously,
42 offer their interpretation.
226 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
Turning to the financial dimension to practicality, Robertson argues that in the 1
civil context the system can be self-financing and self-regulating.54 This is 2
because parties will be incentivised to make full use of it, and will be required to 3
pay the experts directly (not the intermediary organisation) for the privilege. 4
One might wonder why the parties would have an incentive to use a system in 5
which they have no control over whether the expert supports their interpre- 6
tation of the case. The reason, according to Robertson, is that they will be 7
trapped in a Prisoners’ Dilemma: it would probably be best for both parties to 8
agree that neither of them will use the system and to stick with hired-gun 9
experts, but if they did that one party could gain a strategic advantage by unilat- 10
erally breaking their promise and using the system to get an answer in their 11
favour. So, in the end, both parties end up using the system to their mutual disad- 12
vantage, but to the court’s advantage. 13
14
The success of Robertson’s argument depends on a couple of conditions. First, 15
judges and juries will need to be made aware of the epistemic advantages 16
bestowed by the blind protocol. This would require some sort of educational 17
programme, even if it does not rise to the level envisaged by the introduction of a 18
formal reliability test. Secondly, the parties will need to be given the option of 19
using the system multiple times, with the caveat that the number of times they 20
use the system will be discoverable by the opposing side. These conditions seem 21
reasonable and attainable, and if the natural antagonism between the two sides to 22
the litigation can be coerced into the strategic dynamics of a Prisoners’ Dilemma, 23
this would create a significant incentive to use the system and should, by proxy, 24
improve the reliability of the evidence being presented to the court. 25
26
Moving on to Koppl’s proposal, one might suppose that the creation of multiple 27
forensic labs or the performance of multiple tests within the one lab, along with 28
some system of regulation, would give rise to serious costs that would have to be 29
discharged from the public purse. However, Koppl offers some reasons for 30
thinking that the cost implications associated with his proposal are not as severe 31
as might first be thought. Indeed, he suggests that—at least in the case of finger- 32
print matching—the triplicate performance of the forensic analysis may lead to a 33
net reduction in costs. The precise details of his costing are somewhat intricate,55 34
but the basic logic is easily expressed. First, he notes that all forensic techniques 35
have certain rates of false positive and false negative errors associated with 36
them. False positive errors, i.e. errors that falsely link a suspect to a crime scene, 37
38
39
54 Robertson, above n. 5 at 230–41 sets down these arguments. 40
55 They can be found in Koppl, ‘Romancing Forensics: Legal Failure in Forensic Science Adminis- 41
tration’, above n. 5.
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 227
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 will give rise to false convictions and these false convictions will give rise to costs
2 (associated with imprisonment and prosecution). Performing multiple forensic
3 tests also gives rise to costs, but because the performance of multiple tests can
4 reduce the number of false positives, it can also reduce the number of false
5 convictions. This could, in turn, lead to a net reduction in costs. Using US-based
6 statistics on the numbers and costs of convictions arising from the use of finger-
7 print evidence, and estimating a false positive rate of 0.8 per cent, Koppl shows
8 how triplicate fingerprint matching tests should result in significant annual
9 savings (upwards of $60 million per year).56 One can always quibble with the
10 assumptions used in costings of this sort, but with (for example) the annual
11 per-prisoner costs of incarceration standing at 77,222 in Ireland in 2009,57 it
12 would surely be prudent and possibly illuminating to perform detailed costings
13 to see whether or not significant savings could be made from implementing
14 Koppl’s proposal.
15
16 Moving now to other practical issues, one concern is that some varieties of scien-
17 tific evidence will be ill-suited to the blinding procedure, at least in the strict
18 form envisaged by Robertson. For example, fMRI-based lie detection tests have
19 recently been offered as evidence in criminal trials in the United States.58 The
20 questions that make up these tests need to be formulated through close consul-
21 tation with the person being tested59 and because of the interaction between the
22 tester and the subject of the test there is a real potential for biasing.60 Given the
23 personal nature of the test, it is difficult to see how this potential could be
24 prevented from being realised. Neverthelss, some blinding is usually possible.
25 For example, in the case of a lie detector test different people can administer and
26 interpret the results of the test (this can be aided by computer programs that
27 decode the relevant data and which are free of bias) thereby ensuring some level
28
29
30
31 56 Ibid. Koppl includes the possibility of false exonerations in his analysis, but the number is small.
57 Irish Prison Service, Annual Report 2009, 4, available at <http://www.irishprisons.ie/documents/
32
AnnualReport2009PDF.pdf>, accessed 3 May 2011.
33 58 See United States v Lorne Allan Semrau, 31 May 2010, US District Court for the Western District of
34 Tennessee (Eastern Division) Case No. 07-10074 M1/P.
35 59 Nothing I say here should be taken to indicate a belief in the actual reliability or validity of lie
detection tests. I am actually deeply sceptical about it. Nonetheless, the example is one with which
36 I am familiar and it seems appropriate to discuss it here. I should note that fMRI lie detection
37 evidence seems to be the perfect example of a fringe area of science with self-selecting experts who
38 have a vested interest in its success.
60 Indeed, this seems to have been an issue in the Semrau case. The person performing the test was
39
willing to discount one set of results that did not happen to be favourable to his client’s interests
40 on the (controversial) grounds that the client was too tired for the results to be accurate. There are
41 further problems with lie detection tests due to the way in which the questions that are asked
42 during the test are formulated after a consultation between the tester and the testee.
228 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
of impartiality.61 At a minimum, the possibilities for blinding should be explored 1
in these cases. 2
3
There are some further miscellaneous practical advantages to the blind expertise 4
solution. First, as Robertson argues, the system might be preferable to the intro- 5
duction of a system of court-appointed experts.62 His argument is premised on the 6
fact that a court-appointed system has had a negligible effect in the American 7
states that have introduced it. It seems that, in practice, few judges are willing to 8
employ their powers to appoint experts.63 This may be due to deeply engrained 9
norms of legal practice which favour the adversarial process. Robertson argues 10
that his proposal, insofar as it takes advantage of the competitive, adversarial 11
nature of the trial process, is more compatible with legal tradition. We can add to 12
this the fact that a single, monopolistic court-appointed expert would not realise 13
the potential of the jury theorems discussed earlier. One could, of course, imagine 14
multiple court-appointed witnesses being selected via some blinding procedure, 15
but this would seem to be redundant if the same effect can be achieved through 16
the adversarial process. It would also seem to rule out the self-financing advan- 17
tages of Robertson’s proposal. 18
19
In conclusion then, although there are practical hurdles facing the introduction 20
of a system of blind expertise, they are not insurmountable. Indeed, such a system 21
may be less difficult and less costly to implement than the reliability test solution 22
currently favoured by law reform agencies in Ireland and England since such 23
proposals would require the introduction of substantial judicial education 24
programmes. Furthermore, the solution may have certain financial advantages 25
associated with it, both in terms of its ability to reduce the costs associated with 26
erroneous incarcerations (in the criminal context) and its capacity to be 27
self-financing (in the civil context). 28
29
Should a system of blind expertise be introduced? 30
31
I think the answer to this question is a firm ‘yes’ with a ‘but’. The qualification is a
32
sop to the fact that there are really three questions to be considered here: (1)
33
Should a system of blind expertise be introduced on its own? (2) Should it be intro-
34
35
61 Or consider the proposal from Michael Bishop and J. D. Trout suggesting that, wherever possible, 36
statistical prediction rules should be used instead of human judgment. See M. Bishop and J. D. 37
Trout, ‘50 Years of Successful Predictive Modeling Should Be Enough: Lessons for Philosophy of 38
Science’ (2002) 69 Philosophy of Science: PSA 2000 Symposium Papers (supplement) S197–S208. And see
also M. Bishop and J. D. Trout, Epistemology and the Psychology of Human Judgment (Oxford University
39
Press: Oxford, 2005). 40
62 Robertson, above n. 5 at 198–201 and Section V. 41
63 Ibid. at 198–201.
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 229
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
1 duced in addition to a reliability test? (3) Should it be introduced in preference to a
2 reliability test?
3
4 The first question is really asking: assuming no other changes to the existing legal
5 position with respect to scientific evidence are possible, should a system of blind
6 expertise be introduced? The answer to this is straightforwardly ‘yes’. The existing
7 legal position, in Ireland at any rate, is that the courts simply check the qualifica-
8 tions of the expert and the relevance of his or her evidence. This system does little
9 to resolve the problems of bias and incomprehensibility outlined in the second
10 part of this article. Introducing a system that does something to resolve at least
11 one of those problems is clearly going to be better than doing nothing. The
12 position in England is little better with judges asked to determine whether there is
13 a body of knowledge that is sufficiently well organised to be considered reliable.
14
15
The second question is really asking: given that the plan is to introduce a
16
reliability test, should a system of blind expertise also be introduced? I think the
17
answer to this is a straightforward ‘yes’. The reliability test proposal that was
18
outlined at the beginning of this article is clearly intended to resolve the problem
19
of incomprehensibility: it provides the court with a set of criteria that should
20
allow it to ascertain how reliable the evidence being presented really is. As we saw,
21
the problem of incomprehensibility is only one half of the problem of scientific
22
evidence. A system of blind expertise is focused on resolving the other half of the
23
problem. Thus, the reliability test and the system of blind expertise could be
24
almost perfect complements to one another and their joint introduction might
25
constitute an almost ‘ideal’ solution to our problems. That said, it is important to
26
emphasis that it would not actually constitute an ideal. Residual problems would
27
certainly remain. For example, a reliability test and an associated programme of
28
judicial education might help judges comprehend scientific evidence, but it does
29
nothing, by itself, to help the juries who will also have to understand the nature of
30
such evidence.
31
32
33 Finally, the third question is asking: if we had to choose between a system of blind
34 expertise and the introduction of a reliability test, which should we opt for? This
35 is, of course, the most difficult question to answer. It asks us to decide between
36 two options that are, as just pointed out, not direct alternatives but, rather,
37 complements. That said, accepting that having both might be preferable, the
38 correct answer is still the system of blind expertise. Why? Primarily, because the
39 truth-directedness and practicality of blind expertise seem to me to be superior to
40 those of the reliability test. Both of these have been discussed already. To reiterate,
41 a system of blind expertise makes use of protocols that are already widely accepted
42 in scientific practice to ensure that more reliable information is presented to the
230 THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF
BLIND EXPERTISE AND SCIENTIFIC EVIDENCE
court. A reliability test merely gets judges to perform a similar check. 1
Furthermore, the proposals discussed above seem to have reasonable cost-savings 2
associated with them when compared to the introduction of a reliability test. The 3
only serious question mark is whether the necessary infrastructures could be 4
readily put in place. I have given some reasons for thinking that the obstacles 5
towards doing so are not great, but the devil, as always, will be in the details. Thus, 6
although it would not be ideal, I submit that the introduction of a system blind 7
expertise instead of a reliability test should be given serious consideration. 8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
THE INTERNATIONAL JOURNAL OF EVIDENCE & PROOF 231