A provocative experiment pits AI against federal judges

1 Advances in artificial intelligence can prompt heady predictions of utopia and apocalypse.

2 But they can also prompt reflection about the purpose of the human institutions AI threatens to replace.

3 Take a recent University of Chicago paper pitting real federal judges against ChatGPT.

4 The bottom line is that real judges appear to be more easily swayed by “legally irrelevant” factors than artificial intelligence presented with the same material.

5 That result, however, contains a delicious duality: It highlights either the manifest fallibility of human judges or their superior wisdom.

1 In 2015, 31 federal judges attending a Harvard Law School professional workshop agreed to participate in an experiment.

2 The judges had 55 minutes to decide whether to affirm or overturn the defendant’s conviction and give reasons.

3 The researchers varied the materials in two ways.

4 First, sometimes the defendant was presented, subtly, as more sympathetic, including by expressing “deep regret at all bloodshed in this tragic war.”

5 Second, some judges were presented with a precedent hinting that the defendant’s conviction was legally flawed, while others were presented with a precedent hinting that the conviction was valid.

1 The result: The precedents didn’t matter much either way.

2 Instead, the judges’ decisions were strongly correlated with whether the defendant was presented sympathetically (even though the judges wrote that their decisions were based on the law).

1 Now Eric A. Posner and Shivam Saran at the University of Chicago Law School have released a working paper comparing the performance of the judges with that of a modern AI.

2 In contrast to the judges tested a decade ago, they found, GPT’s likelihood of overturning a conviction is “unaffected by whether the defendant is portrayed as sympathetic or unsympathetic.”

3 And: “GPT follows precedent more consistently, demonstrating a higher likelihood of affirming when the precedent supports affirmation and a lower likelihood when precedent supports reversal.”

1 GPT’s legal stringency might seem impressive.

2 But there’s another twist: Lowly law students are stringent, too.

3 Spamann and Klöhn, who designed the 2015 experiment on judges, more recently ran the same test on law students at three elite schools.

4 Like GPT, the law students’ decisions to affirm or throw out the defendant’s conviction were strongly influenced by which precedent was put in front of them.

5 After all, to impress their superiors, law students need to be able to demonstrate their knowledge of precedents in exercises like these.

1 But federal judges, lords of their courtrooms, might feel more leeway to exercise discretion, knowingly or not.

2 Does GPT’s greater “formalism” make it a superior judge?

3 Nonsense, conclude Posner and Saran, unless you think law students are ideal judges, too.

4 Sometimes the law is ambiguous; in this experiment, the precedents were purposely designed to be suggestive rather than conclusive.

1 The pressing issue, in any case, isn’t whether AI or human judges are better.

2 Human judges are in the Constitution, and they aren’t going anywhere.

3 But this research is a reminder that the legal system isn’t just about doctrinaire rule-following.

4 Law involves a good deal of puzzle-solving, which AI excels at.

5 But the legal system is intertwined with human qualities that maintain its legitimacy and coherence.

6 These qualities are still far from being understood, much less replicated by any machine.