In a 2015 case, Kansas v. Nebraska, Antonin Scalia wrote a separate opinion to offer a scathing assessment of the value of Restatements:

I write separately to note that modern Restatements—such as the Restatement (Third) of Restitution and Unjust Enrichment (2010), which both opinions address in their discussions of the disgorgement remedy—are of questionable value, and must be used with caution. The object of the original Restatements was “to present an orderly statement of the general common law.” Over time, the Restatements’ authors have abandoned the mission of describing the law, and have chosen instead to set forth their aspirations for what the law ought to be. Restatement sections such as that should be given no weight whatever as to the current state of the law, and no more weight regarding what the law ought to be than the recommendations of any respected lawyer or scholar. And it cannot safely be assumed, without further inquiry, that a Restatement provision describes rather than revises current law. [citations omitted]

Two years later, Will and I wrote a paper with our colleague Anup Malani arguing that anyone trying to make descriptive claims about the state of the law on a given topic should consider conducting a so called “systematic review.”

The basic idea of a systematic review is to transparently lay out the process you used to reach your conclusion. For instance, if I wanted to know Scalia’s opinion was on Restatements, I could either just say “Scalia hated Restatements, see Kansas v. Nebraska.” Or, I could conduct a systematic review by searching every opinion and academic article written by Scalia for mentions of the Restatement, and then document whether he discussed Restatements favorably or not. Obviously, that kind of comprehensive review is a lot more time intensive, but it also reduces error. So if I really cared about Scalia’s opinion of Restatements (which, for the record, I don’t), it might be worth investing the effort.

In the article, we suggested that Restatements are exactly the kind of instance where the costs of conducting systematic reviews are worth it. When writing our paper, we were inspired in part by the methods that the reports for the Restatement of Consumer Contracts — Oren Bar-Gill, Omri Ben-Shahar, and Florencia Marotta-Wurgler — were using to code cases as part of their efforts to accurately describe the law.

We thought that reviews of the law would be more accurate because of this kind of coding, but also that the added transparency would allow replication. Or, as we put it at the time: “being explicit about the method is almost as important as the method itself, because transparency allows others to replicate the review author’s work, ensuring that the review was not manipulated and increasing confidence in the review’s conclusions.”

Since writing that, it’s been exciting to see that kind of replication play our with the Restatement of Consumer Contracts. The reporters were transparent about their process of coding cases, and Gregory Klass and others then sought out to reassess their claims through replication. In doing so, they raised a number of concerns with the original analysis, which in turn lead to a response from Oren, Omri, and Florencia explaining how the replication effort still came to the same substantive conclusions.

And although there has been some controversy over the Restatement of Consumer Contracts, their work demonstrates the kind of concrete academic exchange that is possible when reporters take the time to be rigorous. So although I’m not sure on the answer to every questions Will raised in his post, but I know that whatever reporters do, they should be transparent.