Research as Consumption

The only thing bad about Will’s recent post, Unhelpful Tips for Junior Scholars, is the title. The advice is all helpful. And in addition to giving a lot of great tips that junior scholars should follow, Will also hit on my main piece of advice for people deciding which junior scholars to hire: focus on people that view research as consumption.

Let me explain why this so important. The mission of academic departments is the production and dissemination of knowledge. Even after tenure, you can force people to disseminate knowledge (e.g. teach). But it’s hard to force someone to produce it (e.g. research). This problem is true in all academic departments, but it’s exacerbated by several structural features of law schools: tenure clocks are short and denial rates are low; salary ranges are compressed; and there are non-research, legal related opportunities for faculty to spend their time on that will provide them with money, prestige, and stimulation.

So although it’s critical to hire people that are going to continue to be active researchers and help produce knowledge, it’s hard to compel them to do it for very long. The solution is to focus on hiring scholars that love research and would be doing it regardless of the professional rewards. The kind of people that love nothing more than a day alone in their office, that fantasize about starting new projects, and that love learning about other new research because they find it interesting. These are the people that are worth betting on as being future great scholars when making massive long-run investments in their hiring.

But identifying people that love research isn’t just important because its a good characteristic to focus on when predicting that individual’s own future scholarly output. Having people that love research around is massively important for peer effects. I’m pretty sure there is research on this point, but it’s definitely been my experience anecdotally.

When someone is an active researcher, there are things they’re learning about that few other people in the world know. So, over a meal, I can almost always learn something novel from someone that has a lot of active research interests. Often, the things I learn make me want to start new projects or help me improve current ones. And being around great, productive researchers makes me think I need to push harder to produce good scholarship.

But when I’m at a meal with an academic that is not actively engrossed by the researcher process, I find that that dinner conversations slide toward whatever is currently being covered on the landing page of the NY Times or the TV that people are watching. This might make for a fun evening, but it never does much to improve my projects. And it makes me think I should read more news and watch more TV.

Great News: Someone is Reining in the IRB

A few years ago, some collaborators and I wrote a paper trying to assess whether law clerks systematically influence votes at the Supreme Court. The specific research question was: do justices cast more liberal votes in years they have particularly liberal clerks and more conservative votes in years they have particularly conservative clerks?

The tricky part of answering this question is that Justices likely hire clerks that reflect their ideology at the time. For instance, if a justice is trending to the right over time, the ideology of the clerks they hire might trend to the right over time too. Fortunately for us though, the fact that Supreme Court clerks are hired during earlier terms makes it possible to account for this. I’ll spare you the equations, but we could control for the Justices’ ideology in the prior term as a way to account for whether Justices preferences for hiring particularly conservative or liberal clerks was changing.  

Our key assumption was thus that Supreme Court Clerks are hired either during the term before they started or earlier. But it would be a problem for us if most clerks were hired and started working on cases immediately. How did we know that rarely happened? We just do. 

Despite our assertion that we were sure that was the way that it worked, during the peer review process we ran into some referees that didn’t want to take our word for it. So we decided to email a random sample of 10 percent of Supreme Court clerks in our database and ask them when they were hired to clerk. Here’s the email we decided to send to the former clerks:

Dear [Former Clerk],

I am law professor at the University of Chicago Law School, and I am currently conducting research on the Supreme Court. As part of that research, I am trying to understand when Supreme Court clerks were hired for their clerkship. It is my understanding that you are a former Supreme Court clerk. I was hoping you would be willing to answer two short questions for me:

1. What is the month and year that you were offered your Supreme Court clerkship?

2. What is the month and year that you started your Supreme Court clerkship?

If you do not remember, any information would be helpful (as well as simply knowing that you do not remember). Your answers will be kept confidential. I am simply trying to document the average amount of time Supreme Court clerks are hired before they started, and will not in any way reveal personalized information.

Thank you for your time and help,

Adam

But sending out emails like this to get data for research is human subject research, and thus subject to IRB review. (Don’t know what an IRB is? Then this post isn’t for you.) So I submitted the survey to the University of Chicago’s IRB and explained that I wanted to ask 102 of the countries most sophisticated lawyers the date they were offered their clerkship just to confirm something that is widely known in legal circles.

In response, the IRB asked me to first send the former clerks a consent form to get their agreement to participate in our research. I went back and forth with the IRB explaining that requiring a consent form before soliciting answers would ensure that no one responded to my short email. Finally, we reached a compromise. I would include this in every email:

This research has been approved by the University of Chicago Institutional Review Board. If you have any questions about participating in this research, you can contact the Social & Behavioral Sciences Institutional Review Board at the University of Chicago, 1155 E. 60th Street, Room 418, Chicago, IL 60637. Phone: (773) 834-7835. Email: sbs-irb@uchicago.edu. 

Although I’m complaining about it now, getting IRB approval for this project wasn’t a huge deal. But it did it involve filling our forms and several rounds of emails. And, more importantly, it held up the research a few weeks. And this story isn’t an outlier. Talk to any researcher at an American University that collects data in any way, and they’ll have plenty of IRB red tape horror stories.

That’s why I’m excited that my colleague Omri Ben-Shahar just wrote an essay to announce a pilot he is launching to rein in the IRB at the University of Chicago. The basic idea of the pilot is that the federal law exempts many kinds of research from IRB review, but right now IRB across the country have decided they should be the one to make the determination of whether a project falls into one of the exemptions. And when they make that determination, even if the project qualifies as exempt, they can ask for revisions. (That’s what happened in my story: even though the IRB agreed my harmless email to former clerks was exempt, they still had me add the IRB language to the email.)  

In this pilot, researchers will get to make the determination of whether their research qualifies for one of the IRB exemptions for themselves. But that’s not all the pilot is doing: it’s randomizing which professors at the University of Chicago get to be part of the pilot. That way, in a few years, Omri and others can look back and see if the people in the control group (that remained subject to IRB review) committed any few instances of harmful research. In other words, the IRB reform is being done as a research project.

Like with our survey to clerks, I’m pretty sure I already know what Omri’s research will find: there will not be any differences in research misconduct between the professors in the pilot and professors subject to the normal IRB regime. But it’s great that someone is running this experiment to generate the kind of evidence we need to rein in the IRB for everyone.

---- 

Oh, and in case you are in suspense about the results of the survey: the people that responded confirmed they were hired a while before they showed up for their clerkship.

Note: I edited this post because I misspelled “rein” as “reign” and law twitter dragged me for it. On the plus side, I finally found a way to get people to engage with one of my blog posts.

The Case for a Signature Prize for Legal Scholarship

There was no Pulitzer prize for fiction awarded in 2012. This decision not only upset eligible authors, but it also was terrible for bookstores. By failing to award a prize, the Pulitzer committee had essentially sent the message to the broader public that there weren’t any new novels worth reading that year. As a result, book sales suffered across the board.

I was thinking about this example when I attended a talk a few weeks ago for one of this year’s Nobel Prize winners in economics, Abhijit Banerjee. The audience filled an auditorium that’s probably normally used for plays or concerts (I actually have no idea what usually happens in the auditorium; it took an economics talk being scheduled in the space for me to learn it existed). Although economists are revered here at the University of Chicago, that kind of audience is still extremely unusual.

Abhijit’s talk not only highlighted his own research projects, but also the work of many other economists. It thus not only increased the awareness of the randomized control trials on development that this year’s laureates are now famous for, but the discipline as a whole.

Without his recent publicity for winning the Nobel, however, I’m confident that Abhijit would have spoken in front of a few dozen people instead of a few hundred. (One reason for that confidence: space was not a concern when Oliver Hart presented at our law and economics workshop in a basement seminar room just before he won the Nobel Prize in 2016.)

By failing to have major prizes like other academic disciplines (not to mention literature, music, and movies), legal scholars miss out on the opportunity to highlight our work to the broader public. This not only deprives recognition for the superstars at the top of our profession that would win the award, but also means that the public will not be exposed to legal scholarship more generally. Or, in other words, by not establishing a signature prize, we are sending the message to the broader public that there isn’t any legal scholarship worth learning about. But instead of sending that message one year, we send it every year.

Restatements Should be Transparent

In a 2015 case, Kansas v. Nebraska, Antonin Scalia wrote a separate opinion to offer a scathing assessment of the value of Restatements:

I write separately to note that modern Restatements—such as the Restatement (Third) of Restitution and Unjust Enrichment (2010), which both opinions address in their discussions of the disgorgement remedy—are of questionable value, and must be used with caution. The object of the original Restatements was “to present an orderly statement of the general common law.” Over time, the Restatements’ authors have abandoned the mission of describing the law, and have chosen instead to set forth their aspirations for what the law ought to be. Restatement sections such as that should be given no weight whatever as to the current state of the law, and no more weight regarding what the law ought to be than the recommendations of any respected lawyer or scholar. And it cannot safely be assumed, without further inquiry, that a Restatement provision describes rather than revises current law. [citations omitted]

Two years later, Will and I wrote a paper with our colleague Anup Malani arguing that anyone trying to make descriptive claims about the state of the law on a given topic should consider conducting a so called “systematic review.”

The basic idea of a systematic review is to transparently lay out the process you used to reach your conclusion. For instance, if I wanted to know Scalia’s opinion was on Restatements, I could either just say “Scalia hated Restatements, see Kansas v. Nebraska.” Or, I could conduct a systematic review by searching every opinion and academic article written by Scalia for mentions of the Restatement, and then document whether he discussed Restatements favorably or not. Obviously, that kind of comprehensive review is a lot more time intensive, but it also reduces error. So if I really cared about Scalia’s opinion of Restatements (which, for the record, I don’t), it might be worth investing the effort.

In the article, we suggested that Restatements are exactly the kind of instance where the costs of conducting systematic reviews are worth it. When writing our paper, we were inspired in part by the methods that the reports for the Restatement of Consumer Contracts — Oren Bar-Gill, Omri Ben-Shahar, and Florencia Marotta-Wurgler — were using to code cases as part of their efforts to accurately describe the law.

We thought that reviews of the law would be more accurate because of this kind of coding, but also that the added transparency would allow replication. Or, as we put it at the time: “being explicit about the method is almost as important as the method itself, because transparency allows others to replicate the review author’s work, ensuring that the review was not manipulated and increasing confidence in the review’s conclusions.”

Since writing that, it’s been exciting to see that kind of replication play our with the Restatement of Consumer Contracts. The reporters were transparent about their process of coding cases, and Gregory Klass and others then sought out to reassess their claims through replication. In doing so, they raised a number of concerns with the original analysis, which in turn lead to a response from Oren, Omri, and Florencia explaining how the replication effort still came to the same substantive conclusions.

And although there has been some controversy over the Restatement of Consumer Contracts, their work demonstrates the kind of concrete academic exchange that is possible when reporters take the time to be rigorous. So although I’m not sure on the answer to every questions Will raised in his post, but I know that whatever reporters do, they should be transparent.

How Should Restatements Restate the Law?

Two weeks ago I spent the day at a meeting for the project to create a Third Restatement of Conflict of Laws. For the reasons I discussed on another blog long ago, I remain unsure that another Restatement of Conflict of Laws is a good idea, but working on the project has made me think about how tricky the project of Restatements is.

One much-discussed problem is the relationship between positive and normative analysis. Most of the time the restatements describe the majority rule, but sometimes they instead recommend a minority rule or a new rule. As I understand it, the general practice is that this is fine, so long as it is explicitly disclosed and normatively justified. But even putting that kind of explicit normative change to the side, a number of confusing problems come up in deciding what a Restatement should be restating:

One issue is the denominator problem. Restatements are very long and have lots and lots of detail. So for any given rule, it’s possible that most states simply haven’t clearly stated what their rule is on that specific topic. If so, the “majority rule” might actually be only a handful of cases. But those cases may or may not be representative of the logic or likely outcomes in most states.

A related issue is the combination problem. Suppose a majority of jurisdictions follow rule X. And a majority of jurisdictions follow rule Y. That doesn’t mean that a majority of jurisdictions follow rule X+Y. Indeed, depending on the denominator problem, it’s possible that none do. And of course this issue scales up across the whole Restatement.

A final issue is the relevance of statutes. Restatements generally focus on common-law topics, but sometimes the state legislatures have adopted statutes changing the common-law rule. Should one simply restate the common-law rule that would have applied in the absence of the statute, or use only the non-statutory states as the denominator? Or should one try to Restate a new common law rule that matches the prevalent statutory rule? My instinct is that the former is better, because it treats the decision whether to pass a statute as meaningful, but I think the latter is common.

Even the purely descriptive parts of a Restatement project can subtly transform the common law.

The Market for FedSoc

Will’s post arguing that the Federalist Society is a network reminds me of a long running debate about why there isn’t a liberal network of lawyers of similar stature. Sure, most law schools have American Constitution Society chapters; but it’s common to hear liberal students lament that joining the ACS isn’t even a path to getting a good fed courts outline, that alone a good clerkship. FedSoc, on the other hand, has always had the well-earned reputation of being extremely effective at opening doors for its members. (At least, that’s the impression from the outside—I’m not a member.)

What explains the difference?

In an article published in Politico earlier this year, Evan Mandery presented the standard explanation of why the FedSoc is more influential than the ACS. Mandery explains that the FedSoc has three advantages over the ACS: (1) it is older, (2) advances an agenda more appealing to rich donors, and (3) has a unifying ideological commitment (originalism) that brings conservatives together.

These arguments all miss the mark. Simply put, there is a market for FedSoc; there isn’t a market for a liberal equivalent.

The reason for this discrepancy is that the legal profession is overwhelmingly liberal. In The Political Ideologies of American Lawyers, my collaborators and I find that over 60 percent of lawyers are liberal. And our subsequent research on law clerks and law professors suggests that more like 75 to 85 percent of elite lawyers are liberal.

The result of this ideological skew in the profession is that anyone hiring lawyers without respect to ideology is going to hire liberals most of the time. This means that people trying to hire for ideological reasons are going to make a lot of “mistakes” if they hire without knowing the candidates ideology. Additionally, the people that are trying to hire for ideological reasons—e.g., the people in charge of selecting political appointees or picking new judges—know perfectly well that lawyers are smart enough to be coy about their ideological commitments when good opportunities depend on coming off as conservative for a short period.

So how can people trying to hire conservatives make sure they don’t accidentally give good jobs to liberal lawyers? Make conservative lawyers send a costly signal that demonstrates their ideological bonafides. Joining FedSoc and attending talks, debates, and social events for years is that costly signal. There was a demand for providing that signal, and the FedSoc met it.

Liberal decision-makers don’t need to rely on the same kind of costly signals when hiring; they can just assume lawyers are liberal and be right most of the time. And they especially don’t need to bother with costly signals once conservatives are taking on the heavy lifting of doing the ideological sorting. Want to hire a liberal law student as your law clerk? Just make sure you don’t hire someone in FedSoc.

So any explanation for why ACS is less influential than FedSoc that focuses on why it is tough for liberals to organize misses the point. The difference in influence isn’t because liberals can’t get their act together; it exists because they don’t need to.

FedSoc is a They, not an It

The Federalist Society has been in the news (and in my Twitter feed) a lot lately, as people criticize both things that happened at a national convention last week, and things that have been said and done by a couple of its officials, especially Leonard Leo and Steven Calabresi.

This has led to claims that the Society is in fact a partisan organization because of its supposed role in picking judges, to calls that the organization disavow or denounce various things, and to arguments that members of the society have some moral culpability for what other members of the society do.

I am a member of the Federalist Society, but I don’t see things this way and thought I’d try to explain why. As I see it, the Federalist Society is essentially a network that connects thousands of scholars, students, and lawyers. There is obviously some intellectual valence to that network — it is not a random network — but it’s usually a mistake to discuss the network as a collective noun.

Thus, I don’t think it’s right to say that the Federalist Society picks judges. Some judges have been members of the Federalist Society, and so have some people who participated in the selection process. And sharing a network may well make some of those judges more likely to be picked by others in the network. (This is not going to happen to me, to be clear.) But the society doesn’t do anything. Individuals like Leonard Leo and Don McGahn do.

Similarly, I think it’s a mistake to expect the Federalist Society to take official positions beyond, perhaps, its relatively open-ended mission statement. Because the Society is not a legislative, adjudicative, or deliberative body it doesn’t really have a mechanism for taking positions. The positions are held by members of the network. And for the same reason, the fact that one member of the Society, or even an official of the society, has taken a position doesn’t attribute it to the others or to the group.

Finally there is the question of collective responsibility. Unlike the previous two points, I don’t think we can dismiss that out of hand. Maybe there is some kind of collective responsibility to abandon a network or group if you disagree with enough people in the group over enough sufficiently profound issues. Or maybe there is at least a duty to publicly comment on the behavior of other members of the group. But I find thinking of the group as a network helpful in framing these questions. By being part of a network the main thing one is offering is not political power or official endorsement, but one’s own willingness to freely associate.