A few years ago, some collaborators and I wrote a paper trying to assess whether law clerks systematically influence votes at the Supreme Court. The specific research question was: do justices cast more liberal votes in years they have particularly liberal clerks and more conservative votes in years they have particularly conservative clerks?
The tricky part of answering this question is that Justices likely hire clerks that reflect their ideology at the time. For instance, if a justice is trending to the right over time, the ideology of the clerks they hire might trend to the right over time too. Fortunately for us though, the fact that Supreme Court clerks are hired during earlier terms makes it possible to account for this. I’ll spare you the equations, but we could control for the Justices’ ideology in the prior term as a way to account for whether Justices preferences for hiring particularly conservative or liberal clerks was changing.
Our key assumption was thus that Supreme Court Clerks are hired either during the term before they started or earlier. But it would be a problem for us if most clerks were hired and started working on cases immediately. How did we know that rarely happened? We just do.
Despite our assertion that we were sure that was the way that it worked, during the peer review process we ran into some referees that didn’t want to take our word for it. So we decided to email a random sample of 10 percent of Supreme Court clerks in our database and ask them when they were hired to clerk. Here’s the email we decided to send to the former clerks:
Dear [Former Clerk],
I am law professor at the University of Chicago Law School, and I am currently conducting research on the Supreme Court. As part of that research, I am trying to understand when Supreme Court clerks were hired for their clerkship. It is my understanding that you are a former Supreme Court clerk. I was hoping you would be willing to answer two short questions for me:
1. What is the month and year that you were offered your Supreme Court clerkship?
2. What is the month and year that you started your Supreme Court clerkship?
If you do not remember, any information would be helpful (as well as simply knowing that you do not remember). Your answers will be kept confidential. I am simply trying to document the average amount of time Supreme Court clerks are hired before they started, and will not in any way reveal personalized information.
Thank you for your time and help,
Adam
But sending out emails like this to get data for research is human subject research, and thus subject to IRB review. (Don’t know what an IRB is? Then this post isn’t for you.) So I submitted the survey to the University of Chicago’s IRB and explained that I wanted to ask 102 of the countries most sophisticated lawyers the date they were offered their clerkship just to confirm something that is widely known in legal circles.
In response, the IRB asked me to first send the former clerks a consent form to get their agreement to participate in our research. I went back and forth with the IRB explaining that requiring a consent form before soliciting answers would ensure that no one responded to my short email. Finally, we reached a compromise. I would include this in every email:
This research has been approved by the University of Chicago Institutional Review Board. If you have any questions about participating in this research, you can contact the Social & Behavioral Sciences Institutional Review Board at the University of Chicago, 1155 E. 60th Street, Room 418, Chicago, IL 60637. Phone: (773) 834-7835. Email: sbs-irb@uchicago.edu.
Although I’m complaining about it now, getting IRB approval for this project wasn’t a huge deal. But it did it involve filling our forms and several rounds of emails. And, more importantly, it held up the research a few weeks. And this story isn’t an outlier. Talk to any researcher at an American University that collects data in any way, and they’ll have plenty of IRB red tape horror stories.
That’s why I’m excited that my colleague Omri Ben-Shahar just wrote an essay to announce a pilot he is launching to rein in the IRB at the University of Chicago. The basic idea of the pilot is that the federal law exempts many kinds of research from IRB review, but right now IRB across the country have decided they should be the one to make the determination of whether a project falls into one of the exemptions. And when they make that determination, even if the project qualifies as exempt, they can ask for revisions. (That’s what happened in my story: even though the IRB agreed my harmless email to former clerks was exempt, they still had me add the IRB language to the email.)
In this pilot, researchers will get to make the determination of whether their research qualifies for one of the IRB exemptions for themselves. But that’s not all the pilot is doing: it’s randomizing which professors at the University of Chicago get to be part of the pilot. That way, in a few years, Omri and others can look back and see if the people in the control group (that remained subject to IRB review) committed any few instances of harmful research. In other words, the IRB reform is being done as a research project.
Like with our survey to clerks, I’m pretty sure I already know what Omri’s research will find: there will not be any differences in research misconduct between the professors in the pilot and professors subject to the normal IRB regime. But it’s great that someone is running this experiment to generate the kind of evidence we need to rein in the IRB for everyone.
----
Oh, and in case you are in suspense about the results of the survey: the people that responded confirmed they were hired a while before they showed up for their clerkship.
Note: I edited this post because I misspelled “rein” as “reign” and law twitter dragged me for it. On the plus side, I finally found a way to get people to engage with one of my blog posts.