Anup Malani and I hosted a conference last week in New Delhi on finding ways to “Improve the Lives of India’s Urban Poor.” One of the presenters at the conference, Neelanjan Sircar, motivated his presentation with the above slide illustrating India’s disturbingly low female labor force participation (FLFP). As India has gotten richer, women have dropped out of the labor force at a rate higher than other South Asian countries. Neelanjan persuasively argued that understanding the drivers of this trend has massive welfare consequences, and given the size of India, is a question of first order importance.
Of course, a conference on urban poverty in India, FLFP isn’t the only problem of first order importance that came up. After all, India is facing dire consequences caused by climate change; there are frequent cases of violence against women and religious minorities; and there are still hundreds of millions of people in extreme poverty.
But here’s the most persuasive argument I heard last week about what problem should be the highest priority for researchers to tackle: making the academic publication process more efficient. Anup was the person that made this argument, and he was primarily talking about the peer review process in social science. Here’s the case he made.
Right now it frequently takes years for a paper to make it through the peer review process in political science, economics, or law & economics. Although this process sometimes makes the paper better, often it is just years of lateral moves. For instance, if you submit to Journal A, a reviewer may suggest to switch piece of evidence X with piece of evidence Y. If you’re rejected from Journal A, it’s reasonable to take a few months making the changes that the reviewers suggested before you submit to Journal B. After you get the referee reports back from Journal B months later, it’s all too common for a different reviewer to say it makes no sense to use piece of evidence Y instead of X. And then the process repeats.
The result is that authors spend years making tweaks on the same paper instead of starting new projects. If you believe that there is social value to the production of knowledge, this inefficient process is a massive social loss. But it may be the most important problem for academia to tackle, because whatever you think is the most important questions for researchers to study, they could be doing dramatically more of it if we could speed up the peer review process and allow them to move on to new projects. Think Neelanjan is right that FLFP in the developing world is a high priority problem to address? Fix the publication system so that researchers like Neelanjan can write twice as many papers instead of spending their time tweaking existing ones.
This brings me to the AALS proposals to reform the law review publication process that Will blogged about yesterday. The big advantage of the law review publication process is its speed. Established authors can submit their papers to dozens of law reviews in February and know with an extremely high degree of confidence that one of the journals will accept their paper by April. It would be a shame for any law review reform to give up this advantage over peer reviewed journals (although, making the process a month or two longer wouldn’t be a huge deal).
But the big drawback with the law review process is that the editors making publication decisions do not have any expertise in the subject matter of the articles specifically, or a good sense of what makes for a good article more generally. Peer reviewed journals typically don’t let second year graduate students review papers, and those students spend their time taking seminars where academic articles are read and debated. The view that a second year law student could assess the relative merits of papers after a year and a half of classes where they mostly read appellate cases is simply not credible.
The result is that law review placement is not a meaningful signal of article quality. It’s true that articles in the “Prestigious Law Journal” might be better on average than the “Low Ranked Journal”, but there is so much noise in the placement that it’s tough to look at a law professors list of publications and know much of anything other than how frequently they like to submit papers.
This creates its own kind of inefficiencies. It is most difficult for lower placed professors to successfully lateral when they do not have the advantage of their articles having a shot at top journals through blind peer review. The result is that the law professors that are best at producing research have difficultly moving to schools with better resources that would, in turn, allow them to produce more scholarship. Like the peer review process, the law review process is thus unnecessarily reducing the production of knowledge.
The only way to fix this problem is to introduce some form of peer review. Which is why it is nice to see the AALS proposals include a section on a possible peer review pool. Many of their ideas are sensible. But I would go further.
For one, I would require any author that submits an article to write referee reports for three other articles. Only after the author writes three reviews, and their paper has been reviewed three times, would the article be released to journals. Additionally, I would only allow law reviews access to the pool of peer reviews if they commit to not publishing articles that have not gone through the peer review process. The list of journals that have made that commitment would be posted online, and sticklers like me would know to not take any ones placements seriously if they publish in journals that have not made the commitment.
The AALS proposal had a versions of both these ideas, but it did not make them requirements or a cornerstone of their plan. These requirements, combined with some of proposed reforms (for instance, the requirement to take the first placement offered) could help to improve the signal in article placement without having law schools take on all of the problems of full peer review models.