If a tree falls in the forest, and no one hears it, does it matter if the peer review was open or closed?
On Media Commons there is a draft of a white paper on Open Review, written by Kathleen Fitzpatrick and Avi Santo in collaboration with a large advisory team. The authorial group is seeking commentary, so I hope you will give them some. I read this paper as a strong rhetorical effort directed at skeptics of the open review process. In other words, I'm not the audience here as I am not opposed to open review. I invite your review here all the time. This rhetorical disconnect though is a little strange for me in this white paper. The underlying purpose of white paper, as I read it, is to assuage fears. For example, they write "Creating a successful open review process begins with clearly establishing roles and expectations for participants, not for technologies, which must always take a backseat to human systems." There is extensive discussion of hierarchies in the review process and maintaining the rigor/quality of the scholarship. I support the exploration of open review, for reasons I will detail below, but here is my primary comment on this white paper:
What is the problem with existing scholarly review procedures that the open review process seeks to solve?I
It's a tricky rhetorical position. If one says that moving to open review is not a big deal, then why do it? If, as this white paper argues, a well-designed open review process can give us results that are equivalent to existing review processes, then why go to all the trouble to get the same results? On the other hand, if open review gives us different results, then why are those results more desirable?
If you are uncertain what is wrong with the current, closed system of peer review for humanities journals then you are not alone. To get an understanding of the problem, you'd first have to open up one of those journals and start reading the articles, and not a lot of people do that. What's wrong with those articles? As the white paper notes, "peer review in the humanities often focuses on originality, creativity, depth of argument, and the ability to communicate connections." We might also mention "rigor." I do not mean to suggest that humanities articles are not rigorous. I know that my colleagues work very hard on their scholarship and that they approach their work with passion and integrity. I am also sure that articles, generally speaking, meet some standard for originality, creativity, etc., though obviously these are all relative standards. My point is that even if we accept the premise that peer review reasonably achieves its stated goals, it has a deeper problem.
The humanities publish work of little interest.
I'm not saying there's no interest in it, just not enough to sustain the existing economic models. One of the key points that the white paper raises is the issue of the labor involved in peer review. And it's not only the reviewers but the editors and authors as well as the publishers. It is labor that is largely paid for by universities when they pay faculty salaries. But even when the faculty work at no cost to the publishers doing the writing, reviewing, and (sometimes) editing, the publishers still can't remain viable. Even when the publishers are university presses and are largely supported by the university (rather than depending only on sales), they are still failing. Maybe the problem is simply overproduction. We don't publish to communicate; we publish to get tenure and promotion. We follow our narrow scholarly interests with little thought for audience. In which case one could say that we simply need to be more rigorous. Instead of 100 articles on a given subject we should only publish 10 or 20 or whatever. But that's not the problem. In the end "rigor" or quality is just some arbitrary set of criteria. It's not a matter of finding the few, the proud; it's a matter of realizing that the whole system suffers from a common rhetorical problem. We can see this when the white paper suggests that "the test of open review’s 'rigor' comes from determining how closely community members, including editors, authors and reviewers, adhere to their publicly-stated parameters." But the problem isn't exactly standards. Even if open review as just a rigorous or even more rigorous than closed review, it would still result in the same publishing problem.
The value of open review will not lie in its ability to carry out the existing goals and practices of scholarly review but in its potential to serve as a lever in reinventing humanities scholarship as something more sustainable and, dare we imagine, even more interesting. Curiously, that potential appears in this white paper as a series of problems related to "etiquette:"
Beyond concerns about civility, open review raises unique challenges for community etiquette when it comes to interactions between authors and reviewers, as well as amongst reviewers. In trying to achieve a mutuality of participation, an open review community may wish to determine: (a) whether authors and reviewers should be allowed to interact either directly or indirectly, (b) whether interaction is required or merely encouraged, (c) what response time for interactions is appropriate, (d) whether all reviewer feedback must be acknowledged, and if so, in what manner; or if not, how choices should be made about which comments must be addressed, (e) how disagreements amongst reviewers should be resolved in the revision process, and (f) when scholarly conversation has strayed too far off topic to be considered part of the review process.
For most humanities scholars (and when I say most, I mean 99%+), review feedback is the most substantive (and often only) conversation they encounter regarding their work. We know something like 95% of humanities articles go uncited. Even when an article is cited, there's no assurance that the citation represents a substantive engagement with one's text. So there is rarely much intertextual conversation that would be akin to editorial feedback. Be honest: have you ever published an article that received the degree of attention I've given this white paper here?
What we need to recognize is that open review isn't the process that precedes publication. It is publication. You write something. You publish it online. I read it, and then I tell you what I think about it. Maybe I tweet others the link. Maybe I just comment on your text. Maybe I write something myself (like this). If no one comes to read it, well then, you figure out what that means… Perhaps the issues raised in the white paper are "challenges," but they are not unique to review. They are the rhetorical challenges inherent in communication. Essentially what they say is that when you write and publish something online in this open environment, you have few assurances about how your audience will respond.
On the other hand, you have plenty of assurances with print publication. If you publish in a print journal you are virtually guaranteed that your audience (if there is one) will not respond at all.
Interestingly, it is quite possible that humanities scholars prefer it that way. They don't want responses and they don't (really) want to communicate (not if it means taking the risks inherent in communication). As the white paper astutely observes:
The “openness” of open review can also prove daunting when it comes to evaluating scholarship for the purposes of tenure and promotion, where even the slightest hint of criticism can sink a scholar’s chances. While unrelated to the actual practices of open review, the inflation in standards for evaluating scholars (it is a cliché that almost every tenure review letter ranks candidates as amongst the top 5% of scholars in their field — and that any deviation from this will be regarded as a “red flag” by review committees) can have a chilling effect on participation in a process that makes critique and disagreement visible.
To which, the only rational response is to say, are you serious?!? I don't disagree that this is an accurate portrayal of tenure review, but this is simply nuts. Do we not know each other at all? I am fairly certain that if my colleagues aren't disagreeing with me then I am probably just saying something they don't care about. As I've said here a dozen times, critique is interminable. Everything can always be critiqued. People devote large portions of their lives to critiquing magazine ads for soap. If they can't be bothered to offer some off-hand critique of something you wrote then they are treating it with some serious disdain.
Perhaps we want to hold on to closed review and print publication because it allows us to hold on to the illusion that we are "amongst the top 5% of scholars" and that our work actually matters because the editor told us so and it appeared in print.
Open review forces us to look under that rock. And that's why we need it.
Alex Reid 