Longer version:
Harnad,  S. (2000) The Invisible Hand of Peer Review, Exploit Interactive, issue 5, April 2000
<http://www.exploit-lib.org/>:
http://www.cogsci.soton.ac.uk/~harnad/nature2.html
http://www.princeton.edu/~harnad/nature2.html

ABSTRACT: The refereed journal literature needs to be freed from both paper and its costs, but not from peer review, whose "invisible hand" is what maintains its quality. The residual cost of online-only peer review is low enough to be recovered from author-institution-end page charges, covered from institutional subscription savings, thereby vouchsafing a toll-free refereed research literature for everyone, everywhere, forever.

So it is in civic matters, and it is no different in the world of Learned Inquiry. The "quis custodiet" problem among scholars has traditionally been solved by means of a quality-control and certification [QC/C] system called "peer review" (Harnad 1985): The work of specialists is submitted to a qualified adjudicator, an editor, who in turn sends it to fellow-specialists, referees, to seek their advice about whether the paper is potentially publishable, and if so, what further work is required to make it acceptable. The paper is not published until and unless the requisite revision can be and is done to the satisfaction of the editor and referees.
 

Pitfalls of peer policing

Neither the editor nor the referees is infallible. Editors can err in the choice of specialists (indeed, it is well-known among editors that a deliberate bad choice of referees can always ensure that a paper is either accepted or rejected, as preferred); or editors can misinterpret or misapply referees' advice. The referees themselves can fail to be sufficiently expert, informed, conscientious or fair.

Nor are authors always conscientious in accepting the dictates of peer review. (It is likewise well-known among editors that virtually every paper is eventually published, somewhere (Lock 1985; Harnad 1986): There is a quality hierarchy among journals, based on the rigour of their peer review, all the way down to an unrefereed vanity press at the bottom. Persistent authors can work their way down until their paper finds its own level, not without considerable wasting of time and resources along the way, including the editorial office budgets of the journals and the freely given time of the referees, who might find themselves called upon more than once to review the same paper, sometimes unchanged, for several different journals.)

The system is not perfect, but it is what has vouchsafed us our refereed journal literature to date, such as it is, and so far no one has demonstrated any viable alternative to having experts judge the work of their peers, let alone one that is at least as effective in maintaining the quality of the literature as the present imperfect one is (Harnad 1982).
 

Self Policing?

There is a way to test our intuitions about the merits of this sort of proposal a priori, using a specialist domain that is somewhat more urgent and immediate than abstract "learned inquiry"; if we are not prepared to generalise this intuitive test's verdict to scholarly/scientific research in general, we really need to ask ourselves how seriously we take the acquisition of knowledge: If someone near and dear to you were ill with a serious but potentially treatable disease, would you prefer to have them treated on the basis of the refereed medical literature or on the basis of an unfiltered free-for-all where the distinction between reliable expertise and ignorance, incompetence or charlatanism is left entirely to the reader, on a paper by paper basis?

A variant on this scenario is currently being tested by the British Medical Journal <http://www.bmj.com/cgi/shtml/misc/peer/index.shtml>, but instead of entrusting entirely to the reader the quality control function performed by the referee in classical peer review, this variant, taking a cue from some of the developments and goings-on on both the Internet and Network TV chat-shows, plans to publicly post submitted papers unrefereed on the Web and to invite any reader to submit a commentary; these commentaries will then be used in lieu of referee reports as a basis for deciding on formal publication.
 

Expert Opinion or Opinion Poll?

Is this peer review? Well, it is not clear whether the self-appointed commentators will be qualified specialists (or how that is to be ascertained). The expert population in any given speciality is a scarce resource, already overharvested by classical peer review, so one wonders who would have the time or inclination to add journeyman commentary services to this load on their own initiative, particularly once it is no longer a rare novelty, and the entire raw, unpoliced literature is routinely appearing in this form first. Are those who have nothing more pressing to do with their time than this really the ones we want to trust to perform such a critical QC/C function for us all?

And is the remedy for the possibility of bias or incompetence in referee-selection on the part of editors really to throw selectivity to the winds, and let referees pick themselves? Considering all that hangs on being published in refereed journals, it does not take much imagination to think of ways authors could manipulate such a public-polling system to their own advantage, human nature being what it is.
 

Peer Commentary vs. Peer Review

And is peer commentary (even if we can settle the vexed "peer" question) really peer review? Will I say publicly about someone who might be refereeing my next grant application or tenure review what I really think are the flaws of his latest raw manuscript? (Should we then be publishing our names alongside our votes in civic elections too, without fear or favour?) Will I put into a public commentary -- alongside who knows how many other such commentaries, to be put to who knows what use by who knows whom -- the time and effort that I would put into a referee report for an editor I know to be turning specifically to me and a few other specialists for our expertise on a specific paper?

If there is anyone on this planet who is in a position to attest to the functional difference between peer review and peer commentary (Harnad 1982, 1984), it is surely the author of the present article, who has been umpiring a peer-reviewed paper journal of Open Peer Commentary (Behavioral and Brain Sciences [BBS] <http://www.princeton.edu/~harnad/bbs.html>, published by Cambridge University Press) for over 2 decades (Harnad 1979), as well as a brave new online-only journal of Open Peer Commentary, likewise peer-reviewed (Psycoloquy, sponsored by the American Psychological Association, <http://www.princeton.edu/~harnad/psyc.html>), which entered its second decade with the millennium.

Both journals are rigorously refereed; only those papers that have successfully passed through the peer review filter go on to run the gauntlet of open peer commentary, an extremely powerful and important supplement to peer review, but certainly no substitute for it. Indeed, no one but the editor sees [or should have to see] the population of raw, unrefereed submissions, consisting of some manuscripts that are eventually destined to be revised and accepted after peer review, but also (with a journal like BBS, having a 75% rejection rate) many manuscripts not destined to appear in that particular journal at all. Referee reports, some written for my eyes only, all written for at most the author and fellow referees, are nothing like public commentaries for the eyes of the entire learned community, and vice versa. Nor do 75% of the submissions justify soliciting public commentary, or at least not commentary at the BBS level of the hierarchy.

It has been suggested that in fields such as Physics, where the rejection rate is lower (perhaps in part because the authors are more disciplined and realistic in their initial choice of target journal, rather than trying their luck from the top down), the difference between the unrefereed preprint literature and the refereed reprint literature may not be that great; hence one is fairly safe using the unrefereed drafts, and perhaps the refereeing could be jettisoned altogether.
 

Successful Test-Site in Los Alamos

Is this really evidence that peer review is not indispensable after all? Hardly, for the "Invisible Hand" of peer review is still there, exerting its civilising influence: Just about every paper deposited in Los Alamos is also destined for a peer reviewed journal; the author knows it will be answerable to the editors and referees. That certainly constrains how it is written in the first place. Remove that invisible constraint -- let the authors be answerable to no one but the general users of the Archive (or even its self-appointed "commentators") -- and watch human nature take its natural course, standards eroding as the Archive devolves toward the canonical state of unconstrained postings: the free-for-all chat-groups of Usenet <http://tile.net/news/listed.html>, that Global Graffiti Board for Trivial Pursuit -- until someone re-invents peer review and quality control.
 

A subversive proposal

Now it is no secret that I am a strong advocate of a free literature along the lines of Los Alamos (Okerson & O'Donnell 1995). How are we to reconcile the conservative things I've said about QC/C here with the radical things I've advocated elsewhere about public author archiving (Harnad 1998a, 1998b, 1999)?

The answer is very simple. The current cost of the refereed paper journal literature is paid for by Subscription, Site License and Pay-Per-View (S/L/P). Both the medium (paper) and the method of cost-recovery (S/L/P) share the feature that they block access to the refereed literature, whereas the authors, who contribute their papers for free, would infinitely prefer free, universal access to their work.

The optimal (and inevitable) solution is an online-only refereed journal literature, which will be much less costly (less than 1/3 of the current price per page) once it is paper-free  <http://amsci-forum.amsci.org/archives/september98-forum.html> and resides in open archives
<http://www.openarchives.org>but still not entirely cost-free, because the peer review (and editing) still needs to be paid for (Odlyzko 1998). If those residual QC/C costs are paid at the author-institution-end (not out of the author's pocket, of course, but out of instiutional publication funds redirected from 1/3 of the 3/3 annual institutional savings from serial S/L/P cancellations), the dividend will be that the papers are all accessible for free for all (via interoperable open archives such as CogPrints <http://cogprints.soton.ac.uk > --  integrated seamlessly into a single global "virtual" archive, mirrored worldwide, which will then have an unrefereed preprint sector and a refereed, published, reprint sector, tagged by journal name). Journal publishers will continue to provide and be paid for their QC/C while the public archive will serve as the "front end" for both journal submissions (tagged "unrefereed prepints") and published articles (tagged "refereed reprints [plus journal name, etc.]"<http://www.eprints.org/index.html>.
 

Streamlining peer review for the airwaves

To distribute the load among referees more equitably (and perhaps also to protect editors from themeselves), the journal editor can formally approach a much larger population of selected, qualified experts about relevant papers they are invited to referee if they have the time and inclination. Referee reports can be emailed or deposited directly through a password-controlled Web interface. Accepted final drafts can be edited and marked up online, and the final draft can then be deposited in the public Archive for all, superseding the preprint.
 

Galactic hitch-hiking, PostGutenberg

Referee reports can be revised, published and linked to the published article as commentaries if the referee wishes; so can author rebuttals. And further commentaries, both refereed and unrefereed, can be archived and linked to the published article, along with author responses. Nor is there any reason to rule out postpublication author updates and revisions of the original article -- 2nd and 3rd editions, both unrefereed and refereed. Learned Inquiry, as I have had occasion to write before (Harnad 1990) is a continuum; reports of its findings -- informal and formal, unrefereed and refereed -- are milestones, not gravestones; as such, they need only be reliably sign-posted. The discerning hitch-hiker in the PostGutenberg Galaxy can take care of the rest (Harnad 1991).

Overall, the dissemination of learned research, once we have attained the optimal and inevitable state described here, will be substantially accelerated, universally accessible, and incomparably more interactive in the age of Scholarly Skywriting than it was in our own pedestrian, papyrocentric one; Learned Inquiry itself -- and hence all of society  --will be the chief beneficiary.
 
 

Ginsparg, P. (1996) Winners and Losers in the Global research Village. Invited contribution, UNESCO Conference HQ, Paris, 19-23 Feb 1996.
http://xxx.lanl.gov/blurb/pg96unesco.html

Ginsparg, P. (1994) First Steps Towards Electronic Research Communication. Computers in Physics. (August, American Institute of Physics). 8(4): 390-396.
http://xxx.lanl.gov/blurb/

Harnad, S. (1979) Creative disagreement. The Sciences 19: 18 - 20.

Harnad, S. (ed.) (1982) Peer commentary on peer review: A case study in scientific quality control, New York: Cambridge University Press.

Harnad, S. (1984) Commentaries, opinions and the growth of scientific knowledge. American Psychologist 39: 1497 - 1498.

Harnad, S. (1985) Rational disagreement in peer review. Science, Technology and Human Values 10: 55 - 62.

Harnad, S. (1986) Policing the Paper Chase. (Review of S. Lock, A difficult balance: Peer review in biomedical publication.) Nature 322: 24 - 5.

Harnad, S. (1990) Scholarly Skywriting and the Prepublication Continuum of Scientific Inquiry. Psychological Science 1: 342 - 343
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad90.skywriting.html
ftp://ftp.princeton.edu/pub/harnad/Harnad/HTML/harnad90.skywriting.html

Harnad, S. (1991) Post-Gutenberg Galaxy: The Fourth Revolution in the Means of Production of Knowledge. Public-Access Computer Systems Review 2 (1): 39 - 53
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad91.postgutenberg.html
ftp://ftp.princeton.edu/pub/harnad/Harnad/HTML/harnad91.postgutenberg.html

Harnad, S. (1996) Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals. In: Peek, R. & Newby, G. (Eds.) Scholarly Publishing: The Electronic Frontier. Cambridge MA: MIT Press. Pp. 103-118.
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html
ftp://ftp.princeton.edu/pub/harnad/Harnad/HTML/harnad96.peer.review.html

Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292. Short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared in the University of Toronto Bulletin: 51(6) P. 12. http://citd.scar.utoronto.ca/EPub/talks/Harnad_Snider.html

Harnad, S. (1998a) For Whom the Gate Tolls? Free the Online-Only Refereed Literature. American Scientist Forum. http://www.cogsci.soton.ac.uk/~harnad/amlet.html

Harnad, S. (1998b) On-Line Journals and Financial Fire-Walls. Nature 395(6698): 127-128. http://www.cogsci.soton.ac.uk/~harnad/nature.html

Harnad, S. (1999) Free at Last: The Future of Peer-Reviewed Journals. D-Lib Magazine 5(12) December 1999
http://www.dlib.org/dlib/december99/12harnad.html

Hitchcock, S. Carr, L., Jiao, Z., Bergmark, D., Hall, W., Lagoze, C. & Harnad, S. (2000) Developing services for open eprint archives: globalisation, integration and the impact of links. Proceedings of the 5th ACM Conference on Digital Libraries. San Antonio Texas June 2000.
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad00.acm.htm

Carr, L., Hitchcock, S., Hall, W. & Harnad, S. (2000) A usage based analysis of CoRR [A commentary on "CoRR: a Computing Research Repository" by Joseph Y. Halpern] ACM SIGDOC Journal of Computer Documentation. May 2000.
http://www.cogsci.soton.ac.uk/~harnad/Papers/Harnad/harnad00.halpern.htm

Lock, Stephen (1985) A difficult balance : editorial peer review in medicine London : Nuffield Provincial Hospitals Trust.

Odlyzko, A.M. (1998) The economics of electronic journals. In: Ekman R. and Quandt, R. (Eds) Technology and Scholarly Communication Univ. Calif. Press, 1998. http://www.research.att.com/~amo/doc/economics.journals.txt

Okerson A. & O'Donnell, J. (Eds.) (1995) Scholarly Journals at the Crossroads; A Subversive Proposal for Electronic Publishing. Washington, DC., Association of Research Libraries, June 1995. http://www.arl.org/scomm/subversive/index.html

Youngen G.K.  (1998) Citation patterns to traditional and electronic preprints in the published literature. College and Research Libraries 59(5): 448-456
http://www.physics.uiuc.edu/library/preprint.html