Increasing the predictive power of the UK Research Assessment Exercise while making it cheaper and easier
Stevan Harnad, Les Carr, Tim Brody (Southampton University) & Charles Oppenheim (Loughborough University)
Being the only country with a national
research assessment exercise http://www.hero.ac.uk/rae/submissions/,
the UK is today in a unique position to make a very small procedural
that will generate some very large benefits. The Funding Councils
that in order to be eligible for Research Assessment
and funding, all UK
research-active university staff must maintain
(I) a standardised online
RAE-CV, including all designated RAE
performance indicators, chief among
them being (II) the full-text of
every refereed research paper, self-archived
in the university’s
online Eprint Archive and linked to the CV for online
digitometric analysis and assessment. This will (i) give the
Research Assessment Exercise (RAE) far richer, more sensitive and more
predictive measures of research productivity and impact, for far less
and effort (both to the RAE and to the universities preparing their
(ii) increase the uptake and impact of UK research
output, by increasing
its visibility, accessibility and usage, and (iii)
set an example for the
rest of the world that will almost certainly be
emulated, in both respects:
research assessment and research access.
Figure 1: Predicting RAE Ratings from Citation Impact (Smith & Eysenck 2002)
The UK already has a Research Assessment Exercise (RAE), every 4 years. The RAE costs a great deal of time and energy to prepare and assess, for both universities and assessors (time and energy that could be better used to actually do research, rather than preparing and assessing RAE returns).
For many areas of research, an important and predictive measure of research impact is the "Journal Impact Factor" (JIF) of the journal in which the article appears: the average number of citations its articles receive annually. (For core journals in all subject areas the JIF can be obtained from the Institute of Scientific Information's Journal Citation Reports service, for which the UK has a national site license: http://wos.mimas.ac.uk/jcrweb/.) The number of times a paper has been cited (hence used) is a measure of the importance and uptake of that research.
Figure 2: Predicting Citation Impact From Usage Impact (Physics ArXiv)
The JIF figures only indirectly in the RAE: Researchers currently have to submit 4 publications for the 4-year interval. It is no secret that departments (informally) weight candidate papers by their JIFs in deciding on what and whom to submit. Although it is always stressed by the RAE panels that they will not judge papers by the journals in which they appeared (but by the quality of their content), it would nevertheless be the strange RAE reviewer who was indifferent to the track-record, refereeing standards, and rejection-rate of the journal whose quality-standards a paper has met. (For books or other kinds of publications, see below; in general, peer-reviewed journal- or conference-papers are the coin of the research realm, especially in scientific disciplines.)
Statistical correlational analyses on the numerical outcome of the RAE using average citation frequencies predicts departmental outcome ratings remarkably closely. Smith & Eysenck (2002), for example, found a correlation of as high as .91 in Psychology (Figure 1). Oppenheim and collaborators (1995, 1998; Holmes & Oppenheim 2001) found correlations of .80 and higher in other disciplines.
The power of the indirect journal-based JIF has not yet been tested for predicting RAE rankings, but it is no doubt correlated with the well-demonstrated RAE predictive power of direct author-based citation counts (average or total). Journal-impact is the blunter instrument, author- or paper-impact the sharper one (Seglen 1992). But a natural conclusion is that the reliability and validity of RAE rankings can and should be maximized by adding and testing as many candidate predictors as possible within a weighted multiple regression equation.
Figure 3: New Online Performance Indicators
Nor is there any reason why the RAE should be done, at great effort and expense, every 4 years! Since the main determining factor in the RAE outcome ratings is research impact, there is no reason why research impact should not be continuously assessed, using not only author- and paper-citation counts and the JIF, but the many other measures derivable from such a rich research-performance-indicator database. There is now not only a method to assess UK research impact (i) continuously, (ii) far more cheaply and effortlessly for all involved, and (iii) far more sensitively and accurately (Figures 2-4), but doing the RAE this new way will also dramatically enhance UK research impact itself, (iv) increasing research visibility, usage, citation and productivity, simply by maximizing its accessibility (Figure 5-8).
Figure 4: Time-Course of Citations and Usage (Physics ArXiv)
The method in question is to implement the RAE henceforth online-only, with only two critical components: (a) a continuously updated and continuously accessible RAE-standardized online CV (containing all potential performance indicators: publications, grants, doctoral students, presentations, etc.) http://paracite.eprints.org/cgi-bin/rae_front.cgi for every researcher plus (b) a link from each CV to the full digital text of every published paper -- books discussed separately below – self-archived in that researcher's university Eprint Archive (an online archive of that institution's peer-reviewed research output). http://www.eprints.org/self-faq/#institution-facilitate-filling (See the free, open-source software developed at Southampton to allow universities to create their own institutional Eprint Archives: http://software.eprints.org/ )
Figure 5: Open Online Full-Text Access Enhances Citations by Dramatically (Computer Science)
Currently, university peer-reviewed research output -- funded by government research grants, the researcher's time paid for by the researcher's institution -- is given, free, by all researchers, to the peer-reviewed journals in which it appears. The peer-reviewed journals in turn perform the peer-review, which assesses,improves and then certifies the quality of the research (this is one of the indirect reasons that the RAE depends on peer-reviewed journal publications) (Figure 7). There is a hierarchy of peer-reviewed journals, from those with the highest quality standards (and hence usually the highest rejection rates and impact factors) at the top, grading all the way down to the lowest-quality journals at the bottom http://wos.mimas.ac.uk/jcrweb/. The peers review for free; they are just the researchers again, wearing other hats. But it costs the journals something to implement the peer reviewing: http://preprints.cern.ch/archive/electronic/other/agenda/a01193/a01193s5t11/transparencies/Doyle-peer-review.pdf.)
Figure 6: The Vast and Varied Influence of Research Impact
Partly because of the cost of peer review, but mostly because of the much larger cost of print-on-paper and its dissemination, plus online enhancements, journals charge tolls (subscriptions. licenses, pay-per-view) for access to researchers' papers (even though the researchers gave them the papers for free http://www.ecs.soton.ac.uk/~harnad/Tp/resolution.htm). The result is a great loss of potential research impact, because most institutions cannot afford to pay the access-tolls for most peer-reviewed journals (there are 20,000 in all, across disciplines), but only to a small and shrinking proportion of them http://www.arl.org/stats/index.html.
Hence the second dramatic effect of revising the RAE to turn it into online continuous assessment based on the institutional self-archiving of all UK peer-reviewed research output is that it will make all that UK research accessible to all would-be users worldwide whose access is currently blocked by access-toll-barriers (Figure 8). If RAE mandates self-archiving, university departments will mandate it too. Here, for example, is the draft Southampton self-archiving policy: http://www.ecs.soton.ac.uk/~lac/archpol.html
The UK full-text peer-reviewed research archives will not only be continuously accessible to all potential users, but the access will be continuously assessable, in the form not only of continuously updated impact estimates based on the classical measure of impact, which is citations, but usage will also be measured at earlier stages than citation, namely downloads ("hits," Figure 2) of both peer-reviewed “postprints” and pre-refereeing “preprints”. Many powerful new online measures of research productivity and impact will also develop around this rich UK research performance database (Figure 3,4), increasing the sensitivity and predictiveness of the RAE analyses more and more. (See the online impact-measuring scientometric search engines we have developed at Southampton: http://citebase.eprints.org/cgi-bin/search and http://opcit.eprints.org )
And all that is needed for this is for RAE to move to online submissions, mandating online CVs linked to the full-text draft of each peer-reviewed publication in the researcher's institutional Eprint Archive. http://www.eprints.org/self-faq/#research-funders-do Reference-link-based impact-assessment engines like citebase and Web of Science http://wos.mimas.ac.uk/ can then be used by RAE to derive ever richer and more accurate measures of research productivity and impact (Figure 3), available to the RAE continuously. Universities could continuously monitor and improve their own research productivity and impact, using those same measures. And the rest of the world could see and emulate the system, and its measurable effects on research visibility, uptake and impact.
Just a few loose ends: Books are usually not give-aways, as peer-reviewed research is, so full-text self-archiving is probably not viable for book output (apart from esoteric monographs that produce virtually no royalty revenue). But even if the book's full-text itself cannot be made accessible online, its metadata and references can be! Then the citation of books by the online peer-reviewed publications becomes a measurable and usable estimate of their impact too! For disciplines whose research and productivity does not consist of text but of other forms of digital output, both online usage counts and citations by text publications can still be used to estimate impact; and there are always the further kinds of performance indicators in the standardized RAE-CV that can be used to design discipline-specific metrics.
The UK is uniquely placed to move ahead with this and lead the world, because the RAE is already in place. The Netherlands has no formal RAE yet, but it is about to implement a national system of open research archiving for all of its universities called DARE: http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2356.html It is just a matter of time before they too realize that a marriage between a national network of DARE-style institutional Eprint Archives and CVs plus a national RAE-style research assessment exercise make a natural, perhaps even an optimal combination.
But although the naturalness and optimality -- indeed the inevitability -- of all this is quite transparent, it is a fact that research culture is slow to change of its own accord, even in what is in its own best interests. That, however, is precisely why we have funding councils and research assessment: To make sure that researchers do what is best for themselves, and best for research, and hence also best for the supporters (and beneficiaries) of research, namely, tax-paying society: The institutional self-archiving of research output, for the sake of maximizing research access and impact, has been much too slow in coming, even though it has already been within reach for several years. The UK and the RAE are now in a position to lead the world research community to the optimal and the inevitable. http://www.eprints.org/self-faq/#research-funders-do
We at Southampton and Loughborough, meanwhile, keep trying to do our bit to hasten the optimal/inevitable for research and researchers. At Loughborough we are clearing the way for universal self-archiving of university research output by sorting out the copyright issues (and non-issues http://www.lboro.ac.uk/departments/ls/disresearch/romeo/index.html). At Southampton we are planning to harvest all the metadata form the submissions to RAE 2001 http://www.hero.ac.uk/rae/submissions/ into RAEprints, a "meta-archive" that is intended to demonstrate what RAE returns would look like if this RAE upgrade proposal were adopted. Of course (i) RAEprints will contain only four papers per researcher, rather than their full peer-reviewed research output, (ii) it will only contain the metadata for those papers (author, title, journal-name), not the full-text and the all-important references cited. But we will also try to enhance the demo by adding as much of this missing data as we can – both from Journal Citation Reports http://wos.mimas.ac.uk/jcrweb/ and from the Web itself, to at least give a taste of the possibilities: Using paracite http://paracite.eprints.org/ an on-line citation-seeker that goes out and tries to find peer-reviewed full-text papers on the web, we will "stock" RAEprints with as much as we can find -- and then we will invite all the RAE 2001 researcher/authors to add their full-texts to RAEprints too!
We hope that the UK Funding Councils will put their full weight behind our recommended approach (Figure 9) when they publish their long-awaited review of the RAE process http://www.ra-review.ac.uk/
Figure 7: The Limited Impact Provided by Toll-Based Access Alone
Figure 8: Maximizing Research Impact Through Self-Archiving of University Research Output
Figure 9: What Needs to be Done to Fill the Eprint Archives
Berners-Lee, Tim. & Hendler, Jim. (2001). Scientific publishing on the "semantic web." Nature, 410, 1023-1024
Garfield, Eugene (1979). Citation indexing: Its theory and applications in science, technology and the humanities. New York. Wiley lnterscience
Harnad, S. (2001) The Self-Archiving Initiative. Nature 410: 1024-1025 http://www.nature.com/nature/debates/e-access/Articles/harnad.html
Harnad, Stevan, & Carr, Les (2000). Integrating, navigating, and analysing open eprint archives through open citation linking (the OpCit project). Current Science. 79(5). 629-638
Harnad, Stevan. (2001) Research access, impact and assessment. Times Higher Education Supplement 1487: p. 16. http://www.ecs.soton.ac.uk/~harnad/Tp/thes1.html
Harnad, S. (2003) Electronic Preprints and
of Library and Information
Marcel Dekker, Inc.
Harnad, S. (2003) Online Archives for Peer-Reviewed Journal Publications. International Encyclopedia of Library and Information Science. John Feather & Paul Sturges (eds). Routledge. http://www.ecs.soton.ac.uk/~harnad/Temp/archives.htm
Hodges, S., Hodges, B., Meadows, A.J., Beaulieu, M. and Law, D. (1996) The use of an algorithmic approach for the assessment of research quality, Scientometrics, 35, 3-13.
Holmes, Alison & Oppenheim, Charles (2001) Use of citation analysis to predict the outcome of the 2001 Research Assessment Exercise for Unit of Assessment (UoA) 61: Library and Information Management. http://www.shef.ac.uk/~is/publications/infres/paper103.html
Jaffe, Sam (2002) Citing UK Science Quality: The next Research Assessment Exercise will probably include citation analysis. The Scientist 16 (22), 54, Nov. 11, 2002. http://www.the-scientist.com/yr2002/nov/prof1_021111.html
Ingwersen, P., Larsen, B. and Wormell, I. (2000) Applying diachronic citation analysis to ongoing research program evaluations. In: The Web of Knowledge : a Festschrift in Honor of Eugene Garfield / Cronin, B. & Atkins, H. B. (eds.). Medford, NJ: Information Today Inc. & The American Society for Information Science.
Oppenheim, Charles (1995) The correlation between citation counts and the 1992 Research Assessment Exercises ratings for British library and information science departments, Journal of Documentation, 51:18-27.
Oppenheim, Charles (1996) Do citations count? Citation indexing and the research assessment exercise, Serials, 9:155-61, 1996.
Oppenheim, Charles (1998) The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology, Journal of Documentation, 53:477-87. http://dois.mimas.ac.uk/DoIS/data/Articles/julkokltny:1998:v:54:i:5:p:477-487.html
Seglen, P. O. (1992). The skewness of science. Journal of the American Society for Information Science, 43, 628-638 http://www3.interscience.wiley.com/cgi-bin/abstract/10049716/START
Smith, Andrew, & Eysenck, Michael (2002) “The correlation between RAE ratings and citation counts in psychology,” June 2002 http://psyserver.pc.rhbnc.ac.uk/citations.pdf.
Warner, Julian (2000) Research Assessment and Citation Analysis. The Scientist 14(21), 39, Oct. 30, 2000. http://www.the-scientist.com/yr2000/oct/opin_001030.html
Thelwall, Mike (2001) Extracting macroscopic information from Web links. Journal of the American Society for Information Science 52(13) (November 2001) http://portal.acm.org/citation.cfm?id=506355&coll=portal&dl=ACM
Zhu, J., Meadows, A.J. & Mason, G. (1991) Citations and departmental research ratings. Scientometrics, 21, 171-179
 Both usage statistics and citation statistics are open to potential abuse. See http://citebase.eprints.org/help/ - impactwarning and http://www.ecs.soton.ac.uk/~harnad/Hypermail/Amsci/2642.html The data nevertheless still have considerable signal value, its effect-size can be estimated, and ways of detecting and correcting for abuses will evolve as these new measures become more important.