0.31 European elephants in the room (are they the ones with the  (Page 3/9)

The arts and humanities have largely stayed on the margins of this debate.  In fact, when the United Kingdom decided to scrap its hugely influential and much maligned Research Assessment Exercise (RAE) in favor of a metrics-based system, a special expert group had to be convened to determine how the arts and humanities would be treated.  This group came up with a very long list of possible forms of research output that could be measured (although the report did not always define or specify how one might capture these measurements) and determined that a basket of these measures, including a “peer review lite” approach, could be chosen closer in time to the first round of the new assessment exercise in 2013.  This report was largely an admission of failure in the face of a task for which the system had no easy or accessible tools.

This is not to say that there have not been attempts to develop frameworks to enable a more systematic quality assessment of research outputs in the arts and humanities.  The European Science Foundation funded the development of a project called the European Reference Index for the Humanities.  The ERIH was intended to provide a relatively uncomplicated impact mark for the journal in which an article was published on a discipline-by-discipline basis.  In spite of the project leaders’ insistence that the A, B, and C rankings given to journals were intended to indicate their degree of internationalization and profile, not quality (whatever that is), the use of the simple letter grading became a source of great controversy and discontent with the project.  The real threat this index posed to smaller publishers and national journals, which were very appropriate places for certain kinds of research to appear, resulted in the project being largely abandoned after its initial development funding ran out.

But even if ERIH had been embraced by the community, what then?  It didn’t include the publishers, so had no capacity to aid in the evaluation of monographs, chapters, editions, or articles in edited collections.  It was a misleading, unauthoritative tool, and although the scholarly community reviled the project, there were still enough cases seen of research councils and institutions attempting to use it that these bodies’ desire for some easy way of making even an initial judgment of quality or impact was clearly perceived. 

This extended anecdote is intended to illustrate an environment that holds both opportunities and threats for the development of scholarly communications in Europe.  Institutional autonomy is low, and the pressure to contribute to the system in certain particular ways is very real.  “Is it RAE-able” was the question that always used to be asked about potential project outputs in the UK, and it is doubtful that the question will change much under the new system; only the terminology used to describe acceptability to the higher powers will shift.  On the other side, however, the existence of national systems with gatekeepers who are unsure how to evaluate scholarship most effectively also means that these gatekeepers may also be willing to work with their community to define reasonable metrics and approaches to assessment.  If the traditional analog scholarly communications formats are to be transcended, then success will be guaranteed only if this transcendence can be translated into terms comparable with those applied to the known scholarly output forms.  There have been calls for years for the creation of “kite markers” and other ways of recognizing the scholarly contribution made by non-traditional output forms.  Certainly in Europe, one has the sense that if the scholarly community took the lead in establishing norms and measures for best practice where these norms don't already exist, then their lead would be both appreciated and followed.