Response to Joint Funding Bodies’ Review of Research Assessment

November 2002

This Association represents teachers, researchers, and students
in its fields within UK Higher Education. Its predecessor
body was instrumental in arguing for the formation of a panel
within the Research Assessment Exercise that could properly
recognise and assess research in our fields. This panel (65)
was created for the first time in the 1996 Exercise. Previously
the field had been inadequately assessed as a division of
work within a broader category of research including information
science and other, rather disparate, research areas.

Our response to the Research Assessment Exercise is guided
by the following principles:

  1. Any such exercise must be designed to secure improvement
    in the quality of research in universities. The costs,
    both human and material, of undertaking the Assessment,
    for both units of assessment and for those acting as
    assessors, must be less than the demonstrable benefits.
    If the primary purpose of the RAE is for resource allocation,
    then it has already gone further in concentrating resources
    than we believe to be healthy or productive, and in our
    judgement, further assessment leading to further concentration
    could only damage rather than enhance overall research
    quality.

  2. The purpose of the exercise is brought into disrepute
    when it does not fulfil the declared purpose of supporting
    excellence in research. The 2001 exercise clearly demonstrated
    improvements in the quality of research, with more work
    being judged to be at national and international level.
    In many instances, financial restrictions have meant
    that this improvement was not rewarded, and this must
    call into question the ability of staff to maintain and
    improve current levels.

  3. Many fields, ours included, do not always, or even predominantly,
    require concentrations of either facilities or of people
    to enable high quality research. Lone scholars, or small
    groups of researchers, frequently make significant contributions
    to research. Any Assessment Exercise must recognise this,
    while the funding consequences of assessment must ensure
    such activities are protected.

  4. Our field is a relatively new and certainly dynamic
    one. Assessment must offer maximum opportunity for the
    demonstration of research excellence in newer departments
    or groups, and must also reflect the frequent changes
    in the location and dispersion of research activity across
    the sector.

  5. We regard good teaching and opportunities for research
    to be inextricable. The QAA assessment of our field made
    extremely clear the benefits of teaching programmes delivered
    by staff with active research programmes. We would oppose
    any consequences of research assessment which led to
    damaging and unnecessary concentration of research, or
    of a drift to a division between research active and
    inactive centres.

  6. Our field, like several others, incorporates a wide
    range of approaches to “original investigation
    undertaken in order to gain knowledge and understanding”.
    We accept and endorse that definition of research. However
    we feel further discussion, across a number of subject
    areas, is required to move to a more widely accepted
    and understood interpretation of this definition as it
    applies to a diversity of work, including professional
    practice and non-traditional forms of output. The relationship
    between practice and research remains a problem to which
    different panels seemed to take different approaches.
    Further work needs to be done if a better understanding
    is to be reached about how departments and institutions
    which encourage a diversity of work are to be appropriately
    assessed either in the RAE or outside it.

With these principles in mind, the Association offers the
following comments on the Review paper.

  1. Of the four models for assessment offered we feel none
    alone adequately captures any acceptable definition of
    research quality. We do regard peer judgment as a sound
    basis for assessment. This depends on the methods employed
    being transparent and widely endorsed, and the experts
    exercising the judgements being both accepted as such
    by the constituency and adequately representative of
    the diversity of work within the field. Such judgements
    anchor the assessment in academic evaluation, a well
    established principle of work within the academy, and
    the least worst of the approaches suggested.

  2. We do not, however, regard peer judgement as sufficient.
    It will need to be buttressed by substantial elements
    of evidential support for statements about the context
    and culture within which research is conducted. For such
    evidence we would reject many metrics. For example, citation
    indices are a wholly inadequate, and largely misleading,
    indicator of research excellence in our field. Input
    factors such as external earnings can be an indirect
    indicator of peer judgement, but are less significant
    by far than the quality of outputs.

  3. We cannot envisage how a system based on self assessment
    could be effective. Either it would depend on the extensive
    use of auditable metrics, which we regard as wholly insufficient,
    or it would be based on the judicious use of rhetoric
    and promotion, which would simply require a further assessment
    exercise to adjudicate.

  4. It would be beneficial and equitable if assessment covered
    all funding council supported staff who are required
    to undertake research. To facilitate this strategy and
    to avoid penalising staff unfairly, careful consideration
    needs to be given to the criteria for defining both the
    nature of research and of the submissions expected. In
    this area, two specific problems need to be carefully
    addressed. The first relates to practice work as outlined
    in the principles above. The second problem relates to
    staff who during the period in question have been developing
    work rather than bringing it to fruition. At the moment
    they may either be pressured to produce work prematurely
    or not be entered. In this context, it may be helpful
    to strengthen those elements of the assessment which
    relate to the culture of research and the support for
    specific long term projects, however they are funded.

  5. The distribution of resources between subject areas
    cannot be derived from the RAE alone or even primarily.
    We have previously submitted to HEFCE evidence to suggest
    under-funding of our field because of the insufficient
    allocation to what is termed our ‘subject pot’.
    This was corrected in relation to teaching following
    work by the Media Studies Advisory Group, but has not
    been adjusted for research. The allocation across subject
    areas should be on the basis of the demonstrable average
    costs of research in given areas, which, while difficult
    to establish, is not impossible, and by the volume of
    research active staff.

Posted by Administrator