Question 1
How far do you agree that the descriptor provided
by the sub-panel describes the main subject areas of the
UOA?
(Note: following a consultation conducted by the
HE funding bodies, the units of assessment for the 2008
RAE are fixed and are not subject to change – see RAE
02/2004 Panel configuration and recruitment and RAE
03/2004 Units of assessment and recruitment of panel members.
Respondents should focus on whether the sub-panel’s
description of the UOA and of its boundaries with other
UOAs is comprehensive)
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Comments:
In general, MeCCSA welcomes the clearer and wider definition
of research now adopted and the greater standardization of
procedures and criteria compared to the last RAE.
We also welcome the inclusive definition of the Communication,
Cultural and Media Studies remit to include theoretical,
historical and empirical studies across our field.
MeCCSA is pleased to note the range of research forms and
outputs to be accepted by the sub-panel and assessed against
the indicators of excellence and degrees of quality described
in paragraphs 15 – 22 of the main panel statement.
However, MeCCSA members have noted a degree of confusion
in the following paragraph:
Communication, Cultural and Media Studies proposes to
adopt an inclusive definition of its remit, and will assess
research addressing or deploying theory, history, institutional,
policy, textual, critical and/or empirical analysis, or
practice within communication, culture, media, journalism
and film studies. Within UK higher education much, but
not all, of this work is likely to emanate from units or
departments in Communication Studies, Cultural Studies,
Media Studies, Journalism or Film and Television Studies.
This work will include research on print media, broadcasting,
the moving image, ‘new media’ including computer-mediated
communication, popular culture, information and communication
technologies, and journalism, which will be variably titled
and organized. Much will also be conducted in units or
departments situated elsewhere within the social sciences,
arts or humanities. The Sub-Panel will assess research
as defined above which will include (but is not confined
to):
- policy for regulation of culture and the media;
- the organization, institutions, political economy and
practice of cultural production;- media and cultural texts, forms and practices;
- media and cultural audiences, consumption and reception,
including questions of power, identity and difference.
This outline has two related lists, which are not consistent.
In sentence 1 ‘history’ appears, but it does not appear in
the bulleted list. However, the bulleted list does repeat
other topics in Sentence 1 affording them greater prominence.
We also note the absence of certain areas such as public
relations from the specific research areas mentioned. We
ask for further clarity in this paragraph and that either
all subject areas be given explicit recognition as contributing
to the communication, cultural and media studies field to
eliminate the possibility of any disadvantage or a broader
definition is incorporated (such as below) that does not
invite ambiguity with two inconsistent lists.
Clarity of this paragraph could be improved by a redrafting.
For example, sentence 1 could open with:
“Communication, Cultural and Media Studies proposes
to adopt an inclusive definition of its remit, and will assess
research which will include (but is not confined to): work
addressing or deploying theory; the history of the media
and cultural practices; media and cultural policy; media
and cultural texts, forms and practices; critical and/or
empirical analysis; the organization, institutions, political
economy and practice of cultural production within communication,
culture, media, journalism and film studies; creative and
reflective work in media practice and media and cultural
audiences, consumption and reception, including questions
of power, identity and difference”.
The paragraph could then continue as it is, ending at ‘…humanities’.
Question 2
How far do you agree that the sub-panel’s
proposed weightings for research outputs, research environment
and research esteem are appropriate to the UOA?
(Note: in line with RAE team guidance, research
outputs must be weighted at not less than 50% and esteem
and environment must each be weighted at not less than
5% and the three must sum to 100%; main panels are encouraged
to adopt consistent weightings for their sub-panels unless
variation is justifiable.)
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Comments:
The weighting between components has left the Research Environment
at no more than 20%. This has raised concerns regarding the
impact of early-career researchers. The RAE has declared
a wish to minimize the benefits of tactical buying-in of
big names, and to reward those institutions which work hard
to bring on young researchers. However, at present, the only
acknowledgement given to this is in the provision that early
career researchers may not return four outputs. We would
argue for the introduction of a final, ‘balancing’ consideration.
Where a returning Unit is graded very high in its research
environment (that is, in all those processes whereby it supports
and develops those within its remit), but is returning over
50% of early career researchers, in the final stage of judging
quality, the Sub Panel should consider adding a weighting
for emergent potential in awarding its final profile.
In addition MeCCSA would like more explanation of how the
proposed 70% quality profile is to be broken down. How will
the 5% bands be related to the overall starring system? Will
the 70% itself be a conversion from a scaling system of 0-100?
Question 3
How far do you agree that the sub-panel’s
range of indicators for excellence is appropriate for the
UOA in assessing the submitted research outputs?
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Comments:
We would welcome a more detailed statement about how research
outputs will be allocated to the star grades. This would
be helpful in determining which items to select in each individual’s
submission and also in deciding which researchers to include
in the submission. For example:
- Is it the case that the global reach or location of the
publisher will be taken into account in determining the
claims to international significance?
- Will the evidence provided in the ‘esteem indicators’ be
used to cross reference to the research outputs in order
to assess the degree of recognition that a work has enjoyed,
and therefore its quality? If so, how will this affect
the evaluation of very recently published work whose impact
is not yet measurable? - How will books that are designed to be used as student
textbooks be regarded? In relation to this point we would
want to draw the panel’s attention to a recent Publishers
Association conference ‘Publishing for the RAE 2008
and beyond’ (University College London – Wednesday
29th June 2005) where this issue was discussed. John B
Thompson, the Cambridge media sociologist, identified academic
publishers increasing reluctance to (or even termination
of) publishing scholarly monographs (because the market
is too small) and keen interest in ‘textbooks’ to
be at direct odds with the RAEs valuing of monographs and
devaluing of ‘textbooks’. Thomson argued that
the RAE had effectively stopped the writing of first rate,
research informed, textbooks by the best UK academics.
However, this was an area of publishing in which there
had, in the past, been some very important historical achievements,
especially in areas such as sociology, literary studies,
and history and which had represented and articulated the
relation of research and teaching that should be at the
heart of Universities. We would argue that just because
a book might be addressed to students it should not obscure
the degree to which they may be works of research
and original thought, particularly within new areas of
enquiry. Cultural and Media Studies has an historical link,
in its founding texts and arguments (Williams, Hoggart,
Hall, Whanell, Thompson), with educational processes and
a popular but non-reductive address to readerships. The
panel for UOA 66 might wish to take a more explicitly inclusive
view of the research content of ‘textbooks’ in
the manner that the UOA 62 (History) panel has indicated.
We quote from the History panel’s draft guidelines: ‘1.
No form of output will be regarded as intrinsically inferior
to any other. 2. Where there is a visible contribution
to research, the following forms of scholarly outputs (listed
alphabetically) will be evaluated: books, including textbooks
which incorporate considerable personal research or substantially
advance the subject area’
MeCCSA would also welcome detailed guidance on joint authored
pieces. We note that History UoA (62) paragraph 19.p.4 states ‘joint
authored pieces will be treated as a single-authored piece…joint
authored output may be listed by more than one individual
in a department’s submission’.
There is some concern at the way in which the RAE will deal
with the question of über-texts, that is, submitted
outputs which claim to be worth more than one. We understand
the motivation behind this – that the RAE does not
wish to disadvantage or discourage long-term and ambitious
research plans. However, the way in which this is currently
being proposed, will turn it into a high-risk strategy. As
currently proposed, it will be in the hands of a returning
University to claim that one output should count for two
or more, but if their claim is not accepted, that person
will be counted as having returned less than four, and thus
penalized and awarded 0* for the missing item(s).
We would argue for a change to this. If the RAE wishes to
sustain this principle, then it should be amended to remove
this risk, so that an institution could submit four items,
but make a claim that one of them should be considered for
more than one. Should the claim be rejected by the Sub-Panel,
the remaining items would still be available for consideration.
Should it be accepted, one or more of the others could have
been marked as the one(s) to be deleted from consideration.
MeCCSA notes that departments are not required to include
all their eligible staff in their submission but there is
not necessarily a financial incentive to exclude people.
We would like this to be made more explicit and explained
in more detail in the sub-panel statement.
More information is sought about how Category B staff will
figure in the assessment of outputs and the scoring of profiles.
The treatment of Category B staff could affect departments
where some staff who will have retired by the census date
will be entered.
We also require further clarification on the scoring system
for excellence in research outputs. The Panel 66 document
says they will examine in detail 50% or more of outputs (each
by 2+ readers). Each will each be assessed from 0-4 on the
3 indicators of excellence for outputs (significance, originality,
rigour). Individual outputs will thus be given a quality
level, i.e. an overall score of 0-4. But it doesn’t say whether
the 3 indicators will be equally weighted in arriving at
this.
Question 4
How far do you agree that the sub-panel’s
range of indicators for excellence is appropriate for the
UOA in assessing the research environment component of
a submission?
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Comments:
In general, we would like clearer indications of how this
is to be reported. There is a generally box-ticking approach
taken here and we have no indication how the absence of information
on particular points will be treated, should that information
not be deemed to be relevant for a given department’s
submission by those making the submission. We question the
relevance of supplying information under:
14. Strategy c. iii Research grant applications etc, where
it is asked for the numbers of grant application and the
number of successful applications. We cannot see
how asking for information on unsuccessful applications
can do anything other than damage a department’s
standing. Furthermore, criteria that equate getting money
from research councils with high level research are problematic.
The end result is that certain kinds of research are rewarded
twice further increasing funding differentials.Strategy f.i. Other research activities, The Achievement
of research staff etc. This would seem to be a redundant
category as information will have been supplied on this
elsewhere in the submission.
We ask that Paragraph 14(d)iv is redrafted to include NGOs
and that Paragraphs 14(f) iv and v are redrafted to include
organisations of civil society.
MeCCSA welcomes the undertaking in Paragraph 13 that the
numbers of research-active staff submitted and their levels
of experience will be taken account of. This may discourage
the exclusion of individuals from departmental submissions
which is potentially damaging for individual careers. Further
advice on how this will be factored-in would offer more assurances
and lead to less ambiguity and second-guessing.
Paragraph 15(c) notes that the sub-panel would ‘allocate
a further 25% on an holistic basis’. This is rather vague.
MeCCSA would welcome some examples of indicators that will
be employed.
Question 5
How far do you agree that the sub-panel’s
range of indicators for excellence is appropriate for the
UOA in assessing evidence of esteem within a submission?
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Comments:
Paragraph 17 notes that ‘The assessment will be focused
on outcome rather than income’. The paragraph then
goes on to emphasise the importance of research income, and
in particular, it says the sub-panel will ‘take account
of the total number of grants relative to the size of the
department’. We question the relevance of this crude
numerical criterion, which presupposes that research income
is necessarily relevant to a quality assessment.
Paragraph 19 (c) Benefit. We wish to point out that while
this paragraph says ‘examples may include’, all
those included are in fact valued in money terms. This encourages
the impression that the criteria to be applied will be narrow.
Paragraph 19 (b) Influence. We would like this to include
executive and non-executive positions on boards of relevant
organisations of civil society.
Question 6
How far do you agree that the sub-panel has identified
appropriate criteria for assessing the vitality and sustainability
of the research described in each submission, including
its criteria for assessing the contribution of researchers
at different career stages?
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Comments:
We would like to repeat the comments made in section 2 above.
The weighting between components has left the Research Environment
at no more than 20%. This has raised concerns regarding the
impact of early-career researchers. The RAE has declared
a wish to minimize the benefits of tactical buying-in of
big names, and to reward thoseinstitutions which work hard
to bring on young researchers. However, at present, the only
acknowledgement given to this is in the provision that early
career researchers may not return four outputs. We would
argue for the introduction of a final, ‘balancing’ consideration.
Where a returning Unit is graded very high in its research
environment (that is, in all those processes whereby it supports
and develops those within its remit), but is returning over
50% of early career researchers, in the final stage of judging
quality, the Sub Panels should consider adding a weighting
for emergent potential in awarding its final profile. Without
provisions such as the above the RAE will continue to have
a serious impact on the academic culture of young scholars.
Question 7
If relevant in this UOA, how far do you agree that
the sub-panel’s criteria and working methods are
appropriate for identifying and assessing applied research
equitably alongside other forms of research?
(Note paragraph
37b of RAE team’s Guidance to panels)
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Comments:
MeCCSA welcomes clarification of the role of ‘user experts’
on the panel.
In Paragraph 23. MeCCSA would like to include recognition
of policy submissions to government departments, public bodies
and NGOs as valid examples of applied research.
Question 8
If relevant in this UOA, how far do you agree that
the sub-panel’s criteria and working methods are
appropriate for identifying and assessing practice-based
or practice-led research equitably alongside other forms
of research?
(Note paragraph
37b of RAE team’s Guidance to panels)
Strongly agree / Agree / Disagree / Strongly
disagree / Don’t know
Question 9
General comments on any other aspect of the sub-panel’s
criteria and working methods. Where appropriate, respondents
might wish to comment here, for example, on any non-standard
data or data analyses that the sub-panel has requested:
Working methods:
Paragraph 33b. We note that the panel will examine a minimum
of 25% of the research outputs. We would ask for further
clarification as to how these outputs will be selected and
for reassurances that the process followed will not be to
the potential detriment of researchers. Further clarification
of how members of the panel assigned to the task of examining
particular outputs will divide their labour would be of help.
Will they read the same outputs or different ones?
MeCCSA notes that care has been taken over questions of
conflict of interest. However, while this will work effectively
at the point where assessments of individual institutions
are taking place, it cannot work so effectively at the point
when Sub-Panels, and indeed Panels, are discussing the relations
between assessments – otherwise, at this point, meetings
will be inevitably be stripped of most members. MeCCSA asks
that the panels/sub-panels clarify how they plan to manage
this final stage of the assessment process.
Clarity is also required concerning the role of the wider
sub-panel in debating the assessments of the individual teams – where,
for instance, sub-panel members might disagree with the assessment
of particular outputs by those allocated to assess them.
MeCCSA is also concerned about confidentiality. Although
the star-rating of individuals is intended to be confidential
it is difficult to foresee how this is possible in a field
as small as media, communication and cultural studies. We
would like further information on the processes of ensuring
confidentiality of individual staff ratings.
General Comments:
MeCCSA has some concerns over the way in which Sub-Panel
66 has set out its remit. This is done in a slightly confusing
way, since it seems to say most, but not all, things twice.
We welcome the evident will to be inclusive, but hope that
it might be tidied up a little.
It is crucial that departments have the earliest indications
possible of the length of RA5 and how this relates to vast
array of information required. The likely effort of preparing
RA5 is quite disproportionate to the final weight assigned
to it in the overall quality assessment.
We also wish to underline the widely felt concern in the
field that the RAE process is going to lead to a further
concentration of research capacity, which we do not regard
as the best way to ensure pluralistic and creative advances.
The RAE has had a disastrous impact on the UK higher education
system, leading to the closure of departments with strong
research profiles and healthy student recruitment. The RAE
has been responsible for job losses, discriminatory practices,
widespread demoralization of staff, the narrowing of research
opportunities through the over-concentration of funding and
the undermining of the relationship between teaching and
research. This is of particular importance for media, communication
and cultural studies, given its still relatively early stage
of development and the need to maintain as wide a research
base as possible for the nurturing and growth of research
talent.
Despite attempts to discourage games-playing the current
exercise appears to be stimulating even more competitive
recruitment and ‘restructuring’ driven purely by attempts,
ill-fated or otherwise, to maximize RAE income. As in the
past, the exercise will, without doubt, have a serious impact
on departmental and institutional practices, research planning
and collegiality, further distorting and disrupting the system
and devaluing the professional contribution of many staff
to teaching and research.
Finally, although MeCCSA appreciate that the decision on
funding is entirely out-with the purview of the sub-panel
we would like to register our continued frustration that
departments will have no idea in advance of how assessment
scores will translate into financial awards.