Responses to Questions For Communication, Cultural and Media Studies (UOA 65)

Question 1

How far do you agree that the descriptor provided by the sub-panel describes the main subject areas of the UOA?

(Note: following a consultation conducted by the HE funding bodies, the units of assessment for the 2008 RAE are fixed and are not subject to change – see RAE 02/2004 Panel configuration and recruitment and RAE 03/2004 Units of assessment and recruitment of panel members. Respondents should focus on whether the sub-panel’s description of the UOA and of its boundaries with other UOAs is comprehensive.)

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

Comments:

In general, MeCCSA welcomes the clearer and wider definition of research now adopted and the greater standardization of procedures and criteria compared to the last RAE.

MeCCSA is pleased to note the range of research forms and outputs to be accepted by the sub-panel and assessed against the indicators of excellence and degrees of quality described in paragraphs 15 - 22 of the main panel statement.

Top of page

Question 2

How far do you agree that the sub-panel’s proposed weightings for research outputs, research environment and research esteem are appropriate to the UOA?

(Note: in line with RAE team guidance, research outputs must be weighted at not less than 50% and esteem and environment must each be weighted at not less than 5% and the three must sum to 100%; main panels are encouraged to adopt consistent weightings for their sub-panels unless variation is justifiable.)

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

Comments:

The weighting between components has left the Research Environment at no more than 20%. This has raised concerns regarding the impact of early-career researchers. The RAE has declared a wish to minimize the benefits of tactical buying-in of big names, and to reward those institutions which work hard to bring on young researchers. However, at present, the only acknowledgement given to this is in the provision that early career researchers may not return four outputs. We would argue for the introduction of a final, ‘balancing’ consideration. Where a returning Unit is graded very high in its research environment (that is, in all those processes whereby it supports and develops those within its remit), but is returning over 50% of early career researchers, in the final stage of judging quality, Sub Panels should be invited to consider adding a weighting for emergent potential in awarding its final profile.

In addition MeCCSA would like more explanation of how the proposed 70% quality profile is to be broken down. How will the 5% bands be related to the overall starring system? Will the 70% itself be a conversion from a scaling system of 0-100?

Top of page

Question 3

How far do you agree that the sub-panel’s range of indicators for excellence is appropriate for the UOA in assessing the submitted research outputs?

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

Comments:

We would welcome a more detailed statement about how research outputs will be allocated to the star grades. This would be helpful in determining which items to select in each individual’s submission and also in deciding which researchers to include in the submission. For example:

  • Is it the case that the global reach or location of the publisher will be taken into account in determining the claims to international significance?

  • Will the evidence provided in the ‘esteem indicators’ be used to cross reference to the research outputs in order to assess the degree of recognition that a work has enjoyed, and therefore its quality? If so, how will this affect the evaluation of very recently published work whose impact is not yet measurable?

  • How will books that are designed to be used as student textbooks be regarded? In relation to this point we would want to draw the panel’s attention to a recent Publishers Association conference ‘Publishing for the RAE 2008 and beyond’ (University College London - Wednesday 29th June 2005) where this issue was discussed. John B Thompson, the Cambridge media sociologist, identified academic publishers increasing reluctance to (or even termination of) publishing scholarly monographs (because the market is too small) and keen interest in ‘textbooks’ to be at direct odds with the RAEs valuing of monographs and devaluing of ‘textbooks’. Thomson argued that the RAE had effectively stopped the writing of first rate, research informed, textbooks by the best UK academics. However, this was an area of publishing in which there had, in the past, been some very important historical achievements, especially in areas such as sociology, literary studies, and history and which had represented and articulated the relation of research and teaching that should be at the heart of Universities. We would argue that just because a book might be addressed to students it should not obscure the degree to which they may be works of research and original thought, particularly within new areas of enquiry. Cultural and Media Studies has an historical link, in its founding texts and arguments (Williams, Hoggart, Hall, Whanell, Thompson), with educational processes and a popular but non-reductive address to readerships. The panel for UOA 65 might wish to take a more explicitly inclusive view of the research content of ‘textbooks’ in the manner that the UOA 62 (History) panel has indicated. We quote from the History panel’s draft guidelines: ‘1. No form of output will be regarded as intrinsically inferior to any other. 2. Where there is a visible contribution to research, the following forms of scholarly outputs (listed alphabetically) will be evaluated: books, including textbooks which incorporate considerable personal research or substantially advance the subject area’

MeCCSA would also welcome detailed guidance on joint authored pieces. We note that History UoA (62) paragraph 19.p.4 states 'joint authored pieces will be treated as a single-authored piece…joint authored output may be listed by more than one individual in a department's submission'.

There is some concern at the way in which the RAE will deal with the question of über-texts, that is, submitted outputs which claim to be worth more than one. We understand the motivation behind this – that the RAE does not wish to disadvantage or discourage long-term and ambitious research plans. However, the way in which this is currently being proposed, will turn it into a high-risk strategy. As currently proposed, it will be in the hands of a returning University to claim that one output should count for two or more, but if their claim is not accepted, that person will be counted as having returned less than four, and thus penalized and awarded 0* for the missing item(s).

We would argue for a change to this. If the RAE wishes to sustain this principle, then it should be amended to remove this risk, so that an institution could submit four items, but make a claim that one of them should be considered for more than one. Should the claim be rejected by the Sub-Panel, the remaining items would still be available for consideration. Should it be accepted, one or more of the others could have been marked as the one(s) to be deleted from consideration.

MeCCSA notes that departments are not required to include all their eligible staff in their submission but there is not necessarily a financial incentive to exclude people. We would like this to be made more explicit and explained in more detail in the sub-panel statement.

More information is sought about how Category B staff will figure in the assessment of outputs and the scoring of profiles. The sub-panel 65 criteria refer to scrutiny of Category A and C staff outputs but there does not seem to be reference to Category B. This could affect departments where some staff who will have retired by the census date will be entered.

Top of page

Question 4

How far do you agree that the sub-panel’s range of indicators for excellence is appropriate for the UOA in assessing the research environment component of a submission?

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

In general, we would like clearer indications of how this is to be reported. There is a generally box-ticking approach taken here and we have no indication how the absence of information on particular points will be treated, should that information not be deemed to be relevant for a given department’s submission by those making the submission. We question the relevance of supplying information under:

14. Strategy c. iii Research grant applications etc, where it is asked for the numbers of grant application and the number of successful applications. We cannot see how asking for information on unsuccessful applications can do anything other than damage a department’s standing.

Paragraph 17 notes that ‘The assessment will be focused on outcome rather than income’. The paragraph then goes on to emphasise the importance of research income, and in particular, it says the sub-panel will ‘take account of the total number of grants relative to the size of the department’. We question the relevance of this crude numerical criterion, which presupposes that research income is necessarily relevant to a quality assessment. Furthermore, criteria that equate getting money from research councils with high level research are problematic. The end result is that certain kinds of research are rewarded twice further increasing funding differentials.

Strategy f.i. Other research activities, The Achievement of research staff etc. This would seem to be a redundant category as information will have been supplied on this elsewhere in the submission.

We ask that Paragraph 14(d)iv is redrafted to include NGOs and that Paragraphs 14(f) iv & v are redrafted to include organisations of civil society.

MeCCSA welcomes the undertaking in Paragraph 13 that the numbers of research-active staff submitted and their levels of experience will be taken account of. This may discourage the exclusion of individuals from departmental submissions which is potentially damaging for individual careers. Further advice on how this will be factored-in would offer more assurances and lead to less ambiguity and second-guessing.

Paragraph 15(c) notes that the sub-panel would 'allocate a further 25% on an holistic basis'. This is rather vague. MeCCSA would welcome some examples of indicators that will be employed.

Top of page

Question 5

How far do you agree that the sub-panel’s range of indicators for excellence is appropriate for the UOA in assessing evidence of esteem within a submission?

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

Comments:

Paragraph 17 notes that ‘The assessment will be focused on outcome rather than income’. The paragraph then goes on to emphasise the importance of research income, and in particular, it says the sub-panel will ‘take account of the total number of grants relative to the size of the department’. We question the relevance of this crude numerical criterion, which presupposes that research income is necessarily relevant to a quality assessment.

Paragraph 19c Benefit. We wish to point out that while this paragraph says ‘examples may include’, all those included are in fact valued in money terms. This encourages the impression that the criteria to be applied will be narrow.

Paragraph 19(b) Influence. We would like this to include executive and non-executive positions on boards of relevant organisations of civil society.

Top of page

Question 6

How far do you agree that the sub-panel has identified appropriate criteria for assessing the vitality and sustainability of the research described in each submission, including its criteria for assessing the contribution of researchers at different career stages?

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

Comments:

We would like to repeat the comments made in section 2 above. The weighting between components has left the Research Environment at no more than 20%. This has raised concerns regarding the impact of early-career researchers. The RAE has declared a wish to minimize the benefits of tactical buying-in of big names, and to reward thoseInstitutions which work hard to bring on young researchers. However, at present, the only acknowledgement given to this is in the provision that early career researchers may not return four outputs. We would argue for the introduction of a final, ‘balancing’ consideration. Where a returning Unit is graded very high in its research environment (that is, in all those processes whereby it supports and develops those within its remit), but is returning over 50% of early career researchers, in the final stage of judging quality, the Sub Panel should consider adding a weighting for emergent potential in awarding its final profile. Without provisions such as the above the RAE will continue to have a serious impact on the academic culture of young scholars.

Top of page

Question 7

If relevant in this UOA, how far do you agree that the sub-panel’s criteria and working methods are appropriate for identifying and assessing applied research equitably alongside other forms of research?

(Note paragraph 37b of RAE team’s Guidance to panels)

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

Comments:

In Paragraph 23. MeCCSA would like to include recognition of policy submissions to government departments, public bodies and NGOs as valid examples of applied research.

Top of page

Question 8

If relevant in this UOA, how far do you agree that the sub-panel’s criteria and working methods are appropriate for identifying and assessing practice-based or practice-led research equitably alongside other forms of research?

(Note paragraph 37b of RAE team’s Guidance to panels)

Strongly agree / Agree / Disagree / Strongly disagree / Don't know

Comments:

Practice as research has been covered by the generic heading of Applied Research and the same text appears in each sub-panel's document. We are concerned that the specificity of arts practice is not well represented and that calling everything an 'artefact' is too loose. Further clarity is required.

Top of page

Question 9

General comments on any other aspect of the sub-panel’s criteria and working methods. Where appropriate, respondents might wish to comment here, for example, on any non-standard data or data analyses that the sub-panel has requested:

Working methods

Paragraph 33b. We would like further clarification as to how research outputs will be selected and for reassurances that the process followed will not be to the potential detriment of researchers. Further clarification of how members of the panel assigned to the task of examining particular outputs will divide their labour would be of help. Will they read the same outputs or different ones?

MeCCSA notes that care has been taken over questions of conflict of interest. However, while this will work effectively at the point where assessments of individual institutions are taking place, it cannot work so effectively at the point when Sub-Panels, and indeed Panels, are discussing the relations between assessments – otherwise, at this point, meetings will be inevitably be stripped of most members. MeCCSA asks that the panel/sub-panels clarify how they plan to manage this final stage of the assessment process.

MeCCSA is also concerned about confidentiality. Although the star-rating of individuals is intended to be confidential it is difficult to foresee how this is possible in a field as small as media, communication and cultural studies. We would like further information on the processes of ensuring confidentiality of individual staff ratings.

Clarity is needed too concerning the role of the wider sub-panel in debating the assessments of the individual teams – where, for instance, sub-panel members might disagree with the assessment of particular outputs by those allocated to assess them.

General Comments

It is crucial that departments have the earliest indications possible of the length of RA5 and how this relates to vast array of information required. The likely effort of preparing RA5 is quite disproportionate to the final weight assigned to it in the overall quality assessment.

We also wish to underline the widely felt concern in the field that the RAE process is going to lead to a further concentration of research capacity, which we do not regard as the best way to ensure pluralistic and creative advances. The RAE has had a disastrous impact on the UK higher education system, leading to the closure of departments with strong research profiles and healthy student recruitment. The RAE has been responsible for job losses, discriminatory practices, widespread demoralization of staff, the narrowing of research opportunities through the over-concentration of funding and the undermining of the relationship between teaching and research. This is of particular importance for CCMS, given its still relatively early stage of development and the need to maintain as wide a research base as possible for the nurturing and growth of research talent.

Despite attempts to discourage games-playing the current exercise appears to be stimulating even more competitive recruitment and 'restructuring' driven purely by attempts, ill-fated or otherwise, to maximize RAE income. As in the past, the exercise will, without doubt, have a serious impact on departmental and institutional practices, research planning and collegiality, further distorting and disrupting the system and devaluing the professional contribution of many staff to teaching and research.

Finally, although MeCCSA appreciate that the decision on funding is entirely out-with the purview of the sub-panel we would like to register our continued frustration that departments will have no idea in advance of how assessment scores will translate into financial awards.