Response—The funding of research and research training in Australian universities

On 13 September 2002, the Australian Academy of Science made the following response to the Commonwealth Department of Education, Science and Training's issues paper on the funding of research and research training in Australian universities.

Introduction

1. The Australian Academy of Science has already provided comments on the initial Ministerial Discussion Paper, which reiterated the Academy’s stated policy positions in a number of relevant areas. This submission focuses in more detail upon the funding of Australian higher education research and related issues such as performance assessment.
2. The Academy is surprised at how little attention research and related issues received in the Discussion Papers and have had in the debate. Australian universities are a very significant component of the Australian innovation system. Annually they produce in excess of 3800 PhD graduates [1]. Higher education R&D (HERD) amounts to $2774.6 million or 27 per cent of all Australian R&D [2].
3. How this funding is allocated is as critical as the quantum allocated. Indeed, the Academy is skeptical that unless it is possible to demonstrate that the funding mechanisms for higher education research are such that, to the greatest possible extent, the most innovative research of the greatest value to Australia is supported, arguments for increased funding for research will be unsuccessful.
4. Substantial reforms to the funding of higher education research and research training were introduced following the Government’s 2000 White Paper – Knowledge and Innovation. These reforms did increase the extent to which research funding is allocated on performance and did, through the Backing Australia’s Ability initiatives, result in a substantial increase in research funding. Nevertheless, the Academy believes that these reforms did not go far enough and substantial deficiencies remain in the current funding mechanisms for higher education research and research training. These deficiencies detract from the ability of Australian universities to build and maintain internationally competitive research strengths in key areas of importance to Australia.
5. Perhaps even more seriously, these deficiencies detract from the ability of Australia to engage with and access the international R&D effort. Australia may carry out around 1 per cent of the world's R&D but that effort is vital if Australia is to access the other 99 per cent [3]. However that access does not come free. As a recent UK analysis [4] the economic benefits of publicly funded research concluded: 'No nation can "free-ride" on the world scientific system…A nation needs the capability to understand the knowledge produced by others and that understanding can only be developed through performing research.'
6. The main thrust in this submission is based upon the premise that the White Paper reforms have established a funding structure for higher education research and research training that can in principle deliver sustainable improvements in research capabilities, research training and research outcomes (including commercialisation). The policy action required is simply to modify some critical aspects of the existing framework relating to how performance is measured and how this assessment is used to influence funding. This submission focuses on these critical issues.

The Need for Further Reform

7. Currently Australian higher education research and research training is supported through a dual funding scheme made up of ‘direct funding’ from agencies, determined on the basis of competitive peer review (according to the agencies' objectives and criteria) and a block institutional grant that is performance based. The Academy believes that this is, in principle, the most effective approach to funding higher education research in Australia. However as currently implemented the dual funding scheme suffers from a serious and critical deficiency – an inadequate focus on quality and outcomes in the allocation of the block grant.
8. Last October the Academy published a report [5] which showed that Australia’s share of scientific publications had increased markedly over the 1990s but the relative impact of Australia’s publications, as measured by citations, had declined and continues to fall behind most other OECD countries. Even more disquieting are the findings reported by Linda Butler in her submission [6] to this review that this increase in university output has occurred disproportionably in journals of lower impact. Significantly a similar effect is not seen in the output of other research agencies such as CSIRO. Butler hypothesises that these results are consequences of a funding algorithm, introduced originally in the early 1990s and maintained in the post-White Paper reforms, that involves a volume-based publication measure that does not discriminate on any quality measures. In contrast to Australia the citation impact of UK science has increased over recent years [7], plausibly because excellence is the major factor that drives the allocation of block research grants to UK universities.
9. The Academy concurs with Butler. Given the close link that Narin [8] has demonstrated between high quality science, as measured by citations, and patenting and thus innovation and commercialisation, the Academy believes that it is vitally important that this unfortunate trend is addressed urgently. As Narin concluded: 'There is no field in which mediocre research stands much chance of contributing to innovation.'
10. The discredited volume measure should be dispensed with immediately and consideration given to the design of a block funding scheme that encourages and rewards outcomes and impact. Designing such a scheme would take some time – some possible ideas are discussed below. Until that is possible it is the Academy’s view that it would be far more preferable for the current block grant mechanisms be used without the publication component.

The ARC Submission

11. The Australian Research Council in its submission to this review [9] has also recognised this weakness in current arrangements but has suggested a far more radical reform. The crux of their approach is for the ARC to be resourced – possibly by clawing funding off the existing university operating grants – to fund the full cost of the projects that they fund, including the research component of the salary of chief investigators who are university academic staff members. In essence, Australia moves to a US-style funding regime.
12. The Academy does not, on the arguments advanced by the ARC, believe that the case for such a radical reform has been established or that the resulting benefits would outweigh the very significant transition costs. Some concerns that the Academy believes need to be addressed before it could be seriously contemplated include:
    • Implementation. To be effective a total reconstruction of all research funding arrangements, not just the ARC’s, would be necessary. There are also very serious difficulties that would need to be addressed concerning the funding of humanities and social sciences. It would almost certainly result in a dramatic upsurge in applications to the ARC and the other funding agencies, thereby adding significantly to their workloads.
    • Industrial relations. As noted also by Frank Larkins in his response it would be unrealistic, within current Australian award structures, for part of the salary of academic teaching and research staff to be so directly linked to project funding. Indeed, the suggestion is even contrary to some recent sensible reforms of higher education employment conditions and the desirability of institutions to strategically manage their staff to achieve agreed strategic outcomes.
    • A neglect or underestimation of the considerable 'leverage' between the different functions. It is likely that such a total separation of teaching and research would result in the discovery that the parts cost far more separately than they do together.
    • Cost. To be effective, significant additional funding would need to be injected. The salary arrangements are arguably not the most significant factor in the US system that has led to the emergence of strong research-intensive universities as the major drivers of US innovation. The high indirect costs (up to 100 per cent of direct costs) and paid by all Federal agencies would appear to be a more likely factor.

The Design of an Alternative Funding Scheme for Higher Education Research

13. Rather than such a radical and disruptive move to a US-style funding scheme, the Academy continues to argue for an Australian model that is closer to the UK than the US but might borrow from that system (and also the Canadian) where appropriate [10]. Any such move must be evolutionary to avoid major dislocation at a time when, notwithstanding the Backing Australia’s Ability initiatives, there is evidence that Australia’s investment in R&D as a percentage of GDP continues to slip behind many competitors (total R&D was 1.53 per cent of GDP in 2000-01 and was ranked 12th in terms of this ratio in 1998-99) [11]. And it must be cost effective.
14. In 2002 the Commonwealth Government allocated [12] in excess of $900 million through the block research grant. This block funding comprises the Institutional Grants Scheme (IGS), the Research Training Scheme (RTS), the Research Infrastructure Block Grant (RIBG) and the Regional Protection Fund. It is meant to cover postgraduate research training and contribute to institutions’ research, in particular through the support of research infrastructure costs.
15. The components of the research block (other than the regional protection fund) are allocated [13] on various research performance indicators that are a mix of research income, higher degree by research (HDR) load, HDR completions and the volume-based publications measure referred to above. While an improvement on the pre-White Paper arrangements, the total lack of any credible assessment of outcome or impact remains.
16. In addition, research is supported through the salary support of chief investigators who are employed as academic staff on the Commonwealth operating (teaching) grant or other institutional funding (eg, from international students). The ARC reports that the overall value of the operating grant estimated by universities to be spent on research activities is in the order of $570 million, and that around $47 million in 2002 was associated with the time spent by chief investigators on ARC research project work [14].

How might this current dual funding system be reformed?

17. Relative to current funding arrangements some possible options for reform may be that:
    1. Part of the imputed salary component (possibly of the order that the ARC has identified) should be moved from the teaching block to the research block.
    2. The existing RIBG ($113.7 million in 2002) be transferred to the funding agencies (as was originally argued in Kemp's Green Paper) but with the adoption of a US-style mandated but negotiated figure that would apply to ALL Commonwealth agencies. Some thought would need to be given to the CRC program, where a totally different approach currently exists to the funding of infrastructure costs – usually as an 'in-kind' contribution from the partners.
    3. An allocation mechanism be developed for the total block research grant, based on a set of performance indicators that are influenced to a greater degree than currently by some form of outcome assessment.
18. As already noted in paragraph 10 above, designing such a scheme would take some time and it must be cost effective. However with nearly a billion dollars at stake, the relative costs of ensuring that it is effectively allocated are rather different than in the case of the old Research Quantum which was of the order of $200 million. In its response to the West Report in 1997 the Academy proposed a modification of the UK-RAE that drew on the work of Paul Bourke and Michael Barber but was adapted to the smaller scale of Australia. That proposal is attached as Appendix A.
19. Other approaches may be worth considering. The Academy is, for example, aware of and strongly endorses CSIRO’s efforts to develop – within the context of the Government’s outcome-outputs framework – a suite of performance indicators consistent with relevant Government policies and its own Strategic Action Plan and Strategic Business Development Plan [15].
20. The extension of this approach to the assessment, monitoring and evaluation of the activities and outcomes of other contributors to Australian research – including specifically the universities – would seem quite feasible. The Academy would be pleased to play any role it could in developing such a scheme.

Major Research Infrastructure

21. In its 2001 Statement on priorities for the next Australian government the Academy stressed that 'there is an ongoing need to maintain internationally competitive infrastructure for R&D. There is a real weakness in policy in this area, because investment in infrastructure is not tied directly to national competitive grants' [16]. Long-term planning for investment in major national research facilities, with improved coordination between Federal and State governments, was put forward as one key aspect of a coherent approach to research infrastructure. The statement recommended that 'the ad hoc nature of the Major National Research Facility program must end by inclusion of a one-line budget item in the Science and Technology Budget each year, even if there are competitive rounds on a less frequent basis than annual.'
22. The Academy has continued to highlight the importance of adequate investment in research infrastructure and has recently circulated a discussion paper on this issue [17]. This paper builds upon the Academy’s statement on priorities by suggesting how a more coordinated approach to infrastructure investment could be achieved in practice and by explaining how more adequate investment in research infrastructure would benefit Australian science.
23. The effectiveness of investment in research infrastructure in the higher education sector would be improved if the current arrangements were simplified. Better long-term planning of investment in research infrastructure, including Major National Research Facilities, remains as a critical issue for university research.

Appendix A: An Australian-style Research Assessment Exercise

To devise an effective 'new Composite Index', it is of course necessary to be aware of the real and perceived limitations of the current Composite Index. Of the three components constituting the current index, the publications component has incurred the most criticism. These criticisms centre on:

• the cost involved in the collection itself and in assuring its veracity;
• the fact that the index is purely volume based and lacks any quality component;
• the devaluation of the output of particular disciplines; and
• a perception that it is influencing undesirable behaviour at both individual and institutional levels.

One response is to abolish the publications index completely. However, such a step would be a disappointing outcome of the sector’s attempt to emphasise output in the allocation of research funding. A better response is to seek an alternative that is both administratively practical and quality based. The most obvious alternative is the UK’s Research Assessment Exercise (RAE). The feasibility of implementing the RAE in Australia has been considered by Paul Bourke [18]. This report, which constitutes a comprehensive account of the RAE as implemented in Britain, recommended unequivocally against the suitability of this approach for Australia, on the grounds that Australia has neither the infrastructure required to mount such an exercise, nor sufficient scale in its university research activities to warrant the cost.

Whilst his analysis could not justify a UK-style RAE in Australia, Bourke advocated that a fraction (25 per cent, or approximately $55 million) of the existing Research Quantum should be allocated on the basis of a series of National Discipline Reviews. Given the cost of allocating such a small amount, this recommendation is unlikely to be accepted as practical. Nevertheless, the idea of a funding scheme based on a process in which a university’s performance on a field of research basis is assessed has a significant number of useful features for the Australian higher education sector as a whole. In addition, if the Research Quantum was to be increased significantly, the relative cost becomes less.

How might such a scheme be implemented? The following proposal draws on Bourke’s suggestions and incorporates some aspects of the 1993-5 Quality Assessment Reviews, particularly the 1995 round that focused on research [19].

1. Institutions nominate for assessment those major fields of research [20] in which they exceed minimum performance thresholds, eg coverage of a minimum range of sub-fields, a minimum research income, minimum number of research student completions, and particularly, a minimum number of research-active staff. The performance thresholds should be specific to a particular field of research.
2. The notion of 'research-active staff' is important both as a threshold for nomination and as the basic 'unit' for subsequent funding allocation. To qualify as research-active an academic staff member would need to have produced a minimum quantity of research output over a specified period. Again the criteria involved should be specific to a particular field of research. For these purposes, an author’s contribution to a paper should not be fractionated in the case of multiple authors, thereby encouraging inter-disciplinary, collaborative and cooperative research.
3. Assessment would be at two levels:
   • Institutionally on the basis of a portfolio demonstrating the institution’s performance over at least a minimum number of sub-fields of research. The portfolio would need to address specified criteria which might, in addition to quality, significance and impact of research outcomes, include issues such as innovation in research, the postgraduate research environment and research administration and management. As in the 1995 Quality Review, research improvement should be a criterion.
   • At the level of a centre, in cases where institutions do not meet the criteria allowing institutional assessment across the whole field of research. In those circumstances, institutions could nominate centres that represent pockets of strength and research activity in selected (narrower) areas of research. Again minimum levels of performance should be specified for nomination.
4. The results in both cases would be an assessment of either the institutional or centre activity on a three point scale [21], say Q, QQ, QQQ. The lowest level could simply involve confirmation of the claims for research-active staff and no portfolio assessment. Under those circumstances it should be possible to design a relatively simple statistical audit process to confirm a university’s claim without checking every individual academic involved.
5. Given this assessment, the index would be constructed by summing over all fields of research, the product of Q_aC_aN_a. Here N_a is the number of research-active staff; C_a reflects the relative costs of the different fields of research; and Q_a is a quantification of the outcome assessment exercise, say, 1 for Q, 2 for QQ and 4 for QQQ.

Unlike the UK’s RAE, in which lowly ranked departments gain little or no funding, universities would receive some 'base level' funding for the output of all research-active staff under this model. In addition, all universities would continue to gain funding through the other two components of the proposed new Composite Index. The resulting distributions would be more egalitarian than in the UK and more appropriate to Australia, where research activity is now fairly widely distributed across the system [22]. However, the proposed model would also provide considerable incentives for institutions to build concentrations of research activity and to reward and recognise high-quality performance.

Notes

• [1] In 2000, 3858 doctorates were completed. Source: DEST, Students 2001: Selected Higher Education Statistics, Table iii (2002)
• [2] ABS 8112.0, 2001-02 Research and Experimental Development, All Sector Summary (2002)
• [3] Based on the R&D expenditure reported by OECD nations and the OECD’s estimates of the R&D expenditure of non-OECD countries. OECD, Main Science and Technology Indicators, November (2001)
• [4] A. J. Salter and B. R. Martin, The economic benefits of publicly funded basic research: a critical review, Research Policy 30, 509-532 (2001)
• [5] L. Butler, Monitoring Australia's Scientific Research, Australian Academy of Science, October 2000
• [6] Submission 66
• [7] J Adams, Research assessment in the UK, Science 296, 805, 2002
• [8] F. Narin, CHI’s Research, Vol VIII, No.1, July 2000
• [9] Submission 341
• [10] Australian Academy of Science, Priorities in research and innovation for the next Australian government (2001).
• [11] ABS 8112.0, 2001-02 Research and Experimental Development, All Sector Summary (2002)
• [12] Higher Education Report for the 2002 to 2004 Triennium, Department of Education, Science and Training, December 2001, www.dest.gov.au/archive/highered/he_report/2002_2004/html/contents.htm
• [13] See Ref. 5 for details of the precise algorithms used
• [14] Submission 341
• [15] Review of the External Earnings Targets Policy applying to the Science Authorities (CSIRO, ANSTO and AIMS). A submission to the Chief Scientist, 18 December 2001. Available from the Australian Academy of Science on request.
• [16] Australian Academy of Science, Priorities in research and innovation for the next Australian government (2001). Page 7
• [17] Matthews, M., Providing the machinery of science: defining a whole-of-government strategy for securing access to critical research facilities. Discussion Paper. Australian Academy of Science. (2002). The paper puts forward the suggestion that we identify the ‘critical research facilities’, both within Australia and overseas, for which adequate access is a pre-requisite for conducting effective research. The identification of these critical research facilities would then form the basis for long-term planning, including investment in Major National Research Facilities.
• [18] Evaluating University Research: The British Research Assessment Exercise and Australian Practice, ARC/HEC Commissioned Report No. 56, July 1997.
• [19] Report on 1995 Quality Review, Committee for Quality Assurance in Higher Education, November 1995
• [20] As classified by the ABS these are Chemical Sciences; Agricultural Sciences; Applied Sciences & Technologies; Mathematical Sciences, Engineering; Information, Computing and Communication Technologies; Physical Sciences; Biological Sciences; Earth Sciences, Medical & Health Sciences; Humanities and Social Sciences. Some combination of these fields might be warranted.
• [21] Other grading scales with more points, such as the seven point scale of the UK RAE, could be considered
• [22] See: Patterns of Research Activity in Australian Universities, Commissioned Report No. 47, NBEET, 1996