ERA Evaluation of Data

Evaluation of ERA data will take place on a 4-digit FoR code level. There is a minimum threshold for analysis of a 4-digit FoR group, and if this is not reached, the data will be aggregated upwards, and evaluated at a 2-digit level. If there is insufficient data for analysis in a 2-digit FoR group, evaluation will not take place for that 2-digit FoR code for that institution.

For the purposes of national reporting for a discipline, evaluations will be undertaken of disciplines aggregated across institutions at the two-digit and four-digit levels regardless of the volume of research at those levels within individual institutions. This information will not be identifiable at an institution level. The evaluations will be informed by three broad categories of indicators:

1. Indicators of research quality

Research quality is considered on the basis of ranked outlets, citation analysis and peer-reviewed Australian and international research income. Peer review is also incorporated where necessary.

2. Indicators of research volume and activity

Research volume and activity is considered on the basis of total research outputs and research income within the context of the eligible researcher profile.

3. Indicators of research application

Applied research is considered on the basis of research commercialisation income and other applied measures.

4. Indicators of recognition

Research recognition is considered on the basis of esteeem measures.

The Higher Education Research Data Collection (HERDC) process managed by DIISR will continue to inform research block grant allocations until such a time that the Australian Government considers and implements any new mechanism.

Research Evaluation Committees (REC)

Research Evaluation Committees (RECs), comprising experienced, internationally recognised experts selected by the ARC, will evaluate the overall research performance of disciplines. There will be one REC for each cluster and membership of the REC will not be made public until after the ERA submission is complete.

After the ERA submission closes, the ARC will assign material to appropriate reviewers, who will submit a preliminary evaluation of the material. These preliminary evaluations, along with all the relevant ERA indicators, will then be compiled into reports for the RECs to consider. Each REC will then meet and agree on a final evaluation, which they will report to the ARC.

REC membership for ERA 2012 can be found here

Minimum threshold for evaluation

To ensure that there is a meaningful level of data to be evaluated, a low volume threshold exists for each Unit of Evaluation in ERA.

For disciplines where citation analysis is used, the low volume threshold is 50 apportioned indexed journal articles. This means that, if the number of apportioned indexed journal articles over the six year research outputs reference period is fewer than 50 in any four-digit or two-digit FoR at an institution then no evaluation will be conducted for that FoR at that institution.

For disciplines where citation analysis is not used, the low volume threshold is the equivalent of 50 submitted apportioned research outputs. This means that, if the number of submitted apportioned research outputs over the six year research outputs reference period is equivalent to fewer than 50 in any four-digit or two-digit FoR at an institution, no evaluation will be conducted for that FoR at that institution. For these disciplines, books are given an effective weighting of 5:1, compared with other research outputs. Books are weighted only for the purposes of determining the low volume threshold.

An institution may meet the low volume threshold for a two-digit FoR regardless of whether or not it has met the low volume threshold for any of the four-digit FoRs within that two-digit FoR. This is because outputs from all the four-digit FoRs within that two-digit FoR are aggregated for evaluation purposes to the two-digit level.

Citation analysis

Scopus was used as the provider of citation data, and therefore only articles indexed in Scopus will be eligible for citation analysis. Articles not indexed in Scopus were still submitted for inclusion in ERA, since citation analysis is only one indicator used in the evaluation.

Year-specific Australian and international citation benchmarks (PDF) were developed by the ARC for each 4-digit FoR code, and the relative citation impacts (RCI); and centile distributions were calculated.

• Relative citation impact: if the average citation of articles published in 2006 in a given FoR code is 10, and a particular 2006 article in that FoR code had been cited 20 times, it's RCI is 2.
• Centile distribution: the number of articles in the top 1%, 5%, 10% most cited articles in that FoR code in the world in that FoR group.

Role of Expert Review and ERA Peer Review

Expert review is central to ERA evaluations. ERA evaluations are conducted by RECs, comprised of internationally recognised researchers with expertise in research evaluation and broad disciplinary expertise.

ERA indicators will be presented to RECs as both profile and trend data, and where relevant will be informed by such discipline benchmarks as are appropriate to the indicator. RECs will also have access to underlying information associated with a research output (e.g. the bibliographic information associated with a journal article). Contextual information, such as the relevant two-digit Explanatory Statement, will also be provided as secondary information for evaluations.

ERA Peer Review

ERA peer review is conducted by reviewing a sample of research outputs rather than by reviewing every output.

For each four-digit FoR that is subject to ERA peer review, institutions must nominate 30% of the research outputs within this four-digit FoR for ERA peer review.Institutions must nominate a sample of research outputs for ERA peer review at the four-digit level, regardless of whether the low volume threshold is met at the four-digit level. ERA peer review will also be conducted at the two-digit level and nominated outputs from all relevant four-digit FoRs will be aggregated to the two-digit level for this purpose.

Each non-traditional research output nominated for ERA peer review must be accompanied by a statement identifying the research component of the output. The ERA peer review sample is to be comprised of a representative spread across all research output types.

Updated: 14 December 2011