9th March 2010: ERA Strategies for ‘Disappeared’ Academic Publication Records

Two separate meetings on career directions: Where do you want to be in 3-to-5 years? What actions can you take to move toward these goals?

 

Collaborator Ben Eltham has written a piece on how the 2010 final rankings for Excellence for Research in Australia (ERA) has affected his academic publishing record: ‘When Your Publication Record Disappears’. A title reminiscent of Nine Inch Nails‘ song ‘The Day The Whole World Went Away.’

 

For the past year I have been dealing, professionally, with issues that Ben raises.Whilst outside academia, journal publications are often viewed as irrelevant, they are crucial to the academic promotions game, and to getting external competitive grants. A personal view:

ERA is the Rudd Government’s evaluation framework for research excellence, developed by the Australian Research Council, to include a ranked list of academic journals and discipline-specific conferences. The ARC released the final ranked list in February 2010. It may be revised and updated in the future, but not this year.

 

The ARC’s goal for this ranked list was to ensure it was comprehensive, peer-reviewed,
would stand up to international scrutiny, and would provide guidance to administrators, managers and researchers on quality research outputs.

 

In the near-term ERA’s 2010 final rankings will require adjustments to our academic publication records. Some of the journals we have published in such as M/C were revised down or excluded, probably because of perceived issues with their peer review process. More starkly, ERA’s guidelines for academic publications filters out most of my writings over the past 15 years: magazines and journals that no longer exist (21C, Artbyte), websites (Disinformation), magazine articles with original research (Desktop, Marketing, Internet.au), unrefereed conference papers, technical reports, and contract research. It also does not usually include textbooks, research monographs, and working papers. The ‘disappearance’ effect that Ben describes also happens elsewhere: when Disinformation upgraded its site to new servers, we sometimes lost several articles during the transition that writers had no back-ups of.

Others are in a tougher position: mid-career academics who have taught and not published or applied for external competitive grants, or who understandably focussed on quantity of articles for DEST points rather than ERA’s focus on quality ranked journals and ‘field of research’ codes. ERA has caused a dramatic re-evaluation for some mid-career and senior academics of their publication record, impact factors, and other esteem measures.

 

In response to Ben’s piece, I mentioned the following possible strategies:

 

1. Know your University’s policy and procedure on ‘research active’ status and how it is calculated. There may be variations of this at Faculty and School level. Once you finish your PhD and have Early Career Researcher status for the next 5 years, focus on building your publication record, internal grants as a rehearsal for external grants, forming a collaborative team, and establishing networks to have industry and government partners. The ARC does not want ERA to be used for academic performance reviews, but this is likely to happen.

 

2. Send in all relevant research outputs to your University’s annual HERDC data collection. Although there is usually at least 12 months delay in this, HERDC outputs mean you contribute to the block grant funding that your University can get for research. Some of this is usually passed on to individual researchers for School and Faculty level research accounts. Where you can, include citation data using ISI Web of Knowledge or Scopus.

 

3. Develop a ‘program of research’ with a 3-to-5 year time-frame. The ‘program’ should encompass multiple projects, collaborations, and creative work or research outputs. This helps the post-PhD transition to ECR status, and ensures you don’t try to put everything into one or two journal articles. One challenge is to first conceptualise what this ‘program of research’ might be, and then translating it into ‘field of research’ codes that are used as institutional metadata. A second is to be able to articulate to others how your approach differs from others in the field; what your distinctive, significant and original contributions may be; and how you will achieve your goals, on-budget, and within the specified time-frame.

 

4. Scholarly published books, i.e. by academic publishers, are counted for both ERA and HERDC data collection. The problem Ben notes for history academics is also a problem for political scientists, who may publish in top journals, but whose life-work usually goes into a major book for Cambridge, Princeton, Routledge, Georgetown, Harvard, Yale, or a similar academic publisher. The ARC does not have a list of academic publishers.

 

A second problem:

 

The ‘research active’ policies and procedures at many universities give a book the same points as two or three articles published in an A* or A-level journal. This points system seriously underestimates the work involved to conceptualise and write the book, and then to get it through the publisher’s development editing process. So, as an incentives scheme it may have subtle and unanticipated effects on knowledge creation.

 

5. Get your research outputs into your University’s institutional repository. This may be run by IT Services or Library staff. The repository may have different policies and scope of what it will accept: I have publications at both Victoria and Swinburne universities, and each
institution is slightly different. Take the time to include the relevant metadata for each submission, especially the 4-to-6 digit ‘field of research’ codes. Keep the last version of the article you submitted to an academic journal, because due to publisher copyright and intellectual property contracts, often this is the only version that an institutional repository can publish.

6. Archive your ‘primary’ research and develop a stream of publications. Ben probably approached his excellent Meanjinwork with the mindset of a journalist and long-form essayist. He did 20 interviews for one piece. This is more work than goes into many articles for B- and C-level journals, and even for some A-level ones. He could easily reuse and revisit this ‘primary’ research, for the next three or four years in academia. For example, a paper that reviews current frameworks to identify a knowledge gap or research problem, could then lead to a methodology paper, then to comparative case studies, and then to an evaluation or meta-analysis study.

Jacob Weisberg’s Possible Fallacies

Slate‘s Jacob Weisberg recently surveyed a range of sociopolitical issues from nuclear proliferation to the China Century where the expert consensus might be wrong.

Weisberg’s survey sample includes macroeconomic aggregates (home ownership, asset investment classes, international competition), geostrategic stability (China, nuclear proliferation), and longrun environmental issues (climate change, fossil fuels). In each, Weisberg contrasts a prevailing view, hypothesis or expert with a challenger.

Below are some thoughts on Weisberg’s analyses and observations on research methods in journalism
.

· Selection and Framing of Experts: Weisberg mentions the late realist Samuel P. Huntington‘s thesis on political order in changing societies, to raise concerns about China’s near-future macroeconomic growth. He also refers to neorealist Kenneth Waltz‘s views that nuclear proliferation is inevitable. Huntington and Waltz both represent dominant traditions within international relations theory, and neither are as new or radical as Weisberg seems to portray. The selection and framing of experts is crucial: it would be even more interesting to compare their views with other schools of thought, such as liberal democratic, critical or constructivist theories, which have different deductive premises and levels of analysis. After all, neoconservative fears about Iraq were not just that nuclear weapons acquisition was a defensive action, but also the security orientation of ‘Axis of Evil’ regimes and their potential connections to non-state actors. Perhaps Weisberg could have checked with a nuclear proliferation specialist such as Graham Allison or Jessica Stern. Equally, a China specialist might convincingly show that Hu Jintao’s Chinese government is aware of Huntington’s thesis and has plotted a different future trajectory

· Heretics & Mavericks: Weisberg cites Freeman Dyson as a heretic of climate change models, although Dyson’s scientific expertise is primarily as a physicist and cosmologist. Other mavericks such as the late Federal Bureau of Investigation counterterrorism expert John O’Neill and United Nations weapons inspector Scott Ritter were ostracised in organisational politics, whilst economist Nouriel Roubini strengthened his reputation by foreseeing the global financial crisis. Perhaps it’s also a matter of luck, timing, and having an effective image makeover.

· Incomplete Deductive Arguments: Weisberg observes that market analysts are re-evaluating house ownership, stock investments and the global competitiveness of car manufacturers. Yet the common assumptions that Weisberg mentions are really incomplete deductive arguments with hidden premises. First, home ownership does not necessarily lead to greater community involvement, and the negative factors mentioned (financial risk, labour market mobility, commute time) are weighted during the purchase decision or emerge later in decision regret. Second, evaluating and comparing the risk premia of bonds versus stocks requires further details on the time horizon, sampling frame, weightings and volatility. Shocks such as the 1973 OPEC oil crisis, the 1995-2000 dotcom bubble, the 1997 Asian currency crisis and the current global financial crisis may affect the comparison of bond and stock returns. Third, although the Detroit Three have cut costs and launched several international joint ventures, their debts and liabilities are partly the result of earlier decisions. Weisberg’s argument that these balance sheet issues are the Detroit Three’s main barrier is not really new: asset management and private equity firms have targeted them for over two decades in their acquisition, reengineering and turnaround attempts. Perhaps that’s why the Obama administration has hired media banker Stephen Rattner.

· Inferences from Small Samples: In his sections on long-run stocks, climate change and fossil fuels Weisberg quotes from a single academic study. Whilst this establishes a challenger hypothesis it also probably means that the sample is too small for inferences that would establish a definitive Kuhnian paradigm shift in a knowledge field. Weisberg would have a more robust argument if he referenced meta-analyses which evaluated a group of studies for their sample size and other effects.