31st May 2011: Dropped

For the past several years, in a developmental editing role, I have worked with academics on their grant applications and publication track records. The Australian Research Council’s Excellence for Research in Australia (ERA) initiative has been one external driver of this work. Minister Kim Carr’s announcement on 30th May that he is ending ERA’s journal ranking system has renewed debate, from incisive critics like Anna Poletti and Josh Gans.

The ARC originally conceived ERA’s 2010 journal rankings to bring evidence-based metrics and greater transparency to the higher education sector. Its Excel spreadsheet of 19,000 ranked journals was a controversial but useful tool to discuss with academics their ‘target’ journals and in-progress work. The team that built the Excel spreadsheet benchmarked the project against similar exercises in the United Kingdom, Europe and New Zealand. Whilst there was confusion about the final rankings of some journals, ERA 2010 was a move in the direction of Google’s analytics and ‘chaordic’ projects.

Minister Carr gave the following reason for ending the journal rankings:

“There is clear and consistent evidence that the rankings were being deployed inappropriately within some quarters of the sector, in ways that could produce harmful outcomes, and based on a poor understanding of the actual role of the rankings.

“One common example was the setting of targets for publication in A and A* journals by institutional research managers.”

Consider a more well-known ranking alternative to ERA: Hollywood’s Academy Awards. Studios invest hundreds of thousands of dollars in lavish marketing campaigns for their films. The nominees gain visibility and negotiation bargaining power in the film industry and for ancillary marketing deals. The winners gain substantive, long-term career and financial benefits, and not just a guest appearance on the television series Entourage. Success goes to the resilient. A similar dynamic to ERA 2010 plays out in the quarterly rankings of mutual fund managers, and in subcultures like the 1978-84 post-punk or ‘new wave’ music movement which ushered in MTV’s dominance.

ERA’s developers appear to have made three mistakes. First, there were inconsistencies between the draft and final rankings which remain unexplained, and that galvanised public criticism from academics. Second, its developers may not have considered the ‘unintended’ yet ‘real-world’ decisions that institutional research managers would make using ERA data: poaching high-performance researchers from competitors, closing low-ranked journals, reengineering departments, and evaluating the research components of promotions applications. If this sounds scary, you probably haven’t worked on post-merger integration or consortia bids. Third, the choice of letter codes – A*, A, B, C and unranked – rather than a different descriptive measure, introduced subtle anchoring, framing and representativeness biases into the ERA 2010 journal rankings.

Academics often knew what ERA sought to explicitly codify yet this tacit knowledge could be fragile. For instance, Richard Slaughter spent significant time during a Swinburne Masters in strategic foresight distinguishing between the field’s flagship journal (Elsevier’s Futures), the savvy new entrant (Emerald’s Foresight), and the critical vanguard (Tamkang University’s Journal of Futures Studies). Each journal had its own history, editorial preferences, preferred methodologies, and delimits. You ‘targeted’ each journal accordingly, and sometimes several at once if an article was controversial. ERA’s draft rankings reflected this disciplinary understanding but the 2010 final rankings did not. Likewise, to get into the A*-ranked International Security journal or to get a stellar publisher for international politics – Cambridge, Princeton, Yale, MIT – can take several years of drafting, re-drafting, editing, seminars and consulting with colleagues and professional networks. An influential book from one of these imprints can take up to five to seven years, from ideation to first journal reviews. The “quality is free” in the final manuscript.

This presented a challenge to institutional research managers and to university workload models. This developmental time can inform teaching, seminars, conference panels with exemplars, and peer networking. But it doesn’t necessarily show up quickly as a line-item that can be monitored by managers or evaluated by promotions committees. Instead, it can look like ‘dead time’ or high-reward gambits which have not paid off. Thus, the delays can be potentially detrimental and could affect institutional perceptions on academic performance. Institutional research managers also may not have the scope to develop the above tacit knowledge outside their disciplinary training and professional expertise.

So, like Hollywood producers, the institutional research managers possibly resorted to the A* and A journal codes as visible, high-impact, high-reward rankings. It was a valuable, time-saving short-cut through complex, messy territory. An academic with 15 A* and A level publications looked more convincing on paper than academic with 30 B and C level papers over the same period. A research team with A* and A level publications would be well positioned for ARC Discovery and Linkage grants. Australian Government funds from the annual research data collection had halo effects and financial benefits to institutions, like the Academy Award nominees have for film studios. It can be easier to buy-in expertise like professors and ambitious young researchers than to try and develop would-be writers. Rather than a “poor understanding”, I suggest the institutional research managers had different, perhaps less altruistic goals.

This was clearly a different role to what Carr and the ERA developers had intended, and conveyed to me at a university roadshow meeting. It was a spirited and valuable discussion: I pointed out to the ARC that a focus largely on A* and A level articles meant that 80% of research outputs were de-prioritised, including many B-ranked sub-field journals. However, there were alternatives to scrapping the system outright (or shifting to Field of Research codes and strengthened peer review): Carr might have made the inclusion and selection criterion for journals more public; could have addressed open publishing, and new and online journals; changed the ranking system from letter codes to another structure; and accepted some of the “harmful outcomes” as Machiavellian, power-based realpolitik which occurs in universities: what the sociologist Diane Vaughan calls “institutional deviance”. This may still happen whatever solution Carr and the ERA developers end up devising.

Perhaps if Carr had read two management books he would have foreseen the game that institutional research managers played with the ERA 2010 journal rankings. Jim Collins’ Good To Great (HarperCollins, New York, 2001) counselled managers to “get the right people on the bus”: A* and A level publishing academic stars. Michael Lewis’ Moneyball (W.W. Norton & Co, New York, 2003) examined how Oakland A’s general manager Billy Beane used sabermetrics – performance-based sports statistics – to build a competitive team, improve his negotiation stance with other teams, and maximise his training budget. Beane had to methodologically innovate: he didn’t have the multi-million dollar budgets of other teams. Likewise, institutional research managers appear to have used ERA 2010 like sabermetrics in order to devise optimal outcomes based on university research performance and other criteria. In their eyes, not all academics have an equal performance or scholarly contribution, although each can have a creative potential.

To me, the ERA 2010 journal rankings are still useful, depending on the appropriate context. They can inform discussions about ‘target’ journals and the most effective avenues for publications. They can be eye-opening in providing a filter to evaluate the quantity versus high-impact quality trade-offs in some publication track records. They have introduced me to journals in other disciplines that I wasn’t aware of, thus broadening the ‘journal universe’ being considered. They can be a well-delivered Platonic shock to an academic to expand their horizons and time-frames. The debate unleashed by Carr’s decision will be a distraction for some who will, instead, focus on the daily goals and flywheel tasks which best leverage their expertise and build their plausible, preferred, and personal futures.

3rd May 2011: Al Qaeda’s Strategic Culture

1. Initial Explanations

Al Qaeda-related literature focuses on several explanations for the terrorist organisation’s survival, growth and influence. A subset focuses on Osama Bin Laden as a charismatic leader and the Bin Laden family (Bergen 2006; Coll 2007). AQ evolved from Bin Laden’s financing of Abdullah Azzam’s Maktab al-Khidamat during the 1979-1989 Soviet-Afghanistan war and involvement in Saudi Arabia, Yemen, Sudan and Afghanistan (Scheuer 2002; Coll 2004; Wright 2004; Bergen 2011). Other explanations advance theories about AQ’s entrepreneurship (Reeves 1999); its perceived similarity to earlier terrorist groups (Gunaratna 2001); the possibility of broader movements in the Middle East (Burke 2003); the Hamburg cell responsible for the September 11 attacks (McDermott 2006); the influence of regional intelligence agencies (Bodansky 2001) and an unsuccessful Balkans jihad (Kohlmann 2004). Scheuer (2002) made historical analogies with the Founding Fathers of the American Revolution whilst Johnson (2000) suggested September 11 was ‘blowback’ from the Afghanistan mujahideen.

2. Strategic Culture Explanations

Alastair Johnston’s ‘three generations’ (1995, 1998) provide one framework to examine if and how AQ has a strategic culture. The post-September 11 portrayal of AQ as an existential threat echoes ‘first generation’ work on US-Soviet rivalry in nuclear weapons proliferation (Gray 1979, 1984; Snyder 1977). The numerous Bin Laden biographies and those of Ayman Al-Zawahiri (Al-Zayyat 2004) and Abu Mus’ab Al-Suri (Lia 2008) highlight a tradition of political profiling of terrorist group leaders (Post 2005; Post 2008; Hudson 2001). These histories suggest pivot points such as Bin Laden’s feud with the House of Saud during the 1990-91 Gulf War about the presence of US military forces. Scheuer’s (2002) analogical study of AQ and Patrick Porter’s Reciprocity of War thesis (2007, 2009) are different responses to the concerns that Ken Booth (1979, 2007) raised in ‘second generation’ literature about ethnocentric profiling. Coll (2004), Wright (2004) and Bergen’s (2011) reportage maps the doctrinal, ideational and organisational factors that other ‘second generation’ analysts like Elizabeth Kier (1992), Bradley Klein (1994), and Michael Desch (1998) have raised in their scholarship. As yet there is no empirical, falsifiable theory for AQ as Johnston (1995, 1998) posits, although Long (2006) contends that ‘fourth generation’ analysts focus on AQ’s covert development of chemical, biological, radiological and nuclear weapons. Strategic culture insights can also be applied to Bin Laden’s communiqués and public statements (Bin Laden and Lawrence 2005; Kepel and Milelli 2010), and to his interviews with journalists (Fisk 1996; Miller 1997; Arnett 1997; Fisk 1997; Ummat 2001).

3. Other Explanations

The ‘self-images’ of Al Qaeda appear to have evolved in proximity to the September 11 attacks and to the role of idea entrepreneurs (Mueller 2009; Mueller 2010). They may also reflect the potential biases of analysts from different intelligence agencies (Tversky and Kahneman 1979; Scheuer 2002; Treverton 2009; Jervis 2010) evaluating AQ as a complex, ambiguous problem. Snyder (1977) also noted this about Soviet elites’ possible use of nuclear weapons during the late Cold War.

2nd May 2011: Osama Bin Laden Dead

Obama Announces UBL's Death (Doug Mills/New York Times)

Al Qaeda’s leader Osama bin Laden was killed Sunday during an attack on Abottabad, Pakistan (NY Times story).

Prior to the White House’s official confirmation, the story leaked onto Twitter (NY Times story).

Bin Laden’s death changes a case study chapter I was beginning to outline about Al Qaeda.

Some resources:

Transcript of Obama’s comments on Osama bin Laden’s death.

New York Times obituary of Osama bin Laden.

• George Packer: Better Late Than Never.

• David Remnick: Obama and Osama.

• Lawrence Wright: Hey, Hey, Goodbye.

• Google Maps display of Bin Laden’s compound.

• PBS Frontline episode Hunting Bin Laden.

• Bin Laden biographers: Peter Bergen, Michael Scheuer, and Lawrence Wright.

• Steve Coll on Young Osama (New Yorker).

• Lawrence Wright on Rebellion Within Al Qaeda (New Yorker).