20th October 2012: The Big Chill



Australia’s budget austerity means the Australian Research Council (ARC) might freeze or put on hold all competitive grant funding rounds and awards. Australian academics are upset.


For two decades the ARC has been one of Australia’s most important funding agencies for competitive research grants. ARC funding success provides career prestige and national visibility for an academic researcher or collaborative research team. The ARC’s Discovery and Linkage rounds have a success rate of 17-19% whilst the DECRA awards for Early Career Researchers (ECR), or first five years after PhD completion, has been 12%. An ARC grant is a criterion that many promotions committees use for Associate Professor and Professor positions. This competitiveness means that a successful ARC grant can take a year to write. The publications track record needed for successful applicants can take five to seven years to develop.


The ARC’s freeze decision is symptomatic of a deeper sea change in Australian research management: the rise of ‘high finance’ decision-making more akin to private equity and asset management firms.


Richard Hil’s recent book Whackademia evoked the traditional, scholarly Ivory Tower that I remember falling apart during my undergraduate years at Melbourne’s La Trobe University. Hil’s experience fits a Keynesian economic model of universities. Academics got tenure for life or switched universities rarely. There was no ‘publish or perish’ pressure. There was a more collegial atmosphere with smaller class sizes. Performance metrics, national journal rankings, and research incentive schemes did not exist. The “life of the mind” was enough. Universities invested in academics for the long-term because they had a 20-30 year time horizon for research careers to mature. There was no intellectual property strategy to protect academics’ research or to create different revenue streams.


‘High finance’ decision-making creates a different university environment to Hil’s Keynesian Ivory Tower. Senior managers believe they face a hyper-competitive, volatile environment of disruptive, low-cost challengers. This strategic thinking convinced University of Virginia’s board chair Helen Dragas to lead a failed, internal coup d’etat against president Teresa Sullivan with the support of hedge fund maven Paul Tudor Jones III. The same thinking shapes the cost reduction initiatives underway at many Australian universities. It creates a lucrative consulting market in higher education for management consulting firms. It influences journalists who often take the public statements made at face value instead of doing more skeptical, investigative work.


The ARC has played a pivotal role in the sectoral change unfolding in higher education. Its journal rankings scheme the Excellence for Research in Australia (ERA) provided the impetus for initial organisational reforms and for the dominance of superstar economics in academic research careers. ERA empowered research administrators to learn from GE’s Jack Welch and to do forced rankings of academics based on their past research performance. ARC competitive grants and other Category 1 funding became vital for research budgets. Hil’s professoriate are now expected to mentor younger, ECR academics and to be ‘rain-makers’ who bring in grant funding and other research income sources. Academics’ reaction to the ARC’s freeze decision highlights that the Keynesian Ivory Tower has shaky foundations.


The make-or-buy decision in ‘high finance’ changes everything. Hil’s Ivory Tower was like a Classical Hollywood film studio or a traditional record company: invest upfront in talent for a long-term payoff. Combining ERA’s forced rankings of academic staff with capital budgeting and valuation techniques creates a world that is closer to private equity or venture capital ‘screening’ of firms. Why have a 20-to-30 time-frame for an academic research career when you can buy-in the expertise from other research teams? Or handle current staff using short-term contracts and ‘up or out’ attrition? Or your strategy might change in several years to deal with a volatile market environment? Entire academic careers can now be modeled using Microsoft Excel and Business Analyst workflow models as a stream of timed cash-flows from publications, competitive grants, and other sources. Resource allocative decisions can then be considered. ARC competitive grants and research quality are still important — but ‘high finance’ decision-making has changed research management in universities, forever.


Today’s young academics face a future that may be more like auteur film directors or indie musicians: lean, independent, and self-financing.


Photo: missdarlene/Flickr.

11th April 2012: How Academia Kills Writing

I recently had some productive exchanges with Roy Christopher and Axel Bruns on academic writing strategies. Christopher wrote-up his insights:


I am sympathetic to all of these conditions, but I have found it important to cultivate the ability to write at any time, in any circumstance — even if it’s just collecting thoughts about something. I keep a pen and paper in my pocket at all times, pen and pad by my bed, notebook(s) in my backpack and all over the house. I do find that I need large chunks of uninterrupted time to surmount larger writing tasks, but the ubiquity of computers, portable or otherwise, makes writing anywhere a much more viable option. [emphasis added]


Christopher’s insight led to an email exchange on the barriers that academia poses for writers. I think about this a lot in my current university gig as a developmental editor. I also work with a talented copy-editor. Here are six ways that academia kills writing:


1. Perverse incentive structures. Christopher and I are both intrinsically motivated writers who approach it as a craft. We blog, write journal articles and in-progress PhD dissertations, and Christopher has several book projects. In contrast, some academics I know write only for performance-based incentives. They play games such as writing fake conference papers, sending book manuscripts to vanity publishers, and publishing in obscure international journals. This leads the university research administrators to change the incentives structures. It also introduces scoping problems into competitive grants: the journal article(s) only get written if the money is awarded. It’s very rare that I find an intrinsically motivated writer: maybe an Early Career Researcher who has just finished their PhD, or a senior academic intent on making a contribution to their field or discipline. I wish academics had a more hip-hop or punk sensibility and just did the work, regardless of the institutional incentives.


2. Misuse of university research metrics. The Australian Research Council‘s Excellence for Research in Australia shifted the research conversation to performance and quality-based outputs. This also lead to games such as poaching academics who had ERA publishing track records. However, it also sometimes led to a narrow focus on A* and A-level journals without changes to the workload models or training investment for academic skills and robust research designs. Not everyone is Group of 8, Harvard or Stanford material, or at least not at their career stage. Metrics use must be counter-balanced with an understanding of intellectual capital and development strategies. To-date the use of ERA and Field of Research metrics is relatively unsophisticated, and it can often actually de-value academic work and publishing track records.


3. A failure to understand and create the conditions for the creative process. The current academic debate about knowledge creation swings between two extremes. On the one hand, budget-driven cost-cutting similar to GE’s Work-Out under Jack Welch or private equity turnarounds. On the other, a desire to return to a mythical Golden Age where academics are left alone with little accountability. Both views are value destructive. The middle ground is to learn from Hollywood studios, music producers, and academic superstars about the creative process, and to create the conditions for it. This means allowing time for insights to emerge or for academics to become familiar with new areas. It means not relying on conferences and being pro-active in forming collaborative networks. It means treating academic publications as an event and leveraging them for maximum public impact and visibility. Counterintuitively, it can also mean setting limits, stage gates, and ‘no go’ or ‘abandon’ criteria (real options theory can be a useful tool). This is one reason why Christopher and I sometimes exchange stories of the strategies that artists use: to learn from them. This is a different mentality to some university administrators who expect research publications to emerge from out of nowhere (a view often related to the two barriers above).


4. Mystifing the blind peer review process. What differentiates academic research from other writing? Apart from the research design, many academics hold up the blind peer review process to be a central difference. Usually, a competitive grant or a journal article goes to between two and five reviewers, who are often subject matter experts. The identities of both the author(s) and the reviewers are kept secret from each-other. Supposedly, this enhances the quality of the review process and the candour of the feedback provided. Having studied the feedback of 80 journal articles and 50 competitive grants, I disagree. The feedback quality is highly reviewer dependent. Blind peer review provides a lack of transparency that allows reviewers to engage in uber-critical reviews (without constructive or developmental feedback), disciplinary in-fighting, or screeds on what the reviewer wished had been written. Many academic journals have no rejoinder process for authors to respond. These are problems of secrecy and can be avoided through more open systems (a lesson from post-mortems on intelligence ‘failures’).


5. Being set up to fail through the competitive grants process. A greater emphasis on research output metrics has prioritised success in competitive grants. Promotions committees now look for a track record in external grants for Associate Professor and Professor roles. Australian universities do not often have endowed chairs or institutional investment portfolios — so they are more reliant on grant income. Collectively, these trends translate into more pressure on academics to apply for competitive grants. However, success is often a matter of paying close attention to the funding rules, carefully scoping the specific research project and budget, developing a collaborative team that can execute on the project, and having the necessary track record in place. These criteria are very similar to those which venture capitalists use to evaluate start-ups. Opportunity evaluation, timing, and preparatory work is essential. Not meeting this criteria means the application will probably fail and the grant-writing time may be wasted: most competitive grants have a 10-20% success rate. Some universities have internal grant schemes that enable new academics to interact with these dynamics before applying to an external agency. In all cases, the competitive grant operates as a career screening mechanism. For institutions, these grants are ‘rain-making’ activities: they bring money in, rather than to the individual academic.


6. A narrow focus on A* and A-level journals at the expense of all other forms of academic writing. The ARC’s ERA and similar schemes prioritise peer reviewed journals over other forms of writing. (This de-valued large parts of my 18-year publishing history.) The 2009 and 2010 versions of ERA had a  journal ranking list which led many university administrators I know to focus on A* and A-level journals. I liked the journal ranking list but I also saw it had some perverse effects over its 18 months of use. It led to on-the-fly decisions made because of cumulative metrics in a publishing track record. It destroyed some of the ‘tacit’ knowledge that academics had about how and why to publish in particular journals. It de-valued B-ranked journals that are often sub-discipline leaders. It helped to create two groups of academics: those with the skills and training to publish in A* and A-level journals, and those who did not. It led to unrealistic expectations of what was needed to get into an A* journal like MIT’s International Security: a failure to understand creative and publishing processes. The narrow emphasis on journals ignored academic book publishers, CRC reports, academic internet blogs, media coverage, and other research outputs. Good writers, editors and publishers know differently: a high-impact publication can emerge from the unlikeliest of places. As of April 2012, my most internationally cited research output is a 2009 conference paper, rejected from the peer review stream due to controversy, that I co-wrote with Ben Eltham on Twitter and Iran’s 2009 election crisis. It would be excluded from the above criteria, although Eltham and I have since written several articles for the A-level journal Media International Australia.


Awareness of these six barriers is essential to academic success and to not becoming co-dependent on your institution.

2nd April 2012: Bridging The Academic/Policy Divide

Foreign Policy‘s Dan Drezner on academics and policy wonks:

I think the academic/policy divide has been wildly overblown, but here’s my modest suggestion on how to bridge it even further.  First, wonks should flip through at recent issues of APSR and ISQand hey, peruse International Organization, International Security, and World Politics while you’re at it.  You’d find a lot of good, trenchant, policy-adjacent stuff.  Second, might I suggest that authors at these journals be allowed to write a second abstract — and abstract for policymakers, if you will?  Even the most jargonesed academic should be able to pull off one paragraph of clean prose.  Finally, wonks should not be frightened by statistics.  That is by far the dominant “technical” barrier separating these articles from general interest reader.


I get APSR and ISQ mailed every few months, and am still working my way through their research designs. I read International Security and World Politics for PhD research. A lot of the policy-relevant journals were B-ranked in the Australian Research Council‘s Excellence for Research in Australia 2010 exercise.

16th February 2012: Academic Blogging

fred and academic blogging


The Lowy Institute’s Sam Roggeveen contends that Australian academics would benefit from blogging their research (in response to The Australian‘s Stephen Matchett on public policy academics).


I see this debate from several perspectives. In a former life I edited the US-based alternative news site Disinformation (see the 1998-2002 archives). I also work at Victoria University as a research administrator. I’ve blogged in various forums since 2003 (such as an old LiveJournal blog). In contrast, my PhD committee in Monash’s School of Political and Social Inquiry are more likely to talk about book projects, journal articles, and media interviews.


As Roggeveen notes, a major uptake barrier is the structure of institutional research incentives. The Australian Research Council’s Excellence for Research in Australia (ERA) initiative emphasises blind peer reviewed journal articles over other forms. Online blogging is not included as an assessable category of research outputs although it might fit under ‘original creative works’. Nor is blogging included in a university’s annual Higher Education Research Data Collection (HERDC) outputs. University incentives for research closely follow ERA and HERDC guidelines. The ARC’s approach is conservative (in my view) and focuses on bibliometrics.


I know very few academics who blog. Many academics are not ‘intrinsic’ writers and are unused to dealing with developmental editors and journals. University websites often do not have blog publishing systems and I’ve seen several failed attempts to do so. Younger academics who might blog or who do use social media are often on casual or short-term contracts. The ones who do blog like Ben Eltham have a journalism background, are policy-focused, and are self-branded academic entrepreneurs.


Roggeveen is correct that blogging can potentially benefit academics — if approached in a mindful way. I met people like Richard Metzger and Howard Bloom during my publishing stint. I am regularly confused with QUT social media maven Axel Bruns — and we can now easily clarify potential queries. Blogging has helped me to keep abreast of sub-field developments; to build networks; to draft ideas for potential journal articles and my PhD on strategic culture; and has influenced the academic citations of my work and downloads from institutional repositories.


Problem is, HERDC or ERA have no scope for soft measures or ‘tacit’ knowledge creation — so blogging won’t count to many universities.


That Roggeveen needs to make this point at all highlights how much the internet has shifted from its original purpose to become an online marketing environment. Tim Berners-Lee’s proposal HyperText and CERN (1989) envisioned the nascent internet as a space for collaborative academic research. The internet I first encountered in 1993-94 had Gopher and .alt newsgroups, and later, web-pages by individual academics. Regularly visited example for PhD research: University of Notre Dame’s political scientist Michael C. Desch and his collection of easily accessible publications.  It’s a long way from that free environment to today’s “unlocking academic expertise” with The Conversation.


Photo: davidsilver/Flickr.

27th January 2012: Handling Article Rejection

I recently got negative reviews for two articles submitted to the Journal of Futures Studies (JFS). Many academics I know find article rejection to be highly stressful. Below are some comments and strategies addressed to three different audiences: academic authors; reviewers; and university administrators. Attention to them may improve the probability that your article is accepted for publication in an academic journal.


Academic Authors


1. Be very familiar with your ‘target’ journal: its editors and review panel, its preferred research design and methodologies, and how it handles controversies and debates in your field. Look for an editorial or scoping statement that explains what kinds of articles the journal will not accept.


2. Before submission do a final edit of your article. Define all key terms or cite past definitions if you have referred to the scholarly literature. Check paragraph structure, connecting sentences, section headings, and that the conclusions answer the key questions you have raised in the beginning. Cite some articles from the target journal if possible. Consider who is likely to review your article and factor this into your discussion of key debates. Use redrafting for honing the article and for self-diagnosis of mental models.


3. Ask if the journal has a rejoinder process for authors to reply to the blind peer review comments. A rejoinder is not an invitation to personal attacks or to engage in flame-wars. Rejoinders do enable authors to address situations in which one or more reviewers misunderstand the article, frame their comments in terms of an article they wish the author had written (rather than the actual article), or where there are concerns about the methodologies used, the research design, or data interpretation. An effective rejoinder process respects all parties, maintains the confidentiality of the blind peer review process, and provides an organisational learning loop. A rejoinder response does not necessarily reverse an editorial decision not to publish.


4. If the journal does have a rejoinder process then carefully examine the feedback pattern from reviewers. Highlight where one reviewer answers the concerns that another reviewer raised: this should neutralise the negative comments or at least show that varied opinions exist. It is more difficult when several reviewers raise the same concerns about an article.


5. Set a threshold limit on the amount of editing and rewrites you will do: you have other opportunities. A rejected article might fit better with another journal; with a substantial rewrite; with a different research design; or could be the stepping stone to a more substantive article. Individual reviews also reflect the particular reviewer and their mental models: this can sometimes be like an anthropological encounter between different groups who misunderstand each-other. Sometimes reviewers like critics just get it wrong: one of my most highly cited publications with international impact was dropped from the blind peer review stream.




1. Use the ‘track changes’ and ‘comment’ function of your word processor to provide comments. It can be difficult for authors to read comments that you provide in the body text and that is written in the same font. Be time-responsive: authors hate waiting months for feedback.


2. Do a first read of the article without preconceptions: focus on the author’s state intent, their narrative arc, the data or evidence, and their conclusions. Be open to the article you have been asked to review, rather than the article that you wish the author had written. Be open to innovation in data collection, methodologies, and interpretation. Even do a self-review of your own comments before you send your feedback to the journal editors.


3. Know your own mental models. That is, how you see the field or discipline that you are reviewing in; your preference for specific methodologies and research designs; your stance on specific controversies and debates; and what kind of material you expect the journal to publish. Be aware of situations in which you are asked to review articles because you have a particular stance: the tendency is to write lukewarm reviews which focus on perceived deficiencies or ‘overlooked’ material. Be careful of wanting to ‘police’ the field’s boundaries.


4. Use your feedback as a developmental opportunity for the author. Don’t just give negative feedback, faulty sentence construction or grammar. If you don’t like something then explain why so that the author can understand your frame of reference. Focus also issues of research design, methodologies, and data interpretation. If there are other external standards or alternative perspectives (such as on a controversy or debate) then mention them. Articles often combine several potential articles or can have scope problems so note them. Highlight sections where the author makes an original, scholarly contribution, including new insights or where you learned something. It’s important to provide developmental feedback even when you reject an article for publication. A developmental review may evoke in authors the ‘moment of insight’ that occurs in effective therapy. The mystique of the blind peer review process ultimately comes down to the reviewer’s attention to the craft of providing constructive yet critical feedback that sets up future opportunities for the academic to advance their career.


5. Poison pen reviews have consequences. This is clearer in creative industries like film and music where bad reviews can kill a project or career. Pauline Kael and Lester Bangs are honoured in film and music circles respectively because they brought sensitivity and style to their reviews, even when they hated an artist. In academia, the blind peer review process can lead to internecine wars over different methodologies or research designs: problems that don’t usually arise in open publishing (because all parties know who is making the comments) or that can be handled through editorial review standards and a rejoinder process. Nevertheless, a negative review will have consequences. The author may not revise the article for publication. They may publish in a different journal. They may drop the project. In some cases, they may leave the field altogether. Consider how to frame the review so that you address the developmental need in a constructive manner.


University Administrators


1. Know the norms, research designs and methodologies, leading research teams, and the most influential and international journals in at least one discipline. This gives you a framework to make constructive inferences from. You will develop awareness of these factors in other disciplines through your interviews with different academics.


2. Understand the arc or life-span of academic careers: the needs of an early career researcher and the professor will differ, and this will influence which journals they seek to publish in. Every successful publication navigates a series of decisions. Know some relevant books and other resources that you can refer interested academics to.


3. Have some awareness of international publishing trends which affect journals and their editorial decisions. These include the debate about open publishing, the consolidation of publishing firms, and the different editorial roles in a journal. Be aware of the connection between some journals and either professional associations or specific university programs.


4. Know what to look for in publication track records. These include patterns in targeting specific journals; attending conferences; building networks in the academic’s discipline; and shifts in research programs. An academic may have a small number of accepted articles when compared with the number that have been written and rejected by specific journals. Use the publication track record as the basis for a constructive discussion with the individual academic, honoring their experience and resources, and using solution-oriented therapeutic strategies.


5. Understand that quality publications require time which equates to university investment in the academic’s career. The journal letter rankings in the Australian Research Council’s Excellence for Research in Australia led some university administrators to advise academics only to publish in A* and A-level journals. But not everyone will realistically achieve this. There can be variability of effort required: one A-level article I co-wrote required a substantive second draft; another took months to discuss, a day to do the first draft, and it was then accepted with minor changes. On the other hand, articles accepted in the A* journal International Security (MIT) have usually gone through multiple rounds of blind peer review, the authors are deeply familiar with the field’s literature, and have work-shopped the article extensively with colleagues, in graduate school seminars, and at international conferences. This takes a median two to five years to occur. The late Terry Deibel took almost 20 years to conceptualise and refine the national security frameworks he taught at the United States National War College for Foreign Affairs Strategy: Logic for American Statecraft (Cambridge: Cambridge University Press, 2007) and Deibel also spent two years of sabbatical — in 1993 and 2005-06 — to write it. John Lewis Gaddis spent 30 years of research on George F. Kennan: An American Life (New York: The Penguin Press, 2011) and five years to write it. Both books make substantive scholarly contributions to their fields; both books also required the National War College and Yale University to make significant financial investments in the authors’ careers. Are you making decisions based on short-term, volume-driven models or helping to create the enabling conditions that will help academics to have a similar impact in their respective fields?

9th March 2010: ERA Strategies for ‘Disappeared’ Academic Publication Records

Two separate meetings on career directions: Where do you want to be in 3-to-5 years? What actions can you take to move toward these goals?


Collaborator Ben Eltham has written a piece on how the 2010 final rankings for Excellence for Research in Australia (ERA) has affected his academic publishing record: ‘When Your Publication Record Disappears’. A title reminiscent of Nine Inch Nails‘ song ‘The Day The Whole World Went Away.’


For the past year I have been dealing, professionally, with issues that Ben raises.Whilst outside academia, journal publications are often viewed as irrelevant, they are crucial to the academic promotions game, and to getting external competitive grants. A personal view:

ERA is the Rudd Government’s evaluation framework for research excellence, developed by the Australian Research Council, to include a ranked list of academic journals and discipline-specific conferences. The ARC released the final ranked list in February 2010. It may be revised and updated in the future, but not this year.


The ARC’s goal for this ranked list was to ensure it was comprehensive, peer-reviewed,
would stand up to international scrutiny, and would provide guidance to administrators, managers and researchers on quality research outputs.


In the near-term ERA’s 2010 final rankings will require adjustments to our academic publication records. Some of the journals we have published in such as M/C were revised down or excluded, probably because of perceived issues with their peer review process. More starkly, ERA’s guidelines for academic publications filters out most of my writings over the past 15 years: magazines and journals that no longer exist (21C, Artbyte), websites (Disinformation), magazine articles with original research (Desktop, Marketing, Internet.au), unrefereed conference papers, technical reports, and contract research. It also does not usually include textbooks, research monographs, and working papers. The ‘disappearance’ effect that Ben describes also happens elsewhere: when Disinformation upgraded its site to new servers, we sometimes lost several articles during the transition that writers had no back-ups of.

Others are in a tougher position: mid-career academics who have taught and not published or applied for external competitive grants, or who understandably focussed on quantity of articles for DEST points rather than ERA’s focus on quality ranked journals and ‘field of research’ codes. ERA has caused a dramatic re-evaluation for some mid-career and senior academics of their publication record, impact factors, and other esteem measures.


In response to Ben’s piece, I mentioned the following possible strategies:


1. Know your University’s policy and procedure on ‘research active’ status and how it is calculated. There may be variations of this at Faculty and School level. Once you finish your PhD and have Early Career Researcher status for the next 5 years, focus on building your publication record, internal grants as a rehearsal for external grants, forming a collaborative team, and establishing networks to have industry and government partners. The ARC does not want ERA to be used for academic performance reviews, but this is likely to happen.


2. Send in all relevant research outputs to your University’s annual HERDC data collection. Although there is usually at least 12 months delay in this, HERDC outputs mean you contribute to the block grant funding that your University can get for research. Some of this is usually passed on to individual researchers for School and Faculty level research accounts. Where you can, include citation data using ISI Web of Knowledge or Scopus.


3. Develop a ‘program of research’ with a 3-to-5 year time-frame. The ‘program’ should encompass multiple projects, collaborations, and creative work or research outputs. This helps the post-PhD transition to ECR status, and ensures you don’t try to put everything into one or two journal articles. One challenge is to first conceptualise what this ‘program of research’ might be, and then translating it into ‘field of research’ codes that are used as institutional metadata. A second is to be able to articulate to others how your approach differs from others in the field; what your distinctive, significant and original contributions may be; and how you will achieve your goals, on-budget, and within the specified time-frame.


4. Scholarly published books, i.e. by academic publishers, are counted for both ERA and HERDC data collection. The problem Ben notes for history academics is also a problem for political scientists, who may publish in top journals, but whose life-work usually goes into a major book for Cambridge, Princeton, Routledge, Georgetown, Harvard, Yale, or a similar academic publisher. The ARC does not have a list of academic publishers.


A second problem:


The ‘research active’ policies and procedures at many universities give a book the same points as two or three articles published in an A* or A-level journal. This points system seriously underestimates the work involved to conceptualise and write the book, and then to get it through the publisher’s development editing process. So, as an incentives scheme it may have subtle and unanticipated effects on knowledge creation.


5. Get your research outputs into your University’s institutional repository. This may be run by IT Services or Library staff. The repository may have different policies and scope of what it will accept: I have publications at both Victoria and Swinburne universities, and each
institution is slightly different. Take the time to include the relevant metadata for each submission, especially the 4-to-6 digit ‘field of research’ codes. Keep the last version of the article you submitted to an academic journal, because due to publisher copyright and intellectual property contracts, often this is the only version that an institutional repository can publish.

6. Archive your ‘primary’ research and develop a stream of publications. Ben probably approached his excellent Meanjinwork with the mindset of a journalist and long-form essayist. He did 20 interviews for one piece. This is more work than goes into many articles for B- and C-level journals, and even for some A-level ones. He could easily reuse and revisit this ‘primary’ research, for the next three or four years in academia. For example, a paper that reviews current frameworks to identify a knowledge gap or research problem, could then lead to a methodology paper, then to comparative case studies, and then to an evaluation or meta-analysis study.