Academic Moneyball

Inside Higher Ed reports that a group of Massachusetts Institute of Technology academics have created a Moneyball style data analytics model to predict and evaluate tenure track applications and performance.

Here are some of my thoughts on Academic Moneyball from December 2014:

The Moneyball Thesis

 

Oakland A’s coach Billy Beane faced a dilemma: competing teams poached his high-performance players and the baseball team faced budget restraints. Beane used Bill James’ sabermetrics – the statistical analysis of baseball player skill – to build a team of contrarian baseball players using performance-based optionality; that experienced positive asymmetric payoffs (a winning streak of successful games); and that achieved a dramatic change in competitive tournament rankings.

 

Definition of Academic Moneyball: develop a Moneyball valuation framework for academic research programs.

 

Definitions

 

Academic: a higher education staff member who does teaching, research, and academic service.

 

Researcher: a higher education staff member whose main work activity Is to conduct research.

 

Research Program: a programmatic suite of research activities including collaboration, dissemination, publishing, and contract research.

 

Structure

 

  1. The University’s Enterprise Bargaining Agreement defines the Minimum Standards for Academic Levels (MSALs): the expected scope and observable impact of an Academic’s teaching, research, and academic service activities (i.e. MSALs establish the baseline for expected performance).

 

  1. An Academic’s work can be modelled as a Present Value cashflow stream of value creation activities from: (i) undergraduate and postgraduate student fees for the Academic’s units of study; (ii) Higher Education Research Data Collection (HERDC) funding, and Research Support and Research Training program funding, received from the Australian Government for the Academic’s eligible research publications, competitive grants, Higher Degree by Research student completions, and other eligible research income (Category 1 to 4); (iii) consulting and contract research income that is non-HERDC eligible; (iv) intellectual property rights income such as patents, joint ventures, and spin-outs; and (v) Academic contribution to Field of Research scores in national and international performance rankings, such as the Excellence for Research in Australia and Shanghai Jiao Tong ranking exercises.

 

  1. The gap between the MSALs and the Academic’s Present Value cashflow stream creates a potential spread in terms of work expectations versus value creation activities that create arbitrage conditions. The University can extract Alpha (excess return above a benchmark) through active management of an Academic’s research program, such as gaining new contract research income, exercising options on intellectual property rights, or creating new, scalable research enterprises. Alternatively, the University can extract Beta (passive returns from a market index) through cultivating an Academic cohort / group’s expertise, and matching it to relevant grant and consulting income sources.

 

  1. Developing an Academic’s research program is one way to change the Future Value cashflow stream from research activities, in a structured process.

 

  1. Moneyball strategies for research programs include: (i) scalability and international scope; (ii) target journals for publication; (iii) book publisher selection for publication; (iv) research methodology training (as a Doctoral candidate and as a Post-Doc); (v) co-authors or collaborative research team; (vi) international conference attendance for collaborative network and referral building; (vii) matching to grant and consulting income opportunities; and (viii) research enterprise development.

Academic Inequality in University Hiring

Slate‘s Joel Warner and University of Colorado’s Aaron Clauset have detailed a study by Clauset and his colleagues on academic inequality in university hiring.

 

Clauset and his colleagues found:

The data revealed that just a quarter of all universities account for 71 to 86 percent of all tenure-track faculty in the U.S. and Canada in these three fields. Just 18 elite universities produce half of all computer science professors, 16 schools produce half of all business professors, and eight schools account for half of all history professors.

 

Clauset’s findings echo what David D. Perlmutter (Promotion and Tenure Confidential) and Karen Kelsky (The Professor Is In) have written about the tenure track process in United States universities. What matters is doctoral degree prestige, having a fundable research program, productivity in research publications, and making a definable impact on a field or domain of knowledge, that is validated through academic networks.

 

These findings mirror what I have seen in research management:

 

  1. The doctoral experience is important to a subsequent research career. It provides an opportunity for the doctoral candidate to immerse themselves in a field or domain, to read its best journals, to be mentored by established researchers, to present research findings at conferences, to master research methods, and to develop a personal voice as a professional author.
  2. Having a research program that is focused and fundable is important to success in competitive grant schemes, and to attracting contract research income. The research program needs to address a significant research or social problem. You need to be aware of the four or five other research teams in the world that are exploring the same research or social problem. You need to address current scholarly debates. The research program should develop expertise that can be shared with government policymakers and industry partners.
  3. Knowledge of journals gained in doctoral education, and a focused research program, can support productivity in research publications. A key is to get a stream of publications that advance the research or social problem identified in the research program. Being strategic about co-authored and collaborative publications is also important. Your publications track record should have a narrative to it that implements your research program. It should systematically build your expertise at national and international levels.
  4. The sum of the first three activities is that, over time, you have a definable impact on a field or domain. Universities look for this definable impact in the minimum standards for academic levels criteria used in promotions and tenure applications. This definable impact can be measured in several ways. Academic publisher prestige can be important for books. National and international conferences can be used to build networks and to workshop material that will later be published in a book or refereed journal article. Social media outreach and the careful use of sites like Academia.edu, ResearchGate and Google Scholar can increase citations. Academic networks can validate the definable impact.

 

Clauset’s study findings support earlier research: Lazear and Rosen’s work on labour tournaments, and Robert K. Merton’s work on the Matthew Effect or success to the successful. Doctoral candidates and Post-Docs who follow the above advice will improve the probability of career and research funding success.

Analytical Investments

In his essay ‘The Only Game In Town‘ the investment manager Jack Treynor identified three kinds of traders: naive, liquidity market-makers, and informed.

 

One way to understand Treynor’s distinction is to examine the different analytical investments that each trader makes. Naive traders invest in public information and simple models of how markets work like popular technical analysis indicators. Liquidity market-makers rely on market microstructure to monitor order flow. Informed traders have more sophisticated models of how markets work and may develop proprietary information sources.

 

Treynor’s distinction also applies to how effective research programs are developed. Effective research programs build on knowledge gaps in publicly available information and in scholarly communities and networks. To develop these capabilities a researcher must be aware of relevant signals from journal publications, other research teams, and the priorities of funding agencies. Collecting and evaluating this information – and noting the knowledge gaps – enables researchers to move closer to Treynor’s ideal of informed traders who have proprietary knowledge and skills. Making the necessary analytical investments as part of a research program – from methodology to cultivating new information sources – is key.

31st January 2012: Public Policy Stars

From an email to Harvard’s Stephen M. Walt in reply to a blog post on public policy stars:

 

1.     Scholars like Anthony Giddens, John Lewis Gaddis and Clayton M. Christensen had a research program with attention to research design/methods. Gaddis’s 1968 PhD, Strategies of Containment (1982) and We Now Know (1997) were hallmarks of Cold War strategy, historical methods, and multi-archival research. Christensen’s Innovator’s Dilemma (1997) based on his PhD was more robust than other dotcom era theories on market change. All of these authors could transition their research program into commercial publishing which then had an impact on broader audiences. Publisher selection was important — such as in the HarperBusiness’  branding of Jim Collins’ management work. Paul Krugman, Robert Putnam, Joseph Stiglitz, Charles Perrow and Herbert Simon also fit this mould. Robert K. Merton, Aaron Wildavsky, James March and Anthony Downs continue to have an impact in emerging sub-fields such as strategic foresight and sustainability.

2.       Each of the scholars mentioned above had a supportive Faculty. This created a ‘success to the successful’ dynamic which means we remember the contribution of these scholars over others. It meant they could combine  basic research in academia, and more applied research in consulting,  publishing, and spin-out ventures. For instance, Gaddis is well-known for  Yale’s grand strategy program, whilst Christensen has the consulting firm  Innosight and a network of collaborative co-authors on recent books.

3.       Media involvement (Krugman and Stiglitz in ‘quality’ media  outlets); collaborative research projects (Gaddis and the Cold War  International History Project); and event-based serendipity (the Cold War’s  end and Francis Fukuyama’s ‘end of history’ thesis) can all act as  amplifiers of their ideas.

27th January 2012: Handling Article Rejection

I recently got negative reviews for two articles submitted to the Journal of Futures Studies (JFS). Many academics I know find article rejection to be highly stressful. Below are some comments and strategies addressed to three different audiences: academic authors; reviewers; and university administrators. Attention to them may improve the probability that your article is accepted for publication in an academic journal.

 

Academic Authors

 

1. Be very familiar with your ‘target’ journal: its editors and review panel, its preferred research design and methodologies, and how it handles controversies and debates in your field. Look for an editorial or scoping statement that explains what kinds of articles the journal will not accept.

 

2. Before submission do a final edit of your article. Define all key terms or cite past definitions if you have referred to the scholarly literature. Check paragraph structure, connecting sentences, section headings, and that the conclusions answer the key questions you have raised in the beginning. Cite some articles from the target journal if possible. Consider who is likely to review your article and factor this into your discussion of key debates. Use redrafting for honing the article and for self-diagnosis of mental models.

 

3. Ask if the journal has a rejoinder process for authors to reply to the blind peer review comments. A rejoinder is not an invitation to personal attacks or to engage in flame-wars. Rejoinders do enable authors to address situations in which one or more reviewers misunderstand the article, frame their comments in terms of an article they wish the author had written (rather than the actual article), or where there are concerns about the methodologies used, the research design, or data interpretation. An effective rejoinder process respects all parties, maintains the confidentiality of the blind peer review process, and provides an organisational learning loop. A rejoinder response does not necessarily reverse an editorial decision not to publish.

 

4. If the journal does have a rejoinder process then carefully examine the feedback pattern from reviewers. Highlight where one reviewer answers the concerns that another reviewer raised: this should neutralise the negative comments or at least show that varied opinions exist. It is more difficult when several reviewers raise the same concerns about an article.

 

5. Set a threshold limit on the amount of editing and rewrites you will do: you have other opportunities. A rejected article might fit better with another journal; with a substantial rewrite; with a different research design; or could be the stepping stone to a more substantive article. Individual reviews also reflect the particular reviewer and their mental models: this can sometimes be like an anthropological encounter between different groups who misunderstand each-other. Sometimes reviewers like critics just get it wrong: one of my most highly cited publications with international impact was dropped from the blind peer review stream.

 

Reviewers

 

1. Use the ‘track changes’ and ‘comment’ function of your word processor to provide comments. It can be difficult for authors to read comments that you provide in the body text and that is written in the same font. Be time-responsive: authors hate waiting months for feedback.

 

2. Do a first read of the article without preconceptions: focus on the author’s state intent, their narrative arc, the data or evidence, and their conclusions. Be open to the article you have been asked to review, rather than the article that you wish the author had written. Be open to innovation in data collection, methodologies, and interpretation. Even do a self-review of your own comments before you send your feedback to the journal editors.

 

3. Know your own mental models. That is, how you see the field or discipline that you are reviewing in; your preference for specific methodologies and research designs; your stance on specific controversies and debates; and what kind of material you expect the journal to publish. Be aware of situations in which you are asked to review articles because you have a particular stance: the tendency is to write lukewarm reviews which focus on perceived deficiencies or ‘overlooked’ material. Be careful of wanting to ‘police’ the field’s boundaries.

 

4. Use your feedback as a developmental opportunity for the author. Don’t just give negative feedback, faulty sentence construction or grammar. If you don’t like something then explain why so that the author can understand your frame of reference. Focus also issues of research design, methodologies, and data interpretation. If there are other external standards or alternative perspectives (such as on a controversy or debate) then mention them. Articles often combine several potential articles or can have scope problems so note them. Highlight sections where the author makes an original, scholarly contribution, including new insights or where you learned something. It’s important to provide developmental feedback even when you reject an article for publication. A developmental review may evoke in authors the ‘moment of insight’ that occurs in effective therapy. The mystique of the blind peer review process ultimately comes down to the reviewer’s attention to the craft of providing constructive yet critical feedback that sets up future opportunities for the academic to advance their career.

 

5. Poison pen reviews have consequences. This is clearer in creative industries like film and music where bad reviews can kill a project or career. Pauline Kael and Lester Bangs are honoured in film and music circles respectively because they brought sensitivity and style to their reviews, even when they hated an artist. In academia, the blind peer review process can lead to internecine wars over different methodologies or research designs: problems that don’t usually arise in open publishing (because all parties know who is making the comments) or that can be handled through editorial review standards and a rejoinder process. Nevertheless, a negative review will have consequences. The author may not revise the article for publication. They may publish in a different journal. They may drop the project. In some cases, they may leave the field altogether. Consider how to frame the review so that you address the developmental need in a constructive manner.

 

University Administrators

 

1. Know the norms, research designs and methodologies, leading research teams, and the most influential and international journals in at least one discipline. This gives you a framework to make constructive inferences from. You will develop awareness of these factors in other disciplines through your interviews with different academics.

 

2. Understand the arc or life-span of academic careers: the needs of an early career researcher and the professor will differ, and this will influence which journals they seek to publish in. Every successful publication navigates a series of decisions. Know some relevant books and other resources that you can refer interested academics to.

 

3. Have some awareness of international publishing trends which affect journals and their editorial decisions. These include the debate about open publishing, the consolidation of publishing firms, and the different editorial roles in a journal. Be aware of the connection between some journals and either professional associations or specific university programs.

 

4. Know what to look for in publication track records. These include patterns in targeting specific journals; attending conferences; building networks in the academic’s discipline; and shifts in research programs. An academic may have a small number of accepted articles when compared with the number that have been written and rejected by specific journals. Use the publication track record as the basis for a constructive discussion with the individual academic, honoring their experience and resources, and using solution-oriented therapeutic strategies.

 

5. Understand that quality publications require time which equates to university investment in the academic’s career. The journal letter rankings in the Australian Research Council’s Excellence for Research in Australia led some university administrators to advise academics only to publish in A* and A-level journals. But not everyone will realistically achieve this. There can be variability of effort required: one A-level article I co-wrote required a substantive second draft; another took months to discuss, a day to do the first draft, and it was then accepted with minor changes. On the other hand, articles accepted in the A* journal International Security (MIT) have usually gone through multiple rounds of blind peer review, the authors are deeply familiar with the field’s literature, and have work-shopped the article extensively with colleagues, in graduate school seminars, and at international conferences. This takes a median two to five years to occur. The late Terry Deibel took almost 20 years to conceptualise and refine the national security frameworks he taught at the United States National War College for Foreign Affairs Strategy: Logic for American Statecraft (Cambridge: Cambridge University Press, 2007) and Deibel also spent two years of sabbatical — in 1993 and 2005-06 — to write it. John Lewis Gaddis spent 30 years of research on George F. Kennan: An American Life (New York: The Penguin Press, 2011) and five years to write it. Both books make substantive scholarly contributions to their fields; both books also required the National War College and Yale University to make significant financial investments in the authors’ careers. Are you making decisions based on short-term, volume-driven models or helping to create the enabling conditions that will help academics to have a similar impact in their respective fields?