Microstructure (AssetPriceChange)

Tonight, I started programming a knowledge base in the Prolog computer language used in artificial intelligence. This will also form part of how I map out the underlying logics of two PhD case studies on strategic culture before I get to specific research methods (process tracing for one case study and thematic coding analysis for the other). Whilst coding I read Cullen Roche’s post on why stockmarket speculation is neither investment nor usually wealth generation:


Now, this doesn’t mean it’s impossible to become wealthy picking stocks on a secondary market.  There will always be people who “beat the market” and ride stocks to riches.  But that’s not the point of this story.  The point is, in the aggregate, the market returns the market return and the market return isn’t likely to make investors rich quickly in the aggregate.  In fact, you’re almost certainly buying an asset that has already made someone else rich well before you ever had the opportunity to own a claim on that asset’s cash flows.  This doesn’t mean that buying stocks and bonds is bad.  It doesn’t even mean you can’t “beat the market”, but we should be careful about the concept of “investing” and how it actually leads to us becoming wealthy.  You’re much more likely to become wealthy investing in your own ability to generate future production than you are by buying an asset that was actually someone else’s investment. [emphasis added]


Buried in Roche’s analysis is a comparison of cowboy intraday traders / momentum traders / trend-followers and Jack Bogle style index investment. This is like a battle between a discredited, naive form of active management and low-fee passive management that disrupts mutual funds. It’s a narrative that turns up in a lot of introductory books for retail investors. 

But there are at least two other options found more in funds management: (1) using arbitrage to find or to create market inefficiencies; and (2) trading the asset price changes that occur in market microstructure due to the order flow (and transaction / execution algorithms) of different market participants (such as proprietary trading desks; hedge funds; sovereign wealth funds; company directors; and noise traders). 

The first is a goal of activist hedge funds who use behavioural finance to create situations of crowded trades and rational herding; the second informs quantitative trading techniques like machine learning and statistical arbitrage. 

The first leaks to the media; the second uses news and text analytics to anticipate microstructure changes in asset prices. I spent a decade studying the first in media analysis, publishing and research. Now, I am interested in the quantitative insights of the second. This feels like a self-disruption of past work. 

System designers combine both options mentioned above in funds management: the first uses human intelligence; the second relies on computational intelligence. 

It will take a few years to complete the Prolog knowledge base project. 

When I’m done, I will have decomposed the relevant literature into white box models (Bayesian decision rules). Then, I will start on the black box version with live data. I have access to live data now – but I need more of a background in algorithms, machine learning, stream processing, and high-frequency econometrics – to really begin work on the black box version. 

The foundations are in place.

Thematic Analysis of a Reading List on Investment Alpha

I recently did a thematic analysis of a reading list on investment alpha, which involves:


1. Excess return.

2. Active management.

3. Adjusted risk.


The following themes emerged from the reading list, and from also checking the rankings of several hundred books at Amazon.com:


1. Excess return: fund type (hedge fund, private equity, venture capital); return drivers (including asset class); and quantitative models.


2. Active management: discretionary (human trading, portfolio composition and rebalancing, options, technical analysis) and algorithmic (algorithmic trading; complex event / stream processing; computational intelligence; genetic algorithms; machine learning; neural nets; and software agents).


3. Adjusted risk: Bayesian probabilities; investor psychology; market microstructure; and risk management models (such as Monte Carlo simulation, Value at Risk, and systematic risk)


This core work suggests the following query line:


SELECT return drivers (Bayesian belief network) (multi-asset) (portfolio) (fund)



WHERE risk (Bayesian probability) (exposures) (exposures – investor decisions) (exposures – market microstructure) AND trade (algorithms)


ORDER BY Bayesian (belief network, probability); return drivers (multi-asset); risk (exposures); and trading (algorithms).


This thematic analysis will help to focus my post-PhD research on the sociology of finance into the following initial research questions:


1. What is the spectrum of possible return drivers in a multi-asset world?


A good model for this is David Swensen’s Yale endowment portfolio detailed in Pioneering Portfolio Management: An Unconventional Approach to Institutional Investment (New York: The Free Press, 2009). Antti Ilmanen’s magisterial Expected Returns: An Investor’s Guide to Harvesting Market Rewards (Hoboken, NJ: John Wiley & Sons, 2011) has information on the return drivers of specific asset classes. Matthew Hudson’s recent Funds: Private Equity, Hedge Funds, and All Core Structures (Hoboken, NJ: John Wiley & Sons, 2014) deals with global fund structures.


2. What specific risk exposures might these multi-assets face, and under what conditions?


Richard C. Grinold and Ronald Kahn’s Active Portfolio Management: A Quantitative Approach for Producing Superior Returns and Controlling Risk (New York: McGraw-Hill, 1999) is the classic book on institutional portfolio models. Morton Glantz and Robert Kissell’s Multi-Asset Risk Modeling: Techniques for a Global Economy in an Electronic and Algorithmic Trading Era (San Diego, CA: Academic Press, 2014) is a recent book I will look at. Charles Albert-Lehalle and Sophie Larulle’s Market Microstructure in Practice (Singapore: World Scientific Publishing Company, 2014), and Thierry Foucault, Marco Pagano, and Ailsa Roell’s Market Liquidity: Theory, Evidence, and Policy (New York: Oxford University Press, 2013) deal respectively with the practice and theory of contemporary financial markets. There are many books on behavioural finance and investor psychology: two recent ones are H. Kent Baker and Victor Ricciardi’s collection Investor Behavior: The Psychology of Financial Planning and Investing (Hoboken, NJ: John Wiley & Sons, 2014), and Tim Richards’ Investing Psychology: The Effects of Behavioral Finance on Investment Choice and Bias (Hoboken, NJ: John Wiley & Sons, 2014).


3. How can algorithmic trading and computational techniques model the risk-return dynamics of alpha generation?


Despite its flaws Rishi K. Narang’s Inside the Black Box: A Simple Guide to Quantitative and High Frequency Trading (New York: John Wiley & Sons, 2013) opened my eyes to the structures needed for alpha generation. The Bayesian approach is detailed in David Barber’s Bayesian Reasoning and Machine Learning (New York: Cambridge University Press, 2012). Barry Johnson’s Algorithmic Trading and DMA: An Introduction to Direct Access Trading Strategies (London: 4Myeloma Press, 2010) and Robert Kissell’s The Science of Algorithmic Trading and Portfolio Management (San Diego, CA: Academic Press, 2013) deal with order types in algorithmic trading. Christian Dunis, Spiros Likothanassis, Andreas Karathanasopoulos, Georgios Sermpinis, and Konstantinos Theofilatos have edited a recent collection on Computational Intelligence Techniques for Trading and Investment (New York: Routledge, 2014). Eugene A. Durenard’s Professional Automated Trading: Theory and Practice (New York: John Wiley & Sons, 2013) covers software agents. For retail trader-oriented applications of data mining, machine learning, and Monte Carlo simulations there is Kevin Davey’s Building Algorithmic Trading Systems: A Trader’s Journey from Data Mining to Monte Carlo Simulation to Live Trading (New York: John Wiley & Sons, 2014), and David Aronson and Timothy Masters’ Statistically Sound Machine Learning for Algorithmic Trading of Financial Instruments: Developing Predictive-Model-Based Trading Systems Using TSSB (CreateSpace, 2013).


What this means is that for an investment of about $US1,000 a new researcher can gain some of the core books on institutional, quantitative portfolio and risk management; behavioural finance and market microstructure as potential sources for edges; and some recent practitioner-oriented literature on algorithmic / automated trading that uses computational intelligence.


In deference to Mao and McKenzie Wark’s vectoralist class:


Let a thousand algorithmic / quantitative micro-funds bloom.

The Toronto-Dominion Centre Working

2:30pm – 3:30pm, 30th March 2014

Toronto-Dominion Centre and Bay St financial district, Toronto, Canada


Preparation material: Francis James Chan’s The Prop Trader’s Chronicles: Short-Term Proprietary Trading Strategies for Both Bull and Bear Markets (Hoboken, NJ: John Wiley & Sons, 2013).




(i) Understand the geography of Toronto’s financial district.

(ii) Make a psychological connection to Toronto’s bank prop traders.




Chan’s book on intraday trading at a Toronto-based proprietary trading firm alludes to inter-firm competition amongst Bay St trading firms. On arrival in Bay St it became clear that Canada’s five major banks — Bank of Montreal, Scotiabank, the Canadian Imperial Bank of Commerce, the Toronto-Dominion Bank, and the Royal Bank of Canada — dominate the area.


The dominance of bank proprietary trading desks explains several aspects that Chan had omitted from his description of intraday trading strategies. Chan and others relied on contracts for difference without overnight holdings. They attempted to understand the order flow of market microstructure using Level II quotes from the NYSE and NASDAQ exchanges rather than technical analysis charts. In game theory terms this was Chan’s attempt to use the best available dominated strategies in a predator-prey ecosystem that the banks dominated.


The Toronto-Dominion Centre evokes this institutional banking power in Ludwig van der Rohe’s modernist, international architecture. The TD Bank Pavilion, TD North, and TD West buildings impose themselves on the surrounding area. Their tenants include banking, financial services, investment banking, investment brokerage, and private equity firms.


On 11th October 2011, I had visited the Tokyo Stock Exchange and formally began a personal research program “to develop a private, low-key, personal vehicle for long-term self-sufficiency.” The Toronto Stock Exchange was closed so I was unable to repeat the experience. Instead, I stood in the TD Bank Pavilion and grasped the essence of institutional banking power evoked in Adam Smith’s satirical book The Money Game (London: Michael Joseph, 1968).


Later that afternoon I visited the Toronto Eaton Centre and the Indigo Books & Music store. Indigo’s business and investment book section was a mix of inspirational biographies; retail investor primers; and technical analysis books. Much of this is outdated information from an institutional banking perspective which relies on non-public trade secrets. I bought a paperback copy of Nassim Nicholas Taleb’s Antifragile: Things That Gain From Disorder (New York: Penguin, 2012) as a reminder of the tacit knowledge that a trader may create through personal experience, research, and reflection.


The next day I read the new Michael Lewis book Flash Boys: A Wall Street Revolt (New York: W.W. Norton & Co., 2014) which features former Royal Bank of Canada trader Brad Katsuyama – founder of the IEX Group dark pool – and critic of high-frequency trading. Lewis describes RBC as a sleepy backwater compared to Wall Street but this wasn’t my sense when walking past the RBC Centre in Wellington Street West, Toronto.


Several days later I learned of a new University of Toronto study (PDF) on how retail traders and high-frequency traders interacted on the Toronto Stock Exchange in 2012. The study felt like a research counterpoint to the Lewis book. The study found that retail investors largely benefited from the market microstructure of high-frequency trading firms.


I resolved to do two things over the next five years:


1. To develop a greater awareness of how bank proprietary trading desks affect market microstructure using dominant trading strategies in a predator-prey ecosystem.


2. To continue to develop a personal knowledge base and decision heuristics akin to Nassim Nicholas Taleb’s published work.

Norway’s Sovereign Wealth Fund on HFT

Oyvind G. Schanke (New York Times)
Oyvind G. Schanke (New York Times)

Norway’s sovereign wealth fund Norges Bank Investment Management has released a new report about high-frequency trading (HFT). NBIM’s report finds that HFT firms front-run the large orders of asset management firms; that there is “transient liquidity” due to cancelled quotes; and that exchanges benefit “low latency ultra HFT strategies.” NBIM trader Oyvind G. Schanke told The New York Times: “It has become much more a market trading for trading’s sake.”

High-Frequency Trading: A Reading List

I first heard of high-frequency trading (HFT) via Charles Duhigg’s New York Times article in July 2009.


Over the past few years I have investigated facets of HFT. Below is an introductory reading list to HFT and the related area of algorithmic trading, which has recently ‘crossed the chasm’ from institutional to retail investors. It covers an historical overview; some relevant theory; and the use of computer algorithms and machine learning. Large-scale HFT firms spend millions on their computing and technological infrastructure.


The introductory reading list hopefully shows how you can use research skills to Understand a media debate or knowledge domain in greater detail.


This work connects with the Chaos Rules school of thought that I helped write for the Smart Internet 2010 Report (2005). One academic told my boss that Chaos Rules thinking “did not exist.” The two decades long research into HFT — and related areas like Bayesian econometrics and market microstructure — shows otherwise.

HFT Introductions

Dark Pools: The Rise of AI Trading Machines and the Looming Threat to Wall Street by Scott Patterson (Cornerstone Digital, 2012). (TS-3). A history of algorithmic and high-frequency trading on Wall Street, and the emergence of dark pools.

Inside The Black Box: A Simple Guide to Quantitative and High Frequency Trading (2nd ed.) by Rishi K. Narang (John Wiley & Sons, 2012). (TS-3). An introduction to quantitative trading models and coverage of the media debate about high-frequency trading. For a counter-view see Haim Bodek’s The Problem of HFT: Collected Writings on High Frequency Trading and Stock Market Structure Reform (CreateSpace, 2013) (TS-3), who was a source for Patterson’s Dark Pools.

HFT Theory: Bayesian Econometrics, High-Frequency Data, and Machine Learning

Empirical Market Microstructure: The Institutions, Economics, and Econometrics of Securities Trading by Joel Hasbrouck (New York: Oxford University Press, 2007). (TS-4). Hasbrouck explains the empirical approaches to market microstructure that underpin high-frequency trading.

Market Liquidity: Theory, Evidence and Policy by Thierry Foucault, Marco Pagano, and Ailsa Roell (New York: Oxford University Press, 2013). (TS-4). The current debates on how high-frequency trading has affected liquidity and price discovery in markets, and the growth of market microstructure frameworks.

Bayesian Reasoning and Machine Learning by David Barber (New York: Cambridge University Press, 2012). (TS-4). An introduction to Bayesian probability and data analysis using filters and machine learning. For an introduction to machine learning see Peter Flach’s Machine Learning: The Art and Science of Algorithms That Make Sense of Data (New York: Cambridge University Press, 2012) (TS-4).

Econometrics of High-Frequency Data by Nikolaus Hautsch (New York: Springer, 2011). (TS-4). An advanced overview of high-frequency data and relevant econometric models for liquidity, volatility, and market microstructure analysis.

Handbook of Modeling High-Frequency Data in Finance by Frederi G. Viens, Maria C. Mariani, and Ionut Florescu (New York: John Wiley & Sons, 2011). (TS-4). An advanced reference on how to model high-frequency data.

HFT Algorithmic Trading

The Science of Algorithmic Trading and Portfolio Management by Robert Kissell (Academic Press, 2013). (TS-4). An advanced introduction to how algorithmic trading influences market microstructure, and is used for the transaction and execution systems of high-frequency trading. For an earlier introduction see Barry Johnson’s Algorithmic Trading & DMA: An Introduction to Direct Access Trading Strategies (4Myeloma Press, 2010) (TS-4).

Professional Automated Trading: Theory and Practice by Eugene A. Durenard (New York: John Wiley & Sons, 2013). (TS-4). Insights from mathematics and computer science about how to develop, test, and automate the algorithmic trading strategies, using agent-based learning.

Statistically Sound Machine Learning for Algorithmic Trading of Financial Instruments: Developing Predictive-Model-Based Trading Systems Using TSSB by David Aronson and Timothy Masters (CreateSpace, 2013) (TS-4). The authors developed the TSSB software program that uses machine learning to implement algorithmic trading strategies.

6th January 2013: The Failure Test Entry Working

The Failure Test Entry Working

3:30-8:30pm, Saturday 5th January 2013

Melbourne, Australia


Preparation Material: Adam H. GrimesThe Art and Science of Technical Analysis (New York: John Wiley & Sons, 2012); Margery Mayall’s University of Queensland sociological research on technical analysis; BusinessSource database search on academic research into technical analysis, and trader development and learning; and MarketPsych.com behavioural finance and psychological tests.




(i) Identification of trading personal goals for 2013.

(ii) Illustrative understanding of technical analysis as a trading methodology for alpha generation.

(iii) Consideration of learning barriers to trader development.


Technical analysis (TA) is the study of group psychology in financial market using price, sentiment, and volume indicators, and pattern recognition. It arose in a modern context due to Charles H. Dow and Richard Schabacker’s study of market patterns in the late 1800s-early 1900s. Robert D. Edwards and John Magee’s Technical Analysis of Stock Trends became the TA bible of market patterns later promulgated in variations by Martin Pring and others. Richard D. Wyckoff (the Wyckoff Method), Robert Prechter (Elliott wave theory), and other TA theoreticians have made influential contributions. TA focuses on identification of trends, retracements, breakouts, pullbacks, support and resistance. It anticipated some aspects of current academic research programs on behavioural finance and market microstructure but from a trader or practitioner viewpoint.


Academics and traders remain divided on TA’s efficacy. In 1934, Alfred Cowles contended that a ‘buy and hold’ strategy beat Dow Theory trading. Early studies from 1966 to 1970 by Eugene Fama and his University of Chicago colleagues found that TA filter rules were unprofitable once transaction and execution costs were considered. Fama’s finding led academics to focus on the Efficient Markets Hypothesis, and, ultimately, mutual fund and passive index fund products. In contrast, TA became popular in the mid-late 1970s amongst trend-following Commodity Trading Advisors on volatile commodities and foreign exchange markets. The ‘housewives of Tokyo’ who speculated on currency movements now challenged the ‘gnomes of Zurich’ or institutional investment managers. Victor Sperandeo who traded for George Soros used Dow Theory. The bootlegged PBS documentary ‘Trader’ (1987) shows Paul Tudor Jones II and Peter Borish using Elliot wave theory and 1929 price data to predict a stockmarket crash in early-mid 1988. Finance theories in academic journals and hedge fund manager practices diverged into parallel universes.


Recent academic research has shed new light on this academic-practitioner divide. In a review of 95 academic studies on TA from 1960 to 2004, Cheol-Ho Park and Scott H. Irwin found that “56 studies find positive results regarding technical trading strategies” (“What Do We Know About the Profitability of Technical Analysis?, Journal of Economic Studies 21:4 2007, p. 786). They note data snooping problems with Edwards & Magee-style pattern recognition which other academic researchers have also identified. Importantly, Park and Irwin found that TA was profitable in spot foreign exchange and futures contracts “from the late 1970s to the early 1990s” involving “unlevered annual net returns of 2-10%” (Park & Irwin 2007, p. 795). This finding reflects the period when Sperandeo, Jones, Borish, and other non-TA traders like Martin Zweig were ascendant in financial markets. It contradicts the earlier findings of Cowles and Fama that TA has always been unprofitable.


Park and Irwin’s finding about TA’s period of profitability is also mirrored in other post-1988 academic studies. These find that the traders used arbitrage on anomalies; the transmission shocks of central bank monetary policies; the anchoring, crowded exits and rational herding of institutional investors; and changes to the international monetary system and political economy. However, these studies often fail to link their finding to the practitioner literature which offers independent confirmation, such as Jones II’s interview in Sebastian Mallaby’s More Money Than God: Hedge Funds and the Making of a New Elite (London: Bloomsbury Publishing, 2010). TA practitioners like Jones II were also often aware of the speculative bubble literature—Charles Mackay, Gustave Le Bon, Charles P. Kindleberger, John Kenneth Galbraith, and Hyman Minsky—which has inspired contemporary research in behavioural finance. This is why Gordon Gekko’s apartment in Wall Street: Money Never Sleeps (2010) had pictures from the Dutch Tulip bubble (1636-37). The conceptual gap between TA and behavioural finance is perhaps not as large for financial market practitioners as some academic researchers believe.


The decline in TA profitability after the early 1990s can be attributed to changes in central bank policy coordination, market microstructure, and the growth of algorithmic trading. For instance, the Wyckoff Method identifies institutional trading and market patterns also found in Robert Shiller’s study of ‘irrational exuberance’ and speculative bubbles. But the growth of new trading—options, futures, and high-frequency systems—have altered what the Wyckoff Method found in pre-World War II financial markets.  Collectively, the above developments over the past two decades have changed markets and volatility from trending to more range-bound dynamics. Edwards & Magee’s TA indicators, and support and resistance levels, can now be programmed into algorithms that actively trade against institutional and retail traders who still use traditional TA methods. This Darwinian-like evolution has led to the demise of dotcom era day traders (1995-2000), and trend followers who benefited from asset price valuations due to housing and commodities speculative bubbles (2003-2008).


Academic researchers rarely refer to the TA practitioner literature beyond introductory books by Alexander Elder, Van Tharp, and other authors. Academics often state incorrectly that TA remains unstructured as a knowledge domain: Edwards & Magee, the Wyckoff Method, Elliott wave, Fibonacci, Japanese Candlesticks, and other major TA methods and schools each have their exponents and adherents. Instead, TA now involves an industry of books, consultants and custom indicators targeted at the retail investor. University of Queensland sociologist Margery Mayall found that TA indicators shaped the self-beliefs, mindsets, and decisions of the Australian retail traders who she interviewed. Some of Mayall’s retail traders became focused on the never-ending Holy Grail Quest to find the ‘right’ TA indicator or system.


In contrast, proprietary trading desks now combine TA with behavioural finance, game theory, and market microstructure. Professional traders seek what Michael Steinhardt called contrarian ‘variant perception’ in financial markets compared with the ‘consensus perception’ of retail traders. There is always someone else on the other side of the trade even if it is a market-making algorithm. Academic researchers could bridge the gap with TA practitioners if the popular models were evaluated and back-tested in a more rigorous manner. However, recent work by Andrew Lo and other authors on rehabilitating TA remains at the interview or memoir stage, rather than using a robust empirical research design. Recent TA practitioner work by Adam H. Grimes, Xin Xie, Charles D. Kirkpatrick II, Julie R. Dahlquist, L.A. Little, David R. Aronson, and others looks promising. Grimes links TA and trader development to George Leonard’s Aikido model of self-mastery; to Daniel Kahneman’s prospect theory and behavioural finance study of cognitive biases; and to Mihaly Csikzentmihalyi’s study of creativity, flow, and optimal experience. This augments earlier work by the late Ari Kiev, Brett N. Steenbarger, and Mark Douglas on trading and performance psychology.


Since circa 1992, a subset of TA academic research has also used genetic algorithms and high-frequency tick data analysis to identify trading rules. The findings from this research often either remain proprietary or reflect mathematical and quantitative models. Hedge fund managers who use TA are closer to Aaron C. Brown’s Bayesian risk managers who revise and update their beliefs. Such hedge fund managers are often aware of confirmation bias, the disposition effect, overconfidence, model risk, and other cognitive biases identified in the behavioural finance literature. Hedge fund managers and professional traders now use TA in a mixed methods approach – if they have not already been replaced by algorithmic trading systems. Another problem with the genetic algorithms research is that whilst it identifies trading rules it often does not include trader learning, risk and money management practices. These are what Sperandeo, Jones II, Borish and other TA traders use, and thus these practices modify the efficacy of the trading rules identified. For instance, the PBS ‘Trader’ documentary (1987) shows Jones II using deception and rumour – closer to the Chinese 36 Strategies – to mask his order size and to influence other traders. Academic researchers using genetic algorithms and other methods have often overlooked this cunning or metic intelligence.


I resolved in 2013 to integrate TA’s relevant insights into a personal knowledge base and bespoke trading system for alpha generation. Academic research rigour can be combined with professional trading insights whilst retail trading myths promulgated by the TA industry and self-styled trading coaches can be avoided. A mixed methods research approach looks promising: where TA sees trends and retracements – a market microstructure researcher may see the interaction of strategic traders, order flow, and order types – and a behavioural finance proponent may find specific cognitive biases and decision heuristics. All three approaches look at the same market data via different lenses and vantage points. I took several MarketPsych.com tests to identify and to understand personal cognitive biases and psychological preferences. Once identified, I then compared the personal cognitive biases with past trades using an after action review approach. This illustrative research will inform operative action research to improve decision heuristics, mental models, and risk preferences for future alpha generation.

24th December 2011: Journal of Financial Markets: Observations

Last night, I read articles from the 2007-2012 issues of the Journal of Financial Markets (Elsevier). I check-in with this academic research to keep abreast of developments in market microstructure, trading mechanisms, and institutional trading strategies.

Some of the academic research findings I noted:


• Growth managers are momentum traders.

• Style neutral and value managers are contrarian traders.

• Trades with high price impact – medium sized orders, large trading volume, trade early in morning.

• Limited capital – traders tend to use momentum and value strategies.

• Morning losses for day-traders – afternoon selling trades – attempt to meet daily price targets.

• Psychological biases lead to correlated trading of individual investors – buy stocks with strong recent performance (momentum); refrain from selling stocks held for a loss; and net buyers of stocks with highly unusual trading volumes.