Effective Backtesting

Main points:

backtesting is an attempt to learn the future by looking at the past
the connection between past and future is chaotic
if we look at the wealth curve of a backtest, everything we infer about that result could be wrong
mimicking the backtest with random trades provides a true picture of when the strategy works and when it doesn’t
it is useful to do the backtest with multiple starting portfolios
the only truly out-of-sample test is real trading
herd risk is a problem
we played the minority game
optimism bias is a problem

Presented 2010 November at the Thalesians.