Analyzing Trading Systems: Too Good to Be True?

Readers do not need us to tell them the old common-sense idea, "If it sounds too good to be true..." Or so we would think.

Meanwhile, those watching financial television are bombarded with self-serving ads from brokerage firms suggesting that the intelligent viewer can design his/her own trading system with their free software.

Other ads offer a pre-packaged approach, like Robert Taylor and the Xyber9 system. We can't find the specific claims on the website, but the TV ads talk about accuracy in the 80's. It is a great marketing program, with references to a Nobel Prize nomination and the Harvard Business Review.

So we wonder....

Analyzing Trading Systems

A standard method for analyzing a system includes some back testing. Everyone does it, so the question is not "whether?" but "how?".

When we were developing and analyzing trading systems in the late 80's, there was an interesting phenomenon.

Every system we saw "predicted" the Crash of '87.

We should say "post-dicted." Most of us now understand why, since we have all read Fooled by Randomness ( on our recommended reading list if you missed it). If a system developer did not forecast the Crash, that system would never have seen the light of day. It certainly was not presented to trading firms like ours.

The founder of our group, who assembled a team of top options traders at the Chicago Board Options Exchange [CBOE], was Ralph Katz. In the days before computer-generated option prices, Ralph could glance at a complicated page of quotes and tell you which specific prices were out of line. Ralph understood immediately why the system development process might be wrong. He would insist that the system developer go to a different time period, (e.g., Did the system capture the Gulf War Decline?), one not contemplated by the original developer.

Ralph's approach is now used by the best system testers.

Analyzing Systems

When you get a system that seems "too good", it is time to take a close look. CXO Advisory Group, one of our featured sites, does a great job of determining "guru grades." Taylor's idea of using gravitational pull to predict stock performance was intellectually unsatisfying for us, so we asked CXO for an opinion. They identified the strange definition of "trend" used by Taylor and concluded as follows:

In summary, Robert Taylor's accuracy rate probably derives not from forecasting ability but from defining targets that are very hard to miss. The accuracy rate seems high only if one ignores the peculiar way he defines trends.

Consumers Do Not Understand

The key to building a successful investment management business is marketing, and that seems to include making big claims. The consumer is looking for a home run. This usually means looking at what worked last year, and chasing that performance.

In a very helpful and honest article, Barry Ritholtz described the process of presenting his methods on a road show. (Thanks to Barry for taking the time to send the pointer for an article we remembered.) Barry wrote as follows:

Money raisers and some GPs have long ago figured this out. You have a few choices: you can answer the investors' questions honestly -- or to quote Ray Davies, you can give the people what they want (or think they want):

"We expect gains of 35-45%, with minimal risk or leverage. Our black box algorithms have been backtested, and generate better numbers than that, but we would rather under-promise and outperform."

Read the entire article for the full flavor of this process. Based upon our own experience, it is a very realistic account. This method of choosing managers has led many to illusory low-risk returns based upon very high leverage and minimal true advantage (alpha).

If the big fund of fund managers are using this approach, imagine the challenge for the individual investor.

A Dash of Realism

The best investment managers, those that we admire greatly like Warren Buffett and Bill Miller, have periods where their style does not seem to be working. That says more about the market than it does about the manager.

Any investor who wants to trade equities with little turnover (meaning low transaction costs) needs to understand what results qualify as strong. Beating the market by several points a year is excellent, and it probably comes with some volatility.

Comments

[ponchovilla]

Jeff:
Interesting article but like comparing apples to oranges in juice concentrate fresh organic etc. David Nassar is an options trader. Warren Buffet is an investor. Looking at the guru chart what does "essentially right" or "essentially wrong" mean? Flip a penny 100 times each day for a thousand days and you will be essentially correct 50% of the time. How much do I bet on the outcome?

"We expect gains of 35-45%, with minimal risk or leverage. Our black box algorithms have been backtested, and generate better numbers than that, but we would rather under-promise and outperform."

The mathematics of algorithms are very complex. PHD math and physics (applied math). Back testing should be very accurate with some "smoothing". Predictions of future results are probability.That's the realm of adaptive predictive algorithms How much do I bet on a guru with 13% probability compared to a guru with 68% probability. Then take $10,000, one time or every month, look at transaction costs, dividends and tax effects etc and what are the resullts over 1 year, 10 years? Not an easy answer, eh? How much would that study cost? Warren Buffet or David Nassar?