Back-testing is not the final word in investing. But too many investors shrug it off as a useless tool often based on the advice of some financial advisors who don’t use it. Some people feel that you cannot quantify their ‘genius’ into a rule-based system.
While it is true that qualitative factors need to be considered, much of what is said about back-testing is spread by those who have never used it or who do not use a clear stock picking strategy. Consider some of the arguments against back-testing and the counter arguments:
- Complaint 1: You don’t know what sector, industry, or style will next perform best. Picking what worked previously will keep you one cycle behind. Looking in the rear-view mirror will have you picking stocks like Google (GOOG) and Apple (AAPL) after large price run-ups.
Counter-argument 1: That is not how back-testing works with a good practitioner. Back-testing can be created using forward-looking data such as forecast earnings. You can select rising industry groups or even go contrarian and against the trend. Some think it means buying into high-tech stocks because high-tech had a good run-up lately - which is a bad practice unless you are a momentum investor. Think of this as what sort of stock picking strategy outperforms and not which group of stocks has historically risen the most. Of course, the back-testing is only as good as the strategy it is built on.
- Complaint 2: My stock picking doesn’t necessarily rely on any one rule. I look for a variety of factors and pick stocks that have the most positive aspects. Using a rule-based system would remove a good stock because it didn’t meet one criteria while being decent in so many other ways. Ford (F) and GM (GM) have some issues but have strength in other areas.
Counter-argument 2: Use a back-testing service (such as Portfolio123) that uses a ranking system. You can use a variety of rules and filters for a comprehensive scoring system. Then you can pick from the companies that score a minimum of 80 or 90 out of a possible 100. This will ensure that you are getting stocks that meet many of your criteria while not excluding companies that don't meet one or two strict requirements.
- Complaint 3: You can’t quantify my stock picking system.
Counter-argument 3: Maybe the problem lies in not having a clearly defined system. While all aspects cannot be put in a rule-based or ranking system, much of it should be possible. Often this is a result of an investor not digging deep enough and believing that his system is based on intuition. Often, the investor is not challenging himself enough. Why did you pick this stock to buy? If you cannot come up with the reasons - then why are you buying it? If you can come up with the reasons, write them down. Try to make them into a absolute or relative rule if possible. Do you only buy this type of stock under certain conditions? Great! Now you are digging deeper. Write down these conditions. Do you have separate buying and selling rules? When do you hedge and how much? Keep writing.
- Complaint 4: Curve-fitting is a problem. You already know the outcome so you simply create rules around it that will never be repeated again.
Counter-argument 4: Good point and a valid argument. Here are a few tips to help you lower the odds of blatant curve-fitting.
First, you should come up with a theory or strategy before blindly running various rules through the back-testing platform. Have a solid reason why you think this strategy is good. This is your biggest defense against curve-fitting the data. Second, test and optimize in a limited set of data – perhaps a 5-year window (but not the most recent). Keep it simple. The more complex the system is the more likely you are to be cherry picking stocks by curve fitting. Finished? Use your strategy in a different 5-year period (possibly the most recent). Did it hold up with this 'walk forward' strategy? Also use a robustness check that will use random entry points with your strategy to get a feel if the strategy is sound under many conditions or if you have a 'big gain system' because you picked an entry point of March 17, 2009.
- Complaint 5: Back-testing programs suffer from survivorship bias as delisted stocks are removed from the database. In the real world we buy stocks that are removed from exchanges. Also, the program might allow access to data that was not available at the time of purchase, such as a quarterly report.
Counter-argument 5: Institutional grade software generally does not suffer from these problems. Back-testing software aimed at the smaller investor needs to be checked carefully for these issues. I won’t name names, but of the software I tested - one fails to meet the above requirements while the other does not suffer from any of these problems.
A Few Other Considerations
A few things your back-testing software should be able to do:
- Buy a portion of daily liquidity. (This ensures that you are not loading up on illiquid stocks that would be impossible in practice.)
- Separate buy and sell rules. (While you may be strict when buying new stocks, you might sell them based on other criteria such as deterioration of market conditions or sliding fundamentals.)
- Hedging rules. (Many investors hedge instead of selling their positions. Others balance between bonds and stocks. A good program should allow for this.)
- Robustness checking. (You should test your strategy over hundreds of random entry points to ensure consistent out-performance of market.)
- Ranking system. (The new era of back-testing software allows for picking stocks that score high on a variety of criteria. This can be used as a stand-alone screening tool or in conjunction with absolute rules.)
- Incorporation of slippage and trading fees. Preferred position sizing
- No survivorship or look-ahead bias
- Ability to build strategies and back-test both ETFs and stocks
- Ability to program rules that switch investing styles based on market conditions
- Low-cost - as in $50 per month or less
Example of a Back-test
One high-growth strategy I use looks for stocks with accelerating fundamentals and strong price performance for an entry. This is a basic premise of how it works:
- Price, Size, and Liquidity: Bigger than volatile small caps. Strict limits on how much of daily turnover I can buy. Needs to have at least 100,000 shares traded daily.
- Relative Ranking Rules: Three ranking systems are used to pick these stocks. It must rank highly on CAN SLIM style stocks, Fiscal Momentum, and good old price Momentum.
- Absolute Filters: Quarterly and annual earnings above 0. Pick from stocks with more volatility than market but with some stability between quarterly earnings.
- Market Filters: Only buy when market timing rules indicate a probable bull trend.
- Hedging: When market rules show a probable downtrend I hedge by shorting a triple leveraged small cap ETF with 20% of my worth.
Other rules include $7 per trade and 0.3% slippage. My current holdings are Woodward (WWD), Starbucks (SBUX), MercadoLibre(MELI), Lincoln Electric Holdings (LECO), and Intuitive Surgical (ISRG). Below is an example of performance over the past 5 years and all sorts of risk and trading statistics are available that are not posted here.
If you have any other concerns about back-testing that are not addressed here, please feel free to leave a comment.
Disclosure: I have no positions in any stocks mentioned, and no plans to initiate any positions within the next 72 hours.