Please Note: Blog posts are not selected, edited or screened by Seeking Alpha editors.

The Perils Of Backtesting

Analysis of whether the market is overvalued has become more and more popular. As a general matter, it appears that "back testing" (comparing current ratios with past periods in which such ratios arose) is very popular. We have pricing data going back to 1876 and so it is possible to amass a great deal of market data and perform complex calculations demonstrating the degree to which current ratios deviate from prior patterns. The ratios that are most often analyzed are the price earnings ratio (NYSE:PE) and the Shiller 10 year trailing price earnings ratio (NYSEARCA:CAPE). Analysts and pundits purport to be able to make Sigma and standard deviation calculations with detailed accuracy using this data and contend that they can predict the future course of the market based on this back testing. I have seen articles which employ arcane mathematical tools and purport to reach extraordinarily precise conclusions about valuation and future market action. This post will explore some inherent problems in this kind of analysis.

Earnings - Since we focus on price earnings ratios, the key data points are prices and earnings. The price part of the ratio is pretty straightforward; we have good data on the prices of securities and can calculate index prices in a fairly reliable manner. It is when we come to "earnings" that problems may arise. The concept of earnings is inherently subjective and can be affected by changes in accounting practices. In this regard, we had a major change in market regulation in the 1930's requiring publicly traded companies to provide periodic earnings reports and attaching sanctions to misrepresentation. I think it is widely agreed that this deterred fraud and made earnings more reliable. Put another way, a dollar of post- 1934 reported earnings may well be worth more than a dollar of pre-1934 reported earnings. I am not suggesting that pre-1934 data should be thrown out but only that it may not be statistically sound to accord that data the same relevance as post-1934 data. Put another way, if the companies now in the S&P 500 were playing by the same reporting rules that were in effect in 1876 or even in 1928, it is likely that they would be reporting substantially higher earnings.

There is another important point. Even after the SEC was established and reporting was required, enormously complex issues still surrounded the calculation of earnings. In this regard, the accounting profession - with SEC participation - has changed the rules of the game at numerous junctures since the 1930's. For example, we recently had the requirement that share based compensation be expensed (deducted from earnings). At the time, there were howls of protest alleging that earnings of tech companies would be unfairly depressed. I regard the current rule as reasonable but that is not the point here. Rather the point is that the change probably did depress earnings. Looked at another way, if corporations were required to restate their earnings all the way back to 1876 based on the current rules of the game, those earnings might be quite a bit lower (and historic price earnings ratios would be quite a bit higher).

This hasn't been the only change. The calculation of "deferred earnings", the use of various methods of depreciation and amortization, the decision of when to book income, and the treatment of "off balance sheet" entities are all difficult issues whose treatment has changed materially over the years. In short, measuring earnings is not like measuring a person's height or weight. Of course, it may be that changes have cancelled one another out and that earnings data from, for example, the 1960's is reasonably comparable to current data. My own impression is, however, that the rules have been toughened up so that -even if we limit ourselves to the post-1934 period - current earnings are likely to be worth more than earnings before - for example - the passage of Sarbanes Oxley.

Share Value - A purchaser of stock gets a stock certificate (or its electronic equivalent) which is a piece of paper creating certain rights. While the legal definition of those rights has been reasonably stable since at least 1934, the practical value of those rights has changed. A number of events in the late 1970's and the early 1980's had the effect of enhancing shareholder rights. The key factors were - 1. the creation of a new issue junk bond market which facilitated takeovers, 2. the increasing prevalence of LBO's, and 3. the SEC's promulgation of safe harbor share repurchase rules. These changes led, in turn, to an increase in shareholder activism and the emergence of individuals like Carl Icahn who sought to unlock the value in corporations whose stocks were mispriced. As a value investor, I can tell you that the result brings tears to my eyes. The wonderful, cheap, giveaway stocks that Graham and Dodd (and Buffett) were able to find in the 1950's are no longer available. Stocks that are ridiculously cheap are either taken out in LBOs or repriced through aggressive share repurchase activity. In a real sense, the result is that the shareholder has more traction with respect to his claim on owner cash flow. Again, the effect is that pre-1980 data may have to be adjusted if it is to reflect post-1980 realities.

Corporate Earnings as a Percent of GDP - Another claim we often see is that corporate earnings (currently around 11 percent of GDP) are at historically high levels and therefore must decline. In this regard, it is important to recognize that we are talking about "after tax" corporate earnings. If we analyze corporate taxes as a percentage of GSP, we see a fairly consistent decline over time. In the 1950's corporate taxes constituted 4 to 5 percent of GDP and, more recently since the Panic of 2008-09, that percentage has dropped to roughly 1.5. A 3 percent (roughly $500 billion) decline in corporate taxes as a per cent of GDP goes a long way to explaining why "after tax" corporate earnings as a percent of GDP have increased to the current level.

Conclusion - The problems with incommensurate data are such that I think it very misleading to imply mathematical precision in estimating market overvaluation based on back testing data to 1876 or even using only more recent data. I have suggested in articles the use of price/dividend ratios instead of price/earnings ratios because dividends do not involve subjective judgment or changing accounting standards. I also try to use owner cash flow analysis to identify individual stocks which are cheap. Beyond this, I think that statistical analysis may be as likely to lead to mistakes as to profits. In terms of the market as a whole, when I find it harder and harder to find bargains based on enterprise price to owner cash flow ratios, I start moving some money into cash. I am doing that very slowly right now. I sincerely doubt that any more "sophisticated" market timing tool has merit.

Disclosure: I have no positions in any stocks mentioned, and no plans to initiate any positions within the next 72 hours.