Digging Deeper into Historical Market Data 1871-2009

Jul.14.09 | About: SPDR S&P (SPY)

Yesterday I looked at S&P 500 daily data from 1950 on. Today, thanks to the generous Professor Shiller, I'm going to look at historical market data from 1871 to the present for the S&P 500.

Let's jump straight in. The first chart shows one year percentage returns by month for the S&P 500 from 1872 to 2009 (Click to enlarge). I had originally intended on showing only the second chart, but included this chart showing the percentage change as it provides more information. I prefer the simplicity of the second chart which shows the up and down months year on year, YoY, for the S&P 500. I have taken the liberty of forecasting this out to the end of 2009, with a conservative "no up months". All will be unveiled in the coming months, but the first month when there is a probability of the S&P 500 being up year or year is October. The probability of an up month increases every month thereafter.

S&P 500 Monthly Year on Year Change 1872-2009 (Click to Enlarge) S&P 500 Up and Down Months 1872-2009 (Click to Enlarge)Click to enlarge

As I said, I prefer the simplicity of chart two as I like to unleash my brains ability of seeing patterns whether or not they exist. This chart makes me think we're due for an up month. Add to this that up and down months appear to cluster together like tall people at rock concerts and I predict we'll soon have a series of up months. It seems almost so obvious that I feel preposterous even putting it in print, but as there's still a lot of commentators out their spreading doom and gloom, I guess I'm going out a limb. Well maybe a bough more than a limb. Feel free to add constructive criticism, but please don't bore me with renditions of gamblers fallacy or apophenia or prediction addiction as Jason Zweig prefers to call it.

The third chart is another view of the same data, once again click to enlarge. This time I've looked at the consecutive down months. We're currently sitting on 18 down months in a row, which is the seventh equal longest period since 1872. By the end of September this will, I confidentiality predict, be the outright fifth worst longest period of consecutive down months. The longest in period of down months at 35, occurred in the early 1930s. If not for one up month in April 1930 that would have been a streak of 41. The 1870s and 1880s also had long streaks only interrupted by a couple of months.

S&P 500 Consecutive Down Months 1872-2009 (Click to Enlarge)Click to enlarge

Gamblers fallacy. The stock market is not known odds. While this data shows an up month occurs on average 62% or almost two thirds of the time, the odds of an up month occurring on the next iteration are almost never 62%. The odds change all the time and after each iteration. It also makes no sense to only look at the odds of the very next iteration, unless you're a short term trader trying to guess short term direction.

Simplistically applying gamblers fallacy to the stock market is pure folly. Despite what efficient market theorist would have us believe, the market is not random. If the market is not random then gamblers fallacy doesn't really apply. It's still a good theory to keep in mind, as is apophenia, but sometimes seeing is believing. I am not saying real patterns exist in all data, many studies prove they don't and that humans see patterns when they don't exist. However, nothing is black and white; sometimes patterns do exist and sometimes historical data can help calm the fear whipped up by doomsayers and data.

Let's take my comment from above which probably sent gamblers fallacy advocates into a quixotic fit. "the first month where there is a possibility of the S&P 500 being up year or year is October. The probability of an up month increases every month thereafter." Hopefully we all agree the probability of any month being an up month is not actually 62%. If you don't agree with that then may God look on you favourably as it's unlikely any mortal can help you. So what are the odds?

Some people argue that only a fundamental analysis can assist in setting the odds. I'd argue that the chances of fundamental analysis accurately setting the odds is the best definition of random I've heard in a long time. No doubt an economist or fundamental market analyst will accurately foretell the future. After all, as their guesses cover the spectrum one of them has to have it right. Still there are many people who I respect that use fundamental analysis as part of their armoury. Grantham is one and he said 30% of a new low. Buffett is another and he said buy America. Hussman is a third and he says slightly overvalued. They're all in the same ball park, now is a good time to buy. Now means while prices are low. Yes they may go lower, but then they will go higher.

I don't need to set the odds, I only need to be in the ball park. Here is the data for the last year on the S&P 500.

Month S&P 500
2008.07 1257.33
2008.08 1281.47
2008.09 1216.95
2008.10 968.8
2008.11 883.04
2008.12 877.56
2009.01 865.58
2009.02 805.23
2009.03 757.13
2009.04 848.15
2009.05 902.41
2009.06 938.15
Click to enlarge

The S&P 500 closed at 901 Monday. The probability of it jumping 356 points or 40% in a half month is so close to zero. Despite the black swan at my local creek, I'm willing to be categorical for once and say it ain't gonna happen. Once in the last 1660 months, 138 years has the S&P 500 jumped more than 40% in a month. That was August 1932 and while current volatility is high it is not as high as 1932.

The same goes for August and September, almost zero probability of a new YoY high. That takes us to October and a clear possibility of a new high as the market only has to be 7.5% higher by then. Same goes for the following months and while the probability of each iteration being a new YoY high may be around 2/3rds (WAG based on long run probability), the probability of just one of the months being a new YoY high is even higher.

The market trends rather than follows a random walk. Any month has a 93% likelihood of being the same as the proceeding month. The probability of future months being up YoY increases as the probability of a trend change increases.

When I factor in the fundamentals and the psychology of the crowd on top of historical precedence I think we're highly likely to get a new YoY high between October and June. When we do, the probability of a string of new YoY highs is high, but as I pointed out above there have been a number of instances when new YoY highs were mere interruptions to a longer down trend.

Here are some quick replies to the comments on my first article in this series which was published at here at Seeking Alpha.

I didn't and never would say you should solely approach the market logically. While logic plays a part, psychology plays a larger part. I don't need to be Wolverine to smell the fear emanating from the market.

I find it interesting that those who clamour for longer historical data appear to only be able look a few months to a couple years into the future. A bull market will happen again, I don't need graphs to tell me that. If it doesn't start until 2014 or 2018 all the better. I'm accumulating wealth so the longer stocks are on sale the better and the stronger the resulting bull market will be. I'd love the S&P 500 to drop to 600, hey give me 400 and I'll be ecstatic. I simply think it is the least likely scenario and standing on the sidelines hoping it is the likely scenario is the least likely path to success.

Later this week I'll really waste your time and layout my working thesis of the years ahead. You may be surprised that while I believe in market timing I do not believe in market predicting. I think too many people make the leap from doubting the predictability of markets to believing that fully invested is the correct path and adjusting your market exposure is a waste of time.

For now I'll leave you with a couple more graphs. The first is the distribution of YoY monthly changes in the S&P 500. The second is the distribution of month on month changes to the same data, 1871-2009.

S&P 500 Monthly YoY Change histogram 1872-2009 (Click to Enlarge) S&P 500 Month on month price changes histogram (Click to enlarge)Click to enlarge

As you can see, month on month changes are almost always in the +/-10% range, 97.53% of the time. The highest was change 50.3% and the lowest -26.47%, August and April 1932 respectively. To put things into perspective October 1987 was down 12%. Keep in mind I'm talking month on month data here, not start to end of month. The second largest fall since 1871 of -20.39% occurred on... come on, have a guess before you mouse over here to see the answer.

Disclosure: Net long various stock markets, including US markets.