## Summary

- The market is, without a doubt, unpredictable in the short term.
- Even though the market acts randomly, that does not mean that the market is efficient.
- We have a tendency to see patterns even when they don't exist. Data is more useful than gut feeling for testing the usefulness of these patterns.

If there is one question in finance that everyone would love to have the answer to, it's: What is going to happen tomorrow? After all, if you knew exactly what the market was going to do it would be easy to make a ton of money. There is a never-ending debate on how predictable the markets really are. This has a great deal to do with the efficient-market hypothesis (EMH) and whether or not the market can be beaten. I've seen countless articles make bold claims about this; sadly, data is all to often considered optional.

The EMH states, roughly, that the current price of a stock already reflects all of the publicly available information and that the price will rapidly adjust to new information. That means that there is no way to beat the market except by pure luck. Therefore, the EMH implies that the market acts randomly. One of the important points I want to make is that a market that acts randomly does *not* imply that the EMH holds. This point has been made before, but I will provide my own analysis to support the conclusion. Also, in the aftermath of the financial crisis there has been a great deal of discussion about black swans and problems with the simple random walk hypothesis; in my analysis of the EMH question I will touch on these issues as they are related.

For this article, I am using the S&P 500 as a proxy for the stock market and all the analysis uses S&P 500 data unless explicitly stated otherwise. Let's start with the question of randomness, is the market unpredictable? This turns out to be a less straightforward question than it seems to be. Anecdote's about the market are not sufficient, so let's look at the data. In order to analyze the data, I define the "n-day log return" of the S&P 500 as the natural log of the index value on a given day divided by the index value n-days prior to that day.

It's important to keep in mind when looking at charts of log returns that a positive value means the index is higher than it was n-days ago, a negative value means the index is lower than n-days ago. In addition, log returns are symmetric in a way that percentage returns are not; in fact, that is the reason for using them. Specifically, if the index value today is 1,000.00 and tomorrow the log change is 0.05 then the index value is 1,051.27. If the log change the day after that is -0.05 the index value will go from 1,051.27 back to 1,000.00.

*Source: Data taken from Yahoo Finance. I made the graph using the daily close for dates between 1/4/1950 and 6/25/2014.*

The chart above shows the daily close of S&P 500 on the top (logarithmic scale on the axis) and the 1-day log return on the bottom. The 1-day log return chart is worth looking at closely as there are several things worth noting. My initial observations were:

- The 1-day decline on Black Monday is considerably larger than any other 1-day change, positive or negative (the value is -0.228997). The next greatest decline has a value of -0.094695, about 2.4 times less than Black Monday. The greatest 1-day increase had a value of 0.109572, about have the magnitude.
- The 1-day returns look like noise with no discernible signal and there is no visible bias for positive or negative values.
- The volatility is not constant. It's well known that volatility clusters and this chart shows that very clearly. That means the 1-day log return data is only weakly stationary.
- Periods of high volatility appear strongly correlated with bear markets. The 2008 financial crisis is a good example of this.

So is there any correlation between the returns on a give day with the days before. It turns out that there is no correlation between the past values and the future values. Here is my time series analysis of the 1-day log returns to justify that point:

*Source: Same data as the chart above.*

The important part is the autocorrelation, that is the correlation between points in the time series offset by some amount (called lag). The autocorrelation for lag 0 is 1.00 because the value on a given day is perfectly correlated with itself. It's easy to see that there is no correlation with past values. I only showed the lag up to 7 trading days here to save space; the autocorrelation values for higher lag values are all effectively zero.

All of this analysis means that yes, the market is indeed unpredictable in the short-term. However, the market does not follow a simple random-walk (Geometric Brownian Motion, or GBM for short) as many models assume. The most well known model that makes this assumption is the Black-Scholes option pricing model. If the random walk hypothesis were true, the 1-day log returns should be normally distributed.

*Source: Same as above.*

The red line shows the best-fit normal distribution for the given data. It's easy to see that, while the normal distribution is not a horrible guess, it is clearly not great at describing the data. The most visible deviation from a normal distribution is around the average, the data is much more focused around the mean than would be the case for a normally distributed data set. The other major deviation from a normal distribution is shown by the outlier box plot above the histogram. All of those points represent events that should not have happened if the data followed a normal distribution. In other words, the returns have a heavy tail. If you care for the numbers, the summary statistics are given next to the histogram and also indicated the data is not normally distributed.

This all goes back to the original point, the market is random, but it is not efficient. It can be argued that a somewhat weaker form of the EMH does hold, I would agree with that. After all, there is a connection between a stock's price and the underlying company's business. There have been enough value investors with long time horizons who have beaten the market to make this point.

If people were as rational as the EMH makes them out to be than there wouldn't be so many outliers. However, there is some semblance of order, so people are not totally irrational. Clearly, people are trying to make use of known information when pricing stocks and there is no discernible way to take advantage of short-term fluctuations in the market.

So, why would you care about this as an investor? The lesson is not to become distracted by the day to day market movements because they don't mean anything. Also, don't trick yourself into seeing patterns that don't exist. Most importantly, always do your homework. If you see a claim without supporting data, that is a red flag.

There is one more side point I would like to bring up. I focused on the 1-day log returns of the S&P 500, but I also ran the data for other time periods. The result is fairly interesting, but I'll leave it to you to determine what information the chart conveys, if any (see below).

*Source: Yahoo Finance data for dates between 6/21/1961 and 6/25/2014.*

Comments()