Measure, Don't Model: Probability, VIX and Bad Math

This has been my motto, my slogan if you will, for a long, long time. As noted in prior posts, over a decade ago I began to question the validity of the probability assumptions in the bell curve distribution that is the basis of so much of our financial system–from the pricing of options and other derivatives to the Basel Accords that regulate leverage ratios and capital levels for banks, insurance companies and other financial entities worldwide. The reason I questioned these probabilities was due to the reality of far too many 3, 4 and 5 standard deviation moves than the bell curve suggested. In 1996, I created a spreadsheet to analyze these flawed probability assumptions, and began using it to analyze option positions. The assumptions underlying that methodolgy is the key component of our free analytical tool - ODDS Movements. It compares the probability assumed by volatility (which is calculated assuming a normal price distribution) to the actual probability of a certain sized stock movement.

The reason I am writing today is to introduce another analytical tool that I built in a spreadsheet. This new tool was inspired by the ridiculously bad article that Bloomberg published about VIX. I have to tell you that I have no idea who was wrong about the facts in that article. Was it the authors misconstruing what was told to them by the experts, or were the experts unable to properly explain things to the authors? All I know is that the article confused the word improbable with impossible. And that’s a very serious mistake to make.

For instance, in the third paragraph, the authors state that “Money managers relied in the 18-year-old VIX as a guide for the S&P 500 because the gauge correctly predicted the equity index’s range 84% of the time …” That is sort of true, IF you assume that the range being predicted is the range bound by one standard deviation. Here’s how to validate that. You can do it yourself, or you can just take my word for it! First, take the value of VIX, the value of the S&P 500, along with 30 calendar days (since the VIX measures 30-day volatility) and put those values into this equation. You’ll need to do this twice — once for +1 standard deviation and once for -1 standard deviations. [Note that this equation does not assume a logarithmic distribution. It uses a linear distribution, which is distribution used in the Bloomberg article.]

Solving this equation for +/- 1 standard deviation will give you an upper and lower range. Then you wait a month to see if the S&P 500 stayed within that range. Now do this for each and every day for the past 18 years, and then start tallying things up. What you’ll find is that indeed, up until 2008, the index was within the predicted range 85% of the time.

But here’s the issue. According to Guassian/normal/bell curve distribution, how frequently should it have stayed within that range? The answer is something we’re all familiar with. About 2/3 of all outcomes should lie between +/-1 standard deviations. But our calculations showed that the index moved less than +/-1 standard deviation 85% of the time. That means the index was stagnant far more frequently than predicted.

In other words, the VIX was never accurate, at least when it came to 1 standard deviation! VIX and the probability model upon which it is based predicted 68%, when in fact, the actual probability was 85%!!

What this new tool does is take this analysis many steps further. First, it looks at all periods through the end of 2008, and it breaks down the analysis by year so you can see how some years behave differently than others.

The other thing is that it looks at different standard deviations. You can select from 1 to 5 standard deviations in ½ point increments. While the VIX may have improperly modeled 1 standard deviation, what about other standard deviation ranges?

By doing this, you can see how the various probabilities expected by models based on the bell curve distribution compared to what really happened.

Here is a partial screen shot showing the bell curve comparisons using one standard deviation.

Click the image to use the analytical tool yourself. Be sure to read the text at the bottom for a full explanation of all the features and how to interpret the data.

Comments

[deflation r...:]

Hooray for option sellers!

Jan 07 10:16 AM

[CM in MA:]

Good premise. An article that covers a great deal of ground. I'll pick up just one narrow thread, that being calculations using the VIX. Unfortunately, this article seems to perpetuate a myth surrounding the VIX. Specifically, that the VIX predicts an expected return or range of returns. While that may be somewhat true, there's a bit more to it. And that bit more calls into question the calculations and conclusions of the article. Let's leave aside whether the VIX does, in fact, predict anything and deal with what it is supposed to predict.

First, this from the CBOE website: the VIX is a measure of expected volatility calculated as 100 times the square root of the expected 30-day variance (var) of the S&P 500 rate of return. The variance is annualized and VIX expresses volatility in percentage points.

To translate the CBOE definition: the VIX is the expected first standard deviation of the annual S&P 500 rate of return. So, to try to use the VIX for anything, one must first have an "S&P rate of return" for the time period under consideration. That's a big problem since it leads to relying on mean returns, and we know that market returns are lumpy. Since the VIX is already annualized, here's what it tells us if we assume the widely used 8% annual return on the S&P. For a VIX of 25, the expected annual return will be between -17% and +33% at the first standard deviation. At the second standard deviation, it would be between -42% and +58%.

It's easy to calculate the first (or any other) standard deviation of the expected return given any VIX value and time period. The article shows correctly how to do this. For example, at VIX=25 for 30 calendar days, the first standard deviation is 7.17%. But that standard deviation, as well as second, third, etc. deviations, has to be applied to the expected 30-calendar day return. The equation in the article assumes this value to be zero. Perhaps that's ok for very short periods of time (a few days), but at 30-calendar days that is a dangerous assumption. It certainly matters when evaluating the accuracy of 30-calendar day predictions.

All that said, caution is warranted when using means and standard deviations for expected market returns!

Jan 07 12:47 PM

[odds:]

CM in MA, I have a question. What do you mean when you say that "this article seems to perpetuate a myth surrounding the VIX. Specifically, that the VIX predicts an expected return or range of returns."?

Jan 07 03:02 PM

[CM in MA:]

Odds, thanks for the question. First let me say that I thought there was a lot of goodness in Don's article. I agree with much of what he wrote about the wrong distribution underlying many models, and some of his other points. I was writing about one minor nit that I hear and read often regarding the VIX. Since it is a standard deviation value, it needs a mean to go with it. That part is consistently overlooked. I should've written "this article seems to perpetuate a myth surrounding the VIX. Specifically, that the VIX predicts an expected return or range of returns (around a zero mean)."

My point on Don's article was that, given the appearance that his VIX "predictions" lack a mean return to center around (other than zero), I'm not sure what to make of the 85% figure he cites. But it's a tempest in a teapot, really. Using means and standard deviations to build probability functions for market returns is ill-advised. I'm not implying that is what Don is doing. In fact, he seems to be taking issue with just such an approach.

Now, should we be surprised that actual returns fall within the VIX predicted 1SD range 85% of the time? Not really. This is comparing apples (implied volatility) with oranges (actual or historical volatility). I don't have stats handy, but implied volatility consistently overstates actual volatility because options sellers will err on the side of higher volatility when setting prices. Keep in mind, the VIX is the implied volatility based on actual prices. It's a reverse process following many options sellers (and buyers) putting in (modeling) various implied volatilities across many strike prices. Then there is the supply/demand dynamic when markets are moving quickly. As the prices up and down the options strip(s) expand, the VIX goes up (usually, but not always, as the market goes down). For lack of a better term, the VIX is "unified implied volatility." It's just a measure of price spread in options across various strikes. Predictive? I've spent far too many hours measuring (and modeling) the VIX. At low levels of volatility it's predictive until it isn't (i.e. volatility returns). At high levels it's predictive in that at some point volatility will subside, and markets tend to rise when that happens. I guess there are some strategies that can fall out of that.

I don't know if all that makes any more sense. I hope so.

Jan 07 04:52 PM

[Adam Hayes:]

the VIX is so influenced by skew that it becomes irrelevant in reality.. simply a x% down move will increase the VIX while an equal x% up move will lower the VIX.. in a distribution a move up OR down of equal % should be equal but due to Skew there is always more premium built in to the downside.
All options trader know this but when the layman or non-options trader looks at the VIX they do not realize this phenomenon is going on and use it incorrectly as a predictor for the future movements of stocks.

Jan 08 09:00 AM

[odds:]

CM, you need to keep in mind that the probabilities discussed are the probabilities cited in the Bloomberg article. They and their experts are the ones who said that the market "wouldn't fluctuate" by more than "7% in the next 30 days". And that missing expected return portion you cite (also known as drift) is missing in the blog post above because the mean return return is so insignificant over a 30 day period that it's practically meaningless. The same things could be said about the fact that neither the authors of the Bloomberg article nor Don consider the discounted interest rate impact on the index price. But those factors -- including the omission of drift -- are so insignificant that their omission really isn't the significant issue. The issues are: 1) the Bloomberg article confuses improbable with impossible and 2) the normal distribution that the article cites as giving accurate probabilities is not truly representative of the real world.

Remember, this is a blog post, not a research paper. There are space constraints, so insignificant information is purposely omitted.

Finally, and I don't want to sound snarky here, but I do know what implied volatility is, and I do know that the definition of VIX is the annualized standard deviation of returns (price changes) expected by option market participants. That means, by definition, VIX has to be a measure of probability expectations.

Jan 08 09:18 AM

[CM in MA:]

Odds (it's really Don, right?), my comments aren't intended as snarky or to offend either so I hope you don't take them that way. Some of the background info (such as definitions) is intended to get us on some common ground, that's all.

Again, I agree with the thrust of your article. Particularly the shortcomings of the probabilistic reasoning of the cited experts and commentators, and the shortcomings of many models that ultimately are reflected in the VIX.

I'll disagree about the calculations, though. This is not to disagree with the conclusions because, for all I know, the actual returns may fall within the predicted 1SD more than indicated if the expected return is taken into account. I haven't done that analysis. The data set I have handy is Jan 1990 - Dec 2006. The mean for the rolling 30-calendar day return is .75% with a standard deviation of 4.03%. Sticking with the 30-calendar day predicted return range for a VIX of 25 (+-7.17) as an example, the expected return is 10.4% of that predicted range. As I said, I don't know what that means to your facts, figures and conclusions. I would be inclined to take that into account, but I guess we can disagree on that. And, of course, the expected return becomes more pronounced as the time span of the prediction is increased (or as volatilty decreases). Again, using a VIX=25 the prediction for an annual return range would be +-25%. For the 1990-2006 data, the 12-month mean is roughly 10%, so the expected return is 40% of the predicted range. To be fair, the expected return becomes less important as the prediction time span is decreased or as volatility increases.

Jan 08 11:10 AM

[John Lounsbury:]

I have learned more about the VIX in 15-20 minutes reading this article and comment stream than in several hours of prior reading. Of course, I admit that the prior background may have given me what I needed to understand more of the discussion here.

You may have a smaller audience than some other artciles on SA, but for those who do read and understand the subject, this is a great article (and comment stream).

Jan 08 01:31 PM