Defending VAR - But You Still Need Common Sense

In Defense Of Value At Risk And Other Risk Management Methods

In the beginning of the month, New York Times Magazine published an article by Joe Nocera called “Risk Mismanagement” that created quite a stir in the blogosphere and beyond. Despite the watering-down of certain aspects related to risk management tools, as well as the diversity with which these tools are applied practice, the article was a success because of the buzz it created as well as the ensuing debate.

The article portrays a debate over value at risk methodology between well-known practitioners of VAR and the critics of the methodology led by Nassim Taleb. It is hard not to get carried away with Mr. Taleb’s tabloid-like descriptions of VAR as a “fraud” and its practitioners as “intellectual charlatans.”

I love how the debate is construed. The premise is that value at risk and other valuation models (such as Black-Scholes) assume normal distribution of asset returns. Okay, they do that in their most primitive forms, but let’s just accept the oversimplification as a fact for a moment because the debate would hardly exist in this simplistic form if we didn’t go along with the show here.

This is where our hero Mr. Taleb, an experienced options trader no less, emerges to the public mainstream to inform all of us ignorant folks that asset returns do not follow a normal distribution! The horror! The painful realization that this stuff continues to be taught in business schools! All that wasted class time learning statistics!

It is fair to say that this assumption will mislead naïve market participants about the nature of their risk exposures as “Black Swan” events happen a lot more frequently than suggested by Gaussian distributions. The problem is, almost anyone in finance already knows that asset prices are not normally distributed, and many practitioners build models or apply extensions to existing ones in order to take this into consideration.

I decided to give a little background on value at risk in order to get the points across that I feel strongly about. Since I teach VAR in the classroom as part of a risk management curriculum, I feel it is best to give some preliminary information.

A Primer On Value At Risk

Depending on the confidence interval chosen, value at risk, in its simplest form, exists of applying a one-sided test to figure out the loss that a portfolio may weather in a given time period. For instance, a 95% daily VAR of ten million dollars indicates that a portfolio is likely to lose at most that amount of money 95% of the time, or once a month assuming 20 trading days in a given month. At the same time, it displays the LEAST amount of money that the portfolio can lose 5% of the time. I appreciated it when Mr. Nocera mentioned this in his article prepared for general readership. As VAR is unable to tell us about what kind of a loss we should expect in that tail of 5%, the limitation of this metric if taken as gospel becomes apparent even to the untrained eye.

More on the tail risk later. But first, I would like to talk about three established ways of calculating value at risk for one asset and analyze the current risk management crisis within this framework:

Analytical VAR – “Misunderestimating” Risks

Otherwise known as variance-covariance method of calculating the value at risk, this is the well-known method of calculating VAR and the easiest one to apply. It assumes a normal distribution of returns. All it takes to calculate VAR is a standard deviation, which represents the “volatility” of the asset as well as a mean, which is the expected return on the same asset.

This is the VAR that Mr. Taleb seems to conveniently focus on, because it will indeed underestimate the risk at the tails of a negatively skewed or a leptokurtic distribution.

Stock markets in general exhibit negative skewness, which means that the distribution of returns will exhibit a long tail (a few extreme losses) to the left side. They also exhibit leptokurtosis, which means that both tails of the distribution are fatter than implied by normal distribution.

So we could go nuts over how wrong the normal distribution assumption is, and apparently people do. But we should also be very concerned over how sensitive this measure is to the standard deviation as well the mean, both of which are subject to change as markets change especially in the light of the current crisis.

Historical VAR – Good As Long As Future Resembles Past

This method does not need any assumptions about the distribution of returns and is certainly superior to analytical VAR because it is not parametric. The more data there is, the better the measurement. Historical data will exhibit characteristics such as skewness or kurtosis as long as the asset itself exhibits these qualities as well.

Assuming 250 trading days in a given year, in order to measure the 95% daily VAR you need to rank the returns from worst to best and pick the greatest return among those that correspond to the bottom 5% of returns. So the worst 12th return (or you could interpolate between the 12th and13th worst return, since 250 divided by 20 is 12.5, but since VAR itself is an approximation, why bother?) will tell you the maximum percentage loss 95% percent of the time, or the minimum percentage loss 5% of the time. Multiply the loss by your portfolio value and you get the neat VAR value in terms of dollars.

Moreover, the majority of investment houses use historical VAR as the basis for measurement as it is a clear improvement over the analytical VAR. You do not need return assumptions or standard deviation values to come up with this value.

Historical VAR calculations replace parametric assumptions with historical data. This means that if you had positions in mortgage derivative securities and started the year 2007 with models that were built around data of the previous two years encompassing the “peaceful” periods of 2005 and 2006, you would soon be awakened to a world where your VAR measures no longer reflected the reality of the marketplace. Note that such limitations of VAR as an all-encompassing risk measure were visible to any professional who understood risk management models as well as the limitations of historical data that went into them.

As Mr. Nocera’s article conveys, this is precisely what Goldman Sachs (GS) did. When it became obvious that the mortgage markets had changed in fundamental ways and aggressive positions in these securities started bringing in gigantic losses (as opposed to reaping the usual gigantic profits on the back of the ever-rising housing market), the team decided to limit its risk exposure by “getting closer to home.”

I don’t think the article conveys what “getting closer to home” really means. Let me use day trading as an example here. In day trading terms, this means that when your positions start showing huge losses at the end of the day, you accept “defeat” and take your losses as opposed to trying to ride them in the hope that the market will come around. So instead of wishing for market to make a comeback to recoup losses, you close out your open positions, take your losses and go home. Then you go back to the drawing board to strategize for the next day given the new reality of the marketplace.

Of course, looking retrospectively, the decision to limit exposure and take losses as opposed to trying to ride them in the expectation of a housing market turnaround has been the right decision to make. However, as we have seen with many other bubbles, managers do not have the incentive to make the sound trading decisions, nor do they have the incentive to listen to their risk managers as long as they get a huge piece of profits made during the ride and the taxpayer ends up holding the bag when the market finally blows up.

We have seen this movie over and over again. What surprises me is the heavy blame put on models for not reflecting “reality,” whereas those in charge knew that the mortgage bubble was collapsing, they had many opportunities to get rid of their huge exposures to the derivatives securities, but they chose not to do it most likely because of expectations of a market turn around. This is trading 101. If you try to ride your losses, you may make comebacks, but you will eventually blow up.

Now the next episode features critics who tell us that the “models” have been faulty and wrong. Hence the conclusion that value at risk is an erroneous and misleading measure, not to mention a “fraud.”

Ladies and gentleman, we found the “fraud” haunting the trading floor on the street, and it is not a human being: Shame on you, VAR and other risk management tools! Of course, we can blame the car manufacturers for the accident: the car’s faulty speedometer, or its lack of an apparatus to show us the bumps on the road ahead. But why is the culture that is reticent to blame the drunk driver who was clearly intoxicated with the thrill of making green?

These “models” are as guilty as the “accounting” that was used with a sleight of hand to conceal what was really going on behind the curtains during the Enron debacle and others. Of course, given the mathematical complexities of models, the quantitative brainpower needed to understand some of them, and the assumptions required in creating a map of your territory, there is more of an opportunity to either blame the models or to pretend that you didn’t understand them when things turned sour.

As I ventured with this essay, hoping to make my points within the value at risk framework featured in textbooks, I will move on to the third methodology used in calculating the measure.

Monte Carlo Simulation – Anything Goes, But More Of An Art Than Science

Monte Carlo Simulation is especially useful in calculating risk exposures of assets that have either little historical data or whose historical data is rendered irrelevant due to changing economic conditions that affect both the price of securities and the way these securities interact with each other in a portfolio. Also, historical returns of assets with asymmetric payoffs or returns of derivative securities that interact with variables such as interest rates, housing prices, and the like will not reflect the future when factors that influence the return of the security change as the economic climate shifts.

Monte Carlo Simulation does not require a fixed set of assumptions regarding its parameters or regarding its distribution of returns. In its simplest form, the technique generates random outcomes via simulating a large number of market returns. For instance, if 50,000 iterations are used in creating asset returns, the 95% percent VAR would be calculated as the 2500th worst return amongst all those returns that are generated by the computer.

The process explained above is the simplest way of running a Monte Carlo Simulation, and requires fairly simple programming and computational capabilities. However, a trading book may hold thousands of positions of securities with asymmetric payoffs that also interact with each other given changing market conditions. Depending on how many securities an average institution holds in its portfolio, calculation of VAR in this manner may become an enormous computational task.

Monte Carlo Simulation in finance is an extensive and broad field. It is also used in combination with other risk methods to carry out stress testing or to create scenarios that simulate crashes, among other things.

Extensions Of VAR And Must-Have Techniques To Supplement VAR Models

Shortcomings of the VAR metric have received due respect in risk management circles. For instance, given the deficiency of the VAR metric in correctly identifying the risks “stuffed” into the tails, extensions such as tail value at risk, otherwise known as conditional VAR, have been developed in order to deal with the issue.

Tail value at risk is the average loss a portfolio manager can expect in that 5% tail. A technical recipe could be like the following: Instead of looking at the 2500th worst return of your Monte Carlo model of 50,000 iterations to find the 95% VAR value, you sum up all the worst returns ranking from 1st to 2500, take the average of them, and call this number TVAR.

The New York Times Magazine article mentions the possibility of gaming VAR as well. For instance, if the portfolio contained a large out of the money short put whose value is likely to blow 1% of the time and cause a much larger loss than what your VAR is telling you, studying those tails carefully either via scenario analysis or via TVAR would keep the risk manager vigilant regarding the true risks the portfolio is exposed to.

Conclusion

Despite the improvements and extensions to the VAR metric, it remains an absolute necessity to supplement the measure with other methods such as stress-testing, scenario analysis as well as factor push models. Combinations of various methods may be used to simulate worst-case scenarios, and these models must be evaluated on the back of conditions present in the marketplace.

Yet there is a Bloomberg story from one year ago that quotes an SEC filing of Merrill Lynch:

"VaR, stress tests and other risk measures significantly underestimated the magnitude of actual loss from the unprecedented credit market environment,'' Merrill's filing said. "In the past, these AAA ABS CDO securities had never experienced a significant loss in value."

Translation: We did not “stress” our portfolio well enough, carry out the necessary worst-case scenarios, or question the validity of our historical data given the new mortgage environment because it was inconvenient for us to do so. But surely it’s not our fault now, is it?

Okay, I have since made peace with the idea that the taxpayer is footing the bill for the losses incurred by the “club.” When the big guys ruin the fabric of the financial system, the taxpayer pays for it, and that’s been the standard procedure. But I’ve had enough of watching people wash the blame off of their hands via appealing to the inadequacies of the models or the risk management tools they were using. Securities Exchange Commission or major newspapers or the general public may very well accept that kind of explanation, but I refuse the idea that risk models were to blame for the severe losses in this crisis.

A financial model is never a complete representation of what is going on in the markets and was never meant to replace judgment as well as common sense.

A forensic eye combined with qualitative analysis was and has always been good enough of a requirement to evaluate the robustness of models, and it is almost always visible to the practitioners when their models no longer reflect the reality of the markets.

Thus, while we should be aware of the shortcomings of VAR measure as well as any other model we are using, it is erroneous to put the blame on the tools when the crisis at hand remains a failure of human judgment, lack of responsible behavior as well as a collapse of plain old business ethics.

Comments

[sgt548:]

A well written, lucid article. Humans, with all our baggage, run the world, not computer models.

Jan 19 08:23 AM

[cr0bar:]

You're argument for VaR mainly appears to be that people forgot to ignore it?

Taleb's point is that sometimes no information is better than bad information. The use of VaR implies some knowledge of the underlying probability distribution, regardless of which version of the gaussian it is based on. This is a black and white issue, if the models are wrong they are wrong, not a little bit right. They will appear to be good models until the enormous consequential event which they have miscalculated the probablity of occuring occurs. Implying that VaR is useful because,

"A financial model is never a complete representation of what is going on in the markets and was never meant to replace judgment as well as common sense. "

i.e. that they should have known to ignore what VaR was telling them, is hardly a ringing endorsement. How do we know when to ignore it and when not to ignore it? It seems to me that it would be best to ignore it all of the time and in every situation for full security.

Jan 19 12:18 PM

[Suna Reyent:]

"This is a black and white issue, if the models are wrong they are wrong, not a little bit right."

A little quantum mechanics will teach everyone about black and white. Is light a particle or a wave? What about the uncertainty principle? What about electron spin?

Okay, I won't plead my case with physics, but practitioners of models know that mathematics is not going to have the same success at describing markets (human behavior) as the success it has had in describing the laws of nature.

This is old news, and not a novel realization that Taleb currently seems to hold a monopoly on. Unfortunately the point seems to escape those who like the sensationalism that Taleb offers.

The point of the article in one sentence is:

The hysterical fingerpointing on mathematical models ironically relieves responsibility from those very human beings who should be held accountable for what went wrong.

Jan 19 01:10 PM

[cr0bar:]

"A little quantum mechanics will teach everyone about black and white. Is light a particle or a wave? What about the uncertainty principle? What about electron spin? "

Well, having done a little quantum mechanics at university, I can say first of all that the difference between quantum mechanics and VaR is that if an experiment showed some part of the many theories of quantum mechanics to be patently false, it would be abandoned. The same practise is not being followed for VaR. Instead excuses are made.

In any case you make the wrong comparison, in quantum mechanics, the model is that particles exhibit both wave and particle type behavior. There is not one model for the particle behavior which competes with the wave behavior model. In other words, if someone came forward with evidence that particles never act like waves, or never act like particles, the entire model of wave/particle duality would be abandoned.

Similarly with the uncertainty principle, if someone showed that you can in fact make an exact measure of the position and velocity of a particle, the uncertainty principle would be abandoned as false.

What about electron spin? I don't even know what point you're trying to make here? That it exists? That it doesn't exist? That the theory is wrong?

There is a further distinction with these models. If they are wrong, the results are inconsequential, if VaR is wrong the consequences can be enormous! Therefore we must be much more careful with this untestable (although many could argue it has been tested and failed many times) model than an approximate physical model whose results are rigorously tested through experimentation.

Jan 20 08:36 AM

[Suna Reyent:]

Dear Mr. "cr0bar",

I started college with the full intention of a career in physics, including a doctoral degree. I majored in physics and then also in economics with an eye toward getting employment on Wall Street. I graduated Magna Cum Laude, and with that my Physics GPA was considerably higher than my economics one. I never moved on to graduate level, but I can show you official records of advanced undergraduate courses in physics that I took.

Employers on Wall Street did NOT think that I was talking mumbo-jumbo whenever I drew similarities with physics. In fact, I was given more quantitative roles than I may have asked for simply because of my background.

My background as well as experiences are available to anybody who wishes to conduct research, because I'm actually blogging under my real name.

If you wish to accuse people of trying to "intentionally deceive" others, you would enhance your credibility if you left comments with your real identity.

Jan 20 09:51 AM

[Robert H. H...:]

From cr0bar:

"This is a black and white issue, if the models are wrong
they are wrong, not a little bit right."

With all due respect, this assertion has a nice rhetorical ring, but no substance whatsoever except for the nihilist.

A VAR model, like any other model, is an *intentional* simplification of the real world it attempts to describe. We already have the real world, in all its elaborate and bewildering complexity, so we build models that are simplified abstractions to further our understanding. If a model that is "a little bit right" is therefore "wrong" by your lights, then all models are (intentionally) wrong, your position is nihilistic and this debate is pointless.

A sophisticated user of a mathematical model -- whether for risk management, weather prediction, or discovering underlying causes of diseases -- attempts to understand the models strengths and weaknesses (by comparing its predictions to the real world) and takes both into account when drawing conclusions.

It is undoubtedly true that senior bank executives took false comfort from various risk analytics and allowed their firms to take on far too much leverage with disastrous consequences. But false comfort and excessive leverage are not the fault of the models any more than flood damage is *caused* by the model that produces your local weather report.

I assume you occasionally consult the weather report before deciding how to dress in the mornings and whether to carry an umbrella? And you are probably grateful that airlines check the weather before flying even though weather reports are notoriously unreliable and frequently wrong. If so, you are living proof that one can receive probabilistic information and act on it with something resembling free will.

Also, "[sometimes]... no information is better than bad information" really makes no sense without a predicate assumption about how one uses that information and a corollary about the consequences. What matters is what one does with information (or the lack of it). If you mean, "Possessing bad information, believing it without reservation and acting upon it with adverse consequences is worse than having no information, not acting and therefore not suffering the consequences”, I suppose that's trivially true, but only because I've stipulated that the outcome was negative. That's what logicians call "begging the question".

It’s equally true that “Sometimes bad information is better than no information.” If you received a hot tip that Steve Jobs suffered a heart attack, immediately shorted AAPL, closed out your position at a profit and subsequently discovered the rumor was false, you’ve still profited from the “bad” information. In fact, someone in possession of good information that Steve Jobs had NOT suffered a heart attack might have lost money betting the other way. So, I suppose it’s also true that bad information is *sometimes* better than good information, depending on the outcome.

If “bad” information can be worse or better than good information, how are we to decide which bad information we should accept or reject ex ante?

I’d build a model, make some predictions and test the predictions against real-world outcomes.

Jan 21 02:40 PM

[cr0bar:]

I admit I was confrontational in my last comment and I apologise, but this debate is not academic. Thousands of real people can lose their homes, pensions and their livelihoods on the basis of what happens in the real economy and as we have seen with the banking crisis, incorrect risk evaluations have extremely severe consequences. If these models were not held in such high esteem, the banks could not turn to them as reasons to relax legislation in the good times or as excuses for failure in the bad times.

However, specifically what part of the arguments I made are incorrect? Also, if I were to reveal my real identity would this prove or disprove them? If I were joe the plumber would this make my argument false, and if I am Benoit Mandelbrot would it make them correct?

Finally, what then specifically did you mean by your statement:

"A little quantum mechanics will teach everyone about black and white. Is light a particle or a wave? What about the uncertainty principle? What about electron spin"

And what is the analogy to the VaR model or my statements about the use of models? Again I apologise for being rude and invite you to clarify this.

To answer some of R Heaths comments:

You are right that we should make models, and then test them. This is exactly what doesn't seem to happen with VaR. When VaR fails it is assumed that it can be tweaked to fix it, or that this particular failure was some 'special' one-off event which could not be accounted for etc.

"Sometimes bad information is better than no information"

In your shorting scenario, how do you know when to close the short position? You assume you have shorted AAPL and the market has moved by some fixed amount at which you know (or happen by luck) to always close out, and you were not whipsawed by the market when the real news came out. Let's say the next day you receive the same information again, would you take the same action? What you are describing is luck, not the good use of bad information. If you do not act on the information you run no risk, if you do, you could be risking a large part of your capital on false information. I for one do not want my banks to be run like this.

Jan 21 04:10 PM

[morph366:]

Interesting piece but methinks you do protest too much about how quants have been misunderstood. Reminds me of some of the self-serving rhetoric from people like Paul Wilmott and Emanuel Derman.
When you say that Taleb is banging on with old news about how financial time series data is not normally distributed it is no less severe a criticism because it is old news.
The real issue that you seem to be trying to evade is that no amount of sophistication in the modeling and the mathematics can deal with the simple problem of how best to estimate the probability of on one day the consensus view is that the glass is half full and the next day the consensus switches (all at once) to the view that it is half empty. Risk analysis should be a lot more about understanding human emotions - fear, panic greed and peer pressure - and quite a bit less about mathematics and Monte Carlo simulations

Jan 22 12:47 PM

[Robert H. H...:]

morph366 -

I'm not sure which part of Ms. Reyent's conclusion you disagree with. In her words:

"A financial model is never a complete representation
of what is going on in the markets and was never meant
to replace judgment as well as common sense."

In yours,

"Risk analysis should be a lot more about understanding
human emotions - fear, panic greed and peer pressure
- and quite a bit less about mathematics and Monte Carlo
simulations."

You seem to be in violent agreement on this point.

cr0bar -

Thanks for the response.

I encourage you to re-read the Nocera piece in the NYT as I just did. For me, the most interesting bit is the anecdote about Goldman Sachs losing money on their mortgage desk for 10 straight trading days. I'm quite sure that Goldman's risk models (including VAR) would characterize this as an outlier. So what did Goldman do? They apparently compared what was actually happening to their models' predictions and concluded:

1) the world had changed, or
2) the models might be mis-specified, or
3) they couldn't explain what was happening.

Either way, they decided to "get closer to home." That's what I call testing and re-testing the model. Goldman didn't distinguish itself by avoiding the use of risk models; it has come through the current crisis (relatively) unscathed precisely because of *how* it used them.

Regarding your other point, I think we are in agreement that taking no action with regard to a potential wager is risk-free (at least as regards that specific bet). But I don't think introducing the concept of "luck" -- by which you seem to mean a good outcome despite a foolish wager -- informs the debate. The issue I raise is how do you evaluate the risk/reward of an inherently uncertain bet *before* the outcome is known?

For my part, I prefer to have information -- conditioned by my beliefs about the relevance and validity of that information -- as opposed to willful ignorance. As I cannot peer into the future, the information I have is necessarily limited to past experience. So, in my view, using information about the past to make predictions about the future is rational. Willfully ignoring information is irrational.

When Taleb's valid observations about outliers get converted into prescriptions to ignore historical data, I wonder exactly how one is expected to follow this prescription.

Jan 22 03:30 PM

[Suna Reyent:]

Mr. Heath - thank you for eloquently explaining the Goldman episode in the framework of testing a hypothesis.

morph366 - I'm not saying that the quants are misunderstood. I'm saying a lot more than that.

I'm saying that the quants, or more specifically quantitative models, are used as whipping boys in what turns out to be a great lapse in plain old business ethics. It is not the self-serving "quant" rhetoric that I am concerned with, but very much the other way around.

It is rather the self-serving justifications of the failed Wall Street management that concerns me. Titanic was sinking, and its captains skillfully escaped the ship first, while pocketing large bonuses and leaving the rest of the passengers (the American taxpayers) on board. What was the music played by the orchestra? The now famous our-VAR-models-didn't-... tune.

I must admit that the massive PR campaign that so successfully packaged the blame onto quantitative models, as well as the sensationalism offered by Taleb make all of this a lot easier to swallow.

This is not a "protest" piece as it is a piece that aims to give a background on value at risk and hopefully show people how to think about it. I for one use the concept (loosely) in my simple stock portfolio allocation to have an idea on what I can expect to lose, 95% of the time, 99% of the time, or 99.9% of the time. You would be surprised at how well it does, especially if you adjust your variables or data depending on how markets change. Of course, that's also a form of ART as much as investing is.

As I've tried to portray, VAR works a lot better (and is much easier compute) for simple portfolios that take no derivatives positions (especially with assymetric payoffs) or leverage. No, VAR will not save the world. Yes it can be wrong depending on how markets change. Yes you can come up with different values of VAR depending on your assumptions about markets. Yes it requires a human being to interpret the number. And NO it is not acceptable to blame VAR for your losses when things turn sour.

Jan 23 06:17 AM

[Jeff V:]

Good morning Suna,

Thank you for your article. I generally agree with your conclusions, that VaR is just a tool, and using it is more of an art form than an exact science, since returns do not hae a normal distribution.

However, I spend a year as a finance/econ PhD student, and all the theory there is built off of the assumption of normal distributions and perfectly rational individuals. When I questioned these assumptions, the responses I received ranged from anger to mockery to a confused blank stare. All the academic theorists know is normal distributions and rational individuals.

Unfortunately, the “our-VaR-models-didn’t... Wall Street management could probably also add “research from Nobel Prize winners told us this would work! It isn’t our fault!”

- Jeff

Jan 27 11:19 AM

[Corporate G...:]

I am a risk manager. I use VaR (parametric, historical scaled to the VIX, and Monte Carlo) combined with scenario analyses. I also use Conditional VaR/Expected Shortfall. Importantly, I back test static daily P&L against the VaR forecasts to ensure robustness. Over the past two years of perhaps the most volatile markets we have seen that the breaches/exceptions/ut... of VaR fell with in statistical expectation. This was all overlaid with a good dose of common sense risk management, i.e. what are my concentrations, what are my exposures, leverage levels, etc. Furthermore, when stop losses are breached, we held firm and cut losses completely or reduced our exposures. This led to keeping our maximum drawdown very low and our returns positive throughout 2008.

The real problem is not the risk models....it was the leverage and greed throughout the financial system that went unchecked due to poor CORPORATE GOVERNANCE and lack of transparency. Rating agencies and investment banks were in bed together. Tons of risk managers using both quantitative and qualitative risk tools unearthed plenty of evidence that sounded an alarm that sub-prime was a problem waiting to happen, but their voices were drowned out by greedy traders and ignored by senior management who were too concentrated on soaking up fat short-term returns with complete disregard for the sh*tstorm waiting to hit just around the corner.

Jan 27 12:58 PM