Did Portfolio Planning Tools Fail Investors in 2008? 13 comments
-
Font Size:
-
Print
- TweetThis
The Wall Street Journal ran an article on May 2, 2009 called “Odds-On Imperfection: Monte Carlo Simulation.” The sub-title is “Financial Planning Tool Fails to Gauge Extreme Events.” The main point of the article is that Monte Carlo Simulations did not predict the potential for a market meltdown on the scale of what we experienced in 2008. This article reinforces some common misconceptions about Monte Carlo planning tools and probabilistic models in general. As the author of a Monte Carlo planning package, I got quite a few questions about this article.
The main premise of the WSJ piece is that “there is little chance your Monte Carlo simulation, named for the gambling mecca, would have highlighted a scenario like the market slide just seen. Though these tools typically run a portfolio through hundreds or thousands of potential market scenarios, they often assign minuscule odds to extreme market events.” The author then frames this point as a general critique of the use of probabilistic portfolio management tools like Monte Carlo Simulation.
The author is correct that available Monte Carlo models and other risk models assigned very low probability to losses on the scale of what we observed in 2008. I have no problem with this assertion. It is where the argument goes from there that is flawed. It remains unclear as to what odds should have been assigned to an event like 2008. If we have experienced two events on the scale of 2008 in the last hundred years (2008 and 1929), this is a very small sample. There is no way to really ‘validate’ a model’s assigned probabilities of events at this kind of extreme. After a truly extreme event, it should be the case that Monte Carlo models projected that this event was possible, but it is not realistic to believe that it is possible to truly ‘validate’ models on the basis of events that happen once in fifty years or so. Perhaps we should always apply a ‘safety factor’ to estimates of 1-in-50 or 1-in-100 year events (like the safety factors used in structural engineering, for example). There are ongoing efforts to improve the way that extreme events are modeled—including by my firm. This is an important area of research, but focusing just on this issue is a mistake.
The main purpose of Monte Carlo Models for portfolio management and planning is to show how market volatility can impact an investor’s long-term plans. There are a range of risks that can be understood only with a Monte Carlo simulation or other probabilistic tools. These include sequence-of-returns risk and longevity risk. These models provide insight into how investors can build better long-term plans, accounting for these risks. If I were to read the WSJ article without knowing a great deal about these models, I might walk away thinking that the models were essentially useless and perhaps worse than using nothing at all. The author cites well-known experts in making her argument about the limitations in Monte Carlo models, but she never notes that these same people are strong advocates for the use of Monte Carlo Simulation (even with its limitations).
William Bernstein, one of the experts, was one of the early advocates for the use of these tools in retirement planning. Moshe Milevsky, also cited as a critic of Monte Carlo, devotes an entire chapter of a recent book to the use of Monte Carlo Simulations and advises investors to “ask your financial or investment advisor to generate a Monte Carlo illustration of your financial future.”
Another problem with this article is that it focuses exclusively on the issue of ‘fat tails’ and ignores other factors that drive the outputs of these models. The ‘fat tail’ problem refers to the fact that equity market returns generate extreme events (both good and bad) with higher probability than a model that assumes Gaussian returns predicts. The shorter the time horizon, the more this problem is evident. This often occurs because of momentum effects—momentum creates fatter tails in returns.
It is true that fat tails are an issue of concern in Monte Carlo models, but there are a range of other issues that should be of just as great or more of a concern when looking at these models. Monte Carlo models must generate not only the probability distribution of returns (where one may look for fat tails), but also (1) the standard deviations in all assets, (2) correlations between assets, and (3) expected returns for all assets. These three factors will have at least as great an impact on long-term planning as the method used to generate the probabilities of extreme events. When combined together, these variables determine the construction of an “optimal” portfolio.
The fat tails issue is only one part of what makes a good model. Any investor or advisor considering the use of Monte Carlo models would do well to examine the three factors listed above before worrying about how the model captures ‘fat tails.’ FinancialEngines (a well-known provider of Monte Carlo simulations) projects that emerging markets have lower long-term expected returns than domestic equities, whereas Quantext Portfolio Planner projects that emerging markets have higher expected return and higher future risks than domestic equities. Financial Engines projects that small cap stocks will not out-perform large-cap stocks to any meaningful degree over the long-term, whereas Quantext Portfolio Planner consistently shows a size effect in which riskier small-cap stocks out-perform. These sorts of differences will typically have a substantial impact on portfolio planning.
The irony of the WSJ article in suggesting that Monte Carlo tools have failed investors in this time of crisis is that these tools tend to (1) discourage exceedingly risky portfolios, (2) encourage higher savings rates (pdf), and (3) encourage the creation of actively diversified portfolios (as opposed to the ‘passive diversification’ benefits from simply buying some of everything). The first of these is especially important in tempering investors’ tendencies to chase performance. One of the general features of Monte Carlo simulations is to suggest that it is impossible to obtain more than some threshold level of return for a given level of risk over extended periods of time. On that basis, simply betting on asset classes that have generated very high returns over recent years is a mistake. Quantext Portfolio Planner suggested that most major asset classes (and especially real estate and emerging markets) were overvalued in 2007, for example.
While the WSJ article implies that MCS is not useful in its current form and even cites recent research by retirement planning expert Moshe Milevsky in apparent support of this argument, that is not the conclusion that Milevsky reaches in his research. To the contrary, Milevsky’s article (pdf) cited by the author of the WSJ article could be seen as a basic refutation of the WSJ article:
“Here is the bottom line. Instead of condemning the entire Monte Carlo Simulation industry for missing the meltdown, let’s take this unique opportunity to properly harness the full power of stochastic methods.”
Milevsky proposes that investors and advisors should stress test Monte Carlo models by looking at how much a bad sequence of years (he chooses 3 years for his examples) would impact an investor’s long-term plans. Milevsky proposes a simple metric to capture this effect from Monte Carlo output that he calls the Sequence of Returns Downside Exposure [SORDEX]. The measure of the “extreme” event that he proposes is the projected 1% worse event from the Monte Carlo model. Conversely, one might simply choose to use a specific period as the stress test (2007-2008 perhaps). Milevsky’s point is that extreme events can have a major impact on long-term plans and that investors need to be aware of the fact that 1-in-100 events happen and to be prepared to survive such a scenario. I have been doing tests of Milevsky’s stress test metric inside Quantext Portfolio Planner and I am quite impressed at the utility of this approach. Any good Monte Carlo model should support this form of test.
In summary, I believe that the WSJ model does a disservice to investors and advisors in implying that Monte Carlo simulation and other probabilistic models are fatally flawed because they “did not highlight” a scenario like what happened in 2008. There are many ways in which portfolio analytics for long-term planning can be improved—and these models will evolve and improve over time—including with the addition of ‘fat tailed’ outcomes. Whether with additional analytics to focus on extreme events or not, investors should stress test their portfolios using a range of conditions, and I think that some form of Milevsky’s SORDEX test is an excellent approach. 2008 notwithstanding, Monte Carlo simulation and related planning tools have helped their users to estimate the interplay of diversification, longevity risk and market risk in the process of building better portfolios and financial plans than they could have otherwise.
Related Articles
|
-
The Seeking Alpha 100
See allThe top 100 stock
market authors
selected for publication
Seeking Alpha 100 » -
Fastest Climbers
Authors with the largestSee all
increase in followers
in the last 7 days
Fastest Climbers » -
Top Commenters
Top commenters,See all
as ranked by their peers
Top Commenters » -
Top Instabloggers
The 100 most active
Instabloggers.-
1
David Fry
-
2
TraderMark
-
3
Don Dion
-
4
Cliff Wachtel
-
5
Mike Havrilla
Top Instabloggers » -
1
-
Top StockTalkers
TopSee all
StockTalkers.
Top StockTalkers »
This article has 13 comments:
But the problem is not the modeling. The problem is that a once-in-a-century event is thought of as a remote improbability, because "that hasn't happened for many years and everything is different now." Actually, an investor with a remaining life expectancy of 30-35 years has about a 1/3 chance of encountering a 100-year improbability. That is a substantial enough risk that some specialized control device should be added to the plan. If only we could know what that would be.
Be proactive in managing your money or hire someone that will be proactive for you. Don't expect things to go smoothly while you let things run on automatic pilot.
The subtle point I am trying to make is that the real issue is using these tools correctly. Anyone using my MC platform can easily demonstrate this for themselves.
Investors and advisors have a somewhat uneasy relationship to MC models. They are far from perfect, but they are the best tools available. Milevsky's article is must-read stuff for anyone who worries about long-term sustainability of income.
Milevsky's proposed SORDEX statistic is a great solution. The more I work with it, the more impressed I am.
G.
On Jun 03 10:17 PM Alan Young wrote:
> The average investor, being told that their portfolio had a 95 or
> 98% chance of sustaining its value throughout their remaining lifespan,
> would be satisfied. There are no 100% guarantees.
>
> But the problem is not the modeling. The problem is that a once-in-a-century
> event is thought of as a remote improbability, because "that hasn't
> happened for many years and everything is different now." Actually,
> an investor with a remaining life expectancy of 30-35 years has about
> a 1/3 chance of encountering a 100-year improbability. That is a
> substantial enough risk that some specialized control device should
> be added to the plan. If only we could know what that would be.
On Jun 04 12:35 AM Jonas Zamora wrote:
> The bottom line is that these tools cannot fully predict everything.
> Humans make assumtions and input these assumptions into the software
> programs. Therefore, depending on these tools alone to predict outcomes
> makes it a futile excercise.
....
> Be proactive in managing your money or hire someone that will be
> proactive for you. Don't expect things to go smoothly while you let
> things run on automatic pilot.
I would add to what Jonas said - you personally handle the Macro view for your money; your advisor(s) too often will be too caught up in their own agendas to clearly see what is right for you, as I found from (expensive) personal experience.
Great comments all. I would like to see more discussion of the roles and duties of advisors here on SA if that would interest people. This is a major issue and I think that advisors and individual investors would benefit from more dialog. I think that really good advisors help their clients to understand the responsibilities on both sides of desk. I am quite convinced that good advisors add considerable value for many investors. That may or may not include making tactical decisions--depends on the advisor and the client. This debate is very similar to what has gone on in the institutional world for several years.
Some investors want to hand over their money and have no involvement whatsoever. Some investors are proactive. Many of the users of my Monte Carlo platform are individuals and quite a few have introduced their advisors to our tools rather than the other way around.
The models and the use of models is one part of the discussion, but this is a bigger topic obviously.
Regards,
Geoff
Could you give a novice some guidance on how to QPP to stress test? So far I've been just using it to generate the projected return/risk for my portfolio at the end of each week. I than look at the table showing the 80%/50%/20% probablilties. I use the SP # as suggested 8%/15%.
Thanks,
Bob
On Jun 04 08:00 PM Geoff Considine wrote:
> Thanks for your comments. I very much reccomend the article by Milevsky
> linked above. His point is that MC models provided strong warnings
> prior to 2008 if people stress tested the results--as he shows. When
> I do the same tests as Milevsky, I get even more extreme warning
> signals using data available through 2007.
>
> The subtle point I am trying to make is that the real issue is using
> these tools correctly. Anyone using my MC platform can easily demonstrate
> this for themselves.
>
> Investors and advisors have a somewhat uneasy relationship to MC
> models. They are far from perfect, but they are the best tools available.
> Milevsky's article is must-read stuff for anyone who worries about
> long-term sustainability of income.
In my opinion the thing in the monte carlo engine that should have given this away was was the ever declining S&P volatility during the period following the burst of the tech bubble. We went through recession and we went through 2 wars yet the volatility declined. THAT decline was due to the systemic bias that was added by the government which gave us the sandbox where the Tech bubble was turned into the real estate bubble, and that is now being turned into the currency/credit/govern... bailout bubble Another systemic risk, and another reason to NOT invest in America
To blame monte carlo for government violation of the rule of randomization is to absolutely NOT understand what a monte carlo engine does. The example of engineering a bridge for the 100 year flood is the perfect example. You can engineer for that probablility, but if you systematically undermine the quality of your concrete (as in adding systemic risk) your bridge is built on a fraud that is not the fault of the model.
I like your examples but I also suggest that you read Milevsky's paper. You raise a number of important issues. First, the cycle in volatility. Volatility was very low, so people took bigger risks--and this led directly to the crash. This process is well documented in Minsky's instability hypothesis. Quantext Portfolio Planner clearly docuemented this risk well before the crash--and I have written many articles on it here at SeekingAlpha.
Second, Monte Carlo models are attempting to capture a series of unknown future factors in terms of cumulative risks. We do not need to know specifically what these are. I disagree that the recent market crash is beyond the realm of statistics to capture (bad cement vs. 100 year flood). We cannot predict what will happen in terms of the extreme events but we can stress test our plans to see if they will hold up in extreme events--thats the point of my article.
On Jun 08 11:05 AM gasem wrote:
> The issue of the past melt down IS UNRELATED to a probabilistic thing.
> The melt down was due to systemic risk. In effect the market was
> rigged. AAA bonds were NOT AAA loans were being made to people who
> HAD NO ABILITY TO EVER PAY THEM BACK. Government policy was such
> that there was a bias to do financially unsound things. Everyone
> knew this was going on EVERYONE. Because there was a huge amount
> of money to be made the bet was "I'll play the game make my huge
> amount of money AND I'll be smart enough to get out before it crashes."
> Virtually no one was smart enough, so the market unwound all at once.
>
>
> In my opinion the thing in the monte carlo engine that should have
> given this away was was the ever declining S&P volatility during
> the period following the burst of the tech bubble. We went through
> recession and we went through 2 wars yet the volatility declined.
> THAT decline was due to the systemic bias that was added by the government
> which gave us the sandbox where the Tech bubble was turned into the
> real estate bubble, and that is now being turned into the currency/credit/govern...
> bailout bubble Another systemic risk, and another reason to NOT invest
> in America
>
> To blame monte carlo for government violation of the rule of randomization
> is to absolutely NOT understand what a monte carlo engine does. The
> example of engineering a bridge for the 100 year flood is the perfect
> example. You can engineer for that probablility, but if you systematically
> undermine the quality of your concrete (as in adding systemic risk)
> your bridge is built on a fraud that is not the fault of the model.
>
>
My next article shows how to stress test--step by step--in QPP or in any other Monte Carlo model...
Geoff
On Jun 08 10:23 AM BobLSW wrote:
> Geoff,
>
> Could you give a novice some guidance on how to QPP to stress test?
> So far I've been just using it to generate the projected return/risk
> for my portfolio at the end of each week. I than look at the table
> showing the 80%/50%/20% probablilties. I use the SP # as suggested
> 8%/15%.
>
> Thanks,
> Bob
Engineers do stress test the concrete to prove the model. They take some concrete and put it in a press and stress it till it breaks. It is that test that quantifies the risk.
Certainly there will be any number of frauds (rating a bond AAA when it is well below that for example). My point is if you invested and stayed invested in Madoff, you had a 100% chance of failure, while the statistical engine would have I'm sure come in well below a 100% chance of failure.
The way I see it is here is the kind of test you must design: You must design a test that looks at the probability that 1,000,000 cabies who buy million dollar mortgages on houses worth 500,000 dollars will be able to pay for that, or what is the probability that a government that has promised its citizens all the money in the world in 30 years can make good on that promise. That is precisely what the engineer tests when he puts the concrete in the press.
I do not see that just putting in the equation that along the course of time there will be some event like the great depression is enough to improve the reliability of the outcome. For example you could score the great depression as 1 event or 2 events or 3 events that occurred in lets say 3 years 10 years or 16 years and how you score it will make a big difference in what your model calls a "catastrophe". You could as well call this present "catastrophe" as the ongoing result of several events going all the way back to Carter and ongoing through the next crash in 5 years we will experience due to the "government debt" bubble we are now entering.
By this I do NOT mean in any way I am against the monte carlo model or stress testing that model. I'm fully invested according to the risk analysis this model provides I think its the best we have till we make it better and that is why I take the side I do. You can aggressively test stresses if you aggressively frame the boundaries. Planning for the 100 year flood is a good thing, but your bridge won't stand if you forgot to plan for the earth quake. To test for the earth quake engineers shake the heck out of the model to see when it will break. Testing the concrete is testing the assumptions of the model. Shaking the heck out of it is testing the modes of failure within the model.
www.youtube.com/watch?...
How many times have you heard "if they hadn't let Lehman fail!!" Lehman was a long time dead already. It just hadn't started yet to stinketh.
On Jun 09 11:01 AM Geoff Considine wrote:
> Gasem:
>
> I like your examples but I also suggest that you read Milevsky's
> paper. You raise a number of important issues. First, the cycle
> in volatility. Volatility was very low, so people took bigger risks--and
> this led directly to the crash. This process is well documented
> in Minsky's instability hypothesis. Quantext Portfolio Planner clearly
> docuemented this risk well before the crash--and I have written many
> articles on it here at SeekingAlpha.
>
> Second, Monte Carlo models are attempting to capture a series of
> unknown future factors in terms of cumulative risks. We do not need
> to know specifically what these are. I disagree that the recent
> market crash is beyond the realm of statistics to capture (bad cement
> vs. 100 year flood). We cannot predict what will happen in terms
> of the extreme events but we can stress test our plans to see if
> they will hold up in extreme events--thats the point of my article.
>