Stress Testing Your Portfolio

Monte Carlo portfolio planning tools allow investors to account for the effects of volatility on their long-term plans. These models simulate many possible future outcomes and calculate the probability that an investor’s portfolio will be able to provide a long-term income stream or that it will meet some other goal. These models must generate statistical projections for the future risks and returns of all assets in a portfolio, as well as accounting for the relationships between asset classes.

In light of recent years—especially 2008—there has been growing misconception that these tools were inadequate to show investors the potential for substantial losses and thus are of limited use in long-term planning. This may be true for some Monte Carlo models, but certainly not all. In this article, I demonstrate a straightforward way to stress test these models, using Quantext Portfolio Planner (QPP) as an example. The technique that I will present is related to Moshe Milevsky’s SORDEX ratio (pdf). The basic idea behind stress testing is to see how some extreme but possible event will impact long-term plans. Milevsky proposes the following:

1) Design a plan that meets your criteria and has a high probability of meeting your goals using the Monte Carlo Simulation

2) Run the Monte Carlo simulation three years forward

3) Look at the worst 1% outcomes from the Monte Carlo Simulation

4) Calculate the probability of meeting your goal assuming that three years have passed and the worst 1% outcome occurs

5) Look at how much the probability of success has declined from the original analysis

This approach makes a great deal of sense—it is a fairly simple stress test, and simply requires two runs of the Monte Carlo Simulation rather than one. Milevsky shows that a Monte Carlo Simulation run through late 2007 would have clearly demonstrated that a heavy allocation to equities for a recent retiree was very risky under this stress test, even though the baseline run from the Monte Carlo Simulation showed a high probability of success in the original analysis. The motivation for this approach is that people should plan to be able to survive an event with a 1% probability. We buy homeowners insurance which will replace our homes and contents if the house is destroyed, even though this event has a very low probability of occurring. There is a further notion that motivates the stress test. Very low probability events that have severe consequences for one’s well being should be emphasized in planning. There is actually a huge literature on this issue that goes under the name ‘utility theory.’

I am a proponent of stress testing of any planning model, and Milevsky’s article is very timely in this regard. I have developed a closely related stress test that is easier to apply in most Monte Carlo or other planning models. I will demonstrate this using Quantext Portfolio Planner.

Imagine that we are standing at the end of 2007 and trying to build a plan going forward. First off, there were a variety of reasons to believe that most asset classes were over-valued. If we were building a plan in late 2007, how would this go?

Step 1: Build a Strategic Asset Allocation Plan

As always, we need a straw man—and for today, it’s Bob. Bob is 65 and has just retired. He has $1M in invested assets and plans to draw $50K per year, increasing by 3% per year to cover inflation, for his retirement—and he is retiring now. Bob has a fairly generic portfolio—and we have analyzed this using QPP with all default settings and data only through 12/31/2007 to see what the world would have looked like back then.

Bob’s portfolio is shown below:

click to enlarge

The Portfolio Stats box shows the projected future risk and return for this portfolio, and the Average Annual Return column shows the projected average annual return for each asset class.

Based on this allocation, QPP generates the following probabilities that Bob will run out of money by a certain age:

Bob looks like he is in good shape. He has a 20% chance of running out of money by age 93—or, conversely, an 80% chance of making it to 93 without running out of money.

This process is how Monte Carlo is usually run—we generate survival probabilities. These results depend on the portfolio, as well as other information about the individual. The main variables are projected annual savings vs. income draws each year.

Step 2: Estimate the ‘worst case’

The next step is to determine a realistic ‘worst case,’ given that we cannot really know the absolute worst case. I believe that the following process is a good baseline stress test.

1. The expected annual return over an N year period is estimated to be:

(8.7% - 5%) * N

in which the 5% is the estimated draw rate ($50K / $1000K) and the 8.7% is the expected return.

2. We are going to use a 3 standard deviation event as our estimated worst case, which leads to the following downside risk estimate over a 3-year period:

-3 * 10.6% * N 1/2

The factor of three in this equation is because we are looking at a 3 standard deviation event. The 10.6% is the estimated future standard deviation for the portfolio. The square root of N comes in because the standard deviation of a random walk increases as the square root of time. Our total estimated loss over the ‘worst case’ one year period is:

8.7% - 5% - 3*10.6% = - 28%

Step 3: Simulate probability of success, assuming the worst case comes to pass

If the worst case occurs, Bob has only $720K after the next year. When we look at QPP for Bob at age 66, with $720K, his odds of being able to sustain his desired income have plummeted:

Bob now has a 20% probability of failure at age 82—11 years earlier than in the original calculation. Bob has a 50% chance of running out of funds by age 88. It is something of a standard pf practice to manage to an 80% success rate / 20% failure rate, which is why I am focusing on this percentile. The loss of eleven years of funding at this percentile is the impact of the very bad 1-year event.

Milevsky looks at how the probability of failure changes, and I am looking at how many years of projected retirement you lose at the targeted percentile, but the process is the same. I find the number of lost years to be a good basic metric. If you are not confident of survival in the event of this worst case, it is a good idea to consider alternative.

It is important to keep in mind that we have assumed that Bob will continue to draw his $50K in current dollars, no matter that his portfolio has lost 28% of its value.

Step 4: Determine whether the worst case is survivable

In real life, Bob is not going to keep on spending at the same level after a major reduction in his portfolio value. What Bob needs to determine is whether this worst case is survivable. Obviously it has significant odds of not being survivable without some reduction in spending. Bob determines that in this ‘perfect storm,’ he could reduce his living expenses by 25%. We then re-calculate his odds of survival:

If Bob could reduce his annual draw from this portfolio to $40K per year in the event of the worst case outcome, he is now back up to a 20% probability of running out of funds by age 88. If Bob considers this to be acceptable in the event of a ‘Black Swan’ event, his plan is in okay shape. If Bob wants to get all the way back to his original odds of success, he will need to either (1) change his asset allocation, or (2) be willing and able to further reduce his income in the event of the worst case outcome.

Step 5: Mean reversion

I noted near the start of this article that QPP projected that almost every asset class was over-valued as of the end of 2007. Bob’s portfolio was only moderately exposed to this over-valuation because of his 50% allocation to bonds, but his portfolio is projected to generate an average return of 8.7% per year and the most recent three years have returned an average of 9.2%. This suggests that a period of lower-than-expected returns is likely so that the long-term average will converge towards the expected value. This portfolio would need to return an average of 8.2% over the next three years to bring the long-term balance into equilibrium (to average the 9.2% with 3 years of lower returns and get an average of 8.7%). On the other hand, things could revert to the mean a lot faster (as they did in 2008), but we cannot know that ahead of time. For a more extreme test, we might assume that the 3-year excess returns will mean revert over the next year, in which case the expected return over the next year is:

4*8.7% - 3*9.2% = 7.2%

so the new ‘worst case’ estimate becomes:

7.2% - 5% - 3*10.6% = - 29.5%

This will make the probability of running out of funds even worse. In Bob’s case this is not a huge effect. In the case of investors who were overweight sectors like emerging markets, this would have been far more severe.

There is another factor that is worth noting: mean reversion in volatility. The trailing volatility for this portfolio for the three years through 2007 was 6.03%. The projected volatility was 10.6% (see the QPP output shown earlier). If we assume that volatility also mean reverts, we would need to adjust the projected volatility considerably upwards (as we adjusted return down). I will discuss this topic in a future article.

Summary

In 2008, Bob’s asset allocation lost only 14%, but his total portfolio value is down 18%-20%, depending upon the schedule upon which he drew his income. This is more like a 2-standard-deviation event than our estimated ‘worst case’ 3-standard-deviation event. If Bob had run the kind of stress test described above in 2007, he would have had a good idea of his potential exposure to an extreme event—and this is the whole point of stress testing.

Embedded in this process, there are two important ideas. First, a 3-standard-deviation event is a 3-in-1000 event based on the assumption of Gaussian returns (no fat tails). In reality, however, we don’t know the true probabilities of these events. Even if returns are Gaussian, we have what is sure to be only a rough approximation of the true standard deviation of the returns—and this estimation error will raise the probability of under-stating the risks. Thus, the use of a 3-standard-deviation event is a reasonable stress test. If we incorporated the mean reversion in volatility, the projected 28% loss would have been estimated as far more probable than the 3-standard-deviations, of course, but using this as a basic ‘worst case’ that needs to be survivable.

The second important idea is that the consequences of an extreme outcome are so bad, that we need to pay attention to this kind of event, even if it has a very low probability. As I noted earlier, this is an outcome of utility theory.

The simple equation that I provide for estimating worst-case scenarios for stress testing of financial plans will make it easier to apply this approach to various models. Most planning tools do not provide the estimated 1% tails from which to estimate a ‘worst case.’ The reason for this is quite sensible: the estimation error of 1% tails is very high because of model uncertainties and limitations. A more substantial problem is that very few Monte Carlo or other planning tools (with QPP being a notable exception) allow the user to see what the model would have projected from some earlier date.

The number of lost years at the 20th percentile survival rate provides an intuitive metric for seeing the impacts of extreme events, and you can look at extreme events on various time scales. I used 1 year in this case, but Milevsky uses 3 years in his example. While he does not say so, he probably chose 3 years because this is a time horizon on which statistical models tend to do a better job than one year. Putting it another way, fat tails are less extreme on 3 year time horizons than 1 year.

The bottom line, as it were, is that stress testing can help investors and advisors make sure that extreme events are survivable. This does not mean that these models have a magic formula for predicting the ‘true’ probability of events like 2008, but rather that these stress tests provide ‘emergency drills.’ An important take-away is that the potential impact of extreme events is determined by the specifics of the investor (age, income flexibility, planned income, risk tolerance) and the portfolio. Our model investor, Bob, is at the highest risk point in his investing career, the years surrounding the onset of retirement. The long-term impact of a severe market downturn will typically be lower for both older and younger investors.

Comments

[Living4Divi...:]

Nice article, thanks

Jun 12 12:32 PM

[Analyste de...:]

A few thoughts:
1) "Events like 2008" started in October 07 and might not be over (March 09), but focusing on the WORST possible outcome is what everyone should look at during Market peaks (March 00, Oct 07.) They and their advisors rarey do!

2) "[At age 65] Bob’s portfolio was only moderately exposed to this over-valuation because of his 50% allocation to bonds" doesn't reflect most investors' reality AT ALL. In truth, there were many, many more +65yo Buy&Holders with >70% allocation to eqty in 2007. Playing catch-up from the last "event" (2000-02.) If people were more prudent, if only people were saving more, if only people were better diversified... too many ifs.

3) "Age 65" is arbitrary (the Boomer wave starts in 2011) and the 50/50 allocation wasn't realistic either. Bob has $1m in retirement savings, and "we have assumed that Bob will continue to draw his $50K in current dollars" ... Can we ask, how reasonable are those numbers? If you're running survival seminars for Mass Affluents (yes, that's what they WERE called), then reduce the savings by 50% and raise the draw by 50%.

4) Not to pour on your parade, but I'm afraid taxes & inflation will render these presumptions abit unrealistic, too.

This article is written with the best intentions, I know, but why do I have the sense that "retirement planning" for MOST people is now a bad joke? Bob has to be much wealthier (probably older) and likewise have a greater draw and appetite for risk, to be a Client. Or you need to ratchet down, and think "poverty counseling" - there's less money there! Either way, "Bob" needs to be closer to reality with numbers that showed his TRUE age, wealth, experience.

God help us all if hyperinflation hits in 2012-16 or the currency fails sometime in the next decade or so. Stowing away 10-15% in gold bullion certainly isn't as "nutty" as it sounded in 2005 - just look at how much things have changed! So we go back to basics, the frugality of our forefathers - can we Monte Carlo that?

This is the curse of interesting times. Almost all the quant models failed, didn't they? False premises? Mom & Pop investors who believed the 'Buy & Hold' lie lost HALF their invested assets, to say nothing of their property devaluation or remortgaging to buy flashy junk. Now they'll have to re-pay the Pied Piper, somehow. Or do we make-believe it'll all go back to the way it was? Why?

Shouldn't we start with a more accurate description of real investors' experiences?

Jun 12 01:09 PM

[F. Bradeen:]

You have to be kidding me. Isn't all this sophisticated nonsense that got us into this predicament in the first place? Isn't these just quant models that brought down LTCM and now our banks? Ask Tabel, who you quote, whether he agrees with our analysis? More 3 standard deviation rubbish of a defunct normal distribution. It also sounds great, after the fact. But before the fact, you guys were pushing buy-and hold and still are. Any advisor who let their customer's portfolio decline by 50% should be taken out and shot.

Jun 12 01:10 PM

[billb123:]

You guys have some misguided anger. What brought down LTCM was leverage and arrogance, i.e. that the 1% would never happen to them. The author is trying to get someone prepared for the highly unlikely. A tool like this helps quantify the risk that your portfolio is under. If you don't like the numbers, change your allocation until you are comfortable. Is a perfect predictor? No. Does it help gain some vision as to how one's holdings MIGHT behave in the future? Yes. Note: not affiliated with the author, don't use the product.

Jun 12 01:28 PM

[Analyste de...:]

No anger here, just a reality check.

Most 2015 Lifecycle funds (marketed as a "set-it-forget-it" option) lost about -43% (10/9/07-3/9/09).
FFVFX -43.8%
VTXVX -41.6%
TRRGX -48.6%
TCNIX -43.1%

In aggregate, that's also the institutional allocation for products sold to Baby Boomer retirees (born 1946-1950.) For private investors with the relevant time-horizon and risk-tolerance, target date/lifestyle products are a simple benchmark - if you pay a financial advisor to get the same or lower return, you really should shop around for better counsel.

Keep the focus: a competent advisor needs to deliver risk-returns significantly better than retail offerings. S/he can't just provide a clever hypothetical model to dupe the sheeple. What else is the Client paying for, if not superior advice & performance?

There are times to take your clients' money out of (or into) the market, to deliberately & intentionally reduce (or add) risk. Those religiously following quant models got very badly burned - and honestly, its hard to see lots of converts flocking back tomorrow.

Jun 12 02:19 PM

[Jonas Zamora:]

I agree with Analyste de Boston. Trying to package 1% events in a software prigram is the same as putting together cookbook asset allocations for retail investors.

In the real world, you have to actively manage portfolios with discipline and thought. Portfoklios must be managed actively, not just asset allocated and rebalanced once in a while.

Why would I want a portfolio that is projected and built to return 7-8% a year and have a chance of losing 40%? Those are terrible reward/risk numbers. That's what asset allocation/rebalancing portfolios has become.

Jun 12 03:34 PM

[Graham and ...:]

Monte Carlo simulation isn't perfect. But it remains one of the better (known) ways to stress-test a portfolio (investment program), etc. I'm all for using what we have. If someone believes they know of anything better, I'm all ears.

Jun 12 04:00 PM

[gasem:]

I like the idea of stress testing. I would stress test as above and I would in a different way. In the recent crash most "losers" were likely invested in equities, with bonds or cash being the "safe" position. However using QPP you can "what if" inflation. In that case bonds and cash are the unsafe investment, (also in the case where the market is about to explode I think beta auto-correlation and diversification metric say something about that) Thirdly you can "what if" your draw. The way I did that is to bring the draw down until all the top 8 "ages of failure" were equal except for the worst case, the 10% failure rate, which I allowed to come off the the max level. In doing the above 2 tests you can use QPP to predict well I think a very robust retirement. You can double inflation and still not break the bank for example. This analyzes risk in a diversified way for the true black swan. This past downturn in my opinion although painful is not a black swan event.

My guess is the "failure" of monte carlo engines is not in the engine but in the suppositions that are too rosey. The risk that is focused on is the risk of market failure, but the more real risk that tends to be ignored is the risk of taking out too much money too soon. This is the true value of this kind of analysis. If you understand the risks or your advisor helps you to understand the risks I still believe this "tool" to be the best we have. The fact the paper reported the engine didn't "save" you merely means you mis-used the tool and chose too much risk in the draw.

Probably the way to do this also is to re-evaluate every so many years and see how much money you have left and then project your future viability the way it was done above with re-evaluating for a new (older) age, also as was done on Milevsky’s article I read something by Stein and DeMuth that looked at this periodic reevaluation strategy and I think its sound. Its kind of like doing integration by Riemann sums. The smaller you make the period the less error in the integral.

So to recap a tool like QPP can help you determine what is a rational "draw" for a given nest egg, or it can help you determine what the needed nest egg is for a given draw stress tested against catastrophic loss in the market and catastrophic inflation, or a true black swan like changing the reserve currency from $ to Yuan or from $ to oil.

Some examples: if you double inflation you need to double the size of your nest egg to not go bust especially if the market is sluggish and you want to keep your draw the same. The effect of doubling taxes is similar to doubling inflation. If you ask yourself what pot of money has yet to be taxed the answer is retirement funds. Hence the tax rate on retirement funds could be an area of undefined risk. After all we MUST do our "patriotic part". I think QPP is still the best bet as a tool to help protect yourself, you just have to be far more creative in investigating scenarios that could kill you. Certainly all the arrows pointing into the ground can do it. But so can someone taking your money either by running the printing press or by the IRS or by bond yields up to Carter's level.

Maybe others can see other scenarios which QPP might help predict success. The way we protect ourselves from new paper criticism is to have money on our accounts when every one else has IOU's

Jun 12 04:30 PM

[Ron Rowland:]

At the end of the first year of his new plan, Bob was down -19.9% before fees and expenses. Assuming he rebalanced on 12/31/08, then he lost another -8.7% in 1Q 2009.

After five quarters, Bob is down about -27% in this conservative allocation assuming no fees and no withdrawals.

Question: Is Bob still on the plan or did he bail out?

Jun 12 06:05 PM

[Jonas Zamora:]

No one can predict the future. Let's remmeber that financial planning and investment management is as much about art as science. Humans beings are inputing the assumptions on these models and stress tests. You would have to be all knowing to make them predictive and I don't know anybody on earth who is such.

Jun 13 12:31 AM

[RiskReturnO...:]

"Running out of money" is a relative measure ... for US investors, they could have enough money to cover for inflation in US, but a weaked dollar means erosed purchasing power if they ever want to retire in Europe, China, or somewhere else.

Somehow, purchasing power normalization or currency normalization needs to be modeled; otherwise, you risk losing your "relative wealth" to immigrants from China and India if their portfolios are US $ neutral.

In parts of California (many of the weathier cities like Cupertino, Fremont, etc), you are already seeing "non White populations" becoming majorities, as they have grown their wealth measured in US$.

Jun 13 01:27 PM

[Analyste de...:]

Ron, if your "Bob" is a generic/typical retail investor paying an advisor nothing, he's invested in mutual funds ("plan" = 401k) and lost -27% in a conservative allocation, you can chart those MF tickers listed above in Googlecharts, from 12/31/07-3/31/09. Those 2015 allocations lost between -31% to -35% in that period, so your "Bob" did somewhat better and has less to regain : +37% to breakeven.

fwiw, if Jeff's "Bob" was put in a 50/50 allocation and only lost about -24% for the entire Bear Period (I'm extrapolating here) that client is probably much, much better off than most investors of his age/wealth/class.

But I don't believe that 50/50 allocation has been 'optimal' for the rally that many predicted, and IF this is a new Bull Market that allocation won't be satisfactory for long. In asset allocation for the upper-end of the Mass Affluent mkt, the "set it, forget it" days are over. Advisors will need to be much more proactive in managing both risk & return, and prove their value going forward.

My own experience with Monte Carlo simulations was a disappointment (thank god I didn't follow it!) but the program wasnt QPP. I have an issue with the black box features of all quant models, the presumptions & premises behind the output, and until my own performance lags significantly I won't be too interested.

fwiw, I also co-worked with Milevsky on a project for a major insurance company. Although I have the highest respect for his work, go back and see what kinds of allocations he was advocating in 2003-07. (He seemed far more risk-tolerant then; I wasn't.)

Jun 13 03:29 PM

[Fred Swartz:]

Interesting idea for evaluating buy and hold portfolios, but assumptions about return, volatility, and correlations will change these results dramatically. And we know how well 2006's assumptions apply today.

It would be informative to see what results for 12/2006, 12/2007, and 12/2008 would have been.

Jun 13 04:00 PM

[gasem:]

I don't see how using QPP implies a "buy and hold" strategy. In fact Bob's problem was he didn't use QPP to good effect.

1. His portfolio only had about 25% diversification

2 His draw was way too much for the assets he held.

By making a few changes to the asset mix the diversity of Bob's assets could have been raised to 45% and Bob's "losses" over the three years from 6/06 to 6/09 would actually have been slightly positive. In other words he would have made a tiny bit of money.

By stress testing his draw, and adjusting it for something safer, the longevity of his portfolio was dramatically improved. He didn't live quite as luxuriously, but he had money well into his 90's even with higher inflation. Making these changes are form of active management, they just aren't trading.

In fact a program like QPP merely attaches some kind of risk analysis to a gain and a draw. Many of the so called "active managers" just watch Cramer and call that risk analysis.

Jun 14 01:37 AM

[Ron Rowland:]

Analyste de Boston, my Bob and Geoff's Bob are one and the same. I believe there is an error in Geoff's following statement:
"In 2008, Bob’s asset allocation lost only 14%, but his total portfolio value is down 18%-20%, depending upon the schedule upon which he drew his income."

There was also an error in my original numbers. After double checking, Bob's return in 2008 was -18.3% before fees and withdrawals and assuming the reinvestment of dividends. Run the numbers yourself, you will see that the -14% number is incorrect.

After Bob's $50k withdrawal, his 2008 portfolio is down -23%. At the end of Q1 it is down -25% assuming NO withdrawals, down about -30% assuming one $50k withdrawal and down about -35% if he took his second $50k withdrawal.

At that time, it is extremely unlikely that Bob changed his allocation to be more aggressive and participate in the rally.

Remember, Bob was 65 and just retired with $1 million. 15 months later, at the end of 1Q 2009, his portfolio value was down between -30% and -35%. His $1 miilion portfolio, that he was told would average +8.7% per year ($87k in his terms), is down $300k.

At that time Bob is scared to death. I bet Bob shifted many of his equity allocations into CDs paying about 2% about that time.

Monte Carlo simulation is all about odds, and I put the odds at less than 10% that Bob said "I've just seen $350k of my $1 million portfolio wiped out in the first 15 months of my retirement. Let's change the allocation to be more aggressive going forward."

You can run all the paper stress tests you want. You can show investors the stress test results and show them there is a very high probability that their portfolio will experience a 30% drawdown at some point.

But if that big drawdown happens in the first year, then the "real stress test" fails. No paper stress test reveals the emotional implications and the reality of actually seeing your retirement plan ripped to shreds in the first 15 months.

Bob's $50k withdrawals now equate to 7% of his new portfolio value and the bulk of his assets are yielding 3% or 4%. He is now very vulnerable to annuity salesmen and their "guaranteed" 7% annual payments pitch.

You don't have to take my word for it, just ask Bob.

Jun 14 10:26 AM

[Analyste de...:]

Ron-
I think your assessment is spot-on. I would only add that for investors in the game since the Bear Market began, losses would be that much greater (-4% 4Q07, plus 1 presumed withdrawal compounds to... -40%?) Grimmer than first imagined, unfortunately.

My Model is UP +61% in the Bear Market, glad I ignored Monte Carlo projections in defining my allocations. Instincts served my clients much, much better!

Jun 14 12:13 PM

[gasem:]

So what you're saying is buy high sell low doesn't work so good

Good advice

Jun 15 07:48 AM

[Geoff Consi...:]

Hi guys:

I enjoyed the comment stream today, after being away for the weekend. What I find most striking is that this topic has not gotten more attention in the media. There was the really poor WSJ article on Monte Carlo, but you guys are really hashing out the important issues. I agree with the comment on market discipline--the hardest thing for people to do is to stay with a plan--if they have a plan. I am sure that there are lot's of Bob's in the world who sold their equities and purchased an annuity just before the market rallied 35% or so in early 2009. I think that Monte Carlo tools can help investors to be more cognizant of thr risks and thus choose a level of risk that they can live with. Back when QPP was projecting a doubling in volatility over recent years (in 2007), I saw very little commentary on this issue. THAT was the time to prepare. After the crash, there are many people who have been crushed and I am well aware that what Gasem says is true: there were many people aged 60+ with very high allocations to equities--and largely undiversified exposure to equities, too. Monte Carlo is not a crystal ball, but it can provide guidance that helps investors avoid taking too much risk. Also, I am totally sympathetic to the ideas that there are much better portfolios than Bob's--I agree. The specific portfolio is simply a straw man.

Also, Analyst de Boston says that his market timing made his investors 61% in this bear market. First off, I tip my hat :) Second, Monte Carlo does not mean that you do not use tactical asset allocation methods--I have written about this. In the case shown here there is no Tactical component, but this is just one case...

Geoff

Jun 15 10:20 AM

[Geoff Consi...:]

On Jun 12 01:09 PM Analyste de Boston wrote:

>In truth, there were many, many more
> +65yo Buy&Holders with >70% allocation to eqty in 2007.

Yes, but that is precisely the kind of over-aggressive stature that Monte Carlo tools like QPP help to warn against.

Jun 15 06:56 PM

[Analyste de...:]

Thanks, Jeff!

I'm not anti-Monte Carlo per se but against over-reliance and misuse of that tool ("Monte Carlo shows...") as I've witnessed other professionals discussing it. IMO, the caveats are poorly understood and not reiterated as often as necessary. But I think that issue has been addressed in this Comment stream (and perhaps in your other posts.)

I should also correct my estimation of your Bob - with rebalancing, the 50/50 allocation probably lost another -14% between 1/1/09-3/9/09. In the the FULL Bear Mkt, Bob's portfolio has been reduced -49%, including 3 withdrawals in those 17+ months.

fwiw, my portfolio performance is just that: "investment performance" not including any fees, expenses nor withdrawals. NOT apples-to-apples with the hypothetical above! Including those 3 withdrawals, my Frugal Yankee Model would be DOWN -11% from 10/9/07-3/9/09.

Best regards, and thanks again for your excellent articles!

Jun 16 03:02 PM

[gasem:]

In terms of stress testing how do you view changes in a portfolios diversification metric and auto-correlation metric

As I watch things evolve I saw diversity nose dive and auto correlation explode

Jun 16 05:18 PM

[_richard_:]

Milevsky is one of my intellectual heros--you should look at his actuarial writings as well.

For those that can make a strong case for buying bonds, you can make an even stronger case for buying life annuities with an inflation adjustment. While they don't come cheap, they provide excellent diversification and insurance against outliving one's assets.

Actuaries look at retirement differently than financial planners do--finance guys look at "do you have enough money", whereas actuaries look more at mortality and joint mortality. Especially after retirment enters one's 70s and 80s, mortality "bonuses" for annuities make very good sense.

Jun 16 06:50 PM

[Geoff Consi...:]

Guys:

Thanks for all the comments. I am running a bunch more cases right now using this stress test metric. I am looking now at the two year period ending in May 09. This fills out the picture. I will post my next article with more of these results.

Cheers,

Geoff

Jun 16 11:22 PM

[Geoff Consi...:]

I am not accounting for the increase in autocorrelation / cross correlation that occurs in a major meltdown but I am hoping that the 3SD worst estimate is still a good measure. Basically, we are saying that we plan for a really bad (<1/1000) event while ignoring autocorrelation/correl... increases and this is used as a reasonable estimate of the worst case that investors must plan for, given that we know that this will under-estimate the probability of such an event. In light of 2008-2009, this look very reasonable. My estimated worst case 1-year using this simplified approach was worse than 2009-2009 for all the portfolios I have examined.


On Jun 16 05:18 PM gasem wrote:

> In terms of stress testing how do you view changes in a portfolios
> diversification metric and auto-correlation metric
>
> As I watch things evolve I saw diversity nose dive and auto correlation
> explode

Jun 16 11:25 PM

[gasem:]

The problem is that 2008-2009 is likely not going to turn out to be the worst case. Its just a case with over active PR as in public relations and political realities.

My point at bringing up the idea of correlations is if you have a given normal amount of diversification in a portfolio it looks to me like mapping changes in diversity for that given portfolio in a distribution gives you a way to kind of "stick a antenna" into the market. Each security in the portfolio has a correlation AND a magnitude in terms of its percent of the portfolio hence each security can define a vector with the direction defined by the correlation and the magnitude defined by its relative percent as a part of the whole. If you place the vectors all starting at the same point all the vectors in the plane will define a shape being able to watch that shape over time I think will tell you a lot about a portfolio's behavior.

Jun 17 06:39 AM

[BlueOkie:]

Monte Carlo is an excellent tool. The problem is what you input for the variables. Bob is a person and will alter his behavior given a changing environment. Monte Carlo works in hindsight and in a class room. The probability of event "A" in 2006 may not be equal of "A" 's probability in 2007 or even any of the future years. Finally, why would you look at the probability of Bob living to 120?

Jun 17 08:59 AM

[Geoff Consi...:]

BlueOkie:

Nothing in QPP or Monte Carlo is inherently backward looking, though plenty of Monte Carlo implementations are just that way. QPP has been tested extensively as a forward looking tool--and its results speak for themselves. Also, nobody rationally suggests that investors will never change pathes--I wrote an article showing how this relates to Monte Carlo and how a combination of MC and the flexibility to adjust your plans dramatically increases your odds of success. BTW, who said anything about living to be 120?

Geoff


On Jun 17 08:59 AM BlueOkie wrote:

> Monte Carlo is an excellent tool. The problem is what you input for
> the variables. Bob is a person and will alter his behavior given
> a changing environment. Monte Carlo works in hindsight and in a class
> room. The probability of event "A" in 2006 may not be equal of "A"
> 's probability in 2007 or even any of the future years. Finally,
> why would you look at the probability of Bob living to 120?

Jun 17 06:07 PM

[MachineGhost:]

> but I am hoping that the 3SD worst estimate is still a good
> measure. Basically, we are saying that we plan for a really bad
> (<1/1000) event while ignoring autocorrelation/corre

A normal bell curve suggests .13% probability for -3SD, actual market (S&P 500) history is .49% probability at -3SD. You have to go all the way down to -6SD at .10% probability to be equivalent. Using -3SD is false assurance in a retirement planning product!

The problem with all models is GIGO. Garbage in, garbage out. Garbage such as using a heuristic short cut instead of sampling the actual returns distribution. Especially in combination with a short sampling period.

Informed traders/investors don't even bother with standard deviations. Instead, they are concerned with historical maximum drawdowns. That's the absolute worst case scenario, until it is surpassed in the future. Many stocks have up to 80%-90% drawdowns in previous bear markets.

There is a Sharpe Ratio analog that takes into consideration the probabilities of such drawdowns and their underwater equity time. It is called the Ulcer Performance Index It is superior to Sharpe Ratio, but as with standard deviation, you still have the risk being hidden as who knows what the actual probability and magnitude of the next drawdown would be?

I don't like probabilities on my worst case scenario other than 100%. If you plan for that, everything else is gravy.

Jun 19 07:23 AM

[Geoff Consi...:]

Actually, my 3SD loss is worse than what happened over the last two years by notable amount. You are missing the subtlety of Milevsky's proposal. We can never know the tru probability of really extreme events all that well--even if the world is Gaussian, we would have estimation error on the mean and SD. Milevsky's is suggesting taking a model's estimate of 'worst case' and making sure that it is survivable--this is very reasonable. You are saying that the test is 'historical maximum drawdowns' which is also reasonable, but will tend to mack really high loss potential. Many firms have not existed long enough to truly test the extremes in real life--thats why we use models in the first place.


On Jun 19 07:23 AM MachineGhost wrote:

> > but I am hoping that the 3SD worst estimate is still a good

Jun 22 10:07 AM

[Geoff Consi...:]

Correction:

Ron is correct: Bob's portfolio would have lost 19% of its value even without any withdrawals. Before withdrawals, the stress test formula proposed in this article would suggest a 'worst case' of 8.7% - 3*10.6% = -23%. Had Bob done the stress test ahead of time, he could have tuned down his risk level if this was too severe blow. Bob did not have a very well diversified portfolio (according to QPP) and I am in no way saying that this was a great portfolio choice. My point is that a Monte Carlo model, properly stress tested, would have alerted Bob to the risks in his portfolio.

I emphasize: Milevsky's point (and I agree) is that we can never be sure whether we estimate the probability of extreme events well--but we can stress test to see if the 'worst case' events that we can estimate are survivable.

On a related note: there have been some famous cases in which quant models have predicted that the things that hit their portfolios were 25 standard deviation kinds of events--i.e. effectively impossible. This was true in 08 and also in the case of LTCM. The models were clearly wrong. 3 standard deviation events do happen, and we are also saying that we know that they will probably happen with greater frequency in real life--which is why stress testing to ensure survivability of such events is so important.

Jul 10 08:09 PM

[Ronnen.:]

It seems that mean-reversion has not been properly taken into consideration. Here is why: Three assumptions are made:
1) The average annual return of the portfolio is 8.7%
2) At the 'worst case' year the annual return will be -23.1% (A-3SD)
3) The annual return reverts to the mean
Now, if we assume that mean-reversion will occur within the following year, then we have to conclude that the next year will yield 39% (and all subsequent years 8.7%)
And if, for the sake of symmetry we recall that the last 3 years averaged at 9.2% and we assume that reversion to the mean will take 3 years, then we must conclude that the annual return in the next 3 years will be 18.8% (so that the average over the last 3 years + the worst-case year + the next 3 years reverts back to the 8.7%).
Either way, the worst-case year is compensated somewhat by the higher than average returns in the following years so that the resulting outcome is not as bad as was shown.
Ronnen.

Jul 16 05:53 AM