Estimating Future Returns for Retirement and General Investing

by: Doug Short

By Adam Butler and Mike Philbrick

Adam Butler and Mike Philbrick are Directors, Wealth Management and Portfolio Managers with Butler|Philbrick & Associates at Richardson GMP in Toronto, Canada. We have recently corresponded on the topic of historical market valuation. In the post below from their blog, they have used historic values to estimate future returns.
"Mankind are so much the same, in all times and places, that history informs us of nothing new or strange in this particular. Its chief use is only to discover the constant and universal principles of human nature." - David Hume

Long-time readers will know that we do not make predictions in the normal sense. That is, we endorse the decisive evidence that markets and economies are complex, dynamic systems which are not reducible to normal cause-effect analysis. However, we are willing to acknowledge the likelihood that the future is likely to rhyme with the past. Thus, we apply simple statistical models to discover mean estimates of what the future may hold over meaningful investment horizons (10+ years), while acknowledging the wide range of possibilities that exist around these averages.

There are several reasons why it may be useful to have a more robust estimate of future expected returns on stocks:

  • People who are approaching retirement need to estimate probable returns in order to budget how much they need to save.
  • A retiree's level of sustainable income is largely dictated by expected returns over the early years of retirement.
  • Investors of all types must make an informed decision about how best to allocate their capital among various investment opportunities
Many studies have attempted to quantify the relationship between Shiller PE and future stock returns. Shiller PE smooths away the spikes and troughs in corporate earnings which occur as a result of the business cycle by averaging inflation-adjusted earnings over rolling historical 10-year windows.

This study contributes substantially to research on smoothed earnings and Shiller PE by adding three new valuation indicators: the Q-Ratio, and deviations from the long-term price, and total-return, trends. The Q-Ratio measures how expensive stocks are relative to the replacement value of corporate assets. Deviations from the long-term trends of the S&P price and total-return series indicate how 'stretched' values are above or below their long-term averages.

These three measures take on further gravity when we consider that they are derived from three distinct facets of financial markets: Shiller PE focuses on the earnings statement; Q-ratio focuses on the balance sheet; and deviation from trend focuses on a technical price series. Taken together, they have capture a wide breadth of information about markets.

We analyzed the power of each of these 'valuation' measures to explain inflation-adjusted stock returns over subsequent multi-year periods. Our analysis provides compelling evidence that future returns will be lower when starting valuations are high, and that returns will be higher in periods where starting valuations are low.

This last point may seem obvious, but I want to emphasize a critical point about traditional wealth management of which most investors are not aware:

Traditional investment planning does not account for whether markets are cheap or expensive. An investor who visited a traditional Investment Advisor at the peak of the technology bubble in early 2000 would, in practice, be advised to allocate the same proportion of his wealth to stocks as an investor who visited an Advisor near the bottom of the markets in early 2009. This despite the fact that the first investor would have had a valuation-based expected return on his stock portfolio from January 2000 of negative 2% per year, while the second investor would expect inflation-adjusted compound annual returns of 6.5%. For an investor with $1,000,000 to invest, this would represent a difference of more than $1.26 million in cumulative wealth over a decade.

Said differently, traditional wealth advice is rooted in the assumption that the best estimate of future returns is the average long-term return to stocks. No matter where markets are on the continuum from very cheap to very expensive, traditional Advisors will make recommendations on the assumption that investors should expect 6.5% inflation adjusted returns on stocks over all investment horizons.

John Hussman at Hussman funds is careful to qualify the value of this analysis:

Rich valuation is strongly associated with weak subsequent returns, but only reliably so over periods of 7-10 years. In contrast, the present syndrome of overvalued, overbought, overbullish, rising-yield conditions is typically associated with abrupt and often steep losses, but is more commonly resolved over a period of months rather than years. (Hussman, Feb 14, 2011).

Thus, we are not making a forecast of market returns over the next several months; in fact, markets could go substantially higher from here. However, over the next 10 to 15 years, markets are very likely to revert to average valuations, which are much lower than current levels. This study will demonstrate that investors should expect 6.5% returns to stocks only during those very rare occasions when the stock market passes through 'fair value' on its way to becoming very cheap, or very expensive. At all other periods, there is a better estimate of future returns than the long-term average, and this study will quantify that estimate.

Investors should be aware that, relative to meaningful historical precedents, markets are currently expensive and overbought by all three measures, indicating a strong likelihood of low inflation-adjusted returns going forward over periods as long as 20 years.

This prediction is also supported by evidence from an analysis of corporate profit margins. In his recent book, Vitaly Katsenelson provided the following chart (Chart 1.) of long-term profit margins to U.S. companies. Companies have clearly been benefiting from a period of extraordinary profitability.

Source: Vitaly Katsenelson (2011)

The profit margin picture is critically important. Jeremy Grantham recently stated,

Profit margins are probably the most mean-reverting series in finance, and if profit margins do not mean-revert, then something has gone badly wrong with capitalism. If high profits do not attract competition, there is something wrong with the system and it is not functioning properly.

On this basis, we can expect profit margins to begin to revert to more normalized ratios over coming months. If so, stocks may face a future where multiples to corporate earnings are contracting at the same time that the growth in earnings is also contracting. This double feedback mechanism may partially explain why our statistical model predicts such low real returns in coming years. Caveat Emptor.

Modeling Across Many Horizons

Many studies have been published on the Shiller PE, and how well (or not) it estimates future returns. Almost all of these studies apply a rolling 10-year window to earnings as advocated by Dr. Shiller. But is there something magical about a 10-year earnings smoothing factor? Further, is there anything magical about a 10-year forecast horizon?

Kitces (2008) demonstrated that "the safe withdrawal rate for a 30-year retirement period has shown a 0.91 correlation to the annualized real return of the portfolio over the first 15 years of the time period". So there is clearly merit in studying a 15-year forecast horizon as well. Further, the tables below will demonstrate that statistical models have the greatest explanatory power at the 15-year horizon.

This study will attempt to address the question of 'perfect forecast horizon', perfect valuation factor, and 'perfect earnings smoothing factor', by analyzing the explanatory power of earnings, the Q-Ratio, and regressed historical stock returns, over return horizons from 1 to 30 years. We will also put all of the factors together to construct an optimized model.

Table 1. below provides a snapshot of some of the results from our analysis. The table shows estimated future returns based on several factor models over some important investment horizons. The "Best Fit Multiple Regression" is by far the most accurate model, but other results are provided for context.

Table 1. Factor Based Return Forecasts Over Important Investment Horizons
Click for larger image
Source: Shiller (2011), (2011), Butler|Philbrick & Associates (2011)

You can see from the table that every single valuation factor model generates results which suggest a very low future return environment for stocks. Further, the 'Best Fit Multiple Regression', which has historically provided a surprising degree of forecast accuracy, confirms this outlook with a high degree of confidence (see explanation below). Those who are not interested in our process can skip to the bottom sections, 'Putting the Predictions to the Test', and 'Conclusion'.


The following matrices show the R-Squared ratio, regression slope, regression intercept, and current predicted forecast returns for each valuation factor. The matrices are heat-mapped so that larger values are reddish, and small or negative values are blue-ish. Click on each image for a large version.

Matrix 1. Explanatory power of valuation/future returns relationships
Click for larger image
Source: Shiller (2011), Doug Short (2011), Butler|Philbrick & Associates (2011)

You will note that the R-Squared (top chart), which is a measure of the explanatory power of the relationship, is highest for the Q-ratio over all forecast horizons up to 9 years, but that deviations from the real price regression are more predictive over horizons of 10 years or more. The explanatory power of smoothed earnings ratios gets better consistently as we extend the forecast horizon, with peak ratios at the 20-year range. No factors possess any material explanatory ability at forecast horizons less than 5 years, so we have omitted results for these horizons.

Many analysts quote 'Trailing 12-Months' or TTM PE ratios for the market as a tool to assess whether markets are cheap or expensive. If you hear an analyst quoting the market's PE ratio, odds are they are referring to this TTM number. Our analysis slightly modifies this measure by averaging the PE over the prior 12 months rather than using trailing cumulative earnings through the current month, but this change does not substantially alter the results. As it turns out, TTM average earnings have very mild explanatory value over periods greater than 8 years. However, the explanatory power of TTM earnings is substantially less reliable than all other factors studied in this analysis, so investors may wish to pay little heed to this indicator of whether stocks are cheap or expensive.

One interesting take-away from the analysis is that the simple real price series carries much better explanatory power than the real total-return price series, which includes reinvested dividends. The total return series reflects the actual returns to investors over any given period, while the price series ignores dividends altogether. Strangely however, the total return series does a relatively poor job of forecasting future total returns to stocks, and is, statistically speaking, a much less powerful explanatory factor than either the Q-ratio or the simple real price regression. As so often happens in complex fields, the evidence makes a mockery of the theory.

Forecasting Expected Returns

The next matrices provide the slope and intercept coefficients for each regression. We have provided these in order to illustrate how we calculated the values for the final matrix below of predicted future returns to stocks.

Matrix 2. Slope of regression line for each valuation factor/time horizon pair.
Click for larger image
Source: Shiller (2011), Doug Short (2011), Butler|Philbrick & Associates (2011)

Matrix 2. Intercept of regression line for each valuation factor/time horizon pair.
Click for larger image
Source: Shiller (2011), Doug Short (2011), Butler|Philbrick & Associates (2011)

Our final matrix below shows predicted future real returns over each time horizon, as calculated from the slopes and intercepts above, by using the most recent values for each of the 13 earnings series, the Q-Ratio, and the return series as inputs. For statistical reasons which are beyond the scope of this study, we have substituted the ordinal rank for the nominal value for each factor in running our analysis. Therefore, when we solve for future returns based on current monthly data, we apply the monthly rank in the equations.

For example, the 9-year return prediction based on the current Q-Ratio can be calculated by multiplying the ordinal rank of the current Q-Ratio (1266) by the slope from the matrix at the intersection of 'Q-Ratio' and '9-Year Rtns' (-0.0001152), and then adding the intercept at the same intersection (0.1378). The result is -0.0080, or -0.80%, as you can see in the matrix below at the same intersection (Q-Ratio : 9-Year Rtns). Please click the matrix for a larger version.

Matrix 4. Modeled forecast future returns using current valuations.
Click for larger image
Source: Shiller (2011), Doug Short (2011), Butler|Philbrick & Associates (2011)

Finally, at the bottom of the above matrix we show the forecast returns over each future horizon based on our best-fit multiple regression from the factors above. We began testing the multiple regression against the Q-ratio, the 15-year Shiller PE, the price regression, and the total return regression as a 4 factor model. However, we discovered that the 15-year PE and total return regression series provided more noise than signal to the regression (that is, these factors were not statistically significant and reduced the F-score), so we narrowed the regression to include just the Q ratio and the real price series over each forecast horizon. We provided the R-squared for each of these multiple regressions below the forecasts; you can see that at the 15-year forecast horizon, our regression explains 76% of total returns to stocks. Further, the regression is very highly statistically significant, with a p value of effectively zero.

Chart 2 below demonstrates how closely the model tracks actual future 15-year returns. The red line tracks the model's forecast annualized real total returns over subsequent 15-year periods using the Q ratio and deviation from price regression as inputs at each period. The blue line shows the actual annualized real total returns over the same 15-year horizon.

Chart 2 15-Year Forecast Returns vs. 15-Year Actual Future Returns
Click for larger image
Source: Shiller (2011), Doug Short (2011), Butler|Philbrick & Associates (2011)

You can see that 15-year 'Regression Forecast' returns are 0.51% per year, and 10-year returns are forecast to be just -0.48% per year. To be clear, this means our model forecasts negative real total returns to U.S. stocks over the next decade given current levels of valuations.

Putting the Predictions to the Test

A model is not very interesting or useful unless it actually does a good job of predicting the future. To that end, we tested the model's predictive capacity at some key turning points in markets over the past century or more to see how well it predicted future inflation-adjusted returns.

Table 2 Comparing Long-term average forecasts with model forecasts
Click for larger image
Source: Shiller (2011), Doug Short (2011), Butler|Philbrick & Associates (2011)

You can see we tested against periods during the Great Depression, the 1970s inflationary bear market, the 1982 bottom, and the 2000 technology bubble top. The table also shows expected 15-year returns given market valuations at the 2009 bottom, and current levels. These are shaded green because we do not have 15-year future returns from these periods yet. Note real total return forecasts of 6.2% annualized from the bottom of the market in February 2009. This suggests that prices just approached fair value at the market's bottom, but they were nowhere near the level of cheapness that markets achieved at bottoms in 1932 or 1982. As of the end of February 2011, expected future returns over the next 15 years are under 1% annualized.

We compared the forecasts from our model with what would be expected from using just the long-term average real returns of 6.5% as a constant forecast, and demonstrated that estimates form long-term average returns yield over 400% more error than estimations from our model over these 15-year forecast horizons (1.24% annualized return error from our model vs 5.24% using the long-term average). Clearly the model offers substantially more insight into future return expectations than simple long-term averages, especially near valuation extremes.


The 'Regression Forecast' return predictions along the bottom of Matrix 4. are robust predictions for future stock returns, as they account for over 100 different cuts of the data, using 3 distinct valuation techniques, and utilize the most explanatory statistical relationships. The models explain up to 76% of future returns based on R-Squared, and are statistically significant at p=0. It is worth noting, however, that even this model has very little explanatory power over horizons less than 6 or 7 years, so almost anything is possible in the short-term.

Returns in the reddish row labeled "PE1" in Matrix 4 were forecast using just the most recent 12 months of earnings data, and correlate strongly with common TTM PE ratios cited in the media. These expected return numbers are substantially higher than any other numbers in the matrix save the forecasts from the total return series. This anomaly probably helps to explain the general consensus among sell-side market strategists that markets will do just fine over coming years. Just remember that these analysts have no proven ability whatsoever in predicting market returns (see here, here, and here). Further, it can be argued that their firms have a substantial incentive to keep their clients invested in stocks.

Investors would do much better to heed the results of robust statistical analyses of actual market history, and play to the relative odds. This analysis suggests that markets are currently expensive, and asserts a very high probability of low returns to stocks (and possibly other asset classes) in the future. Remember, any returns earned above the average are necessarily earned at someone else's expense, so it will likely be necessary to do something radically different than everyone else to capture excess returns going forward.

Note: I would like to thank Doug Short of the always illuminating for sharing his data on the Q-ratio which he painstakingly compiled from several sources. The most recent data is derived from Federal Reserve data; more information on this, and much more, at