Morningstar CEF Ratings: Worse Than Random

Conclusion: Investors should not rely solely on the Morningstar rating system for closed end funds (CEFs) as a single source of information in making short-term investment decisions. What may be true for your health care may also be true for your investments; it’s always good to get a second opinion.

Summary: Over the past 14 months (March ‘08 to May ‘09), Morningstar’s CEF ratings proved worse than random. With the exception of their lowest rating (M*1Rating) the Morningstar rating system for CEFs was a perfect contrarian indicator; investors would have achieved better average investment performance buying CEFs with lower Morningstar ratings than those with higher ratings.

The graph below illustrates Morningstar’s CEF ratings and their respective average fund type price change during the above referenced period. Additionally, the percentage price change difference of the highest rated CEFs (M*5Rating) less each of the lower rated CEFs (rating “4” through “1”) has been calculated and respectively charted.

Background: Morningstar, Inc. is a provider of independent investment research in America, Europe, Australia, and Asia. Morningstar ranks stocks on a five star rating system by comparing a stock's current market price with Morningstar's estimate of the stock's fair value.

Under the system, 3-star stocks are those that should offer a "fair return," one that adequately compensates for the riskiness of the stock. Five-star stocks, of course, should offer an investor a return that's well above the company's cost of equity. Conversely, low-rated stocks have significantly lower expected returns. Simply put: a 5 rating is good and a 1 rating is bad.

Data: Morningstar provided ratings on 489 CEFs on March 13, 2008. Because the CEF market segment is so diverse and to provide for a more granular analysis, those ratings were further segregated into the 13 fund types used by the Wall Street Journal for categorizing CEFs.

Those rated CEF’s stock prices were then compared from the end of March ‘08 to the end of May ‘09. The resultant average price changes were organized into a matrix of rating levels (“5” through “1”) and fund types. Only fund types that had ratings for all 5 Morningstar ratings were included to keep the data comparable. There were 401 CEFs (82%) that passed that screen representing 7 fund types as illustrated below.

Highlighted in green are the average price changes that outperformed the highest rating (M*5Rating) in each respective fund type. What is particularly interesting is that in 4 out of the 7 fund types, the average price change for the lowest (M*1Rating) did better than the highest rating (M*5Rating).

Caveats: In the investment business there’s a saying, “When you’re predicting the future, predict often.” One could argue ratings are dynamic. To use a single historical rating doesn’t reflect new economic, industry or company events that may cause a subsequent ratings change.

Yet, almost 50% of Morningstar’s CEF ratings remained unchanged from Mar ’08 to May ’09. Additionally, the average rating change was lower by less than one grade. This would seem unusual during a time when the S&P 500 dropped almost 30%.

Glass Houses: This is not to imply that Morningstar’s analysts are anything but bright and hard working, it’s just the stock prediction business is difficult. A recent study by Oya Altinkilic of the University of Pittsburgh and Robert Hansen of Tulane University concluded that analysts “buy” and “sell” tips have almost no effect on share prices (“Study Finds Analyst Tips Don’t Move Prices”; Financial Times; May 17, 2009).

So, in the CEF stock picking business, if you’re looking for a helping hand, you’ll find it at the end of your arm.

Disclosure: Joe maintains a CEF rating system on his website that has been operational for a limited time. As a consequence, there’s currently insufficient data for a similar analysis. Once such data has been accumulated, an analysis will be undertaken and presented. It may in fact prove just as disappointing.

Comments

[YoYoMama:]

That you are developing your own rating system for ETFs sincerely casts doubts on the integrity of your motives in posting this article.

RE:
Disclosure: Joe maintains a CEF rating system on his website that has been operational for a limited time. As a consequence, there’s currently insufficient data for a similar analysis. Once such data has been accumulated, an analysis will be undertaken and presented. It may in fact prove just as disappointing.

Jun 09 10:05 AM

[Joe Eqcome:]

YoYoMama

First of all, impugning people’s integrity is a serious charge. Let’s review the facts.

Morningstar has a CEF ratings system. The article was a review of its results over a limited period of time. I’ve acknowledged in numerous places in the article that predicting stock returns is a difficult business and the people at Morningstar are hardworking, well intentioned, high-integrity folks. I sure if you did the same analysis on other rating systems you’d probably get close to random results.

I’ve also disclosed that I’m in the early development of a rating CEF ratings system. I would place no value on it until it demonstrated its effectiveness--which would probably take a year from this writing. I’ve also indicated that my system might be equally as ineffective once the appropriate data is accumulated. I have not asserted that my ratings are more accurate—they're just there.

I have no economic interest in my rating system as its available free for those who might have a passing interest. I'm hopeful, that I might be able to incorporate suggestions from others to improve its accuracy.

Lastly, the article places an emphasis on taking charge of your investments by doing some of your own homework. It’s always desirable to get more than one data point when making investments which I take as a serious business. That data points doesn’t have to be mine.

This doesn’t sound nefarious to me.

Joe Eqcome



On Jun 09 10:05 AM YoYoMama wrote:

> That you are developing your own rating system for ETFs sincerely
> casts doubts on the integrity of your motives in posting this article.
>
>
> RE:
> Disclosure: Joe maintains a CEF rating system on his website that
> has been operational for a limited time. As a consequence, there’s
> currently insufficient data for a similar analysis. Once such data
> has been accumulated, an analysis will be undertaken and presented.
> It may in fact prove just as disappointing.

Jun 09 12:19 PM

[Alan Young:]

Joe, thanks for this analysis.

If M* can't devote sufficient resources to keeping their ratings current, they shouldn't rate CEFs at all.

My guess is that they are trying to manage the ratings the same way they do mutual funds (OEFs).
CEFs are vastly more complex than OEFs, since the market price and NAV can diverge. Even a well-managed fund can be a bad investment when the market over-prices it, and vice-versa. An intelligent rating system would recognize this, and revisit the pricing every week or two (at least) to keep the evaluation current.

Jun 09 03:13 PM

[Analyste de...:]

M*'s problems - controversies? compromised analysis? - goes waaay back. At a competitor firm (which I'm not promoting) we found a decent number of mutual funds that M* rated favorably which subsequently blew up. I've never had great respect for their MF service, with all its limitations. And listening to Kinnel, it often seems like M* is the greatest enemy of ETFs - again, the $s they get from fund companies may taint their objectivity.

They're fine for 'additional info' but not reliable for primary due-diligence. Caveat Emptor!

Jun 09 04:14 PM

[searcher:]

M* has issued 'mea culpa' in the past when tweaking their system. I've never seen an exposition of a 'system' that didn't make wonderful sense. Unfortunately, the market wasn't as impressed as I.

Jun 09 06:37 PM

[Wildhawk:]

You'd think that the author would have gone to the trouble of actually attempting to understand how Morningstar rates closed-end funds before writing an entire article about how bad the rating system has been. The Morningstar closed-end fund rating system is NOT akin to the way they rate stocks (based on their long-term future investment merit), but instead uses the same methodology as open-end mutual funds (where the rating system is based purely on a backward- looking measurement intended to compare similar funds based on their past performance). All the Morningstar ratings system for CEFs is doing is telling you what funds have performed well in the past relative to funds with similar portfolios. It is not intended to be predicted, like their stock ratings are.

Since, to my knowledge, all closed-end funds are actively managed, rather than indexed, all the above data shows is that managers of closed-end funds were consistently inconsistent in their ability to outperform their peers- the top managers in one time period became the worst managers in a later time period, not unlike what other studies have shown are similar results from open-end actively managed funds.

Anyone who's read a prospectus understands that past performance is not indicative of future returns. If fund investors are using Morningstar's open-end and closed-end fund ratings systems as forward-looking, predictive measures, they're barking up the wrong tree.

Jun 09 11:52 PM

[Joe Eqcome:]

Wildhawk

All rating systems, even if it’s looking from a historical prospective, tacitly imply—and investors infer—some value as to its predictive power. If it doesn’t, what’s the point?

Even the worse of all rating systems, the credit rating agencies (Moody’s, S&P, etc.), implies some predictive power regarding credit worthiness of corporation they analyze. Or why would pension funds and other institutional investors require investment grade ratings for some of their investments if they thought the rating could change tomorrow?

So, if Morningstar CEF rating are only backward looking and do not imply any predictive power they should be discontinued, because it creates more confusion than clarity for the retail investor.

At the very least, they should change the name from “rating”—and get rid of the “stars” to something more descriptive: CEF Historical Performance Indicator (“CEF HPI”).

Additionally, Morningstar should provide a hedge clause anytime that CEF ratings are employed that states: “There is no empirical evidence that Morningstar’s CEF ratings has any investment value at all. It should be used as you would an odometer on your car to be compared with other odometers of your neighbors; which may, or may not, provide you a means to estimate the useful life of your vehicle.”

Joe Eqcome



On Jun 09 11:52 PM Wildhawk wrote:

> You'd think that the author would have gone to the trouble of actually
> attempting to understand how Morningstar rates closed-end funds before
> writing an entire article about how bad the rating system has been.
> The Morningstar closed-end fund rating system is NOT akin to the
> way they rate stocks (based on their long-term future investment
> merit), but instead uses the same methodology as open-end mutual
> funds (where the rating system is based purely on a backward- looking
> measurement intended to compare similar funds based on their past
> performance). All the Morningstar ratings system for CEFs is doing
> is telling you what funds have performed well in the past relative
> to funds with similar portfolios. It is not intended to be predicted,
> like their stock ratings are.
>
> Since, to my knowledge, all closed-end funds are actively managed,
> rather than indexed, all the above data shows is that managers of
> closed-end funds were consistently inconsistent in their ability
> to outperform their peers- the top managers in one time period became
> the worst managers in a later time period, not unlike what other
> studies have shown are similar results from open-end actively managed
> funds.
>
> Anyone who's read a prospectus understands that past performance
> is not indicative of future returns. If fund investors are using
> Morningstar's open-end and closed-end fund ratings systems as forward-looking,
> predictive measures, they're barking up the wrong tree.

Jun 10 01:30 PM

[Gruber:]

If, as you contend, Morningstar's ratings is really comparable to an index, then it should be called that.
I agree with Joe that the Morningstar's ratings for CEFs should be called something else.


On Jun 09 11:52 PM Wildhawk wrote:

> You'd think that the author would have gone to the trouble of actually
> attempting to understand how Morningstar rates closed-end funds before
> writing an entire article about how bad the rating system has been.
> The Morningstar closed-end fund rating system is NOT akin to the
> way they rate stocks (based on their long-term future investment
> merit), but instead uses the same methodology as open-end mutual
> funds (where the rating system is based purely on a backward- looking
> measurement intended to compare similar funds based on their past
> performance). All the Morningstar ratings system for CEFs is doing
> is telling you what funds have performed well in the past relative
> to funds with similar portfolios. It is not intended to be predicted,
> like their stock ratings are.
>
> Since, to my knowledge, all closed-end funds are actively managed,
> rather than indexed, all the above data shows is that managers of
> closed-end funds were consistently inconsistent in their ability
> to outperform their peers- the top managers in one time period became
> the worst managers in a later time period, not unlike what other
> studies have shown are similar results from open-end actively managed
> funds.
>
> Anyone who's read a prospectus understands that past performance
> is not indicative of future returns. If fund investors are using
> Morningstar's open-end and closed-end fund ratings systems as forward-looking,
> predictive measures, they're barking up the wrong tree.

Jun 10 02:52 PM

[ETFdesk.com:]

Great article Joe! While I don't know Morningstars rating system, one would think to choose the exact opposite of what their rating system says. Here's the logic... If you have a peer group and assume all the assets in each fund are basically the same(and at similar discounts), then the worst performing one should be at the biggest discount and thus have a better chance of collapsing the discount. The richest fund is likely the one M* has a top rating on because its share price did better for a period of time and the system is somewhat top ticking it. M* should include NAV performance-share price performance to make a pick...

Jun 16 11:10 PM

[Dan Plettner:]

I conversed with Morningstar's Director of Personal Finance with reference to Morningstar's own "strategy" on Closed-End Fund Research in recent months. I have the impression that the cost of competent analysis for qualitative research would not produce the necessary economic benefits for Morningstar's own business model.

Jun 17 12:29 AM

[Phil S:]

Dan, did you get the impression that Morningstar has a model for CEFs that is reasonably tailored to CEFs as opposed to OEFs/ETFs?

In particular, a good CEF rating system should, IMO, distinguish between market returns and NAV returns, and should take the premium or discount into account, among other considerations.

Jun 17 01:01 AM

[Dan Plettner:]

Phil, Please note that this is not my article. I will be happy to respond to your question because my view on Morningstar's CEF Research is generally consistent with that of the author.

To directly answer your question, I do NOT believe Morningstar has a reasonably tailored model for CEFs. While I personally own a Closed-End Fund with a 5-star rating, my investment thesis on that CEF does not overlap with Morningstar's model.

Closed-End Funds are a unique instrument, significantly different from Open-End Mutual Funds. There is some utility to NAV performance history, but I do not believe Morningstar's CEF research (as it is designed today) lends in any significant way to an investor achieving alpha. And I do strongly believe that Closed-End Funds as a product group foster alpha.


On Jun 17 01:01 AM Phil S wrote:

> Dan, did you get the impression that Morningstar has a model for
> CEFs that is reasonably tailored to CEFs as opposed to OEFs/ETFs?
>
>
> In particular, a good CEF rating system should, IMO, distinguish
> between market returns and NAV returns, and should take the premium
> or discount into account, among other considerations.

Jun 17 04:34 PM