This article is part of an ongoing series exploring flaws in popular investment risk and skill evaluation techniques. We focus on the most common pitfalls that have been particularly costly for investors tracking fund holdings and attempting to replicate the performance of top managers. Since the analysis of fund holdings relies on the ability to identify managers that are skilled, anyone working with holdings data will be interested in these tips. Failure to identify skill can cause an unsuspecting investor to track funds and portfolios that have gotten lucky and are likely to underperform.
Investment risk and skill evaluation frequently relies on returns-based style analysis, and returns-based performance attribution. These techniques perform regressions over one or more historical periods to compute portfolio betas (exposures to systemic risk factors) and alphas (residual returns unexplained by systemic risk factors).
The accessibility of returns-based approach has made it popular. It is often the only practical method for evaluating multi-asset-class portfolios that incorporate commodities, public securities, derivatives, and private investments. This simplicity comes at a heavy cost, which we explore in this and upcoming articles.
The Assumption of Stable Exposures
A key assumption of most returns-based analyses is the constancy of factor exposures over regression period or periods. While this assumption holds for some funds that closely track broad market indices and do not vary bets, it breaks down for more active funds.
A good example of this is the Fairholme Fund (FAIRX). The Fairholme Fund has dramatically varied its bets over the past ten years. A simple linear regression of historical fund returns against the US Market is below. It estimates beta = 1.14 and monthly alpha = 0.12%:
The Fairholme Fund (MUTF:FAIRX) Returns vs. the US Market
This and similar regression approaches form the basis of returns-based analysis. This convenient but overly simplistic analysis does not attempt to estimate market exposure at each point in time. Hence, the beta of 1.14 may not be representative of the current or historical US Market exposures of the fund.
The Variation of Exposure
To verify US Market exposure estimated by the returns-based analysis, we estimated monthly US Market exposures of the fund using the AlphaBetaWorks US Equity Risk Model. For each month, we estimated the betas (exposures) of individual positions to US Market factor and aggregated these into monthly estimates of aggregate portfolio beta. It turns out that over the past 10 years the Fairholme Fund has varied its US Market Exposure between 50% and 170%:
The Fairholme Fund (FAIRX) Market Factor Exposure History
The US Market beta ranged from 0.5 to 1.7 and was infrequently anywhere near 1.1. Turns out, risk estimated using returns analysis is inaccurate most of the time. It is also an inaccurate estimate of the mean exposure over the period, which is 98%:
The Fairholme Fund (FAIRX) Historical US Market Exposure Distribution
Consequences of Varied Exposures
Returns-based analysis can produce deeply flawed estimates of current risk for funds that vary their bets. Even estimates of average risk and style may be inaccurate. In the case of the Fairholme Fund, the returns-based estimate of US Market exposure-around 110%-is well off from the current portfolio exposure-around 140%. To make matters worse, there is a domino effect: Returns-based performance attribution builds upon any errors in returns-based style analysis, compounding them.
Anyone paying for an actively managed investment product must have confidence that expected future active returns exceed fees. Returns-based style analysis and performance attribution are frequently used for this purpose. Can this analysis be trusted?
We estimated cumulative alpha, or residual return, for the Fairholme Fund with a single risk factor for the US Market using returns-based beta = 1.1 calculated above:
The Fairholme Fund (FAIRX) Cumulative Returns-Based Alpha
Unsurprisingly, the errors in the beta estimate lead to a flawed picture of fund's security-selection. Returns-based approach estimates cumulative alpha greater than 10%. Aggregating the betas of the individual portfolio positions throughout the period produces quite the opposite at -10%:
The Fairholme Fund (FAIRX) Cumulative Single-Factor Model Alpha
An investor who used returns-based style analysis and attribution would have estimated significant positive security-selection performance. In reality, an investor would have outperformed by taking the same market risks passively. A capable risk model specifically tuned for skill evaluation and performance prediction, such as the AlphaBetaWorks Statistical Equity Risk Model, averts this and similar pitfalls.
Returns-based analysis can be effective - but only when a passive manager does not significantly vary exposures to the market, sector, and macroeconomic factors.
When an active manager varies bets, returns-based analysis typically yields flawed estimates of portfolio risk. When a manager varies bets, returns-based analysis may not even accurately estimate average or representative portfolio risk.
Returns-based analysis will be the least predictive precisely for the most active managers:
- Estimates of manager's risk may be flawed.
- Skilled funds may be deemed unskilled.
- Unskilled funds may be deemed skilled.
The analysis and aggregation of factor exposures of individual holdings throughout portfolio history using a capable multi-factor risk model addresses these shortcomings.
More subtle but no less dangerous issues with investment risk and skill evaluation using returns-based performance attribution will be discussed in subsequent articles. Awareness of these issues is vital when tracking fund holdings and attempting to replicate the performance of top managers. Ignorance of these issues can cause an unsuspecting investor to track funds and portfolios that are likely to underperform.
Disclosure: The author has no positions in any stocks mentioned, and no plans to initiate any positions within the next 72 hours. The author wrote this article themselves, and it expresses their own opinions. The author is not receiving compensation for it. The author has no business relationship with any company whose stock is mentioned in this article.