Individual investor sentiment reflects current business conditions, but also psychological factors. Moreover, it is the latter which largely explain the ability of investor sentiment to predict future returns. Other publicly available macro-financial data cannot by themselves reproduce the market timing signals that contrarian investors use. By glimpsing the minds of individual investors, sentiment surveys add unique value and enhance the information set for investment decision making. Nevertheless, the usefulness of sentiment-based buy signals is limited because they are both noisy and relatively infrequent.

Each week, the American Association of Individual Investors (AAII) tallies the proportions of its membership that are bullish, neutral and bearish about prospects for the stock market over the next six months. The tallies bounce around and on occasion can become very pessimistic. On 25 occasions over the past 10 years, bullish sentiment has fallen below 25%, comprising more or less the bottom 5% of the distribution and representing bouts of extreme pessimism.

Curiously, these moments of desperation have occurred disproportionately as markets were poised to take off, with the result that an extreme low value of bullish sentiment is a statistically significant predictor of above average future returns (see "A Guide to Contrarian Market Timing" for a more extensive discussion). In Figure 1, the green bars show the expected price increase of SPY over the six weeks following observations of bullish sentiment < 25%, while the yellow bars show the unconditional expectation, i.e. the average value regardless of bullish sentiment. As the figure indicates, a highly pessimistic observation of the bullish sentiment index predicts an excess return of 3-5% over the next six weeks. Remarkably, the predictive value of the signal was even greater during the turbulent second half of the period, 2007-12.

**Figure 1 - Expected return on SPY over 6-week period following Bullish Sentiment <25% compared to unconditional expectation:**

*Click to enlarge.*

This note aims to understand better how episodes of extreme pessimism forecast the market, and also to estimate the economic value of this information.

**What do individual investors know?**

Why do AAII members suffer from recurrent bouts of extreme pessimism? Part of the explanation is that they see publicly available macro-financial data, including recent stock market performance, interest rates, credit spreads, VIX, put-call ratios, and macroeconomic indicators such as unemployment, factory orders, retail sales and inflation. Such indicators of business conditions are clearly linked to investor sentiment. For example, Figure 2 plots the probability of observing Bullish Sentiment < 25% against the return to SPY over the previous two weeks. Bullish Sentiment < 25% was observed on 3 out of 6 occasions (pr. = .5) when SPY fell by 15-20%, on 10 out of 27 occasions (pr. = .35) when SPY fell by 10-15%, and so on. The graph indicates a sharply rising likelihood of extreme pessimism as market performance deteriorates. A similar pattern can be found in many other macro-financial data series.

**Figure 2 - Probability of observing Bullish Sentiment < 25% conditional on change in SPY price over previous two weeks:**

It's hardly surprising that macro-financial data influence investor sentiment. However, they don't fully explain the occurrence of extreme low values. This can be shown easily using regression analysis. Define a dummy variable equal to 1 when Bullish Sentiment is below 25% and 0 at other times, then regress this against a basket of macro-financial series that exhibit a strong linkage to the bullish sentiment index along the lines of Figure 2. The basket used below includes:

- Change in SPY price over previous two weeks
- SPY price relative to average value over previous two weeks
- SPY price volatility over previous 6 weeks
- Credit spread (yield difference between BAA and AAA corporate bonds)
- Change in the yield curve (defined as the difference between 10 year and one month treasuries) over the previous two weeks
- Put-call ratio
- VIX
- CPI inflation
- Change in unemployment rate
- Growth of retail sales

In all cases, care has been taken to allow for the timing of data releases and adjust for differences in frequency (e.g. the CPI is released monthly with a lag of around 2 weeks) in order to approximate the information set actually available to investors at the time of the survey.

Fitting the equation over the period 2002-12, yields an R^{2} of .19; over 2002-07 R^{2} is .08; and over 2007-12 R^{2} is .24 (results from the author on request). Undoubtedly it would be possible to get a better fit by tinkering with the specification and adding more variables, such as earnings surprises on market movers, or monetary policy announcements. It would be an interesting exercise to model expectations as latent or hidden variables, and see whether negative surprises cluster at times of extreme pessimism. But, none of this would likely change the main conclusion that while public information contributes to extreme pessimism, there remains an important residual role for investor psychology.

Now let us ask: Which part of the bullish sentiment index determines predictive power - the factual part, or the psychological part? Simple, ordinary least squares once again gives an answer. The regressions described above project the occurrence of episodes of extreme pessimism onto macro-financial time series data. Thus, the fitted equations are by definition the part which can be "explained" by factual information. The residuals are the part that cannot be explained, and hence capture the psychological dynamics unrelated to current conditions. By regressing the future growth of SPY on the fitted equations and residuals from the first stage regression, we can immediately see what gives rise to the index's predictive power. Elsewhere, I have argued that a six week holding period is a good rule of thumb for contrarian investors, so I'll use as a dependent variable the change in SPY price over the six weeks following publication of the bullish sentiment index. The regression result is as follows:

R2 = .031; t-statistics shown in parentheses.

Because of the way the equation is set up (technically, the regressors are orthogonal), it's easy to apportion the predictive power between the two components, and the result is shown in Figure 3: virtually all of the predictive power of the AAII index stems from the psychological component. Indeed, with a t-statistic of 0.4 the factual component isn't even statistically significant as an explanatory variable.

**Figure 3 - Contributions of the factual and psychological components of extreme pessimism to predictions of 6-week outperformance of SPY:**

One commentator on my previous article wondered whether a more extensive collection of indicators might have better predictive ability than the AAII bullish sentiment index alone. The regression suggests the answer is, no. All of the predictive power of the AAII index seems to derive from the way that AAII members collectively filter and process the information they see rather than from the information itself - a striking example of what one might call the unwisdom of crowds, since their expectations are persistently wrong.

Interestingly, some earlier literature from the 1990s expressed skepticism about behavioral models of investor sentiment because measures such as put-call ratios, reputed to be "fear" indexes, did not give rise to excessive swings with implications for future performance. The results here suggest it may have been looking in the wrong place. Notably, the AAII index would not have had a long enough history at that time to be useful for research purposes.

**How much is the Bullish Sentiment Index worth?**

There's something puzzling about the results just discussed. On the one hand, (the psychological component of) extreme pessimism is robustly predictive: a pessimistic reading raises expected returns by a large amount and the regression coefficient has a large t-statistic, significant at the 0.001% confidence level. Yet, at the same time, the R^{2} of .03 indicates the equation overall forecasts only 3% of future SPY performance. How can low bullish sentiment have strong predictive power in an equation that doesn't predict very much? It's worth being clear about this and understanding the economic value of the information.

First, suppose you are considering a $100 investment in the market for a six week period. How much would you be willing to pay in order to have the next observation of Bullish Sentiment < 25%? For the sake of realism, I'll answer this using more recent data from the period 2007-12 . If Bullish Sentiment < 25% the expected rate of return is 4.7% so you would expect to earn $4.70. If Bullish Sentiment is > 25%, the expected rate of return is -0.6% and you would expect to lose $0.60. The value of an observation of Bullish Sentiment < 25% is thus $5.30 = $4.70 - (-$0.60).

But, from an investment standpoint, it's more meaningful to think about this differently. Suppose you're an investor chasing the best possible rate of return, how much would you be willing to pay to learn the value of the bullish sentiment index? The answer is *not* $5.30, because most of the time what you'll learn is that bullish sentiment > 25%, which won't help you to make money.

Consider two scenarios and for the sake of argument suppose the alternative to investing in the stock market is holding 1 month Treasury bills, currently paying 0.1%. In the first scenario, you don't know the value of bullish sentiment, hence base your investment decision on unconditional expectations. In the second scenario, you learn the value before investing and factor this into your decision. During 2007-12, 7% of observations of bullish sentiment were less than 25%, after which SPY returned 4.7% and 93% of observations were greater than 25% after which SPY returned -0.6%. The scenarios are summarized in Table 1.

Scenario 1 entails choosing between the stock market and T-bills based on unconditional expectations. Multiplying the probabilities by returns gives the column headed "Stock mkt" and summing these up shows that the unconditional expected return on an investment of $100 is a loss of $0.252. By comparison, an investment in T-Bills earns $0.012 (annual rate of 0.1% for a 6 week period). Therefore, the optimal choice is to leave the money in T-bills and earn $.012. The choices are shaded in green in Table 1.

In scenario 2, the value of bullish sentiment is known. 7% of the time the reading is below 25% and the investment earns 4.7%, for an expected contribution to the P&L of $0.314. The other 93% of the time B > 25% yielding an expected return on the stock market of -$0.566, so the money stays in T-bills and earns $.011. The expected return now is $0.325. Thus, the value of the information is $0.313 = $0.325 - $0.012, a small fraction of the $5.30 you'd be willing to pay for a reading of bullish sentiment <25%. The point is, that when extreme low values of bullish sentiment occur they produce sizeable gains (which explains the significant t-statistic), but they occur rarely (which explains the low R^{2}). Note further that these calculations assume risk neutrality. However, if you were risk averse the information would be worth even less because the certainty equivalent of the $0.314 payoff would be lower, possibly much lower.

**Table 1 - Economic value of the information in low readings of the bullish sentiment index:**

**Conclusion**:

The AAII bullish sentiment index is a useful source of information for investors. Episodes of extreme pessimism represent opportunities to earn significant expected excess returns. The index owes its predictive power to consolidating the collective thoughts of AAII members rather than merely repackaging existing public information. As a result it adds unique value to the information set of contrarians, and its signals cannot be reproduced from other macro-financial data, including those which purport to measure investor sentiment such as VIX and put-call ratios.

Although extreme values of the index signal profitable buying opportunities, these signals are transmitted relatively infrequently which reduces the potential returns to a contrarian investment strategy. Factoring in risk would further reduce the value of contrarian buy signals in utility terms. Consider, for instance, that the index is currently flashing a buy signal. However, many investors might feel reluctant at the present time to incur the downside risks of increasing their long positions in US equities.

**Disclosure: **I am long SPY.