I often get questions on why optimization only results in a subset of the universe being used in the final results. This article seeks to explain the mechanics behind optimization and why this is so.

Let's take a look at the following 14 stocks: Alliance Resource Partners (NASDAQ:ARLP), Altera Corp (NASDAQ:ALTR), Amgen (NASDAQ:AMGN), Citrix Systems (NASDAQ:CTXS), Coach (NYSE:COH), Duke Realty Corp (NYSE:DRE), EMC Corp (EMC), Intuitive Surgical (NASDAQ:ISRG), Microsoft Corp (NASDAQ:MSFT), Quest Diagnostics (NYSE:DGX), RF Micro Devices (RFMD), Southern Copper Corp (NYSE:SCCO), Southern Co (NYSE:SO), and Yahoo! (NASDAQ:YHOO).

Using the hybrid rebalancing process described in my article entitled, "The Art of Non-Normal Rebalancing" yields the following performance table:

_{For ease of interpretation we have starting portfolio values of $100, where the blue bars use the hybrid rebalancing process, while the green bars use a buy-and-hold strategy. Taxes, commissions, and the use of a risk-free proxy are not included in the calculations. Price data has been sourced from SeekingAlpha on a dividend-adjusted basis. The results are obtained by going back in time to use only the data available as at each review point, calculating the optimal weights, then going forward to compare the performance between the optimized and equal-weights portfolios. The equal weights portfolio is computed on a buy & hold basis, while the optimized portfolio is rebalanced using the hybrid rebalancing process.}

The stocks and weights that contributed to the performance of the optimized portfolio at each monthly review point are shown in the figure below, where the left axis shows the 48 review points (Review Point 0 being 26 December 2007), the right axis shows the 14 stocks, and the vertical axis shows the weight in fraction form allocated across the 14 stocks.

Whether the weights are rebalanced at a review point is a function of the hybrid rebalancing process, but what should be immediately clear is that not all the 14 stocks are used at any one time.

An optimal (in the efficient sense) portfolio is a portfolio that has the lowest volatility for a given return, or the "best" return for a given volatility. The way it achieves this depends not only on the correlation between the various pairs of securities in the universe, but also on your measures of expected return and volatility for each security. Allow me to explain how this works.

Suppose you had two stocks in your portfolio. You would calculate the weighted return of these two stocks to arrive at the portfolio's expected return. Let the weight for stock1 and stock2 be *w*_{1} and *w*_{2} respectively.

The portfolio volatility where volatility is defined as standard deviation would be the square root of this formula: (*w _{1} x volatility_{1})^2 + (w_{2} x volatility_{2})^2 + 2w_{1}w_{2} x volatility_{1} x volatility_{2} x correlation between stock1 and stock2.*

My purpose in showing this formula is so that you appreciate the fact that the weights are calculated not just on the basis of correlation, but the aggregate result of individual volatilities and the correlation that exists between each and every pair of securities, *at the level of expected return of the portfolio.*

So in the universe above, why was more weight put on only a subset of the 14 stocks in the universe with zero weight in the rest?

It is a combination of reasons.

If any weight was put at all on any of these stocks, it would be because by doing so, it helped reduce portfolio volatility while maintaining the expected portfolio return, which is the weighted average of all the stocks that are included in the portfolio.

And if any of the stocks did reduce portfolio volatility while maintaining the expected portfolio return, it was because it was negatively correlated (or more precisely, less than perfectly positively correlated) to either one or any combination of the other stocks in the portfolio *to the extent* that the reduction in portfolio volatility due to the effects of correlation was greater than any increase in portfolio volatility as a result of the introduction of the stock to the portfolio.

So if a stock or security was not included in the optimal portfolio, it was simply because it did not meet this criteria.

**Optimization In Process**

To elaborate on the process, let's say you started with just two stocks in the optimal portfolio. You then try to include stock3, but it does not pass the criteria for inclusion into the portfolio. Do you then discard stock3? The answer is not just yet.

This is because you will not know if stock3 combines with stock4, or stock5, etc. to form an even lower volatility portfolio at the given return. In other words, you have to analyze all possible combinations before deciding which will be the efficient portfolio (i.e., the lowest portfolio volatility for the given return) that sits on the frontier. Do this process for each and every possible portfolio return, and you will then be able to construct the entire efficient frontier.

Generating this by hand will obviously take forever. And there is no closed-end formula that will give you the answer. Optimizers use an iterative process, which is an algorithm that loops until a certain tolerance is met.

**To Constrain Or Not To Constrain**

It is possible to include all the stocks in the final analysis, but you have to use a constrained frontier, which artificially forces a minimum and/or maximum allocation for each and every stock. In a sentence, it is possible to impose any kind of constraint (even negative ones for short-selling) on the calculations, as long as it makes sense to do so.

Sometimes constraints are added because you wish to impose a maximum percentage on stocks that you have classified as low, medium or high risk; or from different asset classes. At other times, you may wish to include all the stocks in the universe because these are stocks that you have chosen after shortlisting them according to other criteria.

**Conclusion**

Optimization is a simple concept, but it is not easily visualized. A stock can be negatively correlated to those in a portfolio, but still not be eligible for inclusion in the optimal portfolio at a given return.

On the other hand, a stock might be positively correlated but be included, or when combined with another stock that is not yet included in the portfolio, yields a portfolio mix that is also efficient, but at a different return.

Like all power tools, an optimizer must be handled with care. The precision with which it calculates is undeniable, but it can be a double-edged sword -- something I will discuss in a future article.

**Disclosure: **I have no positions in any stocks mentioned, and no plans to initiate any positions within the next 72 hours. I wrote this article myself, and it expresses my own opinions. I am not receiving compensation for it (other than from Seeking Alpha). I have no business relationship with any company whose stock is mentioned in this article.