I went to graduate school to become a macroeconomist and understand business cycles, but became convinced pretty quickly that this problem was not going to be solved any time soon. The current theories today are pretty much the same is when I was in school 20 years ago. There are theories that recessions are caused by monetary shocks, interest rates being too low, insufficient consumption demand, sticky prices, technology shocks, changes in leisure preferences, cyclical investor optimism. None of these explanations are very convincing, which is why economists still have not coalesced around any one explanation. Indeed, if anything it's worse because back in the Keynesians vs. Monetarist days (say 60s and 70s) you had only two camps, highlighting that neither side was right.
Arnold Kling makes a good point when he states that macroeconomic activity consists of sustainable patterns of specialization and trade, and if a recalculation occurs such that the current pattern is recognized as unsustainable, the system re-allocates by exiting those unsustainable businesses. But then, whence the massive recalculation? Do we always have something like Felix Salmon's copulas, unique math errors that pop up every 5 to 10 years?
For those theories that rely on mercurial investment or animal spirits, the cognitive errors that underlie business booms and busts have yet to be identified. Aggregates like investment spending, consumer durables, aggregate average liabilities/assets, do not increase into some unsustainable zone over an expansion. The key puzzle why people always become too optimistic in expansions is like trying to understand why a drunk drinks too much every day. You can't model such investment rationally in traditional models, and if you model it irrationally, you imply a simple market timing model for the stock market should generate abnormal profits, and no such rules exist (.pdf).
My argument is that business cycles are best understood though the framework of Batesian mimicry, an endogenous mechanism for booms and busts thru a misallocation in the horizontal structure of production. In ecosystems, Batesian mimicry is typified by a situation where a harmless species (the mimic) evolves to imitate the warning signals of a harmful species (the model) directed at a common predator (the dupe). For example, venomous coral snakes have red, yellow, and black bands, while the non-venomous scarlet king snake has the same colors in a different order. Animals afraid of venomous snakes would do well to avoid 4 foot long snakes with red, yellow and black stripes, in the process avoiding the scarlet king snake (alternatively, one could remember the rule "Red on yellow, kill a fellow; red on black, friend of Jack").
The process has been observed in insects, reptiles, mammals, and plants, and sometimes occurs between species. By parasitizing the true warning signal of the protected species, the Batesian mimic gains the same advantage without having to go to the biological expense of maintaining a poison. The species being mimicked, on the other hand, is disadvantaged, along with the dupe who misses out on tasty mimic meals. If impostors appear in high numbers, positive experiences by the predator with the mimic may result in the model species losing the benefits of signaling its poison.
Atsushi Yamauchi has shown that when there are density effects on the model species, there is no stable equilibrium. Nonlinear dynamics make the system’s aggregate features unpredictable in specifics, but most importantly, it is not a stable equilibrium to have no mimics over long periods of time: the gains are large to the mimic because predators obey the model’s high-quality signal.
While it’s conceivable one could generate a formal economic model with these qualitative results, note that the ecological literature mainly looks at comparative statics for one species, noting what assumptions generate stable equilibria, and which do not. There is no attempt to generate a dynamic model of the mimic or models success over time, presumably because the highly nonlinear, recursive system is so sensitive to results would merely be qualitative, like the comparative statics.
In an expansion investors are constantly looking for better places to invest their capital, while entrepreneurs are always overconfident, hoping to get capital to fund their restless ambition. Sometimes, the investors (dupes) think a certain set of key characteristics are sufficient statistics of a quality investment because historically they were. Mimic investors seize upon these key characteristics that will allow them to garner funds from the duped investors. The mimic entrepreneurs then have a classic option value, which however low in expected value to the investor, has positive value to the entrepreneur. The mimicry itself may involve conscious fraud, or it may be more benign, such as naïve hope that they will learn what works once they get their funding, or sincere delusion that the characteristics are the essence of the seemingly promising activity. The mimicking entrepreneurs are a consequence of investing based on insufficient information that is thought sufficient, but they make things worse because they misallocate resources that eventually, painfully, must be reallocated.
Once the number of mimics is sufficiently high, their valueless enterprises become too conspicuous and they no longer pass off as legitimate investments. Failures caused by insufficient cash create a tipping point, notifying investors that some of their material assumptions were vastly incorrect. Areas that for decades were very productive, are found to contain exceptional levels of fraud, or operate with no conceivable expectation of a profit. Everyone outside the industry with excessive mimics marvels at how such people—investors, entrepreneurs, and their middlemen--could be so short-sighted, but the key is that the mimics and duped investors chose those business models that seemed most solid based on objective, identifiable characteristics that were, historically, correlated with success. An econometric analysis would have found these ventures a good bet, which is why investors did not thoroughly vet their business models (banks, up through 2007, were one of the best performing industries since industry data has been available in the US, and performed well in the 2001 recession).
In the 1990’s tech firms in general and internet firms in specific were doing very well. The internet bubble was filled with a naïve lack of skepticism that allowed otherwise absurd business ventures to get funding. Using hindsight there were so many businesses with doomed business models, you wondered how they could have been taken seriously, but investors were looking primarily at a few key criteria—net presence, branding—and these did work well for several years until the March 2000 crash, especially using the criteria of their stock price. Consider that Enron was able to engage in negative cash flow activities for at least 5 years while their stock price kept climbing, highlighting that if you hit the key signals investors are naively prioritizing, they can be fooled, just not forever.
AllAdvantage was a website that paid members to surf the Net. It paid to acquire these users, and supposedly leveraged its members’ eyeballs into advertizing dollars. At the initial fundraiser internet capitalist Frank Quattrone, who helped fund Cisco and many other legitimate successful concerns, and President Clinton both paid tribute to AllAdvantage. Yet even then, an investigation into AllAdvantage had determined that the clicks came mostly from programs that were designed to look like human web surfers, to whom AllAdvantage was sending monthly checks. The business model was a sham, but also filled with a lot of general naivete, and the initial IPO for $1B was only stopped by the popping of the internet bubble in March 2000.
Similarly, the housing bubble of 2008 was based on the idea that the borrower’s credit was irrelevant because the underlying collateral, nationwide, had never fallen significantly in nominal terms. This was undoubtedly true. The economics profession, based on what got published in top-tier journals suggested that uneconomical racial discrimination in mortgage lending was rampant, as if lending criteria for borrowers was historically excessively prudent. Well-known economists Joe Stiglitz and Peter Orzag wrote a paper for Fannie Mae arguing the expected loss on its $2 trillion in mortgage guarantees of only $2 million dollars, 0.0001%. Moody’s did not consider it important to analyze the collateral within mortgage CDOs, as if the borrower or collateral characteristics were irrelevant. In short, lots of smart people thought housing was an area with extremely low risk.
Each major bust has its peculiar excesses centered on previously prudent sectors. After the Panic of 1837, many American states defaulted quite to the surprise of European investors, who were mistakenly comforted by their strong performance in the Panic of 1819 (perhaps the first world-wide recession). The Panic of 1893 centered on railroads, which had for a half century experienced solid growth, and seemed tested by their performance in the short-lived Panic of 1873.
Other features of this model explain various stylized facts: For example, subsequent to the 1990 Commercial Real Estate debacle, defaults in this asset class were well below average across the board for the subsequent decade. In the aftermath of that crisis, newly issued Commercial Real Estate Asset Backed securities did very well, because everyone was especially cognizant of the risk factors involved. A similar thing happened in railroads after the Penn Central railroad defaulted in the early 1970s. Fooled once, investors were not fooled again in that specific way, making the key risky asset characteristic of the latest recession a metric of safety going forward.
In 1929, Irving fisher’s famously opined “Stock prices have reached what looks like a permanently high plateau." In 1993, Stanford macroeconomist Robert Hall said about the recession of 1990 that “established models are unhelpful in understanding this recession”. Econometricians James Stock and Mark Watson noted that the 1990 recession, which blindsided their new econometric forecasting model at that time, experienced a sharp fall in consumption, housing, and durable goods; whereas in 2001, the updated model failed to note that the recession was centered on technology investment. Further, the signals change, as yield curves were not characteristically flat or inverted prior to the 1990 recession, and housing permits remained strong throughout the 2001 recession. Stock and Watson note “our conclusion—that every decline in economic activity declines in its own way—-is not new.” I think it's fair to say the latest recession has kept the string of unpredicted recessions perfect.
If mimicry is the essential driver of the misallocation of resources that inevitably must be corrected, it by definition occurs in places that do not have accurate quantitative signals; indeed, it preys upon areas where the signals are misleading, and so have less skepticism. Safety creates risk in that eager overzealous entrepreneurs, once they figure out what sufficient statistics work on investors, quickly jump on these sectors with not merely excess capacity but business models that never stood a chance.
This model explains why business cycles are not forecastable, it’s inherent in the mimic's selection process. The recalculation Arnold Kling mentions relates to an investing error of a particular expansion, which is always unique. Mimicry explains why the biggest winners in a business cycle are also the biggest losers: their productivity was pervaded by fraudulent and incompetent mimics. It explains why the biggest losers of the prior business cycle often do above average in the next recession: investors are wary of mimics, so mimics only thrive where they are not expected. It explains why recessions are concentrated in certain sectors, and why these sectors are different each recession.
Efforts to prevent the next recession face a large difficulty, in that the impetus for the next recession by necessity will be in the area that invites the least concern, because that is where mimics fester. Any risk analysis that can identify risky ventures necessarily identifies safe ones, and when these safe investments become known to the mimics, they will be attacked. Top down risk management, the focus of so much policy talk in Basel, Washington, and wonky journals is futile, because risk grows dangerously only where one does not suspect it (G-7 sovereign debt, anyone?).
This suggests focusing on robustness, as opposed to prediction, because the system works against rational expectations, especially those consensus ideas that come out of large bureaucracies. After all, what better sufficient statistic for a mimic to exploit than some well-known regulatory bullet point that supposedly ensures no risk? Recessions are not going away; they are endogenous because zero mimicry is not an equilibrium among insects, reptiles, or humans. Expect more unexpected recessions, just not real soon, and not in subprime housing.