Not Even Wrong: Why Data-Mined Market Predictions Are Worse Than Useless

|
Includes: JNK, SPY, UUP
by: Mercenary Trader

Summary

The standard deviation for the S&P the past ten days has been the lowest in 15 years. One writer believes this creates an 89% chance of bullish action.

This “89% chance” way of thinking makes no sense to us. In fact, we would consider it “not even wrong.”.

The potential trajectory of equity markets is directly impacted by the trajectory of debt and currency markets (which are the opposite of boring now).

We have no desire to pick fights with fellow members of the financial blogosphere. But as long-time Mercenary readers know, sometimes we just have to rant a little bit. Call it a release valve of sorts - plus a chance to communicate our philosophy and the lens through which we view markets.

The latest piece to trigger the "rant reflex" was from Yahoo Finance Contributor Ryan Detrick, in a piece called "Why Boring is Bullish." The gist goes as follows:

  • Standard deviation for the S&P 500 (NYSEARCA:SPY) the past ten days (as of 09-10-14) has been the lowest in 15 years.
  • This indicates a "historically boring time" (the exact phrase used).
  • Since 1970, there were 18 previous instances, or "signals," of this low-volatility occurrence,
  • There is thus (Detrick's conclusion) an 89% chance the S&P will be higher in three months.
  • (Really ?!?)

Or to sum the data-mined argument even more tightly:

We just had a REALLY boring stretch of days in the S&P 500… going back to 1970 there were only eighteen other such stretches… so based on these eighteen instances, the S&P has an 89% chance of being higher in twelve weeks' time.

This "89% chance" way of thinking makes no sense to us. In fact, we would consider it "not even wrong."

What does "not even wrong" mean? Wikipedia sums it up nicely:

The phrase not even wrong describes any argument that purports to be scientific but fails at some fundamental level, usually in that it contains a terminal logical fallacy or it cannot be falsified by experiment (i.e. tested with the possibility of being rejected), or cannot be used to make predictions about the natural world.

The phrase is generally attributed to theoretical physicist Wolfgang Pauli, who was known for his colorful objections to incorrect or sloppy thinking. Rudolf Peierls documents an instance in which "a friend showed Pauli the paper of a young physicist which he suspected was not of great value but on which he wanted Pauli's views. Pauli remarked sadly, 'It is not even wrong'." This is also often quoted as "It is not only not right, it is not even wrong," or "Das ist nicht nur nicht richtig, es ist nicht einmal falsch!" in Pauli's native German. Peierls remarks that quite a few apocryphal stories of this kind have been circulated and mentions that he listed only the ones personally vouched by him. He also quotes another example when Pauli replied to Lev Landau, "What you said was so confused that one could not tell whether it was nonsense or not."

If an argument is "not even wrong," the conclusion is irrelevant - because the means of arriving at the conclusion are so deeply flawed as to be useless, or worse than useless in being misleading.

Let's visit some problems with the "boring is bullish" argument - and again I invite you to check out the piece in question here - and then we'll look deeper into the problematic mechanics.

A "Historically Boring Time?" Seriously?

We'll begin with a chart from the article, used to support the "boring" conclusion.

Cool chart. I don't know about you, but the first thing I notice looking at that chart is not the potential for more "boring," but rather the gigantic spikes that come along with disconcerting regularity. Interesting we haven't had one in a few years - and haven't had a REALLY big one since 2008, notable in that markets have a multi-decade track record of crashing every four to six years or so.

But that's beside the point. The article directly asserts "we are seeing a historically boring time." And also notes right there in the title, "Why boring is bullish."

So to play devil's advocate, let us begin by asking the following:

Are investors in the high yield market having a "boring" time with junk bonds going into freefall, roaring back, and now in freefall yet again?

Are Europe-based investors having a "boring" time, given the euro's biggest drop in years last week and the British pound's panic over the Scottish independence vote?

Is the US dollar looking "boring" having lifted off like a NASA space-shuttle launch?

Are crude oil investors "bored" as oil prices fall like a hot rock? How about neighbors of Russia? Are they bored with what's up in Ukraine?

Take a quick look at Barclays high yield bond index (NYSEARCA:JNK) and Powershares US Dollar (NYSEARCA:UUP)…

There are multiple points to be made here.

To speak of a "historically boring time," based on observation of one index, is a qualitatively false statement. You might say we are in a low volatility period for the S&P 500 index on a stand-alone basis, but that is saying a qualitatively different thing, and is a much less sweeping statement. At the very minimum, speaking of "boring times" based off one instrument is sloppy. At worst, it is so misleading as to be dangerous. (And by the way, didn't the S&P have its largest single-day drop in two years in July?)

The potential trajectory of equity markets is DIRECTLY IMPACTED BY the trajectory of debt and currency markets (which are the OPPOSITE of boring now). If investors are moving out of junk bonds en masse - if high yield is flashing serious warning signs - can we really say that has no bearing on the potential trajectory for stocks? Or if the dollar is taking off like a rocket ship - and remember, the revenues of giant multinational corporations in the United States are priced in dollars, as is the price of major commodities like oil - can we really say that trade and profit outlooks are not impacted by this, perhaps by a lot? Especially when we know that fears of major sea change on the part of global central bank policies, namely the Fed and ECB, are the big factors driving this volatility? "Calm before the storm boring," maybe. Plain old boring boring? Ah, no.

For a market environment to qualify as boring, we have to know what MULTIPLE key asset classes are doing. Let's say the S&P was sleepy, AND the debt markets were sleepy, AND the currency markets were sleepy, AND the trajectory of monetary policy for global central banks was completely on auto-pilot (as opposed to, I don't know, the BIGGEST SEA CHANGE IN YEARS). If all the related markets that have significant potential impact on the trajectory of equities were showing abnormally low volatility together, then yes, to speak of a historically "boring" environment might be a statement qualified by general empirical data. But this environment? Again, no way…

For all we know, the past sample periods referenced really WERE boring - in a way that this one is NOT - because the S&P was not sitting in the eye of brewing storms. But then again we really don't know, do we? Because the study failed to look at historical volatility correlations for high yield debt, currency movements, monetary policy shifts or lack thereof, or other key factors that would have PLAYED A KEY ROLE in determining statistical validity of an assumed outcome.

To simplify what I am saying here: Markets are far from simple. In fact they are very complex. As such, predictions based on data mining of a single historical variable or single cherry-picked pattern observation are almost always worse than useless because they ignore a core confluence of factors.

It is trivially easy to come up with false relationships based on simplistic or agenda-driven data mining. Say, for example, the International Kale Growers Association came out with a study showing that people who eat kale tend to have lower body fat and better cholesterol. Even if the numbers are valid, one would have to ask: Are those good results attributable to kale? Or are they attributable to the fact that the kind of person who eats kale in the first place is much more likely, statistically speaking, to exercise and lead a consciously healthy lifestyle? Even if the numbers add up, what are the core driving factors that PRODUCED the numbers? Without contextual understanding we simply don't know, and that is the point.

Similarly, when making data-based assumptions about, say, the future outlook for the S&P over 12 weeks' time, we have to look at a broad confluence of environmental factors in proper context. If you can't do that, you are more than likely missing very important correlative or causative factors that may well render your entire conclusion useless, e.g. "not even wrong."

Not to mention the complete dismissal of COMPETING historical pattern factors focused on OTHER variables in isolation that come to OPPOSING conclusions… so is the challenge trying to decide whether to trust single-variable study ABC, which is bullish, or single-variable study XYZ, which is bearish? NO, the answer is to not give credence to logically spurious, cause-and-correlation-blind single-variable studies at all!!!

The 80s and 90s were a little bit different…

Here is another data chart from the "Why Boring is Bullish" piece, showing the dates of the signals used to come up with the "89% chance of being higher" conclusion. Notice what those years tend to have in common…

Okay, here are the two major issues with this "signal set":

Most of the years in which the signals occurred were already quite bullish for macro-driven reasons independent of short-term volatility or lack thereof. The signal set starts in 1985, when markets were cruising toward 1987 highs on strong macro tailwinds. It then picks up in 1988, nearly a year after the '87 crash, after Greenspan's first instance of opening the liquidity taps had a chance to re-engage the still-rather-young leverage and debt supercycle. Then into the nineties we go post-Iraq war as expansion continues…then 1996, when Greenspan gave his "irrational exuberance" speech as markets got, umm, exuberant…then a big jump to 2005 / 06, when private equity and the housing bubble were rocking… and then into the 2012-2014 era, the heart of the "Bernanke Put" aka "Forever QE" years.

Bottom line being, all of these past signal dates were associated with a strong confluence of surrounding bullish macro factors, as such you might be able to throw a dart in those years and see the S&P higher…and worse still, there really aren't enough data points to back the assumption in the first place. We are back to the "Kale makes you healthy" assertion:

  • Does kale empirically improve the health of people who eat it…?
  • Or are healthy people more generally disposed to eating kale?
  • Do low volatility S&P stretches truly imply likelihood of bullish follow-through?
  • Or is the sample set overwhelmed by other completely unrelated bull factors?

By the way, regarding my "throw a dart to see the S&P higher" comment, look at this chart (via ZRH) to see what the S&P was doing in the 1985-2014 signal window.

(Hint - it was going up, on a pretty constant basis…)

Now, can we really draw a statistically valid conclusion that is almost laughably certain - 89% likelihood of being higher - about what the S&P is going to do in twelve weeks' time, based on a handful of instances pulled from an EXTREMELY bullish historical macro backdrop, with no sense of the macro drivers then or the macro drivers now, further based on a qualitative assertion (it is a "historically boring time") that is empirically false?

No we can't. We can't draw any kind statistical validity from a cherry-picked, data-biased, factor-blind, flaw-ridden data mining exercise like that at all.

Which leads to some general assertions:

When it comes to predicting future outputs of complex systems, virtually ALL forms of single-variable statistical thinking are flawed. They are just too prone to be missing something. For any given market event "X," there is not just one cause. There are MANY causes. That is why replication of past price patterns - slicing and dicing statistical data points - is a highly dubious exercise. There are too many ways to completely overlook key hidden factors, get fooled by correlations, or otherwise just miss the real drivers completely.

Predicting the future in general is a foolhardy exercise - and unnecessary for good trading. Good traders, like good poker players, understand that crystal balls are useless. The way to make money is with odds and probabilities. But you determine odds and probabilities through a confluence of interlocking factors, not confidence expressed in a single isolated metric with zero context to support it or logical hypothesis to back it. And you seek to make high probability wagers with attractive payoff expectation over a full cycle, while remaining agnostic as to future predictions. The winning poker player does not "predict" what the turn or river cards will be, or even what the outcome of any particular session will be. He doesn't have to. He simply has to apply high-quality situational analysis, combined with skillful integration of probability, observation and intuition, to make intelligent choices that pay off on a probabilistic basis over the long haul. There is nothing "predictive" in that, and good trading and investing is similar.

If you don't use common sense, it is way too easy to get fooled by spurious or invalid data patterns. Imagine a quantitative value investor who determined that, based on studies of Warren Buffett's most successful investments, the ideal age range for a company CEO was between 46 and 59 years old. Imagine if this quantitative value investor then made capital allocation decisions based on this criterion. On one level "it's in the data" - he may have done a study and come up with eighteen samples, just like in the piece we are criticizing. But just because it is "in the data" doesn't mean the relationship is relevant. It doesn't mean a tight band of CEO age range, or a tight band of compressed S&P volatility, is the factor that really mattered in respect to producing the collection of positive outcomes observed. It could have just been a random quirk (a conclusion that basic logic would support, in the absence of a plausible hypothesis). If you don't have underlying knowledge of the "what" behind cause-effect relationships…and if you lack the intellectual discipline to do a "why" logic-check behind your assumptions…then you are prone to getting bamboozled.

The way to really understand markets is by understanding the deeper structural mechanics. Stanley Druckenmiller once said (paraphrase) that not many Wall Street analysts actually understand what makes a stock go up or down. There are many analysts who get caught up in metrics and factors that may be intellectually engaging, or have relevance to a chosen argument, but are not actually key to material changes in price in the medium term (or sometimes even the long term). How do you get to where you understand "what makes a stock go up or down?" How do you get to where you understand what makes a MARKET go up or down? You focus on the structural mechanics. You learn the real relationships and the true core drivers that directly influence outcomes in a meaningful way. This is hard to do…it takes nuance and subtlety…and it is far from the fool's gold simplicity of firing up a spreadsheet.

Again, there was no desire to single anyone out or trash someone's point of view. But this is a critical point that is hard to emphasize enough: There are a lot of questionable assumptions out there, some of them worse than useless in the manner they misdirect and mislead. The only way to avoid getting fooled by spurious data or superficial thinking is to put real elbow grease into truly understanding what drives markets and why…and once you have that understanding you don't need to cherry pick or data mine because you have something better: The ability to assess a confluence of key factors in the present, as impacting important market relationships here and now.

We expend a lot of effort watching and interpreting what is happening in markets. We frequently reflect on key drivers…where institutional money is flowing…what the big impacting factors are…where the next surprise may come from….and so on. We have also inhaled countless classics of financial history and past market environments - most recently The Plungers and The Peacocks by Dana Thomas.

Yet we spend roughly zero time on data mining, with no interest in statements like "Over the past X years, the S&P did this X percent of the time."

Why this contrast? Because markets are a complex sea of swirling and interlocking variables - and it is the historical drivers and qualitative cause-effect relationships are what have lasting value. It is not the output of a spreadsheet that matters - the pattern-based cherry picking lacking insight as to what created the results - but the qualitative relationships truly attributable to joint causation of various outcomes, on a case-by-case basis, with a very big nod to history and context.

Our views are nuanced and complex. We don't apologize for that because so are markets. All too often, the appropriate answer to a "simple" question is: "It just ain't that simple." The game is hard, and it takes a long time to truly get good. One step forward, though, is developing a clear sense of statistical awareness, which includes awareness of the limits that any historical data-mining approach will quickly run into.

We've barely scratched the surface of this topic - the "philosophy" of how to think about market relationships, past data, and other such things - but for anyone interested in more, you might like this piece, "Lessons From the Palindrome," which gives insight into our way of thinking (as shaped by the legendary George Soros). The water gets deep, but it's a fascinating journey (in our humble opinion) and may be well worth it.

Disclosure: The author is short IWM. Also holds a bearish structured option position on SPY along with short AUDUSD, EURUSD and long USDCAD, long USDJPY.

The author wrote this article themselves, and it expresses their own opinions. The author is not receiving compensation for it. The author has no business relationship with any company whose stock is mentioned in this article.

About this article:

Expand
Want to share your opinion on this article? Add a comment.
Disagree with this article? .
To report a factual error in this article, click here