Every day we are inundated by numbers that seemingly prove something. Yet, many are improperly calculated, selected, analyzed and presented. Our defense is to…
No one group is immune. Even professionals and academics commit statistical mistakes. There are three primary causes of this problem:
First: Ignorance or forgetfulness of proper statistical techniques. With today’s computers and software, statistical analysis is push-button easy. The problem is most data doesn’t qualify for the statistics used, so the results are worthless. (We’ll talk more about this below.)
Second: Improper theoretical basis. Computers also make it easy to throw loads of data at a statistical package, looking for some type of apparent relationship. Then, a plausible rationale (theory) is put together. However, statistical analysis is designed to test a theory, not create one. The backwards process, known as “data mining,” is highly flawed, and the results are not to be trusted. Chance can produce remarkable results simply by happenstance. The problem is no one let’s us know that this was the approach taken, although it is sometimes obvious. (Sports statistics provide numerous examples every weekend. Recently, a hockey goalie was asked about his top-ranked “Saturday game” record. He correctly responded that the only way to properly evaluate his skill is to look at his entire record, not just Saturdays.)
Third: Picking and choosing to support a point. Shopping for the desired result to prove a point (“cherry picking”) is disingenuous (a nice word for an unethical tactic used to mislead). It can be done to give a weak analysis false importance or, worse, to convince someone to believe something and/or take an action.
The most common investment statistical error
Most statistics require an evenly or normally distributed pattern of data. “Normally distributed” means that basic hill shape we envision, like the following coin flip graph illustrates.
Note: To get a better overall picture of a data set, statisticians use not only the average (which they call the “mean”), but also the “median” (the mid-point) and the “mode” (the most numerous group of data). When the data are evenly distributed, the mean, median and mode are approximately the same. When the three numbers are fairly different, the data is somehow “skewed” and cannot be analyzed well using standard statistics.
However, many (most?) types of investment data are not evenly or normally distributed. Applying standard statistics to such information produces unusable results, yet are often cited as meaningful and the basis of some conclusion.
The American Airline’s Boeing 757 reporting illustrates the problem
Boeing produced 757s from 1982 through 2005 – so, an age range from 28 to 5 years. American Airlines (AMR) has been a regular user of 757s and currently has a fleet of 124 planes ranging in age from over 21 years to 8 years (according to AIRFLEETS.com).
This is where statistics enters the picture:
“The Boeing 757 [involved in the incident] is about 20 years old, [Keith] Holloway [a spokesman for the National Transportation Safety Board] said. American Airlines declined to confirm the age of the plane, but said the average age of its 757 fleet is 16 years.”
(“Authorities are investigating a hole that forced an American Airlines emergency landing at MIA” – The Miami Herald, by Jennifer Lebovich, October 30)
“Average” is an innocent sounding statistic that can be misleading for unevenly distributed data. Humorously, there is the joke that a statistician drowned in a pond with an average depth of 6 inches. More seriously for investors is the “barbell” bond portfolio composed of US Treasury Bills and long-term junk bonds that carries average characteristics of intermediate-term and investment grade.
The fact that American Airlines “declined to confirm” the plane’s actual age and presented only an “average” raises a red flag. (Airlines know that passengers would prefer not to fly on old planes, so there is a desire to present their fleet in the best light.)
The following graph shows the yearly age groupings of American’s 757 fleet, giving us a more accurate view of a clearly skewed data set. It also shows that the average is of little help in viewing last week’s incident. The “barbell” clumping at 9 and 10 years pulls the average down. More importantly, the mode is around 20 years, meaning that their 757 fleet has a lot of planes around the age of the one that had the problem.
Note: My purpose is not to single out American Airlines but to use them as an example of how statistics often are used to create a desired, but misleading, impression. Why didn’t they just confirm the 20-year age of the problem plane and say that, because about one-half of their 757 fleet is around that age, they are working closely with NTSB and Boeing to …? Because they worry about negative fallout. So, they chose the disingenuous route. Many take the same approach.
So… As investors we must continually be skeptical of statistics used to prove something, particularly if that “proof” supports a call to action. The age-old warning remains alive and well: “Figures don’t lie, but liars figure.”
Disclosure: No AMR positions