Bureau of Labor Statistics Response to Birth / Death Adjustment Critics

OK, our title is false advertising. The Bureau of Labor Statistics (BLS) crew is not allowed to surf the Net making comments on blogs, nor do they have a blog of their own. It might be a good idea, but it is an unlikely move given budget constraints.

This is too bad, since there is near-universal criticism of their methodology. Many go much further. A Google search will reveal plenty of aggressive name-calling critics. The criticism has been so loud and pervasive that hardly anyone in the blogosphere or trading worlds believes in the monthly non-farm payroll report. Many sites routinely mention the birth/death adjustment so that the reader can mentally subtract these "phantom" or "magical" jobs.

This presents an interesting situation. What if the BLS approach is correct and accurate? Those understanding this would do better in gauging economic changes.

Our Mission

Since the BLS is not going to respond directly to critics, we propose to use their existing results and words to address some of the key points. In this article, we will show the strength of the BLS methods with only indirect references to the many critics. In future articles we will directly analyze and expose pervasive errors on this topic. Reader questions are invited.

We have three steps: Showing the accuracy of the birth/death adjustment, explaining the b/d role in job creation, and showing how the research design effectively captures economic changes.

This article takes up the first of these issues.

Accuracy

Estimating the number of jobs and the monthly change in jobs is a daunting challenge. There is a way of keeping score. As we wrote last October:

Each year the BLS makes a "benchmark revision" to the payroll employment series based on the establishment survey. The purpose of this is to make sure the survey data are consistent with the actual count of jobs from state unemployment insurance tax records.

The state data is much better, of course, but it is not available in a timely fashion. The benchmarking is a reality check. It allows the BLS to see how well they did with the monthly estimates. Each October, along with the report on September employment, the BLS releases the preliminary version of these benchmark revisions.

This is the report card for the BLS.

This should be a non-controversial test, since it relies upon actual state data, not projections. No employer is going to pay extra taxes, so this count does not include any "phantom jobs."

The better the BLS methods, the smaller the benchmark revisions. If the Birth/Death adjustment is effective, it makes the revisions smaller.

And it does!!

Here is a nice chart (click to enlarge) showing the effects.

The blue line is the actual count. Just compare the red line to the green line. The red line shows what the estimate would have reported without any birth/death adjustment. The green line shows the effect of birth/death.

The birth/death adjustment improves the job change estimate in every quarter since it has been introduced.

Conclusion to Part One

Most of the BLS critics have been offering the same complaints for many years, but no one ever asks whether they were correct. The closest the BLS team will come is the paper they published last October.

In this article we have emphasized that something about the birth/death adjustment is good, very good. It improves the job change estimates in every quarter.

This seems counter-intuitive. How can we have new job creation in such difficult economic times? Most people believe their intuition rather than the data.

In the next article in this series we will explain this mystery.

Comments

[Neil459:]

What a load. There is no examination of collection and summarization biases that likely exists. There is no examination of the method of adjustment or factors used in adjustment. No details whatsoever. What this article says is, "hey its seems like it aught to work, so its good."

Now, I am no specialist in this field, so I am willing to be proven wrong. But please it you are going to write authoritatively please provide some backup for your opinion.

Journalistic misconduct at its best.

Jul 07 04:34 PM

[Jeff Miller:]

Neil -- This is a simple statement of the facts. It compares the forecasts to the actual known results. The backup is the actual state unemployment count. As I noted, companies do not lie when they have to pay premiums.

The state data is a good source for any of us who want to be objective on the subject.

Meanwhile, I am curious. Is there any actual data that would convince you? I try to learn from every comment, but it helps if people try to be constructive. I am writing more in this series, and I promise to do my best in covering all questions.

Thanks,
Jeff

Jul 07 05:54 PM

[Neil459:]

On Jul 07 05:54 PM Jeff Miller wrote:
> Neil -- This is a simple statement of the facts. It compares the
> forecasts to the actual known results.

So I looked into the data a bit more. Actually it compares a statistical projection against known results for the last 6 years. It looks like they were doing it in full since 2003 even though it started around 2000 (www.bls.gov/ces/cesbdh...).

From BLS, "The second component is an ARIMA time series model designed to estimate the residual net birth/death employment not accounted for by the imputation. The historical time series used to create and test the ARIMA model was derived from the UI universe micro level database, and reflects the actual residual net of births and deaths over the past five years."

So you are asking use to believe that a statistics model in use for 6 years combined with a lack (at least that I could find) of analysis considering upheaval events is proven robust? This time period does not even include the last downturn in 2001. Also there is no indication how the model changes. Could it be that the model parameters are changed each year so that it produces better results?

This is what I mean by journalists just taking what they read at face value. Basically you reprinted the BLS paper at (www.bls.gov/opub/ils/p...) without challenging any of the assumptions.

I can take almost any time series model and prove it correct for some time period. That does not make it correct for the universe of time periods.

Now, it may be the best theory to-date and you may very well be correct as a matter of coincident so far. But the facts considered by a critical thinker are incomplete enough to NOT be able to say its correct as a matter of fact.

Jul 08 01:21 PM

[Jeff Miller:]

Neil -- Thanks for taking another look.

First, I am not a journalist. Among other things I taught research methods at a top university and my undergrad courses also covered critical thinking, your special interest. I have discussed these topics repeatedly on my blog.

So why would you assume that I did not investigate this research in some detail. If you check out my past work, you might make a guess at the hundreds of hours I have spent on this subject. It is not easy, and I have the advantage of a lot of grad work in econ.

So when I say the paper is good work, it is not just a superficial conclusion. I have spoken with several BLS specialists, and asked plenty of questions. I have reviewed many challenges.

The paper reported here was based upon model development extending back into the 90's. These are real-time results for the entire life of the model. What more could they do? They reported what happened.

Why don't you ask some of your questions to the many critics who attack the BLS work? Check out their data. Just provide a little "equal time :) "

It is true that they improve the ARIMA model each year with data from the prior five years. It is not true that they "cherry picked" a time when their method worked.

I hope you will read Parts 2 and 3 of this series for more information and data.

And thanks again for following up.

Jeff

Jul 08 06:22 PM

[John Lounsbury:]

Jeff - - -

You are courageous to keep trying to educate people, many of whom don't want to be educated. (Neil459, I am NOT talking about you - you obviously are trying to figure things out.) I am referring to other readers who have basically blown your efforts off, indicated by the absence of comments.

What I keep waiting for is someone who can provide an analysis suggesting how the birth/death adjustments have moved the employment statistics away from reality. The BLS has shown (with possible qualifications suggested by Neil) how the adjustments have moved monthly employment situation numbers closer to the final results when state data is finalized up to nine months later.

What is going to be interesting is whether or not the brief history of the B/D adjustment will be found wanting for the current period when the annual update is made in March, 2010. This is the first period with such dramatic employment market dislocations since the B/D research and application started.

This is an important area to keep looking at critically. There may be only a few who will second my request, but please don't stop doing this good work.

Jul 08 08:52 PM

[Ergo:]

Well, I appreciate the information and education from the author and commentators. I'd say more, but I have nothing to add. /carry on.

Jul 08 09:57 PM

[Neil459:]

On Jul 08 06:22 PM Jeff Miller wrote:
> First, I am not a journalist. Among other things I taught research
> methods at a top university and my undergrad courses also covered
> critical thinking, your special interest. I have discussed these
> topics repeatedly on my blog.

Yes, I saw it after posting and you do have some interesting and thoughtful research. Its unfortunate that my first impression was this post.

> So why would you assume that I did not investigate this research
> in some detail.

Because no one does (OK, a lot do not do) the research anymore, the post on seeking alpha was my first introduction to you and (sorry, no disrespect) but it was a puff piece (that I now know you had backup for.) In addition, a lot of research is just political propaganda in disguise. My goal is to root out both of these types of lies. Usually its pretty easy to uncover.

> If you check out my past work, you might make a
> guess at the hundreds of hours I have spent on this subject. It
> is not easy, and I have the advantage of a lot of grad work in econ.
>
> So when I say the paper is good work, it is not just a superficial
> conclusion. I have spoken with several BLS specialists, and asked
> plenty of questions. I have reviewed many challenges.
>
> The paper reported here was based upon model development extending
> back into the 90's. These are real-time results for the entire life
> of the model. What more could they do? They reported what happened.
>
>
> Why don't you ask some of your questions to the many critics who
> attack the BLS work? Check out their data. Just provide a little
> "equal time :) "

Yes, I read a couple and they were worse at analysis.

>
> It is true that they improve the ARIMA model each year with data
> from the prior five years. It is not true that they "cherry picked"
> a time when their method worked.
>
> I hope you will read Parts 2 and 3 of this series for more information
> and data.

Yes, I will.

It may sound boring but to me at least any article that just says "looked at it and its good" is suspect. A couple of sentences along these lines would have given me pause before writing.

- While the statistics are relatively new and have not been proven through a recession they nonetheless appear sound based on our extensive analysis and duplication of the governments work. For more info see ...

- As a part of our research we analyzed the government models and see no reason why the models will not hold up in the current environment. We applied the same models to 1971 and 2001 and found they worked well. For more information on our analysis see ....

You get the point, include some specific stand that you are willing to take in the accuracy of the models, not based on a government publication.

Finally, I have now grown to assume that all government statistics are bad. Look at how CPI is calculated and the negative impact this one statistic has on everything. Your statement, "Since the BLS is not going to respond directly to critics, we propose to use their existing results and words to address some of the key points."

I do NOT want to hear the governments words I want to hear about oversight and confirmation from an independent source. This appears to be what you provided but not what was stated in the article.

Thanks for taking the time to stick with this discussion. I have learned a lot and I hope others have also. I am always willing to change my mind based on real facts and information.

Jul 09 08:59 AM

[Neil459:]

On Jul 08 06:22 PM Jeff Miller wrote:
> It is true that they improve the ARIMA model each year with data
> from the prior five years. It is not true that they "cherry picked"
> a time when their method worked.

But my previous question was, "did they cherry pick model parameters that worked with the time period?" That appears to be yes. So to me the accuracy is still open. There is no data to say it doesn't work, but also no data to prove conclusively that it does under all circumstances. How material are the changes to the model? And what would be the results if the model was not changed each year? Maybe you have a link that would answer these questions?

To me, one of the most important statistics was not in your article. That would be the percent difference between the statistical forecast and the final adjusted real value for each time period and whether or not this difference showed a bias. The government data indicates that this is IIRC less then 1%. The 1 to 500 in the chart does not mean anything to me as a non-specialists in this area.

Being pretty good at math but not an expert, when I look at a chart and do not understand it, I also assume that someone is trying to obfuscate the situation (which was not the case.)

Jul 09 10:24 AM