How was that number calculated, and why does it matter?

When you see a number based on psychological research--say, how happy people in different countries are, or the percentage of women who experience sexual assault--you probably already know to ask, "How was the variable of interest defined and measured?  What did they mean by happiness or sexual assault?"  There's a less obvious question that's almost as important: "How did they calculate this particular number?"

I recently read an economics article that vividly illustrated why it's so important to know how the numbers are calculated.  Ordinarily I don't write about economics as I know very little about it, and so can't spot obvious factual or logical problems in an argument that a real economist would notice instantly.  In this case, it shouldn't matter. 

The Bureau of Labor Statistics (BLS) just reported that the USA added 114,000 jobs since September, and the unemployment rate is now 7.8%, down from 8.1% in August.  This would seem to suggest that unemployment has been dropping.  The problem is, the numbers actually reflect seasonal adjustments in how unemployment rates are calculated, and the rates can't be compared from month to month.  Counterintuitively, we simply can't tell from the current numbers whether unemployment was lower (or higher, or unchanged) in September than August.

How can this be?  John Williams of shadowstats.com explains:

Each month the agency recalculates the series to adjust for regular seasonal patterns tied to the school year or holiday shopping season or whatever is considered relevant. The next month, it does the same thing using another set of seasonal factors. Rather than publish a number that's consistent with the prior month's estimate, it recalculates everything, including the previous month, but it doesn't publish the revised number from the previous month.

The assumption is that the monthly recalculations don't make much difference over time, but they do. The depth and the protraction of the current severe economic downturn have thrown off the annual seasonal-factor adjustments. The result is very volatile seasonal factors month-to-month. That means the new calculations for the September number may have resulted in a very significant revision to the August number. Again, though, the BLS doesn't publish that, so the headline August-to-September 2012 change in the unemployment rate is not consistent and not comparable.

I don't know enough about economics to judge whether the current economic downturn has actually made month-to-month adjustments more volatile.  Either way, numbers are still being reported without context, leading reporters, politicians, and ordinary people to make assumptions about the economy that may or may not actually follow.  Psychologists should be quite familiar with this state of affairs.

If I were a BLS researcher, I would either explain the limitations of seasonally-adjusted numbers, or just give the new August rate along with the September one.  Probably the latter, as reporters want to tell a story and they won't see any use for statistics from which they can't draw conclusions.

Definitely keep this issue in mind the next time you see a headline on the number of kids with autism.