Where have all the fatties gone? Mass DOPH releases flawed study

I've been skeptical of the obesity crisis in this state ever since the media started screaming about it five years ago. What the statistics are telling me simply does not jive with what I have been seeing with my own two eyes.

And finally, I have been able to identify the problem: the methodology.

First, the latest statistics. A Profile of Health Among Massachusetts Adults, 2004 is an annual survey done by telephone sampling. The results were just publicized this week. The report found that in 1990, 10% of the adults statewide were obese. In 2004, the figure had risen to 18%.

Wow, that's terrible, you think. What's wrong with people? Why all the fatties? Newspaper and TV reporters casting about for answers invariably find a doctor or public health expert who points to the Internet, video games, sedentary lifestyles, high-calorie foods as creating a crisis.

I don't deny that there are more fat people now than in 1990, and these factors are likely to blame. But I have serious doubts about the new statistics. The reason has to do with the methodology. The 8,203 people who form the basis of the 2004 state data were surveyed by telephone:

The sampling of the survey population involves a list-assisted, stratified [random-digit-dial] sampling frame, which assures that Massachusetts households with telephone numbers assigned after publication of the current directories, as well as households with deliberately unlisted numbers, are included in the sample in appropriate proportions.

The problem is the methodology does not address major changes in phone technology. I read the whole document, and there is no mention of caller ID or mobile phones, both of which did not exist in 1990 but are now used by a significant portion of the population in this state. I screen all calls to our home phone and simply won't pick up the phone if I don't recognize the number. I am sure most of the people reading this blog are the same. However, 15 years ago, we didn't have this luxury -- we had to answer every call, including those from pollsters. As for mobile phones, not only do most of them have caller ID, but I also suspect many local handsets wouldn't be included in the survey because they use non-local area codes.

What does this mean for the 2004 data? I believe it undercounted more active and more technologically savvy people, who are more likely to use caller ID and mobile phones, while overcounting less active, less healthy, and less technologically savvy people, who are more likely to be using traditional landline phones without caller ID.

The survey methodology claims the final results "partially" takes into account "non-response" by weighting undercounted sections of the population, but it does not list people using caller ID or mobile phones as non-respondents.

This flawed methodology would also help explain the unusual one-year jump in Figure 3.5 in the report, which says the number of obese adults jumped from 14% in 1999 to 17% in 2000. That's about 145,000 people who suddenly joined the rolls of the obese (extrapolating from 2004 U.S. Census totals) in one year, even though there was NO SIGNIFICANT CHANGE between 1998 and 1999 and 2000 and 2001. Yes, certainly more people were spending more time in front of a computer screen in 2000 compared to the previous year, but also more people were using caller ID and mobile phones as well.

I'd also like to slam the Boston Globe and reporter Stephen Smith for doing incomplete reporting on this topic and exaggerating the findings. His report didn't mention the sample size, or any information about problems with the methodology. Like many journalists, he rounds up for the sake of convenience -- saying "nearly one in five adults now dangerously overweight" -- instead of stating the actual findings, which is "18% of Massachusetts adults were obese based on their reported height and weight." The difference between 18% and 20% of adults is 98,044 individuals, based on the U.S. Census figures for Massachusetts for 2004.