Grouping Variables: Scientific Fallacy of the week

First, please read this post about a study on stroke risk.

I read this the other day and got to thinking (I know, dangerous).  There are a lot of ways in which scientific research has steered us wrong over the last 50 years or so that share a very specific commonality that I haven't addressed much before.  In all of them some variable was used and compared to some outcome - positive, like protein synthesis, or negative, like heart attack raate.  In all of them the variable measured failed us because it wasn't specific enough.  Let me explain with what we used to call in my field a thought experiment:

Suppose, for example, that red (and only red) M&M's contained some toxic chemical that caused liver cancer after prolonged exposure, and that nobody knew this.  Suppose some intrepid and well meaning researcher was concerned about liver cancer and spent a lifetime examining epidemiological records (perhaps by studying data collected about the diets and health over a long period of time for a large group of nurses, or doctors, or teachers) to search for possible risk factors for liver cancer.  At some point this researcher might either notice or suspect that chocolate eating was linked to liver cancer - maybe he has some theory to explain this, maybe not.  So he compares the statistics surrounding chocolate eating to liver cancer rates - what do you think he'll find?

Keep in mind that in my imaginary example red (and only red) M&M's contain some unknown substance that causes liver cancer at a moderate rate (that is, not everyone who eats them gets liver cancer, but some do).  I think it's clear our researcher will find a nice correlation of some kind between chocolate eating and liver cancer - people who eat lots of chocolate will get more liver cancer, all else being accounted for, than those who eat none.  Why?  Because people who eat no chocolate certainly aren't eating any red M&M's, while some of those who eat lots and lots of chocolate eat some of that chocolate in the form of red M&M's.  The chocolate eaters are more likely to be exposed to our imaginary toxin.

What would happen if this guy did some real science?  Suppose he took two groups of people and had one group eat more chocolate on purpose than the other, while otherwise keeping all the variables the same.  Some of the chocolate eaters would probably choose M&M's, at least some of the time, and would also eat the red ones.  Some might eat only organic dark chocolate.  The M&M eaters would get cancer at a higher rate, but if the study only compares chocolate eaters to non-chocolate eaters then the rate of liver cancer in the chocolate-eaters would be higher - but ONLY becasue the red M&M eaters would get more liver cancer.  By not specifying the type or source of chocolate you wouldn't be able to tell that from reading the study.

You can imagine what would happen next - this guy would be on the cover of Time magazine telling everybody to stop eating chocolate, the chocolate industry would go into a nosedive, health conscious people would stop eating chocolate.  People might get sicker on average - because they'd be missing out on the health benefits of good chocolate (dark chocolate) in a mistaken attempt to avoid liver cancer, not knowing that they should just be avoiding red M&M's.  It might be decades before doctors were willing to look at new, more sophisticated studies and realize that the problem isn't chocolate in general and recommend that people eat chocolate again, just not the red M&M type of chocolate.

What in real life has run a course like this?  Take fat for a great example.  Good science shows that saturated fat is bad for your heart - if by saturated fat you mean saturated and trans fats grouped together.  Why?  Because some of the people who eat more [saturated and trans fat together] are going to eat a lot of trans fats - and they are bad for you in various ways.  So if you study them together you might blame both saturated and trans fats for causing heart disease when in reality all of the damage is coming from just the trans fats.  This has really happened, in this country, over the past 40 or so years.

Another example?  Carbs.  You can look at high carb diets and show that they lead to insulin resistance.  Then you find groups that eat lots of carbs and don't get insulin resistance!  Why not?  Because it's not carbs, it's fructose.  Carb eaters in general get more fructose than non-carb eaters - but you can eat carbs from non-fructose sources (think starch, like sweet potatoes) and not develop insulin resistance (I'm slightly oversimplfying this, but the truth is close).  Carbs get a bad rap because some carbs cause health problems.  Then health conscious people avoid all carbs, when some might be better off getting some starch every day as long as they get it from non-fructose containing non-grain sources.

Want more examples?  Branched chain amino acids.  It was discovered something like 20 years ago that some proteins cause a boost in muscle protein synthesis.  So people ate tons of those proteins.  Then it was determined that it's not the protein as a whole that's signalling the protein synthesis, it was the branched chain amino acids (BCAA's) that were doing it.  Then people could take the BCAA's alone and possibly skip the whole proteins.  Recently it was discovered that it wasn't the BCAA's - it was the leucine, which is one of the BCAA's, that was doing all the signalling work.  Now those whole proteins and BCAA powders did contain lots of leucine, so the science on them wasn't wrong, it just wasn't specific enough.  The dangers of these misunderstandings aren't significant, this didn't cause widespread health damage, it's just another example.

Want more?  The study I linked to at the top of the page is a good one, and we've seen various versions of this recently.  Processed red meat is like the red M&M of the meat world.  If you measure total meat consumption against health risks for different stuff you're going to see a correlation - because some (or many) of the people eating a lot of meat are getting it processed.  The harm done by the processed meat will make the meat-eating group on average sicker than those who eat no meat - even though the individuals who happen to eat just fresh meat are just fine!  This study actually took sensitive enough data to recognize that, although the idiots writing the conclusions and the bigger idiots writing the news article about the study didn't care enough to include that.

Want more?  Omega-3 fats.  DHA and EPA are omega-3 fatty acids that are super important to your health - you need a good amount of them, and a good ratio of those fats to omega-6's, to keep systemic inflammation under control.  So lots of people go out and eat lots of omega-3 fats.  But not all omega-3 fats are DHA and EPA or are even turned into DHA and EPA in your body in significant amounts.  So people who hear that omega-3's are good for them, then go out and guzzle flax seed oil (which contains omega-3 fats, just not the kind that humans can easily turn into useful forms), they aren't doing themselves any good at all and might be doing lots of harm.  All because they weren't specific enough.

I'm sure there are other cases like this, I just can't think of any right now.  Post to comments if you have others.

Take home message: science is hard and specificity is important.  Pay close attention to the fine print in these studies, especially before you go around changing your eating habits for the worse. 

(Please note that the red M&M thing was just a way to explain my point - I have no reason to think that M&M's of any color are especially harmful.)