Government Nutrition Guidelines vs “One Study Shows”

You never really see bloggers link to government health or nutrition guidelines, or talk about them at all. Why is that?

I can’t say I have the answer to that; perhaps they’re too dull? Perhaps it’s just “not cool”, or perhaps it’s seen as “lazy”?

But I don’t think it should be; I think bloggers, and health bloggers in particular, should talk about government nutrition guidelines, cause they’re kind of important. There’s a good reason we have them.

[This post is part of a series on “How to become a BS detector”; you can view the introduction here]

Try downloading the entire Scientific Advisory Committee on Nutrition (SACN) document on sugar, for example, I dare you. Assuming your internet speed is pretty decent, it’ll open in a few seconds time, and you’ll realise just how insanely long it is. If you haven’t bothered, it’s 384 pages. It includes every single study that was analysed, as well as a lengthy introduction and discussion. Imagine how time-consuming that would be to read. Now imagine how time-consuming that must have been to research and write.

Often in the news we see “one study found…” some interesting result which is intriguing and easy to absorb. Occasionally it’ll be something weird and wonderful like “chocolate helps you lose weight”, and sometimes it’ll go against the scientific consensus and you’ll see people shouting “scientists got it wrong all along!”

Well, no. Not quite. Not even close actually.

A single study, while interesting, tells us sweet F all in the grand scheme of things. Why?

Imagine rolling a dice 10 times. You could roll a 6 every time, and a logical conclusion would be that the dice has a 6 on each side. But is that true? Now imagine you did that experiment a hundred times, getting a slightly different result each time. Your conclusion after that would most likely be something along the lines of the dice being numbered 1 to 6, and each face being equally likely to appear. Would you have realised this from just one try? Maybe, but most likely not.

Similarly, if you study a phenomenon or an effect once, the result you get might not be accurate for the whole population, it could be an outlier, or a complete exception. But you can’t be sure unless you do it again and again and again.

As an example, let’s say we want to know if substance X increases or decreases the risk of cancer. We look at 200 studies (ok, 4) and create this lovely forest plot:

Each square is one study, and the confidence intervals (the area where we can be 95% sure the true result lies) are shown by the horizontal lines. The bigger the square the larger the relative sample size in the study and the more weight it carries. The overall result is shown by the diamond symbol. The vertical line at 1.0 is the point where there is no difference in risk, so at this point substance X neither increases nor decreases the risk of cancer. Results on the left of the vertical line indicate a reduction in risk (OR < 1.0), and results to the right of the line indicate an increase in risk (OR > 1). So based on this example, overall substance X increases the risk of cancer.

This is why guidelines are so powerful; they take all the available studies, giving more weight to studies with more participants and with more robust methodology, and find the overall picture. A picture you couldn’t possibly hope to see from just one study.

In addition, these nutrition guidelines are updated every couple of years to take new research into account, which may or may not affect the overall result.

In the UK the SACN advises Public Health England and other government organisations on matters such as nutritional status of people in the UK, dietary recommendations, nutritional issues affecting public health policy, and so on. You can find out who’s on board, how they’re funded, and even what they talk about each time they meet. They make sure it’s all transparent, and any reports they produce are available for anyone to download.

Single studies are interesting and exciting, but we can’t rely on them to make big decisions like whether something is carcinogenic (cancer causing), or how much of a vitamin we need daily, or what the recommended free sugar intake should be.

So next time you see a headline “study shows…” take it with a pinch of salt, and don’t get too excited.