Making better comparisons

We are always making comparisons. Whether we are deciding what job would be better for us, what political party we want to vote for, what is the best research about a given topic or just where we want to go for dinner. It is a fundamental aspect of daily life and doing it thoroughly is often overlooked. I am going to give an example of a post I read several years ago that I trusted without skepticism only to come back and read it years later to find that the ideas inferred by the comparison were not well reasoned.

One of my favourite authors, Tim Ferriss, has been recycling this post he made a few years ago and is a good example of using subjectivity and selection bias to confirm pre-existing ideas.

Screen Shot 2018-07-18 at 6.05.47 AM

It goes into the science of whether a calorie is a calorie and compares a high carb low fat (hClf) low calorie diet with a high fat low carb (hFlc) low calorie diet. This study comparison is used to taut the benefits of a hFlc diet and in some people’s eyes concludes that it is more effective for human health. The problem is that most people either take the author’s word for it and trust the post or just read the absctract of the study and not the entirety.

When we read both studies and how they were performed, we can get a bird’s eye view of the situation and see whether what the researchers or post author was trying to infer is true.

I suggest people read the studies themselves and if they have the interest then read a book called Bad Science by Ben Goldacre to learn how to read studies. For those who are not interested I will break it down for you but you’ll have to trust this post’s author.

The first study was done in 1944 with 100 conscientious objectors, all male. They fed them 3200 calories a day for 3 months. This part was an attempt to get everyone to an ideal or healthy bodyweight before restriction. For the next 6 months they were severely restricted to a 1560 calorie per day diet which was high carb (macros: 100g protein, 30g fat, 255g carbohydrate). All food was controlled and given to them by the researchers and all participants lost weight to the point of ribs and chest bones showing and developed severe mental and physical illness.

The high fat comparison study (this study is now locked behind a paywall) done in 1960 with 11 subjects of mixed sex was said to do a similar thing. While using the same calories per day but changing the macros so they were eating hFlc with 83g protein, 105g fat and 67g carbohydrate. The subjects ate their normal diets for 2 weeks and were then taught how to eat hFlc and were evaluated on their mental health and body composition for 2 weeks after instruction. The study yielded a different result with subjects improving body composition and claiming they had increased feelings of wellbeing and reduced negative emotions.

Are these studies analogous?

Screen Shot 2018-07-22 at 2.38.39 PM

As you can see we are comparing a very long and well controlled study with a very short and lacking control study. We are also comparing all male to male and female. This poses another issue because 1500 calories per day for most men would be considered a large restriction whereas for most women this wouldn’t be considered restrictive at all.

Should we even take into consideration the results of the hFlc study? Perhaps, but with a grain of salt.

Human error is also rife when it comes to tracking our own calories as people tend to underestimate how much they are actually eating. They might think they are eating 2000cal per day but when their food is actually tracked it turns out they are eating 2500cal per day. Unfortunately they didn’t have MyFitnessPal in the 1960s.

When I first read this post I concluded that it must be correct because I trusted Tim and because I considered him an expert he therefor must be correct. I will say that Tim does say in his post that he is not exactly comparing apples to apples but the second study does confirm his pre-existing beliefs.

For some things we do have to take the expert’s word for it because it may take years of research in order to understand one particular concept and this may be too skeptical of a position. I apply this to things like neuroscience, biology, physics, computer science and much more. But the ability to make good comparisons and read studies is the cure for a lot of bad ideas.

Next time someone provides a study to support their ideas, are you going to take their word for it or look into a bit more?

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: