The Latest Round of Supplement Bashing

Response from Dr. Hickey et. al.

Excluding zero death studies clearly biases the results. However, this is a
minor point, considering the other problems with this paper. The paper
demonstrates a misuse of statistics and a lack of biological understanding.
 
The paper's foremost methodological error is the use of subjective selection.
To change a calculated relative risk ratio, you simply bias the selection
procedure. In removing zero death studies, the risk ratio would remain
unchanged only if the authors excluded studies involving supplement group or
control group deaths with appropriate and proportionate probability.
 
Consider the relative risk, calculated as the ratio of the risks, in each
group. Suppose the risk was 2 in 100 subjects in each group (2% risk in each
group, giving a 1:1 ratio). Now, remove 90 subjects from just the supplement
group: the calculated risk increases to 20% (i.e. 2 out of 10 subjects) and the
ratio of the risks becomes 10:1. There is no evidence that exclusions in the
meta-analysis study were subject to appropriate controls, to prevent this kind
of bias: the paper describes subjective selection of studies!
 
It is inappropriate to exclude preferentially studies where no risk outcome was
observed, because it affects the meaning of the risk ratio and its
interpretation. Specifically, removing studies relates the calculated risk
ratio to a smaller subset (m people) of a larger population (n people where
m<<n). Consider the inconsequential increase in risk from one death in
5,000,000,000 people to two deaths in 5,000,000,000 people (giving a relative
risk of 2). If we remove the majority of unaffected people, we get an
apparently important result: say a change from one death in 10 people to two
deaths in 10 people (relative risk still 2). Removing unaffected people makes
the risk appear more important.
 
The fact that people routinely misapply relative risks, by ignoring the
population to which they apply, is irrelevant here. However, as a general
heuristic, always mistrust relative measures, and do not try to defend a bad
paper using an argument based on relative risk.
 
Relative measures in studies should be avoided, unless the criteria are
carefully stated. Abuse of relative measures often makes a paper seem more
important, or drug effects more powerful. Generally, relative values magnify
effects in a way that is deceptive rather than illuminating. An example might
be a doubling, or 100% increase, in risk, which appears far less if it is
properly expressed as an increase from, say 0.0001 per million of the
population to 0.0002 per million. A risk ratio, considered alone, conveys
almost no information about the actual risk. It has meaning only when the real
magnitude of the risk is also provided. For further examples, try Darrel Huff's
book "How to lie with statistics," or Gert Gigerenzer's "Calculated Risks: How
to Know When Numbers Deceive You".  
 
This objection to our analysis has missed or avoided the main issue, which is
that the selection process in this study was biased. This renders the study
worthless, except as a measure of investigator bias.

Update 3

I also posted a question to the blog of Arthur Devany who is a statistician and Peleo Diet advocate. The question is in the comments of this entry. Devany would be well qualified to comment on the statistical methods used in the study. But no response as of yet.

Update 2

Notice near the beginning of the response from the Linus Pauling Institute, Dr. Frei is quoted as saying: "...the new study's focus on a selected group of clinical trials disregards the results of other more positive trials, as well as huge amounts of laboratory, animal, and human observational and experimental data." The LEF response likewise lists a large body of studies which seemingly contradict this latest meta study. I think this is another factor that should be considered. When a meta-study contradicts a large body of other studies, it should be regarded with a high dose of skepticism. Meta studies can be useful but I wonder if they should be considered ground breaking? My limited understanding suggests that they are only useful in the context of plenty of other more rigorous studies.

Update

Original Post

• Chowka's response...
• Knowledge of Health response...
• Response from the LEF...

At the bottom of Chowka's article are links to other responses. In particular the response by the Linus Pauling institute is very good.

[/health/vitamins]     32 comments     permalink

Contact Me:
bobber at kc0dxf dot net

Music Sites

Guitar Builders and Luthiers

Running

Frequented National Blogs

Internal

Columnists (political)

Columnists (technical)

Health and Nutrition

Big Shark Team Cyclists

Other Cyclists

Health Care Reform

Paleo (Hunter Gatherer) Diet (and related diets)

Christianity

Historic Christian Confessions

Classical Education

Routed Slot Car Tracks

Training With Power

Cycling

Ultrafit

Digital Cameras

Sporting Goods and Bike Shops

Technical

Transcendent Music

Utilities