Science 101: Weight-of-Evidence and Weight-of-Warning
December 22, 2010. What does “weight-of-evidence” mean and how is it calculated?
From Terry’s Thoughts on “weight-of-evidence” and “weight-of-warning”
Health Canada’s “weight-of-evidence” approach means if there is one more study showing no harmful effect of microwave radiation, even if that study is industry-funded, Health Canada can ignore all other science no matter how compelling. It is a numbers game that errs on the side of profit.
However, if we follow the rules of this numbers game when it comes to warnings about wireless devices, we have only one agency – Health Canada-saying WiFi is safe since provincial and municipal health departments defer to Health Canada when it comes to electromagnetic radiation. But we have the American Society for Reproductive Medicine saying microwave causes DNA damage to sperm in only four hours. We have the Royal Society of Canada saying it weakens the blood-brain-barrier in only eight hours. We have the majority of the insurance industry refusing to underwrite manufacturers of wireless devices because of the “incalculable risk to human health”. We have the Canadian Standards Association stating it “does not guarantee WiFi is safe and without a health risk.” We have the WiFi Alliance-a lobby group for the industry-refusing to claim its products are safe, and finally we have the makers of the wireless devices who are silent when publicly asked if their devices are safe, and it we read the fine print of their manuals it says that, in fact, they are dangerous and we should limit our exposure.
So the “weight-of-warning” is on the side of not using WiFi in schools where children have no choice. The “weight-of-warning” is on the side of caution and informed consent. The weight-of-warning is a phrase we may chose to adopt.
Health Canada, with its track record of error causing human death, cannot be the only advisor when the “weight-of-warning” suggests caution and avoidance. The “weight-of-warning”, in this case, means that Health Canada must be ignored until it catches up with the majority.
Response to Terry’s Thoughts on “weight-of-evidence” and “weight-of-warning”
Terry is a parent who believes that WiFi was making his child, and possibly other children, ill at school. He has educated himself about the dangers of microwave radiation and thinks that the studies and methods that Health Canada uses to justify the safety of WiFi needs to be re-examined. I agree.
Health Canada has received many warnings and reports from scientists that show WiFi and other forms of microwave radiation generated by wireless devices are potentially dangerous. They have also heard from people from British Columbia to Newfoundland who become ill when exposed to microwave radiation. Some of these people have been medically diagnosed as having electrohypersensitivity (EHS), which is an adverse reaction to electromagnetic radiation and is a cousin to multiple chemical sensitivity (MCS). Both EHS and MCS are recognized by the Canadian Human Rights Commission as forms of environmental sensitivities (Sears 2007). Click here to download this document. The “weight-of-evidence” method that Terry talks about and the “weight-of-warning” needs to be re-examined . . .
What does “weight-of-evidence” mean and how is it calculated?
The wrong way to calculate “weight-of-evidence” is to collect as many peer-reviewed* scientific publications as possible and then identify which ones show harmful effects and which ones show no effects and place them on a scale. This is the way Health Canada calculates weight-of-evidence. They refer to “thousands” of studies showing no effect.
* [Note: a peer-reviewed publication is one where the editor of a journal sends the submitted paper to at least two experts in the field for their comments about the quality and value of the work submitted for publication. If both experts agree the work is well done and has value it is published and referred to as a peer-reviewed publication.]
The right way to calculate “weight-of-evidence” is to collect as many peer-reviewed scientific publications as possible and then identify which ones show harmful effects and which ones show beneficial effects and then place them on a scale.
The rationale behind this is that an agent (microwave radiation, a drug, or something else) can have three possible effects: harmful, beneficial, or no effect. We use statisical analysis to determine if a harmful or beneficial effect is real or due to chance. If it is due to chance we place it in the “no effects” pile. We use 95% (or higher) as the level of probability in scientific studies. What this means is that if this study was repeated 100 times, we would get the same result 95 times and it would happen by chance 5 times. So we have 95% confidence that this result is “real” and not due to chance.
But it could be due to chance 5% of the time so we have another way of examining the results. We compare as many peer-reviewed publications as we can find and do the correct weight-of-evidence calculation.
Here is how it works.
Let’s assume that out of 100 studies, 35 show a “harmful effect” and 5 show a “beneficial effect ” and the rest (60 studies) show no effect. Some would falsely claim that the weight-of-evidence (35 vs 6o) shows no effect so we have nothing to be concerned about.
The correct comparison is 35 studies “harmful” and 5 studies “beneficial”. So the weight of evidence is 35 studies vs 5 studies or a ratio of 7 to 1. This would add to our confidence that the “harmful” effects are real and not due to chance since you might expect as many studies showing harmful vs beneficial effects to be due to chance. So let us assume, for the time being, that all of the positive studies are due to chance (note this is unlikely) then you might expect that the same number of harmful studies are also due to chance. That still leaves 30 studies showing statistically significant harmful effects. How do you explain these studies? You just can’t sweep them under the rug and say well there are more studies that show no effect because a “no effect” study does NOT cancel one that shows “harmful effects”!
Here is an analogy.
Let’s assume you are a prospector looking for gold and you find a gold nugget in a particular area (equivalent to one study showing a harmful effect of radio frequency radiation). Does that mean this is a place for a gold mine? No, not necessarily. Someone may have dropped the gold nugget. But let us assume that of 100 prospectors, 60 don’t find anything, 35 find gold (35 studies showing a harmful effect) and 5 find fool’s gold (5 studies showing a beneficial effect). Let’s also assume that 5 of the gold miners who thought they found real gold actually found fool’s gold (same number as the original fool’s gold findings). That still leaves 30 miners who found real gold in a particular area. No one in their right mind would compare the 30 successful finds with the prospectors who didn’t find anything and say that the weight-of-evidence shows that there isn’t any gold in “dem dere hills”. Instead what it means is that there is gold in that location. The analogy is that there are harmful effects of the radio frequency radiation because so many studies are documenting them.
To make certain that the results are not biased, it is useful to examine who funded the studies. When this is done we find that studies funded by the telecommunication industry are more likely to show “no effect” than those independently funded. See results below for a study conducted by Dr. Henry Lai, University of Washington. For this very reason, authors are asked to disclose funding sources and any other connections that may be viewed as a “conflict of interest” when they submit papers for publication.
Dr. Henry Lai examined the results of cell phone studies documenting biological effects and found that results depended on who funded the study. Here are his results presented at Lakehead University in Thunder Bay Ontario on February 22, 2010.
This analysis shows that 28% of industry funded research and 67% of non-industry funded research documented biological effects. The weight-of-evidence for non-industry funded studies shows that cell phones radiation is “biologically active” i.e. it has biological effects. In this case the analysis (Chi Squared) documenting a difference in results based on funding source is statistically significant at the 99.9% level. So there is a 0.1% chance that this result is due to chance! According to this table there are 181 studies showing biological effects of cell phones! Should we just ignore these reports?
A similar study was published by Huss et al. in 2007, Source of funding and results of studies of health effects of mobile phone use: Systematic review of experimental results. Environmental Health Perspectives, Vol. 115, pages 1-4. (Click here to download this study).
Here is a direct quote from Huss et al. 2007:
“Studies funded exclusively by industry reported the largest number of outcomes, but were least likely to report a statistically significant result: The odds ratio was 0.11 (95% confidence interval, 0.02–0.78) [MH NOTE: this is statistically significant!], compared with studies funded by public agencies or charities. . . . CONCLUSIONS: The interpretation of results from studies of health effects of radiofrequency radiation should take sponsorship into account.”
This is a polite way of saying we may not be able to trust some industry-funded research.
There are multiple explanations for this outcome and they include some of the following:
(1) Industry funded studies showing an effect are suppressed while those not showing an effect are submitted for publication.
(2) Industry funded studies are designed to not find an effect as was done with the INTERPHONE study published earlier this year. See Lessons from INTERPHONE Study and also see recent comparison of the INTERPHONE study to the Hardell studies (Hardell et al. 2010).
There are less charitable interpretations that I won’t present here.
This type of industry-funded research plays into the hands of Health Canada and other agencies who do weight-of-evidence calculations incorrectly.
Weight-of-Warning
Terry’s concept of a “weight-of-warning” is an interesting one. But we need to bare one thing in mind and that is that science is not a popularity contest. New discoveries and novel understanding starts with a single scientist who, by definition, is in the minority. It takes great courage for that single scientist to share his/her views that may be in conflict with prevailing scientific or religious or financial or political views.
In the past, science faced opposition primarily from religion. For this reason, Copernicus did not allow his book to be published until after his death. Galileo was excommunicated and under house arrest for supporting Copernicus and stating that the earth revolved around the sun.
Today, the greatest opposition to environmental and health science comes from industry. Industry wants to make as much profit as possible and as quickly as possible. Many industries have become profit-junkies. They develop products that may not be properly tested or that pollute the environment or that are harmful to human health. They factor in risk and put money aside for legal fees dealing with future law suits.
Pinto is a good example of this. A design flaw, known to the company, caused the fuel tank to explode in a rear-end collision resulting in the loss of life. Ford weighted the cost of redesigning the flaw and decided it would be cheaper to pay off possible lawsuits resulting in death.
Many governments are power-junkies. They make the rules and, if they are provided with sufficient resources, they enforce the rules. Generally governments don’t like to change guidelines or standards because these changes have ripple effects that they don’t want to deal with including potential legal action. Also, these changes indicate they were wrong. They don’t seem to realize that when you learn something new a change of opinion may be appropriate. Mahatma Gandhi, when he was talking to a reporter, stated in his autobiography, “This is my opinion today. Tomorrow I may learn something and, if that happens, I may change my opinion.” This is exactly how science works. As we stand on the shoulders of giants, we get a better view of our surroundings and that allows us to make advances that may require new theories or new world views to emerge. Admitting you are wrong means you have learned something new.
Today, unpopular views may cost industry money or the government a loss of face so there is considerable opposition and hostility to new ideas and interpretations. Instead of scientists being excommunicated, they lose their research funding or their jobs. Attempts are made to discredit them by calling them names like “fear-mongerer” or “quack”. These are things kids do in elementary school and it is disturbing to see adults stooping to the same level of immaturity.
So, while a “weight-of-warning” may be useful as scientific understanding matures in a particular discipline at the early stages the “weight-of-warning” will favor the status quo. For this reason we have the Precautionary Approach/Principal that encourages action to minimize the potentially harmful effects of an agent, while scientists try to understand what is happening.
We have made so many mistakes with new agents introduced into the environment . . . PCBs, DDT, dioxin, lead, cadmium, asbestos, CFCs, X-rays, . . . Why are we so slow to learn?
Today, with the rapid adoption of wireless technology, we are putting the lives of future generations at risk for the convenience of wireless when wired technology is faster, more secure and safer. Schools have aligned themselves with profit-junkies. They won’t admit their mistake until students and teachers become sick, learning disabilities increase, and test scores decrease. And they can always hide behind Health Canada who states that this technology is safe based on their faulty analysis of the weight-of-evidence.