Some of you will recall the story a few months ago that cell phones had been linked to a type of brain cancer. It made the rounds for several days, was a popular topic of news blogs and discussion forums. But why hasn't there been any followup? Well, actually there has been, but only in research circles, and it's not too surprisingly reversed the previous discovery.
It appears that the prior evidence was overstated. Cell phones may yet be demonstrated to increase the risk of certain cancers, but the key point is that no research has reliably demonstrated this.
This is so typical and expected, at least expected by me, that it can serve as a useful lesson in skepticism of media reporting of science stories. It can also serve as an example of the difference between evidence based medicine and science based medicine. When the initial story hit, the coverage was very focused on authority figures and agency statements, which is typical of this kind of reporting. For more discussion of this phenomenon, I refer you to my video version of Ben Goldacre's essay on Bad Science journalism.
The World Health Organization put cell phones on the list of possible carcinogens, alongside such things as dry cleaning and artificial sweeteners. The level of evidence was ranked very low, which to a scientist puts it on a sort of global "to-do" list of promising research leads, but it was hardly a definitive identification of risk. That takes verification and a different type of analysis.
Let's go over the types of studies that scientists use to establish correlations. My goal is that the next time you're presented with these kinds of shocking discoveries, you'll be able to determine what kind of evidence it's based on.
The first two are a type of epidemiology, retrospective and prospective studies:
1. The retrospective cohort study In vastly simplified terms, grab 1000 people with disease A and compare them to 1000 people with no disease A. Identify factors that may be correlated with a higher risk for disease. The biggest problem with a retrospective study is what is called self-selection bias. Your experimental, "sick" group are already diagnosed with a disease, and because they already have the disease, whatever is the risk factor has already happened. It's very easy to misidentify confounding factors.
If you look at the 1000 people in the cancer group and find that 721 of them are heavy cell phone users, while only 614 of the non-cancer group are, how strong is that evidence, considering all the other possible reasons for the differences?
2. The prospective cohort study Take 1000 people, and follow them for a defined period of time. Document disease outcomes (did the people get brain cancers?), try to identify which risk factors were most predictive of the outcome. This is the retrospective study, but done in advance of the condition being studied. Did the highest quartile of meat-eaters get more gastric cancer, did the lowest exercisers develop more diabetes?
The populations are often heterogenous (or complex), and what looks like a simple association can often be a linked series of categories, like finding people who use cell phones to be at higher risk of neural-derived cancer, when the actual association may be with something about their income or profession or other environmental factor or genetic factor. It could be a multifactorial linkage where stress and urbanization plus better medical care is responsible for the increase, and cell phone use is really just a proxy.
3. Case-control trials In a case-control RCT, or randomized control trial, the researcher gets to control a lot more about the comparisons being made. You can take humans or non-human animals and mix them up until the populations are evenly randomized, then expose half the group to the experimental factor... that eliminates the self-selection and the confounding bias, because the researchers are intervening artificially.
This type of study is not always an option for ethical reasons. We choose not to expose humans to excessive risk if we can help it. This is a big improvement over purely epidemiological studies, but it also has some weaknesses. We still have the multifactorial problem, where multiple behaviors can be linked, and it's very difficult to untangle all the variables of genetics, environment, nutrition, case history, and sheer luck. There could still be a crazy mixture of lots of risks. Which is why we need...
4. Mechanistic research This is the kind of work that I've mostly done. The scientist, guided by less precise observational or epidemiological data, and complementing the experimentally rigorous case-control data, look for ways that the experimental factor could cause the correlated effect. This is usually done in some format that allows for almost complete control. Identical, clonal cells grown in identical flasks, transgenic mice, or sibling animals. Even within this category there are shades of resolution. Molecular biology is usually pretty good at sensitive and reproducible results.
Very small margins of error, and results that can be reproduced by different labs or different techniques.
We focus on something called method concordance, which is where I take my username. Arriving at the same conclusion by multiple independent lines of inquiry... In the case of cell phones, we'd need to identify a plausible and testable way that the radiation or heat or emotional cues associated with cell phones can interefere with normal cellular development. We'd attempt to replicate that finding in the simplified model of genetically identical animals or cell culture.
If the mechanistic outcomes contradict the prospective study, then the smart money is on the more rigorous lab research, less on the epidemiology. Notice how we've gone from the very complex and poorly controlled to the very tightly controlled and rigorous design. That doesn't mean the mechanistic research is better than a prospective study. After all, it would take a very long time to test all the possible hypotheses, and prospective studies provide great clues about what's worth further investigation.
The problem arises when epidemiological data is presented as confirmed fact, rather than the first step in a long process. I also often see a sort of reverse: mechanistic data on some new drug that shows that it can cure cancer or other disease in a flask of cells, or some poor mouse with a gutful of injected cancer cells, is touted as a major breakthrough in cancer research before it goes out to a control trial and then on to prospective studies.
None of these studies is without merit, and they add up to a total picture of the real underlying causality.
How do we differentiate between Evidence based medicine and Science Based Medicine?
Evidence based medicine is any medical practice which is based on a preponderance of evidence for its efficacy and safety. It's a pretty broad umbrella, though, and it includes things that we don't understand well, or possibly applications of practices that seem to work, but shouldn't. There's nothing inherently wrong with EBM, but it's not on the same footing as science based medicine.
Science based medicine is a subclass of EBM. It includes only those practices that we can understand to a finer degree. We know that they're effective, but we also understand WHY they're effective. The mechanisms are clear enough to us that we can understand how best to use them in the clinic, when they might not be effective, and what improvements can be made to improve efficacy and safety.
Key to distinguishing the two is that we have more than just evidence that something causes an improvement, we can evaluate that epidemiological data in light of a plausible mechanism. For example, laetrile, a popular alternative cancer treatment derived from the cyanide-like compounds found in peach pits, is well characterized in terms of chemical structure and bioactivity.
It possesses no properties that would make it a good cancer drug. We can predict that with some confidence. However, it has a long and storied history with quack medicine of the mid-1920's, and so when evaluated in randomized control trials, sometimes it show evidence of effectiveness, and sometimes it doesn't. It depends on the power of the test and who's running it.
Laetrile might be considered evidence based medicine, albeit very, very weak evidence. It is not, however science based medicine, because there is absolutely nothing about how it acts in the body that would make it effective. Cases like laetrile, chelation therapy for heart disease or autism, or vitamin supplements are sometimes indicated by evidence, but not by a deep evidence of mechanisms.
So, returning to the example of cell phones, a big challenge is the lack of mechanism. The amount of radiation given off by modern cell phones is too small to reliably damage DNA or cross-link proteins. There's never been a good, well controlled study on cells that was able to replicate the effect suggested by prospective or retrospective epidemiological studies.
This makes scientists a little hesitant to jump behind a single study of a type that is notorious for false associations and lacks rigorous experimental control. Now, there is no process to show that something does NOT cause cancer, that's simply an untestable hypothesis in practical terms. However, how does a proper skeptic view the outcomes of these contradictory studies? Simple. With an open mind, and a rational estimate of risks.
We don't have to cast our vote for guilty or not guilty before the trial is finished. Like good jurors, we can wait until all the evidence is in. That's what rational skepticism is all about: withholding judgment until sufficient evidence is presented to accept or reject a given hypothesis. We understand this when we're in the jury box, but forget it the moment we step out of the courtroom.
It's more than a coincidence that so many scientists are skeptics and rationalists. The process becomes a way of life, a way of thinking, and a way of viewing the world. Turning those gears off on the weekends is just something some of us can't do. We care about what's true, and we know a reliable process for determining this. Credited:
C0nc0rdance
