Evidence can be a tricky thing.
When someone conducts an experiment, they get a specific bit of data which represents the result of that experiment performed at that time in exactly that way. These pieces of scientific evidence then have to be assembled together with the existing theoretical framework
However, sometimes a single piece of evidence doesn't fit well in the framework. This actually happens fairly frequently. The scientific framework is constantly being amended in ways large and small to make it more comprehensive. The way that science adapts to this sort of evidence is what makes it unique--and superior--to other ways of thinking about facts. Science is designed to adapt to fit the facts as they are gained.
One of my favorite explanations of science is defined entirely in terms of this evidence-based approach, in the words of physicist Richard Feynman:
Science is a way to teach how something gets to be known, what is not known, to what extent things are known (for nothing is known absolutely), how to handle doubt and uncertainty, what the rules of evidence are, how to think about things so that judgments can be made, how to distinguish truth from fraud, and from show.
Neutrinos: A Case Study in Good Evidence
The most recent popular example of evidence that failed to match the existing theoretical framework was the suggestion of faster-than-light neutrinos, which is more and more looking like it was the result of equipment failure. In my recent blog post on the subject (Another Lightspeed Limit Update), I suggested that despite being wrong, the OPERA scientists did things the way they were supposed to. In the words of CERN Research Director Sergio Bertolucci:
Whatever the result, the OPERA experiment has behaved with perfect scientific integrity in opening their measurement to broad scrutiny, and inviting independent measurements. This is how science works.
The scientists at the OPERA experiment were very tentative about their results. They checked and re-checked their assumptions and equipment. They performed a more refined version of the experiment, which seemed to confirm their initial evidence. Then they continued to check their evidence, getting input from the broader body of scientists. When they discovered a possible disparity, they announced it publicly.
Now another group, using their equipment to check the earlier OPERA results, is finding no unusual results. It's very likely that this piece of data is in error, probably due to equipment problems.
Standalone Evidence Isn't Proof
Still, even if we'd never identified the source of the error, eventually the scientific method would have given us a verdict. Either we would get more results that showed neutrinos traveling faster than the speed of light or we would not have. If we had failed to get confirming evidence, we'd be in a curious position:
We'd have this one bizarre set of experimental results that didn't match anything else performed before or after.
What is unfortunate is that a lot of people would put substantial weight on this one set of experimental results. Within days of the experiment, I was receiving e-mails from independent physicists (and physics cranks) explaining how their theory accounts for the anomaly. Mind you, this was before we had any confirmation that the anomaly was even real.
Even if we never again see evidence of a neutrino moving faster than the speed of light, there will be people who point at this one experiment for decades as evidence of their pseudoscience claims. Why is this faulty? For one simple reason:
A single piece of data does not establish a law of nature.
Praying Away Tornadoes
It's easy to be misled by a powerful single example of something masquerading as valid evidence of something.
This occurred to me recently when, in the wake of the tornadoes that swept the Midwest, my wife was told by an acquaintance that there was video evidence that prayers to Jesus could stop tornadoes. "Just look on YouTube," she said.
Being always willing to consider evidence, I did look on YouTube. Searching for "pray" and "tornado" led me to one video, which gained popularity because it was also on CNN. (Here's a link to the video on YouTube, as well.) It shows a very intimidating tornado coming toward a woman's home in West Liberty, KY. She can be heard praying passionately in the background for Jesus to turn away the tornado, and you can see someone's hand in the air, as if it's raised to ward off the tornado.
Now, I want to be clear about something (even though I know it's futile, and I'm going to get hate mail regardless):
I am not even going to attempt to convince anyone that prayers don't affect tornadoes.
If you believe that they do, it's a matter of faith, and it's beyond the scope of this blog post to try to change that opinion. (I leave that to my colleague over at About.com Atheism ... an excellent blog, by the way.)
What I do hope to convince you, however, is the following:
This video does not prove that praying affects tornadoes.
In fact, not only does the video not constitute proof ... it doesn't even constitute evidence that praying affects tornadoes.
"But how can that be?" you may ask. The video shows a tornado which appears to be coming toward the woman, she prays, and the tornado moves away. Surely, this is solid evidence that the prayer affects the tornado.
It's not, for a variety of reasons. For the purpose of this analysis, I'll assume that the video hasn't been doctored and that everything demonstrated is exactly as it happened. Remember, the woman who directed me to this video made a very specific claim (or hypothesis, in science jargon):
Hypothesis: God will intervene to protect a believer who prays.
This woman isn't alone. Pat Robertson, of the 700 Club, recently made a very similar claim. While Pat Robertson hasn't, to my knowledge, specifically cited this video, I feel confident that he'd claim it proved his case.
The problem with the video is that there's an inherent selection bias involved in it. The video came to prominence precisely because the tornado doesn't strike the woman who is praying. The only videos like this that will ever make it on television--or even into high YouTube search rankings--are cases where a person is praying and the tornado spares them. If a tornado is coming toward a person, they pray, and the tornado hits them, the video isn't particularly interesting. It's just sad.
But this video gets circulated precisely because it feels like there's a connection between the prayer and the tornado. Two things happen at a similar time and are linked together meaningfully in our minds, in a case of what psychologist Carl Jung referred to as synchronicity. To constitute actual physical evidence about the workings of the world, we need some clear constraints on how the data is obtained, to remove the emotional aspect from our analysis as much as possible.
Here are some examples of what we'd need before we could even call this evidence:
- An analysis of the ways that tornadoes move which includes some sort of mechanism for defining what is usual or unusual behavior for a tornadoes change in severity and direction. (Because, frankly, I watched the video the first time with the sound off and didn't even think it looked like the tornado did anything unusual.)
- A controlled experiment would involve recording both non-believers and believers (ideally of various faiths) praying to turn tornadoes away.
- Another set of videos would include recordings of tornadoes in remote areas, ideally where there is no chance of a believer being nearby to pray.
- As part of a double-blind experiment, a group of weather experts would be asked to watch the videos without sound and rate the behavior of the tornadoes based upon the analysis rating mechanism defined in Step 1.
- Statistical analysis of these ratings would determine whether there is any difference between the prayed-for tornadoes and the tornadoes in remote areas.
- Multiple analyses could compare the success rates of each individual faith category.
If there is a statistically-significant difference between the cases where believers pray for tornadoes and the cases where no one is around, then that would be evidence that prayer affects tornado movement.
This would still not constitute proof, mind you (because, as Feynman says, nothing is known for certain), but it would be evidence that you could validly cite to support your prayer-tornado link hypothesis.
Short of that, or a similar study, you've got no evidence ... you've only got a video of a lucky woman praying, without any evidence of why she was lucky and other people were unlucky. You can still believe that prayer has an impact, but you believe that without evidence. (That's why it's called faith, after all.) Trying to claim that it's evidence or proof is misleading.
Evidence and Conclusions
And, of course, even if you had evidence then sometimes, as in the case of the OPERA experiment, the evidence can lead you to a false conclusion. That's one of many reasons why science is so great, because it has the means to sort out the false conclusions from the true ones. It requires discipline, but the methodology will always win out in the end.
Do you have an example of when a single piece of evidence led you to a faulty conclusion? How did you ultimately discover that it was faulty? Share it with us in the comments!