So I'm working on a scientific paper that is trying to report the meagre results I got on a project that was canceled halfway through. While doing so I came across the following articles by Norvig, which hopefully will be useful in attacking my own assumptions and make the paper stronger:

Warning Signs in Experimental Design and Interpretation:

When an experimental study states "The group with treatment X had significantly less disease (p = 1%)", many people interpret this statement as being equivalent to "there is a 99% chance that treatment X prevents disease." This essay explains why these statements are not equivalent. For such an experiment, all of the following are possible:

• X is in fact an effective treatment as claimed.
• X is only effective for some people, in some conditions, in a way that the experiment failed to test.
• X is ineffective, and only looked effective due to random chance.
• X is ineffective because of a systematic flaw in the experiment.
• X is ineffective and the experimenters and/or reader misinterpreted the results to say that it is.

There is no way to know for sure which possibility holds, but there are warning signs that can dilute the credibility of an experiment.

The companion paper:

Evaluating Extraordinary Claims: Mind Over Matter? Or Mind Over Mind?
A relative of mine recently went in for minor surgery and sent out an email that asked for supportive thoughts during the operation and thoughtfully noted that since the operation was early in the morning when I might be sleeping, that

It doesn't matter, according to Larry Dossey, M.D. in Healing Words, whether you remember to do it at the appropriate time or do it early or later. He says the action of mentally projected thought or prayer is "non-local," i.e. not dependent on distance or time, citing some 30+ experiments on human and non-human targets (including yeast and even atoms), in which recorded results showed changes from average or random to beyond-average or patterned even when the designated thought group acted after the experiment was over.

I was perplexed. On the one hand, if there really was good evidence of mind-over-matter (and operating backwards in time, no less) you'd think it would be the kind of thing that would make the news, and I would have heard about it. On the other hand, if there is no such evidence, why would seemingly sensible people like Larry Dossey, M.D. believe there was? I had a vague idea that there were some studies showing an effect of prayer and some showing no effect; I thought it would be interesting to research the field. I was concurrently working on an essay on experiment design, and this could serve as a good set of examples.

The only thing that I quibble with is the term "extraordinary" in the title of the second article. In my experience, "extraordinary" is a word people use to signal that something has challenged one of their beliefs and they're going to run it over the coals, which Norvig does with the efficacy of intercessory prayer in his article (in a very balanced and fair way I think). However, part of the point of Norvig's very evenhanded essay is that these kinds of problems can happen to you on things that you do believe:

After reading Tavris and Aronson's book Mistakes Were Made (but not by me), I understand how. Dossey has staked out a position in support of efficacious prayer and mind-over-matter, and has invested a lot of his time and energy in that position. He has gotten to the point where any challenge to his position would bring cognitive dissonance: if his position is wrong, then he is not a smart and wise person; he believes he is smart and wise; therefore his position must be correct and any evidence against it must be ignored. This pattern of self-justification (and self-deception), Tavris and Aronson point out, is common in politics and policy (as well as private life), and it looks like Dossey has a bad case. Ironically, Dossey is able to recognize this condition in other people -- he has a powerful essay that criticizes George W. Bush for saying "We do not torture" when confronted with overwhelming evidence that in fact Bush's policy is to torture. I applaud this essay, and I agree that Bush has slipped into self-deception to justify himself and ward off cognitive dissonance. Just like Dossey. Dossey may have a keen mind, but his mind has turned against itself, not allowing him to see what he doesn't want to see. This is a case of mind over mind, not mind over matter.

So, at least as working scientists are concerned, I would suggest Norvig's second essay should be retitled "Evaluating Claims."

Or put another way, with all due respect to Carl Sagan, I think "extraordinary claims require extraordinary evidence" is a terrible way to think for a scientist: it prompts you to go around challenging all the things you disagree with. In contrast, I think claims require evidence, and for a scientist you must start at home with the things you're most convinced of, because you're least likely to see your own claims as extraordinary.

This is the most true, of course, for papers you're trying to get published. Time to review my results and conclusions sections...

-Anthony

Labels: Intelligence, Thank You Carl Sagan