Disconfirming Evidence
by Joseph Rowlands
Years ago in school I learned about an important method in science. Since then, I've mentioned it to a few people and they told me that they were never actually taught it in school. The lesson went something like this:
A guy is given a set of numbers. The numbers given are 4, 6, 8, 12, and 14. He is told that there is a larger set of numbers, and these are mere examples. It is his job to formulate a theory for what numbers are in this larger set and verify his theory. He can do this by simply asking whether a specific number is also in the set. He can ask as many times as he wants, but this is the only kind of question he can ask.
The guy examines the set of number. He notices that all of these number are even. He formulates a theory that the set includes even numbers. There were some not mentioned, though. So he asks whether 2 is in the set. Yes it is. Okay, how about 10? Yes, it is. Okay, if it is even numbers, how many even numbers is it? Is 16 in the set? Yes. 18? Yes. How about 102? Yes. 500,214? Yes. A billion and two? Yes. Well, he thinks he's done a good job of confirming his theory. So he provides his answer. It is the set of even numbers! And he fails the test.
And here is where the lesson begins. When offered the chance to ask whether a number was in the set, he only looked for examples that fit his theory. He could have asked a million more questions about even lines, and he would still not have come any closer to the truth. A million data points wouldn't have helped because he only looked for data points that confirmed his initial theory.
If he was really interested in testing the theory, he could have asked whether the number 3 was in the set. If the answer was yes, it would have invalidated his initial theory entirely and he would have to come up with a new one. Maybe it is all of the natural numbers. How about 1, 5, and 7? Yes, yes, yes. But again, if he keeps asking about odd numbers, he is likely not to learn anything. So maybe he asks about negative numbers. Or fractions. Or irrational numbers. Each time he creates a new theory that explains the data, he looks for an example that would invalidate the theory.
This number example is just an instance of a wider principle. For a theory to be tested well, you have to look for disconfirming evidence. You can't just look for evidence that supports your initial impressions. You have to figure out what wouldn't meet those expectations and look for the evidence that would disconfirm the theory.
This is all tied to Karl Popper's view that real science must be falsifiable. He formulated this idea when he notices some popular "scientific" theories of his day were compatible with any possible outcome. The supporters of these theories thought that this was a good thing. They could explain any situations with their theory, so their theory must be very potent and complete. Popper declared that instead of being a strength, an inability to falsify a theory was a weakness.
There's a serious methodological flaw in creating a theory that is sufficiently vague so that it can be seen as consistent with any outcome. The theory lacks any kind of predictive ability. It may be seen as explaining the results, but it can only do so after the fact.
This means a theory cannot be considered good just because it explains a variety of facts. If you focus only on whether a theory covers the current set of facts, you haven't actually tested the theory at all. You can always create a theory that "explains" the facts. But until you start looking for disconfirming evidence, the theory has no real support.
While all of this is true in science, in may ways its far more important in the field of philosophy. A incorrect scientific theory will likely lead to unexpected consequences. Even if you aren't searching for disconfirming evidence, you may be confronted by it anyway.
Philosophy is not so easily tested. It deals with a framework for understanding the world. Errors in thinking can lead to contradictions, but those contradictions could remain hidden indefinitely if you don't have the habit of looking for them.
I have seen many examples of people proposing an idea or explanation and putting all of their effort into making sure that it successfully deals with the issues being discussed. Surprisingly, the proposals are easily shown to be faulty with just a little analysis, but there just isn't any focus on anything that might disconfirm the idea.
Choosing examples of this is difficult because the problem is so common. A person would excuse religious belief by saying that everyone just believes what they were taught growing up, while ignoring the fact that he doesn't believe what his parents taught him. This particular variant is widely used. Many people make claims about "people" or "humanity", but mysteriously exclude themselves. This is an easy example of disconfirming evidence. If you say something applies to everyone, ask if it applies to you. People aren't objective? They can't be logical? Their beliefs are simply a product of their personal desires? Are you speaking for yourself?
Other examples abound. One person claimed that the freest country is the one with the fewest laws. In his mind China is freer than the United States because they don't have seatbelt laws. Without trying to decide which is actually freer, the theory itself ignores the possibility that some laws have a larger impact than others. A myriad of little laws that govern free speech or a free press would have to be compared to a country that permits neither. A pointless argument and absurd conclusions could be avoided with the smallest effort.
There are many examples of someone stating that something can never happen. They are drawn to those kinds of arguments out of the belief that it strengthens their position in a related topic. But it often takes only a moment to think of a counterexample.
Good philosophy requires an effort to look for flaws in your own ideas. Instead of looking for examples that prove your point, you should take some time to look for examples that might disprove it. Often, the effort is minimal and the result is a much clearer understanding of the topic.
Discuss this Article (12 messages)
|
