About Non-Empirical 'Science'

I appreciate reading content from people whose views are unique, but whom I disagree with a healthy percentage of the time.  The unique part is important, because if you're saying the same thing I can get a hundred other places I know you're not putting much individual thought into the message.  This is perhaps part of why I've never signed on - even nominally - to any of the major political movements or parties.  If one of them had all the answers, it feels like they would be dominant by now.  Not because everyone 'saw the light', but because if an idea works in practice more people do it, until everyone is doing it.

I imagine there are reasons this wouldn't be the case.  For example, both major parties in the USA often claim that their reforms/policies 'would have worked if it hadn't been for those meddling people from the other party'.  And while that's an unfalsifiable claim that might well be correct, it doesn't really help convince me to sign onto the ideology.  If your proposed path to success requires pure adherence to your principles by everyone then it's not something that can be implemented in the real world.  Like an Ayn Rand novel, it's a purely hypothetical situation divorced from how real people operate.  And since it can't be falsified, it's impossible to challenge it with new evidence and thereby constantly improve it.

This is why I prefer to read from people whose ideological perspectives are mostly independent of traditional labels.  Even if I don't agree with them, they often follow lines of logic I might never have considered otherwise.  I have to turn my brain on in a way I don't when someone is regurgitating the party-line opinion in their own words.  The benefit of this is that my ideas might change in the opposite direction of the author's position, if only because I'm forced to seriously consider the idea.  Even when they don't change, I'm at least better able to articulate my viewpoint after engaging with the material than before.

When Empiricism Isn't Available

One of the websites that fits this description of 'don't always agree with, but regularly produces unique ideas' is SlateStarCodex.  The author, Scott, recently posted a pair of articles trying to understand what to do with hypotheses that don't have any empirical evidence to back them up.  (Or more particularly, where they can't be tested by obtaining new empirical evidence against their claims.)  The conventional scientific explanation of what you do when you can't figure out how to test your hypothesis is that you need to pause the scientific method until you can test it.  After all, a hypothesis that is untested can't be relied upon as a predictive theory, since we haven't confirmed whether it can actually predict new information yet.  Scott has an interesting habit of trying to break conventional wisdom by proposing exceptions, and in these two posts he tries to do that with hypothesis testing.

He poses a few interesting hypotheticals in each article while seeking to understand what to do with things that aren't fundamentally testable.  Now, I'll give him credit that he's doing something very much in the spirit of my Open Questions series when he says this:
The untestable is a fundamental part of science, impossible to remove. We can debate how to explain it. But denying it isn’t an option.
This is true in the sense that we often run into a problem where it's not obvious how to test a hypothesis.  We want to test it, because otherwise we're stalled in running through the steps of the scientific method, but we can't test it for one reason or another.  (Much of the debate on the SSC website revolves around the Many Worlds Interpretation of Quantum Mechanics, but there are plenty of other examples where we can't fundamentally test a hypothesis.  Thus, this isn't an isolated problem with MWI, but rather a broad philosophical question with far-reaching implications.)

But science - and the natural world it seeks to understand - doesn't care about what we want.  It's not graded on a curve based on how expensive an experiment would need to be to reach statistical significance, or how large a supercollider we'd have to build to test our hypothesis.  When I was in graduate school, I'd have discussions about this with my fellow students.  They had some data that was suggestive of a certain effect - it correlated with their preferred hypothesis - but they didn't have any good data pointing toward causation.  I'd point out that the data was inadequate to support the conclusion they wanted to make, to which I'd get the reply, "It would cost ten million dollars to run the experiment the way you're talking about.  Nobody would ever do it."  Which is true, but kind of irrelevant.  The argument was, "your data don't support your conclusion".  You can't convince a scientist with the reply "This is the best I could get under the circumstances, so it'll have to do."

So what do you do with a hypothesis when you can't run the experiment?  Do you throw up your hands and give up on understanding the subject altogether?  I don't think so.  We've run into this problem in the past, and we didn't reject every hypothesis that wasn't immediately testable.  On the other hand, we also didn't just accept the hypothesis as proven until we could find a good way to test it.  We treated it differently, but we didn't ignore it.  A good example of this is Einstein's theories of special and general relativity.  When they came out, they made a big impact on physics, but many physicists worried that it wasn't possible to test the implications of Einstein's ideas.  Later experiments confirmed his ideas (and indeed corrections for relativity are key to ensuring GPS satellites work such that today relativity contributes meaningfully daily for billions of people), but before they were tested his ideas were treated differently from how they were treated afterward.

This will make more sense if we define the difference between a hypothesis you can test (science), and a hypothesis that you can't test yet (philosophy).  Astute readers will note here that science is a branch of philosophy, which is true enough, but for the moment let's think of it as a kind of 'tool' of philosophy.  What is this tool, and how does it enable us to do different things in philosophy than if we didn't have it?

As I discussed in a previous post, science was predated by natural philosophy as established by Aristotle.  Natural philosophy was an exercise in making observations and telling a story based on those observations.  Philosophers could make additional observations and refine their story, and in some sense they were doing something akin to science.  But the process of just telling a story to describe evidence after the fact can lead you down strange paths.  For the Greeks it lead to things like geocentrism, four humors theory, belief that everything was made of four 'elements', and other absurdities that have since been laughably disproven.  The problem with natural philosophy is that you can tell a convincing story after the fact about any set of observed phenomena.  Many a creative Greek natural philosopher went on at length about how certain observations they made had to be right, because their philosophical reasoning dictated it must be so.

Today, Galen's writing about four humors might seem completely useless.  And in the context of understanding how the human body works, it is.  But there's a different way in which it's highly instructive.  Going back and reading the work of any proponent of an idea we now know to be completely wrong is a fun experience.  This is because they completely believe the arguments we modern readers consider absurd.  Indeed, the philosophy is sound, as far as they're concerned, so it's a bizarre experience to watch them explain how clearly their reasoning establishes their ideas as obviously true.

What makes the scientific method different that allows us to have more confidence in it?  Hypothesis testing.  Because it's so easy to tell a variety of just-so stories that explain all the historical data but are ultimately wrong, we need a way to reject bad stories.  We can't pretend that after looking at a limited set of data we we'll be able to tell some story of Ultimate Truth about the universe.  Instead, the genius of the scientific method is to ask whether a story is strong enough to predict future observations.

That's it.  That's the tool we use to deepen our philosophy to the point we can rely on it.  That's what drives discovery and accelerates the pace of change.  When a story is capable of predicting future observations we gain more confidence that it will continue to predict future outcomes.  When it doesn't, we're forced to modify our story until it is again able to predict future outcomes.

The idea is deceptively simple, but there's a lot of power behind it.  I've mentioned before that it's easy to tell a convincing but false story about historical data.  It's very easy to explain a phenomenon and fool yourself (and many intelligent people around you) that your story represents a deeper understanding of Universal Truth.  Again, go back and read Galen and Aristotle with a discerning eye.  You'll see why the people who followed after them for the next thousand years believed in their ideas.  They had theories that were simple to understand and just so philosophically beautiful they had to be true.

This kind of reading of outdated natural philosophy is difficult for a modern reader to do well, but it's worth the perspective it gives.  You might ask, "Why didn't they realize [X] can't be true?  They had the tools back in the day to test it, so why didn't they?"  But the practitioners of medicine and natural philosophy of the day didn't think that way because they lacked the falsification by experimentation toolkit.  They applied the tools of natural philosophy, but didn't have the tools of science.

(I'm not saying the ancients never performed this kind of falsification testing.  I'm saying there's a difference between a Scientific Method that requires you to reject/modify your hypothesis in the face of new evidence, versus a philosophical school that holds new evidence as only one of a number of factors to weigh together.)

This takes us back to the overarching question raised by Scott over at SSC: What do we do when there's no way to empirically test a scientific claim?

The answer is simple when we think of the scientific method as a tool in our toolkit.  If we can't use it, we must default to philosophy.  However, as with any other building project that requires the use of tools, if there's some tool or component missing that means there's part of the structure we can't build.  We might still be able to build other parts, but we can't complete the parts that require the missing tool.  If I don't have a furnace or ductwork I can build a fine house but it will lack central heating and air.

Similarly, since hypothesis testing is how we gain confidence in an explanation's ability to predict future observations, if we can't use that tool we can't have confidence in the power of our story to predict.  We have other tools we can use.  For example, we can test whether a hypothesis has internal logical inconsistencies.  That might allow us to reject an idea even without having to test it.  Indeed, even where we can test a hypothesis it's often useful to reject the worst ones prior to the testing phase by applying the useful tools of philosophy and logic.  That way we pare down our options so we're only testing the ideas that have a shot at being right.  This is especially important if testing a hypothesis is expensive or time-consuming.  But no matter how expensive or time-consuming - or even impossible - hypothesis testing is, if we skip this step we can't have confidence in our story's predictive power.

No matter how beautiful or complete it feels when we hear it, confidence comes only when we rigorously test an idea.  So we have to be cautious about even the most internally consistent hypotheses - even if they get called a 'theory' because they've been examined so much.  They still have a high probability of being wrong.  Maybe this isn't intuitive to you yet.  Let me finish with an analogy here.

The Bible has probably been the most widely-read book of the past few hundred years.  And yet there are at least a hundred different interpretations of the messages contained in its pages.  For any random chapter, you could probably assemble a room full of people who'll be willing to argue completely different interpretations of multiple passages, words, translations, etc.  I know many religious people who look at this situation and think the answer to arriving at the "correct interpretation" is simply to apply enough logic to it.  They often see discussion of any specific Bible concept as a game of confronting the person who has a "wrong" interpretation with references within the text that contradict their opponent's point of view.  They're wrong.

I have observed a few debates of this type, where two people are arguing a finer point of doctrine by claiming the other side has overlooked one verse or another.  Invariably, the debate ends with both sides arguing semantics, or what so-and-so 'must have really meant' when they chose a certain phrasing.  And no, defaulting to the original Greek/Hebrew doesn't help.

As I was observing one such debate, I remember one side pulled out a verse they believed would obviously prove the point they were trying to make.  The person leaned back in their chair with a self-satisfied grin, as though they'd just delivered the final argument that would settle the debate.  The other party stared at the verse, puzzled.  "Why are you bringing this up?  It hurts your case.  Jesus is clearly arguing the exact opposite of what you're saying."

It was at this point I realized that any religious denomination that has had over a hundred years to dedicate its members, apologists, and paid clergy to the task of analyzing the Bible will have found a way to interpret every single verse consistent with their worldview.  There are many interpretations of the Bible, yes, but the insight I want you to come away with is that they are all internally logically consistent!  (I'm sure there are some that aren't, but most established mainstream and near-fringe interpretations are this way.)

Take this back to the story you told about observed natural phenomena.  Lacking the tool of hypothesis testing, you revert to other philosophical tools.  Through a rigorous application of logic and mathematics you refine your idea.  In the end it passes all the philosophical tests you apply to it.  It's internally logically consistent.  That's a great work, and a laudable achievement.  But you still can't reject it.  All that work made your hypothesis better, but it didn't tell you whether it was more 'true' than a hundred other hypotheses we could have created from the same set of observable phenomena.  Hypotheses that - if we dedicated enough time to - would be just as internally logically consistent as the one you prefer.  You can't have confidence in your hypothesis until it has been tested.  There's no way around it.  'Non-empirical science' isn't science.  It's just philosophy.

I understand the frustration surrounding this type of problem.  It feels like the restrictions shouldn't apply.  We put so much work into the hypothesis.  It's close enough, right?  Can't we fudge that last little difference - at least until we can finally figure out how to formally test it?  But you only think you're 'close enough' because you haven't tested your hypothesis.  Experimental scientists understand this better.  They've witnessed a beautiful idea fall apart when it runs up against the real world.  It's frustrating, especially when you have to start over again, but wise researchers know that way lies some of the most profound discoveries.

Accepting non-empirical schemes as 'close enough' is the wrong direction.  That way lies epicycles and the four humors.  We're so used to the Universe being amenable to experimentation that when it isn't we get indignant.  It's like Nature is refusing to play by the rules of the game so we want to make it comply by shifting the rules just a little bit.  But there's no universal law stating the Universe has to give up all its secrets to us.  When it doesn't - if even temporarily - we don't change the rules to let our pet theories in.  Instead, we do the hard intellectual work of holding back even the most beautiful explanations as open questions.  Because despite what we want to believe, they don't have to be right.


  1. I think you're being far too black and white here and overlooking a huge amount of grey in between. Yes, there are some theories in physics, like string theory where to test them you'd have to build a supercollider the size of the solar system, and without that test we should not call string theory science. It is in fact philosophy. And on the other hand there's the Theory of Relatively which was hard to test until it turned out to exactly predict the Perihelion precession of Mercury. On stuff like this I completely agree with you, but what about something like the minimum wage. You can run some experiments, but they're not going to be very good because you don't have a "control world" available where everything is the same except the minimum wage was unchanged. So you run these experiments and you get a lot of contradictory answers. Certainly enough so that no one would talk about "the settled science of setting minimum wages". And yet people still have to act, laws still have to be passed, something has to be done, even if it's abolishing the minimum wage.

    All of which is to say people have to make decisions about numerous things where science doesn't provide any answers, or it provides really crappy answers which are barely distinguishable from random noise. We can't set those decisions aside until science catches up, they still have to be made. People want to be treated for their illness and they had to decide between the four humors framework, or nothing. As it turns out nothing was probably better, but that's also a decision. And also I'm a lot more sympathetic to the conservative argument of change anything if you're not sure, but that doesn't seem to be what you're saying...

    It's easy to say, as you did, if you can't test it, it's not science, what's much harder and more useful is to provide some framework for what should be done when science is no help.

    1. Right. I don't think we disagree here. If we can't do science, we don't give up we just do natural philosophy and hope for the best (while adjusting our confidence profile accordingly). That's as it should be.

      My point above is that what we shouldn't do is pretend we can change that natural philosophy into science without good hypothesis testing. I see people try to do this (most recently Scott over at SSC) and it's a significant error. If you convince yourself that non-science is close enough to call it science you assign too much confidence in the results. Like with unlimited biblical interpretations, just because the philosophy is sound and not contradicted doesn't mean it's right.

      We can still do the work even if the field isn't scientific (like with epidemiology). We just shouldn't treat it like it is.


Post a Comment

Popular posts from this blog

Nutrition Graph: Cereal Comparisons

Reverse Engineering Life

Cancer update: precision oncology comes into its own