Open Questions: Introduction

In the interest of brevity, I've decided to spin this post off into a sort of mini-series.  I first sat down to write some thoughts, but then realized I needed to explain other concepts that required other concepts and ...  I figure I'll capture each point in its turn, but in the process I want to get across an overall idea about open questions, hence the title of the mini-series.

To start, I'd like you to indulge me in a small experiment.  It's not a well-controlled experiment, but humor me.  First, go read this list someone generated of open questions.  Then come back here.  you don't have to read the whole thing, but get through at least six or seven of them.

I'll wait.

Done?  Great.  Did you notice that the author attempts - and generally fails - to identify explanations for each of the phenomena?  Other times there are multiple acceptable explanations and you the reader are left to guess at which is true.  Since the author was trying to write a list of unsolved mysteries, she did a pretty good job of withholding her own judgement of what hypotheses might best explain each one.

Now I want you to honestly answer about your own internal thought process while reading the article.  Specifically, this was an article about unsolved mysteries.  It's about open questions for which nobody knows the answer.  So what did you think about while reading the entries?  Did you formulate your own best-guess for what might be the answer to each open question?  Are there a few of the open questions for which you have a favorite hypothesis?

Open questions are incredibly important in science, but for some reason they're very difficult to look at.  They almost require you to train your brain in order to look at them, similar to these pictures about trypophobia.  It's not that you can't see them, it's that looking at them directly is viscerally uncomfortable.  You look away and try not to think about it, or formulate an answer you think 'probably fits' and then go along your way as though the problem is solved.  And this concept of 'holding a question open' is itself counter-intuitive.  We like to think we're very open minded about open questions.  Indeed, we deal with open questions all the time!

The problem is there is a strong tendency to try and close open questions; perhaps to accept that there's no 'official answer', but of course there's an answer we prefer - one that is probably right.  Certainly in science we're working to close open questions.  But it's important along the way not to close them too early - specifically, not to close a question before the facts come in.  This is the hard part, given the strong natural tendency to close open questions.  Part of the lure, I think, comes from  the idea that a hypothesis fits the evidence very well.  The hypothesis must be true; otherwise how do we explain why so many pieces of evidence fit together so well?

The answer to this question is similar to a phenomenon I discussed in a previous post about why studies often produce results that don't replicate.  The researchers think they're looking at something specific, when really they're pulling from a much larger pool of possibility.  Something similar is happening with hypotheses.  When I describe phenomena to you in the context of some hypothesis, you might be persuaded to believe that hypothesis is well-founded.  After all, there's all this data to support it!  This ignores some fundamental principles of the scientific method.

Without looking it up, answer this question: what's the first step of the scientific method?  If you said, "formulate a hypothesis", you're wrong.  What is that hypothesis based on?  The first step of the scientific method is to make an observation - or a series of observations - about the world around you.  After you make observations, you formulate a hypothesis.  These two step are so natural we have a tendency to overlook the first step, rolling it into the second one, skipping the third step (design an experiment) and assuming our original observations are the fourth and fifth steps (collect and analyze data).  It's easy to assume a hypothesis is supported because the evidence we used to create the hypothesis fits it really well.  Of course it does!  It was literally designed to fit the observations.  It's not until that hypothesis predicts additional evidence - and is not disproved by other evidence - that we should have any confidence in it.

Yet even a completely unproven hypothesis (by which I mean it has not predicted new observations; repeats of our original observation don't count here) still feels like it might be true.  It feels like maybe we should put some weight behind it, even if it's unproven.  Let me give you an example that has become famous in the field of cancer research.

A Radical Hypothesis

After anesthetics were discovered, surgery became a viable treatment for a number of conditions - since your patient wasn't in constant pain throughout the procedure - and especially for cancer.  Cancer may seem like a complicated disease, since each different type of cancer can produce different types of symptoms, but the way most cancers kill you is simply by growing somewhere they're not supposed to.  Digestive system not working?  If cancer is involved, it's probably because a tumor is blocking the road.  Difficulty breathing?  A tumor may be crowding the space the lungs use to expand.  Headaches, personality changes, etc?  A tumor in the brain may be crowding out space in the skull.

The simple solution to these problems is to just go in and cut the cancer out.  Surgeons were on hand to do just that.  Problem solved.

Except the tumors would often come back.  And indeed, they might sprout additional tumors, sometimes in new tissues.  Surgeons observed this and made the obvious hypothesis: we didn't get all the cancer with the original surgery.  This was followed by an additional hypothesis: we can get all the cancer cells by cutting out some of the surrounding tissue.

There's an analogy here to pulling weeds.  You might get the large, visible stalk, leaves, and flower of a weed but still miss the root deep down underground.  There are no visible signs of the weed on the surface, but it will grow back - from the root - stronger than ever.  If you can get the root (Latin radix) of the cancer it won't grow back.  That means digging down in the soil for a weed, and digging down into the healthy tissue for cancer.  Since they were looking for the root, they used the Latin origin and called these surgeries "radical".  In the lead on this hypothesis were breast cancer surgeons, who routinely performed radical mastectomies.

As mentioned above, when they first started, they were just pulling out the tumor.  They gradually began to take some of the healthy tissue beside that, extending to the whole breast, then to both breasts, then to the muscles below the breast.  Eventually surgeons were pulling tissue from the arms, neck, and taking major muscles from the chest.  The more tissue they removed, the better, since the cells harboring the cancer could be hiding in any of these places.  Women came from these surgeries horribly disfigured, but they were encouraged that the results were in the service of fighting the cancer, and keeping them alive.  Sometimes the cancer came back anyway, in some tissue the surgeon chose not to cut out for one reason or another, and the patient would die.  Why hadn't he taken more tissue from the neck?  If he had, there wouldn't be a tumor there, and the patient might be alive today.  Clearly the problem was the surgeon hadn't been aggressive enough at clearing out the last of the root of the tumor.

Many years after this became common practice, some researchers noticed the hypothesis - that a radical mastectomy was better than a simple lumpectomy (just taking the cancerous cells and leaving everything else alone) - had never been formally tested in a clinical trial setting.  They designed a clinical trial to compare the two approaches.  One group of women would be randomly assigned the radical mastectomy, and another group would get the lumpectomy.  It couldn't be blinded, of course, since you'd notice if both breasts and a bunch of neck tissue were there or not, but it was good enough to at least test the hypothesis.  They began enrolling patients.

Or at least, they tried to.  Many physicians balked at the proposal.  It was like testing whether a tourniquet worked to save a life by allowing one set of patients to bleed out and applying basic first aid to the 'lucky' patients in the treatment group.  Sure, science is important, but it's simply unethical to put patients' lives at risk to test a hypothesis.  Especially a hypothesis that was so obviously right.  How could cutting extra tissue out not save lives?  This experiment was irresponsible.

Eventually, after finding some doctors who agreed to participate - against the advice of many of their peers - and extending their search for these doctors into Canada, the researchers found enough patients to fill their study.  They ran the numbers and published the results - which rocked the world of surgical oncologists everywhere.  There was no difference in survival between patients who had the simple lumpectomy versus the radical mastectomy.  The hypothesis justifying the radical mastectomy was wrong.

On the other side of the experiment, it seems obvious doctors shouldn't have given weight to an unproven hypothesis.  The argument that it was unethical to even ask this question is bad logic.  Many women had horribly disfiguring surgeries that were entirely unnecessary because a hypothesis was implemented without being tested first.  That seems like the step along the way that was unethical.  (Please note I'm not claiming the surgeons who invented the radical mastectomy were unethical.  It was common before the popularization of 'evidence-based medicine' to implement ideas based only on a bare hypothesis.  Indeed, this is still common to this day.  These surgeons were doing the same thing everyone else was.)

Yet where would we be if a brave group of researches hadn't fought against an unpopular test of a hypothesis that everyone assumed must be right?  This, then, is the allure of a good hypothesis that has never been tested.  It feels wrong to even need to test it.  It's a waste of time.  Of course it's true.

Unless it's not.

How do you inoculate yourself against the allure of a persuasive hypothesis?  It's probably impossible to become completely immune to overconfidence in a hypothesis.  Current theories about how the brain works suggest this way of thinking - extrapolating from hypothesis, explaining the world around us with stories - is a normal part of how the human brain works through problems.  However, there are some things you can do to train your brain to be more cautious.

The first is to be aware of the problem to begin with.  Hopefully this post checks that box.

Second, understand that the nature of most hypotheses is to be wrong.  For every correct hypothesis that explains all the previous data, there are dozens - perhaps hundreds - of incompatible hypotheses you could conjure up to explain the same observations.  Coming up with the hypothesis is actually the easy part.  As I mentioned above, it's an automatic mechanism your brain engages in when it comes across an unexplained set of observations.  In my experience, the process of discovery is a pattern of disproving hypothesis after hypothesis until you fail to disprove one.

This is a difficult intuition to develop unless you've done some experimental work in a laboratory setting.  However, if you know any scientists, try asking them about their research sometime.  If you can survive the boring bits, you'll notice something about the way they talk.  First, they'll tell you about all the observations that led them to their current hypothesis.  Then, as they transition to talk about their own work, notice how the words they use subtly shift.  Their confidence in what others discovered will be much higher than their confidence in their own work.  This is not false modesty.  After you've disproved a dozen hypotheses, it's natural to assume your current hypothesis will also be disproved with the next experiment.  In the lab, scientists learn through experience this important lesson: unsupported hypotheses cannot be trusted.

Third, practice holding some open questions in your mind without accepting any one explanation.  This is hard to do.  After you get good at developing the intuition that most hypotheses are wrong, you'll be tempted to develop a new hypothesis in place of a hypothesis you don't like.  This can lead to selective application of the principle and isolated demands for rigor.  Or it can lead you to say that nothing is ever truly 'knowable' and wash your hands of empiricism.  Both are bad intellectual habits to form.  You can avoid falling into them by allowing open questions to remain unanswered in your mind.  This is difficult, but  rewarding.  One of the biggest obstacles to understanding a subject that is not currently understood is the strong tendency to close open questions.  If you're used to holding a question open in your mind it's easier to create new hypotheses as new evidence comes in, and easier to jettison old ones as they're disproved.

This is where that natural tendency to formulate new hypotheses becomes very useful.  When you can hold a question open, you'll be more open to developing multiple new hypotheses for any given unsolved problem.  As you collect new observations, some of these hypotheses will be disproved while others survive much longer.  One thing is certain, though.  You'll develop a unique ability to see different solutions than you did before you developed this skill.

To give you some practice at holding open questions in your mind, I'm starting a series where I discuss some open questions.  Keep in mind, as you read these, that these are (as of the time I write them) still open.  Each subsequent explanation has some proponents.  They have problems, or are unconfirmed.  Your task is not to accept any single explanation, but to question all of them at once and develop new explanations of your own.  Then question those.  You'll know you're successful when you start disproving a few hypotheses - even if the only hypotheses you disprove are your own.

Here's a list of some ideas I'm currently working on.  I'll also add a section in the archive so you can find them more easily as they come out.  Feel free to add more in the comments below.

  • Origin of life
  • Molecular evolution
  • Fermi's paradox (dark forest)

Comments

Popular posts from this blog

Reverse Engineering Life

Cancer update: precision oncology comes into its own