Start with an Observation

My wife is a science teacher, and when we first started dating she was teaching high-school science to freshmen. On our first date, she described her difficulties teaching the scientific method to her students. By way of demonstration, she asked me, "What are the steps in the Scientific Method?"

I said, "Well, you start with an observation, then-"

"Exactly! You have to start with an observation. They don't get this concept. They think they start with a hypothesis. 'Maybe you didn't start with an observation, maybe you were just thinking one day and you formed a hypothesis.' But why did you form that hypothesis? If you think back through the process, you did it because you started with an observation. Somewhere down the line you observed something and it caused you to ask questions."

I pointed out that this powerful idea isn't just difficult to understand for high-school science kids. It's also difficult for some researchers to understand. Some people have observed the rise of 'big data' and hypothesized that the kinds of things we will be able to learn from new data analytics techniques will allow us to conduct 'hypothesis-free' science. The idea is that we'll discover things we never would have through observation. Answers will, in effect, 'fall out' of the data set from complicated algorithms or new AI technologies. All without first starting with an observation and generating a hypothesis.

When I first heard about this idea, my response was, "They're just finding new ways to generate hypotheses. Those hypotheses still need to be validated through experiment. These researchers clearly don't understand the scientific method they think they're upending." They're just pulling their observations out of existing data.

It is common to use existing data as the foundation of a new hypothesis, and this can feel to some scientists like it's not starting with an observation, but just going from one experiment to another. But the results of the last experiment can form the foundation of the observation for the next hypothesis. And while this process can be nuanced and interesting, I want to make a more subtle point about the importance of starting with an observation.

This idea - that we really do need to start with an observation - is more powerful than it appears at first glance. Indeed, sometimes you run into problems as a researcher because you unthinkingly push forward from existing data. But sometimes the hypotheses you should be testing can't be formulated just by looking at the data someone has already been collecting. You find you need to refocus on the observation step before moving forward.

I read a book  a few years ago that illustrated this idea perfectly. The author was a sociologist in Chicago in the mid 1990's. He was trying to answer the question, "why are inner city people poor?" So he created some surveys and walked into some of the poorest areas he could find. The account of his trip to the Projects is both hilarious and instructive. People looked at him with his clipboard and laughed. He told them he was researching poverty and they asked to see the survey. One leader of a local gang scoffed at the questions themselves, passing them around to others who also laughed at the idea he could learn anything with this survey. Why did he think he was going to learn anything like this? What he needed to do was spend some time with the people in the projects. Clearly, he didn't even know the right questions to ask.

In other words, he needed to start with an observation.

The rest of the book is worth reading in its own right. The author starts from a place where the questions he's asking don't even make sense to the people he's studying. For example, he asks things like, "Why don't you call the police when someone commits a crime against you?" The people he's studying can't conceive of a world where calling the police would help in that situation. How, then are they to explain how the world they know is different from a world they don't know? But because the researcher hadn't started with an observation, he couldn't even understand the answer to his question. Which came in the form of, "Why would we call the police?"

All this brings me to an important observation. I often see researchers scoff at anecdotes. "The plural of anecdote isn't data," is the common refrain you'll hear in the halls of academia. And that's true! But it's also true that the singular of data is not anecdote. I've written before about this idea. When we generate statistics, we're making observations of a group. But those observations don't describe the individuals in the group. Perhaps we think, "Sure this doesn't describe everyone, but it at least describes most people." This is wrong. Statistics don't even describe very many people in the group! That's because there's something unique about each person that can be difficult to pull out when you lump them together with everyone else. If you start all your observations by looking only at statistics, you're bound to miss many of the important details. By ignoring anecdotes, you find yourself asking questions that you won't even be able to understand the answers to, because the 'people' your mental model is based on don't exist.

(I'm going to put a placeholder here for a future post that outlines this idea perfectly. I'll probably never write this post because it's far too technical, so I'll outline the basic idea here and you can either research it yourself or wait around indefinitely.

One of the main techniques immunologists employ is a technology called flow cytometry. The basic idea is that we have a machine capable of measuring cells single-file based on certain characteristics. These characteristics all go on a plot and we can describe the effects of all kinds of treatments, infections, diseases, responses, etc. by the different types of immune system cells that show up on these plots. It gets complicated very quickly. As an immunologist, I've observed multiple major breakthrough discoveries that followed the following format:

'Look at this flow diagram. You think this group of cells are all the same thing. But we did some fancy experiments and found that a small subset of these cells aren't anything like the others! This changes everything we thought we knew about how the immune system responds to the thing we're studying.' In other words, the statistic we were looking at lumped all the important data into one measurement, assuming the effect we were looking for was in the data we were already measuring. It wasn't. The important targets weren't being measured.)

With this in mind, I'd like to share a few timely anecdotes that I think could help us generate better hypotheses. I think we need to start dealing with specific anecdotes in order to better understand how to understand how Starting With an Observation works in practice. I have friends who keep sharing statistics about police bias. They claim there is no such thing, and people who are upset about it are wrong. The data doesn't support the hypothesis that policing is biased, they say. We can't skip steps, though. Let's back up a moment so we can start with an observation. We'll see how that changes the discussion entirely.

1. Abuse of power

I have an uncle who, like me, is as white as the driven snow. He's also very conservative. Even so, he told me about an experience he had while vacationing in a town in the Western US. He drove there, so his license places were clearly from out of state. On his way into town, he got pulled over by a police officer who asked, "Are you visiting from out of town?"

He was very polite to the officer, telling him where he was from and that yes he was on vacation. He had both hands on the wheel, and was doing everything right.

"Have you been drinking?"

"No, I don't drink."

"Any alcohol in the vehicle?"

"No. Is there a reason you pulled me over?"

No. The officer just wanted to make sure everything was in order.

My uncle drove into town and again got pulled over. This was by a different police officer, but the story was the same. Out of town? Yes. Drugs or alcohol? No, why'd you pull me over? No reason in particular. Be on your way.

Finally, while he was leaving he got pulled over a third time. He wasn't there that long, to get pulled over so much. The officer approached his window and asked, "Are you from out of town?"

My uncle was understandably frustrated. He's just trying to get from one place to another. This is a free country, and he hasn't done anything wrong or even suspicious and yet if he tries to go anywhere he has to stop by the side of the road for ten minutes or more while the police do their thing. They freely admit that they don't even have a reason to pull him over! They just want to find something on him because he's from out of town. His response to the officer's question was admittedly unwise, "No shit, Sherlock. How'd you come to that conclusion?"

The officer asked him to get out of the car. Could he search the trunk? Would my uncle submit to a field sobriety test? The stop took considerably longer than it might have if my uncle had responded the way he knows you're supposed to when you get pulled over. He knew he'd been dumb. He knew what he should have done, which is exactly what he did on those first two stops. He knew his own actions led to the subsequent requests by the officer to get out of the car, to the search request, and the field sobriety test.

Yet he doesn't entirely blame his own lack of self-control for the situation. Saying that the situation was created by the snarky comment ignores the larger pattern of behavior that led to it. The comment was unwise, given the power dynamic, but it's not correct to say that the comment was 'wrong'. The abuse of power that led up to it was wrong. The comment was just a dumb way to react to that abuse of power. Still, can you blame him for being upset - and voicing his objections - at being the victim of such an egregious abuse of power?

2. A higher power

This same uncle was golfing with the head coach of the local university's football program. The coach told him about a player on a full-ride scholarship who tendered his resignation from the team and from the university. "I'm leaving town. I just can't live here." The player was black, and in his letter he complained he'd been pulled over three times in the week since he'd moved to town. There had never been any charges or even any reason to pull him over. The police were just trying to figure out if there should be a crime attached to him. Like my uncle, he was constantly a suspect of criminal activity but the suspicion was based on something unrelated to crime.

The coach didn't want to lose the player, who was a star recruit. So he asked if he could ride around with the player for a bit. They drove about, and sure enough the player got pulled over again. The officer gave him the same litany of questions. The player was let go again, but not before the coach got the officer's name and badge number.

The coach called the mayor. "I have a player here who keeps getting pulled over." He gave the name and badge number, then said, "Look this needs to stop. Otherwise I'm going to take this to the press and you're going to have a major PR problem on your hands." The player didn't get pulled over anymore.

Was this because the police department changed its tactics, or was it because they attached a note in their system to the license place of that one player? I don't know. At least the player was lucky enough to have someone in a position of power who could address the injustice in his specific case.

Formulate a hypothesis is a different step in The Method

My uncle doesn't get pulled over in his home town. The football player wanted to go home because he doesn't get pulled over in his home town. In both cases, not only were the people involved not doing anything wrong, they were never charged with anything.

It's clearly wrong - and unconstitutional - to treat a class of people as though they are suspects of a crime that hasn't even been identified yet. This kind of policing is less in the spirit of "Protect and Serve" and more in the spirit of "Find and Enforce".

The anecdotes aren't statistical data. The point in sharing them isn't to say, "Therefore all police departments do this thing." They're our first step observation we need to work from. We start here and build with the next step in the scientific method: formulate a hypothesis. From the above observations, we have several options. We could hypothesize that all police everywhere do profiling of groups they consider 'outsiders' or 'likely offenders'. We could also hypothesize that some local police departments enforce the law unevenly, based on external characteristics that have nothing to do with suspicion of criminal activity.

We might choose to follow either hypothesis, but for now let's follow the first one.

There have been several attempts to study racial bias in police encounter rates. It looks like it's difficult to identify, and the results are inconsistent. Maybe because it depends on the specific police department you're looking at. Some people have looked at this literature and concluded that there's no strong evidence of police bias, so the whole matter should be dismissed.

Okay, but what about the observations we started with?

"We looked into it, and there's no evidence to support the hypothesis."

Sure, but there are a lot of anecdotes here. Lots of people are saying the way they've been treated by the police feels like it's not just random noise.

"The plural of anecdote is not statistics."

I want to stress here that I'm not trying to elevate anecdotes above statistical or other methods we generally associate with 'legitimate scientific inquiry'. I'm saying there is a strong tendency to dismiss anecdotes due to their hypothesis-generating role. We test the hypothesis, reject it, and move on. But the hypothesis isn't the observation. And just because we used the observation to generate the hypothesis doesn't mean it's attached to it in some way, and must be rejected if the hypothesis is rejected. 'Form a hypothesis' is step two, not step one.

We can reject the hypothesis we reasoned out in our heads, but that doesn't explain our initial observation. Instead, it calls anew for answers to that observation. In other words, when the statistics don't match the hypothesis, we reject the hypothesis and go back to the observation in search of a new hypothesis. We do not reject the observation. We reject the hypothesis. I'm stressing this point because it is at the heart of so many failed attempts at applying the scientific method.

So what we should do in the profiling case, instead of ignoring the policing observations, is go back to those anecdotes and pick up where we left off. I mentioned a second hypothesis earlier: that local police departments enforce the law unevenly, based on external characteristics that have nothing to do with suspicion of criminal activity. This hypothesis fits other anecdotal evidence pretty well, and it would explain why so many people are upset at 'the police' as an institution.

How would evidence for or against this new hypothesis show up in crime statistics? From our observations, it seems both clear and obvious that some police departments engage in unethical policing tactics while others don't. The specific tactics will vary from one region to another, so attempts to pull out a pattern using nationwide statistics are unlikely to find the individual police departments that pull people over for having out-of-state plates, or that pull over black students repeatedly without ever charging them. The individual studies listed above might find no differences due to methodological choices, or more likely because they were looking in the wrong place. A sample of one police department in Kansas may not tell us what we want to know about the police in Michigan or Minnesota, or even elsewhere in Kansas.

Maybe there's a way to track this kind of thing more directly, though. Don't we keep data on individual police stops? I would think a systematic review of various police department records should be able to pull out patterns where certain individuals keep getting pulled over without ever getting charged with anything. Those are departments that should be suspect of having informal biases that result in abuse of power. I'd like to see this kind of methodology being employed, since it directly addresses the observations we started from: repeat stops not based on suspicious activity.

I don't know this literature well enough to tell who is doing this research. It may be that there's a lot of county-level data, and that after exhaustively reviewing the evidence we can't find support for our second, more nuanced, hypothesis. In that case, we still need to explain the anecdotal evidence. Our recourse is not to reject the observation, but to return to it. We need a hypothesis to fit our observations. And - critically - we need evidence to support that hypothesis.

I've heard some people dismiss all these anecdotes as 'situational', 'non-representative', or 'selection bias'. But those dismissals are presented without evidence. They are new hypotheses that would need to be tested before we can begin to believe that they explain the observation we started from. Dismissing an observation is different from providing evidence that explains the source of that observation.

As recent news reports confirm, the observations don't just go away. Ignoring them keeps us from moving forward, making new discoveries, and solving problems. Instead, we need to start with an observation, and move forward from there.

Comments

  1. Great post. It was illuminating, and an interesting way to come at the current crisis. Two things:

    1- I've been hearing about "Gang Leader for a Day" forever but despite the volume of books I read I had never picked that one up. This post, and your recommendation was what finally pushed me over the edge.

    2- I thought this sentiment was particularly valuable: "We do not reject the observation. We reject the hypothesis. I'm stressing this point because it is at the heart of so many failed attempts at applying the scientific method." I'll be using this idea for sure.

    ReplyDelete

Post a Comment

Popular posts from this blog

Nutrition Graph: Cereal Comparisons

Covid-19: Epidemiology is useful

Open Questions: The Origin of Life