Love and p-hacking

The following is a real conversation I had with a fellow graduate student (name and identity appropriately changed).

I knew that our relationship was doomed, and we were only being self-destructive by continuing it behind everyone’s back. However, my phone was once out of battery, and I was stranded in the middle of the city. Suddenly, I saw my boyfriend in the crowd. I called out to him, and he helped me get back home.

What are the chances that the one time I’m in desperate need of help, I would see my boyfriend? That had to be a sign from a higher power that we were meant to be together.

This graduate student was a star student from one of the Indian Institutes of Technology, and was set to have a great academic career. However, she was embroiled in a relationship that was destroying both her academic standing and her mental peace. Because she’d seen a “sign” from god.

This reminded me of my classmate from my Masters program, who was widely considered a “brilliant but lazy” mathematics student. He saw a “sign” from God, and decided that he must move to a certain city where his crush had moved. He took a large loan to fund his further education in that city. He never met his crush in that city, who refused to respond to his messages, and also failed out of his program.

How could the “signs” from God have been so wrong?

P-hacking is concept that is well-known but less well understood. Suppose you want to show a correlation between the rise of gun violence in American cities and the rise in chocolate consumption. You take 100 cities in America, and study their gun violence and chocolate consumption rates. Out of these 100 cities, it is possible through random variation alone that one city may show a positive correlation between the two variables. You then isolate that city, and claim to all your peers that you have indeed discovered that your hypothesis is true. “The more you eat chocolates in Denver, the more people you shoot!”, you may shout from rooftops, perhaps threateningly brandishing a Hersheys bar if you’re in Denver.

In other words, you didn’t start with “Let me explore whether a higher chocolate consumption leads to more gun violence in Denver.” You just took a large database, which may contain some untrue correlations purely by chance, and isolated those patterns.

P-hacking is akin to a scam in the statistical paper-publishing industry. Moreover, it is also effectively what my two friends above were doing.

The female graduate student didn’t start with “If I am ever stranded without my phone, and I see my boyfriend, I will take it to be a sign that we are meant to be together.” She didn’t have a hypothesis to begin with, which she could then have tested. Over the course of years (and hence a large dataset), she just saw an unlikely event, and retroactively interpreted that to be a sign. Similarly, my friend from my masters program hadn’t narrowed down his definition of what a sign from God could be.

This may seem to be a flimsy premise to write a blogpost on, but I often see intelligent people, who I’d love to see succeed, self-obliterate grandly because they may have seen some of these “signs”- that they’re meant to be with that one person who is effectively as nourishing to their dreams as cyanide. If you know these people in your life, shake them by the shoulders and make them read this post (or perhaps a better post on p-hacking). Our friends deserve our condescension and forced blogpost reading.

What’s an example of a “sign” that is not a result of p-hacking? It is one in which you first choose what the “sign” should be, and then observe whether you’re given such a sign. Thereby, this.

Advertisement