I spent the last month watching, with alternating apprehension and delight, as President Trump’s cynical legal efforts to overturn the presidential election deteriorated into absurdity. After dozens of lawsuits were thrown out of court, and votes were certified in contested states, I thought we’d reached the end of the road. But it turns out there was one gut punch left to deliver, a bright red line no science-minded person like myself can bear to see crossed. That’s right, Donald Trump misused statistics.
Subscribe to WIRED and stay smart with more of your favorite Ideas writers.
The Texas attorney general filed a lawsuit Monday asking the US Supreme Court to intervene in the election. Before your heart rhythm changes too dramatically, I should tell you that legal experts consider the case “doomed.” That doesn’t mean the lawsuit can’t be dangerous. It introduced the strange-but-real number “quadrillion” into the political discourse for a couple of news cycles and seeded a new set of numerical conspiracy theories that could live on for years as so-called proof of election fraud. On Tuesday, as 18 more states prepared to back the Texas lawsuit, press secretary Kayleigh McEnany tweeted out one of its central claims: “Chances of Biden winning Pennsylvania, Michigan, Georgia, Wisconsin independently after @realDonaldTrump’s early lead is less than one in a quadrillion.” She then proceeded to type out the number with all of its 15 glorious zeroes.
Given that president-elect Biden won all of those states, the chance of his winning them is 100 percent. Nevertheless. The way this statistic was created, and then disseminated in seemingly authoritative documents, is all too familiar to me as a doctor who relies on the scientific literature. I want to suggest that baseless lawsuits and the medical research we use to guide treatments should not be using the same statistical tricks.
Science is challenging in the same way as political polling. We are asked to explain how the whole world works when we can only see one small part of it. A pollster wants to know how the country will vote by calling up a few people. Similarly, if we want to know whether a treatment improves a medical condition, we can only afford to test it in hundreds or thousands of people—although it might ultimately be given to millions. Modern statistics has tools to handle these situations.
The economist who derived the one-in-a-quadrillion election estimate, Charles Cicchetti, was using one of these tools, called “null hypothesis significance testing.” The idea is simple but insidious: Can we use statistics to prove that a hypothesis about the way the world works is compatible with what we are actually observing? The insidious part is how you pick your hypothesis.
I am assuming that the basic math behind Quadrilliongate is correct. If the group of votes counted on election night and the group of votes counted later on were pulled at random from the same pot, with the same mix of Trump and Biden voters, then yes, sure, you’d expect the outcomes to be about the same. And, sure—[math, math, math]—maybe the odds that an early lead for Trump would have been flipped upside-down are very small, like one-in-a-quadrillion small. But the trouble comes from the hypothesis and what the plaintiffs seem to think it means. Cicchetti set out to prove that “the votes tabulated in the two time periods could not be random samples from the same population of all votes cast.” See the problem yet? This is exactly what we were told would happen months in advance: a “blue shift” arising from the fact that Democrats favored mail-in ballots and Republicans leaned toward in-person voting. Hardly evidence of fraud. Cicchetti admits there is “some speculation that the yet-to-be-counted ballots were likely absentee mail-in ballots.” Of course, it’s not speculation. The day after the election, for example, the Georgia secretary of state announced there were around 200,000 mail-in ballots left to count.
Good science means asking a good question and accepting the results. Bad science, as exemplified by this disingenuous lawsuit, is when you already have an answer in mind and pick your question to best achieve that answer. The lawyers in this case wanted to show there was something fishy with the Democratic-leaning votes counted after election night, so they selected an irrelevant hypothesis guaranteed to achieve a “significant” result.
My profession—medicine—is more susceptible to this bad science than you might think. In designing a clinical trial, for example, a drug company may manipulate the control group so it can show a statistical difference. In essence, drug companies are intentionally asking the wrong question. A recent analysis in the journal JAMA Oncology found that a startling number of cancer drugs approved by the Food and Drug Administration in recent years achieved success in just that way. Instead of asking whether the new drugs were more effective than the best-available alternatives, the researchers had asked a different, shadier question: Were the new drugs better than a treatment that is second-rate? It doesn’t stop there. Even once a study is complete, researchers can keep changing their hypothesis until they find a statistically significant one, a process known as HARKing—hypothesizing after the results are known.
The enduring appeal of these tricks is that humans are impressed by big numbers. Yes, size does matter—the smaller the better. I’m talking about p-values, of course. This is the final product of these statistical machinations: a single number that suggests just how likely it is that the data fits your hypothesis. (There’s actually an endless debate over how best to describe what p-values really mean. Please don’t email me about this.) The way this is constructed in medical research, the “null” hypothesis is bad; it means two groups are the same. The drug doesn’t work better than a placebo. You can’t tell benign from malignant. A low p-value suggests you did detect an important difference.
While statistical chicanery may be laughed out of court, it’s still at home in scientific journals.
For arbitrary historical reasons, science has settled on a p-value of 0.05 as the cutoff for showing that a treatment or observation is real. That means we are comfortable with having a chance of 5 percent or less that we’re mistaken. A smaller percent is even more attractive. Doctors and scientists proudly proclaim just how many zeroes their p-value has (0.0001!). Cicchetti’s p-value of one in a quadrillion puts us all to shame.
This obsession with achieving low p-values at any cost has led to what is widely considered a replication crisis in science. But picking an irrelevant comparison is only one of the ways we can p-hack our data. The late statistician Douglas Altman called medical research a “scandal” because of this widespread reliance on poor methods, and one influential paper put it even more bluntly, claiming that “most published research findings are false.” In short, scientific results have become one-off events that fail to tell us something enduringly true. While statistical chicanery may be laughed out of court, it’s still at home in scientific journals.
Slowly, scientists are working to change these misleading incentives and methods. But the problems are not just out there in the scientific or legal worlds; they’re also within us. When I want to know the chance of some patient outcome, I immediately run to the scientific literature for an estimate. The appeal of numerical objectivity is irresistible. We crave a world that fits neatly into facts and figures. If life is going to be uncertain, we want a number to 15 decimal places telling us just how uncertain it is. Yet life is even more uncertain than that. Sometimes a treatment that seems like a sure thing fails. Sometimes a patient lives longer than anyone expected. Sometimes an obscure virus jumps from an animal to a human and paralyzes the globe. And sometimes Joe Biden wins Georgia.
More Great WIRED Stories
- 📩 Want the latest on tech, science, and more? Sign up for our newsletters!
- One man’s search for the DNA data that could save his life
- The race to crack battery recycling—before it’s too late
- AI can run your work meetings now
- Spoil your cat over the holidays with our favorite gear
- Hacker Lexicon: What is the Signal encryption protocol?
- 🎮 WIRED Games: Get the latest tips, reviews, and more
- 🏃🏽♀️ Want the best tools to get healthy? Check out our Gear team’s picks for the best fitness trackers, running gear (including shoes and socks), and best headphones
WIRED is where tomorrow is realized. It is the essential source of information and ideas that make sense of a world in constant transformation. The WIRED conversation illuminates how technology is changing every aspect of our lives—from culture to business, science to design. The breakthroughs and innovations that we uncover lead to new ways of thinking, new connections, and new industries.
More From WIRED
- Wired Staff
- Press Center
- Contact Us
- Customer Care
- Send a tip securely to WIRED
- Site Map
- Accessibility Help
- Condé Nast Store
- Condé Nast Spotlight
- Do Not Sell My Personal Info