Social scientists are closing in on some answers. The personality traits known as the "Dark Triad" — that's narcissism, psychopathy, and a tendency to see the world in black-or-white terms — play a part. So do political beliefs, particularly populism and a tolerance for political violence. Cognitive biases, like believing only evidence that confirms what you already think, also make people more vulnerable.

But according to new research, it isn't ignorance that makes people most likely to buy into conspiratorial thinking, or social isolation or mental illness. It's a far more prevalent and pesky personality quirk: overconfidence.

The more you think you're right all the time, a new study suggests, the more likely you are to buy conspiracy theories, regardless of the evidence. That'd be bad enough if it applied only to that one know-it-all cousin you see every Thanksgiving. But given how both politics and business reward a faith in one's own genius, the news is way worse. Some of the same people this hypothesis predicts will be most prone to conspiracy thinking also have the biggest megaphones — like an ex-president who believes he's never wrong, and a CEO who thinks that building expensive cars makes him some sort of visionary. It'd be better, or at least more reassuring, if conspiracy theories were fueled by dumb yahoos rather than self-centered monsters. Because arrogance, as history has repeatedly demonstrated, is a lot harder to stamp out than stupidity.

A decade or so ago, when Gordon Pennycook was in graduate school and wanted to study conspiracist thinking, a small but powerful group of unelected people got together to stop him. It wasn't a conspiracy as such. It was just that back then, the people who approved studies and awarded grants didn't think that "epistemically suspect beliefs" — things science can easily disprove, like astrology or paranormal abilities — were deserving of serious scholarship. "It was always a kind of fringe thing," Pennycook says. He ended up looking into misinformation instead.

It still isn't entirely clear whether more people believe conspiracy theories today. Maybe there are just more theories to believe. But researchers pretty much agree that crackpot ideas are playing a far more significant role in politics and culture, and they have a flurry of hypotheses about what's going on. People who believe in conspiracies tend to be more dogmatic, and unable to handle disagreement well. They also rate higher on those Dark Triad personality traits. They're not stark raving mad, just a tick more antisocial.

But at this point, there are just way too many believers in cuckoo theories running around for the explanation to be ignorance or mental illness. "Throughout most of the 1970s, 80% — that's eight zero — believed Kennedy was killed by a conspiracy," says Joseph Uscinski, a political scientist at the University of Miami. "Would we say all of those people were stupid or had a serious psychological problem? Of course not."

"Open-minded thinking isn't just engaging in effortful thought," Pennycook observes. "It's doing so to evaluate evidence that's directed toward what's true or false — to actually question your intuitions." Pennycook wanted to know why someone wouldn't do that. Maybe it was simple overconfidence in their own judgment. 

Sometimes, of course, people are justified in their confidence; after four decades in journalism, for example, I'm right to be confident in my ability to type fast. But then there's what's known as "dispositional" overconfidence — a person's sense that they are just practically perfect in every way. How could Pennycook's team tell the difference?

Their solution was pretty slick. They showed more than 1,000 people a set of six images blurred beyond recognition and then asked the subjects what the pictures were. Baseball player? Chimpanzee? Click the box. The researchers basically forced the subjects to guess. Then they asked them to self-assess how well they did on the test. People who thought they nailed it were the dispositional ones. "Sometimes you're right to be confident," Pennycook says. "In this case, there was no reason for people to be confident."

Sure enough, Pennycook found that overconfidence correlated significantly with belief in conspiracy theories. "This is something that's kind of fundamental," he says. "If you have an actual, underlying, generalized overconfidence, that will impact the way you evaluate things in the world."

"That's often what happens with these really wealthy, powerful people who sort of fail upwards," says Joe Vitriol, a psychologist at Lehigh University who has studied the way people overestimate their own expertise. "Musk is not operating in an environment in which he's accountable for the mistakes he makes, or in which others criticize the things he says or does." 

But what makes Pennycook's finding significant is the way it covers all the different flavors of conspiracists. Maybe some people think their nominal expertise in one domain extends to expertise about everything. Maybe others actually believe the conspiracy theories they spread, or simply can't be bothered to check them out. Maybe they define "truth" legalistically, as anything people can be convinced of, instead of something objectively veridical. Regardless, they trust their intuition, even though they shouldn't. Overconfidence could explain it all.

Pennycook's findings also suggest an explanation for why conspiracy theories have become so widely accepted. Supremely overconfident people are often the ones who get handed piles of money and a microphone. That doesn't just afford them the means to spread their baseless notions about Democrats running an international child sex-trafficking ring out of the basement of a pizza parlor, or Sandy Hook being a hoax. It also connects them to an audience that shares and admires their overweening arrogance. To many Americans, Pennycook suggests, the overconfidence of a Musk or a Trump isn't a bug. It's a feature.

The next step, or course, is to figure out how to fight the spread of conspiratorial nonsense. Pennycook is trying; he spent last year working at Google to curb misinformation; his frequent collaborator David Rand has worked with Facebook. They had some meetings with TikTok, too. That pop-up asking whether you want to read the article before sharing it? That was them.

And what about the bird site? "Twitter? Well, that's another thing altogether," Pennycook says. He and Rand worked on the crowdsource fact-check function called Community Notes. But now? "It's all in flux, thanks to Elon Musk."

But Pennycook's new study suggests that the problem of conspiracy theories runs far deeper — and may prove far more difficult to solve — than simply tweaking a social-media algorithm or two. "How are you going to fix overconfidence? The people who are overconfident don't think there's a problem to be fixed," he says. "I haven't come up with a solution for that yet."

Through our Discourse journalism, Insider seeks to explore and illuminate the day’s most fascinating issues and ideas. Our writers provide thought-provoking perspectives, informed by analysis, reporting, and expertise. Read more Discourse stories here.