A Novel Defense of Bad Social Psychology Studies

Normally a mini-essay by a journeyman reporter for the New York Times would not be worth rebutting with another mini-essay. We can all agree that the world has quite enough mini-essays as it is. But the recent piece by science writer Benedict Carey is a landmark in its own small way. It demonstrates two related cultural dilemmas—a crisis in social science, usually called “the replication crisis,” and a crisis in the news business, as yet unnamed. And it shows how our “thought leaders” hope to evade both of them.

The crisis in the social sciences has grown so obvious that even mainstream social scientists have begun to acknowledge it. In the past five years or so, disinterested researchers have reexamined many of the most crucial experiments and findings in social psychology and related fields. A very large percentage of them—as many as two-thirds, by some counts—crumble on close examination. These include such supposedly settled science as “implicit bias,” “stereotype threat,” “priming,” “ego depletion” and many others known to every student of introductory psychology. At the root of the failure are errors of methodology and execution that should have been obvious from the start. Sample sizes are small and poorly selected; statistical manipulations are misunderstood and ill-performed; experiments lack control groups and are poorly designed; data are cherry-picked; and safeguards against researcher bias are ignored. It’s a long list.

The second dilemma has to do with the first, though it is less often discussed. The great bulk of journalism—what used to constitute the stuff of a large metropolitan daily newspaper—involves only a handful of general subjects. We read sports, politics, weather, celebrity doings, and pop science. Without them the trade would collapse. Readers and editors alike especially love stories that begin “A new study finds . . . ” or “Scientists have discovered . . . ” This last sort of news—easily digested findings that scientifically explain the mysteries of human behavior—is fed and constantly replenished by the same social science whose elemental assumptions are withering before our eyes. This is bad news for the news.

The circle is vicious indeed. Journalism craves pop-science stories from researchers, who like publicity and must get their work into print, according to the pitiless mandate of publish or perish. The researchers’ urgency encourages corner-cutting and conclusion-jumping, which conveniently tend to produce flashy findings, which are inhaled by news outlets, which publish them under the headline “Researchers find!” and then turn back to the researchers to demand more, more, more.

The growing realization of this unhealthy co-dependency is the kind of thing that can ruin a science writer’s day—his livelihood, too. For Benedict Carey, the Times science writer, the collapse of social psychology is an understandably painful subject. The tone of his mini-essay is mournful, as if he’s watching an old friend walk to the electric chair.

He opens his article by mentioning the 50-year-old Stanford Prison Experiment, a simulation designed to prove that people in positions of power are more likely to behave cruelly than the Dorothy Gales among us, the small and meek. The prison experiment, which required psychology students to play-act as prisoners and prison guards, launched a thousand other experiments that used its findings as an unquestioned premise. The unique dangers of power disparities—as found, for example, in capitalist societies—became a theme of social science, confirming the leftish, class-based politics of social psychologists.

In the last 10 years, thanks to several whistle-blowing researchers working independently, the prison experiment and its findings have been largely discredited. The editor of at least one popular textbook has removed mentions of it. It turns out that the behavior of college students in role-playing exercises under the watchful eye of their professors doesn’t tell us much about the behavior of ordinary people in the real world, no matter how powerful or powerless they are. This has surprised social psychologists. Many of them still refuse to believe it.

Carey also mentions another famous, and much cuter, experiment called the Marshmallow Test. It, too, he notes glumly, has been subverted by further examination. In the marshmallow test, young and adorable children were filmed as they tried not to eat marshmallows. The researchers concluded that children who were taught the ability to delay gratification would, thanks to this single trait, grow up to have happier and more successful lives. On the basis of the marshmallow experiment, policymakers over the next generation developed character-building programs that became all the rage in the fad factories of public education. Teach a kid self-control when he’s 5, went the thinking, and 20 years later you’ll have a college graduate on your hands.

Anyone uncontaminated by social science would understand this proposition to be laughably mechanical and simplistic. And even social scientists are now seeing that the study was severely limited in application. Almost all the kids in the test were white and well-to-do; the results didn’t take into account family stability, the level of parents’ education, the behavior of peers, or any of the other infinite factors that form a child’s character. For nearly 30 years the “marshmallow effect” was science. Now it’s folklore.

Carey could have picked dozens of other examples. Every few weeks, it seems, another established truth of social science comes a cropper. But Carey is a man of faith, as believers in social science must be. He doesn’t want to let go. He is wounded by critics who think the replication crisis somehow undermines social psychology’s standing as science. “On the contrary,” he writes. The crisis proves social science is self-correcting, just the way real sciences are.

“Housecleaning is a crucial corrective in science,” Carey writes. This is true. He also says “psychology has led by example.” This is not true. A science cannot correct itself unless its findings are subjected to replication, but even now such self-examination is rare in social science—indeed, it is often deemed seditious. Reformers and revisionists who question famous findings are subjected to personal and professional abuse from colleagues online and elsewhere.

Still, Carey insists, psychology is a science. It’s just not a science in the way that other, fussier sciences are science. “The study of human behavior will never be as clean as physics or cardiology,” he writes. “How could it be?” And of course those farfetched experiments aren’t like real experiments. “Psychology’s elaborate simulations are just that.”

These are large concessions, but Carey doesn’t seem to realize how subversive they are. Those “elaborate simulations” are held up by social scientists as experiments on a par with the controlled experiments of real science. We are told they re­-create the various circumstances that human beings find themselves in and react to. The only reason anyone pays attention to social psychology is that its findings are supposed to be widely, even universally, applicable, as the findings of the physical sciences are. Otherwise it’s unlikely news outlets would hire reporters to write about social science.

Carey’s defense of social psychology fits the current age. It is post-truth, as our public intellectuals like to say. “[Social psychology’s] findings are far more accessible and personally relevant to the public than those in most other scientific fields,” Carey writes. “The public’s judgments matter to the field, too.”

Okay, but are the findings true? Carey’s answer is: Who cares? The headline over his piece summarizes the point. “Many famous studies of human behavior cannot be reproduced. Even so, they revealed aspects of our inner lives that feel true.”

Feeling true is what’s important. “It is one thing,” Carey goes on, “to frisk the studies appearing almost daily in journals that form the current back-and-forth of behavior research. It is somewhat different to call out experiments that became classics—and world-famous outside of psychology—because they dramatized something people recognized in themselves and in others.”

The public likes them. They’re famous. They’re classics! And they feel true.

Or true-ish, anyway. This is good enough for the New York Times and its guardians of science. They have adapted the scientific method to the Trump era.

Leave a Reply