Pangram Has Emerged as the Gold Standard of AI Detection. Should You Trust It? - WIRED

PANGRAM IS AN AI startup with 24 employees, headquartered above a Popeyes in Brooklyn. It has raised $13 million to date—about 0.0072 percent of what OpenAI has—and was virtually unknown until earlier this year. But if Pangram is to be believed, it’s the one force standing in the way of a full machine takeover of the written word.

Pangram promises to detect how much AI was involved in the generation of any given text, and it does so as a best-guess percentage (using AI, of course). In January, speculation swirled on Reddit and YouTube that the author Mia Ballard may have used AI to write Shy Girl, a self-published novel picked up for traditional publication by Hachette. Although Ballard denied it, Pangram’s CEO posted on X that the book was 78 percent AI-generated. Hachette later canceled the release.

A barrage of similar accusations followed. The New York Times was called out for running an AI-generated installment of its Modern Love column. Pangram score: 100 percent. Then came a winner of the Commonwealth Short Story Prize (100 percent), the novel Daggermouth (60 percent), and a thriller titled Call Me, I’ll Hide the Body that had sold for $2.4 million (97 percent). In late July, Substack announced it was integrating Pangram into its platform so readers could quickly determine possible AI use.

Not all writers see this as a good thing. “There is such distaste and anger at the AI detection software,” says Jane Friedman, an author and publishing expert. “There’s this feeling like they are just as evil, if not more evil, than the AI companies themselves.” Outside of publishing and higher education, Pangram isn’t a household name, but I suspect it’s about to be; the demand to distinguish LLM churn from human writing grows with each passing day. The question to ask now is: How much can the company be trusted?

MAX SPERO, PANGRAM’S 30-year-old cofounder and CEO, dials into our Google Meet from his phone, takeout box in one hand and skyscrapers in the background. He’s wearing a beige T-shirt and has short dark hair, a boyish expression, and a wide smile. He says he’s rushing home with lunch and will call back. Ten minutes later, Spero reappears inside his Brooklyn apartment. He’s been on hiring calls all morning, he says. In July, Pangram raised $9 million and announced the launch of its newest model, Pangram 4. The company’s website lists six new job openings, which would increase headcount by 25 percent.

I ask Spero about growing up in California, a softball question that makes him visibly uncomfortable. Discussing the nuances of Pangram’s tech comes more naturally to him than answering personal queries—or speaking to the company’s involvement in literary scandals. Spero takes a bite of his meal and says he was raised in the Los Angeles suburb of La Crescenta. He loved programming and joined his school’s robotics team. As an undergrad at Stanford, he met Bradley Emi, his eventual Pangram cofounder. (My several emailed requests to interview Emi were ignored by the company’s comms team.) After school, Spero went to Google, where he worked on FLoC, a technology designed to replace third-party cookies by grouping Chrome users by interests. (Google killed off FLoC in 2022 after the product was met with privacy concerns.) Spero later worked for the autonomous car company Nuro, while Emi made his way through Tesla and AI biotech company Absci. After ChatGPT launched in 2022, the duo saw a business opportunity to confront a future filled with AI content. In 2023, they founded Checkfor.ai, then renamed the company Pangram a year later. By then, at least a dozen other companies were crowding the detection space, including Originality.ai, GPTZero, and Turnitin; Pangram performed well in some early, independent testing and emerged as a front-runner.

As our interview progresses, Spero’s responses seem sticky, stopping and starting, and not just because he’s eating. I ask about his hiring ethos. Spero murmurs “hmm” before turning away without apology to microwave his food. Fifteen long seconds pass in silence. He finally faces me again and says, “The average person is at Pangram because they care about the mission.”

To detect AI, Pangram uses a method called “synthetic mirroring” by which it takes human writing and has LLMs generate a close match. This teaches its model how AI writes. Pangram also uses “hard negative mining,” searching datasets for false positives that it can synthetically mirror and use to augment its training set—using mistakes to retrain the machine. “All of our datasets are properly licensed, which I think is kind of rare in the AI world today,” Spero says, though that’s partly because Pangram’s product is far less data-hungry than ChatGPT or Claude. “I don’t want to completely throw the AI companies under the bus,” he adds, “but I think they’ve lost a lot of trust, especially in the world of creatives.” Today, Pangram caters to several industries, including education, the legal field, and recruitment, but creative writing constitutes the largest segment of the training text.

The Shy Girl story put Pangram on the map. Spero had already been calling out suspected AI writing from his social accounts, so it wasn’t a surprise when, in January, he was tagged in a Reddit post and subsequently sent a PDF of Ballard’s manuscript. “I put it in [Pangram], I posted it, and then later The New York Times asked me for comment,” he says. “I think people overstate the importance of Pangram in the Shy Girl story.” Except that wasn’t exactly how it happened. Like a game of AI-scandal telephone, it was a Pangram account executive who discussed the story with a publishing industry analyst who, in turn, brought it to the Times.

Another twist: Critics of Pangram, including the investigative project The Drey Dossier, pointed out that Spero’s copy of the manuscript came from a pirating website. When I ask Spero about this, he says, “Yeah. And, like, yeah … it is what it is. I hadn’t looked too closely. I didn’t go read the whole PDF. I just put it straight into Pangram.”

Pangram hasn’t exactly shied away from controversy since then. After the Commonwealth Short Story Prize winner returned a high Pangram score, the company analyzed every winner since 2012, calling out three more potential AI uses. Still, Spero downplays Pangram’s impact, particularly on axed book deals. “Basically, with every book deal, to my knowledge, it hasn’t really been about the Pangram score,” he says. “That is a part of it, but if you talk to anyone involved, it is only a small, small part of the bigger picture.”

We move on to Pangram’s Substack integration. “People shouldn’t be afraid about disclosing this because your work should stand on its own as quality and something that people want to read, regardless of … ” Spero trails off. “Well, how do I want to put that?” He pauses. “I think I … oh, I know what I said the other day.” He gains confidence and his voice speeds up. “If the value of your work is dependent on deceiving the end user into thinking it wasn’t written by AI, then”—he hesitates again—“that’s going to be a problem.” (Later, I realize his intonation changed because he was quoting his own post on X.)

Substack would not confirm any specifics of its Pangram partnership structure or whether Substack pays the company per scan. “Substack’s philosophy is not anti-AI,” a representative tells me. “We simply believe you should know what you’re consuming.” Some authors are wary. There’s a sense, one tells me, that an entire career can be destroyed with the click of a button.

BOOK PUBLISHING, A notoriously slow industry, has been particularly slow to reckon with AI. Still, the reality is that AI is used by a growing subset of authors, publishers, and agents. Last year, Gotham Ghostwriters polled 1,481 working writers and found that 61 percent use AI tools, with 7 percent saying they’ve published AI-generated text. When Tuhin Chakrabarty, an assistant professor of computer science at Stony Brook University, used Pangram earlier this year to analyze 14,419 self-published novels, he found that nearly 20 percent had substantial AI-detection scores. His working paper has been widely cited, and that data was the original source for the Daggermouth accusations.

I reached out to the Big Five publishers to ask if they use AI detection. Simon & Schuster and HarperCollins declined to comment; Hachette and Macmillan did not respond. A spokesperson from Penguin Random House confirmed that its editors may use approved AI-detection tools “as one additional means of identifying potential AI-generated content,” but that such tools are “not determinative and are only one component of a broader editorial process.” Some agents are using detection too—and for every public execution of a book deal, others are being quietly killed offstage. “Normally, agents say nothing,” Friedman says.

I wanted Chakrabarty’s perspective. He first heard about Pangram from one of Spero’s posts in late 2024. After he signed up, Spero sent him API credits. The two have since become “close friends,” Chakrabarty tells me, and Pangram has continued to supply Chakrabarty with credits to support his research. Chakrabarty posts Pangram results on social media and regularly defends the company. He says he has met with publishers to talk about detection in the wake of AI scandals. “Pangram should not be the de facto judgment,” he says. “But I think your own discretion coupled with Pangram’s judgment cannot be wrong.”

Chakrabarty is also in a relationship with Todd Shuster, the co-CEO of Aevitas, a top New York literary agency. Chakrabarty encouraged Shuster to meet with Pangram. “It was instantly a very helpful tool,” Shuster tells me. “Right away, we started having to have difficult conversations [with authors] because Pangram was showing their works to be anything from 50 percent AI-written to 95 percent.” Shuster now uses Pangram to analyze manuscripts and book proposals. He also consults for Pangram, a role that includes making introductions between Pangram and publishers. When authors are confronted, Shuster says some are defensive or seem to fib, while others are honest about their AI use. Shuster says he has asked authors to rewrite work with their own voice and ideas.

Critics point to two major issues with Pangram: the damage done by false positives and the potential bias built into its machine learning. “I don’t think [AI detectors] work,” says Sam Illingworth, a professor of critical AI literacy at Edinburgh Napier University. “I think that detectors are prejudiced against certain people.” One study showed that detectors were more likely to identify work by non-native English writers as AI-generated, and neurodiverse writers have also claimed that their writing patterns are disproportionately flagged. (That study predated Pangram, and Spero points to internal research countering it.) The three novels mentioned earlier—Shy Girl, Daggermouth, and Call Me, easily publishing’s biggest AI-detection scandals—were all written by writers of color. (I reached out to these authors; two didn’t respond and one declined to comment, citing legal concerns.) “As an African American agent, I do think we really need to pay attention to some of the inequities within the industry,” says Regina Brooks, the president of the Association of American Literary Agents. Whose work gets scanned, and who gets publicly accused, is after all an inherently human process—and therefore vulnerable to bias.

In some cases where writers have claimed that their work was wrongfully flagged as AI-generated, Spero has offered a cash reward if they can prove they wrote it themselves. “I’ve offered the bounty in cases where I’m almost certain the person is lying,” he says. “Or, if they’re not lying, then it’s very valuable for us to know.” So far, he says, nobody has taken him up on it.

“Of course, you can fool Pangram,” Chakrabarty says. Some writers have tweaked text or written like AI to elicit false results. A new working paper from Notre Dame, titled “Why AI Detection Fails for Academic Integrity,” notes that Pangram’s 3.2 model flagged light AI editing on academic abstracts as AI writing 64 to 80 percent of the time. However, after putting AI-generated text through a humanizer—a tool that makes AI work seem more human—Pangram caught AI writing less than 4 percent of the time.

Pangram does disclose that its tool performs worse on shorter blocks of text, particularly anything under 100 words. This is a complication, considering that Pangram’s free service offers a limited number of credits per day, totaling about 2,000 words. The median input is 350 words. The company also concedes that an identical passage can get different verdicts depending on whether it’s scanned alone or within a larger context—which is exactly what might happen if a short excerpt is copy and pasted from a novel.

All told, Pangram claims false positives only happen 0.0041 percent of the time with its new model (an improvement over a previously reported rate of 0.01 percent). If anything, Spero says, Pangram is conservative. If text falls on the borderline, the tool errs on the side of declaring it human-written. “This is an intentional choice that we made,” Spero says. It’s partly responsible for the low false-positive rate—and means AI-generated work is more likely to slip under the radar. By Pangram’s measure, human essays that are heavily rewritten by AI are still detected as human-written 41.37 percent of the time.

DURING OUR INTERVIEW, Spero pulls up several social posts to illuminate points he wants to make, pasting them in our chat with such rapid speed that I wonder if he has his own AI system running in the background.

According to Rod Breslau, a journalist and former Pangram contractor, Spero is “personally online all day. He is going to reply to your tweet; he is going to reply to your post on Reddit. He’ll probably reply to your post on LinkedIn.” Breslau first approached Spero at the beginning of the year, offering his services as an “attack dog.” He was sick of seeing AI content flood social media. Breslau’s contract work for Pangram included consulting on its social media strategy and securing interviews for Spero, but he also scrolled the internet with the Pangram extension, publicly calling out suspected AI users. “I wanted to take a harsher approach because I thought that people were getting off too easy,” Breslau says.

Pangram parted ways with Breslau in part because it changed strategies—the company now predicts that AI use will become more accepted. Spero seems to have lost interest in chasing down just anyone who uses AI, but he’ll be vocal, he says, in cases where writers are being “deceptive.” He adds that Pangram will soon be able to flag instances where AI was used even for light editing. “That’s the road map,” Spero says. “Higher granularity, better detail, provide more information to the end users.” As far as I can tell, Pangram ultimately wants to be the go-to arbiter of originality for every sector: publishers, schools, and everyone else.

For now, Spero’s small team includes former educators and one former publishing professional. (A Pangram spokesperson also notes that a researcher has a degree in English literature.) There have been hints of defamation suits brewing from accused authors but, when asked about any lawsuits, Spero says, “I don’t believe anything has come to pass.” He plans to continue publishing technical reports on Pangram’s models and training. “Everybody else is sort of this black box, so I think our strategy here is really just build trust through transparency, be as open as possible, and engage with the community online as much as possible.”

As our conversation wraps, I reflect on the fact that Spero has come across as less than 100 percent engaged. Cagey, even. He roamed around his apartment throughout the interview. He took his laptop to sit near his living room, then to a window with a clutter of houseplants, then a different corner of his kitchen. At several points, his face floated halfway out of frame. He slipped on his headphones, then took them off again. At one point, while discussing model training, he gently burped.

I hadn’t expected this. Was he just busy and tired? Did he not take me seriously? Had the deluge of media coverage rendered interviews rote—or annoying? In the end, I could make assumptions, but there’s only so much anyone can ascertain from an hourlong interaction. Pangram aims to take all the nuances, variables, and unknowns from a piece of communication and return a number. Humans, of course, are far more ambiguous than all that.

Let us know what you think about this article. Submit a letter to the editor at mail@wired.com.