Once popular for attacking AI, ASCII smuggling is embraced by spammers - Ars Technica

A once-overlooked block of unicode that’s invisible to humans is gaining ever wider use.

A clever technique used to hide malicious prompts in attacks on AI agents has been adopted by spammers to evade filters on email platforms that are designed to flag unwanted messages used in mass campaigns.

The technique is broadly known as ASCII smuggling. It gained attention two years ago as a means of making a class of AI attack known as prompt injections more stealthy. Malicious instructions embedded in emails or other untrusted content to be processed by an LLM aren’t written in ordinary text. Instead, they’re rendered by a special range of Unicode tags. For example, the tag point U+E0041 mirrors “A,” and U+E0061 mirrors “a.”

The block of 128 tags mimics a portion of the American Standard Code for Information Interchange almost perfectly, with one major difference: the characters they encode are readable by computers but, by design, are almost completely invisible to humans. By expressing the malicious prompts in these tags, LLMs detect the instructions, but people reading the email never see them. There’s much more about ASCII smuggling here.

Earlier this year, Microsoft started seeing a massive increase in spam messages that used the technique. Beginning on one day in early February, the number of ASCII smuggling signatures detected by Microsoft Defender for Office spiked from roughly 21,000 per day to more than 1.3 million. Within four days, signature detections jumped to 2.5 million. The deluge persisted for months and then fell off sharply in mid-May.

“Because tag characters are invisible to humans but exist at the text-processing level, the same property that makes them useful for smuggling instructions into a model also makes them useful for obfuscating keywords before a detector evaluates them,” Microsoft explained Thursday. “The intent is inverted, but the mechanism is similar, and a user’s suspicions are not raised.”

Spammers are embedding Unicode in an attempt to evade filters that search for text, such as dollar amounts and the words “credit” and “term” that are commonly found in their mass emails. By sprinkling the invisible text into the middle of the word “funding,” for example, filters may read the words “fun” and “ding” instead. The receiver, meanwhile, sees the word “funding.”

Using special text to camouflage certain trigger words isn’t new. Spammers have used zero-width spaces and non-breaking spaces for decades to achieve similar results. The characters can thwart searches matching a literal string and alter the byte sequence that regex filters hunt for. The spammers likely adopted the hidden Unicode tags because some spam filters had yet to be programmed to detect them. A bigger likely reason for its use is to counteract the advantages made possible by machine learning (ML) and natural language processing (NL) LLMs for use in spam detection.

The bigger prize for the attacker, though, is not preventing the literal string matches; it is the ML- and NLP-based models that increasingly drive modern spam and phishing classification. Unless a filtering system takes a picture of a message and does OCR extraction over the visual image, it may miss this type of attack. A standard email classifier may not reason over whole words exactly as a human sees them; for efficiency, they can first split text into tokens or sub-word pieces. A clean lure term such as funding may be represented as a familiar token or a familiar sequence of sub-tokens. Insert an invisible U+E0020 into the middle, however, and the tokenizer may no longer see that same familiar unit. It might split the text into fun, an unexpected tag character, and ding; it might emit rare or unknown sub-tokens; or, if normalization runs first, it simply removes the U+E0020 character, leaving funding.

Thursday’s post provided guidance on ways developers can program filters to better account for ASCII smuggling in spam.