Anomaly Detection¶

has_anomalies / inspect_anomalies flag text that carries out-of-place characters disguising a real word — a cross-script homoglyph, a bidi-direction conflict, leet, a single-letter segmentation, a zero-width / bidi control, or zalgo. Like is_suspicious_hostname, the detector reports a technical fact and leaves the malicious-or-not judgement to the caller — it never claims intent.

Detected classes¶

Fifteen kinds fire. Thirteen need no lexicon: only leet and segmentation do.

The table below is grouped by kind, not by evaluation order. control is checked first, ahead of the ASCII fast-path, because NUL, ESC, BEL and DEL are themselves ASCII: a check placed after that fast-path would never see the vectors it exists for. The per-token branches after it split on !tok.is_ascii(): mixed_numbers, invisible, bidi, enclosing_mark, zalgo, duplicate_mark, bidi_mixed, mixed_script and compat_fold only run on non-ASCII tokens, leet and segmentation then run on everything, and confusable runs last, on non-ASCII tokens. deletion and smuggled are not per-token at all: they read the whole text.

Most branches are script-agnostic and port across writing systems. mixed_script is the exception — it is anchored on Latin, and fires on Latin combined with Cyrillic or Greek.

The bidi kind is a judgement, not a census (#778)¶

The row above spares LRM everywhere and RLM/ALM outside a number run, on purpose: a lone directional mark is ordinary in right-to-left text, and reporting one would fire on any page that uses it. So the kind answers 9 of the 12 UAX #9 controls.

When the question is the census rather than the judgement — a filename, an identifier, a source file, anywhere the caller has already decided their input should carry no bidi control at all — use has_bidi_control, which answers all twelve and applies no context:

from disarm import has_bidi_conflict, has_bidi_control, inspect_anomalies

assert has_bidi_control("\u200e")  # a mark counts
assert inspect_anomalies("\u200e").kinds == []  # and is deliberately not an anomaly

assert has_bidi_control("invoice\u202egpj.exe")  # so does an override
assert not has_bidi_conflict("invoice\u202egpj.exe")  # which reads letters, not controls

The three predicates are disjoint answers to different questions: has_bidi_control is the raw set, inspect_anomalies is the judged subset, and has_bidi_conflict reads strong-direction letters and is structurally blind to controls altogether.

Shapes other detectors answer differently (#1165)¶

A leading U+FEFF reports invisible. It is what a file read as utf-8 and not utf-8-sig begins with, and two other detectors read it as a byte order mark. disarm keeps reporting it because <BOM>paypal is not paypal to an exact match. Open a file with utf-8-sig if the mark is expected.

One private-use character beside a letter or on its own reports nothing (#812), because that is an icon-font glyph. One between two Latin letters of the same case reports, because pay + U+E000 + pal is a split word, and canonicalize returns paypal (#1212). A glued label changes case at the join, so Save + U+E001 + File stays clean, and so does Pay + U+E000 + Pal, where is_canonical still returns False. Between any other letters it takes eight, and a CJK letter on either side does not count, since legacy encodings for Han used this block. of + U+F001 + ce reports too, and that is correct, because canonicalize damages it to ofce.

A ZWJ at the end of a text, after an emoji, reports nothing. One joiner cannot spell a byte, and a truncated message leaves one by accident, so this is a broken emoji sequence and not a carrier. The same goes for a lone zero-width character beside an emoji, with no letter to split or hide. Beside a letter, both report.

U+200B between two digits reports nothing (#1204). The rule for one invisible character reads letters, so 12 U+200B 34 stays clean while ab U+200B cd and a1 U+200B 2b report. A digit group has no word to split, and UAX #14 makes U+200B a line-break opportunity, which a long number has no other way to carry. The cost falls on a filter that matches digits in raw text, since 4111 U+200B 1111 gets past it. Run such a filter on canonicalize output, which removes the character, or ask is_canonical, which returns False.

tests/test_rival_vectors.py holds every string other detectors' test suites state a verdict on, and says for each one whether disarm agrees, differs by decision, or differs with an issue open.

Levels (#946)¶

anomaly_severity(kind) gives the level a finding of that kind carries, by the name a report holds: "error" or "warning", which are SARIF's names. smuggled is the one "error", because it fires only on a run that decodes to readable text. Every other kind reports a technical fact and leaves the judgement to the caller, and that includes deletion, whose presence is not a matter of degree: a classic Mac OS file with CR line endings is a known false positive. The Node, Ruby, Java and C ABI bindings take the same name. The scanner's SARIF output reads these levels.

One verdict for every canonically equivalent spelling¶

\u00e9 is one code point in NFC and two in NFD, and a reader cannot tell them apart. The detector gives both spellings the same report: it classifies each token in its composed form (NFC), and the tests that ask for an ASCII letter read a letter through its canonical decomposition, so \u00e9 counts as the e it is. Before this, \u00e9t\u00e9 followed by an isolate was clean in NFC and bidi in NFD, and the NFC spelling, the one nearly all text arrives in, was the unreported one (found by the Lean model in formal/lean/Detection, Finding 3). tests/exhaustive_anomalies.rs checks the property over every Unicode scalar.

The exception is a character NFC replaces with a different one rather than composing: the canonical singletons, such as U+212A KELVIN SIGN (to K) and U+037E GREEK QUESTION MARK (to ;), and duplicate encodings such as U+1FEE. Those are not a spelling of the letters they stand for, they are different characters that look like them, so the detector judges them as spelled: \u212Aey reports compat_fold where Key reports nothing, and canonicalize rewrites it. Neither NFC nor NFD contains such a character, so any text's two normal forms still get one verdict. A caller who must refuse bytes that are not already normalized wants is_normalized or is_canonical, not this detector.

The leet and segmentation branches take a caller-supplied lexicon — a set of common words for the language being protected. The defining rule: a real leet attack substitutes a letter, whereas win32 carries a literal number that maps to no letter, so requiring every out-of-place character to be a real letter-substitution that yields a common word rejects the literals.

Usage¶

from disarm import has_anomalies, inspect_anomalies

words = {"free", "paypal"}

# leet: "fr33" decodes to "free"
assert has_anomalies("get fr33 now", words)
# a literal number is not a substitution, so "win32" is spared
assert not has_anomalies("the win32 api", words)

report = inspect_anomalies("log in to paypаl", {"paypal"})  # Cyrillic а
assert report.anomalous
assert report.kinds == ["mixed_script"]
assert report.findings[0].kind == "mixed_script"

use disarm::api::{self, AnomalyKind};
use std::collections::HashSet;

let words: HashSet<String> = ["free", "paypal"].iter().map(|s| s.to_string()).collect();

assert!(api::has_anomalies("get fr33 now", &words));
assert!(!api::has_anomalies("the win32 api", &words));

let report = api::inspect_anomalies("log in to paypаl", &words);
assert!(report.anomalous);
assert_eq!(report.kinds, vec![AnomalyKind::MixedScript]);

require "disarm"

# the lexicon is a common-word collection (Array or Set)
Disarm.has_anomalies?("get fr33 now", ["free"])  # => true
Disarm.has_anomalies?("the win32 api", ["free"]) # => false

Disarm.inspect_anomalies("log in to paypаl", ["paypal"])[:kinds] # => ["mixed_script"]

import { hasAnomalies, inspectAnomalies } from 'disarm'

// the lexicon is a Set or array of common words
hasAnomalies('get fr33 now', ['free'])  // => true
hasAnomalies('the win32 api', ['free']) // => false

inspectAnomalies('log in to paypаl', ['paypal']).kinds // => ['mixed_script']

The report¶

inspect_anomalies returns a report with anomalous, kinds (the anomaly kinds that fired, in first-appearance order), findings, and reason (the first finding's plain-language sentence). Each finding carries the offending kind, token, byte start/end span, detail (the codepoint, the scripts, or the decoded word), and its own reason.

A False result is not a safety guarantee — it means only that none of the fifteen kinds fired on the lexicon you supplied. Compose this with your own policy, as you would the hostname analysis.

Checking a transform at the seam¶

Run has_anomalies on the output of a transform and it tells you whether that transform left something behind. The check needs no new API and is the cheapest way to find out you picked the wrong function.

One input carrying three different hazards — a right-to-left override, a zero-width space and a Cyrillic ԁ standing in for Latin d:

from disarm import (
    canonicalize,
    has_anomalies,
    inspect_anomalies,
    ml_normalize,
    normalize_confusables,
)

hostile = "\u202eexample\u200b.com\u0501"

# canonicalize clears all three: nothing is left to report.
assert canonicalize(hostile) == "example.comd"
assert has_anomalies(canonicalize(hostile)) is False

# ml_normalize is not a security preset. The override survives, and the seam
# check is what tells you so.
assert has_anomalies(ml_normalize(hostile)) is True
assert inspect_anomalies(ml_normalize(hostile)).kinds == ["bidi"]

# normalize_confusables folds the homoglyph and nothing else.
assert inspect_anomalies(normalize_confusables(hostile)).kinds == ["invisible"]

The guidance only runs one way¶