🛡️ TailGuard corpus report

WARNINGS training-corpus-v2 · TailGuard v0.1.0

Metric	Baseline	Current	Δ
Documents	1500	1450	—
Tokens	325426	232221	—
Vocabulary	27072	19935	—
Vocab / 10k tokens	831.9	858.4	+3.2% ✓
Type-token ratio	0.0832	0.0858	+3.1% ✓
Bigram diversity	0.5641	0.5112	-9.4% ⚠
Trigram diversity	0.8908	0.7921	-11.1% ⚠
Mean burstiness	0.457	0.449	-1.8% ✓
Near-duplicate rate	0.0	0.0	—
Mean synthetic score	16.2	15.5/100	-4.3% ✓
Flagged synthetic	0.1	0.1%	+0.0% ✓

Synthetic-score distribution

0–9

332

10–19

843

20–29

221

30–39

40–49

50–59

60–69

70–79

80–89

90–99

Most-repeated 5-grams (boilerplate check)

love love love love love
in this paper we propose
this paper we propose a
food food food food food
paper we propose a novel
state of the art methods
in terms of segmentation accuracy
with the help of her

Distribution-level corpus health, per "Contamination Detection Is Not Collapse Monitoring" — per-document detection misses collapse; these metrics catch it. habsburg-ai.vercel.app