DOC-foundry: Adversarial Agents That Prove Sigma Rules Before You Ship Them

DOC-foundry showed up on Hacker News on November 13, 2026 with a sharp pitch: models can draft Sigma rules, but they cannot hand you a proof that the rule works, an adversary that provably evades the weak version, and an honest record of how the rule hardened until it held. The GitHub repository ships two pipelines — one that proves coverage with a deterministic engine, and one that reviews like a SOC workflow — on top of LangGraph, Pydantic, and FastMCP.

If you build detection-as-code or automate purple-team work with coding agents, this project is worth a weekend install. It is early — roughly two stars when explainx.ai cloned it — but the architecture is unusually explicit about agents propose, engines prove, the same split security teams need as ARTEX-style offensive agents scale up parallel probing.

TL;DR

Agents that act on your behalf need rules. See how explainx helps secure your agents →

Why detection engineers care now

Sigma rules are portable YAML, but production quality still depends on telemetry shape, benign baselines, and evasion creativity. Teams already feel pressure from two directions:

1. Offensive automation — LLM pentest orchestrators iterate exploits faster than humans can retune detections, a pattern FBI FLASH messaging on ARTEX and Cloudflare's 30× traffic spike both describe.
2. Generative drafting — models happily emit Sigma syntax that looks authoritative yet fails on rename tricks, shell obfuscation, or parent-process blind spots.

DOC-foundry's response is not another chat wrapper. It runs a self-repair loop where Blue re-authors after a brittle or broken verdict, while Red commits to two documented attacks in round one and re-throws them every round so goalposts do not drift mid-run.

Green and gray shields facing off, illustrating DOC-foundry Blue vs Red adversarial detection loop

That fixed-adversary discipline mirrors how explainx.ai treats benchmark claims elsewhere: bounded tests, explicit labels, no silent passes when the engine cannot evaluate a construct.

Architecture: propose vs prove

The classic graph, documented in the README, looks like this in prose:

1. Ingest → extract → telemetry — Threat Analyst normalizes input; Telemetry agent fixes the test set.
2. Blue author — Detection Engineer drafts behavior plus command-line intent, not one brittle string match.
3. Evaluate → battery → red_propose → red_verify → verdict — Deterministic referee scores FP/FN, runs mutation battery, and only counts Red bypasses the engine confirms.
4. Repair or exit — If verdict is not real-coverage and rounds remain, Blue gets concrete evasions and false positives as evidence.
5. Kartikeya gate → bundle — Governance scores blast radius times capability delta; egress needs a human.

Four LLM personas live in src/agents.py: Threat Analyst, Blue Detection Engineer, Telemetry, and Red-Team Adversary. Everything that decides truth — matching, adversarial mutations, coverage labels — sits in engine.py, evaluator.py, and graph.py with same input → same verdict → same hash.

What the engine refuses to fake

The README's honesty contract matters for builders who have been burned by evaluators that silently ignore hard clauses. engine.py raises on constructs it cannot truly evaluate — CIDR filters, aggregations, invented |in operators — rather than scoring them as covered. An un-evaluable rule costs a broken round, and Blue gets the engine's error text as repair evidence.

That is the opposite of vibe-based "LGTM" reviews and closer to how serious teams treat schema validation and default deny on APIs.

Green probe pressing a crack in a shell, illustrating Red-team evasion testing against Sigma rules

The two pipelines: proof vs stakeholder review

DOC-foundry runs two workflows with the same cast but different deliverables. Confusing them would mislead your SOC.

Classic proof loop (default for python src/run.py)

Outputs land in Outputs/<rule-slug>/ as:

• rule.yml — Sigma artifact that survived adversarial rounds
• report.md — Round-by-round convergence narrative
• proof.json — Positive corpus, benign baseline, adversarial mutations, coverage label (real-coverage / brittle / broken), FP/FN numbers, reproducibility hash

The canned certutil T1105 fixture demonstrates a honest two-round arc: round one is brittle after rename bypass; round two reaches real-coverage after Blue anchors on rename-resistant fields. That offline path is what CI should gate on.

Review chain (--chain and default Web UI)

The chain asks what would Red Team and SOC Lead say, then ships final-rule.yml, review-history.md, and review.json. The README states plainly: this path does not invoke the proof core, never executes telemetry, and issues no coverage verdict.

The Joomla iCagenda CVE-2026-48939 example is the doc to read first. Red Team pushes Blue to broaden against evasion; SOC Lead pushes back on noise and triage fields. The tension is the product: hardening for evasion raises alert volume.

Run the UI with:

pip install -r requirements.txt
python -m uvicorn webui.server:app --port 8000

FastAPI streams node events and token events over SSE. Cached replays are labeled REPLAY (cached) so demos cannot masquerade as live inference — a small integrity choice that matters when executives watch agent consoles.

Grounding: RAG without touching the hash

src/knowledge_base.py plus rule_corpus/ provide offline BM25 retrieval:

• Blue authors ground in SigmaHQ-style peer rules
• Red attacks ground in MITRE ATT&CK defense evasion and LOLBAS

Retrieval grounds proposals only; it never enters the reproducibility hash. That separation keeps audits clean: creative text may drift; verdict math may not.

docs/OSS-PARTNERS.md maps future integrations — Atomic Red Team for positives, EVTX-ATTACK-SAMPLES for benign baselines, pySigma for multi-SIEM conversion. The SigmaRuleArtifact.backends slot already exists for backends you have not wired yet.

How to run it today

From the repository root:

pip install -r requirements.txt

# Offline proof demo (no API key)
python src/run.py --threat fixtures/certutil_t1105.json

# Deterministic test suite
python src/run.py --evals

# Live threat with Claude
export ANTHROPIC_API_KEY=...
python src/run.py --claude "CVE-2026-45659 SharePoint deserialization RCE"

# Positive corpus from Atomic Red Team commands
python src/run.py --corpus atomic

# Human approval before deploy gate
python src/run.py --approve alice

Governed deploy lives behind FastMCP in src/mcp_server.py: the graph cannot fetch arbitrary URLs or push YAML to production without a token bound to the bundle hash.

Kartikeya gate: blast radius as code

src/governance.py implements Kartikeya's Gate as Blast Radius × Capability Delta. Structural invariants include:

• Egress always requires a human
• Non-real coverage floors blast radius

That is philosophical naming, but the implementation detail matters for teams deploying agent harnesses next to SIEM credentials. Least privilege belongs in code, not prompt prayers — the same lesson as MCP hardening patterns and NIST, NSA, and CISA agent control practices on explainx.ai.

What people are asking

Is this replacing my detection engineering team?

No. The README defines real-coverage as a validated first-version baseline with a fixed adversary, not forever bulletproof coverage. Your team still extends corpora, tunes for volume, and maps to Splunk, Elastic, or Sentinel via pySigma paths you own.

How does this compare to OpenAI's red-team factories?

OpenAI's Daybreak and cyber model work targets model robustness and exploit reasoning. DOC-foundry targets detection YAML against your telemetry category for one log source per run — a narrower, SOC-shaped problem. They can complement each other if you keep secrets off agent context windows.

Can Red "win" by refusing the adversary role?

Yes. Safety-tuned models sometimes decline. The UI records refusals, and graph.node_verdict refuses to award real-coverage when nobody attacked. "No evasions found" and "adversary never ran" must not look alike — another honesty seam worth copying in internal agent evals.

Why only one log source per run?

Majority vote picks a single telemetry category so Blue cannot overfit labels it never sees. A rule keyed on TargetFilename against process-creation events auto-fails — recall zero for the right reason.

Limits and licensing

• Early repo maturity — Few stars and three commits when indexed; treat APIs as volatile until releases exist.
• License — Read LICENSE and NOTICE.md in the repo before production use; explainx.ai does not render legal advice.
• Keys and data — Live runs send threat text to Anthropic or Google APIs; run fixtures locally first.
• SIEM deploy — Proof does not equal change management; pair with your existing detection-as-code pipeline, similar to practices in the AISOC self-host guide.

What this means if you ship agentic security tools

DOC-foundry is a template for governed agent loops in security:

If you are comparing offensive and defensive agent stacks, read it beside the OpenAI Codex Security CLI for code scanning and the November 2026 cyberattacks tracker for agentic pentest scale — offense and detection authoring are merging, but proof artifacts are still rare on the defensive side.

Related reading

• Cloudflare Modern Security Model vs ARTEX traffic
• FBI FLASH on agentic AI cyberattacks
• OpenAI Codex Security CLI
• AISOC open-source security operations center
• What is MCP — model context protocol
• Loop engineering for coding agents
• Check Point ARTEX autopilot pentest research

Repository details, dependency versions, and model names are accurate as of November 14, 2026 per the DOC-foundry README.