Building an Editorial Intelligence Pipeline: Signal Ingestion, Doctrinal Review and Publishing in Secure AI Atlas

Why this exists

Secure AI Atlas is not just a content site — it is an attempt to run an editorial process for a security-focused knowledge base the way a small, disciplined newsroom would, except that most of the newsroom is code. External signals come in (feeds, news, generic search), they get classified against a proprietary threat taxonomy (TR-*), a doctrinal review decides whether they change what the site claims to be true, and only then does anything get published.

The interesting engineering problem is not “call an LLM and publish the output.” It’s: how do you build a pipeline where a language model can participate in classification and drafting, without ever being the thing that silently decides what becomes public truth on a security portal?

This entry documents the current shape of that pipeline, the six structural flaws we found and fixed in a v2 audit, and what we learned running local models as a first-pass filter in front of a cloud model.

The seven scripts and the four risk bands

The system is organized as seven scripts, each with one job, connected through a shared STATE_DIR where every script writes to its own subdirectory and reads JSON contracts from the others. No script is allowed to write outside its lane.

01 atlas-intelligence-cycle   → external radar: ingest, classify, decide action
02 atlas-web-scan             → snapshot of what's actually published
03 atlas-conceptual-scan      → diff between the web and doctrinal truth
04 atlas-conceptual-review    → doctrinal decisions on concept candidates
05 atlas-source-review        → promotes/rejects candidate sources
06 atlas-editorial-apply      → the only script allowed to touch src/content
07 atlas-publish-release      → git push, social, email

What makes this more than a linear pipeline is that each script is assigned to one of four risk bands, and a script in a higher band can never fire automatically as a side effect of one in a lower band:

• Observation (web-scan, conceptual-scan) — read-only, fully autonomous, cron with no human in the loop.
• External intelligence (intelligence-cycle) — ingestion and classification run unattended; only the final “decide action” step touches the boundary of the next band.
• Doctrinal decision (conceptual-review, source-review) — decisions are generated automatically but applied only behind a human gate, during the current phase. A defined, objective exit criterion (30 days of reviewed history, <5% discrepancy rate, zero reverted approvals) is the only path to loosening that gate — and even then, low-confidence decisions still require a human.
• Public action (editorial-apply, publish-release) — always on-demand, always an explicit confirmation, no autonomy path defined at all in this version.

The orchestrator (atlas-run-all) enforces this: it will run the whole external-intelligence chain unattended, including the step that proposes creating content, but it will never chain that proposal into editorial-apply without an explicit --confirm-publish-chain --reason "...". That single rule is doing most of the safety work in the system — not model alignment, not prompt engineering, just “a script from a riskier band cannot be a side effect.”

Walking the chain

01 — atlas-intelligence-cycle is five internal jobs: ingest-signals (deduplicated RSS/Atom pull from registered sources, no classification) → agentic-review preliminary (keyword-based relevance filter, deterministic by default, optional LLM for finer severity/relevance) → generic-search (conditional — only runs when a signal needs more context, or weekly for source discovery, never as an open-ended crawl) → agentic-review final (merges context, assigns severity and confidence, flags doctrinal candidates) → decide-action.

decide-action is the piece that didn’t exist as a formal spec before v2. It mirrors the decision model of conceptual-review deliberately: six possible outcomes (create_content, update_content, flag_for_conceptual_review, watch, ignore, needs_human_review), built around the same governing principle — an external signal does not become published content directly. Its one non-negotiable rule: a high-severity signal from a source that hasn’t yet cleared source-review is forced to needs_human_review, full stop, even with the LLM enabled. That’s the same shape of rule as the conflict-resolution table in script 04 — the system keeps re-deriving the same instinct at every layer: unvalidated + high-impact = a human decides, no exceptions engineered around.

02 — atlas-web-scan is deliberately the most boring script in the system, and that’s the point. It walks the five content collections (blog, risks, controls, frameworks, learning-log), hashes each file, and classifies it new/changed/carried/removed against a persistent index. It never touches doctrine, never calls a model, never touches the network beyond the local filesystem. Its only real design challenge was distinguishing a legitimately empty first run from a lost index — solved by checking for prior run history before trusting an empty index, and refusing to silently rebuild without a warning.

03 — atlas-conceptual-scan compares the web snapshot against the truth repository and produces doctrinal signals (missing_concept, uncovered_doctrinal_tag, etc.). This is where the first of the six v2 fixes lives (see below): per-concept fingerprinting instead of one global hash.

04 — atlas-conceptual-review is the doctrinal gatekeeper. Its current implementation is conservative by design: it validates already-generated conceptual analysis against a strict quote_ids evidence contract, applies six deterministic invariants (insufficient evidence caps confidence at 0.4, no conceptual delta short-circuits, evidence without verifiable citation IDs gets stripped, promotional content is capped regardless of what the model says, contradictions require a literal truth proposition plus a literal incompatible claim), and queues everything for human review. It does not write truth, changelog, or content — that authority hasn’t been granted yet in this phase.

05 — atlas-source-review decides whether a candidate source graduates to the official registry, using a fully deterministic point score (HTTPS, category match, repeat appearances, historical score, feed health) against fixed thresholds. No model involved at all in the current implementation — the spec explicitly reserves room for one later, but with a hard rule: a model can never promote a source by itself.

06 — atlas-editorial-apply is the only script allowed to touch src/content, and it’s the convergence point for two independent decision chains — the external-intelligence chain’s decision.json and the doctrinal chain’s editorial-impact.json. This convergence is exactly where v1 had a silent bug (more below): nothing stopped both chains from targeting the same file in the same run.

07 — atlas-publish-release is the last gate: build validation, social draft, git push, Bluesky, email — each behind its own enable flag, with automatic git rollback if a later stage in the chain fails, and manual follow-up flagged for channels that can’t be unpublished (you can’t un-post to Bluesky).

The six structural flaws v2 fixed

An architecture audit surfaced six real gaps in v1, none of them exotic — all six were instances of the same underlying failure mode: the system assumed a case wouldn’t happen instead of specifying what happens when it does.

1. No priority rule when an internal signal and an external candidate point at the same concept with different evidence. Fixed with an explicit resolution table in script 04 (§15): exact duplicates merge, complementary evidence merges, contradictions force human review and block promotion — never silently resolved either way.

2. No merge rule in editorial-apply when both decision chains touch the same file. This was the most consequential gap, because it’s the one place the two chains actually meet. Fixed with a mandatory detect-overlap phase before any write: compatible actions (a content revision plus a tag addition) merge in order; two independent content revisions on the same file, or two independent “create this file” proposals, always fail the run rather than guess which one wins.

3. No doctrinal rollback mechanism. If an approved concept later turns out to be wrong, there was no defined path back. Fixed with a rollback_of changelog field and a reversal procedure that can generate revert_reference editorial actions.

4. Global truth fingerprint forced a full portal reanalysis on cosmetic changes. A single typo fix anywhere in the 26+-page truth document invalidated every page. Fixed with per-concept fingerprinting: only pages whose related TR-XXX concept actually changed get reanalyzed; a reformatting pass with zero doctrinal changes now triggers zero reanalysis.

5. Silent loss of the persistent index had no active detection, only a declared prohibition. Fixed with a check: if prior run history exists but the current index is empty or has dropped more than a configurable threshold (default 50%) without explanation, the run fails with index_loss_suspected instead of quietly doing a full rescan nobody asked for.

6. No circuit-breaker or timeout policy at the orchestrator level, only per-script timeouts. Fixed with a chain-level timeout, safe process termination (SIGTERM then SIGKILL), and — critically — the band-limit rule described above, which is really a circuit breaker for risk rather than for time.

What we’re still learning: local models as a triage layer

Part of the ongoing work is figuring out how much of the classification and review work a local model (running on an RTX 5070 via Ollama) can absorb, to preserve cloud quota (DeepSeek, and Codex for architectural work) for the cases that actually need it.

The benchmarking so far has produced a few durable conclusions, even without every number finalized yet:

• GPT-OSS:20b emerged as the strongest local candidate, but explicitly in a review_candidate role — a first-pass opinion, never an authority. Its output can escalate a decision but never override a rule-based one downward; a model cannot turn publish into watch or ignore, no matter how confident it claims to be. That asymmetry exists specifically so that manipulating or fooling a local model can’t be used to lower scrutiny on a signal, only to raise it.
• A hard binary gate (local model as a yes/no filter before DeepSeek) was tried and rejected after a costly false negative — a relevant signal the local model discarded outright never reached the cloud model at all. The design that replaced it is conditional escalation: specific trigger conditions (an invariant firing unexpectedly, a detected contradiction, a new concept candidate, a promotional signal with a non-null verdict, an invented concept_id, or local infrastructure being down) route the case to DeepSeek instead of trusting local judgment. The lesson generalizes: a local model is useful as evidence, risky as a gatekeeper.
• A silent infrastructure bug produced what looked like model failure. Ollama’s default context window (num_ctx) was truncating prompts without any warning, and the truncated input produced outputs that read like hallucinations. The fix wasn’t a better prompt or a better model — it was checking the context window configuration before attributing anything to model quality. This is probably the most transferable lesson from the whole exercise: when a local model’s output looks inexplicably bad, check the plumbing before you blame the model.

[pending: exact benchmark figures — discrepancy rates, false-negative counts, latency comparisons — to be filled in from the underlying benchmark reports]

What “human in the loop” actually means here

It would be easy to describe this system as “AI does the work, a human approves it,” but that undersells what’s actually enforced. The gate isn’t a checkbox at the end — it’s structural:

• A model can propose, but a rule always decides whether that proposal is even eligible to apply (source validation status, evidence contract compliance, invariant checks).
• Two independent decision chains are never allowed to silently overwrite each other’s work on the same file.
• Nothing crosses from “doctrinal decision” to “public action” without an explicit, reasoned, logged confirmation — not a default, not a timeout, not a high-confidence score alone.
• Vocabulary has to match across scripts (type, target, priority) specifically so that the overlap-detection logic between two independently-run chains can actually compare their outputs at all — a governance requirement expressed as a data-contract requirement.

That last point turned out to matter more than it sounds: if the external-intelligence chain and the doctrinal chain used different words for the same kind of action, the entire detect-overlap phase in script 06 — the one piece of the system built specifically to prevent silent content collisions — simply couldn’t function. Terminology consistency isn’t a style preference in this system; it’s load-bearing.

Open threads

• Finishing the triage/mapping layer between classified TR-X signals and catalogue actions (matched_entry_id | new_candidate | needs_human_review), with a deliberately higher bar for creating a new catalogue entry than for updating an existing one, and mandatory human approval for anything genuinely new.
• Closing the three remaining gaps in script 01: enforcing needs_human_review for unvalidated high-severity sources in code (not just spec), implementing merge_model_review_with_rules(), and retiring the legacy action vocabulary (draft_blog, update_frameworks) that currently prevents detect-overlap from seeing intelligence-cycle decisions correctly.
• Publishing the local-vs-cloud triage methodology itself here in the Learning Log, including the failures — the false negative from the binary gate and the context-window bug are more useful to a technical reader than a clean success story would be.

The throughline across all of this: incrementality, explicit failure modes instead of assumed absence of edge cases, and a hard separation between detection, decision, editing, and publication that no amount of model capability is allowed to collapse back into one step.