Engineering Log: Can a Local LLM Replace a Cloud Model for Conceptual Security Triage?

Project: Secure AI Atlas — automated signal intelligence pipeline Period: June 2026 Status: ongoing, no production changes deployed

Summary

We ran a structured comparison between local open-weight models (Ollama, RTX 5070, 16GB VRAM) and DeepSeek’s cloud API for the conceptual-analysis stage of our security signal pipeline — the step that decides whether an incoming news item represents a genuine extension of our threat taxonomy (TR-* concepts) or should be filtered out. This log documents what we tried, what broke, why it broke, and what we fixed, including the failures that didn’t make it into a clean changelog.

The short version: local models are viable for literal evidence extraction and structured classification, but not yet for autonomous conceptual judgment. The interesting part isn’t that conclusion — it’s the specific failure modes we found along the way, several of which were bugs in our own validation logic, not in the models.

Baseline: Qwen3-14B vs. DeepSeek on conceptual analysis

Setup: 10 hand-curated signals (6 real security items across OWASP/MITRE ATLAS categories, 4 synthetic adversarial cases — irrelevant content, near-duplicates, and a fabricated concept with no supporting evidence).

Model	Contract compliance	Repairs needed	Avg. latency
Qwen3:14b	10/10	0	14.265s
DeepSeek-chat	10/10	0	6.169s

Both models hit 100% schema compliance — JSON Schema-constrained decoding via Ollama’s format parameter eliminated the malformed-output problem we’d seen earlier. The real divergence was semantic, not structural:

• Qwen fabricated evidence, mechanisms, and requirements not present in source text; confused taxonomy concept IDs; produced near-identical reasoning across distinct signals; and failed the adversarial test, accepting a fabricated concept (“latent agent resonance”) with no supporting evidence.
• DeepSeek grounded claims better, correctly rejected irrelevant signals, and correctly rejected the fabricated concept — but was overconfident on promotional/marketing-style summaries and over-assigned taxonomy concepts.

DeepSeek’s per-run cost for the 10-signal batch: $0.0198 (53,127 input tokens, 4,930 output tokens, official API pricing).

Initial recommendation: keep conceptual judgment on DeepSeek; restrict Qwen to conservative triage, never to taxonomy assignment or action decisions.

Detour: a context-window bug nearly invalidated the comparison

Before scaling to multiple local models, we discovered that Ollama’s default context window (2048-4096 tokens depending on model) was silently truncating our prompt. With the default window, qwen3.5:9b and gpt-oss:20b could only generate a single token; qwen3:14b’s input was also being clipped without any explicit error.

This matters because the original “Qwen fabricates wildly” result was partly an artifact of truncation, not solely a capability limitation. After setting num_ctx=8192 explicitly and re-running, the extreme hallucinations (fabricated thresholds like “1,000 decision nodes,” invented audit requirements) disappeared entirely. What remained was a more specific, more useful, and more honest failure mode: overconfidence and unjustified concept creation, not wholesale fabrication.

Lesson: when a local model’s output looks unreasonably bad, check the context window before concluding the model is unreasonably bad.

Three-model local comparison (corrected context)

With num_ctx=8192, think=false, and temperature=0.15 applied consistently:

Metric	Qwen3-14B	Qwen3.5-9B	GPT-OSS-20B
Schema-valid responses	10/10	6/10	10/10
Repairs attempted	0	4	0
Avg. latency	8.894s	9.972s	11.458s
Output tokens	4,698	8,690	2,294
Poor-evidence signals correctly flagged	0/4	0/4	1/4
Fabricated-concept test	Inconsistent	Rejected, with internal contradictions	Cleanly rejected

GPT-OSS-20B emerged as the strongest local candidate on grounding quality — no fabricated mechanisms were observed, and it cleanly rejected the adversarial concept. Qwen3.5-9B, despite the smallest parameter count and the largest context window (262K), failed schema compliance on 40% of responses: it consistently emitted conceptual_delta: null even after a repair pass, and conflated topical irrelevance with active contradiction of existing taxonomy — a category error that suggests a contract/semantic mismatch rather than a capability gap.

None reached a state we’d call decision_candidate. The remaining gap wasn’t primarily about what the models said — it was about whether their structured output was internally consistent.

Building deterministic invariants

Rather than relying on prompt engineering to make the model self-correct, we moved the consistency checks into code that runs after generation, before acceptance. The principle: some properties of the output must always hold, and we enforce them deterministically rather than hoping the model gets them right.

First pass — six invariants, applied to GPT-OSS-20B output:

Invariant	Mutations applied	Verdict
1. Normalize insufficient_evidence state	3	Correct
2. Empty arrays on no_conceptual_delta	2	Correct
3. Require literal citation for claimed impacts	18	Over-destructive
4. Require justification against existing taxonomy	1	Formally correct, weak heuristic
5. Cap confidence on promotional content	0	Ineffective
6. Verify contradiction claims	0	Not exercised (no contradictions generated)

Invariants 1 and 2 worked cleanly: they close exactly the kind of internal contradiction we saw earlier (a model claiming “insufficient evidence” while simultaneously proposing new taxonomy concepts and declaring no further evidence needed).

Invariants 3 and 5 revealed bugs in our own validation design, not in the model:

Invariant 3 failure: we validated the entire observed_evidence field (free-text interpretation) against the source, requiring near-literal match. This destroyed 18 of 18 legitimate impacts across four relevant signals — every one of them was a reasonable paraphrase of real source content, not a fabrication. Example: source text said “hijack agents”; the model wrote “malicious skills can take control beyond intended scope.” Semantically faithful, lexically non-literal. Our check couldn’t tell the difference, so it deleted real evidence at a 100% false-positive rate.

Invariant 5 failure: we capped confidence on promotional content unless the model itself reported evidence_sufficient=true and needs_full_article=false. The model reported exactly those flags on the promotional signals we were trying to catch — the rule was validating the model’s judgment using the model’s own judgment as the gate. It never fired.

Fixing the invariants

Fix for #3: split the contract into two fields — evidence_quote (a short literal fragment, verified against source) and observed_evidence (free interpretation, not validated). Only the former is checked for literalness.

Fix for #5: move the “promotional” flag out of model-generated fields entirely. It now comes from a deterministic signal computed at ingestion time (in this test, manually labeled to simulate that ingestion signal), independent of anything the model reports.

Re-running on the same 10 signals:

Metric	Before fix	After fix
Citations evaluated	18 paraphrases	19 explicit quotes
Citations verified	0	17
Impacts incorrectly deleted	18	2
False-positive reduction	—	88.9%
Poor-evidence signals meeting strict criteria	0/4	2/4
Promotional signals capped at confidence ≤0.4	0/2	2/2

The two remaining false positives are worth naming precisely: one was a defensible paraphrase that didn’t meet the literal-match threshold; the other was a sentence assembled from real source fragments recombined in a way the source never stated — closer to genuine confabulation than paraphrase, and a useful reminder that even a well-designed citation check has a residual failure mode. The clean fix isn’t a looser threshold — it’s having the model reference pre-extracted, pre-verified quotes by ID rather than generating quote text freely.

Where this stands

What’s confirmed:

• JSON Schema-constrained decoding solves format compliance reliably across model families.
• Literal evidence extraction (quote-level, not paragraph-level) is something a 9-14B local model does well when validated programmatically — this tracks with independent benchmark data (Vectara’s hallucination leaderboard places Qwen3-14B at 5.4% hallucination on short-document summarization/extraction, ahead of several larger models).
• Disabling “thinking” mode and lowering temperature for extraction-style tasks measurably reduces variance — this is consistent with published findings that reasoning mode increases hallucination specifically on summarization/extraction tasks, even as it helps on open-ended analysis.
• A binary triage gate using a local model as a hard filter is not safe: in a follow-up experiment, Qwen rejected a real, doctrinally relevant OWASP signal (memory/context poisoning, ASI06) before it ever reached DeepSeek — a single false negative with high editorial cost, for roughly $0.01 of API savings across 10 signals and a 147% increase in total pipeline latency.
• GPT-OSS-20B, with corrected invariants, is the strongest local candidate as a non-authoritative review_candidate — useful for generating drafts a human or a stronger model reviews, not for autonomous decisions.

What’s still open:

• A 10-signal (now 30-signal) dataset is not large enough to make production claims about precision/recall.
• Invariant 4 (justification against existing taxonomy) uses a shallow heuristic (checks for the presence of a TR-*-style identifier) that can be satisfied by inserting an ID without improving the actual reasoning.
• Invariant 6 (contradiction verification) has never been exercised — none of our test signals triggered a contradiction verdict, so the logic remains unvalidated until we build a dedicated adversarial set for it.
• We haven’t tested whether the same invariant framework generalizes to OWASP/ASI categories beyond the ones in the original 10-signal set (prompt injection, system prompt leakage, unbounded consumption, agent goal hijacking, and several others under the 2026 OWASP Agentic Top 10).

In progress: scaling to 30 signals

We’re currently re-running the corrected-invariant pipeline against an expanded 30-signal dataset, adding categories not covered in the original set and a wider range of adversarial cases: a fabricated concept written in convincing academic-paper language (testing whether sophistication of phrasing alone can bypass evidence requirements), a signal mixing one real and one fabricated concept in the same text (testing selective acceptance rather than all-or-nothing judgment), a signal using correct security terminology applied to the wrong context, and a signal with real technical substance presented through promotional framing (testing whether our deterministic “promotional” flag distinguishes degrees of marketing content rather than treating all press releases identically).

We’ll publish the results once that run completes — including, as has been the pattern in this project, whatever in our methodology turns out to be wrong.

---

This log reflects internal experimentation for Secure AI Atlas. No findings here have been deployed to production signal classification or publication decisions.