Compare commits

..

No commits in common. "218a65068a2d02ac1e3cb661a80504b112f6af80" and "4f8eb711f32bdeb070ced0aa81a7d0ff9b0bbd40" have entirely different histories.

430 changed files with 7518 additions and 9666 deletions

View File

@ -3,79 +3,6 @@
---
## 2026-06-24 (cont.) — R4 analyst mode: MITM-everything + media reverse-catcher + clone (#736)
New "R4" doctrine — visibility over performance. Delivered + live on gk2:
- **Splice flip**`tls-splice-seed.conf` reduced from a media-CDN perf list to
breakers-only (`api.anthropic.com`); splice now applied ONLY where MITM provably
breaks (cert pinning). Banner reaches every page; catcher sees media URLs. Live:
learned splices cleared, autolearn gated (`tls_splice=off`).
- **sbxmitm media reverse-catcher** (`cmd/sbxmitm/mediacatch.go`, toolbox-ng 0.1.20)
— 2xx MITM'd flows → cloneable media URLs (HLS/DASH manifests, direct A/V,
googlevideo videoplayback) appended to `/run/secubox/media-catch.jsonl` (URLs
only, deduped, atomic, fail-open). `--media-catch` default on; worker unit
`ReadWritePaths=/run/secubox`.
- **mediaflow Discovered Media + Clone** (2.1.0) — `/discovered`, `/clone`
(yt-dlp→ffmpeg queue, lazy worker for the aggregator), `/library`,
`/download/{id}`, DELETE; dashboard cards. Verified: HLS caught → ffmpeg →
464 MiB mp4 in library. yt-dlp installed.
- Also fixed the empty mediaflow dashboard (2.0.2 contract + 2.0.3 cumulative
services): cards/streams live, Top Media Services from DPI cumulative store.
KEY: dashboard routes via the **aggregator** (in-process import) — restart
`secubox-aggregator` to pick up mediaflow code changes.
- Phase 4 done — R4 button added to the banner topbar (R0..R4) + set-level + by-MAC
validation + analytics buckets; gated to the wg path like R3 (secubox-toolbox 2.7.20).
- yt-dlp upgraded 2023.03.04 → 2026.06.09 (standalone binary; YouTube works).
- Recos: catcher now captures YouTube watch **pages** (kind=page, toolbox-ng 0.1.22);
Discovered Media persisted off tmpfs into a durable capped store (mediaflow 2.1.1);
yt-dlp packaged (Recommends + weekly refresh timer + postinst).
- **Catch-log ownership bug**`/run/secubox/media-catch.jsonl` was created
`secubox`-owned while the worker runs as `secubox-toolbox`, so O_APPEND failed
silently → nothing captured. Fixed with a tmpfiles.d entry pre-creating it owned
by the writer every boot (zz-secubox-toolbox-ng.conf). Live: rm + worker recreate.
## 2026-06-24 (cont.) — Banner on nonce-CSP sites + Claude API splice + YouTube unblock (#728)
Three distinct root causes behind "no banner on youtube / news", fixed in order:
1. **Trusted Types** (0.1.17) — `require-trusted-types-for` blocked DOM injection. Stripped.
2. **Nonce-based CSP** (0.1.18) — the banner is *inlined* (service-worker-proof), but a CSP
nonce/hash makes `'unsafe-inline'` IGNORED → the bare inline `<script>` was silently
blocked. `relaxCSPForLoader` now **borrows the page's own nonce** and stamps it on the
injected `<script nonce=…>` (surgical: page CSP/nonces/hashes untouched), falling back to
forcing `unsafe-inline` (drop nonce/hash/strict-dynamic) only when there's no nonce.
Nonce validated to base64 charset (attribute-breakout guard). Threaded nonce through
injectIntoBody → injectHTML → injectInlineBanner. Tests rewritten for inline semantics.
3. **YouTube wholly blocked** (runtime) — autolearn false-positive put `youtube.com` in
`/var/lib/secubox/toolbox/learned-trackers.txt``Decide()` returned `block` (204) →
page never loaded. Removed from learned + added to `ad-allowlist.txt` (hot-reloaded).
Latent-bug tracker: **#735** (autolearn must not block apex/first-party nav targets).
**Claude API splice** (user request) — `api.anthropic.com` added to `tls-splice-seed.conf`
(+ live seed): cert-pinned Claude API/SDK clients reject the MITM CA, so pass them through;
`claude.ai` web stays MITM'd (browser trusts the CA → still gets the banner).
Verified end-to-end on gk2: YouTube 200 + banner nonce == page nonce; lemonde/lefigaro
banner via unsafe-inline fallback. DPI confirmed healthy — collector writes to
`/var/lib/secubox/dpi/` (state.json/cumulative.json fresh), `/exfil` returns categorized
flows; the earlier "empty" was me checking the wrong paths (`/run/secubox/dpi`).
## 2026-06-24 — DPI YouTube bannering: strip Trusted Types CSP (#728)
- **Root cause** — YouTube serves a standalone `Content-Security-Policy:
require-trusted-types-for 'script'` header. sbxmitm's `relaxCSPForLoader` already
relaxed `script-src` (drop `strict-dynamic`, add `'self'`/`'unsafe-inline'`) so the
banner loader runs, but Trusted Types still blocked the banner's DOM injection →
banner silently never mounted on YouTube.
- **Fix** (`cmd/sbxmitm/csp.go`, toolbox-ng 0.1.17) — drop `require-trusted-types-for`
and `trusted-types` directives during the relax; omit the resulting empty CSP header
line. Local Go unit tests cover both the relax and the empty-header drop.
- **DPI capture half** — collector `state.json` was stale (frozen 09:44); restarted
`secubox-dpi-flowcap` → fresh windows, YouTube/media flows now visible in mediaflow.
- Deployed to gk2; R3 workers `secubox-toolbox-ng-worker@1..4` restarted on 0.1.17.
- Filed for later: #729 wireguard peers/tabs, #730 yacy, #731 lyrion, #732 magicmirror,
#733 firewall dashboard misreport, #734 webui.conf hardcoded-route cleanup.
## 2026-06-22 — DPI exfil engine + Netrunner report (HTML+PDF) + sbxmitm fixes
Big session: full per-device DPI exfiltration pipeline, the kbin report reborn as a

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 60 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 54 KiB

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 71 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 42 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 15 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 42 KiB

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 36 KiB

After

Width:  |  Height:  |  Size: 47 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 47 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 47 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 38 KiB

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 52 KiB

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 30 KiB

After

Width:  |  Height:  |  Size: 31 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 40 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 35 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 44 KiB

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 53 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 47 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 33 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 40 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 38 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 50 KiB

After

Width:  |  Height:  |  Size: 42 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 40 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 31 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 32 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 28 KiB

After

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 25 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 51 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 29 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 39 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 49 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 29 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 45 KiB

After

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 43 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 27 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 48 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 36 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 49 KiB

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Some files were not shown because too many files have changed in this diff Show More