Compare commits
No commits in common. "218a65068a2d02ac1e3cb661a80504b112f6af80" and "4f8eb711f32bdeb070ced0aa81a7d0ff9b0bbd40" have entirely different histories.
218a65068a
...
4f8eb711f3
|
|
@ -3,79 +3,6 @@
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 2026-06-24 (cont.) — R4 analyst mode: MITM-everything + media reverse-catcher + clone (#736)
|
|
||||||
|
|
||||||
New "R4" doctrine — visibility over performance. Delivered + live on gk2:
|
|
||||||
- **Splice flip** — `tls-splice-seed.conf` reduced from a media-CDN perf list to
|
|
||||||
breakers-only (`api.anthropic.com`); splice now applied ONLY where MITM provably
|
|
||||||
breaks (cert pinning). Banner reaches every page; catcher sees media URLs. Live:
|
|
||||||
learned splices cleared, autolearn gated (`tls_splice=off`).
|
|
||||||
- **sbxmitm media reverse-catcher** (`cmd/sbxmitm/mediacatch.go`, toolbox-ng 0.1.20)
|
|
||||||
— 2xx MITM'd flows → cloneable media URLs (HLS/DASH manifests, direct A/V,
|
|
||||||
googlevideo videoplayback) appended to `/run/secubox/media-catch.jsonl` (URLs
|
|
||||||
only, deduped, atomic, fail-open). `--media-catch` default on; worker unit
|
|
||||||
`ReadWritePaths=/run/secubox`.
|
|
||||||
- **mediaflow Discovered Media + Clone** (2.1.0) — `/discovered`, `/clone`
|
|
||||||
(yt-dlp→ffmpeg queue, lazy worker for the aggregator), `/library`,
|
|
||||||
`/download/{id}`, DELETE; dashboard cards. Verified: HLS caught → ffmpeg →
|
|
||||||
464 MiB mp4 in library. yt-dlp installed.
|
|
||||||
- Also fixed the empty mediaflow dashboard (2.0.2 contract + 2.0.3 cumulative
|
|
||||||
services): cards/streams live, Top Media Services from DPI cumulative store.
|
|
||||||
KEY: dashboard routes via the **aggregator** (in-process import) — restart
|
|
||||||
`secubox-aggregator` to pick up mediaflow code changes.
|
|
||||||
- Phase 4 done — R4 button added to the banner topbar (R0..R4) + set-level + by-MAC
|
|
||||||
validation + analytics buckets; gated to the wg path like R3 (secubox-toolbox 2.7.20).
|
|
||||||
- yt-dlp upgraded 2023.03.04 → 2026.06.09 (standalone binary; YouTube works).
|
|
||||||
- Recos: catcher now captures YouTube watch **pages** (kind=page, toolbox-ng 0.1.22);
|
|
||||||
Discovered Media persisted off tmpfs into a durable capped store (mediaflow 2.1.1);
|
|
||||||
yt-dlp packaged (Recommends + weekly refresh timer + postinst).
|
|
||||||
- **Catch-log ownership bug** — `/run/secubox/media-catch.jsonl` was created
|
|
||||||
`secubox`-owned while the worker runs as `secubox-toolbox`, so O_APPEND failed
|
|
||||||
silently → nothing captured. Fixed with a tmpfiles.d entry pre-creating it owned
|
|
||||||
by the writer every boot (zz-secubox-toolbox-ng.conf). Live: rm + worker recreate.
|
|
||||||
|
|
||||||
## 2026-06-24 (cont.) — Banner on nonce-CSP sites + Claude API splice + YouTube unblock (#728)
|
|
||||||
|
|
||||||
Three distinct root causes behind "no banner on youtube / news", fixed in order:
|
|
||||||
|
|
||||||
1. **Trusted Types** (0.1.17) — `require-trusted-types-for` blocked DOM injection. Stripped.
|
|
||||||
2. **Nonce-based CSP** (0.1.18) — the banner is *inlined* (service-worker-proof), but a CSP
|
|
||||||
nonce/hash makes `'unsafe-inline'` IGNORED → the bare inline `<script>` was silently
|
|
||||||
blocked. `relaxCSPForLoader` now **borrows the page's own nonce** and stamps it on the
|
|
||||||
injected `<script nonce=…>` (surgical: page CSP/nonces/hashes untouched), falling back to
|
|
||||||
forcing `unsafe-inline` (drop nonce/hash/strict-dynamic) only when there's no nonce.
|
|
||||||
Nonce validated to base64 charset (attribute-breakout guard). Threaded nonce through
|
|
||||||
injectIntoBody → injectHTML → injectInlineBanner. Tests rewritten for inline semantics.
|
|
||||||
3. **YouTube wholly blocked** (runtime) — autolearn false-positive put `youtube.com` in
|
|
||||||
`/var/lib/secubox/toolbox/learned-trackers.txt` → `Decide()` returned `block` (204) →
|
|
||||||
page never loaded. Removed from learned + added to `ad-allowlist.txt` (hot-reloaded).
|
|
||||||
Latent-bug tracker: **#735** (autolearn must not block apex/first-party nav targets).
|
|
||||||
|
|
||||||
**Claude API splice** (user request) — `api.anthropic.com` added to `tls-splice-seed.conf`
|
|
||||||
(+ live seed): cert-pinned Claude API/SDK clients reject the MITM CA, so pass them through;
|
|
||||||
`claude.ai` web stays MITM'd (browser trusts the CA → still gets the banner).
|
|
||||||
|
|
||||||
Verified end-to-end on gk2: YouTube 200 + banner nonce == page nonce; lemonde/lefigaro
|
|
||||||
banner via unsafe-inline fallback. DPI confirmed healthy — collector writes to
|
|
||||||
`/var/lib/secubox/dpi/` (state.json/cumulative.json fresh), `/exfil` returns categorized
|
|
||||||
flows; the earlier "empty" was me checking the wrong paths (`/run/secubox/dpi`).
|
|
||||||
|
|
||||||
## 2026-06-24 — DPI YouTube bannering: strip Trusted Types CSP (#728)
|
|
||||||
|
|
||||||
- **Root cause** — YouTube serves a standalone `Content-Security-Policy:
|
|
||||||
require-trusted-types-for 'script'` header. sbxmitm's `relaxCSPForLoader` already
|
|
||||||
relaxed `script-src` (drop `strict-dynamic`, add `'self'`/`'unsafe-inline'`) so the
|
|
||||||
banner loader runs, but Trusted Types still blocked the banner's DOM injection →
|
|
||||||
banner silently never mounted on YouTube.
|
|
||||||
- **Fix** (`cmd/sbxmitm/csp.go`, toolbox-ng 0.1.17) — drop `require-trusted-types-for`
|
|
||||||
and `trusted-types` directives during the relax; omit the resulting empty CSP header
|
|
||||||
line. Local Go unit tests cover both the relax and the empty-header drop.
|
|
||||||
- **DPI capture half** — collector `state.json` was stale (frozen 09:44); restarted
|
|
||||||
`secubox-dpi-flowcap` → fresh windows, YouTube/media flows now visible in mediaflow.
|
|
||||||
- Deployed to gk2; R3 workers `secubox-toolbox-ng-worker@1..4` restarted on 0.1.17.
|
|
||||||
- Filed for later: #729 wireguard peers/tabs, #730 yacy, #731 lyrion, #732 magicmirror,
|
|
||||||
#733 firewall dashboard misreport, #734 webui.conf hardcoded-route cleanup.
|
|
||||||
|
|
||||||
## 2026-06-22 — DPI exfil engine + Netrunner report (HTML+PDF) + sbxmitm fixes
|
## 2026-06-22 — DPI exfil engine + Netrunner report (HTML+PDF) + sbxmitm fixes
|
||||||
|
|
||||||
Big session: full per-device DPI exfiltration pipeline, the kbin report reborn as a
|
Big session: full per-device DPI exfiltration pipeline, the kbin report reborn as a
|
||||||
|
|
|
||||||
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 26 KiB |
|
Before Width: | Height: | Size: 31 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 34 KiB After Width: | Height: | Size: 35 KiB |
|
Before Width: | Height: | Size: 50 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 48 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 25 KiB |
|
Before Width: | Height: | Size: 53 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 60 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 31 KiB |
|
Before Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 32 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 34 KiB After Width: | Height: | Size: 28 KiB |
|
Before Width: | Height: | Size: 54 KiB After Width: | Height: | Size: 40 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 71 KiB |
|
Before Width: | Height: | Size: 32 KiB After Width: | Height: | Size: 51 KiB |
|
Before Width: | Height: | Size: 31 KiB After Width: | Height: | Size: 28 KiB |
|
Before Width: | Height: | Size: 42 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 15 KiB After Width: | Height: | Size: 31 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 29 KiB |
|
Before Width: | Height: | Size: 50 KiB After Width: | Height: | Size: 35 KiB |
|
Before Width: | Height: | Size: 42 KiB After Width: | Height: | Size: 27 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 36 KiB After Width: | Height: | Size: 47 KiB |
|
Before Width: | Height: | Size: 50 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 30 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 39 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 26 KiB |
|
Before Width: | Height: | Size: 43 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 40 KiB |
|
Before Width: | Height: | Size: 44 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 65 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 26 KiB |
|
Before Width: | Height: | Size: 47 KiB After Width: | Height: | Size: 31 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 31 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 28 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 26 KiB |
|
Before Width: | Height: | Size: 47 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 44 KiB After Width: | Height: | Size: 29 KiB |
|
Before Width: | Height: | Size: 38 KiB After Width: | Height: | Size: 26 KiB |
|
Before Width: | Height: | Size: 30 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 44 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 52 KiB After Width: | Height: | Size: 38 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 44 KiB After Width: | Height: | Size: 28 KiB |
|
Before Width: | Height: | Size: 30 KiB After Width: | Height: | Size: 31 KiB |
|
Before Width: | Height: | Size: 40 KiB After Width: | Height: | Size: 28 KiB |
|
Before Width: | Height: | Size: 34 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 35 KiB |
|
Before Width: | Height: | Size: 44 KiB After Width: | Height: | Size: 41 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 46 KiB |
|
Before Width: | Height: | Size: 53 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 30 KiB |
|
Before Width: | Height: | Size: 47 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 33 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 27 KiB |
|
Before Width: | Height: | Size: 40 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 38 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 38 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 50 KiB After Width: | Height: | Size: 42 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 40 KiB |
|
Before Width: | Height: | Size: 31 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 25 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 32 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 21 KiB |
|
Before Width: | Height: | Size: 28 KiB After Width: | Height: | Size: 33 KiB |
|
Before Width: | Height: | Size: 51 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 25 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 27 KiB |
|
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 30 KiB |
|
Before Width: | Height: | Size: 51 KiB |
|
Before Width: | Height: | Size: 29 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 39 KiB |
|
Before Width: | Height: | Size: 49 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 29 KiB |
|
Before Width: | Height: | Size: 48 KiB After Width: | Height: | Size: 34 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 48 KiB |
|
Before Width: | Height: | Size: 45 KiB After Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 37 KiB |
|
Before Width: | Height: | Size: 43 KiB |
|
Before Width: | Height: | Size: 16 KiB After Width: | Height: | Size: 27 KiB |
|
Before Width: | Height: | Size: 48 KiB |
|
Before Width: | Height: | Size: 36 KiB |
|
Before Width: | Height: | Size: 49 KiB After Width: | Height: | Size: 28 KiB |
|
Before Width: | Height: | Size: 46 KiB After Width: | Height: | Size: 44 KiB |