mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-06-29 07:08:34 +00:00
Compare commits
12 Commits
a76da4f783
...
740cbd291f
| Author | SHA1 | Date | |
|---|---|---|---|
| 740cbd291f | |||
| d203b9aa8f | |||
| 52d358c9d8 | |||
| 20fca011a1 | |||
| fb90349670 | |||
| 90a7df6f4b | |||
| f997d1c9a9 | |||
| 24fa9da107 | |||
| ff439d9395 | |||
| 6100a3e8ed | |||
| 3473320ad0 | |||
| 3e8b3e80fd |
307
docs/superpowers/plans/2026-06-27-ads-aggregate-mitm-stats.md
Normal file
307
docs/superpowers/plans/2026-06-27-ads-aggregate-mitm-stats.md
Normal file
|
|
@ -0,0 +1,307 @@
|
||||||
|
# Aggregate MITM protection stats in the #ads card — Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax.
|
||||||
|
|
||||||
|
**Goal:** Turn the #ads card's single narrow "blocked" number into an honest labeled breakdown of the full MITM protection: ad-block 204s (existing) + trackers detected (social_edges) + pages cosmetically cleaned (new sbxmitm counter) + network drops (blacklist nft).
|
||||||
|
|
||||||
|
**Architecture:** Task 1 is Python-only (trackers + network_drops added to the ad-stats payload + a WebUI breakdown) — immediate value, no engine redeploy. Task 2 adds a Go `cosmeticPages` counter in sbxmitm, flushed via the existing ad-event channel, stored, and surfaced.
|
||||||
|
|
||||||
|
**Tech Stack:** Python/FastAPI + sqlite3 (toolbox), Go (sbxmitm), vanilla JS (toolbox WebUI), pytest + `go test`.
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
- New Python files carry `# SPDX-License-Identifier: LicenseRef-CMSD-1.0`. New Go logic keeps the file's existing header.
|
||||||
|
- **Honest breakdown, not a sum:** the four metrics are different units (blocks / trackers / pages / drops) — label them separately; never add them into one number.
|
||||||
|
- Reuse existing helpers: `store._conn()`, the `social_edges` table (same toolbox.db), the existing ad-event flush (`adstats.go` payload + `/__toolbox/ad-event` handler), the blacklist nft drops parse (api.py `admin_blacklist`).
|
||||||
|
- `trackers_seen` = `COUNT(DISTINCT cookie_id_hash)` over `social_edges` in the window (exclude empty cookie ids).
|
||||||
|
- Commits reference `(ref #755)`. No "Claude Code"/"Generated with" strings.
|
||||||
|
- Tests: `cd packages/secubox-toolbox && python -m pytest tests/<file> -v` ; Go: `cd packages/secubox-toolbox-ng && GOFLAGS=-mod=vendor go test ./cmd/sbxmitm/ -count=1`.
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
- Modify `packages/secubox-toolbox/secubox_toolbox/store.py` — `ad_stats` adds `trackers_seen` + `pages_cleaned`; new `record_cosmetic_pages`.
|
||||||
|
- Modify `packages/secubox-toolbox/secubox_toolbox/api.py` — `admin_ad_stats` adds `network_drops`; `toolbox_ad_event` ingests `cosmetic_pages`.
|
||||||
|
- Modify `packages/secubox-toolbox/www/toolbox/index.html` — the #ads card breakdown.
|
||||||
|
- Modify `packages/secubox-toolbox-ng/cmd/sbxmitm/adstats.go` + `main.go` — the `cosmeticPages` counter + flush.
|
||||||
|
- Tests: `packages/secubox-toolbox/tests/test_ads_aggregate.py`, `packages/secubox-toolbox-ng/cmd/sbxmitm/cosmetic_count_test.go`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Python — trackers_seen + network_drops + WebUI breakdown
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `packages/secubox-toolbox/secubox_toolbox/store.py` (`ad_stats`)
|
||||||
|
- Modify: `packages/secubox-toolbox/secubox_toolbox/api.py` (`admin_ad_stats`)
|
||||||
|
- Modify: `packages/secubox-toolbox/www/toolbox/index.html` (#ads card)
|
||||||
|
- Test: `packages/secubox-toolbox/tests/test_ads_aggregate.py`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Produces: `store.ad_stats(...)` dict gains `trackers_seen: int` (and `pages_cleaned: int`, defaulted 0 here — Task 2 fills it); `admin_ad_stats(...)` dict gains `network_drops: int`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing test**
|
||||||
|
|
||||||
|
Create `packages/secubox-toolbox/tests/test_ads_aggregate.py`:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
"""Tests for the #ads aggregate breakdown (ref #755)."""
|
||||||
|
import sqlite3
|
||||||
|
import time
|
||||||
|
from secubox_toolbox import store
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_db(tmp_path, monkeypatch):
|
||||||
|
db = tmp_path / "toolbox.db"
|
||||||
|
c = sqlite3.connect(str(db))
|
||||||
|
c.executescript(
|
||||||
|
"CREATE TABLE ad_block_stats(ad_host TEXT, site TEXT, action TEXT, hits INTEGER, bytes INTEGER, last_seen REAL, PRIMARY KEY(ad_host,site,action));"
|
||||||
|
"CREATE TABLE ad_block_client_host(mac_hash TEXT, ad_host TEXT, hits INTEGER, last_seen REAL, PRIMARY KEY(mac_hash,ad_host));"
|
||||||
|
"CREATE TABLE social_edges(ts INTEGER, client_mac_hash TEXT, src_site TEXT, tracker_domain TEXT, cookie_id_hash TEXT, ja4_hash TEXT, consent_state TEXT);"
|
||||||
|
)
|
||||||
|
now = int(time.time())
|
||||||
|
# two distinct cookie-trackers in window, one duplicate, one stale (>24h)
|
||||||
|
for cid, ts in [("A", now-60), ("A", now-30), ("B", now-60), ("C", now-90000), ("", now-10)]:
|
||||||
|
c.execute("INSERT INTO social_edges(ts,client_mac_hash,src_site,tracker_domain,cookie_id_hash,ja4_hash,consent_state) VALUES(?,?,?,?,?,?,?)",
|
||||||
|
(ts, "m", "s", "t", cid, "j", "none_seen"))
|
||||||
|
c.commit(); c.close()
|
||||||
|
monkeypatch.setattr(store, "DB_PATH", db)
|
||||||
|
return db
|
||||||
|
|
||||||
|
|
||||||
|
def test_ad_stats_trackers_seen_distinct_in_window(tmp_path, monkeypatch):
|
||||||
|
_seed_db(tmp_path, monkeypatch)
|
||||||
|
out = store.ad_stats(hours=24)
|
||||||
|
# distinct non-empty cookie ids in the last 24h = {A, B}; C is stale, "" excluded
|
||||||
|
assert out["trackers_seen"] == 2
|
||||||
|
assert out["pages_cleaned"] == 0 # no cosmetic_events table yet → 0 (Task 2 fills it)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run it to verify it fails**
|
||||||
|
|
||||||
|
Run: `cd packages/secubox-toolbox && python -m pytest tests/test_ads_aggregate.py -v`
|
||||||
|
Expected: FAIL — `KeyError: 'trackers_seen'`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Implement store.ad_stats additions**
|
||||||
|
|
||||||
|
In `store.py`, in `ad_stats`, inside the `with _conn() as c:` block (after the existing `top_visitors` query, before the function returns `out`), add:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# #755 — trackers detected/poisoned by the MITM in the window: distinct
|
||||||
|
# cross-site cookie-identifier hashes seen on social_edges. This is the
|
||||||
|
# "Trackers" half of the card (the 204 ad-block is the "pubs" half).
|
||||||
|
try:
|
||||||
|
r = c.execute(
|
||||||
|
"SELECT COUNT(DISTINCT cookie_id_hash) FROM social_edges "
|
||||||
|
"WHERE last_seen IS NULL AND 0", # placeholder replaced below
|
||||||
|
).fetchone()
|
||||||
|
except sqlite3.Error:
|
||||||
|
r = None
|
||||||
|
out["trackers_seen"] = 0
|
||||||
|
try:
|
||||||
|
out["trackers_seen"] = int(c.execute(
|
||||||
|
"SELECT COUNT(DISTINCT cookie_id_hash) FROM social_edges "
|
||||||
|
"WHERE ts >= ? AND cookie_id_hash IS NOT NULL AND cookie_id_hash <> ''",
|
||||||
|
(int(cutoff),),
|
||||||
|
).fetchone()[0] or 0)
|
||||||
|
except sqlite3.Error:
|
||||||
|
out["trackers_seen"] = 0
|
||||||
|
# #755 — pages where the cosmetic ad-hide style was injected (Task 2 writes
|
||||||
|
# cosmetic_events; absent table → 0).
|
||||||
|
out["pages_cleaned"] = 0
|
||||||
|
try:
|
||||||
|
out["pages_cleaned"] = int(c.execute(
|
||||||
|
"SELECT COALESCE(SUM(pages),0) FROM cosmetic_events WHERE ts >= ?",
|
||||||
|
(cutoff,),
|
||||||
|
).fetchone()[0] or 0)
|
||||||
|
except sqlite3.Error:
|
||||||
|
out["pages_cleaned"] = 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Remove the dead placeholder block (the first `try/except` with `WHERE last_seen IS NULL AND 0`) — it was only to show the shape; keep ONLY the two real queries (`trackers_seen` and `pages_cleaned`). (`cutoff` is the existing local `cutoff = time.time() - hours*3600` already computed at the top of `ad_stats`.)
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run the test to verify it passes**
|
||||||
|
|
||||||
|
Run: `cd packages/secubox-toolbox && python -m pytest tests/test_ads_aggregate.py -v`
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 5: Add network_drops to the endpoint**
|
||||||
|
|
||||||
|
In `api.py`, change `admin_ad_stats` (currently `return store.ad_stats(hours=h)`) to:
|
||||||
|
|
||||||
|
```python
|
||||||
|
async def admin_ad_stats(hours: int = 24) -> dict:
|
||||||
|
"""Contextual ad-block metrics for the #ads tab (read-only, kbin-safe)."""
|
||||||
|
h = max(1, min(int(hours if hours is not None else 24), 168))
|
||||||
|
out = store.ad_stats(hours=h)
|
||||||
|
# #755 — network-layer drops (blacklist nft sets). Best-effort; 0 when the
|
||||||
|
# blacklist is inert or unreadable. Reuses the admin_blacklist parse.
|
||||||
|
try:
|
||||||
|
bl = await admin_blacklist()
|
||||||
|
out["network_drops"] = int(bl.get("drops", 0) or 0)
|
||||||
|
except Exception:
|
||||||
|
out["network_drops"] = 0
|
||||||
|
return out
|
||||||
|
```
|
||||||
|
|
||||||
|
(Confirm `admin_blacklist` is defined ABOVE `admin_ad_stats` in api.py; it is referenced at module scope so definition order at call time is fine since both are coroutines resolved at runtime.)
|
||||||
|
|
||||||
|
- [ ] **Step 6: WebUI — render the breakdown**
|
||||||
|
|
||||||
|
In `packages/secubox-toolbox/www/toolbox/index.html`, find the line building the #ads KPI (around line 620, the `kpi.innerHTML = ...` that shows `Trackers & pubs bloqués ${d.total_blocked}`). Replace that assignment with a labeled breakdown that keeps the existing "pubs bloquées" + bytes and ADDS the three new metrics:
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
kpi.innerHTML = `<span class="k">Pubs bloquées (204)</span> <span class="v">${d.total_blocked||0}</span>`
|
||||||
|
+ ` <span class="k">Trackers détectés</span> <span class="v">${d.trackers_seen||0}</span>`
|
||||||
|
+ ` <span class="k">Pages nettoyées</span> <span class="v">${d.pages_cleaned||0}</span>`
|
||||||
|
+ ` <span class="k">Drops réseau</span> <span class="v">${d.network_drops||0}</span>`
|
||||||
|
+ ` <span class="k" title="estimation : un contenu bloqué n'est jamais téléchargé, on ne peut pas mesurer les octets réels — ~45 Ko/blocage">Ko évités <span style="opacity:.6">(est.)</span></span> <span class="v">~${Math.round((d.total_bytes||0)/1024)}</span>`;
|
||||||
|
```
|
||||||
|
|
||||||
|
(Keep the surrounding code that builds `hostRows`/`siteRows` tables unchanged.)
|
||||||
|
|
||||||
|
- [ ] **Step 7: Run the test once more + commit**
|
||||||
|
|
||||||
|
Run: `cd packages/secubox-toolbox && python -m pytest tests/test_ads_aggregate.py -v`
|
||||||
|
Expected: PASS.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add packages/secubox-toolbox/secubox_toolbox/store.py packages/secubox-toolbox/secubox_toolbox/api.py packages/secubox-toolbox/www/toolbox/index.html packages/secubox-toolbox/tests/test_ads_aggregate.py
|
||||||
|
git commit -m "feat(toolbox): #ads breakdown — trackers_seen + network_drops (ref #755)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Go cosmetic-pages counter + flush + store + surface
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `packages/secubox-toolbox-ng/cmd/sbxmitm/adstats.go` (counter + payload field + flush)
|
||||||
|
- Modify: `packages/secubox-toolbox-ng/cmd/sbxmitm/main.go` (increment on injection)
|
||||||
|
- Modify: `packages/secubox-toolbox/secubox_toolbox/api.py` (`toolbox_ad_event` ingest)
|
||||||
|
- Modify: `packages/secubox-toolbox/secubox_toolbox/store.py` (`record_cosmetic_pages`)
|
||||||
|
- Test: `packages/secubox-toolbox-ng/cmd/sbxmitm/cosmetic_count_test.go`, extend `packages/secubox-toolbox/tests/test_ads_aggregate.py`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `store.ad_stats`'s `pages_cleaned` query (Task 1, reads `cosmetic_events`); the ad-event flush (`flushOnce`).
|
||||||
|
- Produces: `(*adStats).recordCosmetic()`, the `cosmetic_pages` JSON field on the ad-event payload; `store.record_cosmetic_pages(n: int)`.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing Go test**
|
||||||
|
|
||||||
|
Create `packages/secubox-toolbox-ng/cmd/sbxmitm/cosmetic_count_test.go`:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
package main
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestCosmeticCounterSnapshotClears(t *testing.T) {
|
||||||
|
a := newAdStats()
|
||||||
|
a.recordCosmetic()
|
||||||
|
a.recordCosmetic()
|
||||||
|
if got := a.snapshotCosmetic(); got != 2 {
|
||||||
|
t.Fatalf("snapshotCosmetic = %d, want 2", got)
|
||||||
|
}
|
||||||
|
if got := a.snapshotCosmetic(); got != 0 {
|
||||||
|
t.Fatalf("snapshot must clear; second call = %d, want 0", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run it to verify it fails**
|
||||||
|
|
||||||
|
Run: `cd packages/secubox-toolbox-ng && GOFLAGS=-mod=vendor go test ./cmd/sbxmitm/ -run TestCosmetic -v`
|
||||||
|
Expected: FAIL — `a.recordCosmetic undefined`.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Implement the Go counter**
|
||||||
|
|
||||||
|
In `adstats.go`: add a field to the `adStats` struct (find `type adStats struct {`): a counter guarded by the existing struct mutex, or a dedicated `sync/atomic` int64. Use atomic to avoid touching the existing lock scope:
|
||||||
|
|
||||||
|
Add import `"sync/atomic"` if absent. Add to the `adStats` struct:
|
||||||
|
```go
|
||||||
|
cosmetic atomic.Int64 // #755 — pages where the cosmetic ad-hide style was injected
|
||||||
|
```
|
||||||
|
Add methods:
|
||||||
|
```go
|
||||||
|
// recordCosmetic tallies one R3 HTML page that received the cosmetic ad-hide style.
|
||||||
|
func (a *adStats) recordCosmetic() { a.cosmetic.Add(1) }
|
||||||
|
|
||||||
|
// snapshotCosmetic atomically reads-and-clears the cosmetic page counter.
|
||||||
|
func (a *adStats) snapshotCosmetic() int64 { return a.cosmetic.Swap(0) }
|
||||||
|
```
|
||||||
|
In the ad-event payload struct (`adEventPayload`), add the field:
|
||||||
|
```go
|
||||||
|
CosmeticPages int64 `json:"cosmetic_pages,omitempty"`
|
||||||
|
```
|
||||||
|
In `flushOnce` (where the payload `p` is assembled before marshal), set:
|
||||||
|
```go
|
||||||
|
p.CosmeticPages = a.snapshotCosmetic()
|
||||||
|
```
|
||||||
|
Note: if `flushOnce` early-returns when ad block/candidate maps are empty, ensure a non-zero cosmetic count still gets POSTed — adjust the "is the snapshot empty?" guard to also consider `p.CosmeticPages > 0` so cosmetic-only windows still flush.
|
||||||
|
|
||||||
|
- [ ] **Step 4: Increment on injection (main.go)**
|
||||||
|
|
||||||
|
In `main.go`, in `mitmPipeline`, in the block `if out, ok := injectIntoBody(body, resp.Header.Get("Content-Encoding"), scriptBody, cspNonce, wg); ok {` — inside the `ok` branch (after `body = out`), add:
|
||||||
|
```go
|
||||||
|
px.ads.recordCosmetic() // #755 — this R3 HTML page got the cosmetic ad-hide style
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run the Go test + build**
|
||||||
|
|
||||||
|
Run: `cd packages/secubox-toolbox-ng && GOFLAGS=-mod=vendor go test ./cmd/sbxmitm/ -run TestCosmetic -v && GOFLAGS=-mod=vendor go build ./... && GOFLAGS=-mod=vendor go vet ./cmd/sbxmitm/`
|
||||||
|
Expected: PASS, build OK, vet clean.
|
||||||
|
|
||||||
|
- [ ] **Step 6: Python — store.record_cosmetic_pages + ad-event ingest**
|
||||||
|
|
||||||
|
In `store.py`, add (near `record_ad_blocks`):
|
||||||
|
```python
|
||||||
|
def record_cosmetic_pages(pages: int) -> None:
|
||||||
|
"""#755 — append one cosmetic-hide tally (pages cleaned since the last flush).
|
||||||
|
ad_stats sums these over the window. Best-effort; never raises."""
|
||||||
|
try:
|
||||||
|
n = int(pages)
|
||||||
|
if n <= 0:
|
||||||
|
return
|
||||||
|
with _conn() as c:
|
||||||
|
c.execute("CREATE TABLE IF NOT EXISTS cosmetic_events(ts REAL, pages INTEGER)")
|
||||||
|
c.execute("INSERT INTO cosmetic_events(ts, pages) VALUES(?, ?)", (time.time(), n))
|
||||||
|
except Exception as e:
|
||||||
|
log.debug("record_cosmetic_pages failed: %s", e)
|
||||||
|
```
|
||||||
|
|
||||||
|
In `api.py`, in `toolbox_ad_event`, after the existing `store.record_ad_blocks(...)` / `record_ad_candidates(...)` calls, add:
|
||||||
|
```python
|
||||||
|
cp = payload.get("cosmetic_pages")
|
||||||
|
if cp:
|
||||||
|
store.record_cosmetic_pages(cp)
|
||||||
|
```
|
||||||
|
(Match the variable name the handler uses for the parsed JSON body — the brief's `payload` may be named `body`/`data` in the actual handler; use whatever it is. Guard so a missing/zero field is a no-op.)
|
||||||
|
|
||||||
|
- [ ] **Step 7: Extend the Python test (pages_cleaned now populated)**
|
||||||
|
|
||||||
|
Append to `tests/test_ads_aggregate.py`:
|
||||||
|
```python
|
||||||
|
def test_record_cosmetic_pages_summed_in_window(tmp_path, monkeypatch):
|
||||||
|
_seed_db(tmp_path, monkeypatch)
|
||||||
|
store.record_cosmetic_pages(3)
|
||||||
|
store.record_cosmetic_pages(2)
|
||||||
|
out = store.ad_stats(hours=24)
|
||||||
|
assert out["pages_cleaned"] == 5
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 8: Run both suites**
|
||||||
|
|
||||||
|
Run: `cd packages/secubox-toolbox && python -m pytest tests/test_ads_aggregate.py -v` → PASS (3 tests).
|
||||||
|
Run: `cd packages/secubox-toolbox-ng && GOFLAGS=-mod=vendor go build ./... && GOFLAGS=-mod=vendor go test ./cmd/sbxmitm/ -count=1` → build OK, all PASS.
|
||||||
|
|
||||||
|
- [ ] **Step 9: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add packages/secubox-toolbox-ng/cmd/sbxmitm/adstats.go packages/secubox-toolbox-ng/cmd/sbxmitm/main.go packages/secubox-toolbox-ng/cmd/sbxmitm/cosmetic_count_test.go packages/secubox-toolbox/secubox_toolbox/store.py packages/secubox-toolbox/secubox_toolbox/api.py packages/secubox-toolbox/tests/test_ads_aggregate.py
|
||||||
|
git commit -m "feat(toolbox): cosmetic-pages counter → #ads 'Pages nettoyées' (ref #755)"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Self-Review notes
|
||||||
|
- **Spec coverage:** trackers_seen (Task 1) ✓; network_drops (Task 1) ✓; pages_cleaned cosmetic counter end-to-end Go→store→ad_stats (Task 2) ✓; WebUI labeled breakdown, not a sum (Task 1 Step 6) ✓; honest units (each labeled) ✓.
|
||||||
|
- **No placeholders:** Step 3 explicitly instructs deleting the illustrative dead block; the only "match the actual var name" notes (the ad-event handler's body var) are verify-in-context, not gaps.
|
||||||
|
- **Type consistency:** `trackers_seen`/`pages_cleaned`/`network_drops`/`cosmetic_pages` keys are identical across store → api → WebUI → Go payload → store ingest.
|
||||||
|
- **Out of scope:** real DNS-sinkhole per-window counter (no endpoint exists; `network_drops` uses the blacklist nft drops, 0 until that layer reports) — flagged in the issue.
|
||||||
21
docs/superpowers/plans/2026-06-27-sw-revalidation-nudge.md
Normal file
21
docs/superpowers/plans/2026-06-27-sw-revalidation-nudge.md
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
# #757 — SW revalidation nudge (strip conditional headers for allow-listed hosts)
|
||||||
|
|
||||||
|
**Goal:** stale-while-revalidate SWs on the sw-neuter allow-list update their cache
|
||||||
|
with a banner'd shell WITHOUT being neutered — by forcing a full 200 (not a 304)
|
||||||
|
on their HTML revalidation fetch so the MITM's existing injection lands.
|
||||||
|
|
||||||
|
**Design (approved):** in `mitmPipeline`, BEFORE the upstream proxy (`up.Do(req)`),
|
||||||
|
for an allow-listed host (`swNeuter.Match`) on an HTML document request, strip
|
||||||
|
`If-None-Match` + `If-Modified-Since` → upstream returns 200 with body → MITM
|
||||||
|
injects the banner → the SW caches the banner'd version on next background
|
||||||
|
revalidation. Gated to the allow-list (bounds the full-200 bandwidth cost).
|
||||||
|
Limitation: only helps SWs that revalidate; cache-first still needs #753's neuter.
|
||||||
|
|
||||||
|
## Task 1 (single)
|
||||||
|
**Files:** modify `cmd/sbxmitm/swneuter.go` (add `requestWantsHTML`), `cmd/sbxmitm/main.go` (the strip); test `cmd/sbxmitm/swneuter_test.go`.
|
||||||
|
|
||||||
|
- requestWantsHTML(req): true if `Sec-Fetch-Dest: document` (case-insensitive) OR `Accept` contains `text/html`; nil-safe false.
|
||||||
|
- Strip in mitmPipeline before `resp, err := up.Do(req)`:
|
||||||
|
`if px.swNeuter != nil && requestWantsHTML(req) && px.swNeuter.Match(host) { req.Header.Del("If-None-Match"); req.Header.Del("If-Modified-Since") }`
|
||||||
|
- Test: requestWantsHTML true/false cases (Sec-Fetch-Dest document, Accept text/html, neither, nil).
|
||||||
|
- Verify: go build + vet + go test ./cmd/sbxmitm/.
|
||||||
|
|
@ -28,6 +28,7 @@ import (
|
||||||
"net/url"
|
"net/url"
|
||||||
"regexp"
|
"regexp"
|
||||||
"sync"
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
@ -132,10 +133,12 @@ type adCounter struct {
|
||||||
// adStats is the lock-guarded in-memory aggregator. blocks is keyed by
|
// adStats is the lock-guarded in-memory aggregator. blocks is keyed by
|
||||||
// (adHost,site); clients by (macHash,adHost). The keys are small structs so the
|
// (adHost,site); clients by (macHash,adHost). The keys are small structs so the
|
||||||
// maps stay allocation-light and comparable without string concatenation.
|
// maps stay allocation-light and comparable without string concatenation.
|
||||||
|
// cosmetic counts HTML pages where the cosmetic ad-hide style was injected (#755).
|
||||||
type adStats struct {
|
type adStats struct {
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
blocks map[adKey]*adCounter
|
blocks map[adKey]*adCounter
|
||||||
clients map[cliKey]*adCounter
|
clients map[cliKey]*adCounter
|
||||||
|
cosmetic atomic.Int64 // #755 — pages where the cosmetic ad-hide style was injected
|
||||||
}
|
}
|
||||||
|
|
||||||
type adKey struct{ adHost, site string }
|
type adKey struct{ adHost, site string }
|
||||||
|
|
@ -181,6 +184,12 @@ func (a *adStats) recordAdBlock(adHost, site, macHash string) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// recordCosmetic tallies one R3 HTML page that received the cosmetic ad-hide style.
|
||||||
|
func (a *adStats) recordCosmetic() { a.cosmetic.Add(1) }
|
||||||
|
|
||||||
|
// snapshotCosmetic atomically reads-and-clears the cosmetic page counter.
|
||||||
|
func (a *adStats) snapshotCosmetic() int64 { return a.cosmetic.Swap(0) }
|
||||||
|
|
||||||
// ── wire payload (mirrors the portal /__toolbox/ad-event JSON contract) ──────
|
// ── wire payload (mirrors the portal /__toolbox/ad-event JSON contract) ──────
|
||||||
|
|
||||||
type adBlockRow struct {
|
type adBlockRow struct {
|
||||||
|
|
@ -207,9 +216,10 @@ type adCandidateRow struct {
|
||||||
}
|
}
|
||||||
|
|
||||||
type adEventPayload struct {
|
type adEventPayload struct {
|
||||||
Blocks []adBlockRow `json:"blocks"`
|
Blocks []adBlockRow `json:"blocks"`
|
||||||
Clients []adClientRow `json:"clients"`
|
Clients []adClientRow `json:"clients"`
|
||||||
Candidates []adCandidateRow `json:"candidates,omitempty"`
|
Candidates []adCandidateRow `json:"candidates,omitempty"`
|
||||||
|
CosmeticPages int64 `json:"cosmetic_pages,omitempty"` // #755 — pages cleaned since the last flush
|
||||||
}
|
}
|
||||||
|
|
||||||
// snapshot atomically reads-and-clears both maps, returning the accumulated rows.
|
// snapshot atomically reads-and-clears both maps, returning the accumulated rows.
|
||||||
|
|
@ -234,7 +244,7 @@ func (a *adStats) snapshot() adEventPayload {
|
||||||
|
|
||||||
// empty reports whether a payload carries no rows (nothing to POST).
|
// empty reports whether a payload carries no rows (nothing to POST).
|
||||||
func (p adEventPayload) empty() bool {
|
func (p adEventPayload) empty() bool {
|
||||||
return len(p.Blocks) == 0 && len(p.Clients) == 0 && len(p.Candidates) == 0
|
return len(p.Blocks) == 0 && len(p.Clients) == 0 && len(p.Candidates) == 0 && p.CosmeticPages == 0
|
||||||
}
|
}
|
||||||
|
|
||||||
// adEventClient is a short-timeout fire-and-forget client for the ad-event POST.
|
// adEventClient is a short-timeout fire-and-forget client for the ad-event POST.
|
||||||
|
|
@ -260,6 +270,7 @@ func (a *adStats) flushOnce(portal string, cand *adCandidates) adEventPayload {
|
||||||
if cand != nil {
|
if cand != nil {
|
||||||
p.Candidates = cand.snapshot()
|
p.Candidates = cand.snapshot()
|
||||||
}
|
}
|
||||||
|
p.CosmeticPages = a.snapshotCosmetic() // #755 — pages cleaned in this window
|
||||||
if p.empty() {
|
if p.empty() {
|
||||||
return p
|
return p
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -39,6 +39,13 @@ const cosmeticGuard = "sbx-ghost-style"
|
||||||
// CONSERVATISM note above). The rule mirrors the Python _style_for:
|
// CONSERVATISM note above). The rule mirrors the Python _style_for:
|
||||||
// display:none + visibility:hidden, both !important, collapsing the slot.
|
// display:none + visibility:hidden, both !important, collapsing the slot.
|
||||||
const cosmeticStyle = `<style id="sbx-ghost-style">` +
|
const cosmeticStyle = `<style id="sbx-ghost-style">` +
|
||||||
|
// #756 — restore scroll. When we display:none a paywall/consent overlay, the
|
||||||
|
// site's JS has often already scroll-locked the page (document.body.style.
|
||||||
|
// overflow='hidden', no inline !important). A stylesheet !important overrides
|
||||||
|
// that, so scroll returns (Bloomberg etc.). Tradeoff: a legitimate modal that
|
||||||
|
// locks body scroll will let the page scroll behind it — acceptable for a
|
||||||
|
// page-cleaning MITM whose purpose includes defeating paywall/consent locks.
|
||||||
|
`html,body{overflow:auto!important}` +
|
||||||
// ── ads (ported from _COSMETIC["ads"]) ──────────────────────────────────
|
// ── ads (ported from _COSMETIC["ads"]) ──────────────────────────────────
|
||||||
`[id^="google_ads"],` +
|
`[id^="google_ads"],` +
|
||||||
`[id^="div-gpt-ad"],` +
|
`[id^="div-gpt-ad"],` +
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,16 @@
|
||||||
|
// SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
package main
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestCosmeticCounterSnapshotClears(t *testing.T) {
|
||||||
|
a := newAdStats()
|
||||||
|
a.recordCosmetic()
|
||||||
|
a.recordCosmetic()
|
||||||
|
if got := a.snapshotCosmetic(); got != 2 {
|
||||||
|
t.Fatalf("snapshotCosmetic = %d, want 2", got)
|
||||||
|
}
|
||||||
|
if got := a.snapshotCosmetic(); got != 0 {
|
||||||
|
t.Fatalf("snapshot must clear; second call = %d, want 0", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -400,6 +400,15 @@ func (px *Proxy) mitmPipeline(tconn *tls.Conn, rawClient net.Conn, host, verdict
|
||||||
Transport: newUchromeTransport(dialHost, host),
|
Transport: newUchromeTransport(dialHost, host),
|
||||||
}
|
}
|
||||||
req.RequestURI = ""
|
req.RequestURI = ""
|
||||||
|
// #757 — SW revalidation nudge: for an allow-listed (sw-neuter) host, strip the
|
||||||
|
// conditional headers off an HTML navigation / SW-revalidation request so the
|
||||||
|
// upstream returns a full 200 (not a 304) → the MITM injects the banner →
|
||||||
|
// a stale-while-revalidate SW caches a banner'd shell WITHOUT being neutered.
|
||||||
|
// Cache-first SWs that never revalidate still need the #753 neuter.
|
||||||
|
if px.swNeuter != nil && requestWantsHTML(req) && px.swNeuter.Match(host) {
|
||||||
|
req.Header.Del("If-None-Match")
|
||||||
|
req.Header.Del("If-Modified-Since")
|
||||||
|
}
|
||||||
resp, err := up.Do(req)
|
resp, err := up.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeRaw(tconn, 502, "Bad Gateway", nil, nil)
|
writeRaw(tconn, 502, "Bad Gateway", nil, nil)
|
||||||
|
|
@ -526,6 +535,9 @@ func (px *Proxy) mitmPipeline(tconn *tls.Conn, rawClient net.Conn, host, verdict
|
||||||
scriptBody, _ := fetchInlineBanner(px.portal, clientHash, wg, cspBypassed)
|
scriptBody, _ := fetchInlineBanner(px.portal, clientHash, wg, cspBypassed)
|
||||||
if out, ok := injectIntoBody(body, resp.Header.Get("Content-Encoding"), scriptBody, cspNonce, wg); ok {
|
if out, ok := injectIntoBody(body, resp.Header.Get("Content-Encoding"), scriptBody, cspNonce, wg); ok {
|
||||||
body = out
|
body = out
|
||||||
|
if wg && px.ads != nil {
|
||||||
|
px.ads.recordCosmetic() // #755 — cosmetic style is wg-only (injectHTML gates it)
|
||||||
|
}
|
||||||
// Keep framing consistent with the served bytes (only the length changed).
|
// Keep framing consistent with the served bytes (only the length changed).
|
||||||
resp.Header.Set("Content-Length", strconv.Itoa(len(body)))
|
resp.Header.Set("Content-Length", strconv.Itoa(len(body)))
|
||||||
resp.ContentLength = int64(len(body))
|
resp.ContentLength = int64(len(body))
|
||||||
|
|
|
||||||
|
|
@ -115,6 +115,20 @@ func (s *SWNeuter) snapshotCandidates() []string {
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// requestWantsHTML reports whether req is for an HTML document (a navigation or a
|
||||||
|
// Service-Worker document fetch) — Sec-Fetch-Dest: document, or an Accept that
|
||||||
|
// advertises text/html. Used by the #757 revalidation nudge so we only force a
|
||||||
|
// full 200 on document fetches, never on subresources.
|
||||||
|
func requestWantsHTML(req *http.Request) bool {
|
||||||
|
if req == nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if strings.EqualFold(req.Header.Get("Sec-Fetch-Dest"), "document") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return strings.Contains(req.Header.Get("Accept"), "text/html")
|
||||||
|
}
|
||||||
|
|
||||||
// isSWScriptRequest reports whether req is a Service-Worker SCRIPT fetch.
|
// isSWScriptRequest reports whether req is a Service-Worker SCRIPT fetch.
|
||||||
// Browsers send the spec-mandated `Service-Worker: script` header on the
|
// Browsers send the spec-mandated `Service-Worker: script` header on the
|
||||||
// register() fetch and every update check — reliable and host-agnostic.
|
// register() fetch and every update check — reliable and host-agnostic.
|
||||||
|
|
|
||||||
|
|
@ -85,3 +85,25 @@ func TestSWFlushCandidatesClears(t *testing.T) {
|
||||||
t.Fatal("empty buffer → flush returns nil, no POST")
|
t.Fatal("empty buffer → flush returns nil, no POST")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRequestWantsHTML(t *testing.T) {
|
||||||
|
mk := func(setter func(h http.Header)) *http.Request {
|
||||||
|
r, _ := http.NewRequest("GET", "https://x/", nil)
|
||||||
|
if setter != nil {
|
||||||
|
setter(r.Header)
|
||||||
|
}
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
if !requestWantsHTML(mk(func(h http.Header) { h.Set("Sec-Fetch-Dest", "document") })) {
|
||||||
|
t.Fatal("Sec-Fetch-Dest: document → true")
|
||||||
|
}
|
||||||
|
if !requestWantsHTML(mk(func(h http.Header) { h.Set("Accept", "text/html,application/xhtml+xml") })) {
|
||||||
|
t.Fatal("Accept text/html → true")
|
||||||
|
}
|
||||||
|
if requestWantsHTML(mk(func(h http.Header) { h.Set("Accept", "image/png") })) {
|
||||||
|
t.Fatal("non-html Accept → false")
|
||||||
|
}
|
||||||
|
if requestWantsHTML(nil) {
|
||||||
|
t.Fatal("nil → false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,26 @@
|
||||||
|
secubox-toolbox-ng (0.1.26-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
|
* #757 SW revalidation nudge: for sw-neuter allow-listed hosts, strip
|
||||||
|
If-None-Match/If-Modified-Since on HTML document fetches so a
|
||||||
|
stale-while-revalidate Service Worker re-fetches a full 200 (banner injected)
|
||||||
|
and caches a banner'd shell WITHOUT being neutered. Cache-first SWs still use
|
||||||
|
the #753 neuter.
|
||||||
|
|
||||||
|
-- Gerald KERMA <devel@cybermind.fr> Sat, 27 Jun 2026 10:00:00 +0000
|
||||||
|
|
||||||
|
secubox-toolbox-ng (0.1.25-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
|
* #756 cosmetic restores scroll: the cosmetic ad-hide <style> now prepends
|
||||||
|
html,body{overflow:auto!important} so a paywall's JS scroll-lock
|
||||||
|
(body.style.overflow='hidden') is overridden and the page scrolls again
|
||||||
|
(Bloomberg etc.). No-op on normal pages; modal-scroll-behind is the accepted
|
||||||
|
tradeoff.
|
||||||
|
* #755 cosmetic-pages counter: sbxmitm tallies each WG/R3 HTML page that got
|
||||||
|
the cosmetic style (wg-gated, atomic) and flushes it via the ad-event channel
|
||||||
|
(cosmetic_pages) → portal → the #ads "Pages nettoyées" metric.
|
||||||
|
|
||||||
|
-- Gerald KERMA <devel@cybermind.fr> Sat, 27 Jun 2026 09:30:00 +0000
|
||||||
|
|
||||||
secubox-toolbox-ng (0.1.24-1~bookworm1) bookworm; urgency=medium
|
secubox-toolbox-ng (0.1.24-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
* #753 targeted Service-Worker neuter: PWA news sites (leparisien, cnn,
|
* #753 targeted Service-Worker neuter: PWA news sites (leparisien, cnn,
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,22 @@
|
||||||
|
secubox-toolbox (2.7.24-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
|
* ui: remove the redundant ♥ Liveness dashboard card (generic status/version);
|
||||||
|
the version stays in the top version-badge (loadHealth still drives it).
|
||||||
|
|
||||||
|
-- Gerald KERMA <devel@cybermind.fr> Sat, 27 Jun 2026 10:00:00 +0000
|
||||||
|
|
||||||
|
secubox-toolbox (2.7.23-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
|
* #755 #ads card → honest labeled MITM-protection breakdown: Pubs bloquées
|
||||||
|
(204), Trackers détectés (distinct cross-site cookie-trackers from
|
||||||
|
social_edges), Pages nettoyées (cosmetic-hide pages, fed by sbxmitm via the
|
||||||
|
new cosmetic_events table), Drops réseau (blacklist nft). store.ad_stats gains
|
||||||
|
trackers_seen + pages_cleaned; admin_ad_stats gains network_drops;
|
||||||
|
record_cosmetic_pages ingests the sbxmitm flush. No metric is summed across
|
||||||
|
units.
|
||||||
|
|
||||||
|
-- Gerald KERMA <devel@cybermind.fr> Sat, 27 Jun 2026 09:30:00 +0000
|
||||||
|
|
||||||
secubox-toolbox (2.7.22-1~bookworm1) bookworm; urgency=medium
|
secubox-toolbox (2.7.22-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
* #753 portal ingest for the SW-neuter auto-learn: POST /__toolbox/sw-candidate
|
* #753 portal ingest for the SW-neuter auto-learn: POST /__toolbox/sw-candidate
|
||||||
|
|
|
||||||
|
|
@ -250,6 +250,11 @@ async def toolbox_ad_event(request: Request) -> Response:
|
||||||
store.record_ad_client_blocks(client_rows)
|
store.record_ad_client_blocks(client_rows)
|
||||||
if cand_rows:
|
if cand_rows:
|
||||||
store.record_ad_candidates(cand_rows)
|
store.record_ad_candidates(cand_rows)
|
||||||
|
# #755 — cosmetic-pages counter: Go engine reports how many R3 HTML pages
|
||||||
|
# received the cosmetic ad-hide style in this flush window.
|
||||||
|
cp = body.get("cosmetic_pages")
|
||||||
|
if cp:
|
||||||
|
store.record_cosmetic_pages(cp)
|
||||||
except Exception as e: # never raise into the engine's fire-and-forget POST
|
except Exception as e: # never raise into the engine's fire-and-forget POST
|
||||||
log.debug("ad-event ingest failed: %s", e)
|
log.debug("ad-event ingest failed: %s", e)
|
||||||
return Response(status_code=204)
|
return Response(status_code=204)
|
||||||
|
|
@ -3090,7 +3095,15 @@ async def admin_protective() -> dict:
|
||||||
async def admin_ad_stats(hours: int = 24) -> dict:
|
async def admin_ad_stats(hours: int = 24) -> dict:
|
||||||
"""Contextual ad-block metrics for the #ads tab (read-only, kbin-safe)."""
|
"""Contextual ad-block metrics for the #ads tab (read-only, kbin-safe)."""
|
||||||
h = max(1, min(int(hours if hours is not None else 24), 168))
|
h = max(1, min(int(hours if hours is not None else 24), 168))
|
||||||
return store.ad_stats(hours=h)
|
out = store.ad_stats(hours=h)
|
||||||
|
# #755 — network-layer drops (blacklist nft sets). Best-effort; 0 when the
|
||||||
|
# blacklist is inert or unreadable. Reuses the admin_blacklist parse.
|
||||||
|
try:
|
||||||
|
bl = await admin_blacklist()
|
||||||
|
out["network_drops"] = int(bl.get("drops", 0) or 0)
|
||||||
|
except Exception:
|
||||||
|
out["network_drops"] = 0
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
@router.get("/admin/ad-stats/client/{mac_hash}")
|
@router.get("/admin/ad-stats/client/{mac_hash}")
|
||||||
|
|
|
||||||
|
|
@ -127,6 +127,20 @@ def ad_client_stats(mac_hash: str, hours: int = 24, top: int = 25) -> dict:
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def record_cosmetic_pages(pages: int) -> None:
|
||||||
|
"""#755 — append one cosmetic-hide tally (pages cleaned since the last flush).
|
||||||
|
ad_stats sums these over the window. Best-effort; never raises."""
|
||||||
|
try:
|
||||||
|
n = int(pages)
|
||||||
|
if n <= 0:
|
||||||
|
return
|
||||||
|
with _conn() as c:
|
||||||
|
c.execute("CREATE TABLE IF NOT EXISTS cosmetic_events(ts REAL, pages INTEGER)")
|
||||||
|
c.execute("INSERT INTO cosmetic_events(ts, pages) VALUES(?, ?)", (time.time(), n))
|
||||||
|
except Exception as e:
|
||||||
|
log.debug("record_cosmetic_pages failed: %s", e)
|
||||||
|
|
||||||
|
|
||||||
def record_ad_candidates(rows) -> None:
|
def record_ad_candidates(rows) -> None:
|
||||||
"""rows: iterable of (host, site, hits)."""
|
"""rows: iterable of (host, site, hits)."""
|
||||||
rows = [r for r in rows if r and r[0]]
|
rows = [r for r in rows if r and r[0]]
|
||||||
|
|
@ -180,6 +194,28 @@ def ad_stats(hours: int = 24, top: int = 25) -> dict:
|
||||||
"SELECT mac_hash, SUM(hits) FROM ad_block_client_host "
|
"SELECT mac_hash, SUM(hits) FROM ad_block_client_host "
|
||||||
"WHERE last_seen>=? AND mac_hash<>'' GROUP BY mac_hash "
|
"WHERE last_seen>=? AND mac_hash<>'' GROUP BY mac_hash "
|
||||||
"ORDER BY SUM(hits) DESC LIMIT ?", (cutoff, top))]
|
"ORDER BY SUM(hits) DESC LIMIT ?", (cutoff, top))]
|
||||||
|
# #755 — trackers detected/poisoned by the MITM in the window: distinct
|
||||||
|
# cross-site cookie-identifier hashes seen on social_edges. This is the
|
||||||
|
# "Trackers" half of the card (the 204 ad-block is the "pubs" half).
|
||||||
|
out["trackers_seen"] = 0
|
||||||
|
try:
|
||||||
|
out["trackers_seen"] = int(c.execute(
|
||||||
|
"SELECT COUNT(DISTINCT cookie_id_hash) FROM social_edges "
|
||||||
|
"WHERE ts >= ? AND cookie_id_hash IS NOT NULL AND cookie_id_hash <> ''",
|
||||||
|
(int(cutoff),),
|
||||||
|
).fetchone()[0] or 0)
|
||||||
|
except sqlite3.Error:
|
||||||
|
out["trackers_seen"] = 0
|
||||||
|
# #755 — pages where the cosmetic ad-hide style was injected (Task 2 writes
|
||||||
|
# cosmetic_events; absent table → 0).
|
||||||
|
out["pages_cleaned"] = 0
|
||||||
|
try:
|
||||||
|
out["pages_cleaned"] = int(c.execute(
|
||||||
|
"SELECT COALESCE(SUM(pages),0) FROM cosmetic_events WHERE ts >= ?",
|
||||||
|
(cutoff,),
|
||||||
|
).fetchone()[0] or 0)
|
||||||
|
except sqlite3.Error:
|
||||||
|
out["pages_cleaned"] = 0
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
log.debug("ad_stats failed: %s", e)
|
log.debug("ad_stats failed: %s", e)
|
||||||
return out
|
return out
|
||||||
|
|
|
||||||
39
packages/secubox-toolbox/tests/test_ads_aggregate.py
Normal file
39
packages/secubox-toolbox/tests/test_ads_aggregate.py
Normal file
|
|
@ -0,0 +1,39 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
"""Tests for the #ads aggregate breakdown (ref #755)."""
|
||||||
|
import sqlite3
|
||||||
|
import time
|
||||||
|
from secubox_toolbox import store
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_db(tmp_path, monkeypatch):
|
||||||
|
db = tmp_path / "toolbox.db"
|
||||||
|
c = sqlite3.connect(str(db))
|
||||||
|
c.executescript(
|
||||||
|
"CREATE TABLE ad_block_stats(ad_host TEXT, site TEXT, action TEXT, hits INTEGER, bytes INTEGER, last_seen REAL, PRIMARY KEY(ad_host,site,action));"
|
||||||
|
"CREATE TABLE ad_block_client_host(mac_hash TEXT, ad_host TEXT, hits INTEGER, last_seen REAL, PRIMARY KEY(mac_hash,ad_host));"
|
||||||
|
"CREATE TABLE social_edges(ts INTEGER, client_mac_hash TEXT, src_site TEXT, tracker_domain TEXT, cookie_id_hash TEXT, ja4_hash TEXT, consent_state TEXT);"
|
||||||
|
)
|
||||||
|
now = int(time.time())
|
||||||
|
# two distinct cookie-trackers in window, one duplicate, one stale (>24h)
|
||||||
|
for cid, ts in [("A", now-60), ("A", now-30), ("B", now-60), ("C", now-90000), ("", now-10)]:
|
||||||
|
c.execute("INSERT INTO social_edges(ts,client_mac_hash,src_site,tracker_domain,cookie_id_hash,ja4_hash,consent_state) VALUES(?,?,?,?,?,?,?)",
|
||||||
|
(ts, "m", "s", "t", cid, "j", "none_seen"))
|
||||||
|
c.commit(); c.close()
|
||||||
|
monkeypatch.setattr(store, "DB_PATH", db)
|
||||||
|
return db
|
||||||
|
|
||||||
|
|
||||||
|
def test_ad_stats_trackers_seen_distinct_in_window(tmp_path, monkeypatch):
|
||||||
|
_seed_db(tmp_path, monkeypatch)
|
||||||
|
out = store.ad_stats(hours=24)
|
||||||
|
# distinct non-empty cookie ids in the last 24h = {A, B}; C is stale, "" excluded
|
||||||
|
assert out["trackers_seen"] == 2
|
||||||
|
assert out["pages_cleaned"] == 0 # no cosmetic_events table yet → 0 (Task 2 fills it)
|
||||||
|
|
||||||
|
|
||||||
|
def test_record_cosmetic_pages_summed_in_window(tmp_path, monkeypatch):
|
||||||
|
_seed_db(tmp_path, monkeypatch)
|
||||||
|
store.record_cosmetic_pages(3)
|
||||||
|
store.record_cosmetic_pages(2)
|
||||||
|
out = store.ad_stats(hours=24)
|
||||||
|
assert out["pages_cleaned"] == 5
|
||||||
|
|
@ -97,10 +97,6 @@
|
||||||
<h2>📊 Live metrics (24h)</h2>
|
<h2>📊 Live metrics (24h)</h2>
|
||||||
<div class="kv" id="metrics"><span class="k">loading…</span><span class="v"></span></div>
|
<div class="kv" id="metrics"><span class="k">loading…</span><span class="v"></span></div>
|
||||||
</div>
|
</div>
|
||||||
<div class="card">
|
|
||||||
<h2>♥ Liveness</h2>
|
|
||||||
<div class="kv" id="health"><span class="k">loading…</span><span class="v"></span></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
|
@ -435,15 +431,10 @@ async function loadClientDetail(macHash) {
|
||||||
}
|
}
|
||||||
|
|
||||||
async function loadHealth() {
|
async function loadHealth() {
|
||||||
|
// Liveness card removed (redundant generic status); /health still drives the
|
||||||
|
// top version-badge.
|
||||||
const h = await J('/health');
|
const h = await J('/health');
|
||||||
const el = document.getElementById('health');
|
if (h && h.version) document.getElementById('version-badge').textContent = `v${h.version}`;
|
||||||
if (h.__error) { el.innerHTML = `<span class="k">err</span><span class="v">${h.__error}</span>`; return; }
|
|
||||||
el.innerHTML = `
|
|
||||||
<span class="k">status</span> <span class="v">${h.status}</span>
|
|
||||||
<span class="k">module</span> <span class="v">${h.module}</span>
|
|
||||||
<span class="k">version</span> <span class="v">${h.version}</span>
|
|
||||||
`;
|
|
||||||
document.getElementById('version-badge').textContent = `v${h.version}`;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async function loadFilters() {
|
async function loadFilters() {
|
||||||
|
|
@ -617,10 +608,11 @@ async function loadAds() {
|
||||||
const kpi = document.getElementById('ads-kpi');
|
const kpi = document.getElementById('ads-kpi');
|
||||||
const esc = s => String(s).replace(/&/g,'&').replace(/</g,'<').replace(/>/g,'>');
|
const esc = s => String(s).replace(/&/g,'&').replace(/</g,'<').replace(/>/g,'>');
|
||||||
if (!d || d.__error) { kpi.innerHTML = `<span class="k">err</span><span class="v">${(d&&d.__error)||'no data'}</span>`; return; }
|
if (!d || d.__error) { kpi.innerHTML = `<span class="k">err</span><span class="v">${(d&&d.__error)||'no data'}</span>`; return; }
|
||||||
kpi.innerHTML = `<span class="k">Trackers & pubs bloqués</span> <span class="v">${d.total_blocked||0}</span>`
|
kpi.innerHTML = `<span class="k">Pubs bloquées (204)</span> <span class="v">${d.total_blocked||0}</span>`
|
||||||
+ ` <span class="k" title="estimation : un contenu bloqué n'est jamais téléchargé, on ne peut pas mesurer les octets réels — ~45 Ko/blocage">Ko évités <span style="opacity:.6">(est.)</span></span> <span class="v">~${Math.round((d.total_bytes||0)/1024)}</span>`
|
+ ` <span class="k">Trackers détectés</span> <span class="v">${d.trackers_seen||0}</span>`
|
||||||
+ ` <span class="k">Silenced</span> <span class="v">${(d.by_action&&d.by_action.silent)||0}</span>`
|
+ ` <span class="k">Pages nettoyées</span> <span class="v">${d.pages_cleaned||0}</span>`
|
||||||
+ ` <span class="k">Fenêtre</span> <span class="v">${d.window_hours||24}h</span>`;
|
+ ` <span class="k">Drops réseau</span> <span class="v">${d.network_drops||0}</span>`
|
||||||
|
+ ` <span class="k" title="estimation : un contenu bloqué n'est jamais téléchargé, on ne peut pas mesurer les octets réels — ~45 Ko/blocage">Ko évités <span style="opacity:.6">(est.)</span></span> <span class="v">~${Math.round((d.total_bytes||0)/1024)}</span>`;
|
||||||
const hostRows = (d.top_hosts||[]).slice(0, 5).map(r=>`<tr><td><code>${esc(r.host)}</code></td><td>${r.hits}</td><td>~${Math.round((r.bytes||0)/1024)}</td></tr>`).join('');
|
const hostRows = (d.top_hosts||[]).slice(0, 5).map(r=>`<tr><td><code>${esc(r.host)}</code></td><td>${r.hits}</td><td>~${Math.round((r.bytes||0)/1024)}</td></tr>`).join('');
|
||||||
const siteRows = (d.top_sites||[]).slice(0, 5).map(r=>`<tr><td><code>${esc(r.site)}</code></td><td>${r.hits}</td></tr>`).join('');
|
const siteRows = (d.top_sites||[]).slice(0, 5).map(r=>`<tr><td><code>${esc(r.site)}</code></td><td>${r.hits}</td></tr>`).join('');
|
||||||
document.getElementById('ads-hosts').innerHTML = hostRows
|
document.getElementById('ads-hosts').innerHTML = hostRows
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue
Block a user