nsaspy/hackmode

Fork 0

Default HTTP capture + browser-profile egress for spawned providers #38

New issue

Open

opened 2026-09-02 17:24:36 +00:00 by nsaspy · 2 comments

nsaspy commented

2026-09-02 17:24:36 +00:00

Owner

Outcome

Make operation-scoped HTTP(S) execution use one consistent transport/capture policy instead of each provider inventing proxy, headers, and client identity independently.

Parent capture stack: #26
Provider migration: #2
Hackpert active orchestration: #27/#29
ZAP API/provider work: #21/#22/#23

Operator direction — 2026-08-31

an operation-scoped mitmproxy/mitmdump capture service should be started and available by default;
HTTP-capable spawned tools/providers should use that capture path by default where protocol semantics permit;
spawned HTTP clients should use explicit browser-consistent client profiles rather than obvious default tool user agents;
prefer a pinned, maintained curl-impersonate implementation for curl-like HTTP acquisition where its fingerprint semantics are actually preserved;
ZAP should not become a second HTTP-capture authority: when ZAP is used, configure its outbound/upstream proxy through the operation capture service and keep ZAP API control for ZAP-specific contexts/alerts/spider/active-scan jobs.

Important boundary: capture vs origin-side TLS fingerprint

Full TLS interception and end-to-end client TLS impersonation are not the same mode.

If curl-impersonate performs TLS to a local intercepting proxy, the origin sees the proxy's upstream TLS fingerprint, not curl-impersonate's original browser TLS fingerprint. Do not claim both properties simultaneously.

Model this explicitly, for example:

intercept — decrypted HTTP request/response evidence is captured; origin-side TLS fingerprint belongs to the capture proxy;
tunnel — CONNECT/raw passthrough preserves the provider/browser TLS handshake toward the origin but the capture layer cannot decrypt/store the HTTP bodies;
direct/unsupported — only for capabilities where the proxy is technically inapplicable or policy explicitly permits it; execution evidence must say so rather than pretending capture occurred.

The operation may still keep the capture service running in every mode.

HTTP client profile

Define one typed profile selected/frozen for a provider execution, carrying at least:

stable profile ID/version;
browser family/version identity;
User-Agent;
compatible Accept, Accept-Language, Accept-Encoding, fetch/client-hint headers where supported;
HTTP protocol preference;
transport backend identity (curl-impersonate, browser/CDP, ordinary client, etc.);
capture mode;
provenance/version sufficient to reproduce why a request looked the way it did.

Do not randomize only User-Agent per request while leaving an obviously incompatible header/TLS profile. A coherent frozen profile is less noisy and more reproducible.

Protocol-inapplicable tools (DNS, raw port scanning, etc.) do not get fake HTTP user-agent settings.

mitmproxy/IPX lifecycle

The capture process is a supervised Hackmode provider/service boundary, not a database primitive.

operation
   -> capture supervisor
      -> mitmdump process
      -> versioned append-only IPX spool
      -> parser/tailer
      -> canonical typed :http-exchange evidence

The Python addon/process writes the versioned spool only. It never writes Tek9 directly. Existing #26 checkpoint/quarantine/rotation and execution-graph persistence remain canonical.

ZAP relationship

Keep #21 for typed ZAP API semantics and jobs.

Do not add a separate ZAP HTTP-history capture pipeline if the same traffic can be routed:

client/ZAP -> mitmproxy/IPX -> origin

ZAP alerts remain ZAP-specific typed evidence and may link to canonical HTTP exchanges. Active ZAP scanning remains explicitly operation-scoped per #21.

Required proofs

starting an operation can realize one supervised capture session with stable operation/session identity;
HTTP-capable provider receives the selected capture endpoint and client profile through typed configuration;
proxy/capture failure is observable and does not silently downgrade to uncaptured HTTP unless policy explicitly permits fallback;
browser-profile headers are internally consistent in deterministic fixtures;
curl-like acquisition selects the impersonating backend when configured/available;
intercept vs tunnel mode is visible in execution provenance;
ZAP outbound traffic can use the same capture proxy while ZAP API control remains separate;
raw secrets/API keys are never injected into execution logs/profile snapshots;
no second operation database, capture database, or provider execution authority is introduced.

Ownership handoff

This is provider/runtime implementation, not source/hackmode-database/**. The existing database worker already supplies typed HTTP exchange/checkpoint/quarantine/rotation persistence under #26. Consumer/process/profile implementation belongs in the provider/Hackpert runtime lane.

Canonical execution checklist

Use this section as the RAGE pickup map. Do not recreate these requirements as parallel issues.

#135 — supervised operation-scoped mitmproxy / mitmdump provider
#136 — versioned mitmproxy addon -> provider-neutral IPX spool
#137 — typed HTTP transport + coherent browser client profiles for spawned providers
#138 — typed curl-impersonate requester integrated with capture/IPX
#139 — chain OWASP ZAP through canonical mitmproxy/IPX capture; correlate ZAP alerts without a second HTTP-history authority
#140 — Hackmode-native visual web reconnaissance / GoWitness replacement
lost-rob0t/tek9#12 — configurable LMDB map-size autogrow with bounded safe retry

Worker ordering

Land #135 and #136 together or in an order that keeps the capture contract testable.
Land #137 before migrating arbitrary HTTP-capable providers so every provider consumes one typed transport/profile policy.
Implement #138 against #137 rather than adding curl-specific policy.
Implement #139 against the canonical capture endpoint and preserve ZAP only as its typed scanner/control surface.
Drive #140 from the same browser-profile/capture policy; do not add a GoWitness side database.
Tek9 autogrow remains a Tek9 storage-engine task and must not be reimplemented in Hackmode.

RAGE completion rule

This umbrella is not complete because individual tools merely exist. Completion requires the integrated proof: one authorized operation starts canonical capture, representative HTTP/browser tooling receives coherent transport/profile configuration, curl-like acquisition records its actual backend/capture mode, ZAP traffic can traverse the same capture authority, visual reconnaissance stores replay-safe canonical evidence, and failures never silently downgrade transport/capture policy.

## Outcome Make operation-scoped HTTP(S) execution use one consistent transport/capture policy instead of each provider inventing proxy, headers, and client identity independently. Parent capture stack: #26 Provider migration: #2 Hackpert active orchestration: #27/#29 ZAP API/provider work: #21/#22/#23 ## Operator direction — 2026-08-31 - an operation-scoped `mitmproxy`/`mitmdump` capture service should be started and available by default; - HTTP-capable spawned tools/providers should use that capture path by default where protocol semantics permit; - spawned HTTP clients should use explicit browser-consistent client profiles rather than obvious default tool user agents; - prefer a pinned, maintained `curl-impersonate` implementation for curl-like HTTP acquisition where its fingerprint semantics are actually preserved; - ZAP should not become a second HTTP-capture authority: when ZAP is used, configure its outbound/upstream proxy through the operation capture service and keep ZAP API control for ZAP-specific contexts/alerts/spider/active-scan jobs. ## Important boundary: capture vs origin-side TLS fingerprint Full TLS interception and end-to-end client TLS impersonation are not the same mode. If `curl-impersonate` performs TLS to a local intercepting proxy, the origin sees the proxy's upstream TLS fingerprint, not curl-impersonate's original browser TLS fingerprint. Do not claim both properties simultaneously. Model this explicitly, for example: - `intercept` — decrypted HTTP request/response evidence is captured; origin-side TLS fingerprint belongs to the capture proxy; - `tunnel` — CONNECT/raw passthrough preserves the provider/browser TLS handshake toward the origin but the capture layer cannot decrypt/store the HTTP bodies; - `direct`/`unsupported` — only for capabilities where the proxy is technically inapplicable or policy explicitly permits it; execution evidence must say so rather than pretending capture occurred. The operation may still keep the capture service running in every mode. ## HTTP client profile Define one typed profile selected/frozen for a provider execution, carrying at least: - stable profile ID/version; - browser family/version identity; - `User-Agent`; - compatible `Accept`, `Accept-Language`, `Accept-Encoding`, fetch/client-hint headers where supported; - HTTP protocol preference; - transport backend identity (`curl-impersonate`, browser/CDP, ordinary client, etc.); - capture mode; - provenance/version sufficient to reproduce why a request looked the way it did. Do not randomize only `User-Agent` per request while leaving an obviously incompatible header/TLS profile. A coherent frozen profile is less noisy and more reproducible. Protocol-inapplicable tools (DNS, raw port scanning, etc.) do not get fake HTTP user-agent settings. ## mitmproxy/IPX lifecycle The capture process is a supervised Hackmode provider/service boundary, not a database primitive. ```text operation -> capture supervisor -> mitmdump process -> versioned append-only IPX spool -> parser/tailer -> canonical typed :http-exchange evidence ``` The Python addon/process writes the versioned spool only. It never writes Tek9 directly. Existing #26 checkpoint/quarantine/rotation and execution-graph persistence remain canonical. ## ZAP relationship Keep #21 for typed ZAP API semantics and jobs. Do **not** add a separate ZAP HTTP-history capture pipeline if the same traffic can be routed: `client/ZAP -> mitmproxy/IPX -> origin` ZAP alerts remain ZAP-specific typed evidence and may link to canonical HTTP exchanges. Active ZAP scanning remains explicitly operation-scoped per #21. ## Required proofs - starting an operation can realize one supervised capture session with stable operation/session identity; - HTTP-capable provider receives the selected capture endpoint and client profile through typed configuration; - proxy/capture failure is observable and does not silently downgrade to uncaptured HTTP unless policy explicitly permits fallback; - browser-profile headers are internally consistent in deterministic fixtures; - curl-like acquisition selects the impersonating backend when configured/available; - intercept vs tunnel mode is visible in execution provenance; - ZAP outbound traffic can use the same capture proxy while ZAP API control remains separate; - raw secrets/API keys are never injected into execution logs/profile snapshots; - no second operation database, capture database, or provider execution authority is introduced. ## Ownership handoff This is provider/runtime implementation, not `source/hackmode-database/**`. The existing database worker already supplies typed HTTP exchange/checkpoint/quarantine/rotation persistence under #26. Consumer/process/profile implementation belongs in the provider/Hackpert runtime lane. ## Canonical execution checklist Use this section as the RAGE pickup map. Do not recreate these requirements as parallel issues. - [ ] #135 — supervised operation-scoped `mitmproxy` / `mitmdump` provider - [ ] #136 — versioned mitmproxy addon -> provider-neutral IPX spool - [ ] #137 — typed HTTP transport + coherent browser client profiles for spawned providers - [ ] #138 — typed `curl-impersonate` requester integrated with capture/IPX - [ ] #139 — chain OWASP ZAP through canonical mitmproxy/IPX capture; correlate ZAP alerts without a second HTTP-history authority - [ ] #140 — Hackmode-native visual web reconnaissance / GoWitness replacement - [ ] lost-rob0t/tek9#12 — configurable LMDB map-size autogrow with bounded safe retry ### Worker ordering 1. Land #135 and #136 together or in an order that keeps the capture contract testable. 2. Land #137 before migrating arbitrary HTTP-capable providers so every provider consumes one typed transport/profile policy. 3. Implement #138 against #137 rather than adding curl-specific policy. 4. Implement #139 against the canonical capture endpoint and preserve ZAP only as its typed scanner/control surface. 5. Drive #140 from the same browser-profile/capture policy; do not add a GoWitness side database. 6. Tek9 autogrow remains a Tek9 storage-engine task and must not be reimplemented in Hackmode. ### RAGE completion rule This umbrella is not complete because individual tools merely exist. Completion requires the integrated proof: one authorized operation starts canonical capture, representative HTTP/browser tooling receives coherent transport/profile configuration, curl-like acquisition records its actual backend/capture mode, ZAP traffic can traverse the same capture authority, visual reconnaissance stores replay-safe canonical evidence, and failures never silently downgrade transport/capture policy.

nsaspy commented

2026-09-02 17:24:49 +00:00

Author

Owner

Database handoff: PR #134 merged the provider-neutral canonical HTTP-header evidence boundary. make-http-exchange-record now accepts bounded sanitized request/response header pairs, snapshots them into immutable operation-scoped execution evidence, and rejects secret-bearing header names (Authorization/cookies/API-key families) rather than persisting their values. Provider/runtime capture work in #133 can target this boundary without writing Tek9 directly or creating a provider-specific HTTP schema.

Database handoff: PR #134 merged the provider-neutral canonical HTTP-header evidence boundary. `make-http-exchange-record` now accepts bounded sanitized request/response header pairs, snapshots them into immutable operation-scoped execution evidence, and rejects secret-bearing header names (Authorization/cookies/API-key families) rather than persisting their values. Provider/runtime capture work in #133 can target this boundary without writing Tek9 directly or creating a provider-specific HTTP schema.

nsaspy commented

2026-09-02 17:24:49 +00:00

Author

Owner

Split this umbrella into dependency-ready RAGE slices:

#135 supervised operation-scoped mitmproxy/mitmdump provider
#136 versioned mitmproxy addon -> provider-neutral IPX spool
#137 typed transport + coherent browser client profile policy for spawned providers
#138 typed curl-impersonate HTTP requester
#139 ZAP -> canonical mitmproxy/IPX chaining + alert correlation

Keep #133 as the cross-cutting transport/capture policy. Child workers should claim the narrow issues rather than implementing this umbrella wholesale.

Split this umbrella into dependency-ready RAGE slices: - #135 supervised operation-scoped mitmproxy/mitmdump provider - #136 versioned mitmproxy addon -> provider-neutral IPX spool - #137 typed transport + coherent browser client profile policy for spawned providers - #138 typed curl-impersonate HTTP requester - #139 ZAP -> canonical mitmproxy/IPX chaining + alert correlation Keep #133 as the cross-cutting transport/capture policy. Child workers should claim the narrow issues rather than implementing this umbrella wholesale.

nsaspy referenced this issue

2026-09-04 20:37:21 +00:00

Hackpert: establish explicit passive/active engine authority #134