nsaspy/hackmode

Fork 0

Investigation views: define Hackmode-local Tek9 MapReduce views over operation data #15

New issue

Closed

opened 2026-08-16 05:23:07 +00:00 by lost-rob0t · 2 comments

lost-rob0t commented

2026-08-16 05:23:07 +00:00

(Migrated from github.com)

Problem

Hackmode needs fast operation-scoped investigation queries, but view ownership must be split correctly: Tek9 should provide generic materialized-view primitives, while Hackmode defines investigation-specific view semantics.

Current behavior

operation-local data is persisted in Tek9.
Tek9 already supports durable map/reduce view definitions and incremental update.
Tek9 issue lost-rob0t/tek9#2 tracks a missing public read/query API; current consumers would otherwise need raw LMDB access.

Desired behavior

Define Hackmode-owned views after the Tek9 public read API exists.

Initial local views:

assets/by-type
assets/by-operation
assets/by-domain
assets/by-host
assets/by-ip
assets/by-source
assets/by-tag
domains/by-root
domains/subdomains
urls/by-domain
urls/by-status
urls/by-content-type
hosts/by-port
hosts/by-service
findings/by-severity
findings/by-operation
documents/by-operation
documents/by-source
relationships/by-source
relationships/by-target
ingest/by-state
ingest/failed

Only implement views whose underlying canonical fields exist. Do not fabricate schema just to satisfy the list.

Architecture

generic view storage/query stays in Tek9
local operation views are defined/registered in Hackmode
central CouchDB views stay in starintel-server where they serve central datasets
callers use Hackmode/Tek9 public query APIs, never raw LMDB
maintain views incrementally from canonical operation writes where possible

Dependencies

#6 canonical operation/asset lifecycle
#12 durable ingest/outbox for ingest-state views
lost-rob0t/tek9#2 public materialized-view read API
#3 canonical StarIntel asset/document mapping for document/relationship view fields

Acceptance criteria

first useful subset (assets/by-type, assets/by-tag, ingest/by-state) is defined with fixtures
repeated writes update views without duplicate rows
view rebuild and incremental update produce equivalent results
no raw LMDB code in Hackmode
unsupported candidate views are explicitly deferred rather than implemented against guessed fields

Verification

Fixture operation with multiple asset types/tags/outbox states; test exact and range queries, incremental updates, and full rebuild equivalence.

## Problem Hackmode needs fast operation-scoped investigation queries, but view ownership must be split correctly: Tek9 should provide generic materialized-view primitives, while Hackmode defines investigation-specific view semantics. ## Current behavior - operation-local data is persisted in Tek9. - Tek9 already supports durable map/reduce view definitions and incremental update. - Tek9 issue lost-rob0t/tek9#2 tracks a missing public read/query API; current consumers would otherwise need raw LMDB access. ## Desired behavior Define Hackmode-owned views after the Tek9 public read API exists. Initial local views: - `assets/by-type` - `assets/by-operation` - `assets/by-domain` - `assets/by-host` - `assets/by-ip` - `assets/by-source` - `assets/by-tag` - `domains/by-root` - `domains/subdomains` - `urls/by-domain` - `urls/by-status` - `urls/by-content-type` - `hosts/by-port` - `hosts/by-service` - `findings/by-severity` - `findings/by-operation` - `documents/by-operation` - `documents/by-source` - `relationships/by-source` - `relationships/by-target` - `ingest/by-state` - `ingest/failed` Only implement views whose underlying canonical fields exist. Do not fabricate schema just to satisfy the list. ## Architecture - generic view storage/query stays in Tek9 - local operation views are defined/registered in Hackmode - central CouchDB views stay in `starintel-server` where they serve central datasets - callers use Hackmode/Tek9 public query APIs, never raw LMDB - maintain views incrementally from canonical operation writes where possible ## Dependencies - #6 canonical operation/asset lifecycle - #12 durable ingest/outbox for ingest-state views - lost-rob0t/tek9#2 public materialized-view read API - #3 canonical StarIntel asset/document mapping for document/relationship view fields ## Acceptance criteria - first useful subset (`assets/by-type`, `assets/by-tag`, `ingest/by-state`) is defined with fixtures - repeated writes update views without duplicate rows - view rebuild and incremental update produce equivalent results - no raw LMDB code in Hackmode - unsupported candidate views are explicitly deferred rather than implemented against guessed fields ## Verification Fixture operation with multiple asset types/tags/outbox states; test exact and range queries, incremental updates, and full rebuild equivalence.

nsaspy commented

2026-09-02 17:24:41 +00:00

Owner

Database slice landed via #166 at merge 04f65eb17ffd00198776af6770096268b95f514d.

Completed in this slice:

Hackmode-owned materialized assets/by-type and assets/by-tag views over canonical operation documents;
canonical asset writes maintain the views after persistence and before observer publication;
first registration rebuilds existing operation data, so legacy stores do not appear empty;
typed query-assets now consumes assets/by-type rather than full-scanning the operation database;
public reads stay on Tek9's exported view API; no raw LMDB boundary leak;
deterministic tests cover type/tag selection, idempotent rediscovery, stable row order, and rebuild equivalence.

RED: 872639d130feeddcee3a6328f66795f3e4f297c9 failed the real Common Lisp gate on the intentionally absent HACKMODE:ENSURE-INVESTIGATION-VIEWS contract.
GREEN: exact implementation head d23073e6df8bc8f060e84877d510d5569e12df93 passed fresh non-draft PR core, monorepo, and agent-framework-boundary workflows before expected-head squash merge.

Remaining #15 work stays open. In particular, ingest/by-state cannot correctly materialize the durable outbox yet because Tek9 view application is currently hard-wired to the primary DB while the outbox is a named DB. Generic source-database selection is now tracked upstream as lost-rob0t/tek9#15; I did not duplicate/move outbox state or reach around Tek9 internals.

Database slice landed via #166 at merge `04f65eb17ffd00198776af6770096268b95f514d`. Completed in this slice: - Hackmode-owned materialized `assets/by-type` and `assets/by-tag` views over canonical operation documents; - canonical asset writes maintain the views after persistence and before observer publication; - first registration rebuilds existing operation data, so legacy stores do not appear empty; - typed `query-assets` now consumes `assets/by-type` rather than full-scanning the operation database; - public reads stay on Tek9's exported view API; no raw LMDB boundary leak; - deterministic tests cover type/tag selection, idempotent rediscovery, stable row order, and rebuild equivalence. RED: `872639d130feeddcee3a6328f66795f3e4f297c9` failed the real Common Lisp gate on the intentionally absent `HACKMODE:ENSURE-INVESTIGATION-VIEWS` contract. GREEN: exact implementation head `d23073e6df8bc8f060e84877d510d5569e12df93` passed fresh non-draft PR core, monorepo, and agent-framework-boundary workflows before expected-head squash merge. Remaining #15 work stays open. In particular, `ingest/by-state` cannot correctly materialize the durable outbox yet because Tek9 view application is currently hard-wired to the primary DB while the outbox is a named DB. Generic source-database selection is now tracked upstream as lost-rob0t/tek9#15; I did not duplicate/move outbox state or reach around Tek9 internals.

nsaspy commented

2026-09-02 17:24:41 +00:00

Owner

Final database acceptance reconciliation on current master 78a9f94dc80075240afe8e595938c2cec75c984e.

The issue's required first-useful subset is now complete:

assets/by-type — landed via #166;
assets/by-tag — landed via #166;
ingest/by-state — landed via #169 after Tek9 named-source view support became available.

The acceptance invariants are covered by repository tests: repeated writes do not duplicate view rows, view rebuild matches incremental maintenance, reads go through public Tek9 materialized-view APIs, and Hackmode contains no raw-LMDB investigation-view path.

Candidate views beyond this subset remain intentionally deferred until their canonical fields exist. Per the issue contract, I am not fabricating domain/host/IP/source/finding/document/relationship schema solely to populate the original wishlist.

#15 therefore meets its stated acceptance criteria and can close. Future concrete views should be opened from proven canonical fields/query needs rather than keeping this acceptance issue artificially open.

Final database acceptance reconciliation on current master `78a9f94dc80075240afe8e595938c2cec75c984e`. The issue's required first-useful subset is now complete: - `assets/by-type` — landed via #166; - `assets/by-tag` — landed via #166; - `ingest/by-state` — landed via #169 after Tek9 named-source view support became available. The acceptance invariants are covered by repository tests: repeated writes do not duplicate view rows, view rebuild matches incremental maintenance, reads go through public Tek9 materialized-view APIs, and Hackmode contains no raw-LMDB investigation-view path. Candidate views beyond this subset remain intentionally deferred until their canonical fields exist. Per the issue contract, I am not fabricating domain/host/IP/source/finding/document/relationship schema solely to populate the original wishlist. #15 therefore meets its stated acceptance criteria and can close. Future concrete views should be opened from proven canonical fields/query needs rather than keeping this acceptance issue artificially open.

nsaspy closed this issue

2026-09-04 14:35:15 +00:00

nsaspy referenced this issue

2026-09-04 20:36:52 +00:00

test: prove missing ingest-by-state runtime contract #58

nsaspy referenced this issue

2026-09-04 20:36:52 +00:00

views: add canonical ingest-by-state materialization #59

nsaspy referenced this issue

2026-09-04 20:36:53 +00:00

Investigation: materialize operation asset views #61

nsaspy referenced this issue