Version and expose the StarIntel map-reduce view registry #61

New issue

Open

opened 2026-07-26 02:35:09 +00:00 by lost-rob0t · 2 comments

lost-rob0t commented

2026-07-26 02:35:09 +00:00

(Migrated from github.com)

Goal

Make StarIntel map-reduce projections versioned, discoverable, correct, and queryable through both CouchDB and the server API.

Scope

consolidate design documents for core counts, relations, messages, targets, and events
version design document IDs/functions so migrations are explicit
fix legacy dtype/case/key inconsistencies and verify reduce/rereduce behavior
expose /api/v1/views discovery and /api/v1/views/:design/:view query endpoints
support keys, ranges, grouping, reduce, include_docs, pagination, and stale/update controls with validated limits
publish a machine-readable view manifest usable by Quasar's local PouchDB mirror
add install/migrate/status commands and safe index warm-up
test each map/reduce function against v0.9 fixtures and CouchDB

Initial shared projections

core: by dtype, by dataset, review status, dtype/dataset counts
relations: outgoing, incoming, edges, degree counts
targets: by actor, actor counts, target counts, state counts
messages: by user, channel/group, platform, chronological keys
events: type and time projections

Acceptance criteria

Quasar can install the same manifest locally and get equivalent results from PouchDB
server view results use a stable response envelope
every reduce supports rereduce correctly
migration never overwrites an unknown user design document

Depends on #58.

## Goal Make StarIntel map-reduce projections versioned, discoverable, correct, and queryable through both CouchDB and the server API. ## Scope - consolidate design documents for core counts, relations, messages, targets, and events - version design document IDs/functions so migrations are explicit - fix legacy dtype/case/key inconsistencies and verify reduce/rereduce behavior - expose `/api/v1/views` discovery and `/api/v1/views/:design/:view` query endpoints - support keys, ranges, grouping, reduce, include_docs, pagination, and stale/update controls with validated limits - publish a machine-readable view manifest usable by Quasar's local PouchDB mirror - add install/migrate/status commands and safe index warm-up - test each map/reduce function against v0.9 fixtures and CouchDB ## Initial shared projections - core: by dtype, by dataset, review status, dtype/dataset counts - relations: outgoing, incoming, edges, degree counts - targets: by actor, actor counts, target counts, state counts - messages: by user, channel/group, platform, chronological keys - events: type and time projections ## Acceptance criteria - Quasar can install the same manifest locally and get equivalent results from PouchDB - server view results use a stable response envelope - every reduce supports rereduce correctly - migration never overwrites an unknown user design document Depends on #58.

lost-rob0t commented

2026-08-24 08:55:25 +00:00

(Migrated from github.com)

ADARD corpus-driven redesign completed and split into implementation issue #114.

Key finding: this is a correctness repair before expansion. Several current generic views read legacy top-level relation/date/target fields while real Auto-Dig v0.9 records use data.subject/predicate/object, snake_case envelope dates, and investigation-target workflow/data state. #114 captures the v0.9-native registry, explicit time-basis views driven by Auto-Dig #2018, reviewed graph/degree projections, research frontier views, evidence/source projections, public visibility constraints, PouchDB parity, and TDD fixtures copied from real Auto-Dig record shapes.

Decision: no topic-specific views and no cross-document inference in CouchDB. Use compact generic map/reduce projections and leave pathfinding/entity resolution/contradiction adjudication to graph/application/Prolog layers.

ADARD corpus-driven redesign completed and split into implementation issue #114. Key finding: this is a correctness repair before expansion. Several current generic views read legacy top-level relation/date/target fields while real Auto-Dig v0.9 records use `data.subject/predicate/object`, snake_case envelope dates, and `investigation-target` workflow/data state. #114 captures the v0.9-native registry, explicit time-basis views driven by Auto-Dig #2018, reviewed graph/degree projections, research frontier views, evidence/source projections, public visibility constraints, PouchDB parity, and TDD fixtures copied from real Auto-Dig record shapes. Decision: no topic-specific views and no cross-document inference in CouchDB. Use compact generic map/reduce projections and leave pathfinding/entity resolution/contradiction adjudication to graph/application/Prolog layers.

lost-rob0t commented

2026-08-24 09:00:05 +00:00

(Migrated from github.com)

Architecture boundary clarified after ADARD: #114 is generalized core views only. Auto-Dig corpus records are being used to discover real generic query shapes and catch v0.9 field bugs, not to add topic-specific views to server core. Deployment/data-specific CouchDB design docs belong in opt-in installable packs owned by starintel-labs/starintel-infra#36. Core registry/API should distinguish built-in generalized views from installed custom packs.

Architecture boundary clarified after ADARD: #114 is **generalized core views only**. Auto-Dig corpus records are being used to discover real generic query shapes and catch v0.9 field bugs, not to add topic-specific views to server core. Deployment/data-specific CouchDB design docs belong in opt-in installable packs owned by `starintel-labs/starintel-infra#36`. Core registry/API should distinguish built-in generalized views from installed custom packs.