nsaspy/starintel-server

Fork 0

Generalized Views v1: v0.9-native reusable projections and legacy field repair #114

New issue

Open

opened 2026-08-24 08:55:13 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-24 08:55:13 +00:00

(Migrated from github.com)

Parent: #61
Depends on API foundation: #58
Cross-repo evidence: lost-rob0t/starintel-gpt-auto-dig#2018

ADARD decision

Replace the legacy/accidental view surface with a versioned v0.9-native view registry derived from actual Auto-Dig corpus/query behavior. Do not add topic-specific views. Build a compact set of generic projections that answer the queries Auto-Dig, Quasar, dashboards, recursive workers, and public stats actually perform.

This issue is the implementation slice produced by the ADARD research/design loop. It must be TDD-first and exact-head verified.

A — Analyze / corpus evidence

Real corpus scale and pulse problem

Auto-Dig issue #2018 records a 30-day corpus snapshot with 1,254,577 documents across only 13 active days. Bulk materializations create huge spikes. The current dashboard groups only date_added, conflating first-seen/materialization/research-event semantics.

The Auto-Dig corpus dashboard currently recomputes over the complete corpus in Python:

document counts and dtype composition;
dataset counts and 30-day activity;
relation predicate counts;
reviewed relation topology / top connected people;
unique source domains;
evidence-backed findings ranked from evidence/source/confidence/link counts;
latest reviewed activity;
active datasets.

These are direct candidates for indexed server projections.

Real v0.9 record shapes inspected

Representative current canonical records show:

common envelope: _id, dataset, dtype, date_added, date_updated, status, verification, handling, assessment, sources, related_ids, temporal, provenance, workflow, data;
relation: data.subject, data.predicate, data.object, sometimes data.confidence;
investigation-target: workflow.research_status, workflow.recursion_depth, workflow.priority, workflow.run_id, plus data.status, data.depth, data.priority, data.score, data.seed_ids;
research-pass: selected-target/findings/support/counterevidence/unresolved-target structures under data;
dataset-manifest: data.record_count, data.counts_by_dtype, generation/run metadata and coverage status;
people/orgs generally expose canonical display identity through data.name / title, while older split-name fields are not guaranteed.

Existing view defects against real data

source/views/data.json currently contains several legacy-shape projections that do not correctly index canonical v0.9 Auto-Dig records:

relations_by_predicate reads doc.predicate instead of doc.data.predicate;
relations_out_degree reads doc.source instead of doc.data.subject;
relations_in_degree reads doc.target instead of doc.data.object;
relations_self_loops has the same legacy endpoint assumption;
targets_by_actor assumes dtype === 'target' plus top-level actor, which does not represent Auto-Dig investigation-target workflow records;
date_added_stats_by_dataset reads only dateAdded;
bad_docs_missing_meta treats snake-case date_added / date_updated as missing;
persons/by_name assumes lname/fname, while current migrated v0.9 people can carry only data.name;
multiple map views emit entire documents even though callers can request include_docs, increasing index size.

This is correctness debt before it is optimization work.

D — Design

Create versioned design documents / manifest IDs. Exact naming may follow the registry machinery in #61, but semantic namespaces are fixed below.

1. `core-v1`

`count_by_dtype`

Map: [dtype] -> 1; reduce _count or _sum.

`count_by_dataset`

Map: [dataset] -> 1.

`count_by_dataset_dtype`

Map: [dataset, dtype] -> 1.
Supports corpus composition globally or per dataset with group_level.

`documents_by_dataset`

Map-only [dataset, _id] -> null.
Never emit the full document; use include_docs=true when required.

`review_state`

Normalize the same reviewed/unreviewed semantics Auto-Dig currently computes from:

verification.status;
workflow.review_status / workflow status;
top-level status.
Emit [dataset, normalized_review_state, dtype] -> 1.

The normalization rule must be a shared/generated contract, not independently drifting Python and JavaScript token lists.

`verification_status`

Emit [dataset, verification.status || 'unspecified', dtype] -> 1.
Do not collapse detailed verification status into review_state only.

`visibility`

Emit [handling.visibility || 'unspecified', dataset, dtype] -> 1.
This is mandatory for API-side authority filtering and public-mode safety.

`source_presence`

Emit [dataset, has_sources|no_sources, dtype] -> 1.

2. `activity-v1`

Do not keep one ambiguous “documents by day” view.

`by_time_basis`

For every valid timestamp available on a document, emit one row with a bounded basis enum:

["added", YYYY, MM, DD, dataset, dtype]
["updated", YYYY, MM, DD, dataset, dtype]
["observed", ...] from temporal.observed_at
["collected", ...] from temporal.collected_at
["valid_from", ...] when present
["valid_to", ...] when present
["generated", ...] for canonical manifest/pass generation timestamps where the schema explicitly provides one

Reduce _count.

No heuristic substitution: absence of a basis means no emitted row for that basis. Do not rewrite historical timestamps to make charts prettier.

This enables the Auto-Dig #2018 selector without full-corpus scans and preserves exact raw counts even when the UI uses log/symlog/aggregation presentation.

`latest_updated`

Map-only key [date_updated, _id] -> {dataset,dtype} or a minimal fixed projection. Descending range query supplies latest activity without sorting the corpus in application memory.

3. `graph-v1`

All relation fields must use canonical data paths with a narrowly tested compatibility fallback only for proven legacy records.

`outgoing`

For each relation subject endpoint emit:
[subject_id, predicate, object_id, dataset, relation_id] -> null.

`incoming`

[object_id, predicate, subject_id, dataset, relation_id] -> null.

Endpoint helper must support the canonical forms Auto-Dig already handles: string ID, {id: ...}, and arrays.

`predicate_counts`

[dataset, predicate] -> 1, reduce _count.

`degree`

Emit both directions using key [entity_id, direction, predicate, dataset] -> 1, reduce _count.
Callers can group by entity for topological ranking without scanning all relations.

`reviewed_degree`

Only emit relations whose normalized review state is reviewed. This directly supports Auto-Dig's “reviewed graph” / top-connected-people behavior.

`relation_state`

Emit [dataset, review_state, verification_status, predicate] -> 1 for graph QA.

Do not attempt transitive closure, path finding, contradiction detection, or entity resolution in CouchDB map/reduce. Those require graph/application logic.

4. `research-v1`

`targets_by_state`

For dtype === 'investigation-target', normalize state from workflow.research_status, then data.status, then top-level status.
Emit [dataset, state, priority_bucket, recursion_depth, _id] -> null.

Keep raw priority/score in a small value projection for frontier ranking; do not encode arbitrary prose.

`targets_by_run`

Emit [workflow.run_id || provenance.run_id, state, recursion_depth, _id] -> minimal target projection.

`target_counts`

Emit [dataset, state, recursion_depth] -> 1, reduce _count.

`research_passes`

Map-only [dataset, date_updated, _id] -> {run_id, unresolved_count, finding_count}.
Counts are derived only from arrays present in the same research-pass document.

`research_frontier`

For each explicit unresolved target ID present in a research-pass, emit [dataset, unresolved_target_id, pass_id] -> null.
This indexes explicit recorded frontier state; it must not infer missing work from absence.

5. `evidence-v1`

`source_domain`

For each canonical source URL/URI embedded on a document, emit [normalized_host, dataset, dtype] -> 1.
Reduce _count.
Host normalization must be deterministic and shared/tested; malformed URLs go to an explicit invalid bucket or are skipped with fixture coverage.

`source_kind`

Emit [dataset, source.kind || 'unspecified', dtype] -> 1.

`evidence_shape`

Emit [dataset, dtype, evidence_count_bucket, source_count_bucket] -> 1.
This supports corpus QA without trying to decide whether evidence is substantively sufficient.

`confidence`

Emit [dataset, dtype, normalized_confidence_bucket] -> 1 from canonical assessment.confidence when numeric.
No ranking claim is implied by the view.

6. `entities-v1`

`by_name`

For person/org and other name-bearing entity dtypes, use canonical data.name, falling back to title and only then proven legacy name components. Emit a normalized search key plus _id, but never overwrite the canonical display string.

`aliases`

Emit one row per explicit alias: [normalized_alias, dtype, _id] -> canonical_display_name.
This is for candidate lookup only, not automatic merging.

`datasets_for_entity`

Use explicit canonical records/relations only. Do not infer same-person identity across different IDs from matching names.

7. `manifests-v1`

`dataset_manifest`

Map dataset-manifest docs by [dataset, generated_at, _id] with minimal value containing record_count, counts_by_dtype, schema versions, actor/pipeline/run ID, and declared coverage status.

This gives operators a cheap way to compare manifest-declared materialization with live canonical counts, while treating discrepancies as a signal rather than silently trusting either side.

A — Adversarial review

Reject: one giant design document

CouchDB rebuilds indexes for views in a design doc when that design doc changes. Separate stable semantic families so changing experimental research projections does not force the core corpus index to rebuild. Keep manifest/versioning explicit.

Reject: custom reducers for rich objects

Prefer built-ins (_count, _sum, _stats only where truly needed). Reducers must stay associative/rereduce-safe and converge to small fixed output. PouchDB parity is a hard constraint.

Reject: full-document view values

At 1.25M+ corpus scale this needlessly bloats B-trees. Map values must be null, scalar, ID, or a small fixed projection. Consumers request include_docs when they need bodies.

Reject: “conflict detector” map/reduce

Cross-document contradictions, duplicate identity adjudication, stale-vs-current semantics, and claim/fact conflicts are application/Prolog/graph jobs. Views should only expose the indexed evidence needed to perform them.

Reject: silently treating all dates as event time

The observed Auto-Dig pulse failure proves this is misleading. Time-basis is explicit and queryable.

Reject: making public-mode visibility filtering optional

A generic /api/v1/views endpoint must not let an anonymous caller query a projection whose rows were built from non-public documents. Public API view access must be allowlisted and visibility-filtered or use public-only projections. Never rely on clients to filter returned rows.

Reject: topic-specific views

No flock_*, worldcoin_*, cloarida_*, etc. Those investigations prove the need for generic relation, lifecycle/time, evidence, target, and dataset projections.

D — Decision gate

Implement in this order

P0 correctness

versioned registry/manifest skeleton from #61;
v0.9 relation path repair;
snake_case date/meta repair;
canonical person/org name repair;
fixtures from real Auto-Dig records.

P1 corpus/dashboard
6. core-v1;
7. activity-v1 explicit time-basis view;
8. graph-v1 predicate/degree/reviewed-degree;
9. research-v1 target/pass/frontier projections.

P2 evidence/operator
10. evidence-v1;
11. entities-v1 aliases/name lookup;
12. manifests-v1 reconciliation surface.

P3 API/local parity
13. /api/v1/views allowlisted discovery/query contract;
14. Quasar/PouchDB installation of the same manifest;
15. warm-up/status/migration and old-index cleanup tooling.

TDD gate

Tests must be written before each implementation slice.

Fixture sources

Copy/redact-to-minimal deterministic fixtures from actual canonical v0.9 Auto-Dig shapes, including:

a person with only data.name;
a relation using data.subject/predicate/object;
a relation with endpoint objects/arrays;
an investigation-target with queued/expanded workflow state, priority and recursion depth;
a research-pass with explicit unresolved targets;
a dataset-manifest with counts_by_dtype and coverage status;
records with reviewed/unreviewed verification states;
records with each supported time basis;
a valid v0.9 snake_case envelope that bad_docs must not flag;
one legacy fixture for each compatibility fallback we intentionally retain.

Required proofs

current legacy relation views fail the v0.9 fixture before repair;
canonical relation projections return correct outgoing/incoming/predicate/degree rows;
group/group_level counts are identical in CouchDB and PouchDB;
every reducer is rereduce-safe;
no view value unexpectedly contains an entire document;
explicit time bases never substitute one timestamp for another;
a synthetic bulk-ingest day remains an exact raw count while normal days remain queryable;
private/non-public fixture rows cannot be obtained through anonymous view API calls;
map functions tolerate absent/malformed optional fields without aborting index builds;
version migration does not overwrite unknown operator/user design docs;
old registry version remains readable during migration until cutover completes.

Success criteria

Auto-Dig dashboard metrics can be reproduced from server views without a whole-corpus Python scan for the indexed dimensions.
#2018 can consume explicit activity bases rather than ambiguous date_added.
reviewed graph topology uses canonical relation paths and matches Auto-Dig fixture expectations.
recursive worker dashboards can query queued/expanded targets and latest research passes by dataset/run/depth.
public stats can consume registry views without opening non-public data.
Quasar/PouchDB produces equivalent keys and reduced results.
exact-head unit + CouchDB integration + PouchDB parity + API authorization tests are green.

Non-goals

graph pathfinding/transitive closure;
entity merge decisions;
contradiction adjudication;
full-text search replacement;
topic-specific business logic;
rewriting canonical Auto-Dig timestamps.

Parent: #61 Depends on API foundation: #58 Cross-repo evidence: lost-rob0t/starintel-gpt-auto-dig#2018 ## ADARD decision Replace the legacy/accidental view surface with a **versioned v0.9-native view registry derived from actual Auto-Dig corpus/query behavior**. Do not add topic-specific views. Build a compact set of generic projections that answer the queries Auto-Dig, Quasar, dashboards, recursive workers, and public stats actually perform. This issue is the implementation slice produced by the ADARD research/design loop. It must be TDD-first and exact-head verified. --- ## A — Analyze / corpus evidence ### Real corpus scale and pulse problem Auto-Dig issue #2018 records a 30-day corpus snapshot with **1,254,577 documents across only 13 active days**. Bulk materializations create huge spikes. The current dashboard groups only `date_added`, conflating first-seen/materialization/research-event semantics. The Auto-Dig corpus dashboard currently recomputes over the complete corpus in Python: - document counts and dtype composition; - dataset counts and 30-day activity; - relation predicate counts; - reviewed relation topology / top connected people; - unique source domains; - evidence-backed findings ranked from evidence/source/confidence/link counts; - latest reviewed activity; - active datasets. These are direct candidates for indexed server projections. ### Real v0.9 record shapes inspected Representative current canonical records show: - common envelope: `_id`, `dataset`, `dtype`, `date_added`, `date_updated`, `status`, `verification`, `handling`, `assessment`, `sources`, `related_ids`, `temporal`, `provenance`, `workflow`, `data`; - relation: `data.subject`, `data.predicate`, `data.object`, sometimes `data.confidence`; - investigation-target: `workflow.research_status`, `workflow.recursion_depth`, `workflow.priority`, `workflow.run_id`, plus `data.status`, `data.depth`, `data.priority`, `data.score`, `data.seed_ids`; - research-pass: selected-target/findings/support/counterevidence/unresolved-target structures under `data`; - dataset-manifest: `data.record_count`, `data.counts_by_dtype`, generation/run metadata and coverage status; - people/orgs generally expose canonical display identity through `data.name` / `title`, while older split-name fields are not guaranteed. ### Existing view defects against real data `source/views/data.json` currently contains several legacy-shape projections that do not correctly index canonical v0.9 Auto-Dig records: - `relations_by_predicate` reads `doc.predicate` instead of `doc.data.predicate`; - `relations_out_degree` reads `doc.source` instead of `doc.data.subject`; - `relations_in_degree` reads `doc.target` instead of `doc.data.object`; - `relations_self_loops` has the same legacy endpoint assumption; - `targets_by_actor` assumes `dtype === 'target'` plus top-level `actor`, which does not represent Auto-Dig `investigation-target` workflow records; - `date_added_stats_by_dataset` reads only `dateAdded`; - `bad_docs_missing_meta` treats snake-case `date_added` / `date_updated` as missing; - `persons/by_name` assumes `lname/fname`, while current migrated v0.9 people can carry only `data.name`; - multiple map views emit entire documents even though callers can request `include_docs`, increasing index size. This is correctness debt before it is optimization work. --- ## D — Design Create versioned design documents / manifest IDs. Exact naming may follow the registry machinery in #61, but semantic namespaces are fixed below. ### 1. `core-v1` #### `count_by_dtype` Map: `[dtype] -> 1`; reduce `_count` or `_sum`. #### `count_by_dataset` Map: `[dataset] -> 1`. #### `count_by_dataset_dtype` Map: `[dataset, dtype] -> 1`. Supports corpus composition globally or per dataset with `group_level`. #### `documents_by_dataset` Map-only `[dataset, _id] -> null`. Never emit the full document; use `include_docs=true` when required. #### `review_state` Normalize the same reviewed/unreviewed semantics Auto-Dig currently computes from: 1. `verification.status`; 2. `workflow.review_status` / workflow status; 3. top-level `status`. Emit `[dataset, normalized_review_state, dtype] -> 1`. The normalization rule must be a shared/generated contract, not independently drifting Python and JavaScript token lists. #### `verification_status` Emit `[dataset, verification.status || 'unspecified', dtype] -> 1`. Do not collapse detailed verification status into review_state only. #### `visibility` Emit `[handling.visibility || 'unspecified', dataset, dtype] -> 1`. This is mandatory for API-side authority filtering and public-mode safety. #### `source_presence` Emit `[dataset, has_sources|no_sources, dtype] -> 1`. ### 2. `activity-v1` Do **not** keep one ambiguous “documents by day” view. #### `by_time_basis` For every valid timestamp available on a document, emit one row with a bounded basis enum: - `["added", YYYY, MM, DD, dataset, dtype]` - `["updated", YYYY, MM, DD, dataset, dtype]` - `["observed", ...]` from `temporal.observed_at` - `["collected", ...]` from `temporal.collected_at` - `["valid_from", ...]` when present - `["valid_to", ...]` when present - `["generated", ...]` for canonical manifest/pass generation timestamps where the schema explicitly provides one Reduce `_count`. No heuristic substitution: absence of a basis means no emitted row for that basis. Do not rewrite historical timestamps to make charts prettier. This enables the Auto-Dig #2018 selector without full-corpus scans and preserves exact raw counts even when the UI uses log/symlog/aggregation presentation. #### `latest_updated` Map-only key `[date_updated, _id] -> {dataset,dtype}` or a minimal fixed projection. Descending range query supplies latest activity without sorting the corpus in application memory. ### 3. `graph-v1` All relation fields must use canonical `data` paths with a narrowly tested compatibility fallback only for proven legacy records. #### `outgoing` For each relation subject endpoint emit: `[subject_id, predicate, object_id, dataset, relation_id] -> null`. #### `incoming` `[object_id, predicate, subject_id, dataset, relation_id] -> null`. Endpoint helper must support the canonical forms Auto-Dig already handles: string ID, `{id: ...}`, and arrays. #### `predicate_counts` `[dataset, predicate] -> 1`, reduce `_count`. #### `degree` Emit both directions using key `[entity_id, direction, predicate, dataset] -> 1`, reduce `_count`. Callers can group by entity for topological ranking without scanning all relations. #### `reviewed_degree` Only emit relations whose normalized review state is reviewed. This directly supports Auto-Dig's “reviewed graph” / top-connected-people behavior. #### `relation_state` Emit `[dataset, review_state, verification_status, predicate] -> 1` for graph QA. Do not attempt transitive closure, path finding, contradiction detection, or entity resolution in CouchDB map/reduce. Those require graph/application logic. ### 4. `research-v1` #### `targets_by_state` For `dtype === 'investigation-target'`, normalize state from `workflow.research_status`, then `data.status`, then top-level status. Emit `[dataset, state, priority_bucket, recursion_depth, _id] -> null`. Keep raw priority/score in a small value projection for frontier ranking; do not encode arbitrary prose. #### `targets_by_run` Emit `[workflow.run_id || provenance.run_id, state, recursion_depth, _id] -> minimal target projection`. #### `target_counts` Emit `[dataset, state, recursion_depth] -> 1`, reduce `_count`. #### `research_passes` Map-only `[dataset, date_updated, _id] -> {run_id, unresolved_count, finding_count}`. Counts are derived only from arrays present in the same research-pass document. #### `research_frontier` For each explicit unresolved target ID present in a research-pass, emit `[dataset, unresolved_target_id, pass_id] -> null`. This indexes *explicit recorded frontier state*; it must not infer missing work from absence. ### 5. `evidence-v1` #### `source_domain` For each canonical source URL/URI embedded on a document, emit `[normalized_host, dataset, dtype] -> 1`. Reduce `_count`. Host normalization must be deterministic and shared/tested; malformed URLs go to an explicit invalid bucket or are skipped with fixture coverage. #### `source_kind` Emit `[dataset, source.kind || 'unspecified', dtype] -> 1`. #### `evidence_shape` Emit `[dataset, dtype, evidence_count_bucket, source_count_bucket] -> 1`. This supports corpus QA without trying to decide whether evidence is substantively sufficient. #### `confidence` Emit `[dataset, dtype, normalized_confidence_bucket] -> 1` from canonical `assessment.confidence` when numeric. No ranking claim is implied by the view. ### 6. `entities-v1` #### `by_name` For person/org and other name-bearing entity dtypes, use canonical `data.name`, falling back to `title` and only then proven legacy name components. Emit a normalized search key plus `_id`, but never overwrite the canonical display string. #### `aliases` Emit one row per explicit alias: `[normalized_alias, dtype, _id] -> canonical_display_name`. This is for candidate lookup only, not automatic merging. #### `datasets_for_entity` Use explicit canonical records/relations only. Do **not** infer same-person identity across different IDs from matching names. ### 7. `manifests-v1` #### `dataset_manifest` Map dataset-manifest docs by `[dataset, generated_at, _id]` with minimal value containing `record_count`, `counts_by_dtype`, schema versions, actor/pipeline/run ID, and declared coverage status. This gives operators a cheap way to compare manifest-declared materialization with live canonical counts, while treating discrepancies as a signal rather than silently trusting either side. --- ## A — Adversarial review ### Reject: one giant design document CouchDB rebuilds indexes for views in a design doc when that design doc changes. Separate stable semantic families so changing experimental research projections does not force the core corpus index to rebuild. Keep manifest/versioning explicit. ### Reject: custom reducers for rich objects Prefer built-ins (`_count`, `_sum`, `_stats` only where truly needed). Reducers must stay associative/rereduce-safe and converge to small fixed output. PouchDB parity is a hard constraint. ### Reject: full-document view values At 1.25M+ corpus scale this needlessly bloats B-trees. Map values must be `null`, scalar, ID, or a small fixed projection. Consumers request `include_docs` when they need bodies. ### Reject: “conflict detector” map/reduce Cross-document contradictions, duplicate identity adjudication, stale-vs-current semantics, and claim/fact conflicts are application/Prolog/graph jobs. Views should only expose the indexed evidence needed to perform them. ### Reject: silently treating all dates as event time The observed Auto-Dig pulse failure proves this is misleading. Time-basis is explicit and queryable. ### Reject: making public-mode visibility filtering optional A generic `/api/v1/views` endpoint must not let an anonymous caller query a projection whose rows were built from non-public documents. Public API view access must be allowlisted and visibility-filtered or use public-only projections. Never rely on clients to filter returned rows. ### Reject: topic-specific views No `flock_*`, `worldcoin_*`, `cloarida_*`, etc. Those investigations prove the need for generic relation, lifecycle/time, evidence, target, and dataset projections. --- ## D — Decision gate ### Implement in this order **P0 correctness** 1. versioned registry/manifest skeleton from #61; 2. v0.9 relation path repair; 3. snake_case date/meta repair; 4. canonical person/org name repair; 5. fixtures from real Auto-Dig records. **P1 corpus/dashboard** 6. `core-v1`; 7. `activity-v1` explicit time-basis view; 8. `graph-v1` predicate/degree/reviewed-degree; 9. `research-v1` target/pass/frontier projections. **P2 evidence/operator** 10. `evidence-v1`; 11. `entities-v1` aliases/name lookup; 12. `manifests-v1` reconciliation surface. **P3 API/local parity** 13. `/api/v1/views` allowlisted discovery/query contract; 14. Quasar/PouchDB installation of the same manifest; 15. warm-up/status/migration and old-index cleanup tooling. --- ## TDD gate Tests must be written before each implementation slice. ### Fixture sources Copy/redact-to-minimal deterministic fixtures from actual canonical v0.9 Auto-Dig shapes, including: - a person with only `data.name`; - a relation using `data.subject/predicate/object`; - a relation with endpoint objects/arrays; - an investigation-target with queued/expanded workflow state, priority and recursion depth; - a research-pass with explicit unresolved targets; - a dataset-manifest with `counts_by_dtype` and coverage status; - records with reviewed/unreviewed verification states; - records with each supported time basis; - a valid v0.9 snake_case envelope that `bad_docs` must not flag; - one legacy fixture for each compatibility fallback we intentionally retain. ### Required proofs - current legacy relation views fail the v0.9 fixture before repair; - canonical relation projections return correct outgoing/incoming/predicate/degree rows; - group/group_level counts are identical in CouchDB and PouchDB; - every reducer is rereduce-safe; - no view value unexpectedly contains an entire document; - explicit time bases never substitute one timestamp for another; - a synthetic bulk-ingest day remains an exact raw count while normal days remain queryable; - private/non-public fixture rows cannot be obtained through anonymous view API calls; - map functions tolerate absent/malformed optional fields without aborting index builds; - version migration does not overwrite unknown operator/user design docs; - old registry version remains readable during migration until cutover completes. --- ## Success criteria - Auto-Dig dashboard metrics can be reproduced from server views without a whole-corpus Python scan for the indexed dimensions. - #2018 can consume explicit activity bases rather than ambiguous `date_added`. - reviewed graph topology uses canonical relation paths and matches Auto-Dig fixture expectations. - recursive worker dashboards can query queued/expanded targets and latest research passes by dataset/run/depth. - public stats can consume registry views without opening non-public data. - Quasar/PouchDB produces equivalent keys and reduced results. - exact-head unit + CouchDB integration + PouchDB parity + API authorization tests are green. ## Non-goals - graph pathfinding/transitive closure; - entity merge decisions; - contradiction adjudication; - full-text search replacement; - topic-specific business logic; - rewriting canonical Auto-Dig timestamps.

lost-rob0t commented

2026-08-24 08:59:56 +00:00

(Migrated from github.com)

Architecture correction — generalized core only

Superseding any ambiguous interpretation of this issue:

starintel-server owns only generalized, schema-level, reusable StarIntel projections that make sense across arbitrary valid v0.9 corpora.
Dataset-specific, investigation-specific, deployment-specific, experimental, or high-cost custom CouchDB design docs do not belong in server core.
Those custom views are opt-in installable deployment artifacts owned by starintel-labs/starintel-infra under infra issue #36.
Auto-Dig data is evidence for discovering generic query shapes and validating field paths/scale. It is not permission to bake Auto-Dig topic/business-specific projections into the server.

Examples that stay core: dtype/dataset counts, canonical relation incoming/outgoing/degree, explicit timestamp bases, generic verification/review/visibility, generic target/pass state where those are canonical dtypes, generic name/alias/source summaries.

Examples that move to infra packs: Flock lifecycle/sharing summaries, campaign-finance reconciliation, Palantir contract dashboards, per-deployment operator queues, dataset-specific materializations, and other domain/report-specific indexes.

The API registry must distinguish built-in core views from installed custom packs; installation alone must not make a custom view public.

## Architecture correction — generalized core only Superseding any ambiguous interpretation of this issue: - `starintel-server` owns only generalized, schema-level, reusable StarIntel projections that make sense across arbitrary valid v0.9 corpora. - Dataset-specific, investigation-specific, deployment-specific, experimental, or high-cost custom CouchDB design docs do **not** belong in server core. - Those custom views are opt-in installable deployment artifacts owned by `starintel-labs/starintel-infra` under infra issue #36. - Auto-Dig data is evidence for discovering *generic query shapes* and validating field paths/scale. It is **not** permission to bake Auto-Dig topic/business-specific projections into the server. Examples that stay core: dtype/dataset counts, canonical relation incoming/outgoing/degree, explicit timestamp bases, generic verification/review/visibility, generic target/pass state where those are canonical dtypes, generic name/alias/source summaries. Examples that move to infra packs: Flock lifecycle/sharing summaries, campaign-finance reconciliation, Palantir contract dashboards, per-deployment operator queues, dataset-specific materializations, and other domain/report-specific indexes. The API registry must distinguish built-in core views from installed custom packs; installation alone must not make a custom view public.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

nsaspy/starintel-server#114

No description provided.

Rows
Columns

Generalized Views v1: v0.9-native reusable projections and legacy field repair #114

ADARD decision

A — Analyze / corpus evidence

Real corpus scale and pulse problem

Real v0.9 record shapes inspected

Existing view defects against real data

D — Design

1. core-v1

count_by_dtype

count_by_dataset

count_by_dataset_dtype

documents_by_dataset

review_state

verification_status

visibility

source_presence

2. activity-v1

by_time_basis

latest_updated

3. graph-v1

outgoing

incoming

predicate_counts

degree

reviewed_degree

relation_state

4. research-v1

targets_by_state

targets_by_run

target_counts

research_passes

research_frontier

5. evidence-v1

source_domain

source_kind

evidence_shape

confidence

6. entities-v1

by_name

aliases

datasets_for_entity

7. manifests-v1

dataset_manifest