Add a bounded StarIntel search adapter for SearXNG/YaCy #123

New issue

Open

opened 2026-08-25 09:23:37 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-25 09:23:37 +00:00

(Migrated from github.com)

Goal

Integrate the new search plane into starintel-server without turning external search/crawl data into implicit StarIntel ingest.

Dependencies

starintel-infra #58 search host
starintel-infra #59 YaCy
starintel-infra #60 SearXNG

Requirements

Add a narrow service adapter for the private SearXNG endpoint; use YaCy directly only when the design proves a need rather than duplicating engine logic.
Keep current /api/v1/search compatibility in mind; research whether this should extend that contract or add a distinct federated-search capability.
Bounded timeout, result count, response size, retry, and concurrency budgets.
Normalize result metadata while retaining source URL/engine/provenance information.
Search results are ephemeral search-plane results by default. They are not written to CouchDB/StarIntel datasets unless an explicit ingest operation/actor promotes selected material through normal provenance and authorization rules.
No private StarIntel query context, credentials, internal hostnames, or restricted document content may leak to external engines.
Add capability discovery/versioning if a new API surface is introduced.
Emit structured logging through the #121 event contract without recording raw sensitive search queries by default.

Tests

Success/mixed-engine result normalization fixtures.
SearXNG timeout/unavailable/malformed/oversized response tests.
Prove search does not mutate ingest/document state.
Authorization and query-redaction tests.
Exact-head service-backed integration against a controlled SearXNG fixture/service.

Execution mode — AUTO-RAGE

Run the full ADARD/ADADR loop autonomously: inspect current search/API contracts and infra dependencies; design and record the integration boundary; adversarial review + privacy/SSRF/data-provenance threat model; autonomous decision gate; TDD-first implementation; exact-head unit/integration verification; push a focused branch and open/update a PR targeting master. Stop only for a genuinely unresolved requirement/security blocker, inaccessible dependency, or failed verification gate. Do not ask for approval between phases.

## Goal Integrate the new search plane into `starintel-server` without turning external search/crawl data into implicit StarIntel ingest. ## Dependencies - `starintel-infra` #58 search host - `starintel-infra` #59 YaCy - `starintel-infra` #60 SearXNG ## Requirements - Add a narrow service adapter for the private SearXNG endpoint; use YaCy directly only when the design proves a need rather than duplicating engine logic. - Keep current `/api/v1/search` compatibility in mind; research whether this should extend that contract or add a distinct federated-search capability. - Bounded timeout, result count, response size, retry, and concurrency budgets. - Normalize result metadata while retaining source URL/engine/provenance information. - Search results are **ephemeral search-plane results by default**. They are not written to CouchDB/StarIntel datasets unless an explicit ingest operation/actor promotes selected material through normal provenance and authorization rules. - No private StarIntel query context, credentials, internal hostnames, or restricted document content may leak to external engines. - Add capability discovery/versioning if a new API surface is introduced. - Emit structured logging through the #121 event contract without recording raw sensitive search queries by default. ## Tests - Success/mixed-engine result normalization fixtures. - SearXNG timeout/unavailable/malformed/oversized response tests. - Prove search does not mutate ingest/document state. - Authorization and query-redaction tests. - Exact-head service-backed integration against a controlled SearXNG fixture/service. ## Execution mode — AUTO-RAGE Run the full ADARD/ADADR loop autonomously: inspect current search/API contracts and infra dependencies; design and record the integration boundary; adversarial review + privacy/SSRF/data-provenance threat model; autonomous decision gate; TDD-first implementation; exact-head unit/integration verification; push a focused branch and open/update a PR targeting `master`. Stop only for a genuinely unresolved requirement/security blocker, inaccessible dependency, or failed verification gate. Do not ask for approval between phases.