[P1] Add safe bounded text/token iteration to native CouchDB map DSL #28

New issue

Open

opened 2026-08-13 05:15:51 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-08-13 05:15:51 +00:00

(Migrated from github.com)

Goal

Extend the native CouchDB language: "prolog" map DSL so a design view can safely tokenize existing text fields and emit one row per lexical occurrence. The immediate consumer is Star Funds social mention counting over StarIntel SocialMPost.content.

This supersedes the earlier word_counts-array design. Do not require precomputed word-count documents. The view should derive counts directly from the source document.

Required safe DSL

The minimal surface should support expressions/actions equivalent to:

normalize_text(field("content"))
lower(Expr)
tokens(Expr)
for_each(CollectionExpr, Action)
item
item(Path)
time_part("year", field("date_updated"))
time_part("month", field("date_updated"))
time_part("day", field("date_updated"))
time_part("hour", field("date_updated"))

Example:

where(
  eq(field("dtype"), "SocialMPost"),
  for_each(
    tokens(lower(normalize_text(field("content")))),
    emit(item, 1)
  )
).

Time-keyed example:

where(
  eq(field("dtype"), "SocialMPost"),
  for_each(
    tokens(lower(normalize_text(field("content")))),
    emit(array([
      item,
      time_part("year", field("date_updated")),
      time_part("month", field("date_updated")),
      time_part("day", field("date_updated")),
      time_part("hour", field("date_updated"))
    ]), 1)
  )
).

date_updated is used by the Star Funds view as initial knowledge/ingest time; provider-authored time remains a separate source field/semantic.

Token semantics

deterministic, locale-independent-enough behavior suitable for indexing;
normalize Mastodon HTML content to visible text before tokenization;
lowercase before tokenization;
split on non-alphanumeric boundaries;
preserve repeated occurrences ("Trump trump" emits two trump rows);
punctuation does not change lexical identity ("oil, oil!" emits two oil rows);
bound input length and token count.

Security requirements

DSL remains ground; variables are rejected.
No arbitrary call/1, dynamic predicate invocation, source execution, shell access, file access, or executable add_lib behavior.
item/item(Path) is only valid under iterator scope.
for_each/2 collection must evaluate to a JSON/Prolog list.
Bound iterator nesting depth.
Bound normalized text length.
Bound tokens and maximum emissions per source document/map function.
time_part/2 accepts only a fixed unit whitelist and numeric Unix timestamps.
Deterministic errors on invalid item/path/type/limit.
Preserve existing arbitrary-call rejection tests.

Reduce compatibility

Map output must continue to work unchanged with existing sum., count., min., max., and stats. reduce/rereduce behavior.

Tests

direct text tokenization emits one row for every occurrence;
case normalization;
punctuation normalization;
Mastodon-style HTML normalization;
scalar/object collection iteration;
item(Path) lookup;
time-part extraction in UTC;
surrounding where conditions;
reject item outside for_each/2;
reject variables/callable terms;
enforce nesting/token/text/emission limits;
reducer/rereduce compatibility;
existing CI/conflict/replication/observability suites remain green.

Done when

A CouchDB Prolog view can derive [word,...] -> 1 rows directly from SocialMPost.content, safely and deterministically, without precomputed word_counts documents or executable Prolog from CouchDB documents.

## Goal Extend the native CouchDB `language: "prolog"` map DSL so a design view can safely tokenize existing text fields and emit one row per lexical occurrence. The immediate consumer is Star Funds social mention counting over StarIntel `SocialMPost.content`. This supersedes the earlier `word_counts`-array design. **Do not require precomputed word-count documents.** The view should derive counts directly from the source document. ## Required safe DSL The minimal surface should support expressions/actions equivalent to: ```prolog normalize_text(field("content")) lower(Expr) tokens(Expr) for_each(CollectionExpr, Action) item item(Path) time_part("year", field("date_updated")) time_part("month", field("date_updated")) time_part("day", field("date_updated")) time_part("hour", field("date_updated")) ``` Example: ```prolog where( eq(field("dtype"), "SocialMPost"), for_each( tokens(lower(normalize_text(field("content")))), emit(item, 1) ) ). ``` Time-keyed example: ```prolog where( eq(field("dtype"), "SocialMPost"), for_each( tokens(lower(normalize_text(field("content")))), emit(array([ item, time_part("year", field("date_updated")), time_part("month", field("date_updated")), time_part("day", field("date_updated")), time_part("hour", field("date_updated")) ]), 1) ) ). ``` `date_updated` is used by the Star Funds view as initial knowledge/ingest time; provider-authored time remains a separate source field/semantic. ## Token semantics - deterministic, locale-independent-enough behavior suitable for indexing; - normalize Mastodon HTML content to visible text before tokenization; - lowercase before tokenization; - split on non-alphanumeric boundaries; - preserve repeated occurrences (`"Trump trump"` emits two `trump` rows); - punctuation does not change lexical identity (`"oil, oil!"` emits two `oil` rows); - bound input length and token count. ## Security requirements - DSL remains ground; variables are rejected. - No arbitrary `call/1`, dynamic predicate invocation, source execution, shell access, file access, or executable `add_lib` behavior. - `item`/`item(Path)` is only valid under iterator scope. - `for_each/2` collection must evaluate to a JSON/Prolog list. - Bound iterator nesting depth. - Bound normalized text length. - Bound tokens and maximum emissions per source document/map function. - `time_part/2` accepts only a fixed unit whitelist and numeric Unix timestamps. - Deterministic errors on invalid item/path/type/limit. - Preserve existing arbitrary-call rejection tests. ## Reduce compatibility Map output must continue to work unchanged with existing `sum.`, `count.`, `min.`, `max.`, and `stats.` reduce/rereduce behavior. ## Tests - direct text tokenization emits one row for every occurrence; - case normalization; - punctuation normalization; - Mastodon-style HTML normalization; - scalar/object collection iteration; - `item(Path)` lookup; - time-part extraction in UTC; - surrounding `where` conditions; - reject `item` outside `for_each/2`; - reject variables/callable terms; - enforce nesting/token/text/emission limits; - reducer/rereduce compatibility; - existing CI/conflict/replication/observability suites remain green. ## Done when A CouchDB Prolog view can derive `[word,...] -> 1` rows directly from `SocialMPost.content`, safely and deterministically, without precomputed `word_counts` documents or executable Prolog from CouchDB documents.