Add safe direct text tokenization to CouchDB Prolog views #29

Merged

lost-rob0t merged 3 commits from agent/direct-social-token-dsl into main

2026-08-13 05:53:55 +00:00

lost-rob0t commented

2026-08-13 05:48:09 +00:00

(Migrated from github.com)

Implements #28 for direct StarIntel SocialMPost.content indexing.

What changed

adds bounded normalize_text/1, lower/1, tokens/1, for_each/2, item/item(Path), and whitelisted UTC time_part/2 primitives;
preserves ground-only parsing and the existing arbitrary-call rejection boundary;
caps iterator depth, collection size, source text, tokens, and map emissions;
strips Mastodon-style HTML and decodes common entities before tokenization;
keeps existing reducers unchanged;
adds deterministic tests for repeated words, case, punctuation, HTML, item paths, time keys, filtering, and resource limits.

This intentionally does not add support for precomputed word_counts; Star Funds can derive mention rows directly from the existing social-post content.

Validation

CI is authoritative for this branch because the current ChatGPT execution environment does not have a local Nix/SWI checkout available. Merge only after required checks are green.

Implements #28 for direct StarIntel `SocialMPost.content` indexing. ### What changed - adds bounded `normalize_text/1`, `lower/1`, `tokens/1`, `for_each/2`, `item`/`item(Path)`, and whitelisted UTC `time_part/2` primitives; - preserves ground-only parsing and the existing arbitrary-call rejection boundary; - caps iterator depth, collection size, source text, tokens, and map emissions; - strips Mastodon-style HTML and decodes common entities before tokenization; - keeps existing reducers unchanged; - adds deterministic tests for repeated words, case, punctuation, HTML, item paths, time keys, filtering, and resource limits. This intentionally does **not** add support for precomputed `word_counts`; Star Funds can derive mention rows directly from the existing social-post content. ### Validation CI is authoritative for this branch because the current ChatGPT execution environment does not have a local Nix/SWI checkout available. Merge only after required checks are green.