Modernize fediWatch for StarIntel 0.9 #1

2026-07-31T00:41:35Z

lost-rob0t commented

2026-07-31 00:41:35 +00:00

(Migrated from github.com)

What changed

keep fediWatch in Nim and move the build/runtime baseline to Nim 2.2
replace the legacy StarIntel 0.6 flat objects with strict StarIntel 0.9 user, social-media-post, and relation envelopes
generate deterministic SHA-256 document IDs with source, provenance, temporal, and required envelope metadata
represent boosts as reposted relations instead of falsely attributing original content to the boosting account
replace ZeroMQ/starRouter output with runtime-selectable StarIntel HTTP bulk ingestion or RabbitMQ topic publishing
load targets from inline JSON, files, or the StarIntel targets API, accepting both legacy payloads and StarIntel 0.9 target envelopes
advance Mastodon cursors only after complete transport acceptance for at-least-once replay
add bounded batching/retries, optional strict schema validation, verified AMQPS, mandatory routing, and RabbitMQ publisher confirms
modernize Docker, Nix, environment examples, and operational documentation
relicense the repository under AGPL-3.0-only with the complete canonical AGPLv3 text

Dependency modernization

The branch pins the companion Nim 2 Mastodon client modernization from lost-rob0t/fedi. That rewrite fixes malformed endpoint URLs, premature shared-client closure, unsafe rate-limit parsing, dropped headers, unencoded account lookups, numeric cursor handling, non-JSON errors, and the obsolete query-macro dependency stack.

RabbitMQ output uses a small Nim/C bridge over maintained rabbitmq-c instead of the broken third-party Nim AMQP package. It supports amqp:// and verified amqps:// connections across rabbitmq-c 0.11 and newer.

Validation

GitHub Actions validates:

StarIntel 0.9 document and boost/repost semantics
RabbitMQ URL and TLS configuration parsing
Nim 2.2.10 unit tests
release compilation
live routed RabbitMQ publishing with publisher confirms
container image construction

## What changed - keep fediWatch in Nim and move the build/runtime baseline to Nim 2.2 - replace the legacy StarIntel 0.6 flat objects with strict StarIntel 0.9 `user`, `social-media-post`, and `relation` envelopes - generate deterministic SHA-256 document IDs with source, provenance, temporal, and required envelope metadata - represent boosts as `reposted` relations instead of falsely attributing original content to the boosting account - replace ZeroMQ/starRouter output with runtime-selectable StarIntel HTTP bulk ingestion or RabbitMQ topic publishing - load targets from inline JSON, files, or the StarIntel targets API, accepting both legacy payloads and StarIntel 0.9 target envelopes - advance Mastodon cursors only after complete transport acceptance for at-least-once replay - add bounded batching/retries, optional strict schema validation, verified AMQPS, mandatory routing, and RabbitMQ publisher confirms - modernize Docker, Nix, environment examples, and operational documentation - relicense the repository under `AGPL-3.0-only` with the complete canonical AGPLv3 text ## Dependency modernization The branch pins the companion Nim 2 Mastodon client modernization from `lost-rob0t/fedi`. That rewrite fixes malformed endpoint URLs, premature shared-client closure, unsafe rate-limit parsing, dropped headers, unencoded account lookups, numeric cursor handling, non-JSON errors, and the obsolete query-macro dependency stack. RabbitMQ output uses a small Nim/C bridge over maintained `rabbitmq-c` instead of the broken third-party Nim AMQP package. It supports `amqp://` and verified `amqps://` connections across rabbitmq-c 0.11 and newer. ## Validation GitHub Actions validates: - StarIntel 0.9 document and boost/repost semantics - RabbitMQ URL and TLS configuration parsing - Nim 2.2.10 unit tests - release compilation - live routed RabbitMQ publishing with publisher confirms - container image construction

lost-rob0t (Migrated from github.com) reviewed

2026-08-13 03:53:27 +00:00

lost-rob0t (Migrated from github.com) left a comment

Review found two merge-blocking correctness issues. CI is green, but this head should not merge until pagination and federated post identity are fixed.

src/fediwatch/documents.nim

					
				@ -0,0 +124,4 @@

				proc postId*(status: JsonNode, instanceHost: string): string =

				  stableId(

				    "social-media-post",

				    normalizeHost(instanceHost) & ":" & status{"id"}.getStr(

Blocker for federated deduplication: Mastodon status.id is local to the instance serving the API, so hashing instanceHost:id gives different StarIntel _ids for the same federated post observed through different instances. The canonical ActivityPub uri should be the primary identity key (then canonical url, with instanceHost:id only as a last-resort fallback). The same identity rule should be used for reply/quote/reblog relation targets so they resolve to the same post document across observers. Please add a cross-instance test proving the same uri yields the same _id.

**Blocker for federated deduplication:** Mastodon `status.id` is local to the instance serving the API, so hashing `instanceHost:id` gives different StarIntel `_id`s for the same federated post observed through different instances. The canonical ActivityPub `uri` should be the primary identity key (then canonical `url`, with `instanceHost:id` only as a last-resort fallback). The same identity rule should be used for reply/quote/reblog relation targets so they resolve to the same post document across observers. Please add a cross-instance test proving the same `uri` yields the same `_id`.

src/fediwatch/service.nim

					
				@ -0,0 +72,4 @@

				      remote = watcher.target.remote,

				      onlyMedia = watcher.target.onlyMedia,

				      limit = watcher.target.limit,

				      sinceId = watcher.sinceId

Blocker: this can silently drop statuses under normal instance traffic. Both pinned fedi.getTimeline and getStatuses support maxId, but this fetches only one page (max 40) newer than sinceId, then poll() advances the cursor to timeline[0].id. If >40 statuses arrive between polls, the older unseen statuses are never fetched and become permanently skipped. Please paginate all pages newer than the committed sinceId (using maxId while retaining the original lower bound), then advance the cursor only after the full paginated result has been emitted successfully. Add a test with >1 page of new statuses.

**Blocker: this can silently drop statuses under normal instance traffic.** Both pinned `fedi.getTimeline` and `getStatuses` support `maxId`, but this fetches only one page (max 40) newer than `sinceId`, then `poll()` advances the cursor to `timeline[0].id`. If >40 statuses arrive between polls, the older unseen statuses are never fetched and become permanently skipped. Please paginate all pages newer than the committed `sinceId` (using `maxId` while retaining the original lower bound), then advance the cursor only after the full paginated result has been emitted successfully. Add a test with >1 page of new statuses.

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin agent/modernize-starintel-0.9:agent/modernize-starintel-0.9

git switch agent/modernize-starintel-0.9

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master

git merge --no-ff agent/modernize-starintel-0.9

git switch agent/modernize-starintel-0.9

git rebase master

git switch master

git merge --ff-only agent/modernize-starintel-0.9

git switch agent/modernize-starintel-0.9

git rebase master

git switch master

git merge --no-ff agent/modernize-starintel-0.9

git switch master

git merge --squash agent/modernize-starintel-0.9