starintel-labs/starintel-gpt-auto-dig
3
0
Fork 0
Code Issues 998 Pull requests 33 Projects Releases Packages Wiki Activity Actions 2

[GH-#1985] Speed Pages builds with Nim materialization #1016

Open
nsaspy wants to merge 13 commits from agent/nim-site-builder-performance into main
pull from: agent/nim-site-builder-performance
merge into: starintel-labs:main
Conversation 0 Commits 13 Files changed 5 +269 -5
nsaspy commented 2026-09-08 17:06:37 +00:00
Owner
Copy link

Mirrored from GitHub https://github.com/lost-rob0t/starintel-gpt-auto-dig/pull/1985 (GitHub is authoritative for this item).

What changed

  • adds a compiled Nim materializer for the corpus-scale db/*/*.ndjson -> site-input grouping pass
  • integrates the Nim fast path directly into the canonical scripts/build_research_site.py
  • requires the canonical Python corpus validator to pass before native materialization is allowed
  • makes the exact python3 scripts/validate-for-merge.py --site gate use the native materializer when available
  • removes the duplicate full-site build from the Pages workflow: validate once, build once
  • adds a PR workflow that compiles Nim and times the exact complete merge gate against the current corpus

Why

The current corpus is ~341k documents. The old Pages path validates the corpus, performs serial CPython DB materialization, builds the full site during the merge gate, and then builds the full site again for publication.

Safety

The native path is only enabled after validate_repository(..., require_v090=True) succeeds. Direct/local builds without the trusted-corpus flag retain the existing Python materializer. Existing full-site, artifact, size-budget, and evidence-seal checks remain in place.

Validation

  • the first Actions attempt compiled the Nim materializer successfully
  • the current Validate Nim site build run executes the repository-required python3 scripts/validate-for-merge.py --site command through the native path
  • PR remains draft until that complete gate is green
Mirrored from GitHub https://github.com/lost-rob0t/starintel-gpt-auto-dig/pull/1985 (GitHub is authoritative for this item). --- ## What changed - adds a compiled Nim materializer for the corpus-scale `db/*/*.ndjson` -> site-input grouping pass - integrates the Nim fast path directly into the canonical `scripts/build_research_site.py` - requires the canonical Python corpus validator to pass before native materialization is allowed - makes the exact `python3 scripts/validate-for-merge.py --site` gate use the native materializer when available - removes the duplicate full-site build from the Pages workflow: validate once, build once - adds a PR workflow that compiles Nim and times the exact complete merge gate against the current corpus ## Why The current corpus is ~341k documents. The old Pages path validates the corpus, performs serial CPython DB materialization, builds the full site during the merge gate, and then builds the full site again for publication. ## Safety The native path is only enabled after `validate_repository(..., require_v090=True)` succeeds. Direct/local builds without the trusted-corpus flag retain the existing Python materializer. Existing full-site, artifact, size-budget, and evidence-seal checks remain in place. ## Validation - the first Actions attempt compiled the Nim materializer successfully - the current `Validate Nim site build` run executes the repository-required `python3 scripts/validate-for-merge.py --site` command through the native path - PR remains draft until that complete gate is green
track canonical builder in nim gate
Some checks failed
Validate Nim site build / build (pull_request) Failing after 1m44s
Details
4292aed0e6
Some checks failed
Validate Nim site build / build (pull_request) Failing after 1m44s
Details
This pull request has changes conflicting with the target branch.
  • .github/workflows/publish-pages.yml
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/nim-site-builder-performance:agent/nim-site-builder-performance
git switch agent/nim-site-builder-performance

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff agent/nim-site-builder-performance
git switch agent/nim-site-builder-performance
git rebase main
git switch main
git merge --ff-only agent/nim-site-builder-performance
git switch agent/nim-site-builder-performance
git rebase main
git switch main
git merge --no-ff agent/nim-site-builder-performance
git switch main
git merge --squash agent/nim-site-builder-performance
git switch main
git merge --ff-only agent/nim-site-builder-performance
git switch main
git merge agent/nim-site-builder-performance
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
github-mirror
Mirrored from GitHub (authoritative host)

No labels
github-mirror
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starintel-labs/starintel-gpt-auto-dig!1016
No description provided.