starintel-labs/starintel-gpt-auto-dig
3
0
Fork 0
Code Issues 998 Pull requests 33 Projects Releases Packages Wiki Activity Actions 2

[GH-#1893] Import 13k+ Global Shapers and alumni profiles #1019

Open
nsaspy wants to merge 68 commits from agent/global-shapers-13k-directory into main
pull from: agent/global-shapers-13k-directory
merge into: starintel-labs:main
Conversation 0 Commits 68 Files changed 47 +349366 -233
nsaspy commented 2026-09-08 17:06:39 +00:00
Owner
Copy link

Mirrored from GitHub https://github.com/lost-rob0t/starintel-gpt-auto-dig/pull/1893 (GitHub is authoritative for this item).

Goal

Recover at least 13,000 distinct canonical Global Shapers / member / alumni identities and generated bio pages from official public and archival Global Shapers surfaces, with provenance and dedupe strong enough to survive the repository gate.

Do not synthesize missing people, infer alumni from weak name matches, or treat inaccessible archive surfaces as evidence of absence.

Verified baseline

The old historical 4,062-person transport is malformed/truncated. Merged replacement PR #1956 established the reproducible baseline:

  • 2,736 verified historical people are already safely on main;
  • the PR's current public API packet contains 4,652 unique people;
  • reconciliation therefore starts from 7,388 person documents across those two sources before cross-source dedupe;
  • the 13,000-person completion gate remains unmet.

The branch also contains archive/profile discovery material, but archive URLs are evidence/recovery inputs, not proof of distinct people by themselves.

Current work

Current head: 6d2c78ebf499975496a7d4d106015e282b3c0f3a.

Identity reconciliation repaired

The previous reconciler could treat a shared generic official URL as a strong identity key and collapse unrelated people into one component. That result was invalid.

Current head now:

  • only treats person-specific Global Shapers / WEF profile URLs as strong official URL keys;
  • only treats personal LinkedIn /in/ or /pub/ URLs as strong LinkedIn keys;
  • keeps generic organization/community URLs as provenance without using them for identity union;
  • suppresses and reports high-fanout official/LinkedIn key collisions spanning multiple distinct names;
  • includes focused regression tests for the generic-URL collapse and collision cases;
  • runs those regression tests from the reconciliation workflow before rebuilding the ledger.

Publishing CI paused

Publishing is intentionally disabled for now.

  • main commit 950778074cc706dfe5a52b0c1a41517fde1e2cef replaced the Pages/release publisher with an inert, read-only workflow stub;
  • this PR branch carries the same paused publisher;
  • pushes no longer trigger that publishing workflow;
  • a manual dispatch also skips the disabled publishing job;
  • ordinary validation/import/research workflows are not intentionally disabled by this change.

Remaining blockers

This PR stays draft.

  • the branch is currently 66 commits ahead and 455 commits behind main and needs current-main reconciliation before merge;
  • live/archival recovery still has to produce and validate at least 13,000 distinct canonical people;
  • generated person pages must meet the same ≥13k floor;
  • the new exact head currently has no attached workflow runs or commit statuses; historical runs on the old head were action_required;
  • the full current repository validation/site gates still need to pass on the exact reconciled head.

No mercy merge: current-main sync, ≥13k validated corpus/pages, and green exact-head validation remain required.

Mirrored from GitHub https://github.com/lost-rob0t/starintel-gpt-auto-dig/pull/1893 (GitHub is authoritative for this item). --- ## Goal Recover at least **13,000 distinct canonical Global Shapers / member / alumni identities and generated bio pages** from official public and archival Global Shapers surfaces, with provenance and dedupe strong enough to survive the repository gate. Do not synthesize missing people, infer alumni from weak name matches, or treat inaccessible archive surfaces as evidence of absence. ## Verified baseline The old historical 4,062-person transport is malformed/truncated. Merged replacement PR #1956 established the reproducible baseline: - **2,736** verified historical people are already safely on `main`; - the PR's current public API packet contains **4,652** unique people; - reconciliation therefore starts from **7,388** person documents across those two sources before cross-source dedupe; - the **13,000-person completion gate remains unmet**. The branch also contains archive/profile discovery material, but archive URLs are evidence/recovery inputs, not proof of distinct people by themselves. ## Current work Current head: `6d2c78ebf499975496a7d4d106015e282b3c0f3a`. ### Identity reconciliation repaired The previous reconciler could treat a shared generic official URL as a strong identity key and collapse unrelated people into one component. That result was invalid. Current head now: - only treats **person-specific Global Shapers / WEF profile URLs** as strong official URL keys; - only treats **personal LinkedIn `/in/` or `/pub/` URLs** as strong LinkedIn keys; - keeps generic organization/community URLs as provenance without using them for identity union; - suppresses and reports high-fanout official/LinkedIn key collisions spanning multiple distinct names; - includes focused regression tests for the generic-URL collapse and collision cases; - runs those regression tests from the reconciliation workflow before rebuilding the ledger. ### Publishing CI paused Publishing is intentionally disabled for now. - `main` commit `950778074cc706dfe5a52b0c1a41517fde1e2cef` replaced the Pages/release publisher with an inert, read-only workflow stub; - this PR branch carries the same paused publisher; - pushes no longer trigger that publishing workflow; - a manual dispatch also skips the disabled publishing job; - ordinary validation/import/research workflows are not intentionally disabled by this change. ## Remaining blockers This PR stays **draft**. - the branch is currently **66 commits ahead and 455 commits behind `main`** and needs current-main reconciliation before merge; - live/archival recovery still has to produce and validate at least **13,000 distinct canonical people**; - generated person pages must meet the same ≥13k floor; - the new exact head currently has **no attached workflow runs or commit statuses**; historical runs on the old head were `action_required`; - the full current repository validation/site gates still need to pass on the exact reconciled head. No mercy merge: current-main sync, ≥13k validated corpus/pages, and green exact-head validation remain required.
Sync publishing pause with main
Some checks failed
StarIntel conformance / matrix (pull_request) Failing after 8s
Details
Enumerate Global Shapers archives / enumerate (pull_request) Failing after 4m10s
Details
Discover Global Shapers frontend data / discover (pull_request) Failing after 4m17s
Details
Import current Global Shapers API / import (pull_request) Failing after 1m46s
Details
Import complete Global Shapers directory / import (pull_request) Failing after 7m51s
Details
Reconcile Global Shapers identities / reconcile (pull_request) Failing after 1m5s
Details
People directory / build (pull_request) Failing after 8m23s
Details
Validate StarIntel DB / validate (pull_request) Successful in 1m21s
Details
Validate StarIntel DB / Standalone Python v0.9 (3.10) (pull_request) Failing after 40s
Details
Validate StarIntel DB / Nim v0.9 (pull_request) Failing after 6s
Details
Validate StarIntel DB / Standalone Python v0.9 (3.12) (pull_request) Failing after 36s
Details
6d2c78ebf4
This pull request has changes conflicting with the target branch.
  • reports/wef-global-shapers-intact-prefix.json
  • reports/wef-global-shapers-intact-prefix.md
  • scripts/build_research_site.py
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/global-shapers-13k-directory:agent/global-shapers-13k-directory
git switch agent/global-shapers-13k-directory

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff agent/global-shapers-13k-directory
git switch agent/global-shapers-13k-directory
git rebase main
git switch main
git merge --ff-only agent/global-shapers-13k-directory
git switch agent/global-shapers-13k-directory
git rebase main
git switch main
git merge --no-ff agent/global-shapers-13k-directory
git switch main
git merge --squash agent/global-shapers-13k-directory
git switch main
git merge --ff-only agent/global-shapers-13k-directory
git switch main
git merge agent/global-shapers-13k-directory
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
github-mirror
Mirrored from GitHub (authoritative host)

No labels
github-mirror
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starintel-labs/starintel-gpt-auto-dig!1019
No description provided.