[GH-#139] Enforce one scraper issue per discovered membership-list URL #995

New issue

Open

opened 2026-09-08 17:05:43 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-08 17:05:43 +00:00

Owner

Mirrored from GitHub https://github.com/lost-rob0t/starintel-gpt-auto-dig/issues/139 (GitHub is authoritative for this item).

Invariant

Every organization scraper must record every encountered official public URL that exposes a membership, participant, expert, fellow, board, leadership, staff, trustee, delegate, committee, council, or comparable roster list surface.

For each canonical list URL, AutoDig must open exactly one deduplicated GitHub issue containing:

organization and dataset
canonical source URL
discovery source
parser/coverage status
required organization-specific parser work
pagination, year/archive, regional-tab, role, affiliation, and provenance requirements
regression coverage requirement

Individual person/profile URLs, homepages, and ordinary news pages must never generate membership-list issues.

Implementation

PR #1774 adds the repository-wide infrastructure:

scripts/run_with_membership_url_capture.py records every fetched URL and page metadata
scripts/membership_list_surface_candidates.py identifies list roots, boards, annual participant pages, archives, and pagination while rejecting profiles
scripts/sync_membership_list_issues.py drains invalid legacy tickets, deduplicates by canonical-URL SHA-256, and hard-caps issue fan-out
corrected RAND/Bilderberg and institutions/foundations workflows

Incident and cleanup

A superseded classifier treated individual /people/<name> URLs as list surfaces and generated invalid tickets through #1773. The stale job has stopped. PR #1774 installs a repeated direct-list cleanup pass that closes those profile/homepage tickets before any organization scraper resumes.

Acceptance criteria

Capture every fetched public URL and page label
Distinguish roster-list surfaces from individual profiles
Stable URL-hash deduplication
Hard fan-out limit
Preserve issue creation even when later corpus validation fails
Merge PR #1774
Verify invalid generated tickets are closed
Verify only actionable roster-list issues remain open
Apply the invariant to every later organization scraper workflow

Mirrored from GitHub https://github.com/lost-rob0t/starintel-gpt-auto-dig/issues/139 (GitHub is authoritative for this item). --- ## Invariant Every organization scraper must record every encountered official public URL that exposes a membership, participant, expert, fellow, board, leadership, staff, trustee, delegate, committee, council, or comparable roster **list surface**. For each canonical list URL, AutoDig must open exactly one deduplicated GitHub issue containing: - organization and dataset - canonical source URL - discovery source - parser/coverage status - required organization-specific parser work - pagination, year/archive, regional-tab, role, affiliation, and provenance requirements - regression coverage requirement Individual person/profile URLs, homepages, and ordinary news pages must never generate membership-list issues. ## Implementation PR #1774 adds the repository-wide infrastructure: - `scripts/run_with_membership_url_capture.py` records every fetched URL and page metadata - `scripts/membership_list_surface_candidates.py` identifies list roots, boards, annual participant pages, archives, and pagination while rejecting profiles - `scripts/sync_membership_list_issues.py` drains invalid legacy tickets, deduplicates by canonical-URL SHA-256, and hard-caps issue fan-out - corrected RAND/Bilderberg and institutions/foundations workflows ## Incident and cleanup A superseded classifier treated individual `/people/<name>` URLs as list surfaces and generated invalid tickets through #1773. The stale job has stopped. PR #1774 installs a repeated direct-list cleanup pass that closes those profile/homepage tickets before any organization scraper resumes. ## Acceptance criteria - [x] Capture every fetched public URL and page label - [x] Distinguish roster-list surfaces from individual profiles - [x] Stable URL-hash deduplication - [x] Hard fan-out limit - [x] Preserve issue creation even when later corpus validation fails - [ ] Merge PR #1774 - [ ] Verify invalid generated tickets are closed - [ ] Verify only actionable roster-list issues remain open - [ ] Apply the invariant to every later organization scraper workflow

nsaspy added the

github-mirror

label

2026-09-08 17:05:43 +00:00

No labels

github-mirror

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starintel-labs/starintel-gpt-auto-dig#995

No description provided.

Rows
Columns