nsaspy/starintel-auto-research

Fork 0

OSINT cluster research: username/account enumeration engines #180

New issue

Open

opened 2026-08-29 10:28:08 +00:00 by nsaspy · 5 comments

nsaspy commented

2026-08-29 10:28:08 +00:00

Owner

Goal

Deeply compare the major username/account enumeration engines used by modern OSINT stacks, especially Maigret, WhatsMyName, Sherlock, Nexfil, and Blackbird.

Architecture rule

This is one Identity Domain Server capability cluster, not five new actor projects.

The research must extract the distinct techniques and data models each engine contributes, then define one StarIntel capability contract with provider/registry-specific strategies behind it.

Required comparison

For each engine enumerate and verify:

site/platform registry schema
account-existence detection strategies
request templates and substitutions
positive/negative/error markers
status-code handling
redirect behavior
rate limiting and retries
proxy/Tor support where present
concurrency model
site tags/categories
account metadata extraction
profile URL generation
false-positive handling
disabled/dead-site handling
registry update mechanism
custom-site extension mechanism
result formats
machine-readable provenance
confidence/multi-engine confirmation opportunities

StarIntel target

Research a single capability family such as identity.enumerateAccounts with:

a versioned platform registry
typed platform probes
bounded concurrent fan-out
explicit found | notFound | unknown | partial | blocked outcomes
preserved evidence and probe reason
provider health state
deduplication across registries without losing supporting observations

Implementation preference

StarLang first. Extend StarLang where needed. Common Lisp next. Python only as a last-resort adapter.

Output gate

Stage findings in the internal OSINT tool ledger and this issue. Do not modify ARARD/ARADR conclusions until explicit human instruction to add research.

## Goal Deeply compare the major username/account enumeration engines used by modern OSINT stacks, especially Maigret, WhatsMyName, Sherlock, Nexfil, and Blackbird. ## Architecture rule This is **one Identity Domain Server capability cluster**, not five new actor projects. The research must extract the distinct techniques and data models each engine contributes, then define one StarIntel capability contract with provider/registry-specific strategies behind it. ## Required comparison For each engine enumerate and verify: - site/platform registry schema - account-existence detection strategies - request templates and substitutions - positive/negative/error markers - status-code handling - redirect behavior - rate limiting and retries - proxy/Tor support where present - concurrency model - site tags/categories - account metadata extraction - profile URL generation - false-positive handling - disabled/dead-site handling - registry update mechanism - custom-site extension mechanism - result formats - machine-readable provenance - confidence/multi-engine confirmation opportunities ## StarIntel target Research a single capability family such as `identity.enumerateAccounts` with: - a versioned platform registry - typed platform probes - bounded concurrent fan-out - explicit `found | notFound | unknown | partial | blocked` outcomes - preserved evidence and probe reason - provider health state - deduplication across registries without losing supporting observations ## Implementation preference StarLang first. Extend StarLang where needed. Common Lisp next. Python only as a last-resort adapter. ## Output gate Stage findings in the internal OSINT tool ledger and this issue. Do not modify ARARD/ARADR conclusions until explicit human instruction to add research.

nsaspy commented

2026-08-29 10:28:15 +00:00

Author

Owner

Deep finding: WhatsMyName

Upstream: WebBreacher/WhatsMyName.

What it actually is now

WhatsMyName is primarily a community-maintained username detection dataset, not the checker runtime. Since 2023 the project intentionally removed bundled checker scripts and focuses on wmn-data.json.

Feature/mechanism inventory

one versioned JSON registry covering hundreds of public profile sites
each site entry describes the profile/check URL shape
each entry records signals for successful and not-found responses
dataset categories support filtering/grouping
explicit inclusion constraints: public accessibility, username present in URL, no username→opaque numeric-ID transformation
community validation/maintenance workflow for broken detections and new sites
registry can be consumed by many independent runtimes
browser, CLI, desktop, Maltego, SpiderFoot, Blackbird, and other consumers all share the same detection data
false-positive reduction is a core reason the dataset exists
detection drift is expected: profile URLs, response codes, and page content change continuously

What StarIntel needs from it

Do not make WhatsMyName an actor. Split the concept into:

a versioned identityPlatformRegistry dataset
StarLang/Lisp probe execution owned by the Identity Domain Server
automated registry-health validation

The registry should carry at least platform ID, category/tags, profile URL template, probe URL/method, positive/negative/error signatures, expected status behavior, side-effect policy, health state, last validation evidence, and source/license metadata.

This is the cleanest evidence so far that user-hunt should stop owning WMN-specific data semantics and instead consume a generic platform registry.

### Deep finding: WhatsMyName Upstream: `WebBreacher/WhatsMyName`. **What it actually is now** WhatsMyName is primarily a community-maintained **username detection dataset**, not the checker runtime. Since 2023 the project intentionally removed bundled checker scripts and focuses on `wmn-data.json`. **Feature/mechanism inventory** - one versioned JSON registry covering hundreds of public profile sites - each site entry describes the profile/check URL shape - each entry records signals for successful and not-found responses - dataset categories support filtering/grouping - explicit inclusion constraints: public accessibility, username present in URL, no username→opaque numeric-ID transformation - community validation/maintenance workflow for broken detections and new sites - registry can be consumed by many independent runtimes - browser, CLI, desktop, Maltego, SpiderFoot, Blackbird, and other consumers all share the same detection data - false-positive reduction is a core reason the dataset exists - detection drift is expected: profile URLs, response codes, and page content change continuously **What StarIntel needs from it** Do **not** make WhatsMyName an actor. Split the concept into: 1. a versioned `identityPlatformRegistry` dataset 2. StarLang/Lisp probe execution owned by the Identity Domain Server 3. automated registry-health validation The registry should carry at least platform ID, category/tags, profile URL template, probe URL/method, positive/negative/error signatures, expected status behavior, side-effect policy, health state, last validation evidence, and source/license metadata. This is the cleanest evidence so far that `user-hunt` should stop owning WMN-specific data semantics and instead consume a generic platform registry.

nsaspy commented

2026-08-29 10:28:15 +00:00

Author

Owner

Deep finding: Sherlock

Upstream: sherlock-project/sherlock.

Feature/mechanism inventory

username search across 400+ social networks/sites
accepts one or many usernames
supports username similarity templates using {?} substitutions for _, -, and . variants
configurable per-site selection
timeout control
proxy support
optional raw-response dumping for debugging/evidence
output to text, JSON, CSV, and XLSX
browse/open-found-result workflow
local site-data mode
NSFW-site filtering and exclusion controls
result filtering for found/all results
mature site-registry maintenance with frequent false-positive/site-breakage fixes
active handling of transient 5xx/false-positive issues in current maintenance
packaged CLI/runtime separate from the site definitions it consumes

What StarIntel needs from it

Sherlock adds two useful ideas beyond basic WMN consumption:

bounded username-variant generation as a first-class pre-enumeration step
debug/evidence capture for ambiguous probes, including raw response metadata when policy allows

StarIntel should express username variants as generated candidate observations with derivation provenance, then fan them through the same identity.enumerateAccounts contract. Do not make a separate Sherlock capability family.

The provider result must distinguish transient server errors from true not-found; a 5xx cannot collapse to notFound.

### Deep finding: Sherlock Upstream: `sherlock-project/sherlock`. **Feature/mechanism inventory** - username search across 400+ social networks/sites - accepts one or many usernames - supports username similarity templates using `{?}` substitutions for `_`, `-`, and `.` variants - configurable per-site selection - timeout control - proxy support - optional raw-response dumping for debugging/evidence - output to text, JSON, CSV, and XLSX - browse/open-found-result workflow - local site-data mode - NSFW-site filtering and exclusion controls - result filtering for found/all results - mature site-registry maintenance with frequent false-positive/site-breakage fixes - active handling of transient 5xx/false-positive issues in current maintenance - packaged CLI/runtime separate from the site definitions it consumes **What StarIntel needs from it** Sherlock adds two useful ideas beyond basic WMN consumption: 1. **bounded username-variant generation** as a first-class pre-enumeration step 2. **debug/evidence capture for ambiguous probes**, including raw response metadata when policy allows StarIntel should express username variants as generated candidate observations with derivation provenance, then fan them through the same `identity.enumerateAccounts` contract. Do not make a separate Sherlock capability family. The provider result must distinguish transient server errors from true not-found; a 5xx cannot collapse to `notFound`.

nsaspy commented

2026-08-29 10:28:15 +00:00

Author

Owner

Deep finding: NExfil

Upstream: thewhiteh4t/nexfil.

Feature/mechanism inventory

username profile lookup across 300+ platforms
design goal explicitly emphasizes speed with low false-positive volume
single-username mode
comma-separated batch mode
username-list file mode
configurable timeout
automatic result persistence to text
update-check support
proxy configuration surface with single/file modes and HTTP/HTTPS protocol selection
CLI optimized for broad fast fan-out
current README still describes JSON/CSV and Tor as incomplete/coming-soon even though proxy flags exist, so capability claims must be source-versioned instead of assumed

What StarIntel needs from it

NExfil is mostly a performance reference, not a distinct capability. Pull these ideas into identity.enumerateAccounts:

explicit scan budget/profile (fast, balanced, exhaustive)
batch input as native fan-out rather than shell looping
deadline/timeout as part of the plan
transport/proxy policy chosen independently from the site probe registry
benchmark fixtures for precision, latency, and partial completion

Do not port NExfil wholesale unless later benchmarking shows a probe technique absent from the shared registry strategy.

### Deep finding: NExfil Upstream: `thewhiteh4t/nexfil`. **Feature/mechanism inventory** - username profile lookup across 300+ platforms - design goal explicitly emphasizes speed with low false-positive volume - single-username mode - comma-separated batch mode - username-list file mode - configurable timeout - automatic result persistence to text - update-check support - proxy configuration surface with single/file modes and HTTP/HTTPS protocol selection - CLI optimized for broad fast fan-out - current README still describes JSON/CSV and Tor as incomplete/coming-soon even though proxy flags exist, so capability claims must be source-versioned instead of assumed **What StarIntel needs from it** NExfil is mostly a performance reference, not a distinct capability. Pull these ideas into `identity.enumerateAccounts`: - explicit scan budget/profile (`fast`, `balanced`, `exhaustive`) - batch input as native fan-out rather than shell looping - deadline/timeout as part of the plan - transport/proxy policy chosen independently from the site probe registry - benchmark fixtures for precision, latency, and partial completion Do not port NExfil wholesale unless later benchmarking shows a probe technique absent from the shared registry strategy.

nsaspy commented

2026-08-29 10:28:16 +00:00

Author

Owner

Deep finding: Blackbird

Upstream: p1ngul1n0/blackbird.

Feature/mechanism inventory

username search across 600+ platforms
email search in the same CLI
integrates WhatsMyName data rather than maintaining an entirely separate platform universe
smart filtering focused on reducing false positives
PDF and CSV reporting
optional AI profile generation from the set of sites where an identity was found
AI mode intentionally sends only site names rather than raw sensitive target data
automated post-enumeration interpretation layer
current upstream issue history shows the same drift/false-positive maintenance pressure as other username engines

What StarIntel needs from it

Blackbird's useful delta is post-enumeration interpretation, not another checker runtime.

StarIntel should keep raw account observations separate from any inferred behavioral/technical profile. A later analysis capability may consume only an explicitly policy-approved projection of those observations and emit claims with provenance/confidence.

So the target split is:

identity.enumerateAccounts -> factual/observational results
identity.resolveIdentity -> correlation
analysis domain -> optional higher-level profile/summary claims

Do not let an LLM summary become the evidence layer, and do not duplicate WMN enumeration just to reproduce Blackbird.

### Deep finding: Blackbird Upstream: `p1ngul1n0/blackbird`. **Feature/mechanism inventory** - username search across 600+ platforms - email search in the same CLI - integrates WhatsMyName data rather than maintaining an entirely separate platform universe - smart filtering focused on reducing false positives - PDF and CSV reporting - optional AI profile generation from the set of sites where an identity was found - AI mode intentionally sends only site names rather than raw sensitive target data - automated post-enumeration interpretation layer - current upstream issue history shows the same drift/false-positive maintenance pressure as other username engines **What StarIntel needs from it** Blackbird's useful delta is **post-enumeration interpretation**, not another checker runtime. StarIntel should keep raw account observations separate from any inferred behavioral/technical profile. A later analysis capability may consume only an explicitly policy-approved projection of those observations and emit claims with provenance/confidence. So the target split is: - `identity.enumerateAccounts` -> factual/observational results - `identity.resolveIdentity` -> correlation - analysis domain -> optional higher-level profile/summary claims Do not let an LLM summary become the evidence layer, and do not duplicate WMN enumeration just to reproduce Blackbird.

nsaspy commented

2026-08-29 10:28:16 +00:00

Author

Owner

ARDR research checkpoint — READY_FOR_DESIGN

StarIntel problem solved

This research is directly tied to lost-rob0t/starintel-server#45: the dormant source/actor-systems/user-finder.lisp still performs a WMN download at load time, uses duplicated/unbounded concurrency layers, has no robust typed partial/error model, and publishes generated documents directly to Rabbit instead of canonical ingest. The target is therefore not “adopt five OSINT tools”; it is to replace the broken username-enumeration subsystem with one bounded Identity Domain Server capability.

Current implementation evidence

Current user-finder.lisp proves the old architectural boundary is wrong:

*wmn-data* is fetched remotely during load;
WMN schema/runtime are coupled through global variables;
checking reduces outcomes to truthy/nil and logs arbitrary errors rather than returning typed found | notFound | unknown | blocked | partial results;
lparallel:pmapcar wraps additional submit-task calls, creating two concurrency layers;
completion state never becomes terminal;
produced users/relations bypass canonical ingest/provenance flow.

Upstream findings

WhatsMyName

Current WMN is intentionally a data registry, not a checker runtime. Its maintained wmn-data.json describes how to test sites and its JSON schema is a useful source-registry contract. Upstream explicitly focuses on public sites where the username is present directly in the profile URL and detection can distinguish existing vs missing accounts. WMN is therefore a strong registry/provider input, not a StarIntel runtime architecture.

Useful semantics to retain:

versioned site registry;
profile/check URL templates;
positive/negative response markers;
categories/tags;
known claimed/unclaimed fixtures for registry validation;
explicit schema validation;
registry freshness and drift testing.

Sherlock

Sherlock demonstrates multiple per-site detection strategies rather than one universal status-code test. Its site definitions support URL templates, username regex constraints and detection based on status/body/redirect-style behavior. The reusable StarIntel lesson is typed probe strategies, not Sherlock's Python runtime.

Maigret

Current Maigret extends the registry model substantially. Its site model includes stable site metadata, disabled state, tags, identifier type, custom headers, activation requirements, username regex, request method/payload/GET params, response-presence/absence indicators, known claimed/unclaimed usernames and site ranking/filtering. It also supports site health/self-check behavior, retries, tags/country filtering and page-derived identifier pivots. Recent upstream false-positive bot issues show that provider drift is an active operational problem, not an edge case.

Reusable semantics:

versioned registry entries with strategy-specific request data;
provider/site health with self-check fixtures;
automatic demotion, not silent deletion, of drifting probes;
exact identifier type (username, other ids) rather than assuming every probe accepts the same identifier;
per-site tags/rank/disabled state;
headers/method/body/query templates;
typed positive/negative/error evidence;
optional profile parsing/pivot observations kept separate from existence proof.

Blackbird / Nexfil

They remain useful comparison points and downstream registry/probe consumers, but current evidence does not justify making either runtime an architectural dependency. Blackbird is already described by WMN as a WMN consumer, reinforcing the registry/runtime separation. Nexfil-style broad platform-count coverage is not itself a capability StarIntel should encode; coverage belongs in versioned registry data and quality metrics.

StarIntel design conclusions

One Identity Domain Server capability: identity.enumerateAccounts (final spelling follows StarLang conventions).
Do not resurrect user-hunt as a bespoke forever-loop actor.
Do not vendor Maigret/Sherlock/Blackbird/Nexfil as the control plane.
Use one normalized Platform Probe Registry that may import/normalize WMN, Sherlock/Maigret-derived entries, and future registries while preserving source/version/license/provenance.
Each probe declares a closed strategy type and typed request template. Initial strategy families:
- status classification;
- body presence/absence markers;
- final/redirect URL classification;
- bounded combinations of the above;
- API/JSON field classification where a source explicitly exposes it.
Registry entries include positive and negative owned/reference fixtures where upstream provides them. A registry refresh is not active until validation passes.
Execution is bounded fan-out using the shared StarLang orchestration direction from #172 and source-acquisition HTTP semantics from #171. Identity does not implement another HTTP/concurrency stack.
Outcomes are typed: found | notFound | unknown | blocked | unavailable | failed; aggregate run may be success | partial | cancelled | failed and retains every per-provider reason.
notFound requires a probe-specific negative condition. Timeout, challenge, rate-limit, parse drift or ambiguous response can never be coerced to not-found.
Provider/site health is scoped and versioned; health/demotion must not mutate authorization or silently rewrite historical observations.
Found-account evidence produces typed account observations through canonical ingest with registry version, probe identity, request target, response evidence digest/ref, timestamp and source provenance. Correlation/merging remains expert/entity-resolution work.
Profile parsing and recursive pivots are separate optional capabilities. Existence detection must remain usable without broad profile scraping.

StarLang classification

bounded multi-provider fan-out / typed partial aggregation: StarLang extension/shared orchestration, already designed in #172;
HTTP request/deadline/cancellation/rate/response-size semantics: reuse Source Acquisition #171, not Identity-specific syntax;
platform registry parsing/normalization/health: domain/runtime library, not language syntax;
typed probe strategy declaration: small StarLang/domain declaration surface or typed registry schema, whichever avoids executable registry data;
evidence/confidence/corroboration: document/expert layer, not checker runtime;
external Python checker process: not required for the core architecture; allowed only for isolated source-specific probes that cannot reasonably be expressed through the normalized HTTP/browser capability.

Adversarial findings

Rejected:

treating WMN registry truth as account truth;
global boolean provider health;
unbounded “scan every site” execution;
using positive absence of an error string alone where a site fixture cannot validate that rule;
silent registry hot-update during a run;
duplicate platform entries from several upstream registries being executed as independent proof without lineage/dedup;
raw response bodies in ordinary documents/logs;
direct Rabbit publication from the Identity capability;
arbitrary executable callbacks/functions stored in registry JSON;
automatic recursive pivots from discovered usernames without the #172 depth/visited/budget contract.

Research transition

Research state: READY_FOR_DESIGN.

No implementation approval is implied. ARDR may proceed to design under #170, but implementation remains operator-only.

## ARDR research checkpoint — READY_FOR_DESIGN ### StarIntel problem solved This research is directly tied to `lost-rob0t/starintel-server#45`: the dormant `source/actor-systems/user-finder.lisp` still performs a WMN download at load time, uses duplicated/unbounded concurrency layers, has no robust typed partial/error model, and publishes generated documents directly to Rabbit instead of canonical ingest. The target is therefore not “adopt five OSINT tools”; it is to replace the broken username-enumeration subsystem with one bounded Identity Domain Server capability. ### Current implementation evidence Current `user-finder.lisp` proves the old architectural boundary is wrong: - `*wmn-data*` is fetched remotely during load; - WMN schema/runtime are coupled through global variables; - checking reduces outcomes to truthy/nil and logs arbitrary errors rather than returning typed `found | notFound | unknown | blocked | partial` results; - `lparallel:pmapcar` wraps additional `submit-task` calls, creating two concurrency layers; - completion state never becomes terminal; - produced users/relations bypass canonical ingest/provenance flow. ### Upstream findings #### WhatsMyName Current WMN is intentionally a **data registry**, not a checker runtime. Its maintained `wmn-data.json` describes how to test sites and its JSON schema is a useful source-registry contract. Upstream explicitly focuses on public sites where the username is present directly in the profile URL and detection can distinguish existing vs missing accounts. WMN is therefore a strong registry/provider input, not a StarIntel runtime architecture. Useful semantics to retain: - versioned site registry; - profile/check URL templates; - positive/negative response markers; - categories/tags; - known claimed/unclaimed fixtures for registry validation; - explicit schema validation; - registry freshness and drift testing. #### Sherlock Sherlock demonstrates multiple per-site detection strategies rather than one universal status-code test. Its site definitions support URL templates, username regex constraints and detection based on status/body/redirect-style behavior. The reusable StarIntel lesson is **typed probe strategies**, not Sherlock's Python runtime. #### Maigret Current Maigret extends the registry model substantially. Its site model includes stable site metadata, disabled state, tags, identifier type, custom headers, activation requirements, username regex, request method/payload/GET params, response-presence/absence indicators, known claimed/unclaimed usernames and site ranking/filtering. It also supports site health/self-check behavior, retries, tags/country filtering and page-derived identifier pivots. Recent upstream false-positive bot issues show that provider drift is an active operational problem, not an edge case. Reusable semantics: - versioned registry entries with strategy-specific request data; - provider/site health with self-check fixtures; - automatic **demotion**, not silent deletion, of drifting probes; - exact identifier type (`username`, other ids) rather than assuming every probe accepts the same identifier; - per-site tags/rank/disabled state; - headers/method/body/query templates; - typed positive/negative/error evidence; - optional profile parsing/pivot observations kept separate from existence proof. #### Blackbird / Nexfil They remain useful comparison points and downstream registry/probe consumers, but current evidence does not justify making either runtime an architectural dependency. Blackbird is already described by WMN as a WMN consumer, reinforcing the registry/runtime separation. Nexfil-style broad platform-count coverage is not itself a capability StarIntel should encode; coverage belongs in versioned registry data and quality metrics. ### StarIntel design conclusions 1. **One Identity Domain Server capability**: `identity.enumerateAccounts` (final spelling follows StarLang conventions). 2. Do not resurrect `user-hunt` as a bespoke forever-loop actor. 3. Do not vendor Maigret/Sherlock/Blackbird/Nexfil as the control plane. 4. Use one normalized **Platform Probe Registry** that may import/normalize WMN, Sherlock/Maigret-derived entries, and future registries while preserving source/version/license/provenance. 5. Each probe declares a closed strategy type and typed request template. Initial strategy families: - status classification; - body presence/absence markers; - final/redirect URL classification; - bounded combinations of the above; - API/JSON field classification where a source explicitly exposes it. 6. Registry entries include positive and negative owned/reference fixtures where upstream provides them. A registry refresh is not active until validation passes. 7. Execution is bounded fan-out using the shared StarLang orchestration direction from #172 and source-acquisition HTTP semantics from #171. Identity does not implement another HTTP/concurrency stack. 8. Outcomes are typed: `found | notFound | unknown | blocked | unavailable | failed`; aggregate run may be `success | partial | cancelled | failed` and retains every per-provider reason. 9. `notFound` requires a probe-specific negative condition. Timeout, challenge, rate-limit, parse drift or ambiguous response can never be coerced to not-found. 10. Provider/site health is scoped and versioned; health/demotion must not mutate authorization or silently rewrite historical observations. 11. Found-account evidence produces typed account observations through canonical ingest with registry version, probe identity, request target, response evidence digest/ref, timestamp and source provenance. Correlation/merging remains expert/entity-resolution work. 12. Profile parsing and recursive pivots are separate optional capabilities. Existence detection must remain usable without broad profile scraping. ### StarLang classification - bounded multi-provider fan-out / typed partial aggregation: **StarLang extension/shared orchestration**, already designed in #172; - HTTP request/deadline/cancellation/rate/response-size semantics: **reuse Source Acquisition #171**, not Identity-specific syntax; - platform registry parsing/normalization/health: **domain/runtime library**, not language syntax; - typed probe strategy declaration: **small StarLang/domain declaration surface or typed registry schema**, whichever avoids executable registry data; - evidence/confidence/corroboration: **document/expert layer**, not checker runtime; - external Python checker process: **not required for the core architecture**; allowed only for isolated source-specific probes that cannot reasonably be expressed through the normalized HTTP/browser capability. ### Adversarial findings Rejected: - treating WMN registry truth as account truth; - global boolean provider health; - unbounded “scan every site” execution; - using positive absence of an error string alone where a site fixture cannot validate that rule; - silent registry hot-update during a run; - duplicate platform entries from several upstream registries being executed as independent proof without lineage/dedup; - raw response bodies in ordinary documents/logs; - direct Rabbit publication from the Identity capability; - arbitrary executable callbacks/functions stored in registry JSON; - automatic recursive pivots from discovered usernames without the #172 depth/visited/budget contract. ### Research transition Research state: `READY_FOR_DESIGN`. No implementation approval is implied. ARDR may proceed to design under #170, but implementation remains operator-only.

nsaspy referenced this issue

2026-08-29 10:28:16 +00:00

OSINT cluster research: email-account existence and Google identity pivots #179