nsaspy/llm-log
1
0
Fork 0
Code Issues 34 Pull requests 19 Projects Releases Packages Wiki Activity Actions

ARADR-003: Provider census — every LLM provider gets a tracked compatibility issue #6

New issue
Open
opened 2026-08-29 12:37:37 +00:00 by lost-rob0t · 0 comments
lost-rob0t commented 2026-08-29 12:37:37 +00:00 (Migrated from github.com)
Copy link

Goal

Make provider coverage an explicit maintained backlog instead of a handful of hard-coded upstreams.

Every distinct LLM provider / gateway / managed endpoint family discovered by the provider census MUST have a dedicated llm-log compatibility issue. Provider support may share generic protocol adapters in implementation, but no provider gets silently collapsed into “OpenAI-compatible” and forgotten.

This is a census + issue-generation slice first. It does not authorize implementing 200+ one-off adapters.

Initial authoritative census

Start from the live Models.dev provider catalog:

As of 2026-08-29 the provider page exposes 207 provider entries, including regional/token/coding-plan variants where the actual endpoint/auth/runtime contract differs.

Cross-check against:

The census is intentionally open-ended. A provider appearing later in a maintained upstream catalog is backlog, not “out of scope.”

Additional provider/runtime classes that must be represented even when absent from the hosted-provider census

Create dedicated issues for important self-hosted/local/gateway surfaces too, including at minimum:

  • Ollama local
  • llama.cpp server
  • vLLM
  • SGLang
  • LM Studio
  • LiteLLM proxy
  • generic OpenAI-compatible endpoint
  • generic Anthropic-compatible endpoint
  • generic Gemini/Google-compatible endpoint when meaningful
  • ChatGPT/Codex backend transport used by Codex
  • arbitrary user-defined HTTP/SSE/WebSocket upstream

If research finds another materially distinct local runtime or gateway, it gets an issue too.

One provider = one issue

For each census entry create or reconcile exactly one canonical provider-compatibility issue. Regional/plan variants may share a parent only when their transport, auth, path and streaming semantics are genuinely identical; otherwise they remain separate issues.

Suggested title:

Provider: <display name> — proxy/capture compatibility

Each provider issue must record:

  • canonical provider ID + display name
  • upstream/base URL form or endpoint discovery mechanism
  • protocol family: OpenAI Chat Completions / Responses / Realtime, Anthropic Messages, Gemini, Bedrock, Vertex, bespoke, etc.
  • authentication/header behavior and which fields must be redacted from persistence
  • HTTP request/response behavior
  • SSE/streaming behavior
  • WebSocket/realtime behavior where supported
  • path-prefix / query handling
  • model-selection preservation requirements
  • known client surfaces that can target it (OpenCode, Codex, Claude Code, gptel, SDKs, custom clients)
  • whether current generic forwarding already works without code changes
  • gaps requiring adapter/normalization work
  • smallest RED-first conformance test required before implementation
  • upstream documentation/source links

Provider registry artifact

ARADR must persist a machine-readable provider registry in the repository, generated/reconciled from the census rather than maintained only in GitHub issue prose.

The registry needs stable provider IDs and at least:

  • source catalog(s)
  • display name
  • protocol family/package hint
  • default/public endpoint when applicable
  • region/variant identity
  • canonical GitHub issue number
  • research status
  • implementation/conformance status
  • last observed catalog timestamp/version

Prefer a simple declarative format that works for Nix, Python tests and future Prolog projection without requiring network access at runtime.

Drift / “every provider ever” rule

Add a deterministic provider-census check that can compare the checked-in registry against the selected upstream catalog snapshot/API.

A newly discovered provider MUST result in one of:

  1. an existing canonical provider issue linked into the registry; or
  2. a newly created provider issue.

The check must flag untracked providers. It must never silently drop a provider because it looks protocol-compatible with another provider.

Catalog removals must be retained as historical/deprecated registry entries rather than erasing compatibility history.

Auto-ARADR flow

  1. Analyze current proxy/provider assumptions.
  2. Snapshot and normalize the provider census.
  3. Group only by actual wire/API semantics for research efficiency.
  4. Create/reconcile one canonical issue per provider entry.
  5. Research protocol families and identify which providers work through existing generic forwarding.
  6. Produce Org-mode research/design artifacts for provider-registry architecture and conformance strategy.
  7. Hand bounded implementation slices to Auto-RAGE only after the design is coherent.

Auto-RAGE implementation priorities

Do not implement alphabetically just because the census is huge. After issues exist, prioritize reusable protocol coverage:

  1. arbitrary OpenAI-compatible HTTP/SSE upstreams
  2. OpenAI Responses + WebSocket/realtime variants
  3. Anthropic Messages-compatible upstreams
  4. Google Gemini / Vertex contracts
  5. AWS Bedrock
  6. Azure/OpenAI-family cloud endpoint peculiarities
  7. local/self-hosted runtimes
  8. bespoke providers that cannot ride a generic adapter

A generic adapter may satisfy many provider issues at once, but each provider issue must receive explicit conformance evidence before being marked supported.

RED-first acceptance for implementation work

Before changing production proxy behavior for a protocol/provider, add the smallest legitimate failing conformance test proving the missing behavior on the untouched baseline. Provider-specific issues can reuse a shared parameterized conformance suite, but no issue is closed solely because another vendor passed it.

Acceptance criteria for this census slice

  • Checked-in provider registry exists.
  • All providers in the current Models.dev census are represented.
  • Relevant providers found only through secondary catalogs/local-runtime research are represented.
  • Every registry entry links to a canonical GitHub provider issue.
  • No two issues accidentally represent the same provider identity.
  • Regional/plan variants are handled deliberately, not silently merged.
  • Provider protocol families are classified.
  • Existing generic compatibility is distinguished from verified conformance.
  • Drift checker detects a newly introduced catalog provider with no issue.
  • Removed providers remain historical rather than disappearing.
  • Org research/design documents explain registry authority, issue lifecycle and conformance strategy.

Non-goals

  • implementing every provider in one PR
  • storing API keys in the registry/Nix store
  • changing client model pickers
  • provider routing/failover policy
  • ranking providers by quality
  • hiding provider-specific incompatibilities behind false “OpenAI-compatible” green status
## Goal Make provider coverage an explicit maintained backlog instead of a handful of hard-coded upstreams. **Every distinct LLM provider / gateway / managed endpoint family discovered by the provider census MUST have a dedicated `llm-log` compatibility issue.** Provider support may share generic protocol adapters in implementation, but no provider gets silently collapsed into “OpenAI-compatible” and forgotten. This is a census + issue-generation slice first. It does **not** authorize implementing 200+ one-off adapters. ## Initial authoritative census Start from the live Models.dev provider catalog: - https://models.dev/providers/ - https://models.dev/api.json As of 2026-08-29 the provider page exposes **207 provider entries**, including regional/token/coding-plan variants where the actual endpoint/auth/runtime contract differs. Cross-check against: - OpenCode provider catalog / `models.dev` consumption: https://opencode.ai/docs/providers - Portkey provider/endpoints catalog: https://portkey.ai/models/providers - current `llm-log` built-ins and actual client configs in `lost-rob0t/dotfiles` The census is intentionally open-ended. A provider appearing later in a maintained upstream catalog is backlog, not “out of scope.” ## Additional provider/runtime classes that must be represented even when absent from the hosted-provider census Create dedicated issues for important self-hosted/local/gateway surfaces too, including at minimum: - Ollama local - llama.cpp server - vLLM - SGLang - LM Studio - LiteLLM proxy - generic OpenAI-compatible endpoint - generic Anthropic-compatible endpoint - generic Gemini/Google-compatible endpoint when meaningful - ChatGPT/Codex backend transport used by Codex - arbitrary user-defined HTTP/SSE/WebSocket upstream If research finds another materially distinct local runtime or gateway, it gets an issue too. ## One provider = one issue For each census entry create or reconcile exactly one canonical provider-compatibility issue. Regional/plan variants may share a parent only when their transport, auth, path and streaming semantics are genuinely identical; otherwise they remain separate issues. Suggested title: `Provider: <display name> — proxy/capture compatibility` Each provider issue must record: - canonical provider ID + display name - upstream/base URL form or endpoint discovery mechanism - protocol family: OpenAI Chat Completions / Responses / Realtime, Anthropic Messages, Gemini, Bedrock, Vertex, bespoke, etc. - authentication/header behavior and which fields must be redacted from persistence - HTTP request/response behavior - SSE/streaming behavior - WebSocket/realtime behavior where supported - path-prefix / query handling - model-selection preservation requirements - known client surfaces that can target it (OpenCode, Codex, Claude Code, gptel, SDKs, custom clients) - whether current generic forwarding already works without code changes - gaps requiring adapter/normalization work - smallest RED-first conformance test required before implementation - upstream documentation/source links ## Provider registry artifact ARADR must persist a machine-readable provider registry in the repository, generated/reconciled from the census rather than maintained only in GitHub issue prose. The registry needs stable provider IDs and at least: - source catalog(s) - display name - protocol family/package hint - default/public endpoint when applicable - region/variant identity - canonical GitHub issue number - research status - implementation/conformance status - last observed catalog timestamp/version Prefer a simple declarative format that works for Nix, Python tests and future Prolog projection without requiring network access at runtime. ## Drift / “every provider ever” rule Add a deterministic provider-census check that can compare the checked-in registry against the selected upstream catalog snapshot/API. A newly discovered provider MUST result in one of: 1. an existing canonical provider issue linked into the registry; or 2. a newly created provider issue. The check must flag untracked providers. It must never silently drop a provider because it looks protocol-compatible with another provider. Catalog removals must be retained as historical/deprecated registry entries rather than erasing compatibility history. ## Auto-ARADR flow 1. Analyze current proxy/provider assumptions. 2. Snapshot and normalize the provider census. 3. Group only by actual wire/API semantics for research efficiency. 4. Create/reconcile one canonical issue per provider entry. 5. Research protocol families and identify which providers work through existing generic forwarding. 6. Produce Org-mode research/design artifacts for provider-registry architecture and conformance strategy. 7. Hand bounded implementation slices to Auto-RAGE only after the design is coherent. ## Auto-RAGE implementation priorities Do **not** implement alphabetically just because the census is huge. After issues exist, prioritize reusable protocol coverage: 1. arbitrary OpenAI-compatible HTTP/SSE upstreams 2. OpenAI Responses + WebSocket/realtime variants 3. Anthropic Messages-compatible upstreams 4. Google Gemini / Vertex contracts 5. AWS Bedrock 6. Azure/OpenAI-family cloud endpoint peculiarities 7. local/self-hosted runtimes 8. bespoke providers that cannot ride a generic adapter A generic adapter may satisfy many provider issues at once, but each provider issue must receive explicit conformance evidence before being marked supported. ## RED-first acceptance for implementation work Before changing production proxy behavior for a protocol/provider, add the smallest legitimate failing conformance test proving the missing behavior on the untouched baseline. Provider-specific issues can reuse a shared parameterized conformance suite, but no issue is closed solely because another vendor passed it. ## Acceptance criteria for this census slice - [ ] Checked-in provider registry exists. - [ ] All providers in the current Models.dev census are represented. - [ ] Relevant providers found only through secondary catalogs/local-runtime research are represented. - [ ] Every registry entry links to a canonical GitHub provider issue. - [ ] No two issues accidentally represent the same provider identity. - [ ] Regional/plan variants are handled deliberately, not silently merged. - [ ] Provider protocol families are classified. - [ ] Existing generic compatibility is distinguished from verified conformance. - [ ] Drift checker detects a newly introduced catalog provider with no issue. - [ ] Removed providers remain historical rather than disappearing. - [ ] Org research/design documents explain registry authority, issue lifecycle and conformance strategy. ## Non-goals - implementing every provider in one PR - storing API keys in the registry/Nix store - changing client model pickers - provider routing/failover policy - ranking providers by quality - hiding provider-specific incompatibilities behind false “OpenAI-compatible” green status
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/108-full-kb-mcp-20260920
feat/mcp-expert-kb-20260920
feat/105-opencode-profile
feat/104-python-admission
feat/starworld-rsi
feat/58-structured-output-anomaly
feat/58-tool-schema-anomaly
feat/58-tool-call-anomaly
fix/61-alert-policy
fix/61-retry-routing-correlation
fix/61-transport-error-evidence
fix/68-expert-response-ingest
fix/67-stream-integrity
feature/cl-native-bulk-infill
feature/expert-backfill-query-export
feat/79-quota-telemetry
feat/79-provider-quota-telemetry
fix/74-relay-socket-ownership
feat/stats-analytics-api
feat/71-rate-profile-foundation
feat/75-rate-admission
feat/70-request-scheduler
rage/59-init-files-r3
fix/websocket-expert-plane
rage/58-60-quant-anomaly-alerts-r4
rage/58-60-quant-anomaly-alerts-r3
rage/57-openrouter-quant-observability-r2
rage/58-60-quant-anomaly-alerts-r2
rage/59-init-files-r2
rage/58-60-quant-anomaly-alerts
rage/59-init-files
rage/57-openrouter-quant-observability
rage/openrouter-quantization-presets
noop-unused
rage/15-outcome-dataset-task-cost-red
rage/15-outcome-dataset-classification-red
rage/15-outcome-dataset-metadata-red
rage/13-outcome-retry-economics-red
rage/13-outcome-breakdown-red
rage/15-outcome-dataset-red
rage/15-outcome-history-red
rage/15-outcome-evidence-red
rage/13-classification-breakdown-red
rage/13-usage-breakdown-red
rage/13-retry-attempt-red
rage/13-task-rollup-red
rage/13-task-accounting-red
rage/12-classifier-task-id-red
rage/12-classifier-time-range-red
rage/12-classifier-query-red
rage/12-request-classifier-red
rage/10-expert-runtime-red
rage/10-recorder-deps-red
rage/4-unified-config
rage/home-manager-service
rage/1-capture-proxy-prolog-kb
No results found.
Labels
Clear labels   
accessibility
Barrier affecting people with disabilities

  
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/llm-log#6
No description provided.