nsaspy/a0-symbolics

Fork 0

EPIC: Live Docker end-to-end CI and provider compatibility #58

New issue

Open

opened 2026-08-25 02:18:55 +00:00 by lost-rob0t · 2 comments

lost-rob0t commented

2026-08-25 02:18:55 +00:00

(Migrated from github.com)

Parent: #1

Goal

Make CI prove that the real Dockerized a0-symbolics product works end-to-end, including Agent Zero startup, the single Prolog-RLM integration, prompt/context compilation, the actual model transport, OpenRouter, native tool schemas, and a completed live agent turn.

Unit tests and direct Python bridge calls remain useful but are not sufficient release evidence.

Current failure this epic must catch

A real OpenRouter/Z.AI request currently fails with:

Tool choice must be auto

The Responses transport can synthesize tool_choice: required when function tools are present. Provider-specific semantics must be normalized before dispatch so Z.AI/OpenRouter receives auto where required. The live test must reproduce and prevent this class of integration failure.

Live CI contract

CI must build the same Docker image path used by Symbolics development/production and start the real service. The test then exercises the externally reachable application rather than importing plugin bridge classes directly.

Required path:

build DockerfileLocal or the exact production-equivalent Symbolics image;
boot the container with Prolog-RLM enabled;
wait for the real healthcheck and WebUI/API readiness;
assert only one Prolog-RLM plugin is discovered;
execute a real chat/agent turn through Agent Zero;
send a real OpenRouter request using the configured live model;
require at least one real native-tool-capable turn;
verify the turn completes through the normal response-tool path;
verify provider-visible context respects the configured symbolic budget;
retain logs/artifacts on failure;
always destroy containers/volumes created by CI.

Secrets and variables

Required GitHub Actions secret:

OPENROUTER_API_KEY

Non-secret repository variable:

SYMBOLICS_LIVE_MODEL — live CI model override. Keep a cheap known-good default in workflow/config when unset.

No other provider credential should be required for the default live gate.

Test tiers

Pull-request gate

deterministic Docker build/startup
offline/unit/integration tests
one small live OpenRouter smoke when the secret is available
explicit skip reason when secrets are unavailable on untrusted forks

Main/release gate

mandatory live OpenRouter smoke
tool-capable model path
provider compatibility matrix for at least OpenRouter generic + Z.AI-backed route when configured
failure artifacts uploaded

Runtime assertions

Prolog-RLM reports ready authority.
Exactly one Prolog-RLM plugin is enabled/discovered.
Persistent Prolog worker survives more than one turn.
Context projection is used by the real turn, not bypassed.
Minimum starting symbolic context budget is 500 tokens unless a test explicitly overrides it.
Tool schemas remain authorized by Agent Zero policy; visibility never grants authority.
Provider-specific tool_choice is legal for the selected model/provider.
A provider 400/422 is a hard CI failure, not a successful fallback hidden from the gate.
Transport fallback, if intentionally exercised, is visible in test output.

Failure enumeration artifact

Every failed live run should capture enough evidence to enumerate problems quickly:

docker compose ps
health status
supervisor status
container logs
Agent Zero application logs
Prolog worker stderr/status
selected provider/model/mode
sanitized outbound request metadata
response status/error body
context token ledger
enabled plugin list
active native tool names
git SHA/image tag

Never upload API keys, auth cookies, passwords, or secret values.

Acceptance

CI builds and boots the actual Symbolics Docker image.
CI talks to the real running application instead of only importing bridge classes.
A real OpenRouter-backed agent turn completes.
Native tools are present in at least one live request.
Z.AI/OpenRouter tool_choice compatibility is covered by a regression.
Exactly one Prolog-RLM plugin is discovered in the container.
500-token starting symbolic context behavior is asserted.
Failure diagnostics are uploaded as artifacts with secrets redacted.
Main/release cannot go green if the mandatory live test fails.
Local developer commands reproduce the same Docker test path.

Local developer command target

Provide one canonical command, ideally something equivalent to:

OPENROUTER_API_KEY=... SYMBOLICS_LIVE_MODEL=... ./scripts/test-symbolics-live

The script should build/start/test/diagnose/clean up itself so local and CI behavior do not drift.

Parent: #1 ## Goal Make CI prove that the real Dockerized `a0-symbolics` product works end-to-end, including Agent Zero startup, the single Prolog-RLM integration, prompt/context compilation, the actual model transport, OpenRouter, native tool schemas, and a completed live agent turn. Unit tests and direct Python bridge calls remain useful but are not sufficient release evidence. ## Current failure this epic must catch A real OpenRouter/Z.AI request currently fails with: ```text Tool choice must be auto ``` The Responses transport can synthesize `tool_choice: required` when function tools are present. Provider-specific semantics must be normalized before dispatch so Z.AI/OpenRouter receives `auto` where required. The live test must reproduce and prevent this class of integration failure. ## Live CI contract CI must build the same Docker image path used by Symbolics development/production and start the real service. The test then exercises the externally reachable application rather than importing plugin bridge classes directly. Required path: 1. build `DockerfileLocal` or the exact production-equivalent Symbolics image; 2. boot the container with Prolog-RLM enabled; 3. wait for the real healthcheck and WebUI/API readiness; 4. assert only one Prolog-RLM plugin is discovered; 5. execute a real chat/agent turn through Agent Zero; 6. send a real OpenRouter request using the configured live model; 7. require at least one real native-tool-capable turn; 8. verify the turn completes through the normal response-tool path; 9. verify provider-visible context respects the configured symbolic budget; 10. retain logs/artifacts on failure; 11. always destroy containers/volumes created by CI. ## Secrets and variables Required GitHub Actions secret: - `OPENROUTER_API_KEY` Non-secret repository variable: - `SYMBOLICS_LIVE_MODEL` — live CI model override. Keep a cheap known-good default in workflow/config when unset. No other provider credential should be required for the default live gate. ## Test tiers ### Pull-request gate - deterministic Docker build/startup - offline/unit/integration tests - one small live OpenRouter smoke when the secret is available - explicit skip reason when secrets are unavailable on untrusted forks ### Main/release gate - mandatory live OpenRouter smoke - tool-capable model path - provider compatibility matrix for at least OpenRouter generic + Z.AI-backed route when configured - failure artifacts uploaded ## Runtime assertions - Prolog-RLM reports ready authority. - Exactly one Prolog-RLM plugin is enabled/discovered. - Persistent Prolog worker survives more than one turn. - Context projection is used by the real turn, not bypassed. - Minimum starting symbolic context budget is 500 tokens unless a test explicitly overrides it. - Tool schemas remain authorized by Agent Zero policy; visibility never grants authority. - Provider-specific `tool_choice` is legal for the selected model/provider. - A provider 400/422 is a hard CI failure, not a successful fallback hidden from the gate. - Transport fallback, if intentionally exercised, is visible in test output. ## Failure enumeration artifact Every failed live run should capture enough evidence to enumerate problems quickly: - `docker compose ps` - health status - supervisor status - container logs - Agent Zero application logs - Prolog worker stderr/status - selected provider/model/mode - sanitized outbound request metadata - response status/error body - context token ledger - enabled plugin list - active native tool names - git SHA/image tag Never upload API keys, auth cookies, passwords, or secret values. ## Acceptance - [ ] CI builds and boots the actual Symbolics Docker image. - [ ] CI talks to the real running application instead of only importing bridge classes. - [ ] A real OpenRouter-backed agent turn completes. - [ ] Native tools are present in at least one live request. - [ ] Z.AI/OpenRouter `tool_choice` compatibility is covered by a regression. - [ ] Exactly one Prolog-RLM plugin is discovered in the container. - [ ] 500-token starting symbolic context behavior is asserted. - [ ] Failure diagnostics are uploaded as artifacts with secrets redacted. - [ ] Main/release cannot go green if the mandatory live test fails. - [ ] Local developer commands reproduce the same Docker test path. ## Local developer command target Provide one canonical command, ideally something equivalent to: ```bash OPENROUTER_API_KEY=... SYMBOLICS_LIVE_MODEL=... ./scripts/test-symbolics-live ``` The script should build/start/test/diagnose/clean up itself so local and CI behavior do not drift.

lost-rob0t commented

2026-08-27 00:06:27 +00:00

(Migrated from github.com)

Cross-repo generic-contract handoff: the tool_choice: required -> provider rejects with Tool choice must be auto failure is now tracked upstream as lost-rob0t/prolog-rlm#257.

Ownership split:

prolog-rlm#257 owns provider/model capability normalization at the reusable request/transport boundary, with deterministic exact-payload tests plus real OpenRouter evidence when credentials are available;
this Symbolics epic should retain the real Docker/product E2E regression and consume the upstream contract rather than adding an independent provider-policy rewrite in Agent Zero glue.

Upstream TDD explicitly forbids globally forcing auto: routes that support required must preserve it, and compatibility normalization must not grant tool authority/effects.

Cross-repo generic-contract handoff: the `tool_choice: required` -> provider rejects with `Tool choice must be auto` failure is now tracked upstream as `lost-rob0t/prolog-rlm#257`. Ownership split: - `prolog-rlm#257` owns provider/model capability normalization at the reusable request/transport boundary, with deterministic exact-payload tests plus real OpenRouter evidence when credentials are available; - this Symbolics epic should retain the real Docker/product E2E regression and consume the upstream contract rather than adding an independent provider-policy rewrite in Agent Zero glue. Upstream TDD explicitly forbids globally forcing `auto`: routes that support `required` must preserve it, and compatibility normalization must not grant tool authority/effects.

lost-rob0t commented

2026-08-27 02:04:50 +00:00

(Migrated from github.com)

Upstream handoff update: prolog-rlm PR #259 now realizes the generic provider-boundary contract at exact head 6644035d4a86ba9a363448ecd2a182566ff5213b, with deterministic stream/non-stream request normalization plus green REAL and pinned Paid OpenRouter gates.

Important integration detail: the upstream runtime does not infer model legality from route-name strings and does not globally downgrade required. The consuming trusted provider/model profile must declare the known legal modes, e.g. tool_choice_modes([auto]) for the affected Z.AI-backed route. With that profile, simple required normalizes to auto; providers supporting required preserve it; specific-function selectors are not silently weakened.

Keep this issue's Docker E2E as the product proof: configure the affected live profile through the normal trusted Prolog-RLM provider configuration, then require the real Agent Zero turn to complete. Do not add an Agent Zero-local request rewrite. PR #259 is ready but remains unmerged pending the upstream repository's explicit merge-on-green authorization gate.

Upstream handoff update: `prolog-rlm` PR #259 now realizes the generic provider-boundary contract at exact head `6644035d4a86ba9a363448ecd2a182566ff5213b`, with deterministic stream/non-stream request normalization plus green REAL and pinned Paid OpenRouter gates. Important integration detail: the upstream runtime does **not** infer model legality from route-name strings and does not globally downgrade `required`. The consuming trusted provider/model profile must declare the known legal modes, e.g. `tool_choice_modes([auto])` for the affected Z.AI-backed route. With that profile, simple `required` normalizes to `auto`; providers supporting `required` preserve it; specific-function selectors are not silently weakened. Keep this issue's Docker E2E as the product proof: configure the affected live profile through the normal trusted Prolog-RLM provider configuration, then require the real Agent Zero turn to complete. Do not add an Agent Zero-local request rewrite. PR #259 is ready but remains unmerged pending the upstream repository's explicit merge-on-green authorization gate.