EPIC: Live Docker end-to-end CI and provider compatibility #185

New issue

Open

opened 2026-09-09 15:05:29 +00:00 by nsaspy · 0 comments

nsaspy commented

2026-09-09 15:05:29 +00:00

Owner

Parent: #1

Goal

Make CI prove that the real Dockerized a0-symbolics product works end-to-end, including Agent Zero startup, the single Prolog-RLM integration, prompt/context compilation, the actual model transport, OpenRouter, native tool schemas, and a completed live agent turn.

Unit tests and direct Python bridge calls remain useful but are not sufficient release evidence.

Current failure this epic must catch

A real OpenRouter/Z.AI request currently fails with:

Tool choice must be auto

The Responses transport can synthesize tool_choice: required when function tools are present. Provider-specific semantics must be normalized before dispatch so Z.AI/OpenRouter receives auto where required. The live test must reproduce and prevent this class of integration failure.

Live CI contract

CI must build the same Docker image path used by Symbolics development/production and start the real service. The test then exercises the externally reachable application rather than importing plugin bridge classes directly.

Required path:

build DockerfileLocal or the exact production-equivalent Symbolics image;
boot the container with Prolog-RLM enabled;
wait for the real healthcheck and WebUI/API readiness;
assert only one Prolog-RLM plugin is discovered;
execute a real chat/agent turn through Agent Zero;
send a real OpenRouter request using the configured live model;
require at least one real native-tool-capable turn;
verify the turn completes through the normal response-tool path;
verify provider-visible context respects the configured symbolic budget;
retain logs/artifacts on failure;
always destroy containers/volumes created by CI.

Secrets and variables

Required GitHub Actions secret:

OPENROUTER_API_KEY

Non-secret repository variable:

SYMBOLICS_LIVE_MODEL — live CI model override. Keep a cheap known-good default in workflow/config when unset.

No other provider credential should be required for the default live gate.

Test tiers

Pull-request gate

deterministic Docker build/startup
offline/unit/integration tests
one small live OpenRouter smoke when the secret is available
explicit skip reason when secrets are unavailable on untrusted forks

Main/release gate

mandatory live OpenRouter smoke
tool-capable model path
provider compatibility matrix for at least OpenRouter generic + Z.AI-backed route when configured
failure artifacts uploaded

Runtime assertions

Prolog-RLM reports ready authority.
Exactly one Prolog-RLM plugin is enabled/discovered.
Persistent Prolog worker survives more than one turn.
Context projection is used by the real turn, not bypassed.
Minimum starting symbolic context budget is 500 tokens unless a test explicitly overrides it.
Tool schemas remain authorized by Agent Zero policy; visibility never grants authority.
Provider-specific tool_choice is legal for the selected model/provider.
A provider 400/422 is a hard CI failure, not a successful fallback hidden from the gate.
Transport fallback, if intentionally exercised, is visible in test output.

Failure enumeration artifact

Every failed live run should capture enough evidence to enumerate problems quickly:

docker compose ps
health status
supervisor status
container logs
Agent Zero application logs
Prolog worker stderr/status
selected provider/model/mode
sanitized outbound request metadata
response status/error body
context token ledger
enabled plugin list
active native tool names
git SHA/image tag

Never upload API keys, auth cookies, passwords, or secret values.

Acceptance

CI builds and boots the actual Symbolics Docker image.
CI talks to the real running application instead of only importing bridge classes.
A real OpenRouter-backed agent turn completes.
Native tools are present in at least one live request.
Z.AI/OpenRouter tool_choice compatibility is covered by a regression.
Exactly one Prolog-RLM plugin is discovered in the container.
500-token starting symbolic context behavior is asserted.
Failure diagnostics are uploaded as artifacts with secrets redacted.
Main/release cannot go green if the mandatory live test fails.
Local developer commands reproduce the same Docker test path.

Local developer command target

Provide one canonical command, ideally something equivalent to:

OPENROUTER_API_KEY=... SYMBOLICS_LIVE_MODEL=... ./scripts/test-symbolics-live

The script should build/start/test/diagnose/clean up itself so local and CI behavior do not drift.

Mirrored from lost-rob0t/a0-symbolics#58 via tracker sync.

Parent: #1 ## Goal Make CI prove that the real Dockerized `a0-symbolics` product works end-to-end, including Agent Zero startup, the single Prolog-RLM integration, prompt/context compilation, the actual model transport, OpenRouter, native tool schemas, and a completed live agent turn. Unit tests and direct Python bridge calls remain useful but are not sufficient release evidence. ## Current failure this epic must catch A real OpenRouter/Z.AI request currently fails with: ```text Tool choice must be auto ``` The Responses transport can synthesize `tool_choice: required` when function tools are present. Provider-specific semantics must be normalized before dispatch so Z.AI/OpenRouter receives `auto` where required. The live test must reproduce and prevent this class of integration failure. ## Live CI contract CI must build the same Docker image path used by Symbolics development/production and start the real service. The test then exercises the externally reachable application rather than importing plugin bridge classes directly. Required path: 1. build `DockerfileLocal` or the exact production-equivalent Symbolics image; 2. boot the container with Prolog-RLM enabled; 3. wait for the real healthcheck and WebUI/API readiness; 4. assert only one Prolog-RLM plugin is discovered; 5. execute a real chat/agent turn through Agent Zero; 6. send a real OpenRouter request using the configured live model; 7. require at least one real native-tool-capable turn; 8. verify the turn completes through the normal response-tool path; 9. verify provider-visible context respects the configured symbolic budget; 10. retain logs/artifacts on failure; 11. always destroy containers/volumes created by CI. ## Secrets and variables Required GitHub Actions secret: - `OPENROUTER_API_KEY` Non-secret repository variable: - `SYMBOLICS_LIVE_MODEL` — live CI model override. Keep a cheap known-good default in workflow/config when unset. No other provider credential should be required for the default live gate. ## Test tiers ### Pull-request gate - deterministic Docker build/startup - offline/unit/integration tests - one small live OpenRouter smoke when the secret is available - explicit skip reason when secrets are unavailable on untrusted forks ### Main/release gate - mandatory live OpenRouter smoke - tool-capable model path - provider compatibility matrix for at least OpenRouter generic + Z.AI-backed route when configured - failure artifacts uploaded ## Runtime assertions - Prolog-RLM reports ready authority. - Exactly one Prolog-RLM plugin is enabled/discovered. - Persistent Prolog worker survives more than one turn. - Context projection is used by the real turn, not bypassed. - Minimum starting symbolic context budget is 500 tokens unless a test explicitly overrides it. - Tool schemas remain authorized by Agent Zero policy; visibility never grants authority. - Provider-specific `tool_choice` is legal for the selected model/provider. - A provider 400/422 is a hard CI failure, not a successful fallback hidden from the gate. - Transport fallback, if intentionally exercised, is visible in test output. ## Failure enumeration artifact Every failed live run should capture enough evidence to enumerate problems quickly: - `docker compose ps` - health status - supervisor status - container logs - Agent Zero application logs - Prolog worker stderr/status - selected provider/model/mode - sanitized outbound request metadata - response status/error body - context token ledger - enabled plugin list - active native tool names - git SHA/image tag Never upload API keys, auth cookies, passwords, or secret values. ## Acceptance - [ ] CI builds and boots the actual Symbolics Docker image. - [ ] CI talks to the real running application instead of only importing bridge classes. - [ ] A real OpenRouter-backed agent turn completes. - [ ] Native tools are present in at least one live request. - [ ] Z.AI/OpenRouter `tool_choice` compatibility is covered by a regression. - [ ] Exactly one Prolog-RLM plugin is discovered in the container. - [ ] 500-token starting symbolic context behavior is asserted. - [ ] Failure diagnostics are uploaded as artifacts with secrets redacted. - [ ] Main/release cannot go green if the mandatory live test fails. - [ ] Local developer commands reproduce the same Docker test path. ## Local developer command target Provide one canonical command, ideally something equivalent to: ```bash OPENROUTER_API_KEY=... SYMBOLICS_LIVE_MODEL=... ./scripts/test-symbolics-live ``` The script should build/start/test/diagnose/clean up itself so local and CI behavior do not drift. --- *Mirrored from [`lost-rob0t/a0-symbolics#58`](https://github.com/lost-rob0t/a0-symbolics/issues/58)* via tracker sync.