nsaspy/llm-log

Fork 0

Feed response and usage outcomes into expert plane #106

Open

nsaspy wants to merge 14 commits from fix/68-expert-response-ingest into fix/67-stream-integrity

nsaspy commented

2026-09-20 06:13:37 +00:00

Owner

Tracks #68.

Stacked on #86 (fix/67-stream-integrity) so response ingest can rely on explicit stream-integrity metadata without bloating that PR.

What changed

emit one safe observe_response after each immutable completed capture;
emit observe_usage only when provider-reported token counters exist;
expose the already-declared observe_response operation through SubprocessExpertPlane;
persist immutable Common Lisp/Tek9 response:<event-id> projections;
bind response projections to the request event using event ID, provider/model/transport/completion time, and captured response SHA-256;
keep raw response bodies out of expert RPC payloads;
normalize stream state to completed, incomplete, or unknown and preserve provider finish reason when present;
keep every derived expert call best-effort so expert failures cannot block forwarding or suppress independent observations;
replay historical captures through the same response + usage contract so live and backfilled graphs remain aligned;
record the verified invariants in Org-roam and the project Prolog KB.

RED evidence

Test-only head 012c370f8959f47e72beb94ebed8863cedc7a11f: 101 tests ran; only the two new live response/usage ingest contracts failed.
Expanded RED head 186f74a0374219d92c523afaf61150c20a4c8faa: 103 tests ran; exactly four new expectations failed: live response/usage emission, adapter observe_response, and Common Lisp service response projection. Existing tests, including #86 stream integrity, stayed green.

GREEN evidence

Implementation/replay head 7aaf3ddc22e8fe3157f11878125869aaf776f098 passed the expert-plane contracts and quota/package workflow; the full Python and adversarial analytics tests also passed.

Final exact head 6b2013fe8aef247fa44eb03da6994945681d505b passed all three workflows:

Analytics API run 39: success, including full Python regression + adversarial analytics contract.
Quota telemetry exact-head run 44: success, including full Python regression + delivered Python capture package build.
Expert plane contracts run 259: success, including Python expert-policy/adapter/transport contracts, Home Manager wiring, and real Common Lisp + Tek9 + SWI-Prolog integration.

Client-abort behavior remains unchanged: no completed immutable capture means no response/usage projection. This PR does not treat transport 200 as task success; response metadata remains evidence for the separate outcome expert.

Tracks #68. Stacked on #86 (`fix/67-stream-integrity`) so response ingest can rely on explicit stream-integrity metadata without bloating that PR. ## What changed - emit one safe `observe_response` after each immutable completed capture; - emit `observe_usage` only when provider-reported token counters exist; - expose the already-declared `observe_response` operation through `SubprocessExpertPlane`; - persist immutable Common Lisp/Tek9 `response:<event-id>` projections; - bind response projections to the request event using event ID, provider/model/transport/completion time, and captured response SHA-256; - keep raw response bodies out of expert RPC payloads; - normalize stream state to `completed`, `incomplete`, or `unknown` and preserve provider finish reason when present; - keep every derived expert call best-effort so expert failures cannot block forwarding or suppress independent observations; - replay historical captures through the same response + usage contract so live and backfilled graphs remain aligned; - record the verified invariants in Org-roam and the project Prolog KB. ## RED evidence - Test-only head `012c370f8959f47e72beb94ebed8863cedc7a11f`: 101 tests ran; only the two new live response/usage ingest contracts failed. - Expanded RED head `186f74a0374219d92c523afaf61150c20a4c8faa`: 103 tests ran; exactly four new expectations failed: live response/usage emission, adapter `observe_response`, and Common Lisp service response projection. Existing tests, including #86 stream integrity, stayed green. ## GREEN evidence Implementation/replay head `7aaf3ddc22e8fe3157f11878125869aaf776f098` passed the expert-plane contracts and quota/package workflow; the full Python and adversarial analytics tests also passed. Final exact head `6b2013fe8aef247fa44eb03da6994945681d505b` passed all three workflows: - Analytics API run 39: success, including full Python regression + adversarial analytics contract. - Quota telemetry exact-head run 44: success, including full Python regression + delivered Python capture package build. - Expert plane contracts run 259: success, including Python expert-policy/adapter/transport contracts, Home Manager wiring, and real Common Lisp + Tek9 + SWI-Prolog integration. Client-abort behavior remains unchanged: no completed immutable capture means no response/usage projection. This PR does not treat transport 200 as task success; response metadata remains evidence for the separate outcome expert.

nsaspy added 14 commits

2026-09-20 06:13:37 +00:00

test: define RED expert response ingest contract 012c370f89

test: require live observe_response protocol path 186f74a037

feat: expose observe_response on expert adapter 09c63338b6

feat: build safe expert response payloads 5bd9343402

feat: persist immutable response observations b72ab5a26e

build: load response observation dispatch eb0ef3efc9

feat: feed response and usage observations to expert plane 7fadbd3fca

test: cover response replay parity 7aaf3ddc22

docs: record missing expert response observation 04426c348b

docs: record response observation solution 4ba3e342a6

kb: record expert observation invariants 977021585f

docs: index response observation knowledge 7f3017268a

kb: load expert observation invariants 3729fcedd3

fix: omit absent response metadata from expert RPCs

Analytics API / test (pull_request) Failing after 17s

Details

Expert plane contracts / common-lisp-expert-integration-contract (pull_request) Failing after 9s

Details

Expert plane contracts / python-expert-policy-contract (pull_request) Failing after 25s

Details

Expert plane contracts / home-manager-expert-contract (pull_request) Failing after 26s

Details

Quota telemetry exact-head contract / quota-contracts (pull_request) Failing after 12s

Details

Analytics API / package (pull_request) Has been skipped

Details

6b2013fe8a

Analytics API / test (pull_request) Failing after 17s

Details

Expert plane contracts / common-lisp-expert-integration-contract (pull_request) Failing after 9s

Details

Expert plane contracts / python-expert-policy-contract (pull_request) Failing after 25s

Details

Expert plane contracts / home-manager-expert-contract (pull_request) Failing after 26s

Details

Quota telemetry exact-head contract / quota-contracts (pull_request) Failing after 12s

Details

Analytics API / package (pull_request) Has been skipped

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/68-expert-response-ingest:fix/68-expert-response-ingest

git switch fix/68-expert-response-ingest

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch fix/67-stream-integrity

git merge --no-ff fix/68-expert-response-ingest

git switch fix/68-expert-response-ingest

git rebase fix/67-stream-integrity

git switch fix/67-stream-integrity

git merge --ff-only fix/68-expert-response-ingest

git switch fix/68-expert-response-ingest

git rebase fix/67-stream-integrity

git switch fix/67-stream-integrity

git merge --no-ff fix/68-expert-response-ingest

git switch fix/67-stream-integrity

git merge --squash fix/68-expert-response-ingest