nsaspy/llm-log
1
0
Fork 0
Code Issues 34 Pull requests 19 Projects Releases Packages Wiki Activity Actions

Expert: success/failure outcome labeling for dataset construction #15

New issue
Open
opened 2026-08-29 12:49:52 +00:00 by lost-rob0t · 7 comments
lost-rob0t commented 2026-08-29 12:49:52 +00:00 (Migrated from github.com)
Copy link

Parent: #8
Depends on: #10, #13

Goal

Build a provenance-first outcome expert that derives task/request success and failure labels suitable for later training/evaluation datasets.

Outcome states

At minimum:

  • success
  • failure
  • partial
  • cancelled
  • rejected/denied
  • timeout
  • unknown

Do not force weak evidence into success/failure.

Evidence classes

Represent evidence separately from the derived label. Candidate evidence includes:

  • HTTP/provider completion/failure
  • tool result status
  • test/build/lint/deployment result
  • explicit user acceptance/correction/rejection
  • retry after failure
  • Beagle rewrite followed by successful/failed completion
  • task closure/completion state
  • manual authoritative label
  • downstream artifact validation

Each evidence item must carry source ID, timestamp, evidence type, strength/authority and exact observed value.

Rules

Prolog derives outcome assertions from bounded task/request evidence. Store:

  • outcome
  • scope (request/task/subtask)
  • confidence/evidence state
  • rule + version
  • evidence IDs
  • supersedes/replaced-by relation when a later stronger label changes the interpretation

Never delete earlier labels; preserve label history.

Dataset contract

Provide a query/export contract that can select examples by:

  • outcome state
  • evidence strength
  • request classification
  • provider/model
  • task cost
  • presence/absence of Beagle rewrite
  • date/rule version

Export must retain links to the immutable raw capture and derived-label provenance so labels can be audited or regenerated.

RED-first

Fixtures must prove baseline cannot distinguish:

  • provider success but user rejection;
  • partial success followed by retry;
  • explicit test failure;
  • timeout/cancelled from ordinary failure;
  • unknown outcome from weak evidence;
  • later authoritative correction superseding an earlier weak label.

Acceptance

  • deterministic fixture tasks produce correct non-binary outcome states;
  • explicit user/test evidence can outweigh transport-level 200 success;
  • weak evidence remains unknown/partial rather than hallucinated certainty;
  • every exported example includes outcome provenance;
  • label revisions remain historically auditable;
  • Tek9 can query examples by outcome + classification + cost.
Parent: #8 Depends on: #10, #13 ## Goal Build a provenance-first outcome expert that derives task/request success and failure labels suitable for later training/evaluation datasets. ## Outcome states At minimum: - success - failure - partial - cancelled - rejected/denied - timeout - unknown Do not force weak evidence into success/failure. ## Evidence classes Represent evidence separately from the derived label. Candidate evidence includes: - HTTP/provider completion/failure - tool result status - test/build/lint/deployment result - explicit user acceptance/correction/rejection - retry after failure - Beagle rewrite followed by successful/failed completion - task closure/completion state - manual authoritative label - downstream artifact validation Each evidence item must carry source ID, timestamp, evidence type, strength/authority and exact observed value. ## Rules Prolog derives outcome assertions from bounded task/request evidence. Store: - outcome - scope (request/task/subtask) - confidence/evidence state - rule + version - evidence IDs - supersedes/replaced-by relation when a later stronger label changes the interpretation Never delete earlier labels; preserve label history. ## Dataset contract Provide a query/export contract that can select examples by: - outcome state - evidence strength - request classification - provider/model - task cost - presence/absence of Beagle rewrite - date/rule version Export must retain links to the immutable raw capture and derived-label provenance so labels can be audited or regenerated. ## RED-first Fixtures must prove baseline cannot distinguish: - provider success but user rejection; - partial success followed by retry; - explicit test failure; - timeout/cancelled from ordinary failure; - unknown outcome from weak evidence; - later authoritative correction superseding an earlier weak label. ## Acceptance - deterministic fixture tasks produce correct non-binary outcome states; - explicit user/test evidence can outweigh transport-level 200 success; - weak evidence remains unknown/partial rather than hallucinated certainty; - every exported example includes outcome provenance; - label revisions remain historically auditable; - Tek9 can query examples by outcome + classification + cost.
lost-rob0t commented 2026-08-31 22:22:10 +00:00 (Migrated from github.com)
Copy link

Bounded #15 substrate merged via PR #45 at main 00e960a9e75c386054d293eaf72a3678f6fbea5a.

Canonical docs:

  • research/LLM-LOG-RESEARCH-045-outcome-evidence-substrate.org
  • evidence/LLM-LOG-EVIDENCE-031-outcome-evidence-red.org
  • evidence/LLM-LOG-EVIDENCE-032-outcome-evidence-green.org

Verified RED → GREEN:

  • accepted RED 6da93978808564e6d5242ec17ea72e0b80e647f2, run 33445207439: new operation failed with unknown_operation; prior CL/Tek9/SWI contracts passed, Python + HM green.
  • implementation GREEN 287ac94e30327ed3f987e02c03f1aa290ee95a4c, run 33445559581.
  • strengthened restart/provenance GREEN 704417d5ccf822c8fe18d37be9aa301bb4fcfa9a, run 33445653420.
  • final evidence-only exact head 70df878bf514b8e8c6f99a10e8a993e4ec910e38, run 33445728577, all three checks green.

Semantics now pinned: outcome evidence is immutable and separate from derived assertions; SWI-Prolog alone owns outcome_decision; authoritative user rejection beats provider HTTP 200; HTTP 200 alone remains unknown; exact rule/evidence provenance survives expert-host restart. Tek9 public primitives were sufficient, so no Tek9 repo change. No dotfiles mutation, #13 outcome analytics, provider-census work, or PR #5 changes were mixed into this slice.

#15 remains open. Next bounded work should add richer evidence/rules (success/failure/partial/cancelled/timeout), supersession/history query surfaces, then bounded dataset export; #13 outcome analytics should consume those semantics rather than infer from transport.

Bounded #15 substrate merged via PR #45 at main `00e960a9e75c386054d293eaf72a3678f6fbea5a`. Canonical docs: - `research/LLM-LOG-RESEARCH-045-outcome-evidence-substrate.org` - `evidence/LLM-LOG-EVIDENCE-031-outcome-evidence-red.org` - `evidence/LLM-LOG-EVIDENCE-032-outcome-evidence-green.org` Verified RED → GREEN: - accepted RED `6da93978808564e6d5242ec17ea72e0b80e647f2`, run 33445207439: new operation failed with `unknown_operation`; prior CL/Tek9/SWI contracts passed, Python + HM green. - implementation GREEN `287ac94e30327ed3f987e02c03f1aa290ee95a4c`, run 33445559581. - strengthened restart/provenance GREEN `704417d5ccf822c8fe18d37be9aa301bb4fcfa9a`, run 33445653420. - final evidence-only exact head `70df878bf514b8e8c6f99a10e8a993e4ec910e38`, run 33445728577, all three checks green. Semantics now pinned: outcome evidence is immutable and separate from derived assertions; SWI-Prolog alone owns `outcome_decision`; authoritative user rejection beats provider HTTP 200; HTTP 200 alone remains `unknown`; exact rule/evidence provenance survives expert-host restart. Tek9 public primitives were sufficient, so no Tek9 repo change. No dotfiles mutation, #13 outcome analytics, provider-census work, or PR #5 changes were mixed into this slice. #15 remains open. Next bounded work should add richer evidence/rules (success/failure/partial/cancelled/timeout), supersession/history query surfaces, then bounded dataset export; #13 outcome analytics should consume those semantics rather than infer from transport.
lost-rob0t commented 2026-08-31 23:18:42 +00:00 (Migrated from github.com)
Copy link

Bounded #15 slice merged via #47 at merge commit ac58224f1343ce58bb2189f4164e7ab9df6ab721 from exact GREEN head c74fddf94a4a47f2506568e255b117d4d99f6942 (workflow 33450033533). Canonical design: research/LLM-LOG-RESEARCH-046-outcome-supersession-history.org; RED/GREEN evidence: EVIDENCE-033 / EVIDENCE-034. Verified finite Prolog success/failure/partial/cancelled/rejected/timeout rules, HTTP-200-alone remains unknown, cross-scope supersession rejection, immutable supersession projection, and bounded restart-durable query_outcome_history. No Tek9 repo or dotfiles mutation. Keep #15 open for bounded dataset-export/corpus construction work.

Bounded #15 slice merged via #47 at merge commit `ac58224f1343ce58bb2189f4164e7ab9df6ab721` from exact GREEN head `c74fddf94a4a47f2506568e255b117d4d99f6942` (workflow `33450033533`). Canonical design: `research/LLM-LOG-RESEARCH-046-outcome-supersession-history.org`; RED/GREEN evidence: `EVIDENCE-033` / `EVIDENCE-034`. Verified finite Prolog success/failure/partial/cancelled/rejected/timeout rules, HTTP-200-alone remains unknown, cross-scope supersession rejection, immutable supersession projection, and bounded restart-durable `query_outcome_history`. No Tek9 repo or dotfiles mutation. Keep #15 open for bounded dataset-export/corpus construction work.
lost-rob0t commented 2026-09-01 00:34:35 +00:00 (Migrated from github.com)
Copy link

Merged bounded #15 outcome dataset substrate via PR #49 at exact tested head 9bf2d997d10a3b2340858a24f1141fdf655832e9; main is now 2f5e4acb5569a2f54a028e2dec30a857b6eb736b.

Canonical docs:

  • research/LLM-LOG-RESEARCH-047-outcome-dataset-export.org
  • evidence/LLM-LOG-EVIDENCE-035-outcome-dataset-red.org
  • evidence/LLM-LOG-EVIDENCE-036-outcome-dataset-upgrade-red.org
  • evidence/LLM-LOG-EVIDENCE-037-outcome-dataset-green.org

Verified: bounded query_outcome_dataset; immutable rule/expert/evidence/source provenance; superseded labels excluded by default with historical opt-in; truthful truncation; no Prolog re-inference during export; and versioned one-time Tek9 index backfill for pre-upgrade outcome assertions. Final evidence-only run 33455152519 passed Common Lisp/Tek9/SWI-Prolog, Python expert-policy/transport, and Home Manager.

Adversarial correction: temporary Tek9 issue #13 / draft PR #14 were closed unmerged after fresh source review showed Tek9 duplicate traversal was not the defect; the fixture had incorrectly assumed positive user feedback alone derived success. No Tek9 production mutation was required.

#15 remains open. Next bounded work should extend the base dataset surface with provenance-preserving joins/curation/export (classification/provider/model/task-cost as separately versioned projections), before #13 consumes #15 outcomes for expensive-failure and token-burn-before-success analytics.

Merged bounded #15 outcome dataset substrate via PR #49 at exact tested head `9bf2d997d10a3b2340858a24f1141fdf655832e9`; `main` is now `2f5e4acb5569a2f54a028e2dec30a857b6eb736b`. Canonical docs: - `research/LLM-LOG-RESEARCH-047-outcome-dataset-export.org` - `evidence/LLM-LOG-EVIDENCE-035-outcome-dataset-red.org` - `evidence/LLM-LOG-EVIDENCE-036-outcome-dataset-upgrade-red.org` - `evidence/LLM-LOG-EVIDENCE-037-outcome-dataset-green.org` Verified: bounded `query_outcome_dataset`; immutable rule/expert/evidence/source provenance; superseded labels excluded by default with historical opt-in; truthful truncation; no Prolog re-inference during export; and versioned one-time Tek9 index backfill for pre-upgrade outcome assertions. Final evidence-only run 33455152519 passed Common Lisp/Tek9/SWI-Prolog, Python expert-policy/transport, and Home Manager. Adversarial correction: temporary Tek9 issue #13 / draft PR #14 were closed unmerged after fresh source review showed Tek9 duplicate traversal was not the defect; the fixture had incorrectly assumed positive user feedback alone derived `success`. No Tek9 production mutation was required. #15 remains open. Next bounded work should extend the base dataset surface with provenance-preserving joins/curation/export (classification/provider/model/task-cost as separately versioned projections), before #13 consumes #15 outcomes for expensive-failure and token-burn-before-success analytics.
lost-rob0t commented 2026-09-01 03:19:06 +00:00 (Migrated from github.com)
Copy link

Bounded #15 dataset-metadata slice merged via PR #52.

Canonical design: research/LLM-LOG-RESEARCH-050-outcome-dataset-request-metadata.org
RED: evidence/LLM-LOG-EVIDENCE-042-outcome-dataset-request-metadata-red.org
GREEN: evidence/LLM-LOG-EVIDENCE-043-outcome-dataset-request-metadata-green.org

Verified after expert-host restart: query_outcome_dataset can now select request-scoped outcome examples by exact provider/model through bounded Tek9 request→usage lookup and returns usage/provider/model/client/transport metadata. Outcome authority remains the stored #15 assertion; export does not invoke Prolog. Existing Tek9 public primitives were sufficient, so Tek9 was not changed.

Exact evidence-only head 558d714bdbf419d41c786fcb54cc78ddf0d13775 passed all three expert-plane lanes in workflow 33465664814 before expected-head merge. #15 remains open for later bounded corpus/dataset work.

Bounded #15 dataset-metadata slice merged via PR #52. Canonical design: `research/LLM-LOG-RESEARCH-050-outcome-dataset-request-metadata.org` RED: `evidence/LLM-LOG-EVIDENCE-042-outcome-dataset-request-metadata-red.org` GREEN: `evidence/LLM-LOG-EVIDENCE-043-outcome-dataset-request-metadata-green.org` Verified after expert-host restart: `query_outcome_dataset` can now select request-scoped outcome examples by exact provider/model through bounded Tek9 request→usage lookup and returns usage/provider/model/client/transport metadata. Outcome authority remains the stored #15 assertion; export does not invoke Prolog. Existing Tek9 public primitives were sufficient, so Tek9 was not changed. Exact evidence-only head `558d714bdbf419d41c786fcb54cc78ddf0d13775` passed all three expert-plane lanes in workflow 33465664814 before expected-head merge. #15 remains open for later bounded corpus/dataset work.
lost-rob0t commented 2026-09-01 04:25:23 +00:00 (Migrated from github.com)
Copy link

Bounded Auto-ARADR/RAGE slice merged via #54 at bd377f3472cf61405f62cb79ac25c61268b46fea.

Canonical design: research/LLM-LOG-RESEARCH-051-outcome-dataset-classification.org; RED/GREEN evidence: EVIDENCE-044 / EVIDENCE-045.

Accepted RED: exact 241b7d10f9d6fbfa72713a423ce98eb2393684dd, run 33469398054 (Python + Home Manager GREEN; only the new CL/Tek9/SWI contract RED). Implementation GREEN: e1f71731da231f700ce46272667454624db69b4d, run 33469561648. Evidence-only exact head fd6f50a265bab174eb97abfc3a18dd17f7c3696a, run 33469628408, independently GREEN in all three lanes.

query_outcome_dataset now supports exact stored #12 classification dimension/value/state selectors and returns classifier assertion/rule/evidence provenance after expert-host restart. The join is bounded through existing Tek9 request/source indexes; SWI-Prolog is not re-run during export and classification metadata grants no outcome authority. No Tek9 or dotfiles mutation was required. #15 remains open for further bounded dataset/corpus work.

Bounded Auto-ARADR/RAGE slice merged via #54 at `bd377f3472cf61405f62cb79ac25c61268b46fea`. Canonical design: `research/LLM-LOG-RESEARCH-051-outcome-dataset-classification.org`; RED/GREEN evidence: `EVIDENCE-044` / `EVIDENCE-045`. Accepted RED: exact `241b7d10f9d6fbfa72713a423ce98eb2393684dd`, run 33469398054 (Python + Home Manager GREEN; only the new CL/Tek9/SWI contract RED). Implementation GREEN: `e1f71731da231f700ce46272667454624db69b4d`, run 33469561648. Evidence-only exact head `fd6f50a265bab174eb97abfc3a18dd17f7c3696a`, run 33469628408, independently GREEN in all three lanes. `query_outcome_dataset` now supports exact stored #12 classification dimension/value/state selectors and returns classifier assertion/rule/evidence provenance after expert-host restart. The join is bounded through existing Tek9 request/source indexes; SWI-Prolog is not re-run during export and classification metadata grants no outcome authority. No Tek9 or dotfiles mutation was required. #15 remains open for further bounded dataset/corpus work.
lost-rob0t commented 2026-09-01 05:20:43 +00:00 (Migrated from github.com)
Copy link

Bounded #15 task-cost dataset slice merged via #55.

Canonical design: research/LLM-LOG-RESEARCH-052-outcome-dataset-task-cost.org
RED evidence: evidence/LLM-LOG-EVIDENCE-046-outcome-dataset-task-cost-red.org
GREEN evidence: evidence/LLM-LOG-EVIDENCE-047-outcome-dataset-task-cost-green.org

Accepted RED: d8f0a86a58a0337ba8757a681a352c30fa6bdf18, workflow 33472780086 — existing gates stayed green except the new CL/Tek9/SWI contract, which proved task-cost selectors were ignored.

GREEN implementation: 662acc1ca9099a300a9bff01d26d124d50568b78, workflow 33472989667. Final evidence-only exact head 3e53d01de73eda503211589d7d27f207d9f55e32, workflow 33473123450, independently passed Common Lisp/Tek9/SWI-Prolog, Python expert policy/transport, and Home Manager.

Behavior: query_outcome_dataset now supports bounded task cost state/currency/min/max selection by reusing #13 query-task-accounting; numeric bounds only match complete known costs, task projections are cached per query, truncated accounting is rejected, and immutable usage/cost assertion provenance is returned. No export-time Prolog, price recomputation, Tek9 repo mutation, Python transport mutation, or dotfiles mutation.

Merged main: f3f146c160afc91509e96afcde3db4040517550d.

Leaving #15 open for remaining bounded dataset/corpus acceptance work.

Bounded #15 task-cost dataset slice merged via #55. Canonical design: `research/LLM-LOG-RESEARCH-052-outcome-dataset-task-cost.org` RED evidence: `evidence/LLM-LOG-EVIDENCE-046-outcome-dataset-task-cost-red.org` GREEN evidence: `evidence/LLM-LOG-EVIDENCE-047-outcome-dataset-task-cost-green.org` Accepted RED: `d8f0a86a58a0337ba8757a681a352c30fa6bdf18`, workflow `33472780086` — existing gates stayed green except the new CL/Tek9/SWI contract, which proved task-cost selectors were ignored. GREEN implementation: `662acc1ca9099a300a9bff01d26d124d50568b78`, workflow `33472989667`. Final evidence-only exact head `3e53d01de73eda503211589d7d27f207d9f55e32`, workflow `33473123450`, independently passed Common Lisp/Tek9/SWI-Prolog, Python expert policy/transport, and Home Manager. Behavior: `query_outcome_dataset` now supports bounded task cost state/currency/min/max selection by reusing #13 `query-task-accounting`; numeric bounds only match complete known costs, task projections are cached per query, truncated accounting is rejected, and immutable usage/cost assertion provenance is returned. No export-time Prolog, price recomputation, Tek9 repo mutation, Python transport mutation, or dotfiles mutation. Merged main: `f3f146c160afc91509e96afcde3db4040517550d`. Leaving #15 open for remaining bounded dataset/corpus acceptance work.
nsaspy commented 2026-09-15 02:40:47 +00:00
Owner
Copy link

Historical corpus backfill/query/export slice merged via #84 at merge commit 692d3820f638c011fc248a7694ecabd72a6e2895 from exact GREEN head 70561b68ce6346e33f30645ccb8aec556415698c.

Delivered:

  • loopback-only maintenance/query surface using the existing proxy-owned expert child;
  • resumable/idempotent events.jsonl replay with stable IDs and checkpoints;
  • request-level historical usage projection without synthetic task/pricing state;
  • optional weak transport evidence (HTTP success still does not imply task success);
  • cursor-paginated query_outcome_dataset proven across 257 examples;
  • provenance-complete raw-capture JSONL export + SHA-256 manifest;
  • expert-affecting PRs now run the CL/Tek9/SWI + Python + Home Manager expert CI lanes.

Final PR head GREEN: Expert plane run 34921711756, Analytics run 34921711765, Quota run 34921711768. Canonical design/evidence: LLM-LOG-RESEARCH-053 / LLM-LOG-EVIDENCE-048.

Keep #15 open for later bounded training projections such as SFT/DPO/tool-trajectory dataset transforms.

Historical corpus backfill/query/export slice merged via #84 at merge commit `692d3820f638c011fc248a7694ecabd72a6e2895` from exact GREEN head `70561b68ce6346e33f30645ccb8aec556415698c`. Delivered: - loopback-only maintenance/query surface using the existing proxy-owned expert child; - resumable/idempotent `events.jsonl` replay with stable IDs and checkpoints; - request-level historical usage projection without synthetic task/pricing state; - optional weak transport evidence (HTTP success still does not imply task success); - cursor-paginated `query_outcome_dataset` proven across 257 examples; - provenance-complete raw-capture JSONL export + SHA-256 manifest; - expert-affecting PRs now run the CL/Tek9/SWI + Python + Home Manager expert CI lanes. Final PR head GREEN: Expert plane run `34921711756`, Analytics run `34921711765`, Quota run `34921711768`. Canonical design/evidence: `LLM-LOG-RESEARCH-053` / `LLM-LOG-EVIDENCE-048`. Keep #15 open for later bounded training projections such as SFT/DPO/tool-trajectory dataset transforms.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/108-full-kb-mcp-20260920
feat/mcp-expert-kb-20260920
feat/105-opencode-profile
feat/104-python-admission
feat/starworld-rsi
feat/58-structured-output-anomaly
feat/58-tool-schema-anomaly
feat/58-tool-call-anomaly
fix/61-alert-policy
fix/61-retry-routing-correlation
fix/61-transport-error-evidence
fix/68-expert-response-ingest
fix/67-stream-integrity
feature/cl-native-bulk-infill
feature/expert-backfill-query-export
feat/79-quota-telemetry
feat/79-provider-quota-telemetry
fix/74-relay-socket-ownership
feat/stats-analytics-api
feat/71-rate-profile-foundation
feat/75-rate-admission
feat/70-request-scheduler
rage/59-init-files-r3
fix/websocket-expert-plane
rage/58-60-quant-anomaly-alerts-r4
rage/58-60-quant-anomaly-alerts-r3
rage/57-openrouter-quant-observability-r2
rage/58-60-quant-anomaly-alerts-r2
rage/59-init-files-r2
rage/58-60-quant-anomaly-alerts
rage/59-init-files
rage/57-openrouter-quant-observability
rage/openrouter-quantization-presets
noop-unused
rage/15-outcome-dataset-task-cost-red
rage/15-outcome-dataset-classification-red
rage/15-outcome-dataset-metadata-red
rage/13-outcome-retry-economics-red
rage/13-outcome-breakdown-red
rage/15-outcome-dataset-red
rage/15-outcome-history-red
rage/15-outcome-evidence-red
rage/13-classification-breakdown-red
rage/13-usage-breakdown-red
rage/13-retry-attempt-red
rage/13-task-rollup-red
rage/13-task-accounting-red
rage/12-classifier-task-id-red
rage/12-classifier-time-range-red
rage/12-classifier-query-red
rage/12-request-classifier-red
rage/10-expert-runtime-red
rage/10-recorder-deps-red
rage/4-unified-config
rage/home-manager-service
rage/1-capture-proxy-prolog-kb
No results found.
Labels
Clear labels   
accessibility
Barrier affecting people with disabilities

  
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/llm-log#15
No description provided.