Expert: task graph + per-task token/cost accounting #13
Labels
No labels
accessibility
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
No milestone
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
nsaspy/llm-log#13
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Parent: #8
Depends on: #10
Goal
Build a task-tracking expert that turns raw LLM traffic into explicit task/subtask graphs and reproducible usage/cost accounting.
Task model
Persist stable Tek9 entities/edges for:
A task may span multiple provider requests, retries, rewrites, tools, models, and child tasks.
Identity rules
Use explicit task IDs from clients when available. Otherwise derive/assign stable local task IDs from observed message/session boundaries and expert rules. Never silently merge unrelated work solely because it shares a chat/session.
Store why a request was attached to a task.
Usage
Capture and normalize when available:
Missing usage remains unknown.
Pricing/cost
Costs must be reproducible:
unknown, never guessed;Analysis queries
Support at least:
RED-first
Fixtures must prove baseline cannot correctly account for:
Acceptance
Bounded #13 substrate merged via #35 at
bee9fa8dd3ad1d7ff375359104a48fcfb5ce99b0after exact-head GREEN on982f4dea839f459db04799919a55e0389b1b0ac9(workflow 33408766713). Canonical design remainsresearch/LLM-LOG-RESEARCH-040-task-graph-usage-cost.org; RED/GREEN evidence is in EVIDENCE-021/022.This slice establishes typed task/usage/pricing/cost materialization in Common Lisp/Tek9, declared SWI-Prolog
task_cost, explicit unknown pricing, immutable pricing provenance, and bounded request deduplication. No Tek9 repo or dotfiles mutation.#13 stays open. Remaining acceptance is RED-first: recursive parent/child rollups without double counting, restart-preserved totals/query surfaces, retries, and analysis by model/provider/client/classification/outcome including expensive failures and token burn before success.
Bounded #13 rollup/restart slice merged via #39 at
e33ff07889b0095ed9a0f12084d098cbdc3fda44from exact tested head1c8a0947759e0941480da03a3966673d1e075205(run33418411885GREEN across CL/Tek9/SWI, Python expert-policy/transport, and Home Manager). Canonical design:research/LLM-LOG-RESEARCH-041-task-rollup-restart.org; RED/GREEN evidence:evidence/LLM-LOG-EVIDENCE-023-task-rollup-restart-red.organd...024-task-rollup-restart-green.org.Keep #13 OPEN. Remaining bounded acceptance: retry/attempt accounting and analysis/query dimensions by model/provider/client/classification/outcome, including expensive failures and token burn before success. No Tek9 or dotfiles mutation was needed for this slice.
Bounded #13 retry/attempt slice reached an accepted RED gate.
Canonical design:
research/LLM-LOG-RESEARCH-042-task-retry-attempt-accounting.orgEvidence:
evidence/LLM-LOG-EVIDENCE-025-task-retry-attempt-red.orgDraft PR: #41
Executed RED head:
fdc351217621efa6efafc4a7131b1753b2bb0828, workflow 33423647092. Existing #10/#12/#13 integration behavior, Python expert-policy/transport, and Home Manager remained green. The new restart fixture preserved 3 requests, 240 input tokens, 120 output tokens, and 0.48 USD, butattempt_count,retry_request_count,attempt_ids, andretry_request_idswere absent. This verifies that current production bills distinct retry requests but drops explicit attempt/retry identity from the durable usage projection.No production mutation was made in this bounded cycle. Next slice is limited to Common Lisp validation/persistence of explicit attempt/retry metadata plus bounded Tek9 restart reconstruction through the existing task/usage index path. No Python rule engine, outcome dependency, corpus scan, Tek9 repo change, or dotfiles change.
Bounded #13 retry/attempt slice merged via PR #42.
Canonical design:
research/LLM-LOG-RESEARCH-042-task-retry-attempt-accounting.org.Evidence:
evidence/LLM-LOG-EVIDENCE-025-task-retry-attempt-red.organdevidence/LLM-LOG-EVIDENCE-026-task-retry-attempt-green.org.RED/GREEN chain:
fdc351217621efa6efafc4a7131b1753b2bb0828, run33423647092— durable task accounting preserved 3 requests / 240 input / 120 output / 0.48 USD but attempt/retry metadata was absent after restart;184c5963b921775d783d105bcea6094522751fa1, run33429223078GREEN;01954012c761524bea5cabcfffb7e6222e235da5, run33429724537— legacy immutable replay failed specifically withusage_conflict: usage:usage-retry-legacybecause absent optional metadata keys changed projection identity;1d8e50dca6a885b601681d3ac65130311eb4e71c, run33429869506GREEN across Common Lisp/Tek9/SWI, Python policy/transport, and Home Manager;beaab8aa2806d6bb2a7081b6b579a77e781d551c, run33429992265GREEN across all three lanes;3e6ba6775623f9d59e8f67f9f984402e1f38cbbc.Verified behavior: Common Lisp validates and immutably persists explicit attempt identity / ordinal / retry predecessor when supplied; bounded
query_task_accountingreconstructs attempt/retry IDs and counts after restart using the existing Tek9 task/usage indexes; missing attempt metadata stays explicit and does not suppress usage/cost; retries are never guessed from timestamps/session adjacency; old usage projections replay idempotently after upgrade. SWI-Prolog remains pricing-inference owner. No Python retry rule engine, Tek9 repo mutation, or dotfiles mutation was introduced.Keep #13 OPEN. Remaining acceptance is the cross-task analysis/query slice by model/provider/client/classification/outcome, including expensive failures/retries and token burn before eventual success. Outcome-dependent analysis should respect #15's evidence-vs-label boundary rather than smuggling transport success into task outcome.
Bounded #13 provider/model/client analytics slice completed and merged via #43.
Canonical artifacts:
research/LLM-LOG-RESEARCH-043-task-usage-breakdowns.orgevidence/LLM-LOG-EVIDENCE-027-task-usage-breakdown-red.orgevidence/LLM-LOG-EVIDENCE-028-task-usage-breakdown-green.orgAccepted RED:
9f3e54909fa3d9527bc7fc658c9598e9ac7d193f, run 33435473703. Existing aggregate/retry contracts were green after restart; the new fixture failed specifically becausequery_task_accountinghad nobreakdownsprojection.Exact evidence-only GREEN head:
4eaabb3ddc60fc288819d4f4bffb44b89ec708f4, run 33435931708, all Common Lisp/Tek9/SWI-Prolog, Python expert-policy/transport, and Home Manager lanes green. Merged as2c8a07fb5031cb7175925c41bd7d06db6a14070e.Verified restart-durable breakdowns now cover provider/model/client with bounded task-rooted retrieval, token/request counts, known/partial/unknown pricing state, and usage/cost provenance. No Tek9 repo or dotfiles mutation was required.
Keep #13 open: classification breakdowns still need a provenance-preserving join to #12 assertions, and outcome/expensive-failure/token-burn-before-success analytics must wait for #15's explicit outcome evidence/label semantics rather than treating transport success as task success.
Bounded classification-analytics slice merged via #44.
Canonical design:
research/LLM-LOG-RESEARCH-044-task-classification-breakdowns.orgRED evidence:
evidence/LLM-LOG-EVIDENCE-029-task-classification-breakdown-red.orgGREEN evidence:
evidence/LLM-LOG-EVIDENCE-030-task-classification-breakdown-green.orgExact evidence head
c30920ef00153e247deb7a1e454ce2e6bfe9035bpassed Common Lisp/Tek9/SWI-Prolog, Python expert policy/transport, and Home Manager in workflow 33441459372, then #44 merged with expected-head protection as merge commit79cac2906f2a70764b021d6c3c1fe2107601cc71.Now verified after expert-host restart: task-rooted classification breakdowns are joined through bounded Tek9 request/source/assertion indexes; multi-label and ambiguous states remain explicit; usage/cost/classifier assertion/rule/evidence provenance is preserved. No Tek9 repo mutation, Python rule engine, outcome inference, or dotfiles change.
#13 remains open. Provider/model/client/classification analytics are now covered. Outcome/failure/partial breakdowns, expensive-failure analysis, and token burn before eventual success must consume #15's explicit outcome evidence/label semantics rather than infer success from transport status.
Bounded #13 outcome-breakdown slice merged via PR #50.
Canonical design:
research/LLM-LOG-RESEARCH-048-task-outcome-breakdowns.org.Accepted RED:
evidence/LLM-LOG-EVIDENCE-038-task-outcome-breakdown-red.orgat54dd2d97263a6b59c340c8bd47ce9160ae4ad255/ run 33456066504.Implementation GREEN:
8f29d0c153efb3ddbb0fac21c31de00a4de124d0/ run 33457650079.Evidence-only exact head
a1e7ed2b951d50e1d04ced24b5d49915b63c5222/ run 33457811497 passed Common Lisp/Tek9/SWI-Prolog, Python expert policy/transport, and Home Manager before expected-head merge.Verified: bounded request-local join to current stored #15 outcome assertions; explicit
unknownstays distinct fromunlabeled; superseded assertions are excluded; multiple current assertions fail as integrity errors; usage/cost plus outcome assertion/rule/evidence provenance survives restart. No transport-status outcome inference, no Python rule engine, no Tek9 repo mutation, no dotfiles mutation.Keep #13 open. Next bounded analytics boundary is expensive-failure/retry and token-burn-before-eventual-success analysis using explicit #15 labels plus existing retry/cost projections.
#13 acceptance is now complete after merged PR #51 (
d46bfee8a9cc66095cb102a5ceeb75e1f46f6bcd).Canonical final slice:
research/LLM-LOG-RESEARCH-049-outcome-retry-economics.org; RED/GREEN evidence:evidence/LLM-LOG-EVIDENCE-040-task-outcome-retry-economics-red.organd...041-task-outcome-retry-economics-green.org.Final RED:
da000c95c0f466158fce22e615d00278c9a9604b, run33461795855: existing task/retry/outcome behavior, Python expert policy/transport, and Home Manager were green; the new contract failed only becausequery_task_accounting.result.analysiswas absent.Tightened GREEN:
f2cb7ca9f182767c0ced05c6daddfd13e71bf4ea, run33462054052; final evidence-only exact headacb52ff6993b90bc90bc20423d79d3de53e04435, run33462157182, green across Common Lisp/Tek9/SWI-Prolog, Python expert policy/transport, and Home Manager.The completed task-accounting surface now covers bounded task/child rollups, restart durability, retries/attempts, immutable pricing provenance and unknown-price semantics, provider/model/client/classification/outcome breakdowns, explicit expensive-unsuccessful/retry projections, and token/cost burn before successful completion. “Before success” is strict explicit
retry_of_request_idancestry only; HTTP status/timestamps/session adjacency never grant outcome authority. No Tek9 repo mutation or Python rule engine was required.Extension requirement from #98: add stable session identity and session-level cost/token rollups to the existing task accounting substrate. Session analytics must remain restart/backfill durable, preserve immutable pricing provenance, and use bounded indexed queries rather than whole-corpus scans.