Expert: Logical task tracker with per-task cost attribution #14

New issue

Closed

opened 2026-08-29 12:49:40 +00:00 by lost-rob0t · 1 comment

lost-rob0t commented

2026-08-29 12:49:40 +00:00

(Migrated from github.com)

Parent: #7

Goal

Build a task-tracking expert that groups captured LLM activity into logical tasks/subtasks and attributes provider/model/token/tool costs to those tasks for later analysis.

The important object is the task, not the HTTP request. One user goal may span many model calls, retries, tool calls, subagents, conversations, or providers.

Required entities

task
subtask
task-attempt
provider-call
tool-call
usage record
cost record
task transition
attribution evidence

Task inference

Prolog should derive task continuity/splitting from explicit task IDs when available and symbolic evidence when they are not. Signals may include:

conversation/message ancestry
user goal continuity
project/repository identity
rewrite/classification output
tool/issue/PR/artifact references
explicit start/finish/retry markers
time adjacency only as weak evidence

Do not silently merge ambiguous work. Support unknown_task / candidate membership until stronger evidence exists.

Cost model

Track raw usage separately from derived monetary cost:

provider
model
input tokens
cached input/read tokens when exposed
output tokens
reasoning tokens when exposed
request count
tool/subagent execution counts
wall-clock duration
provider-reported cost when available
locally calculated cost with pricing-table version and source

Pricing is time/version dependent. Never overwrite old cost derivations when pricing tables change; store the pricing revision used.

Tek9 graph examples

request -> assigned_to -> task
subtask -> part_of -> task
provider-call -> incurred_for -> task
provider-call -> used_model -> model-version
cost-record -> prices -> provider-call
task-attempt -> attempts -> task
task-attempt -> produced -> outcome

Use secondary indexes for task ID, provider, model, timestamp, project, status and classification labels so aggregate analysis does not require full scans.

Queries this expert must eventually answer

what did task X cost?
cost by model/provider for task X
cost by request classification
cost of successful vs failed attempts
cost of rewrites/retries
which subtasks dominate spend?
what is the cheapest successful model/provider for this task class?
how much spend is unassigned/ambiguous?

RED-first acceptance

Before implementation, tests must fail proving the baseline cannot:

group multiple provider calls under one logical task;
represent nested subtasks;
keep raw usage separate from priced monetary cost;
reprice historical usage without mutating the original cost record;
preserve ambiguous/unassigned attribution rather than guessing;
aggregate task totals deterministically from stored usage/cost records.

Dataset/analysis value

Task IDs become the join key between request classification, rewrites, provider/model usage, outcomes and later training/evaluation datasets. This expert should make it possible to measure not merely cost-per-call but cost-per-successful-task.

Parent: #7 ## Goal Build a task-tracking expert that groups captured LLM activity into logical tasks/subtasks and attributes provider/model/token/tool costs to those tasks for later analysis. The important object is the **task**, not the HTTP request. One user goal may span many model calls, retries, tool calls, subagents, conversations, or providers. ## Required entities - task - subtask - task-attempt - provider-call - tool-call - usage record - cost record - task transition - attribution evidence ## Task inference Prolog should derive task continuity/splitting from explicit task IDs when available and symbolic evidence when they are not. Signals may include: - conversation/message ancestry - user goal continuity - project/repository identity - rewrite/classification output - tool/issue/PR/artifact references - explicit start/finish/retry markers - time adjacency only as weak evidence Do not silently merge ambiguous work. Support `unknown_task` / candidate membership until stronger evidence exists. ## Cost model Track raw usage separately from derived monetary cost: - provider - model - input tokens - cached input/read tokens when exposed - output tokens - reasoning tokens when exposed - request count - tool/subagent execution counts - wall-clock duration - provider-reported cost when available - locally calculated cost with pricing-table version and source Pricing is time/version dependent. Never overwrite old cost derivations when pricing tables change; store the pricing revision used. ## Tek9 graph examples ```text request -> assigned_to -> task subtask -> part_of -> task provider-call -> incurred_for -> task provider-call -> used_model -> model-version cost-record -> prices -> provider-call task-attempt -> attempts -> task task-attempt -> produced -> outcome ``` Use secondary indexes for task ID, provider, model, timestamp, project, status and classification labels so aggregate analysis does not require full scans. ## Queries this expert must eventually answer - what did task X cost? - cost by model/provider for task X - cost by request classification - cost of successful vs failed attempts - cost of rewrites/retries - which subtasks dominate spend? - what is the cheapest successful model/provider for this task class? - how much spend is unassigned/ambiguous? ## RED-first acceptance Before implementation, tests must fail proving the baseline cannot: 1. group multiple provider calls under one logical task; 2. represent nested subtasks; 3. keep raw usage separate from priced monetary cost; 4. reprice historical usage without mutating the original cost record; 5. preserve ambiguous/unassigned attribution rather than guessing; 6. aggregate task totals deterministically from stored usage/cost records. ## Dataset/analysis value Task IDs become the join key between request classification, rewrites, provider/model usage, outcomes and later training/evaluation datasets. This expert should make it possible to measure not merely cost-per-call but **cost-per-successful-task**.

lost-rob0t commented

2026-08-30 00:39:35 +00:00

(Migrated from github.com)

Closed as duplicate. Canonical task/cost expert is #13 under #8 and depends on #10.