[P1 cost safety] Agent cost budgets fail open when pricing is missing or set to zero #132

New issue

Open

opened 2026-07-28 23:00:51 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-28 23:00:51 +00:00

(Migrated from github.com)

Finding

Agent cost limits are presented as hard budgets, but the implementation fails open in several common states.

Missing pricing is treated as free

AgentSystem.jsx supplies an all-zero pricing record whenever a model has no configured price:

return configured || {
  inputPerMillionUsd: 0,
  outputPerMillionUsd: 0,
  cachedInputPerMillionUsd: 0
};

calculateCost() then records every request as $0, so maxCostUsd, daily limits, and monthly limits never stop the run. There appears to be no normal UI path that requires pricing before enabling a paid model.

Explicit zero limits become unlimited

The launch check uses || Infinity:

Number(activeAgent.budget?.maxCostUsd || Infinity)

An explicit 0 therefore becomes unlimited. budgetState() also skips every zero-valued limit:

const limit = Number(policy?.[policyKey] || 0);
if (!limit) continue;

The hard limit is checked after the provider charge

A full model request is made before its calculated cost is checked. One call can exceed the remaining budget by the entire request amount.

Required fix

Fail closed for paid/unknown remote models

Represent unknown pricing explicitly rather than as zero:

function pricingForModel(modelId) {
  const pricing = settings.agentModelPricing?.[modelId];
  if (!pricing) return { known: false };
  return {
    known: true,
    inputPerMillionUsd: Number(pricing.inputPerMillionUsd),
    outputPerMillionUsd: Number(pricing.outputPerMillionUsd),
    cachedInputPerMillionUsd: Number(
      pricing.cachedInputPerMillionUsd ?? pricing.inputPerMillionUsd
    )
  };
}

Before a paid remote run:

if (!pricing.known && Number(agent.budget.maxCostUsd) > 0) {
  throw new Error(
    `Pricing is required before enforcing a cost budget for ${agent.modelId}`
  );
}

Local/free models should be marked explicitly as free; absence of data must not mean free.

Preserve zero as a real hard stop

function numericLimit(value, fallback = Infinity) {
  if (value === undefined || value === null || value === "") return fallback;
  const limit = Number(value);
  if (!Number.isFinite(limit) || limit < 0) throw new TypeError("Invalid budget limit");
  return limit;
}

const maxCost = numericLimit(activeAgent.budget?.maxCostUsd);
if (maxCost === 0) throw new Error("Agent cost budget is disabled");

budgetState() must distinguish absent limits from zero:

if (policy?.[policyKey] == null) continue;
const limit = Number(policy[policyKey]);
if (limit === 0) {
  return { state: "hard-stop", reason: `${label} limit reached`, ratio: 1, usage: next };
}

Bound the next request before sending it

Derive the maximum output tokens from all remaining limits:

const remainingOutputTokens = Math.max(
  0,
  run.budget.maxOutputTokens - run.usage.outputTokens
);

const remainingCostUsd = Math.max(
  0,
  run.budget.maxCostUsd - run.usage.costUsd
);

const costBoundedOutputTokens = pricing.known && pricing.outputPerMillionUsd > 0
  ? Math.floor((remainingCostUsd / pricing.outputPerMillionUsd) * 1_000_000)
  : remainingOutputTokens;

const maxTokens = Math.min(
  8_192,
  remainingOutputTokens,
  costBoundedOutputTokens
);

if (maxTokens < 1) return budgetExhausted();

Because input token cost is not known exactly before sending, reserve a conservative estimate or clearly label the cost cap as approximate and require a configurable safety margin.

Prevent concurrent global-budget races

Daily/monthly budget checking and run creation need a reservation/lease in one serialized application-state operation. Two runs must not both observe the same remaining global budget.

Acceptance criteria

Missing remote-model pricing blocks cost-budgeted runs instead of recording $0.
Explicit zero run/daily/monthly limits prevent execution.
Local/free pricing is explicit.
The next request is bounded by remaining output-token and estimated cost capacity.
Concurrent runs cannot independently spend the same daily/monthly allowance.
UI states whether a cost limit is exact, estimated, or unenforceable.
Tests cover missing pricing, zero limits, a one-call overshoot, and concurrent launches.

## Finding Agent cost limits are presented as hard budgets, but the implementation fails open in several common states. ### Missing pricing is treated as free `AgentSystem.jsx` supplies an all-zero pricing record whenever a model has no configured price: ```js return configured || { inputPerMillionUsd: 0, outputPerMillionUsd: 0, cachedInputPerMillionUsd: 0 }; ``` `calculateCost()` then records every request as `$0`, so `maxCostUsd`, daily limits, and monthly limits never stop the run. There appears to be no normal UI path that requires pricing before enabling a paid model. ### Explicit zero limits become unlimited The launch check uses `|| Infinity`: ```js Number(activeAgent.budget?.maxCostUsd || Infinity) ``` An explicit `0` therefore becomes unlimited. `budgetState()` also skips every zero-valued limit: ```js const limit = Number(policy?.[policyKey] || 0); if (!limit) continue; ``` ### The hard limit is checked after the provider charge A full model request is made before its calculated cost is checked. One call can exceed the remaining budget by the entire request amount. ## Required fix ### Fail closed for paid/unknown remote models Represent unknown pricing explicitly rather than as zero: ```js function pricingForModel(modelId) { const pricing = settings.agentModelPricing?.[modelId]; if (!pricing) return { known: false }; return { known: true, inputPerMillionUsd: Number(pricing.inputPerMillionUsd), outputPerMillionUsd: Number(pricing.outputPerMillionUsd), cachedInputPerMillionUsd: Number( pricing.cachedInputPerMillionUsd ?? pricing.inputPerMillionUsd ) }; } ``` Before a paid remote run: ```js if (!pricing.known && Number(agent.budget.maxCostUsd) > 0) { throw new Error( `Pricing is required before enforcing a cost budget for ${agent.modelId}` ); } ``` Local/free models should be marked explicitly as free; absence of data must not mean free. ### Preserve zero as a real hard stop ```js function numericLimit(value, fallback = Infinity) { if (value === undefined || value === null || value === "") return fallback; const limit = Number(value); if (!Number.isFinite(limit) || limit < 0) throw new TypeError("Invalid budget limit"); return limit; } ``` ```js const maxCost = numericLimit(activeAgent.budget?.maxCostUsd); if (maxCost === 0) throw new Error("Agent cost budget is disabled"); ``` `budgetState()` must distinguish absent limits from zero: ```js if (policy?.[policyKey] == null) continue; const limit = Number(policy[policyKey]); if (limit === 0) { return { state: "hard-stop", reason: `${label} limit reached`, ratio: 1, usage: next }; } ``` ### Bound the next request before sending it Derive the maximum output tokens from all remaining limits: ```js const remainingOutputTokens = Math.max( 0, run.budget.maxOutputTokens - run.usage.outputTokens ); const remainingCostUsd = Math.max( 0, run.budget.maxCostUsd - run.usage.costUsd ); const costBoundedOutputTokens = pricing.known && pricing.outputPerMillionUsd > 0 ? Math.floor((remainingCostUsd / pricing.outputPerMillionUsd) * 1_000_000) : remainingOutputTokens; const maxTokens = Math.min( 8_192, remainingOutputTokens, costBoundedOutputTokens ); if (maxTokens < 1) return budgetExhausted(); ``` Because input token cost is not known exactly before sending, reserve a conservative estimate or clearly label the cost cap as approximate and require a configurable safety margin. ### Prevent concurrent global-budget races Daily/monthly budget checking and run creation need a reservation/lease in one serialized application-state operation. Two runs must not both observe the same remaining global budget. ## Acceptance criteria - Missing remote-model pricing blocks cost-budgeted runs instead of recording `$0`. - Explicit zero run/daily/monthly limits prevent execution. - Local/free pricing is explicit. - The next request is bounded by remaining output-token and estimated cost capacity. - Concurrent runs cannot independently spend the same daily/monthly allowance. - UI states whether a cost limit is exact, estimated, or unenforceable. - Tests cover missing pricing, zero limits, a one-call overshoot, and concurrent launches.