Plugin API: propagate canonical cancellation to running service tools #352

Open
opened 2026-09-08 02:03:37 +00:00 by nsaspy · 12 comments
Owner

Upstream handoff from lost-rob0t/zara-plugins / zara-shell #5

zara-shell now has deterministic bounded timeout and process-group teardown, but its remaining acceptance criterion requires explicit cancellation distinct from timeout.

Current Zara service-plugin tools are ordinary BaseTool instances returned by ServicePlugin.tools(). I could not find a canonical Core cancellation token/signal passed into an in-flight plugin tool invocation. Adding a plugin-specific execution manager + shell.cancel side channel would create a second control plane and make cancellation semantics inconsistent across long-running plugins.

Needed Core contract

  • a running service-plugin tool can observe canonical Zara turn/tool cancellation;
  • cancellation is scoped to the exact principal + invocation, not a reusable user-supplied process ID;
  • model/tool arguments cannot suppress or retarget the cancellation signal;
  • cancellation composes with ToolApprovalController and normal tool lifecycle/unload;
  • Core records a distinct cancelled outcome rather than conflating cancellation with timeout/failure;
  • deterministic compatibility tests can inject cancellation without network/GUI dependencies.

Concrete consumer

zara-shell can map that signal to process-group termination and return structured cancelled=true evidence while preserving its existing runtime/output/cwd/env bounds. Other long-running service tools could reuse the same contract.

No Zara runtime code is being changed from the plugin worker.

## Upstream handoff from `lost-rob0t/zara-plugins` / `zara-shell` #5 `zara-shell` now has deterministic bounded timeout and process-group teardown, but its remaining acceptance criterion requires **explicit cancellation distinct from timeout**. Current Zara service-plugin tools are ordinary `BaseTool` instances returned by `ServicePlugin.tools()`. I could not find a canonical Core cancellation token/signal passed into an in-flight plugin tool invocation. Adding a plugin-specific execution manager + `shell.cancel` side channel would create a second control plane and make cancellation semantics inconsistent across long-running plugins. ### Needed Core contract - a running service-plugin tool can observe canonical Zara turn/tool cancellation; - cancellation is scoped to the exact principal + invocation, not a reusable user-supplied process ID; - model/tool arguments cannot suppress or retarget the cancellation signal; - cancellation composes with `ToolApprovalController` and normal tool lifecycle/unload; - Core records a distinct cancelled outcome rather than conflating cancellation with timeout/failure; - deterministic compatibility tests can inject cancellation without network/GUI dependencies. ### Concrete consumer `zara-shell` can map that signal to process-group termination and return structured `cancelled=true` evidence while preserving its existing runtime/output/cwd/env bounds. Other long-running service tools could reuse the same contract. No Zara runtime code is being changed from the plugin worker.
Author
Owner

Core recovery is now #413 on exact current master ada24b73ab599d417dfbda6c864854c19b3426e3. r4/#401 is closed unmerged because exact-head CI falsified its hidden RunnableConfig propagation. r5 is TDD-first: public read-only token + sync/async/concurrency/approval/unload acceptance tests are running RED while the next implementation step wires Zara's pinned langgraph-prebuilt==1.1.0 ToolNode(awrap_tool_call=...) interceptor. No plugin-specific cancel side channel and zero android/** changes.

Core recovery is now #413 on exact current master `ada24b73ab599d417dfbda6c864854c19b3426e3`. r4/#401 is closed unmerged because exact-head CI falsified its hidden RunnableConfig propagation. r5 is TDD-first: public read-only token + sync/async/concurrency/approval/unload acceptance tests are running RED while the next implementation step wires Zara's pinned `langgraph-prebuilt==1.1.0` `ToolNode(awrap_tool_call=...)` interceptor. No plugin-specific cancel side channel and zero `android/**` changes.
Author
Owner

RAGE r5 exact-head outcome: architecture falsified; do not merge #413.

Exact candidate 4e792febeb7b0ebdf834d14810f0676aa9d50ffb ran CI #1518. Android APK/native/secret inspection and both shared-mic lanes passed, but the Core test job failed deterministically before flake/package.

The uploaded regression-artifacts JUnit report contains 1983 tests / 3 failures, all in t/test_plugin_tool_cancellation.py:

  • sync tool never reaches tool_entered with a visible cancellation token;
  • concurrent first tool never reaches first_started;
  • async tool never reaches entered.

So the ToolNode(..., awrap_tool_call=...) + ContextVar binding does not provide the contract under Zara's exact pinned execution stack. This is design-level evidence, not a flaky runner failure. r5 should be discarded per skills/rage/SKILL.md rather than patched indefinitely.

Next iteration must bind cancellation at Zara's own exact post-approval/post-availability invocation boundary, while preserving the single canonical ToolNode execution path, terminal ToolCancelled authority, exact-invocation isolation, and zero model-visible cancellation arguments/side channels.

RAGE r5 exact-head outcome: **architecture falsified; do not merge #413**. Exact candidate `4e792febeb7b0ebdf834d14810f0676aa9d50ffb` ran CI #1518. Android APK/native/secret inspection and both shared-mic lanes passed, but the Core `test` job failed deterministically before flake/package. The uploaded `regression-artifacts` JUnit report contains 1983 tests / 3 failures, all in `t/test_plugin_tool_cancellation.py`: - sync tool never reaches `tool_entered` with a visible cancellation token; - concurrent first tool never reaches `first_started`; - async tool never reaches `entered`. So the `ToolNode(..., awrap_tool_call=...)` + ContextVar binding does not provide the contract under Zara's exact pinned execution stack. This is design-level evidence, not a flaky runner failure. r5 should be discarded per `skills/rage/SKILL.md` rather than patched indefinitely. Next iteration must bind cancellation at Zara's own exact post-approval/post-availability invocation boundary, while preserving the single canonical ToolNode execution path, terminal ToolCancelled authority, exact-invocation isolation, and zero model-visible cancellation arguments/side channels.
Author
Owner

RAGE r6 exact-head/merge-ref outcome: second ContextVar architecture falsified; do not merge #420.

CI #1540 exercised branch head 18f548a5857d2badda0206a8046895c12ffc7472 as merge ref 45abe6893be10daf725315a41e8d4dbca76055ce against then-current master 27cb2d85a51c735213d807bacfe38a4880994da6.

Compatibility lanes stayed green:

  • Android unit/native Trealla/debug APK/secret inspection: PASS
  • shared mic / Arch: PASS
  • shared mic / Ubuntu: PASS

Core failed deterministically at the acceptance seam: 1979 passed / 3 failed. Sync, async, and concurrent tool bodies all failed to observe the token/enter before the one-second acceptance boundary. The rejection/no-execution invariant passed.

This proves that wrapping ToolNode.ainvoke() in a ContextVar scope is insufficient under Zara's pinned LangGraph/LangChain stack, just as r5's ToolNode interceptor was insufficient. We are retiring ContextVar propagation through ToolNode as the cancellation transport rather than adding more patches.

r7 research is now constrained to a materially different hidden/injected runtime mechanism (or a canonical executor redesign if the pinned stack lacks one), with the same invariants: approval first, exact invocation isolation, signal-before-cancel, no model-visible/control argument, one terminal ToolCancelled event, and no android/** changes.

RAGE r6 exact-head/merge-ref outcome: **second ContextVar architecture falsified; do not merge #420**. CI #1540 exercised branch head `18f548a5857d2badda0206a8046895c12ffc7472` as merge ref `45abe6893be10daf725315a41e8d4dbca76055ce` against then-current `master` `27cb2d85a51c735213d807bacfe38a4880994da6`. Compatibility lanes stayed green: - Android unit/native Trealla/debug APK/secret inspection: PASS - shared mic / Arch: PASS - shared mic / Ubuntu: PASS Core failed deterministically at the acceptance seam: 1979 passed / 3 failed. Sync, async, and concurrent tool bodies all failed to observe the token/enter before the one-second acceptance boundary. The rejection/no-execution invariant passed. This proves that wrapping `ToolNode.ainvoke()` in a ContextVar scope is insufficient under Zara's pinned LangGraph/LangChain stack, just as r5's ToolNode interceptor was insufficient. We are retiring ContextVar propagation through ToolNode as the cancellation transport rather than adding more patches. r7 research is now constrained to a materially different hidden/injected runtime mechanism (or a canonical executor redesign if the pinned stack lacks one), with the same invariants: approval first, exact invocation isolation, signal-before-cancel, no model-visible/control argument, one terminal ToolCancelled event, and no `android/**` changes.
Author
Owner

RAGE r7 is now PR #425 from exact current master 27cb2d85a51c735213d807bacfe38a4880994da6, candidate f94e9b78fd4048c8237854be05b0bf9149135096.

This is materially different from falsified r5/r6 ContextVar designs: it uses LangGraph 1.1.0 InjectedState as the hidden runtime dependency. Plugin tools opt in with public read-only ToolCancellationArg; Core creates a private per-invocation signal only after approval + availability and injects its view into that exact one-call ToolNode state. LangGraph's own reference states that InjectedState is automatically injected and remains invisible to the model tool-calling interface.

Acceptance now covers schema hiding, sync signal-before-side-effect, async signal-before-CancelledError, concurrent isolation, rejection, unload, and spoof attempts. Zero android/** changes. Exact-head CI #1554 is running; both shared-mic lanes are already green. No merge until Core full tests/flake/package + Android gate are green on this exact SHA.

RAGE r7 is now PR #425 from exact current master `27cb2d85a51c735213d807bacfe38a4880994da6`, candidate `f94e9b78fd4048c8237854be05b0bf9149135096`. This is materially different from falsified r5/r6 ContextVar designs: it uses LangGraph 1.1.0 `InjectedState` as the hidden runtime dependency. Plugin tools opt in with public read-only `ToolCancellationArg`; Core creates a private per-invocation signal only after approval + availability and injects its view into that exact one-call ToolNode state. LangGraph's own reference states that `InjectedState` is automatically injected and remains invisible to the model tool-calling interface. Acceptance now covers schema hiding, sync signal-before-side-effect, async signal-before-CancelledError, concurrent isolation, rejection, unload, and spoof attempts. Zero `android/**` changes. Exact-head CI #1554 is running; both shared-mic lanes are already green. No merge until Core full tests/flake/package + Android gate are green on this exact SHA.
Author
Owner

RAGE r9 recovery is now draft PR #446. Immutable Core start was 0a370079f2bf293b320ea72865239e6ee4c08a73; sibling Android then advanced master with #438 only, so no Core file overlap. r8/#431 is closed unmerged after CI #1561 falsified its Any + InjectedState execution. r9 candidate 280cae81e000236ea5293c1e2060fa6467bfa255 keeps the concrete ToolCancellation annotation and uses Pydantic SkipValidation() only at the runtime schema boundary, preserving InjectedState metadata. Exact-head CI #1595 is queued. If the focused/Core lane still cannot inject the signal, this architecture is retired and the next iteration moves to explicit Core-owned manual argument injection at the existing post-approval/post-availability ToolNode boundary. Zero android/** changes.

RAGE r9 recovery is now draft PR #446. Immutable Core start was `0a370079f2bf293b320ea72865239e6ee4c08a73`; sibling Android then advanced master with #438 only, so no Core file overlap. r8/#431 is closed unmerged after CI #1561 falsified its `Any + InjectedState` execution. r9 candidate `280cae81e000236ea5293c1e2060fa6467bfa255` keeps the concrete `ToolCancellation` annotation and uses Pydantic `SkipValidation()` only at the runtime schema boundary, preserving `InjectedState` metadata. Exact-head CI #1595 is queued. If the focused/Core lane still cannot inject the signal, this architecture is retired and the next iteration moves to explicit Core-owned manual argument injection at the existing post-approval/post-availability ToolNode boundary. Zero `android/**` changes.
Author
Owner

RAGE r10 is now draft PR #451 from exact canonical 15499557e5994a05d57cc1c0a3ec97857939efda, head 6f9dda69b7172e937994a101032b2c4c1fe8404b. r9/#446 is closed unmerged after CI #1595 reproduced the same four focused failures, so InjectedState is retired. r10 uses LangChain InjectedToolArg only to hide the argument from the model, then Core explicitly overwrites cancellation with its exact-invocation ToolCancellation after approval/current-binding checks and before the same canonical ToolNode.ainvoke. No ContextVar, no InjectedState, no second executor, zero android/**. Exact-head CI #1607 is queued.

RAGE r10 is now draft PR #451 from exact canonical `15499557e5994a05d57cc1c0a3ec97857939efda`, head `6f9dda69b7172e937994a101032b2c4c1fe8404b`. r9/#446 is closed unmerged after CI #1595 reproduced the same four focused failures, so `InjectedState` is retired. r10 uses LangChain `InjectedToolArg` only to hide the argument from the model, then Core explicitly overwrites `cancellation` with its exact-invocation `ToolCancellation` after approval/current-binding checks and before the same canonical `ToolNode.ainvoke`. No ContextVar, no InjectedState, no second executor, zero `android/**`. Exact-head CI #1607 is queued.
Author
Owner

r14 / PR #458 is retired unmerged. Exact-head CI #1631 was red in Core: 6 failures / 1985 tests. Android compatibility + Arch mic + Ubuntu mic were green; flake/package correctly skipped after Core failure. Focused cancellation failures: sync tool never entered, async tool never entered, concurrent invocation tool never entered, spoof-guard execution failed before plugin body. Two additional regressions showed supports_tool_cancellation() assuming every registered test double has get_input_schema(). Canonical master also advanced to 9bdf9b8d4bcab37da80bbfb31f701e3021b0ef3e via Android #455, so #458 was both red and stale. Do not revive r14 unchanged. Next recovery must prove the transport reaches plugin-body entry before cancellation semantics are evaluated, while retaining one ToolNode/approval boundary and zero android/** changes.

r14 / PR #458 is retired unmerged. Exact-head CI #1631 was red in Core: 6 failures / 1985 tests. Android compatibility + Arch mic + Ubuntu mic were green; flake/package correctly skipped after Core failure. Focused cancellation failures: sync tool never entered, async tool never entered, concurrent invocation tool never entered, spoof-guard execution failed before plugin body. Two additional regressions showed `supports_tool_cancellation()` assuming every registered test double has `get_input_schema()`. Canonical `master` also advanced to `9bdf9b8d4bcab37da80bbfb31f701e3021b0ef3e` via Android #455, so #458 was both red and stale. Do not revive r14 unchanged. Next recovery must prove the transport reaches plugin-body entry before cancellation semantics are evaluated, while retaining one ToolNode/approval boundary and zero `android/**` changes.
Author
Owner

r15 / PR #463 is retired unmerged. Exact-head CI #1647 at 9d6e5fa2d6cddd9095f75a136de5702020703a19 failed the Core Run all tests step; Android unit/native/APK/secret compatibility and both shared-mic lanes were green, while flake/package correctly skipped after Core failed. Canonical master also advanced to 2f4bd28a9428a117f986f06b025d1a35abf5124a via Android #461, so the candidate is both red and stale. Do not recover r15 unchanged. The next #388 iteration must be based on the exact current master and must first isolate the failing Core acceptance from CI #1647 before selecting another cancellation transport. Regression artifact digest: sha256:fad17dc5703029737712c170bb190abc5372cd3d5cf78e2a23c178daaa959224.

r15 / PR #463 is retired unmerged. Exact-head CI #1647 at `9d6e5fa2d6cddd9095f75a136de5702020703a19` failed the Core `Run all tests` step; Android unit/native/APK/secret compatibility and both shared-mic lanes were green, while flake/package correctly skipped after Core failed. Canonical `master` also advanced to `2f4bd28a9428a117f986f06b025d1a35abf5124a` via Android #461, so the candidate is both red and stale. Do not recover r15 unchanged. The next #388 iteration must be based on the exact current master and must first isolate the failing Core acceptance from CI #1647 before selecting another cancellation transport. Regression artifact digest: `sha256:fad17dc5703029737712c170bb190abc5372cd3d5cf78e2a23c178daaa959224`.
Author
Owner

Fresh r15 failure isolation from CI #1647 / artifact sha256:fad17dc5703029737712c170bb190abc5372cd3d5cf78e2a23c178daaa959224: there are exactly 3 Core failures, all before cancellation semantics can meaningfully run. test_running_sync_tool_observes_canonical_turn_cancellation times out waiting for entered; test_async_tool_sees_signal_before_cancelled_error times out waiting for its async entry event; test_cancellation_is_isolated_between_concurrent_invocations times out waiting for the first invocation to enter. So r15 is not primarily a signal-ordering bug: the opted-in wrapper path fails to reach plugin-body entry under the exact pinned execution stack. Android and both shared-mic lanes were green on that run. Next #388 iteration should begin from current master by tracing registry wrapper -> ToolNode invocation -> actual BaseTool _run/_arun dispatch and prove body entry with a no-cancel control before selecting another cancellation transport. Do not recover r15 unchanged.

Fresh r15 failure isolation from CI #1647 / artifact `sha256:fad17dc5703029737712c170bb190abc5372cd3d5cf78e2a23c178daaa959224`: there are exactly **3** Core failures, all before cancellation semantics can meaningfully run. `test_running_sync_tool_observes_canonical_turn_cancellation` times out waiting for `entered`; `test_async_tool_sees_signal_before_cancelled_error` times out waiting for its async entry event; `test_cancellation_is_isolated_between_concurrent_invocations` times out waiting for the first invocation to enter. So r15 is not primarily a signal-ordering bug: the opted-in wrapper path fails to reach plugin-body entry under the exact pinned execution stack. Android and both shared-mic lanes were green on that run. Next #388 iteration should begin from current master by tracing registry wrapper -> ToolNode invocation -> actual BaseTool `_run/_arun` dispatch and prove body entry with a no-cancel control before selecting another cancellation transport. Do not recover r15 unchanged.
Author
Owner

Additional narrowing from the r15 test matrix/source: test_registered_wrapper_reaches_original_body passed; only the three ToolNode-mediated body-entry tests failed. So CancellationTransportTool.ainvoke() works when called directly, but the same registered wrapper does not reach the original body through canonical create_tools_node() / LangGraph ToolNode. Current Core constructs one ToolNode(tools) then dispatches each approved/current-bound call via await tool_node.ainvoke({"messages": [single_call]}, config). The next control test should therefore compare direct wrapper ainvoke(ToolCall) vs ToolNode-mediated invocation with cancellation disabled, before touching signal propagation. That should isolate ToolNode input/dispatch semantics from cancellation itself.

Additional narrowing from the r15 test matrix/source: `test_registered_wrapper_reaches_original_body` **passed**; only the three ToolNode-mediated body-entry tests failed. So `CancellationTransportTool.ainvoke()` works when called directly, but the same registered wrapper does not reach the original body through canonical `create_tools_node()` / LangGraph `ToolNode`. Current Core constructs one `ToolNode(tools)` then dispatches each approved/current-bound call via `await tool_node.ainvoke({"messages": [single_call]}, config)`. The next control test should therefore compare direct wrapper `ainvoke(ToolCall)` vs ToolNode-mediated invocation with cancellation disabled, before touching signal propagation. That should isolate ToolNode input/dispatch semantics from cancellation itself.
Author
Owner

Core r16 characterization is now canonical via #569 at merge bea53760a6c1e91bb0671a509985498d760a49d5 (exact candidate ea00e250489493ff4e717ca97af905d89a461a5d, CI #1861 fully green). The compiled StateGraph(MessagesState) + ToolNode path invokes the exact registered tool object's public ainvoke with the typed tool-call mapping (name, args, id) and reaches the plugin body/result. This rules the next cancellation design toward an explicit registered-tool public invocation wrapper keyed by the call id, not another ContextVar transport. No Android changes.

Core r16 characterization is now canonical via #569 at merge `bea53760a6c1e91bb0671a509985498d760a49d5` (exact candidate `ea00e250489493ff4e717ca97af905d89a461a5d`, CI #1861 fully green). The compiled `StateGraph(MessagesState)` + `ToolNode` path invokes the exact registered tool object's public `ainvoke` with the typed tool-call mapping (`name`, `args`, `id`) and reaches the plugin body/result. This rules the next cancellation design toward an explicit registered-tool public invocation wrapper keyed by the call id, not another ContextVar transport. No Android changes.
Author
Owner

RAGE r17 / PR #619 is retired unmerged after exact acceptance CI #1961 at 465ee3dc329f44121864e6c2e0f974a6ba2f7a22 falsified the task-local helper. Android unit/native Trealla/debug APK/secret inspection PASS; shared mic Arch PASS; shared mic Ubuntu PASS; Core Run all tests FAIL, so flake/package/screenshots correctly skipped. The strengthened acceptance added the missing sync-service-tool requirement: a real sync tool body must observe canonical cancellation before releasing owned work. asyncio.current_task().cancelling() cannot be the full transport across that worker-thread boundary. Do not recover #619 or prior ContextVar/InjectedState variants. r16/#569 remains the positive characterization: canonical ToolNode calls the exact registered tool object's public ainvoke with the whole typed ToolCall (name,args,id) and reaches the plugin body. Next design is therefore constrained to a private per-call signal at that proven public wrapper boundary, covering sync + async + concurrent isolation + approval/unload + spoof resistance + timeout distinction, with one ToolNode and one terminal ToolCancelled authority. Zero android/** changes.

RAGE r17 / PR #619 is retired unmerged after exact acceptance CI #1961 at `465ee3dc329f44121864e6c2e0f974a6ba2f7a22` falsified the task-local helper. Android unit/native Trealla/debug APK/secret inspection PASS; shared mic Arch PASS; shared mic Ubuntu PASS; Core `Run all tests` FAIL, so flake/package/screenshots correctly skipped. The strengthened acceptance added the missing sync-service-tool requirement: a real sync tool body must observe canonical cancellation before releasing owned work. `asyncio.current_task().cancelling()` cannot be the full transport across that worker-thread boundary. Do not recover #619 or prior ContextVar/InjectedState variants. r16/#569 remains the positive characterization: canonical ToolNode calls the exact registered tool object's public `ainvoke` with the whole typed ToolCall (`name`,`args`,`id`) and reaches the plugin body. Next design is therefore constrained to a private per-call signal at that proven public wrapper boundary, covering sync + async + concurrent isolation + approval/unload + spoof resistance + timeout distinction, with one ToolNode and one terminal `ToolCancelled` authority. Zero `android/**` changes.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/zara#352
No description provided.