Add Local Recall visual-context intent and client adapter #56

New issue

Open

opened 2026-07-19 06:38:55 +00:00 by lost-rob0t · 0 comments

lost-rob0t commented

2026-07-19 06:38:55 +00:00

(Migrated from github.com)

Goal

Allow commands such as “Zara, explain what you see” while keeping Local Recall authoritative for all captured desktop data and visual analysis.

Companion Local Recall issue: https://github.com/lost-rob0t/local-recall/issues/60

This is a client integration, not a repository/codebase merger. Zara must not capture the screen, open Local Recall storage, decrypt records, or send screenshot/OCR data through its normal LLM conversation path.

Current Zara integration points

modules/intent_resolver.pl performs deterministic intent resolution.
modules/command_loop.pl dispatches resolved commands and currently uses LLM rewriting as a fallback.
modules/commands.pl owns command execution.
modules/llm_client.pl stores textual conversation history and supports Ollama/OpenAI/Anthropic providers.

Visual context must use a dedicated Local Recall client path rather than llm_query_with_history/2.

Required Zara changes

Add a deterministic explain_visual_context intent for phrases including:
- “explain what you see”
- “what do you see”
- “what am I looking at”
- “summarize my screen”
- bounded recent-context forms such as “what was I looking at five minutes ago”.
Parse only typed selector arguments: current, recent, or a bounded time window.
Add a narrow Local Recall client module using its versioned owner-only IPC contract.
Configure the endpoint through Zara configuration; do not shell out or accept an arbitrary endpoint from speech.
Enforce connection, response-size, and total request deadlines.
Convert Local Recall outcomes into stable Zara command results and spoken/display replies.
Give explicit responses for unavailable daemon, privacy/lock denial, no context, authorization required, timeout, cancellation, incompatible protocol, and provider failure.
Never copy screenshot bytes, OCR, titles, command lines, usernames, secrets, storage paths, or provider prompts into Zara memory, debug output, hooks, or LLM history.
Do not call Zara’s generic LLM provider for this command; Local Recall owns provider selection and visual-context prompting.
Do not persist the returned explanation automatically as a learned fact or memory.
Remove or sanitize current debug output on this path so user text and returned explanation are not dumped implicitly.

Suggested module boundary

:- module(local_recall_client, [
    explain_visual_context/2,
    explain_visual_context_result/2
]).

% explain_visual_context(+TypedSelector, -Explanation)
% explain_visual_context_result(+TypedSelector, -Result)

Result should use a closed Zara-side result shape mapped from the Local Recall protocol. It must not expose arbitrary response dictionaries to the command layer.

Tests

Deterministic intent-resolution tests for supported phrases and time selectors.
Negative tests proving unrelated see, screen, and look phrases do not trigger visual access.
Client fixture tests against the Local Recall protocol examples from issue #60.
Timeout, unavailable endpoint, malformed response, oversized response, protocol mismatch, and sanitized rejection tests.
Tests proving the request and explanation do not enter current_conversation/1, learned memory, debug logs, or command error terms.
Tests proving no generic Zara LLM call occurs for explain_visual_context.

Acceptance criteria

“Zara, explain what you see” resolves deterministically and returns Local Recall’s textual explanation through Zara’s normal reply/voice surface.
Zara receives no image artifacts and has no direct access to Local Recall storage.
All failure cases produce bounded, content-free diagnostics.
The integration remains disabled or reports unavailable until Local Recall issue #60 provides a compatible protocol.

Dependency and ordering

Blocked by lost-rob0t/local-recall#60. Document only for now; do not begin this Zara issue until the Local Recall roadmap reaches the companion integration work.

## Goal Allow commands such as **“Zara, explain what you see”** while keeping Local Recall authoritative for all captured desktop data and visual analysis. Companion Local Recall issue: https://github.com/lost-rob0t/local-recall/issues/60 This is a client integration, not a repository/codebase merger. Zara must not capture the screen, open Local Recall storage, decrypt records, or send screenshot/OCR data through its normal LLM conversation path. ## Current Zara integration points - `modules/intent_resolver.pl` performs deterministic intent resolution. - `modules/command_loop.pl` dispatches resolved commands and currently uses LLM rewriting as a fallback. - `modules/commands.pl` owns command execution. - `modules/llm_client.pl` stores textual conversation history and supports Ollama/OpenAI/Anthropic providers. Visual context must use a dedicated Local Recall client path rather than `llm_query_with_history/2`. ## Required Zara changes - Add a deterministic `explain_visual_context` intent for phrases including: - “explain what you see” - “what do you see” - “what am I looking at” - “summarize my screen” - bounded recent-context forms such as “what was I looking at five minutes ago”. - Parse only typed selector arguments: `current`, `recent`, or a bounded time window. - Add a narrow Local Recall client module using its versioned owner-only IPC contract. - Configure the endpoint through Zara configuration; do not shell out or accept an arbitrary endpoint from speech. - Enforce connection, response-size, and total request deadlines. - Convert Local Recall outcomes into stable Zara command results and spoken/display replies. - Give explicit responses for unavailable daemon, privacy/lock denial, no context, authorization required, timeout, cancellation, incompatible protocol, and provider failure. - Never copy screenshot bytes, OCR, titles, command lines, usernames, secrets, storage paths, or provider prompts into Zara memory, debug output, hooks, or LLM history. - Do not call Zara’s generic LLM provider for this command; Local Recall owns provider selection and visual-context prompting. - Do not persist the returned explanation automatically as a learned fact or memory. - Remove or sanitize current debug output on this path so user text and returned explanation are not dumped implicitly. ## Suggested module boundary ```prolog :- module(local_recall_client, [ explain_visual_context/2, explain_visual_context_result/2 ]). % explain_visual_context(+TypedSelector, -Explanation) % explain_visual_context_result(+TypedSelector, -Result) ``` `Result` should use a closed Zara-side result shape mapped from the Local Recall protocol. It must not expose arbitrary response dictionaries to the command layer. ## Tests - Deterministic intent-resolution tests for supported phrases and time selectors. - Negative tests proving unrelated `see`, `screen`, and `look` phrases do not trigger visual access. - Client fixture tests against the Local Recall protocol examples from issue #60. - Timeout, unavailable endpoint, malformed response, oversized response, protocol mismatch, and sanitized rejection tests. - Tests proving the request and explanation do not enter `current_conversation/1`, learned memory, debug logs, or command error terms. - Tests proving no generic Zara LLM call occurs for `explain_visual_context`. ## Acceptance criteria - “Zara, explain what you see” resolves deterministically and returns Local Recall’s textual explanation through Zara’s normal reply/voice surface. - Zara receives no image artifacts and has no direct access to Local Recall storage. - All failure cases produce bounded, content-free diagnostics. - The integration remains disabled or reports unavailable until Local Recall issue #60 provides a compatible protocol. ## Dependency and ordering Blocked by lost-rob0t/local-recall#60. Document only for now; do not begin this Zara issue until the Local Recall roadmap reaches the companion integration work.