llm-log

Transparent LLM traffic capture for building a durable training corpus and a symbolic Prolog knowledge base.

The first slice is deliberately small: route an LLM client through llm-log, forward the request unchanged, stream the response back immediately, and append the completed exchange to disk.

Documentation

Canonical documentation is Org-mode under research/:

• research/LLM-LOG-RESEARCH-INDEX.org — document map and migration status
• research/LLM-LOG-RESEARCH-013-cl-proxy-runtime-architecture.org — Common Lisp proxy runtime architecture reference
• research/LLM-LOG-RESEARCH-014-configuration-and-operations.org — configuration and operations guide
• research/LLM-LOG-RESEARCH-015-testing-and-verification.org — testing and verification guide

The Common Lisp runtime lives in proxy/; the expert plane in expert/.

Architecture

LLM client
   |
   v
llm-log proxy
   |---------------------------> configured upstream
   |                                OpenAI / OpenRouter / Anthropic / local
   |
   +--> recorder actor --> data/events.jsonl   # lossless corpus
                       +--> data/events.pl      # compact Prolog projection

events.jsonl is the source of truth for future fine-tuning/export. events.pl is the symbolic index used for request classification and later expert-system rules; it intentionally does not duplicate giant prompt/completion blobs.

Authorization, cookie, and API-key header values are forwarded to the upstream but replaced with <redacted> before persistence.

Run

nix develop
python -m pip install -e .
llm-log serve --log-dir ./data

Default upstream prefixes:

Client base URL	Upstream
http://127.0.0.1:8787/openai/v1	https://api.openai.com/v1
http://127.0.0.1:8787/openrouter/api/v1	https://openrouter.ai/api/v1
http://127.0.0.1:8787/anthropic	https://api.anthropic.com

Keep using the provider's normal API-key mechanism in the client. The proxy does not own or store the key.

Custom/local endpoints are explicit:

llm-log serve \
  --log-dir ./data \
  --upstream ollama=http://127.0.0.1:11434 \
  --upstream vllm=http://127.0.0.1:8000

Then point the client at http://127.0.0.1:8787/ollama/... or http://127.0.0.1:8787/vllm/....

Captured event

Each JSONL row includes event/timing IDs, provider/upstream, method/path/query, redacted headers, complete request bytes, complete response bytes, response status, model when discoverable, latency, SHA-256 hashes, and Prolog classifier labels. Non-UTF-8 bodies are stored as base64.

Provider-reported token counters are normalized as input_tokens, output_tokens, and total_tokens. The extractor recognizes OpenAI/OpenRouter-compatible, Anthropic, Gemini, Cohere, and Ollama JSON fields in regular JSON, SSE, and text WebSocket responses. Missing counters remain null; llm-log does not estimate tokens from body size or text.

Analytics API

The capture service exposes a provider-neutral, read-only analytics API on the same listener:

Endpoint	Result
GET /api/v1/stats/summary	total requests, usage coverage, and aggregate token I/O
GET /api/v1/stats/timeline?granularity=minute	exact minute/hour/day buckets for graphs
GET /api/v1/stats/models	token totals grouped by provider and model
GET /openapi.json	OpenAPI 3.1 contract
GET /docs	Swagger UI

All stats endpoints accept optional RFC 3339 start (inclusive), end (exclusive), provider, and model query parameters. The timeline bucket_seconds and exact UTC bucket edge let consumers such as the Qtile telemetry widget calculate token rates without maintaining a second provider-specific history database.

The recorder is a single-writer asyncio.Queue actor. Concurrent proxy requests can complete in parallel, but only the recorder actor appends corpus/KB records, preventing interleaved file writes.

The initial SWI-Prolog classifier is intentionally coarse (coding, research, search, writing, analysis, fallback chat). It is a seed for an evolving expert system, not training truth.

Test locally

nix develop
python -m unittest discover -s tests -v

No GitHub Actions development loop is required for this slice.

Capture boundary

This captures traffic from software you deliberately point at the proxy: gptel, OpenAI-compatible tools, OpenRouter clients, local model clients, and similar configurable callers. It does not magically capture the ChatGPT/Claude web apps or arbitrary HTTPS applications. Doing that later would require a system proxy / TLS interception design and should be a separate security-sensitive slice.

Next ARADR directions

Later slices can derive fine-tuning datasets, mine repeated failure paths, grow Prolog expert rules, add semantic retrieval, route by symbolic intent, and optionally inject search/tool results before forwarding. Those are intentionally outside ARADR-001.