openclaw-debugging

OpenClaw Debugging

Behavior differs between local tests, live models, providers, code mode, Tool Search, Crabbox, CI: get debug signal before guessing.

Read First

• docs/logging.md: log files, openclaw logs, targeted debug flags.
• docs/reference/test.md: local test commands.
• docs/tools/code-mode.md: code-mode exec/wait + tool catalog rules.
• $openclaw-testing: test-lane choice.
• $crabbox: broad, Docker, package, Linux, live-key, CI-parity proof.

Default Loop

1. State suspected boundary: config, tool construction, provider payload, fetch, stream/SSE, transcript replay, worker/runtime, package/dist, CI.
2. Add/enable narrowest signal proving boundary.
3. Reproduce with same provider/model/config. Never randomly switch models unless model itself = tested variable.
4. Compare configured state with actual run activation.
5. Patch root cause.
6. Rerun exact failing probe; broaden only if contract requires.

Model Transport Logs

Model request shape/stream timing: targeted env flags over global debug:

OPENCLAW_DEBUG_MODEL_TRANSPORT=1 openclaw gateway
OPENCLAW_DEBUG_MODEL_PAYLOAD=tools OPENCLAW_DEBUG_SSE=events openclaw gateway
OPENCLAW_DEBUG_MODEL_PAYLOAD=full-redacted OPENCLAW_DEBUG_SSE=peek openclaw gateway

Useful flags:

• OPENCLAW_DEBUG_MODEL_TRANSPORT=1: request start, fetch response, SDK headers, first SSE event, stream done, transport errors at info.
• OPENCLAW_DEBUG_MODEL_PAYLOAD=summary: bounded payload summary.
• OPENCLAW_DEBUG_MODEL_PAYLOAD=tools: all model-facing tool names.
• OPENCLAW_DEBUG_MODEL_PAYLOAD=full-redacted: capped, redacted JSON payload. Only while debugging; prompts/message text may still appear.
• OPENCLAW_DEBUG_SSE=events: first-event + stream-completion timing.
• OPENCLAW_DEBUG_SSE=peek: first five redacted SSE events.
• OPENCLAW_DEBUG_CODE_MODE=1: code-mode tool-surface diagnostics.

Watch logs:

openclaw logs --follow

Common Boundaries

• Config vs activation: enabled config can coexist with disabled run tools, raw run, empty allowlist, missing model tool support. Check actual visible tools before enforcing provider payload invariants.
• Tool surface: inspect final model-visible tool names, not only registry/ config. Code mode = exactly exec + wait only after actual activation.
• Provider payload: before SDK call, log fields, model id, service tier, reasoning, input size, metadata keys, prompt-cache key presence, tool names.
• Fetch vs SSE: fetch response proves HTTP headers arrived; first SSE event proves provider body progress. Gap here = stream/body/provider issue, not tool execution.
• Worker/dist: touching workers, dynamic imports, package exports, lazy runtime boundaries, published paths → pnpm build.
• Live keys: missing provider keys → configured secret workflow before declaring live proof blocked. Env checks presence-only; never print secrets.

Fetching Sessions and Transcripts

Chat-session bug report: fetch actual transcript, sender attribution, attachments as evidence.

CLI first (needs configured install; safe against live gateway):

openclaw sessions list --agent <agentId> --json
openclaw sessions tail
openclaw sessions export-trajectory

Docs: store layout docs/reference/database-schemas.md; CLI https://docs.openclaw.ai/cli/sessions.

Raw store: CLI unavailable (e.g. remote host over SSH) or event-level detail needed:

• Per-agent data plane: ~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite. Canonical schema: src/state/openclaw-agent-schema.sql.
• Hosted/systemd state: service user's home (e.g. /home/openclaw/.openclaw/...), not root's. Root may have separate stray install with different agents. Expected agent dir missing: locate real DB with find / -maxdepth 6 -name openclaw-agent.sqlite.
• Web chat URL session-id fragment: /chat/<agentId>/<slug>-<hex>. Resolve in session_nodes: session_key LIKE '%<hex>%' → current_session_id, display_name. Key shape: agent:<agentId>:<surface>:<uuid>; subagent sessions use surface subagent.
• Transcript: transcript_events (session_id, seq, event_json). event_json.message: role (user/assistant/toolResult) + content (string or parts of type text/toolCall/image).
• Sender provenance: real user messages carry message.__openclaw (senderId, senderName, senderIsOwner); runtime-synthesized inputs do not. Distinguish operator-authored text from injected prompts with this.
• Full-text transcript search: session_transcript_fts.
• Attachments: media://inbound/<file> URLs → <state-dir>/media/inbound/<file>.
• User-attached images ≠ image content parts. Message envelope carries message.__openclaw.media[] entries: url (media://inbound/<file>), contentType, kind, fileName. Parts-only extractor misses every image; grep raw event_json for media://. Remote host: scp files locally (one remote path per scp argument); read there.

Hosts without sqlite3 binary still have Node: node:sqlite needs no dependencies.

node -e 'const {DatabaseSync}=require("node:sqlite");
const db=new DatabaseSync(process.argv[1],{readOnly:true});
console.log(JSON.stringify(db.prepare(
  "SELECT seq,event_json FROM transcript_events WHERE session_id=? ORDER BY seq"
).all(process.argv[2])))' <db-path> <session-id>

• Always open live stores readOnly: true; never write running gateway's state (root AGENTS.md Validation rules).
• Realistic-data work: copy DB into dev state dir first.

Code Pointers

• Model payload + Responses stream: src/agents/openai-transport-stream.ts
• Guarded fetch/timing: src/agents/provider-transport-fetch.ts
• OpenAI/Codex provider wrappers: src/llm/providers/stream-wrappers/openai.ts
• Tool construction, Tool Search, code-mode activation: src/agents/embedded-agent-runner/run/attempt.ts
• Code-mode runtime and worker: src/agents/code-mode.ts src/agents/code-mode.worker.ts
• Tool Search catalog: src/agents/tool-search.ts

Proof Choice

• Single helper/payload bug: local targeted Vitest.
• Docs/logging-only: pnpm check:docs + git diff --check.
• Worker/dist/lazy import/package surface: targeted tests + pnpm build.
• Live provider/model behavior: same provider/model, debug flags, real key if available.
• Docker/package/Linux/CI-parity: current dedicated Linux worker when capable; otherwise $crabbox.
• CI failure: exact SHA, relevant job only; logs only after failure/completion.

Output Habit

Report:

• boundary tested
• exact command/env shape, redacted
• observed signal, e.g. tool names or first SSE event timing
• fix location
• narrow proof + remaining risk