Leyline

Vision agent integration ​

Leyline bundles .pi/extensions/vision-agent/index.ts. The extension registers the vision_agent tool so a model can inspect image files through a separate image-capable model.

When a parent model cannot receive an attached image, Leyline saves the image locally and instructs the model to call vision_agent. The tool call and its result appear in the transcript.

Tool contract ​

The tool accepts these fields:

json
{
  "path": "screenshots/current.png",
  "question": "What error does this screen show?",
  "model": "local/qwen3.6-27b",
  "cwd": "/workspace/project"
}
  • path is a required absolute path or a path relative to cwd.
  • question is optional. The child answers it before it gives surrounding detail.
  • model is an optional model ID, provider/model-id, or inherit value.
  • cwd is optional and defaults to the parent session working directory.

The tool accepts PNG, JPEG, GIF, and WebP files. It returns the child model's final text and includes child-session, usage, model, and error data in the tool result.

Model selection ​

The tool selects a model in this order:

  1. The model value in the tool call.
  2. A session vision override.
  3. A project vision override.
  4. The global vision model.

The value inherit selects the parent model. It succeeds only when that model supports image input.

Agent defaults, Project settings, and Session details edit global, project, and session vision values on the selected backend. They store overrides in its vision_overrides table. The default database path is ~/.local/share/leyline/memory.sqlite. The model selector lists only models that report image input support.

Thinking selection ​

The settings store model and thinking values independently at each scope. Each value uses session, project, then global precedence. A scope can contain a thinking value without a model value.

The Thinking mode control appears when the effective vision model supports reasoning or a saved thinking override exists. It provides supported levels and Match parent session. Global settings offer Default (no override); project and session settings offer inheritance from broader scopes. Match parent session uses the parent's current thinking level when the vision child starts.

A session fork copies both values from its session override.

Composer delegation ​

When a user attaches images to a parent model without image input, Leyline does these actions:

  1. Resolve the configured vision model and confirm that it supports image input.
  2. Save each image in the session attachment directory.
  3. Submit the original prompt and images to pi for transcript persistence.
  4. Replace image blocks in parent-model context with saved file paths and instructions to call vision_agent.

The attachment directory is $LEYLINE_MEMORY_DIR/attachments/<session-id>/. Without LEYLINE_MEMORY_DIR, it is ~/.local/share/leyline/attachments/<session-id>/.

The parent model calls vision_agent as part of its turn. The tool starts a hidden vision child. Its result appears in the transcript. The composer does not wait for a hidden vision-child preflight.

Leyline removes the session attachment directory after it moves a session or project to trash.

Leyline does not delegate images for extension slash commands because a handled extension command does not start a parent-model turn. When the active model lacks image support, the composer blocks those attachments. Do not attach images to an extension slash command.

Persisted context replacement ​

installVisionDelegationContext() wraps the parent session's agent.transformContext function. A pending record covers the first model request before pi has finished writing the custom entry.

After pi saves the user message, Leyline appends a leyline-vision-delegation custom entry. The entry stores the image signature, saved file paths, initial and settled context text, and user-message timestamp.

At each model boundary, the wrapper reloads these records from the active branch. It removes matched image blocks from parent-model context. Until vision_agent calls exist for all saved paths, it adds the stored tool instruction. It then adds neutral text so later turns and reloads do not request another inspection. The persisted user message keeps its original text and images through reloads and branch changes.

Child runtime ​

The server creates a vision child through the subagent runtime path when vision_agent runs. Its isolated custom system prompt retains output-budget and inline Elune capture. It omits context files, skills, MCP, tool search, and the Leyline prompt.

The child has these constraints:

  • Its model must support image input.
  • Its active tool allowlist is empty.
  • Its thinking level uses the effective vision override when one is set.
  • Its session contains the leyline-subagent-session marker and stays hidden from the sidebar.
  • Its JSONL history keeps the vision prompt and image.
  • Its trace capture retains bounded text and usage, with image payloads and thinking content omitted.
  • A session-local settings override permits image input without changing the user's persisted pi image setting.

The configured model provider receives the image and prompt. Provider authentication must be available to the Leyline server process. Leyline does not delete the hidden child session after execution.

Bundled requests carry parentSessionId and _toolCallId so capture can link the child to its parent tool observation. Delayed committed child usage can reconcile parent totals. Capture continues while trace views are closed. See Elune integration for content and accounting limits.

Cancellation ​

The vision_agent call uses the parent run's abort signal. The direct vision execution route aborts its child when the HTTP connection closes before completion.

A child failure becomes an error tool result. The parent model receives that result in its context.

Internal routes ​

The integration uses these routes:

  1. GET /api/pi/vision/config reads visible overrides for the browser.
  2. PUT|DELETE /api/pi/vision/override changes one scope. PUT accepts optional model and thinking values. DELETE removes both values.
  3. POST /api/pi/vision/resolve resolves stored configuration for the extension.
  4. POST /api/pi/vision runs one vision child.

See the API reference for request and response contracts.