dsh-plugin-multimodal

Usage

1. Configure external API models (required)

Fill in apiBase, apiKey, and model in the providers list of the in-package cordis.patch.yml (none of these connection values has a working default). The plugin and DSH Web still start while they are empty; validation happens when the corresponding tool is called. The protocol is OpenAI-compatible, so any vendor works:

providers:
  # Image recognition: multimodal chat/completions
  - id: my-vision
    kind: vision
    apiBase: https://api.openai.com/v1        # or https://api.siliconflow.cn/v1 etc.
    apiKey: sk-xxxx                            # your API key
    model: gpt-4o                              # or Qwen/Qwen2.5-VL-72B-Instruct etc.
    # maxTokens: 1024      # optional
    # temperature: 0.2     # optional

  # Image generation: ONE provider covers text-to-image AND image-to-image
  # (Volcengine Ark Doubao Seedream uses this protocol too)
  - id: my-image
    kind: image
    apiBase: https://api.openai.com/v1        # or https://ark.cn-beijing.volces.com/api/v3
    apiKey: sk-xxxx
    model: gpt-image-1                         # or doubao-seedream-5-0-pro-260628 etc.
    # i2iModel: doubao-seedream-3-0-i2i-250528 # optional: i2i model used when an
    #                                          #   image arg is passed; omit when the
    #                                          #   model handles both (e.g. Seedream 5.0 Pro)
    # watermark: false      # default false (watermark-free), no need to set
    # size: 1024x1024       # optional default size

  # Image generation: Aliyun DashScope native protocol (use when the
  # compatible-mode gateway exposes no image routes; tested: wanx2.1-t2i-turbo
  # is watermark-free). Aliyun t2i/i2i are separate models → i2iModel.
  - id: my-image-wanx
    kind: image
    protocol: dashscope-native
    apiBase: https://dashscope.aliyuncs.com
    apiKey: sk-xxxx
    model: wanx2.1-t2i-turbo
    i2iModel: wanx-v1                          # i2i model (input image must be a public http(s) URL)
    size: '1024*1024'      # DashScope sizes use an asterisk

id must be unique; the same endpoint may host several models under different ids; vision and image generation may point at different vendors. image_generate does not distinguish t2i/i2i: passing an image argument switches to i2iModel (or the same model when unset); omitting it means text-to-image.

2. Verify the plugin is active

Open a new session and ask: "Do you have image_recognize / image_generate tools?" — or have the model run vision_providers to list the configured models.

3. What the model can do

• Recognize — "识别这张图 C:\path\to\photo.png 里有什么" → image_recognize
• Text-to-image — "画一只戴帽子的橘猫，保存到 output 目录" → image_generate returns the saved file path
• Image-to-image — "把这张图 https://.../a.png 里的猫变成蓝色（用 my-image-wanx）" → image_generate with image + provider
• Inspect — "你现在能用哪些图像模型？" → vision_providers

When nothing is configured

DSH Web and the plugin remain available. Only the invoked tool fails, e.g.:

multimodal: provider "my-vision" 未配置 apiBase。请在插件包内 cordis.patch.yml 的 providers 中填写 OpenAI 兼容端点，然后重启 Web profile。

Configuration

Key	Default	Meaning
providers	[] at schema level; the bundled patch contains two empty editable examples	External API models. Each entry: id (unique), kind (vision / image), apiBase, apiKey, and model (required when calling the provider), protocol (image generation only: openai = images/generations (default); dashscope-native = Aliyun DashScope async task API), optional i2iModel / watermark (default false) / size / maxTokens / temperature
outputDir	empty (= /generated)	Default directory for generated images; relative paths resolve against the caller workspace, absolute paths are used as-is
staticPort	3081	Built-in static image server port (127.0.0.1); generated images are exposed at `http://127.0.0.1:
/for the Web GUI to render via markdown.0` disables it (paths only); port conflicts degrade gracefully

image_recognize args: image (required: path / URL / data URI), prompt, provider, max_tokens, temperature. image_generate args: prompt (required), image (optional: image-to-image source, path / URL / data URI), provider, size, n (1–4), output_path (directory ends with / or \).