dsh-local-llm

Configuration

Example configuration:

{
  "model": "mistral-7b-instruct-v0.3-Q4_K_M.gguf",
  "modelPath": "",
  "runtimeUrl": "http://127.0.0.1:8080",
  "contextSize": 8192,
  "autoContextSize": true,
  "port": 8080,
  "downloadDir": "./models"
}

Parameters:

• model - a built-in model name or a direct URL to a .gguf file.
• modelPath - a path to an existing model file. If omitted, the configured model is downloaded automatically.
• runtimeUrl - the URL of an OpenAI-compatible local runtime such as llama-server.
• contextSize - the minimum context size passed to llama-server, in tokens. The default is 8192.
• autoContextSize - automatically choose a model-dependent context size. Enabled by default.
• port - a plugin configuration value; the Harness web server owns its HTTP port.
• downloadDir - the model directory. Defaults to ./models.

Built-in model names include:

• mistral-7b-instruct-v0.3-Q4_K_M.gguf;
• llama-3-8b-instruct-q4_K_M.gguf;
• deepseek-coder-6.7b-instruct-q4_K_M.gguf;
• qwen-2.5-7b-instruct-q4_K_M.gguf.