dsh-memlab

Usage

npm install -g dsh-memlab

cp $(npm root -g)/dsh-memlab/examples/lab.json lab.json   # edit: home, modelPatch, systems
dsh-memlab setup --config lab.json                        # create one profile per system, install plugins
dsh-memlab run   --config lab.json --mode passive --wait 120 --model "" none mine
dsh-memlab rescore --results ~/memlab/home/results/-passive.json --tasks zh   # re-score saved answers

lab.json:

{
  "home": "~/memlab/home",
  "modelPatch": "~/memlab/model.patch.yml",
  "systems": [
    { "id": "none" },
    { "id": "mine", "plugins": ["my-memory-plugin@1.0.0"], "patch": "- id: my-memory\n  config:\n    foo: 1\n",
      "env": { "MY_USER": "memlab-{run}-{mode}" }, "reset": ["my-memory-store"] }
  ]
}

• modelPatch: a Cordis patch file with your model provider and default model, put first in every profile.
• env values may use {run}, {mode}, {system}, {workspace} so remote-backed plugins get a fresh namespace per run.
• reset: paths inside the DSH home removed before each run, for stores that would otherwise carry over (.. and absolute paths are rejected).
• --wait: seconds to wait between the teach phase and probing, for plugins that commit memory asynchronously.