Configuration and models

Use this page after installing Embedder to change the model, device, network address, or queue settings. Run native commands from the repository root. Bash examples work on Linux, macOS, and WSL.

Configuration files and environment variables

Settings are loaded in this order, with later values taking precedence:

Built-in embedder/config.yaml defaults.
local.yaml at the repository root, next to pyproject.toml.
Environment variables matching a setting name, such as MODEL_ID, DEVICE, or PORT.

Create local.yaml with only the values you want to override:

device: cpu
host: 127.0.0.1
port: 8091
model_id: deepvk/USER-bge-m3
model_dir: models/user-bge-m3
request_timeout_seconds: 60
dashboard_enabled: true

Restart the service after changing configuration. For a one-command override:

PORT=18091 REQUEST_TIMEOUT_SECONDS=60 make run

Environment variables are not prefixed with EMBEDDER_ for native execution. Unknown YAML settings fail validation. A relative model_dir is resolved from the working directory, so use an absolute path when launching outside the checkout.

Available settings

Setting / environment variable	Default	Meaning
`model_id` / `MODEL_ID`	`deepvk/USER-bge-m3`	Hugging Face repository ID for downloading; clients must send this exact ID in API requests.
`model_dir` / `MODEL_DIR`	`models/user-bge-m3`	Directory containing the downloaded SentenceTransformer model.
`device` / `DEVICE`	`cpu`	Inference device: `cpu`, `cuda`, or `mps`.
`host` / `HOST`	`127.0.0.1`	HTTP bind address.
`port` / `PORT`	`8091`	HTTP port, from 1 to 65535.
`realtime_batch_max_inputs` / `REALTIME_BATCH_MAX_INPUTS`	`32`	Input-count threshold when combining realtime requests into an inference batch. Positive integer.
`realtime_batch_max_chars` / `REALTIME_BATCH_MAX_CHARS`	`24000`	Character-count threshold for combining realtime requests. Positive integer; characters are not tokens.
`batch_max_inputs` / `BATCH_MAX_INPUTS`	`8`	Input-count threshold for combining background requests. Positive integer.
`batch_max_chars` / `BATCH_MAX_CHARS`	`12000`	Character-count threshold for combining background requests. Positive integer.
`request_timeout_seconds` / `REQUEST_TIMEOUT_SECONDS`	`30`	Positive timeout for waiting for the scheduled embedding result, including queue time.
`dashboard_enabled` / `DASHBOARD_ENABLED`	`false`	Enable the optional queue dashboard.
`pytorch_cuda_index` / `PYTORCH_CUDA_INDEX`	`cu126`	PyTorch wheel index used by native CUDA setup. Changing it requires running setup again.

Batch thresholds control how queued requests are combined; they are not hard request-size limits. The scheduler can run an oversized first request on its own. Split large document collections into smaller API requests to manage memory and latency.

Changing device also requires the matching PyTorch installation. For example, on a supported Linux NVIDIA host:

DEVICE=cuda make setup
DEVICE=cuda make run

An environment override applies only to that command. Set device: cuda in local.yaml to persist it. See the platform installation pages for CUDA and CPU on Linux, MPS on macOS, and Windows through WSL.

Command-line tools

Command	Parameters	Purpose
`make setup`	`PYTHON_VERSION=3.12` when creating the environment; settings through environment variables	Prepare `venv`, install device dependencies, and download the configured model if missing.
`make run`	Settings through environment variables	Check the model exists and start the service.
`venv/bin/dzen-embedder-serve`	No runtime CLI flags	Start directly using YAML and environment settings. It does not download a model.
`venv/bin/dzen-embedder-fetch-model`	Required `--model-id` and `--model-dir`; `--help`	Download a model into a chosen directory.

The server does not implement flags such as --port, --device, or --config. Use the corresponding environment variable or local.yaml instead:

PORT=18091 venv/bin/dzen-embedder-serve
venv/bin/dzen-embedder-fetch-model --help

Changing PYTHON_VERSION does not recreate an existing environment.

Install a different model: native deployment

Choose a repository that can be loaded by SentenceTransformer and fits your hardware. Replace YOUR_ORG/YOUR_MODEL below with its repository ID.

Stop the running service and update local.yaml, preserving your other settings:

model_id: YOUR_ORG/YOUR_MODEL
model_dir: models/my-new-model
device: cpu

Use a new directory for the new model, then prepare and start the service:

make setup
make run

make setup skips downloading when model_dir/modules.json already exists. Changing only model_id while reusing an existing directory can therefore leave the old model files in use.

To download explicitly after dependencies have been installed:

venv/bin/dzen-embedder-fetch-model \
  --model-id YOUR_ORG/YOUR_MODEL \
  --model-dir models/my-new-model

Downloading does not update configuration. Set both model_id and model_dir before starting the service. The fetch command checks that modules.json exists; successful download alone does not establish runtime compatibility or sufficient GPU memory.

Check readiness and send a request using the new ID:

curl --fail http://127.0.0.1:8091/ready
curl --fail-with-body http://127.0.0.1:8091/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"YOUR_ORG/YOUR_MODEL","input":"Test the new model","encoding_format":"float"}'

Update your application's model ID and rebuild stored embeddings with the new model. Use that same model for queries; vectors from different models are not interchangeable. Keep the previous model directory if you need to switch back.

Docker Compose configuration

Compose uses its own .env substitutions. It does not mount the checkout's local.yaml, and arbitrary entries in .env are not automatically passed to the container.

`.env` variable	Default	Purpose
`MODEL_ID`	`deepvk/USER-bge-m3`	Downloaded model and API model ID; stored under `/models/<MODEL_ID>` in the persistent volume.
`EMBEDDER_PORT`	`8091`	Host port forwarded to the container's port 8091, bound to localhost.
`EMBEDDER_TAG`	`0.1.0-cpu`, or `0.1.0-cu126` with the CUDA override	Docker image tag; choose a tag matching the device.
`GPU_DEVICE_ID`	`0`	NVIDIA GPU selected by `compose.cuda.yaml`.

To install another model, change MODEL_ID in .env:

MODEL_ID=YOUR_ORG/YOUR_MODEL

Then run the appropriate command:

# CPU
 docker compose up -d

# NVIDIA CUDA
 docker compose -f compose.yaml -f compose.cuda.yaml up -d

Compose runs model-init to download into the model-specific directory before starting Embedder. Check docker compose logs model-init embedder and readiness. For CUDA, use the same -f files with logs. Update client model IDs and rebuild stored vectors as described above.

For other runtime settings, create compose.settings.yaml:

services:
  embedder:
    environment:
      REQUEST_TIMEOUT_SECONDS: "60"
      DASHBOARD_ENABLED: "true"
      REALTIME_BATCH_MAX_INPUTS: "16"
      BATCH_MAX_INPUTS: "4"

Apply it on CPU:

docker compose -f compose.yaml -f compose.settings.yaml up -d

For CUDA, insert -f compose.cuda.yaml before -f compose.settings.yaml. Keep the same file set for subsequent Compose commands. Keep the container port at 8091 unless you also update its port mapping and health check.

See network access before exposing the service beyond localhost. Configuration sources: settings and validation, native setup, and Compose.