Skip to content

Benefits and trade-offs

Share one model across clients

Your web application and background workers call the same HTTP service instead of loading their own copies of the model. This concentrates model memory and computation in one process and lets you operate Embedder independently.

For NVIDIA deployments, use one service process per GPU. Additional processes load additional model copies; adding processes alone does not guarantee higher throughput.

Give user queries priority

Users can continue asking questions while documents are being indexed. Whenever the realtime queue has work, Embedder selects its next group of tasks from that queue. Batch jobs run when the realtime queue is empty.

An inference operation that has already started is not interrupted. Sustained realtime traffic can delay background processing, so measure capacity with your own request mix. Clients explicitly mark background work with X-Dzen-Embedding-Priority: batch.

Process text on your own hardware

Embedding inference runs in your infrastructure without calling an external embedding API. Dependencies and model files must be downloaded first, so initial setup normally requires internet access. You remain responsible for network access, logging, and protection of the service.

Local inference can be economical for steady, high-volume workloads or when spare hardware is available. There is no universal savings percentage: include server utilization, power, operations, text volume, and acceptable latency in your comparison.

Economics on a GTX 1080 server

Our server inventory, checked on September 10, 2026, is an Intel Core i7-7700 with 64 GB RAM and an NVIDIA GeForce GTX 1080 with 8 GB VRAM. It is an older consumer GPU machine, not a modern datacenter accelerator.

What we measured

The August 31 benchmark used deepvk/USER-bge-m3. At 64 concurrent batch connections it completed 31.039 requests/second × 8 inputs = 248.312 inputs/second, with p95 request latency of 2.081 seconds. This was a short connection-capacity test, not a sustained monthly workload or a benchmark of 500-token documents. Do not multiply that input rate by an assumed document length and present the result as measured token throughput.

Monthly capacity and cost comparison

At 248.312 inputs/second, a fixed 30-day month at 24 hours/day and 100% utilization projects 643,624,704 inputs/month. At an assumed 500 tokens/input, that is 321,812,352,000 tokens/month.

This is an illustrative full-capacity scenario, not measured monthly throughput. The short GTX 1080 test did not establish this speed for 500-token inputs.

Capacity and savings calculator

Compare one server's estimated monthly output with the API cost of processing the same volume. The default API is OpenAI text-embedding-3-large, $0.13 per million input tokens, checked September 10, 2026. The server budget is $100/month. Choose a model or enter your own price per million tokens. The server scenario is fixed: 30 days × 24 hours, 100% utilization, $100/month.

How the calculation works

Monthly inputs = 248.312 × 24 × 30 × 3,600
Monthly tokens = monthly inputs × 500
API cost = monthly tokens / 1,000,000 × API rate
Monthly savings = API cost − $100
Break-even tokens = $100 / API rate × 1,000,000

The $100 budget is a conservative assumption for the whole rented machine, slightly above our stated rental cost. A negative savings result means the API is cheaper for the selected volume. The calculator assumes enough demand to keep the machine fully occupied throughout the month.

Rates in the comparison use standard public pricing from OpenAI small, OpenAI large, Cloudflare, and Voyage, checked September 10, 2026. They exclude free credits, taxes, negotiated pricing, and deferred Batch API discounts. Providers tokenize text differently; equal assumed token counts are a planning comparison, not a promise of equal bills for identical text.

These are different models, not interchangeable quality levels.1 Benchmark real documents and reserve capacity for interactive queries, peaks, and downtime. A short connection test cannot establish sustained throughput for longer inputs.

Choose the model and hardware separately

The model_id and model_dir settings select the model; device selects the backend. You can start on CPU, use MPS on Apple Silicon, or deploy on Linux with an NVIDIA GPU.

A replacement model must load through SentenceTransformer. Evaluate its retrieval quality against your own queries and documents. When changing models, regenerate corpus vectors and use the same model for queries: equal vector dimensions do not make different models' embeddings compatible.

Integrate through a small HTTP API

The API uses the familiar OpenAI embeddings request and response shape. This simplifies integration, but does not imply support for every OpenAI API feature. The current implementation accepts text or a list of strings, requires the configured model ID, and supports encoding_format: float. Its usage fields contain zeros rather than measured token counts.

See the API handler and queue scheduler for the implementation.

Choose an installation method →

  1. This is a cost comparison between different models. OpenAI text-embedding-3-large is accessed through its API; our deployment runs deepvk/USER-bge-m3, not a local copy of the OpenAI model. We have not established that a larger model with comparable quality would fit or run adequately on the GTX 1080's 8 GB VRAM. The calculation does not demonstrate equal retrieval quality or the ability to host OpenAI's model on this GPU. ↩