Agent Broker¶

Description¶

Agent Broker is an OpenAI-compatible endpoint that gives every entitled user a personal Hermes or OpenClaw agent, started on the first request and addressed by the model name hermes or openclaw.

Overview¶

This role deploys the broker together with a container-API socket proxy in Compose and Swarm deployments. Open WebUI reaches the broker as a second OpenAI connection and forwards the caller’s identity. The broker checks the caller’s Keycloak group, creates or starts that caller’s agent under the isolating runtime, forwards the request to it and streams the answer back. Agents reach their model only through the broker, which relays each call to the shared LiteLLM gateway with its own key and tags it with the owner.

Architecture¶

ARCHITECTURE.md draws Agent Broker from its own metadata: the containers it runs under compose, what changes under swarm, how each volume is backed, and every role it depends on or is depended upon by.

Section	Drawn from
Compose	`meta/services.yml`
Swarm	`meta/services.yml` replicas and placement
Volumes	`meta/volumes.yml` against the NFS decision chain
Dependencies	`meta/services.yml` and `meta/main.yml` of every role

Features¶

Agents on demand: The first completion of a user for hermes or openclaw creates that user’s agent; later requests reuse it, and a stopped agent is started again with its state volume re-attached.
RBAC gate: A user needs the agent-user role of web-app-hermes or web-app-openclaw. Open WebUI lists each agent model only to that group, and the broker refuses any other caller with HTTP 403 and stops a running agent of a user who lost the role. Every completion reads the caller’s group membership from Keycloak again, so a grant and a revocation both take effect on the next request.
Isolation: Every agent runs in its own container or single-replica service under the isolating runtime, with its own volume, its own network whose only other member is the broker, and its own bearer key; an agent that would start under runc is stopped and refused. In compose that network is created per agent and the broker attaches to it; in swarm, where a service task cannot be attached to a network it was not created with, the networks are a pool of agents.max_running overlays that the broker joins through its own service spec and each agent claims one of while it runs.
Restricted engine access: The broker reaches the container engine only through a filtered unix socket in a volume shared with the socket proxy, which publishes no port and runs with network_mode: none in compose. The proxy admits only the container, service, task, network, volume and image calls the broker makes, and refuses every other method and path, including exec and delete. It filters methods and paths, not request bodies: volumes/create stays open for the per-agent volume, so the isolation holds against a compromised agent, which reaches no socket at all, not against a compromised broker.
Bind-mount allowlist: One -allowbindmountfrom carries all four CA-trust files the deployment binds into every container as a comma-separated value, and nothing else; the flag is a plain string the proxy splits on commas, so a repeated flag keeps only its last occurrence. ensure_pool re-POSTs the broker’s own service spec to attach the pool network, so the spec’s own mounts have to pass that check; dropping them answers the update with 403 and the broker crash-loops. An empty allowlist is not an option because it switches the restriction off for every call.
Lifecycle settings: agents.idle_stop, agents.idle_minutes, agents.max_running and agents.start_timeout in meta/services.yml control idle stops, the running-agent bound and the start wait; an inventory overrides them under applications.svc-ai-agent-broker.services.agent-broker.agents. An agent serving a request is never swept.
Model and context: agents.model names the gateway alias every agent runs on and defaults to the gateway’s chat model, so agents can run on a larger model than Open WebUI’s default. The context window of that alias comes from the model’s context in the inventory and is written into the agent config as Hermes model.context_length and OpenClaw contextWindow. An agent prompt runs to tens of thousands of tokens, so a CPU-only local model spends minutes per turn on it; point agents.model at a remote model for interactive use. Hermes refuses a window below 64000 tokens and answers HTTP 500 on every prompt, so a platform whose minimum the selected alias misses is not offered at all: it is left out of /v1/models and answers 404, and the deploy says which platform it dropped and why. The rest of the broker keeps working, which is what makes a small model and OpenClaw a valid pairing. An alias that declares no window is reported instead of filtered, because an agent that does not know its window runs.
Model relay: Agents call http://agent-broker:<port>/llm/v1 with their own key; the broker forwards the call to LiteLLM with the broker’s virtual key, sets the OpenAI user field to the owner and logs a JSON relay event with owner, platform, path and upstream status. Only /v1/chat/completions and /v1/models are relayed; any other path answers 404 and logs a relay_refused event, so an agent cannot reach the gateway’s other routes with the broker’s key.

Quick Setup¶

Development¶

Clone, set up the workstation, and deploy Agent Broker onto the local stack:

git clone https://github.com/infinito-nexus/core.git
cd core
make onboard
make compose-deploy mode=reinstall apps=svc-ai-agent-broker,svc-ai-ollama full_cycle=false

Production¶

Run the published image to provision the inventory and deploy Agent Broker to a managed server over SSH. The image logs in as root, so the server must accept root over a public key, and the two key variables are the two halves of one key pair: the private half authenticates the deploy, and the public half is what the deploy keeps authorized for root and installs for the administrator account. Naming a public half that does not belong to the private one locks the running deploy out at its next reconnect.

APP=svc-ai-agent-broker,svc-ai-ollama
HOST="<your-server>"
DOMAIN="<your-domain>"
TLS_MODE=self_signed
SSH_PUBLIC_KEY="<your-ssh-public-key>"
SSH_PRIVATE_KEY="<your-ssh-private-key-file>"
WORKDIR="$HOME/.infinito.nexus"

mkdir -p "$WORKDIR/inventories"
docker run --rm \
  -v "$WORKDIR/inventories:/etc/infinito.nexus/inventories" \
  -v "$SSH_PRIVATE_KEY:/run/secrets/deploy_key:ro" \
  -e APP="$APP" -e HOST="$HOST" -e DOMAIN="$DOMAIN" -e TLS_MODE="$TLS_MODE" -e SSH_PUBLIC_KEY="$SSH_PUBLIC_KEY" \
  ghcr.io/infinito-nexus/core/debian:latest bash -c '
    INVENTORY=/etc/infinito.nexus/inventories/production
    infinito administration inventory provision "$INVENTORY" \
      --inventory-file "$INVENTORY/devices.yml" \
      --host "$HOST" \
      --include "$APP" \
      --vars "{\"TLS_MODE\": \"$TLS_MODE\", \"DOMAIN_PRIMARY\": \"$DOMAIN\", \"SYS_SVC_SSHD_PERMIT_ROOT_LOGIN\": \"prohibit-password\", \"users\": {\"root\": {\"authorized_keys\": [\"$SSH_PUBLIC_KEY\"]}, \"administrator\": {\"authorized_keys\": [\"$SSH_PUBLIC_KEY\"]}}}" &&
    infinito administration deploy dedicated "$INVENTORY/devices.yml" \
      --password-file "$INVENTORY/.password" \
      -u root --private-key /run/secrets/deploy_key \
      --diff -vv'

The inventory under $WORKDIR is written by the container as root, including the vault password at inventories/production/.password. Run sudo chown -R "$USER" "$WORKDIR" afterwards to read it.

Credits¶

Kevin Veen-Birkenbach

Part of the Infinito.Nexus Project and maintained by Kevin Veen-Birkenbach. Licensed under the Infinito.Nexus Community License (Non-Commercial).

This role deploys no third-party application, so the licence above is the only one that applies.