Central Engine Services (swarm + compose)

Root-cause architecture for running stateful “engine” dependencies (cache, queue, search, vector, DNS resolver) the same way central databases already work, so that application roles stay stateless, unpinned and NFS-shareable in swarm while the engines are the only placement: manager (node-local) services.

Problem this solves

NFS placement is derived from placement (see plugins/lookup/compose_volumes.py): an unpinned role’s volumes are bound to the shared NFS mount (survive a reschedule), a pinned role keeps its volumes node-local. Engine on-disk state (redis/rabbitmq/elasticsearch/sqlite/postfix-queue) cannot live on NFS (fsync + locking corruption), so a role that embeds such an engine must be pinned. Pinning whole app stacks to the manager:

  • concentrates load on one node, and

  • for mailu (13 services, 4 overlay networks, a static-IP unbound resolver) triggers a swarm VIP-allocation deadlock (could not find an available IP while allocating VIP + has pending allocations, even on empty /24 networks).

The fix is to externalise the engines into central, pinned svc-db-* / svc-dns-* roles (exactly like svc-db-postgres / svc-db-mariadb) and have apps connect to them. Apps then own only user-data volumes (NFS-safe) and need no pin.

Central engine roles

One central role per engine, mirroring svc-db-postgres:

role

engine

per-consumer isolation

svc-db-postgres (exists)

PostgreSQL

a database

svc-db-mariadb (exists)

MariaDB

a database

svc-db-redis (convert)

Redis

ACL user + key-prefix {entity}:*

svc-db-memcached (convert)

Memcached

key-prefix only (no native ACL)

svc-db-rabbitmq (new)

RabbitMQ

a vhost + user

svc-db-elasticsearch (new)

Elasticsearch

index/alias namespace

svc-db-typesense (new)

Typesense

API-key scoped collections

svc-db-qdrant (new)

Qdrant

per-consumer collection prefix

svc-dns-unbound (new)

Unbound

shared resolver (no per-consumer split)

Each central role provides:

  • meta/services.yml: placement: manager, enabled, shared: true, image/version/ports/name.

  • templates/compose.yml.j2: its own standalone stack.

  • tasks/00_core.yml: deploy the central stack + wait healthy.

  • tasks/01_init.yml: per-consumer provisioning (create the ACL user / vhost / namespace and its credentials), idempotent, mirroring postgres 01_init.

  • a connection lookup mirroring database: lookup('<engine>', consumer_id, want) returning host/port/user/password/url/db for the consumer.

shared toggle (compose + swarm, identical)

Mirrors the existing postgres shared flag:

  • services.<engine>.enabled: true + shared: true -> central: the consumer connects to the central engine over the shared cross-stack network using the <engine> lookup and its own provisioned credentials.

  • services.<engine>.enabled: true + shared: false -> embedded sidecar (current behaviour), for single-host / opt-out.

base.yml.j2 embeds an engine sidecar only {% if enabled and not shared %}.

Cross-stack connectivity

Central engines are shared-net providers (the existing mechanism in utils/networks/render.py / compose_volumes.py); consumers attach to the engine’s shared overlay and the <engine> lookup resolves the cross-stack address (resolve-container-id in swarm, service name in compose), exactly as lookup('database', ...) does today.

mailu root-cause fix

svc-dns-unbound is a single-service, single-network, static-IP, pinned role. mailu’s services drop their own unbound + static IP, attach to the svc-dns-unbound network, and point dns: at its static IP. The static-IP allocation therefore lives in a trivial 1-service stack (no deadlock) instead of mailu’s 13-service / 4-network stack. mailu also stops embedding redis (uses central) -> fewer services -> less VIP pressure.

Placement

  • Central svc-db-* / svc-dns-*: placement: manager (single node).

  • Application roles: no placement -> distributed + NFS-shared.

  • tests/lint/ansible/roles/meta/test_unpinned_no_engine_volume.py stays as the guard: an unpinned role must not declare an engine on-disk data volume.

Lifecycle + ordering

  • Central engine is enabled on demand: deployed only when at least one inventory role consumes it shared: true (same discovery the central DB uses).

  • Deploy order: central engine -> consumer 02_init provisioning -> consumer (run_after, like postgres).

Scope

Test-fresh: the goal is green compose and swarm deploys on fresh act clusters. No embedded->central data migration path (that is a separate prod-upgrade concern).