036 - Native AI Gateway Integration¶
User Story¶
As an operator of a self-hosted deployment, I want every role whose application ships a native AI feature to reach the platform’s own LiteLLM gateway instead of a third-party provider, so that prompts, documents and telemetry never leave the deployment and one credential rotation covers every application.
Context¶
svc-ai-litellm already serves an OpenAI-compatible gateway, and web-app-nextcloud already binds its integration_openai app to that gateway through tasks/addons/integration_openai.yml, a credentials.litellm_api_key entry and a dedicated Playwright spec. That pattern is implemented once and is not required, described or verified for any other role.
A sweep over all 85 application roles for provider references (openai, ollama, llm, anthropic, aiprovider) in meta/, vars/ and templates/ returns only dashboard, flowise, hermes, litellm, matrix, minio, nextcloud, openclaw and openwebui. Everything below the AI-native roles is therefore unwired today.
Confirmed against upstream documentation, each accepting a custom OpenAI-compatible base URL:
Role |
Native surface |
Endpoint |
|---|---|---|
|
Assistant via |
service URL plus API key |
|
Agents plugin |
“OpenAI Compatible” service with API URL; a non-empty key is mandatory even when the upstream ignores it |
|
AI subsystem provider |
core |
|
Discourse AI |
custom LLM; the host must be listed in |
|
Smart Assist |
settings |
|
ChatGPT bridge |
bridge configuration; see the credential defect below |
|
plugin-provided, no core surface |
a connector plugin such as AI Engine exposes a custom OpenAI-compatible endpoint |
|
AI Agent nodes |
a stored |
|
|
provider |
|
AI LLM Application extension |
the role installs the extension and points its OpenAI-compatible provider at the gateway in |
web-app-matrix is the one role that already ships an AI credential and points it at the vendor: meta/secrets.yml declares chatgpt_bridge_openai_api_key with the validation ^sk-[a-zA-Z0-9]{40,}$. That pattern enforces OpenAI’s own key format and therefore rejects a LiteLLM virtual key by construction, so the schema has to change before the bridge can reach the gateway at all.
web-app-wordpress has no AI surface in core; the requirement applies to whichever connector plugin the role installs, and the role MUST pin one rather than leaving the choice to a site administrator. The pinned connector is ai-engine, whose custom engine type takes an OpenAI-compatible base URL and an API key per environment.
web-app-zammad was pinned at 6.5.0, which ships no AI surface at all. The role is bumped to 7.1.2, the newest line whose contract is still the ai_provider / ai_provider_config setting pair; past 7.1.x the provider moves into an AI::ProviderConnection record and the contract changes. 7.1.2 accepts Elasticsearch >= 7.8, < 10, so the role’s existing 8.13.4 pin stands.
The three candidates that vendor documentation claimed were probed against running instances on 2026-08-27, and each is filed below on what the deployed artefact actually offers rather than on the claim:
Role |
Measured at |
Finding |
Disposition |
|---|---|---|---|
|
2.3.3 |
|
excluded |
|
2026.7 |
|
excluded |
|
|
no AI or LLM extension is present in |
confirmed, the role installs the extension in |
web-app-baserow is the one row measured against the pinned source rather than a running instance. Confirm it on the deployed stack with POST /api/database/fields/<field_id>/generate-ai-field-values/ as the seeded superuser: an unlicensed instance answers HTTP 402.
Excluded, with the reason recorded so the question is not reopened:
Role |
Reason |
|---|---|
|
the endpoint is configurable but both prompt-sending views are premium-licence gated; see the measurement above |
|
the OpenAI agent takes no base URL and the |
|
Duo is bound to the vendor service and exposes no self-hosted endpoint |
|
version 19 hardcodes a vendor model for AI server actions; a configurable endpoint needs third-party modules |
|
no AI surface in the core; every option is a separately installed app |
|
Atlassian Intelligence is vendor-hosted |
|
the shipped assistants are vendor services; only third-party modules expose an endpoint |
|
no AI surface found in the product |
|
measured at tag 10.0.4: zero tree paths match |
|
measured at tag v8.2.4 across three independent surfaces: |
infrastructure and static roles |
|
fediverse and media roles |
|
remaining application roles |
|
Acceptance Criteria¶
[ ] Each unverified candidate is probed against a running instance and moved into the confirmed table or the exclusion table with its reason, so the scope rests on measurement rather than vendor claims.
[x] Every role in the confirmed table declares its native AI surface in
meta/services.ymlwithenabledandsharedbound to'svc-ai-litellm' in group_names, so a deployment without the gateway configures no AI surface at all.[x] Every role in the confirmed table carries a
credentials.litellm_api_keyentry inmeta/secrets.yml, and the deploy writes that virtual key into the application’s own AI configuration rather than a shared or hardcoded key.[x] Every role in the confirmed table points its AI base URL at the in-cluster gateway address, and a deploy-time assertion fails when the configured URL resolves outside the deployment.
[x] No role in the confirmed table retains a default that sends requests to a third-party provider when the gateway is enabled.
[x] A Playwright spec per role proves the configured surface answers a prompt through the gateway, mirroring
roles/web-app-nextcloud/files/playwright/addons/integration_openai.spec.js.[x] The deploy fails when a role declares the AI surface enabled while its configuration still carries an empty or unresolved API key, so a silently unauthenticated surface cannot reach a green deploy.
[x] No credential validation in
meta/secrets.ymlencodes a vendor key format, so a gateway-issued virtual key satisfies every AI credential the platform mints.[x]
docs/requirements/027-integration-matrix.mdlists the AI surface of every role in the confirmed table.
Consumers that must name a model (web-app-moodle, web-app-matrix) read LITELLM_CHAT_MODEL, which picks the first non-embedding preload model of svc-ai-ollama and mirrors the branch order of the gateway’s own config.yaml.j2. The consumer resolves that branch from lookup('deployment').inventory, every group the round’s inventory populates, read from the play-wide groups magic var. .running is wrong here: a targeted apps= round sets it to the whitelist, and svc-ai-ollama is pulled in as a dependency rather than named on the command line, so both backend branches fall through to the openrouter/auto vendor default. The gateway then answers Invalid model name passed in model=openrouter/auto with HTTP 400 and no consumer can complete a prompt. .groups is wrong for the mirror-image reason: it carries group_names, which is host-scoped, so on a multi-host round a worker that hosts neither the gateway nor a backend resolves an empty model while the manager resolves the right one.
Which variant exercises the gateway¶
svc-ai-litellm pulls svc-ai-ollama and svc-ai-lmstudio with it, so a consumer that enables the gateway adds roughly 6 GB to its round. Variant 0 is the fallback every higher round lands on, so a heavy shared dependency pinned there appears in every round: with the gateway on variant 0 of web-app-matrix, web-app-mattermost and web-app-zammad, the web-app-nextcloud rounds that pin those partners exceed the 64 GB budget guarded by tests/integration/roles/meta/variants/test_resource_budget.py.
The gateway is therefore pinned on variant 2 for every consumer that ships three variants, and left on variant 0 for the two-variant roles, whose round 2 falls back to it. One make compose-deploy variant=2 run exercises eight of the nine confirmed roles. web-app-nextcloud is the exception: its AI surface has always lived on its own variant 7, alongside the other bridged partners it distributes across budget bins.