071 - Swarm: Per-Application Proxy-Facing Edge Overlay¶
User Story¶
As an operator running Infinito.Nexus in swarm mode, I want every application that trusts the reverse proxy’s X-Forwarded-For to reach the proxy over its own private edge overlay, so that a tenant on another application’s overlay cannot forge a client address that the application, its brute-force throttle, its audit log or its IP allowlist then believes.
This is step 2 of the proxy-trust fix. Step 1 (a separate fix PR) moves the roles that trust all private ranges or * onto proxy_hops. This requirement makes proxy_hops tight under swarm too, so that step 1 is worth something there, and closes the in-stack hops that step 1 had to leave on the old trust.
Problem¶
Under swarm, proxy_hops() in utils/networks/proxy.py returns the full subnet of every overlay the application shares with svc-prx-openresty. The proxy joins exactly one overlay, openresty 192.168.107.0/24 (roles/svc-prx-openresty/meta/networks.yml). Its consumer kind is web_facing, and utils/networks/attachments.py matches that kind against every role id starting with web-app- or web-svc-, so about 95 roles share that one overlay.
An application that trusts 192.168.107.0/24 as its proxy therefore trusts every other tenant on that overlay. The proxy appends with proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for (roles/sys-svc-proxy/templates/headers/forwarded.conf.j2), so a request that a tenant sends straight to the application with a forged X-Forwarded-For: 203.0.113.9 reaches the application as 203.0.113.9, 192.168.107.x, and the application, seeing the last hop inside its trusted subnet, takes the forged value. Impact: brute-force throttles, audit logs and IP allowlists key on an attacker-chosen address.
Some trusted hops also run inside the application’s own stack and have no fixed subnet today:
The oauth2
sso-proxysidecar of a gated application (roles/web-app-keycloak/templates/sso_proxy/container.yml.j2). Under compose it has nonetworks:key and lands on<entity>_default. Under swarm it joins every network the application joins, DB and tenant overlays included.zammad-nginxin front of the zammad rails server. Both share several networks, so Docker’s embedded DNS may answer with the rails service’s address on a tenant or DB network instead of ondefault.For an application that provides its own
shared_net(for exampleweb-app-zammad,web-app-snipe-it),utils/networks/render.pyrendersdefaultwithout a subnet, so no trust setting can name it. Step 1 therefore leavesweb-app-zammadand the gatedweb-app-snipe-ittrusting all private ranges, marked# nocheck: forwarded-trust.
Rejected Alternatives¶
Pin the proxy’s IP and trust one address:
docker stack deploydropsipv4_address(docker/clicli/compose/convert/service.gocopies onlyTarget,AliasesandDriverOptsinto the network attachment), and an IPAMip-rangeapplies to every container on the overlay, not to one service.VIP or endpoint-mode tricks: the proxy’s requests carry its task IP, not a virtual IP, so no VIP identifies the proxy.
Decisions¶
Each trusting application gets one edge link: an overlay joined only by
svc-prx-openrestyand the application-side member, with its own subnet from the network SPOT (meta/networks.ymlof the application, the same source as every othernetworks.local.subnet). The application-side member is the service the proxy’s upstream targets: the application’s frontend service, or, for an application gated by the oauth2 sidecar (sso is_proxy_gated), itssso-proxysidecar.resolve_upstreamtargets an alias that is unique to the link and owned by the application-side member. The renderedproxy_passtarget therefore resolves to that member’s address on the link, and the proxy’s request leaves from the proxy’s own task address on the link. This holds for named sidecar routes too: the<entity>_sso-proxyroute of a gated application resolves through the link, never through the shared overlay.proxy_hopsunder swarm returns the link subnet, plus the stack’sdefaultsubnet when decision 9 applies, and nothing else. Under composeproxy_hopsreturns the same gateway list as before, plus thedefaultsubnet when decision 9 applies.The shared
openrestyoverlay stays for the proxy’s own non-trusting consumers until they migrate. An application that has an edge link MUST leave the shared overlay: neither its frontend nor its sidecar keeps an endpoint there, so a tenant can reach it only through the link, which it cannot join.The proxy stack declares every link as an external-or-owned network of its own. Adding a link changes the proxy service’s network set, which Swarm applies as a stop-first restart of the proxy. The design MUST batch link additions into the deploy that already restarts the proxy, and MUST NOT restart the proxy once per application.
Links MUST exist before either stack deploys. The proxy role creates them in a pre-step ahead of the application stacks, the same ordering as the existing shared overlays.
Each link costs one subnet from the subnet SPOT. The allocator MUST reject an exhausted range at lint time, not at deploy time.
The effect on swarm CI time is measured, not assumed: the requirement records the wall-clock of the swarm deploy job before and after, and the budget is no more than the cost of one proxy restart per deploy.
In-stack hops (the
sso-proxysidecar in front of the frontend, or an in-stack web server such aszammad-nginxin front of its rails server) run over the stack’sdefaultnetwork only. The upstream is pinned todefaultby the<service>.<project>_defaultDNS name, or bydefaultbeing the only network both ends share. An application that provides its ownshared_netgets a second SPOT allocation fordefault(for examplelocal.default_subnetinmeta/networks.yml). Every subnet consumer learns it:utils/networks/render.py,utils/meta/scan.py,utils/update_check/network.py, the network uniqueness test and the address suggest CLI.proxy_hopstrusts exactly that subnet for the in-stack hop.
Topology¶
flowchart LR
subgraph before["Before: one shared overlay"]
direction LR
P1[openresty] --- O1((openresty overlay 192.168.107.0/24))
O1 --- A1[app A]
O1 --- B1[app B]
O1 --- C1[tenant of app C]
C1 -. forged X-Forwarded-For .-> A1
end
subgraph after["After: one edge link per app"]
direction LR
P2[openresty] --- LA((link A))
P2 --- LB((link B))
LA --- A2[app A]
LB --- S2[sso-proxy of gated app B]
S2 -. default .- B2[app B]
C2[tenant of app C] -. no route to link A .-> LA
end
Open Questions¶
svc-dns-unboundsetsipv4_address: {{ UNBOUND_RESOLVER_IP }}intemplates/compose.yml.j2. By the samedocker stack deploybehaviour that rules out the proxy pin, that address may be dropped under swarm, so the swarmdns:ofweb-app-mailu(192.168.208.53, fromroles/svc-dns-unbound/meta/networks.yml) may not hold. This is unverified; a live swarm probe has to confirm or refute it, and it is out of scope here beyond that probe.Whether non-trusting
web_facingroles stay on the shared overlay or all move to links in one wave.Whether the swarm
sso-proxysidecar still needs the shared redis overlay (redis_connection_urlpoints at the shared redis whensvc-db-redisis deployed) or the tor overlay once it is narrowed to the link anddefault.
Acceptance Criteria¶
[ ] Under swarm,
proxy_hopsof a trusting application without an in-stack hop returns exactly its edge link subnet.[ ] Under swarm,
proxy_hopsof a trusting application with an in-stack hop returns exactly its edge link subnet and itsdefaultsubnet.[ ] Under swarm,
proxy_hopsnever returns the sharedopenrestyoverlay subnet for an application that has an edge link.[ ] Under compose,
proxy_hopsof an application without an in-stack hop returns the same gateway list as before this change.[ ] Under compose,
proxy_hopsof an application with an in-stack hop returns the previous gateway list plus itsdefaultsubnet, and nothing else.[ ] Each edge link’s members are exactly
svc-prx-openrestyand the application-side member of decision 1.[ ] Each edge link’s subnet comes from the network SPOT.
[ ] No edge link subnet overlaps any other subnet.
[ ] Under swarm,
resolve_upstreamrenders an alias that is unique to the link.[ ] The rendered
proxy_passtarget resolves to the application-side member’s address on the link.[ ] The rendered
proxy_passtarget resolves to no address on the shared overlay.[ ] For a gated application, the
<entity>_sso-proxyroute resolves to the sidecar’s address on the link.[ ] The frontend of an application with an edge link has no endpoint on the shared overlay.
[ ] The
sso-proxysidecar of a gated application with an edge link has no endpoint on the shared overlay.[ ] The proxy stack declares every link.
[ ] Every link exists before the proxy stack deploys.
[ ] Every link exists before its application stack deploys.
[ ] One deploy performs at most one stop-first restart of the proxy, however many links it adds.
[ ] A live swarm probe container on a foreign overlay fails to connect to the application-side member’s address on the link.
[ ] The same probe container fails to connect to every other address of the frontend, including its former shared-overlay address and any published port reachable from the probe’s network.
[ ] The same probe container fails to connect to every address of a gated application’s
sso-proxysidecar.[ ] A request through the proxy with a forged
X-Forwarded-Forshows the real client address in the application’s view.[ ] Every application that provides its own
shared_nethas adefaultsubnet from the SPOT that overlaps no other subnet.[ ]
web-app-zammadlogs the real client address for a request throughzammad-nginx, under compose and under swarm.[ ] The gated
web-app-snipe-itlogs the real client address for a request through itssso-proxysidecar, under compose and under swarm.[ ] Neither
web-app-zammadnorweb-app-snipe-ittrusts a whole private range, and neither carries# nocheck: forwarded-trust.[ ] The subnet allocator fails at lint time when the link and
defaultsubnets exhaust the SPOT range.[ ] The swarm CI deploy wall-clock before and after is recorded in the PR.
[ ] The swarm CI deploy wall-clock after the change stays within the budget of decision 8.
[ ] The open question on
svc-dns-unboundis answered by a live swarm probe, and the result is recorded here as a follow-up requirement or a closed question.
Tests¶
[ ] Unit test in
tests/unit/python/utils/networks/test_proxy.py: under swarmproxy_hopsreturns only the link subnet when the application also shares theopenrestyoverlay, and raisesValueErrorwhen the application has no link.[ ] Unit test in the same file:
proxy_hopsadds thedefaultsubnet for a gated application and forweb-app-zammad, in both modes, and adds nothing for an application without an in-stack hop.[ ] Unit test for the subnet allocation of links and
defaultsubnets: unique, inside the SPOT range, deterministic.[ ] Render test of
sso_proxy/container.yml.j2under swarm: the sidecar joins exactly the link anddefault.[ ] Swarm end-to-end probe (act swarm run): the forged-header case and every foreign-overlay connection case of the acceptance criteria, run against at least one trusting application and one gated application.