070 - sys-ctl-hlth-runtime: Vulnerability Engines Instead of Version Floors

User Story

As an operator, I want every node to scan its host and its running container images with Trivy, Grype and OSV-Scanner over a Syft SBOM, so that a real, fixable vulnerability is reported, and the daily mirror scan fails on it, instead of a deploy failing on a distribution build whose upstream version number looks old.

Issue: #1324.

Acceptance Criteria

• [ ] HEALTH_RUNTIME_FLOORS, floor_violation, make security-runtimes, cli/contributing/security/runtimes and utils/docker/image/runtime.py are gone, and the php_hardening check of expose_php and session.upload_progress.enabled still runs for every service that mounts php_hardening.

• [ ] roles/sys-ctl-hlth-runtime/meta/services.yml pins Trivy, Grype, OSV-Scanner and Syft as services with image and a semver version, so the GHCR mirror copies them.

• [ ] The same file holds the scan configuration: targets [host, container], engines [trivy, grype, osv, syft], the severity mapping {CRITICAL: fail, HIGH: warn}, fixed_only: true, workers, db_update, cache_dir and database_dir, with no default fallback in the role. The script rejects a severity list, an empty mapping, an unknown severity and any action other than fail or warn.

• [ ] The container target is the image of every container running on the node, read through the read-only Docker socket; the host target is the host root filesystem mounted read-only, without /proc, /sys, /dev, /run, /tmp, the Docker root, the backups, the NFS export and the vulnerability cache.

• [ ] In swarm every node scans its own containers and its own host.

• [ ] Syft builds one SBOM per target; Grype and OSV-Scanner read that SBOM and Trivy reads the target itself; without syft in engines, Grype and OSV-Scanner read the target directly.

• [ ] OSV-Scanner reads a CycloneDX copy whose deb and apk purls name the source package, so libssl3 is matched as openssl and php8.1-cli as php8.1.

• [ ] A configuration with osv and the host target but without syft is rejected, because OSV-Scanner reads no OS packages of a plain directory.

• [ ] A runtime built from source, such as PHP 8.4.26 in an official php image, fails the scan with the CRITICAL CVE-2026-109618 and warns on the HIGH CVE-2026-109619, both fixed in 8.4.27: only Grype sees it, through the CPE of Syft’s binary classifier, so with syft in engines and grype not, Grype still scans the binary-classified packages of every SBOM, and a configuration with neither grype nor syft is rejected.

• [ ] Targets times engines run in parallel, bounded by workers; Trivy runs one target at a time and OSV-Scanner reads every SBOM in one run.

• [ ] Each engine’s JSON becomes findings of engine, target, image or host, package, installed version, fixed version, vulnerability ids and severity; OSV maps the group’s CVSS score to a severity and takes the fixed version from the affected ranges of the same package.

• [ ] An unsuppressed finding whose severity maps to fail is a ❌ line naming engine, target and CVE and fails sys-ctl-hlth-runtime.service, the deploy through sys-service-terminator and the OnFailure alert; one that maps to warn is a ⚠️ line and does not fail the service; a suppressed one is an ℹ️ line. The mapping is the same in production and CI, and no deploy variant overrides it. An engine error, a missing database or a PHP hardening violation fails the service too. The daily scan job of cron-images-mirror-all.yml fails on every unsuppressed mapped finding, fail and warn alike.

• [ ] Every finding at the threshold becomes one record per owner with run_id, timestamp (RFC 3339, UTC), host, environment, target, kind, image (<image name>@<image id> for a container, the host name for the host), scanner, id, severity, package, installed_version, fixed_version, suppressed, suppressed_until and verdict (fail, warn or suppressed), and the four outputs below are built from these records.

• [ ] report.json in DIR_HEALTH_RUNTIME_REPORT (/var/lib/infinito/hlth-runtime) holds the summary and the records of the last run and is replaced atomically.

• [ ] findings.jsonl in DIR_HEALTH_RUNTIME_LOG (/var/log/infinito/hlth-runtime) gets one JSON line per record on every run, appended; the role installs logrotate and a drop-in that rotates it weekly, keeps 12 compressed generations and skips a missing or empty file.

• [ ] The journal gets one prefix-free JSON line per target, {"event": "hlth_runtime.target", "run_id", "target", "kind", "image", "counts": {"fail", "warn", "suppressed"}, "verdict": "fail|warn|ok"}, and last {"event": "hlth_runtime.summary", "run_id", "counts", "errors", "verdict"}, next to the human-readable lines.

• [ ] While web-app-prometheus is deployed, the role passes --metrics-file and the check replaces infinito_hlth_runtime.prom in DIR_METRICS_TEXTFILE atomically with infinito_hlth_runtime_findings{target,kind,image,severity,scanner,verdict} (a count, no CVE label), infinito_hlth_runtime_last_run_timestamp_seconds, infinito_hlth_runtime_last_run_success (1 when the run did not fail) and infinito_hlth_runtime_scanner_errors{scanner}, each with # HELP and # TYPE and escaped label values; without the flag no textfile is written.

• [ ] A database refresh or a provided database that is missing fails the service loudly, and OSV-Scanner reporting no database for an ecosystem is an error.

• [ ] The timer runs daily at 00:00 UTC and the deploy-time run stays.

• [ ] A suppression is services.<service>.security.suppressions of the owning role, each entry either id: CVE-YYYY-NNNN or a quoted image_tag, plus until: YYYY-MM-DD, with a # nocheck: cve <reason> marker directly above it.

• [ ] An id entry suppresses that CVE; an image_tag entry suppresses every finding of the service’s container while the scanned image runs that tag, and a container running any other tag is judged unsuppressed. The Docker image updater removes a service’s image_tag entries, with their markers, when it moves that service to another version.

• [ ] A role’s suppressions apply only to the containers of its own stack or compose project and service; host suppressions are the id entries under the engine entries of sys-ctl-hlth-runtime and apply to that engine’s host findings, while an image_tag entry there covers the engine’s own image.

• [ ] docs/contributing/actions/testing/suppression.md lists the cve rule.

• [ ] The lint under tests/lint/ rejects a marker without a reason, a malformed id, an entry with both or neither of id and image_tag, an image_tag that differs from the service’s version, a missing or past until and an entry without a marker.

• [ ] Each CI deploy job draws one engine (trivy, grype or osv) by rotation; its job name carries 🔺, 🍇 or 🐙, the axis glyph is 🔬, and utils/symbol_glossary.py registers all four.

• [ ] A CI deploy pins engines to [syft, <engine>], reads provided databases and never downloads one; the vulnerability-db job of call-test-deploy.yml downloads the Trivy, Grype and OSV databases once per day into the Actions cache, every deploy job restores the Grype database plus the drawn engine’s, and a missing one fails the job.

• [ ] A local development deploy (make compose-deploy, swarm through act or make swarm-zombie) draws one engine at random, runs it behind Syft plus the Grype runtime pass, refreshes the databases it reads once in the shared host directory INFINITO_VULNERABILITY_DB_HOST_PATH, and every stack and swarm node reads that directory read-only at INFINITO_VULNERABILITY_DB_DIR; no stack and no node downloads a database.

• [ ] Production hosts run every engine of the list and refresh their databases daily in their own cache.

• [ ] The deploy-time run fails on its first failure: sys-service-terminator does not retry a unit that sets system_service_run_once, and only the database refresh inside the scan is retried, three attempts 30 s apart.

• [ ] The threshold is strict from the first run: a CRITICAL with a fix fails the check, a HIGH with a fix warns, and both fail the daily mirror scan.

• [ ] make test-security builds a Syft SBOM of every pinned image of all roles straight from the GHCR mirror refs, cached by image digest, and scans that SBOM tree with Trivy, Grype and OSV at the same threshold and suppressions; it is not part of make test, and the alias i8seru runs it.

• [ ] The scan job of cron-images-mirror-all.yml runs after sync, runs make test-security, keeps exactly one open vulnerability issue per image that lists its CVEs (unsuppressed first, then CRITICAL first, suppressed ones with their suppressed_until day so the issue stays open while a suppression runs, capped below GitHub’s body limit with a pointer to the artifact) and closes any further open one as a duplicate, comments the still unresolved CVEs into it and updates its body whenever the image’s CVE table changed, leaves it alone while the table is unchanged, closes it once no engine reports any CVE for it, holds issues: write only on that job and pins its actions like the existing jobs.

• [ ] Every run of that job writes the failing findings per image and per CVE into the job summary and the vulnerability-findings artifact, and the workflow_dispatch input issues: false makes it a dry run without issue changes; the schedule keeps the issues on.

• [ ] The measured database sizes are: Grype v6 3.2 GB unpacked, Trivy 1.6 GB without the Java database, OSV offline zips Ubuntu 806 MB, npm 220 MB, Debian 80 MB, PyPI 38 MB, and 1.24 GB for the ecosystem list in meta/services.yml.

• [ ] Unit tests cover the parsers of all three engines with trimmed real output, the severity mapping and its validation, the fix-available switch, suppression scoping and expiry, the bounded parallel run with a fake runner, the CI engine pin, the exit code per verdict and each of the four outputs.

• [ ] A negative control shows the removed floor flagging 8.1.2-1ubuntu2.28 while Trivy, Grype and OSV fixtures of that package pass, and a fixture with a fixable CRITICAL fails.