utils.update package

Submodules

utils.update.addons module

Shared SPOT for the unified addon contract.

Both the role-meta lint (tests/lint/ansible/services/test_addons_schema.py) and the opt-in external drift checker (tests/external/update/addons/) read the constants here so the two layers never drift apart: the lint validates that an addon’s declared mechanism/source/update.catalog is one the checker actually understands, and the checker only walks catalogs it has an adapter for.

This module is intentionally ansible-free and dependency-light so it can be imported from the lint layer, the external test layer, and the update automation without dragging deploy-time machinery in.

class utils.update.addons.AddonEntry(role: str, addon_id: str, config_path: Path, spec: Mapping[str, Any])

Bases: object

A single addon declaration materialised from a role’s meta/addons/<id>.yml.

addon_id: str
property catalog: str | None
config_path: Path
property mechanism: str | None
property monitored: bool
role: str
property source: str | None
spec: Mapping[str, Any]
property update: Mapping[str, Any]
property upstream_id: str
property version: str
class utils.update.addons.CatalogCandidate(upstream_id: str, latest: str, mechanism: str, source: str, catalog_url: str, relevant: bool = True, reason: str = '')

Bases: object

A single entry a catalog adapter surfaced for relevance + version review.

catalog_url: str
latest: str
mechanism: str
reason: str = ''
relevant: bool = True
source: str
upstream_id: str
class utils.update.addons.CatalogResult(version_updates: list[VersionSuggestion] = <factory>, new_addons: list[CatalogSuggestion] = <factory>)

Bases: object

What a catalog adapter returned for one discovery/version pass.

new_addons: list[CatalogSuggestion]
version_updates: list[VersionSuggestion]
class utils.update.addons.CatalogSuggestion(role: str, addon_id: str, upstream_id: str, mechanism: str, source: str, catalog: str, catalog_url: str, reason: str)

Bases: object

A new addon surfaced by a catalog adapter that is not yet declared.

addon_id: str
catalog: str
catalog_url: str
mechanism: str
reason: str
role: str
source: str
upstream_id: str
class utils.update.addons.VersionSuggestion(role: str, addon_id: str, current: str, latest: str, config_path: Path, line: int | None = None, catalog: str = '')

Bases: object

An addon whose pinned version is behind the latest upstream release.

addon_id: str
catalog: str = ''
config_path: Path
current: str
latest: str
line: int | None = None
role: str
utils.update.addons.collect_addon_entries(roles_root: Path) → list[AddonEntry]

Return every declared addon across all roles’ meta/addons/*.yml.

Each file’s root IS the addon spec; the filename stem is the addon id. Malformed (non-mapping) payloads are skipped here; the lint layer is the one that reports them, so the checker stays robust against in-progress files.

utils.update.addons.discover_addon_updates(*, role: str, catalog: str, declared: list[AddonEntry], candidates: list[CatalogCandidate]) → CatalogResult

Reconcile a catalog’s live candidates against the declared addons.

Pure, fixture-friendly core of the external drift checker (no I/O): the live adapter fans the registry queries out and hands the results here.

  • A monitored declared addon whose pinned version is behind the candidate’s latest becomes a VersionSuggestion.

  • A relevant candidate with no matching declared addon becomes a CatalogSuggestion (added disabled by update automation).

  • A candidate the curated relevance rules reject is silently dropped so broad marketplaces never produce unreviewable noise.

utils.update.addons.iter_addon_files(roles_root: Path) → Iterator[tuple[str, Path]]

Yield (role_name, meta/addons/<addon_id>.yml path) for every addon file across all roles. The filename stem is the addon id.

utils.update.addons.secret_key_matches(key: str) → bool

Return True iff key looks like a secret-bearing config key.

utils.update.addons.value_is_templated(value: Any) → bool

Return True iff value is a Jinja expression rather than a bare literal.

A templated value (“{{ … }}”), whether a lookup(…, ‘secrets.credentials.<name>’) call or another variable reference, is accepted; a bare scalar literal (string without {{, or a raw int/bool) is a hard-coded secret and is rejected.

utils.update.addons.version_is_newer(current: str, latest: str) → bool

Return True iff latest is a strictly newer semver than current.

A blank current (the addon tracks the app default) never produces a suggestion, and a non-semver latest is ignored so the checker never proposes a non-comparable tag.

utils.update.addons.worker_fetch_limit(default: int = 8) → int

Thread-pool size for live catalog fetches, from INFINITO_WORKER_FETCH.

utils.update.base module

Semver primitives shared by every version-bump backend.

A “version” here is a tag of the shape (<channel>-)?v?<numeric-semver><patch>?(-<flavor>)?, where the numeric part has 1 to 5 dot-separated components, the optional leading <channel>- is a release line written before the number (e.g. the LiteLLM tag main-v1.77.7-stable or the Jitsi tag stable-9646), the optional <patch> is a vendor counter written as a letter and a number (e.g. the Checkmk tag 2.4.0p32), and the optional -<flavor> suffix is an opaque discriminator (e.g. the Docker Official Image tag 5.4.5-php8.3-apache or the npm-style 1.2.3-alpha).

Upgrade candidates MUST share the same depth (1 to 5 components) AND the same flavor as the current tag, so that 5.4.5-php8.3-apache never silently bumps to 5.4.6-php8.4-apache (different runtime) or to 5.4.6 (different depth). Everything that is not the number folds into the flavor: the leading channel, the trailing suffix and the patch letter. That is what keeps main-v1.77.7-stable away from main-v1.77.7-nightly, stable-9646 away from a bare 9646, and 2.4.0p32 ordering against 2.4.0p33 rather than against the four-component 2.4.0.32. Only the patch number joins the ordering key, so p9 sorts below p10.

utils.update.base.captured_versions(body: str, pattern: str, template: str = '') → list[str]

Return the version every match of a pattern names.

Parameters:

  • body – text to search.

  • pattern – regular expression; its first group is the version.

  • template – when set, a str.format template over the pattern’s named groups that assembles the version instead, for a tag built from several upstream values.

utils.update.base.declares_source(config: object, key: str) → bool

Report whether a services entity names its own upstream for a pin.

Parameters:

  • config – one entity of meta/services.yml.

  • key – the pinned key; an update: block without key names version.

utils.update.base.is_maintained(roles_root: Path, role: str) → bool

Return whether the version-bump backends watch a role.

Parameters:

  • roles_root – the roles/ directory that holds the role.

  • role – role name, the directory name under roles_root.

utils.update.base.is_semver(value: str) → bool
utils.update.base.latest_semver(tags: list[str], depth: int, flavor: str = '') → str | None
utils.update.base.newer_version(current: str, found: list[str], assembled: bool) → str | None

Return the version current should move to, or None.

Parameters:

  • current – the pinned version.

  • found – every version the upstream offers.

  • assembled – found was built from several upstream values, so a changed suffix is a move and not another flavour.

utils.update.base.resolve_max_fetch_workers() → int
utils.update.base.version_depth(tag: str) → int
utils.update.base.version_flavor(tag: str) → str

Return the -<flavor> suffix of a versioned tag, or “” when none.

utils.update.base.version_key(tag: str) → tuple[int, ...]

utils.update.collection_source module

Which requirements source scripts/install/ansible.sh should try first.

Both sources carry the same pinned collections, so the only thing separating them is how long they take and whether they answer at all. A single slow day says nothing: galaxy.ansible.com degrades and recovers, and a git clone can stall on one runner. The default is therefore retuned only when a rolling window of daily samples shows one source durably behind the other.

A sample whose install failed is recorded as None and ranks worse than any duration, so an outage counts fully rather than being dropped.

utils.update.collection_source.RATIO = 3.0

How many times slower the preferred source must be before it is swapped.

class utils.update.collection_source.Verdict(preferred: str, proposed: str, medians: dict[str, float], samples: int)

Bases: object

The outcome of comparing the two sources.

Parameters:

  • preferred – the source in force when the comparison ran.

  • proposed – the source the samples favour, equal to preferred when nothing should change.

  • medians – median seconds per source, infinite when every sample failed.

  • samples – how many samples the medians were taken over.

property changed: bool
medians: dict[str, float]
preferred: str
proposed: str
samples: int
utils.update.collection_source.WINDOW = 12

Samples the median is taken over; below this the default is left alone.

utils.update.collection_source.append_sample(history: list[dict[str, float | None]], sample: dict[str, float | None]) → list[dict[str, float | None]]

The history with SAMPLE appended, truncated to the newest WINDOW entries.

Parameters:

  • history – samples recorded so far, oldest first.

  • sample – seconds per source for this run, None where the install failed.

utils.update.collection_source.decide(history: list[dict[str, float | None]], preferred: str) → Verdict

Whether the samples justify swapping the preferred source.

Parameters:

  • history – samples recorded so far, oldest first.

  • preferred – the source currently tried first.

utils.update.collection_source.load_history(path: Path) → list[dict[str, float | None]]

Samples recorded so far, oldest first, or an empty list.

Parameters:

path – JSON file holding the rolling window.

utils.update.docker module

Detect and rewrite outdated Docker image version: tags in roles/*/meta/services.yml.

Counterpart to utils.update.repository (which handles git refs). Both share the semver primitives in utils.update.base; this module owns the Docker-Hub / GHCR registry lookups and the YAML walker that rewrites the matching version: line.

Suppress a check by placing # nocheck: docker-version on the line directly above the version: key.

A version that names its own upstream in an update: block belongs to utils.update.source and is not collected here.

class utils.update.docker.DockerImageVersionEntry(role: 'str', service: 'str', image: 'str', version: 'str', config_path: 'Path')

Bases: object

config_path: Path
image: str
role: str
service: str
version: str
class utils.update.docker.DockerImageVersionUpdate(entry: 'DockerImageVersionEntry', latest: 'str')

Bases: object

entry: DockerImageVersionEntry
latest: str
utils.update.docker.apply_updates(updates: list[DockerImageVersionUpdate]) → list[Path]
utils.update.docker.collect_entries(repo_root: Path) → list[DockerImageVersionEntry]
utils.update.docker.fetch_dockerhub_tags(image: str, max_pages: int = 5) → list[str]
utils.update.docker.fetch_ghcr_tags(image: str) → list[str]
utils.update.docker.fetch_mcr_tags(image: str, max_pages: int = 10) → list[str]
utils.update.docker.find_outdated_updates(repo_root: Path) → list[DockerImageVersionUpdate]
utils.update.docker.ghcr_repo(image: str) → str
utils.update.docker.is_ghcr(image: str) → bool
utils.update.docker.is_mcr(image: str) → bool
utils.update.docker.is_supported_registry(image: str) → bool

Whether image resolves to a registry the generic v2 client can query.

utils.update.docker.mcr_repo(image: str) → str
utils.update.docker.sourced_services(config_path: Path) → set[str]

Return service names whose version declares its own upstream.

Parameters:

config_path – a role’s meta/services.yml.

utils.update.docker.suppressed_services(config_path: Path) → set[str]

Return service names whose version: line is annotated with the unified # nocheck: docker-version marker.

The file root of meta/services.yml IS the services map; there is no services. wrapper to walk into.

utils.update.docker.update_config_versions(config_path: Path, service_versions: dict[str, str]) → bool

Rewrite each <service>.version: value in meta/services.yml.

The file root IS the services map (no services. wrapper), so the walker tracks one nesting level: top-level service keys and their immediate version: field.

utils.update.fetch module

Read the documents a declared version source names.

Every document is fetched once per run, so the pins that read the same release manifest resolve against one snapshot of it.

utils.update.fetch.document(url: str) → str

Return the text of url, fetched once per run.

utils.update.fetch.documents(urls: str | list[str]) → str

Return the text of one URL, or of several as one text.

Parameters:

urls – a URL, or several whose bodies are searched together.

utils.update.fetch.get(url: str, headers: dict[str, str] | None = None) → bytes

Return the body of a declared version source.

Parameters:

  • url – https URL to read.

  • headers – request headers beside the user agent.

utils.update.follow module

Hold a Docker Hub pin to the release a moving tag points at.

A meta/services.yml entity that sets follow: <tag> only moves to a newer tag of the same shape whose digest equals the one <tag> points at. Ubuntu’s latest names the newest LTS release and Fedora’s the newest stable one, while the higher numbers (Ubuntu’s interim releases, Fedora’s rawhide) point elsewhere.

utils.update.follow.followed_tags(image: str, tags: list[str], config: object, current: str) → list[str]

Narrow tags to the newer releases the entity’s follow tag names.

Parameters:

  • image – the entity’s Docker Hub image.

  • tags – the registry’s tags for that image.

  • config – one entity of meta/services.yml; without follow every tag is kept.

  • current – the pinned version.

Returns:

The tags unchanged without follow; otherwise the newer tags of the pin’s depth and flavor that share the followed tag’s digest, and none when Docker Hub cannot be read, which is reported on stderr.

utils.update.follow.tag_digest(image: str, tag: str) → str

Return the digest Docker Hub reports for image:tag.

Parameters:

  • image – Docker Hub reference.

  • tag – the tag to resolve.

utils.update.galaxy module

Pinned Ansible collection versions in requirements/.

Both requirement files name the same collections at the same versions: the Galaxy file is the normal source, the git file the fallback used when galaxy.ansible.com is unreachable. A bump that touches only one of them would make the fallback install something the pin does not describe, so an update is applied to both or to neither.

Upstream releases are resolved from the git file’s own source: URL via git ls-remote --tags, not from the Galaxy API, because the collections are tagged in the repository the fallback already clones.

An entry whose version: is not a semver (a branch name, say) states an intent this cannot second-guess and is skipped.

class utils.update.galaxy.CollectionPin(fqcn: str, version: str, source: str)

Bases: object

One collection pinned in both requirement files.

Parameters:

  • fqcn – fully qualified collection name, namespace and name.

  • version – the version both files pin today.

  • source – upstream git URL taken from the git requirements file.

fqcn: str
source: str
version: str
class utils.update.galaxy.CollectionPinUpdate(pin: 'CollectionPin', latest: 'str')

Bases: object

latest: str
pin: CollectionPin
utils.update.galaxy.apply_updates(repo_root: Path, updates: list[CollectionPinUpdate]) → list[Path]

Write every update into both requirement files.

Parameters:

  • repo_root – repository root holding the requirements directory.

  • updates – the bumps to apply, as returned by find_outdated_updates().

utils.update.galaxy.collect_pins(repo_root: Path) → list[CollectionPin]

Collections pinned to the same semver in both requirement files.

Parameters:

repo_root – repository root holding the requirements directory.

utils.update.galaxy.find_outdated_updates(repo_root: Path) → list[CollectionPinUpdate]

Pins whose upstream carries a higher release tag.

Parameters:

repo_root – repository root holding the requirements directory.

utils.update.pip module

Pinned pip requirements in roles/**/*.yml.

A role that hands sys-pip-install a bare package name follows whatever pip resolves; one that pins name==version freezes it, and nothing thaws it again. The docker updater keeps image tags current and dependabot reads pyproject.toml; neither sees a version inside an Ansible task variable, so a pin set once ages silently.

Only exact == pins are collected. A range or a VCS reference states an intent this cannot second-guess.

Suppress a package by placing # nocheck: pip-version on the line directly above the package_name: key, or on the key itself:

# nocheck: pip-version package_name: “automtu==1.1.1”

class utils.update.pip.PipPinEntry(role: str, package: str, version: str, config_path: Path, line_number: int)

Bases: object

One package_name: "<name>==<version>" line.

Parameters:

  • role – the role directory the pin lives in.

  • package – the distribution name on PyPI.

  • version – the pinned version.

  • config_path – the task file holding the pin.

  • line_number – 1-indexed line of the pin.

config_path: Path
line_number: int
package: str
role: str
version: str
class utils.update.pip.PipPinUpdate(entry: 'PipPinEntry', latest: 'str')

Bases: object

entry: PipPinEntry
latest: str
utils.update.pip.apply_updates(updates: list[PipPinUpdate]) → list[Path]

Rewrite each pin to its latest version and return the files changed.

Parameters:

updates – the updates to apply.

utils.update.pip.collect_entries(repo_root: Path) → list[PipPinEntry]

Return every exact pip pin under roles/.

Parameters:

repo_root – repository root to scan.

utils.update.pip.fetch_pypi_versions(package: str) → list[str]

Return every released version of package, empty when PyPI is unreachable.

Parameters:

package – distribution name.

utils.update.pip.find_outdated_updates(repo_root: Path) → list[PipPinUpdate]

Return one update per pin that PyPI has moved past.

Parameters:

repo_root – repository root to scan.

utils.update.pip.latest_release(package: str) → str | None

Return the highest released version of package, or None.

Parameters:

package – distribution name.

utils.update.repository module

Detect and rewrite outdated git ref: values in roles/*/meta/services.yml.

Counterpart to utils.update.docker (which bumps OCI image version: tags). Both share the semver primitives in utils.update.base; this module owns the recursive walker over the services map and the upstream-tag lookup via git ls-remote --tags <repository>.

Recursive walker: every dict at any depth (top-level entity, sub-entity, plugin map, …) that declares BOTH repository: and ref: is a candidate. Only semver-compatible refs are version-checkable; branches (master, main, stable) and commit SHAs are skipped.

Each unique repository URL is queried once per run; the result is reused for every entry pointing at the same repo.

Suppress a check by placing # nocheck: repository-version on the line directly above the ref: key.

A ref: that names its own upstream in an update: block belongs to utils.update.source and is not collected here.

class utils.update.repository.RepositoryRefEntry(role: 'str', entity_path: 'tuple[str, ...]', repository: 'str', ref: 'str', config_path: 'Path', line: 'int')

Bases: object

config_path: Path
entity_path: tuple[str, ...]
line: int
ref: str
repository: str
role: str
class utils.update.repository.RepositoryRefUpdate(entry: 'RepositoryRefEntry', latest: 'str')

Bases: object

entry: RepositoryRefEntry
latest: str
utils.update.repository.apply_updates(updates: list[RepositoryRefUpdate]) → list[Path]

Apply every update in updates to disk. Returns the list of config files that were actually modified.

utils.update.repository.collect_entries(repo_root: Path) → list[RepositoryRefEntry]

Return every semver-checkable (repository, ref) declaration under roles/*/meta/services.yml and roles/*/meta/addons/*.yml whose ref is not suppressed.

utils.update.repository.find_outdated_updates(repo_root: Path) → list[RepositoryRefUpdate]

Cross-reference every collected entry against git ls-remote --tags and return the entries whose ref is older than the latest semver tag with the same depth and flavor.

utils.update.repository.git_ls_remote_tags(url: str) → list[str]

Return tag names from git ls-remote --tags, stripped of the refs/tags/ prefix and the ^{} peeled-tag marker.

utils.update.repository.suppressed_ref_lines(config_path: Path) → set[int]

Return 1-indexed line numbers of ref: declarations that carry a # nocheck: repository-version marker on the preceding line.

utils.update.repository.update_config_refs(config_path: Path, line_to_new_ref: dict[int, str]) → bool

Rewrite specific ref: lines in config_path. line_to_new_ref maps the 1-indexed line number to the desired new value.

utils.update.repository.walk_repo_ref_pairs(node, path: tuple[str, ...]) → Iterator[tuple[tuple[str, ...], str, str]]

Yield (entity_path, repository, ref) for every dict in the tree that declares both keys with truthy string values.

utils.update.runner module

Bump the pinned GitHub-hosted Ubuntu runner label of every workflow job.

GitHub lists its hosted images in the runner-images README; the newest x64 Ubuntu label whose image carries no preview or beta badge is the target.

utils.update.runner.bump_text(text: str, label: str) → str

Raise every older runs-on: ubuntu-NN.NN in text to label.

Parameters:

  • text – workflow source.

  • label – target label, ubuntu-NN.NN.

utils.update.runner.bump_workflows(workflows_dir: Path, label: str) → list[Path]

Rewrite the workflows under workflows_dir to label; return the changed files.

Parameters:

  • workflows_dir – directory holding the workflow YAML files.

  • label – target label, ubuntu-NN.NN.

utils.update.runner.newest_ubuntu_label(readme: str) → str | None

Return the newest GA x64 Ubuntu label listed in the images table.

Parameters:

readme – text of the runner-images README.

utils.update.source module

Resolve version pins that declare their own upstream source.

The Docker updater watches image plus version, the repository updater repository plus ref. A pin outside both - an app_version, a release, a tag in a registry neither updater speaks - declares where its upstream lives:

stalwart:
  webui_version: "v1.0.5"
  update:
    key: webui_version
    type: git_tags
    repository: https://github.com/stalwartlabs/webui.git

key names the pinned key and defaults to version; an entity with several pins declares a list of such blocks. The types are git_tags (repository), registry_tags (image), npm (package), http_regex (one url or several, pattern, optionally a template over the pattern’s named groups) and script (path, run with the current version and printing the latest one).

An addon in meta/addons/<id>.yml declares the same fields in its update: block, beside monitored, catalog and upstream_id. A monitored addon of the github-releases catalog needs no type: its upstream_id already names the repository whose tags are its versions. An addon’s config.archive URL carries the pinned version and moves with it.

class utils.update.source.VersionSourceEntry(role: str, entity: str, key: str, current: str, source: dict[str, Any], config_path: Path, line: int, addon: bool = False, image: str = '')

Bases: object

One pin and the upstream it declares.

Parameters:

  • role – role that carries the pin.

  • entity – service entity in meta/services.yml, or the addon id.

  • key – pinned key.

  • current – pinned version.

  • source – the update: block.

  • config_path – file that carries the pin.

  • line – 1-indexed line of the pin.

  • addon – whether the pin sits in meta/addons/<id>.yml.

  • image – image the pin tags when it is the entity’s version, pulled directly or as the base of a custom build; a bump to a tag its registry answers 404 for is withheld, an indeterminate answer does not block it.

addon: bool = False
config_path: Path
current: str
entity: str
image: str = ''
key: str
line: int
role: str
source: dict[str, Any]
class utils.update.source.VersionSourceUpdate(entry: 'VersionSourceEntry', latest: 'str')

Bases: object

entry: VersionSourceEntry
latest: str
utils.update.source.addon_sources(spec: Any) → list[dict[str, Any]]

Return the version sources an addon declares.

Parameters:

spec – root mapping of a meta/addons/<id>.yml file.

utils.update.source.apply_updates(updates: Iterable[VersionSourceUpdate]) → list[Path]

Rewrite every pin to its newer version and return the changed files.

utils.update.source.candidates(entry: VersionSourceEntry, repo_root: Path) → list[str]

Return every upstream version the entry’s declared source offers.

match keeps only the versions carrying that prefix, and strip removes it again, so a pin without the upstream’s v stays without it.

utils.update.source.collect_entries(repo_root: Path) → list[VersionSourceEntry]

Return every declared version source that is not suppressed.

utils.update.source.dockerhub_named_tags(image: str, match: str) → list[str]

Return the Docker Hub tags of image whose name contains match.

Parameters:

  • image – Docker Hub reference.

  • match – substring the registry filters on, reaching tags beyond the first pages of the unfiltered listing.

utils.update.source.find_outdated_updates(repo_root: Path) → list[VersionSourceUpdate]

Return every declared source that moved ahead of its pin.

utils.update.source.http_regex_versions(urls: str | list[str], pattern: str, template: str = '') → list[str]

Return every version pattern names in the documents at urls.

utils.update.source.invalid_declarations(repo_root: Path) → list[str]

Return one message per update: block that cannot be resolved.

A block naming a key the entity does not carry, a type the resolver does not know, or a pin that is not a semver would otherwise move nothing and say nothing. A monitored addon the resolver does not read is held to the same semver rule.

Parameters:

repo_root – repository root.

utils.update.source.npm_versions(package: str) → list[str]

Return every published version of an npm package.

utils.update.source.outdated(entries: Iterable[VersionSourceEntry], repo_root: Path) → list[VersionSourceUpdate]

Return one update per entry whose source offers a newer version.

utils.update.source.registry_v2_tags(image: str, match: str = '') → list[str]

Return the tags of image from its registry’s v2 API.

Parameters:

  • image – reference including the registry host, e.g. gcr.io/cadvisor/cadvisor.

  • match – when set and the registry can filter server-side, the substring every returned tag carries.

utils.update.source.script_versions(repo_root: Path, role: str, path: str, current: str) → list[str]

Return what a role-local updater prints for its pin.

Parameters:

  • repo_root – repository root.

  • role – role the script belongs to.

  • path – role-relative path of the script.

  • current – the version currently pinned, passed as the first argument.

Module contents