utils.diagnostics package

Submodules

utils.diagnostics.container module

SPOT rescue diagnostics, called ONLY by the CI workflows on failure (never from Ansible rescue: blocks; the lint forbids those).

Captures a full failure snapshot on the current level (all containers with their logs and inspect, all swarm services with ps and logs, the systemd journal, host resources), then RECURSES: it copies itself into every running container that carries python3 and a container runtime (DiD) and repeats the capture there, pulling each nested snapshot back under containers/<name>/nested/ - from the outermost caller down to the deepest runtime, however deep the nesting goes. RESCUE_SEEN carries the ids already entered, so a runtime that lists itself is cut as a cycle.

Every collector is best-effort: a missing source must never abort the capture. INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DNS53_SAMPLER_LOG are required and never defaulted here, so group_vars/all/05_paths.yml stays the one source. Prints one summary and ALWAYS exits 1 so a failure stays a failure.

Usage:

container.py [APP_ID] [CONTEXT]

utils.diagnostics.container.capture(out: Path, name: str, cmd: list[str], *, timeout: int = 120) → None

utils.diagnostics.container.collect_dnsmasq_introspection(out: Path) → None

fd table and rlimits of every live dnsmasq process.

Run 32118850138 caught dnsmasq alive with zero sockets and nothing in the journal; the socket table alone cannot separate closed listeners from an exhausted fd table or a lost netlink watch, the fd list can.

utils.diagnostics.container.collect_host(out: Path, app_id: str, context: str, stamp: str) → None

utils.diagnostics.container.collect_local_dumps(out: Path) → None

Copy the in-play role dumps next to the snapshot.

out itself lives under the dump dir (both derive from INFINITO_RESCUE_DIAGNOSTICS_DIR), so the walk must skip its own output subtree or copytree recurses into the growing destination until ENAMETOOLONG.

utils.diagnostics.container.collect_networks(out: Path, rt: str) → None

Dump the network definitions behind every container’s resolver.

A container resolves through the runtime’s embedded server on 127.0.0.11, which forwards to whatever the network was created with, so inspect on the container shows the address but never the forwarder behind it.

utils.diagnostics.container.collect_resolver_probes(out: Path) → None

Resolve each probed name, once through the stack and once per nameserver.

getent honours nsswitch and cannot address a server, so on its own it cannot say which of the resolvers in resolv.conf refused. Where the distro ships a query tool, ask every nameserver directly; where it does not, the per-name verdicts still stand on their own.

utils.diagnostics.container.collect_runtime(out: Path, rt: str) → tuple[list[str], list[str]]

utils.diagnostics.container.collect_service_state(out: Path) → None

Dump per-service liveness for every loaded service unit.

systemctl list-units reports a forking unit as active/running while its guessed main process is gone, and the containers carry no ps, so the unit table alone cannot answer whether a daemon is still there. MainPID and NRestarts can.

utils.diagnostics.container.daemon_json_dns() → list[str]

Nameservers pinned under the dns key of /etc/docker/daemon.json.

A Tor node’s resolv.conf names only the local dnsmasq, so probing its entries alone cannot say whether the pinned upstream was still alive when that dnsmasq went silent. Anything unreadable or keyless yields [].

utils.diagnostics.container.list_lines(cmd: list[str]) → list[str]

utils.diagnostics.container.main(argv: list[str]) → int

utils.diagnostics.container.recurse(out: Path, rt: str, app_id: str, context: str, depth: int, seen: list[str], stamp: str) → int

utils.diagnostics.container.run(cmd: list[str], *, timeout: int = 120, stdin: bytes | None = None) → CompletedProcess

Best-effort subprocess wrapper: never raises, captures everything.

utils.diagnostics.container.runtime_bin() → str | None

utils.diagnostics.container.sanitize(name: str) → str

utils.diagnostics.container.source_name(path: str) → str

utils.diagnostics.container.write(path: Path, data: bytes) → None

Module contents