From 8a9fe82461b048509a92bc36600691849caf05b6 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Thu, 25 Jun 2026 16:13:50 +1000 Subject: [PATCH 01/13] Revert "Fix typos" This reverts commit 1d42b2bfb939520bd57dee14a8774f0d8f8081b3. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 08f96db..d4215eb 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Enroll logo -Hi folks. I spent a lot of time working on what was to be 0.7.0 of Enroll, before finding too many potental security risks along the way. After tens of security audits by LLMs and the like, to be told over and over 'this is really solid engineering', I'd end up with one that would find a critical vulnerability. I could no longer assume there weren't more. I am not a good programmer, and AI is an echo chamber of optimism. +Hi folks. I spent a lot of time working on what was to be 0.7.0 of Enroll, before finding too many potental security risks along the way. After tens of security audits by LLMs and the like, to be old 'this is really solid', I'd end up with one that would find a critical vulnerability. I could no longer assume there weren't more. I decided it was better that such a project didn't exist. To that end, I'm removing it from the repos and PyPI. From 680d286a20617d70067829e31e8e8d8f1f53ac5f Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Thu, 25 Jun 2026 16:14:08 +1000 Subject: [PATCH 02/13] Revert "So long, etc" This reverts commit d99ba66951376218553bcf8d53ece553db607212. --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index d4215eb..72997c1 100644 --- a/README.md +++ b/README.md @@ -4,10 +4,8 @@ Enroll logo -Hi folks. I spent a lot of time working on what was to be 0.7.0 of Enroll, before finding too many potental security risks along the way. After tens of security audits by LLMs and the like, to be old 'this is really solid', I'd end up with one that would find a critical vulnerability. I could no longer assume there weren't more. +Hi folks. I spent a lot of time working on what was to be 0.7.0 of Enroll, before finding too many potental security risks along the way. I decided it was better that such a project didn't exist. To that end, I'm removing it from the repos and PyPI. -Please uninstall it. - Thanks for all the love in 2026. From 88e9dba39ac560ce828839e237bb45912b78b22c Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Thu, 25 Jun 2026 16:14:19 +1000 Subject: [PATCH 03/13] Revert "Byebye Enroll" This reverts commit 958f8e3aa7fd8a1d81f8f4e2764b7000749ea71a. --- README.md | 717 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 714 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 72997c1..7f1a6ad 100644 --- a/README.md +++ b/README.md @@ -4,8 +4,719 @@ Enroll logo -Hi folks. I spent a lot of time working on what was to be 0.7.0 of Enroll, before finding too many potental security risks along the way. +**enroll** inspects a Linux machine (Debian-like or RedHat-like) and generates configuration-management code from it, as Ansible (default), Puppet or Salt. -I decided it was better that such a project didn't exist. To that end, I'm removing it from the repos and PyPI. +- Detects packages that have been installed. +- Detects package ownership of `/etc` files where possible +- Captures config that has **changed from packaged defaults** where possible (e.g dpkg conffile hashes + package md5sums when available). +- Also captures **service-relevant custom/unowned files** under `/etc//...` (e.g. drop-in config includes). +- Defensively excludes likely secrets (path denylist + content sniff + size caps). +- Captures non-system users and their SSH public keys. In `--dangerous` mode, it also auto-harvests common shell dotfiles such as `.bashrc`, `.profile`, `.bash_logout`, and `.bash_aliases` when appropriate. +- Captures miscellaneous `/etc` files it can't attribute to a package and installs them in an `etc_custom` role. +- When running as root/sudo, captures live writable sysctl state into a `sysctl` role that manages `/etc/sysctl.d/99-enroll.conf`. +- Captures live ipset and iptables runtime state, when active ipsets/iptables rules are present *and* no corresponding persistent ipset/iptables *files* were found. +- Captures symlinks in common applications that rely on them, e.g apache2/nginx 'sites-enabled' +- Tries to capture Flatpak, Snap, Docker image presence +- Captures snowflake-y things found in /usr/local/bin (for non-binary files) and /usr/local/etc +- Avoids trying to start systemd services that were detected as inactive during harvest. -Thanks for all the love in 2026. +--- + +## Mental model + +`enroll` works in two phases: + +1) **Harvest**: collect host facts + relevant files into a harvest bundle (`state.json` + harvested artifacts) +2) **Manifest**: turn that harvest into configuration-management code such as Ansible, Puppet or Salt. + +Additionally, some other functionalities exist: + +- **Diff**: compare two harvests and report what changed (packages/services/users/files) since the previous snapshot. +- **Single-shot mode**: run both harvest and manifest at once. + +--- + +## Output modes: single-site vs multi-site (`--fqdn`) + +`enroll manifest` (and `enroll single-shot`) support multiple output targets. Ansible is the default target and supports two distinct output styles. + +### Single-site mode (default: *no* `--fqdn`) +Use when enrolling **one server** (or generating a “golden” role set you intend to reuse). + +**Characteristics** +- Roles are more self-contained. +- Raw config files live in the role's `files/`. +- Template variables live in the role's `defaults/main.yml`. + +### Multi-site mode (`--fqdn`) +Use when enrolling **several existing servers** quickly, especially if they differ. + +**Characteristics** +- Roles are shared, host-specific state lives in inventory. +- Host inventory drives what gets managed (files/packages/services). +- Non-templated raw files live per-host under `inventory/host_vars///.files/...`. + +**Rule of thumb** +- “Make this one server reproducible/provisionable” → start with **single-site** +- “Get multiple already-running servers under management quickly” → use **multi-site** + +--- + +## Subcommands + +### `enroll harvest` +Harvest state about a host and write a harvest bundle. + +**What it captures (high level)** +- Detected services + service-relevant packages +- “Manual” packages +- Changed-from-default config (plus related custom/unowned files under service dirs) +- Non-system users + SSH public keys +- In `--dangerous` mode: common per-user shell dotfiles that are likely to represent deliberate account customisation +- Misc `/etc` that can't be attributed to a package (`etc_custom` role) +- Static firewall config files such as nftables, UFW, firewalld, `/etc/iptables/rules.v4`, `/etc/iptables/rules.v6`, and `/etc/ipset*` +- Live writable sysctl state via `sysctl -a`, emitted as `/etc/sysctl.d/99-enroll.conf` at manifest time when running as root/sudo (`sysctl` role) +- Live kernel ipset/iptables state via `ipset save`, `iptables-save`, and `ip6tables-save` as a fallback, but only when the corresponding persistent config was not found (`firewall_runtime` role at manifest time) +- Optional user-specified extra files/dirs via `--include-path` (emitted as an `extra_paths` role at manifest time) + +**Common flags** +- Remote harvesting: + - `--remote-host`, `--remote-user`, `--remote-port`, `--remote-ssh-config` + - `--no-sudo` (if you don't want/need sudo) +- Sensitive-data behaviour: + - default: tries to avoid likely secrets + - `--dangerous`: disables secret-safety checks (see “Sensitive data” below) +- Encrypt bundles at rest: + - `--sops `: writes a single encrypted `harvest.tar.gz.sops` instead of a plaintext directory +- Path selection (include/exclude): + - `--include-path ` (repeatable): add extra files/dirs to harvest (even from locations normally ignored, like `/home`). Still subject to secret-safety checks unless `--dangerous`. + - `--exclude-path ` (repeatable): skip files/dirs even if they would normally be harvested. + - Pattern syntax: + - plain path: matches that file; directories match the directory + everything under it + - glob (default): supports `*` and `**` (prefix with `glob:` to force) + - regex: prefix with `re:` or `regex:` + - Precedence: excludes win over includes. + * Using remote mode and auth requires secrets? + * sudo password: + * `--ask-become-pass` (or `-K`) prompts for the sudo password. + * If you forget, and remote sudo requires a password, Enroll will still fall back to prompting in interactive mode (slightly slower due to retry). + * SSH private-key passphrase: + * `--ask-key-passphrase` prompts for the SSH key passphrase. + * `--ssh-key-passphrase-env ENV_VAR` reads the SSH key passphrase from an environment variable (useful for CI/non-interactive runs). + * If neither is provided, and Enroll detects an encrypted key in an interactive session, it will still fall back to prompting on-demand. + * In non-interactive sessions, pass `--ask-key-passphrase` or `--ssh-key-passphrase-env ENV_VAR` when using encrypted private keys. + * Note: `--ask-key-passphrase` and `--ssh-key-passphrase-env` are mutually exclusive. +- Root PATH safety: + - when run as root, Enroll warns and asks for confirmation if `PATH` contains `.`, an empty/relative entry, or a group/world-writable directory. + - use `--assume-safe-path` for trusted non-interactive automation where that `PATH` is intentional. + +Examples (encrypted SSH key) + +```bash +# Interactive +enroll harvest --remote-host myhost.example.com --remote-user myuser --ask-key-passphrase --out /tmp/enroll-harvest + +# Non-interactive / CI +export ENROLL_SSH_KEY_PASSPHRASE='correct horse battery staple' +enroll single-shot --remote-host myhost.example.com --remote-user myuser --ssh-key-passphrase-env ENROLL_SSH_KEY_PASSPHRASE --harvest /tmp/enroll-harvest --out /tmp/enroll-ansible --fqdn myhost.example.com +``` + +--- + +### `enroll manifest` +Generate configuration-management output from an existing harvest bundle. Ansible remains the default; use `--target puppet` for Puppet output or `--target salt` for Salt output. + +**Inputs** +- `--harvest /path/to/harvest` (directory) + or `--harvest /path/to/harvest.tar.gz.sops` (if using `--sops`) + +**Output** +- In plaintext Ansible mode: an Ansible repo-like directory structure (roles/playbooks, and inventory in multi-site mode). +- In plaintext Puppet mode: a Puppet control-repo style layout with `manifests/site.pp` and generated modules under `modules/`. By default, package and service resources are grouped by Debian Section/RPM Group where possible; `--fqdn` or `--no-common-roles` preserves one generated module per Enroll role/snapshot. +- In plaintext Salt mode: a Salt state tree under `states/`, plus `pillar/` data in `--fqdn` mode. By default, package and service resources are grouped by Debian Section/RPM Group where possible; `--fqdn` or `--no-common-roles` preserves one generated SLS role per Enroll role/snapshot. +- In `--sops` mode: a single encrypted file `manifest.tar.gz.sops` containing the generated output. + +**Common flags** +- `--target ansible|puppet|salt`: choose the manifest target (`ansible` is the default). +- `--fqdn `: enables **multi-site** output style for Ansible, emits Puppet Hiera/node output, or emits Salt top/pillar output targeted at that minion ID. Without `--fqdn`, Puppet emits `node default { ... }` and Salt targets `*` in `states/top.sls`. +- `--no-common-roles`: disables the default grouping of package and systemd-unit roles into Debian Section/RPM Group roles, preserving one generated role per package/unit. `--fqdn` implies this behaviour. + +**Role tags** +Generated playbooks tag each role so you can target just the parts you need: + +- Tag format: `role_` (e.g. `role_services`, `role_users`) +- Fallback/safe tag: `role_other` + +Example: +```bash +ansible-playbook -i "localhost," -c local /tmp/enroll-ansible/playbook.yml --tags role_services,role_users +``` + +--- + +### `enroll single-shot` +Convenience wrapper that runs **harvest → manifest** in one command. + +Use this when you want “get me something workable ASAP”. + +Supports the same general flags as harvest/manifest, including `--target`, `--fqdn`, `--no-common-roles`, remote harvest flags, and `--sops`. + +--- + +### `enroll diff` +Compare two harvest bundles and report what changed. + +**What it reports** +- Packages added/removed +- Services enabled added/removed, plus key state changes +- Users added/removed, plus field changes (uid/gid/home/shell/groups, etc.) +- Managed files added/removed/changed (metadata + content hash changes where available) + +**Inputs** +- `--old ` and `--new ` (directories or `state.json` paths) +- `--sops` when comparing SOPS-encrypted harvest bundles +- `--exclude-path ` (repeatable) to ignore file/dir drift under matching paths (same pattern syntax as harvest) +- `--ignore-package-versions` to ignore package version-only drift (upgrades/downgrades) +- `--enforce` to apply the **old** harvest state locally (requires the relevant config manager tool on `PATH` - defaults to `ansible-playbook`) +- `--target` when using `--enforce`, to set the desired config manager tool to manifest to and run) + +**Noise suppression** +- `--exclude-path` is useful for things that change often but you still want in the harvest baseline (e.g. `/var/anacron`). +- `--ignore-package-versions` keeps routine upgrades from alerting; package add/remove drift is still reported. + +**Enforcement (`--enforce` (`--target`))** +If a diff exists and the config manager defined by `--target` (default: ansible) is on the PATH, Enroll will: +1) generate a manifest from the **old** harvest into a temporary directory +2) run the config manager tool against that manifest +3) record in the diff report that the old harvest was enforced + +Enforcement is intentionally “safe”: +- reinstalls packages that were removed (`state: present`), but does **not** attempt downgrades/pinning +- restores users, files (contents + permissions/ownership), and service enable/start state + +If the config manager tool is not on `PATH`, Enroll returns an error and does not enforce. + +**IMPORTANT**: Only enforce harvest bundles that you trust. Validation checks bundle structure and artifact safety; it does not prove that the described system state is safe to apply, e.g. hasn't been tampered with by another user with sufficient permission to do so! + + +**Output formats** +- `--format json` (default for webhooks) +- `--format markdown` / `--format text` (human-oriented) + +**Notifications** +- Webhook: + - `--webhook ` + - `--webhook-format json|markdown|text` + - `--webhook-header 'Header-Name: value'` (repeatable) +- Email (optional): + - `--email-to ` (plus optional SMTP/sendmail-related flags, depending on your install) + +--- + +### `enroll explain` +Analyze a harvest and provide user-friendly explanations for what's in it and why. + +This may also explain why something *wasn't* included (e.g a binary file, a file that was too large, unreadable due to permissions, or looked like a log file/secret. + +Provide either the path to the harvest or the path to its state.json. It can also handle SOPS-encrypted harvests. + +Output can be provided in plaintext or json. + +--- + +### `enroll validate` + +Validates a harvest by checking: + + * state.json exists and is valid JSON + * state.json validates against a JSON Schema (by default the vendored one) + * Every `managed_file` entry has a corresponding artifact at: `artifacts//` + * That there are no **unreferenced files** sitting in `artifacts/` that aren't in the state. + +#### Schema location + overrides + +The master schema lives at: `enroll/schema/state.schema.json`. + +You can override with a local file or URL: + +``` +enroll validate /path/to/harvest --schema ./state.schema.json +enroll validate /path/to/harvest --schema https://enroll.sh/schema/state.schema.json +``` + +Or skip schema checks (still does artifact consistency checks): + +``` +enroll validate /path/to/harvest --no-schema +``` + +#### CLI usage examples + +Validate a local harvest: + +``` +enroll validate ./harvest +``` + +Validate a harvest tarball or a sops bundle: + +``` +enroll validate ./harvest.tar.gz +enroll validate ./harvest.sops --sops +``` + +JSON output + write to file: + +``` +enroll validate ./harvest --format json --out validate.json +``` + +Return exit code 1 for any warnings, not just errors (useful for CI): + +``` +enroll validate ./harvest --fail-on-warnings +``` + +--- + +## Sensitive data + +By default, `enroll` does **not** assume how you handle secrets in Ansible. It will attempt to avoid harvesting likely sensitive data (private keys, passwords, tokens, etc.). This can mean it skips some config files you may ultimately want to manage. + +Safe-mode content scanning is intentionally conservative. It treats common assignment-style credential keys as sensitive, including names such as `password`, `client_secret`, `secret_key`, `auth_token`, `api_key`, `aws_access_key_id`, `aws_secret_access_key`, `azure_client_secret`, `GOOGLE_APPLICATION_CREDENTIALS`, and service-account key names. + +**IMPORTANT**: Enroll ignores comments in files! If you have commented out *real secrets*, there's still a risk that Enroll could capture that data even without `--dangerous`. If you are in doubt, play it safe: use `--sops` and/or encrypt the output at rest in a way that makes sense to you. + +Automatic harvesting of per-user shell dotfiles is also disabled by default, even when those files differ from `/etc/skel`, because `.bashrc`, `.profile`, `.bash_aliases`, and similar files commonly contain exported tokens, credentials, or aliases/functions with embedded secrets. Use `--dangerous` for automatic shell-dotfile capture, or use targeted `--include-path` patterns for narrower safe-mode review. + +If you wish to opt in to collecting everything, use `--dangerous` mode, but be aware of what it means: + +### `--dangerous` + +**IMPORTANT:** 'dangerous' mode is exactly that: it disables “likely secret” safety checks when harvesting system data. + +This means it can copy private keys, TLS key material, API tokens, database passwords, and other credentials into the harvest output **in plaintext**, including paths that would normally be considered very secret. + +If you intend to keep harvests/manifests long-term on disk away from the host or its usual protected paths, strongly consider encrypting them at rest! + +### Encrypt bundles at rest with `--sops` +`--sops` encrypts the harvest and/or manifest outputs into a single `.tar.gz.sops` file (GPG). This is for **storage-at-rest**, not for direct “Ansible SOPS inventory” workflows. + +⚠️ Important: `manifest --sops` produces one encrypted file. You must decrypt + extract it before running `ansible-playbook`. + +--- + +## JinjaTurtle integration + +If [JinjaTurtle](https://git.mig5.net/mig5/jinjaturtle) is installed, `enroll` can generate templates for ini/json/xml/toml-style config in renderers. + +For Ansible: +- Templates live in `roles//templates/...` +- Variables live in: + - single-site: `roles//defaults/main.yml` + - multi-site: `inventory/host_vars//.yml` + +For Salt: +- Templates live in `states/roles//templates/...` +- `file.managed` uses `template: jinja` with per-file `context` values +- In `--fqdn` mode, template context values are written to pillar with the file metadata + +For Puppet: +- JinjaTurtle will use its 'erb' mode if you are running a recent-enough version. +- Templates will be stored in `modules//templates/.erb` +- In `--fqdn` mode, template context values are written to Hiera data. + +You can force template generation on with `--jinjaturtle` or disable it with `--no-jinjaturtle`. + +--- + +## How multi-site avoids “shared role breaks a host” + +In multi-site mode, roles are **data-driven**. The role tasks are generic (“deploy the files listed for this host”, “install the packages listed for this host”, “apply systemd enable/start state listed for this host”). Host inventory decides what applies per-host, avoiding the classic “host2 adds config, host1 breaks” failure mode. + +--- + +# Install + +## Ubuntu/Debian apt repository +```bash +sudo mkdir -p /usr/share/keyrings +curl -fsSL https://mig5.net/static/mig5.asc | sudo gpg --dearmor -o /usr/share/keyrings/mig5.gpg +echo "deb [arch=amd64 signed-by=/usr/share/keyrings/mig5.gpg] https://apt.mig5.net $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/mig5.list +sudo apt update +sudo apt install enroll +``` + +## Fedora + +```bash +sudo rpm --import https://mig5.net/static/mig5.asc + +sudo tee /etc/yum.repos.d/mig5.repo > /dev/null << 'EOF' +[mig5] +name=mig5 Repository +baseurl=https://rpm.mig5.net/$releasever/rpm/$basearch +enabled=1 +gpgcheck=1 +repo_gpgcheck=1 +gpgkey=https://mig5.net/static/mig5.asc +EOF + +sudo dnf upgrade --refresh +sudo dnf install enroll +``` + +## AppImage +Download it from my Releases page, then: + +```bash +chmod +x Enroll.AppImage +./Enroll.AppImage +``` + +## Pip/PipX +```bash +pip install enroll +``` + +## Poetry (dev) +```bash +poetry install +poetry run enroll --help +``` + +--- + +## Found a bug / have a suggestion? + +My Forgejo doesn't currently support federation, so I haven't opened registration/login for issues. + +Instead, email me (see `pyproject.toml`) or contact me on the Fediverse: + +https://goto.mig5.net/@mig5 + +--- + +# Examples + +## Harvest + +### Local harvest +```bash +enroll harvest --out /tmp/enroll-harvest +``` + +### Remote harvest over SSH +```bash +enroll harvest --remote-host myhost.example.com --remote-user myuser --out /tmp/enroll-harvest +``` + +### Remote harvest over SSH, where the SSH configuration is in ~/.ssh/config (e.g a different SSH key) + +Note: you must still pass `--remote-host`, but in this case, its value can be the 'Host' alias of an entry in your `~/.ssh/config`. + +```bash +enroll harvest --remote-host myhostalias --remote-ssh-config ~/.ssh/config --out /tmp/enroll-harvest +``` + +### Include paths (`--include-path`) +```bash +# Add a few dotfiles from /home (still secret-safe unless --dangerous) +enroll harvest --out /tmp/enroll-harvest --include-path '/home/*/.bashrc' --include-path '/home/*/.profile' +``` + +### Exclude paths (`--exclude-path`) +```bash +# Skip specific /usr/local/bin entries (or patterns) +enroll harvest --out /tmp/enroll-harvest --exclude-path '/usr/local/bin/docker-*' --exclude-path '/usr/local/bin/some-tool' +``` + +### Regex include +```bash +enroll harvest --out /tmp/enroll-harvest --include-path 're:^/home/[^/]+/\.config/myapp/.*$' +``` + +### `--dangerous` +```bash +enroll harvest --out /tmp/enroll-harvest --dangerous +``` + +### Remote + dangerous: +```bash +enroll harvest --remote-host myhost.example.com --remote-user myuser --dangerous +``` + +### `--sops` (encrypt at rest) +```bash +# Encrypted harvest bundle (writes /tmp/enroll-harvest/harvest.tar.gz.sops) +enroll harvest --out /tmp/enroll-harvest --dangerous --sops +``` + +--- + +## Manifest + +### Single-site (default: no --fqdn) +```bash +enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-ansible +``` + +### Multi-site (--fqdn) +```bash +enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-ansible --fqdn "$(hostname -f)" +``` + + +### Container image caches + +If Docker or Podman is available during harvest, Enroll records local image-cache metadata from `image ls` and `image inspect`. Images that expose registry `RepoDigest` values are reproducible by digest, for example `registry.example.net/app@sha256:...`; those are the references rendered into manifests. Local image IDs and tag-only images are preserved as evidence and notes, but are not treated as exact registry pull references. + +For Ansible, digest-pinned Docker images are pulled with `community.docker.docker_image_pull` and digest-pinned Podman images are pulled with `containers.podman.podman_image`; harvested tag aliases are re-applied where possible. The generated `requirements.yml` includes `community.docker` and `containers.podman` alongside any other required collections. In `--fqdn` mode the image list is host-specific inventory data. + +### Puppet target +```bash +enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-puppet --target puppet +``` + +The Puppet target renders native packages, users/groups, managed directories/files/symlinks, basic service state, and the generated sysctl file/apply exec when present. Without `--fqdn`, `site.pp` uses `node default { ... }`; with `--fqdn`, it uses `node '' { ... }`. Run from the generated output directory with the generated modules on Puppet's module path, for example: + +```bash +cd /tmp/enroll-puppet +sudo puppet apply --modulepath ./modules manifests/site.pp --noop +``` + +Or with absolute paths: + +```bash +sudo puppet apply --modulepath /tmp/enroll-puppet/modules /tmp/enroll-puppet/manifests/site.pp --noop +``` + +Docker images with registry digests are currently managed with `exec` statements. I know that's nasty, but the `puppetlabs-docker` module is even nastier and creates non-idempotent bash scripts for executing on every run. Worse, if you then reharvest that host that has Puppet installed, you'll get a File resource collision with that very shell script. Believe me, for the simple use case of 'make sure this Docker image is installed', this simple solution is better. + +### Salt target +```bash +enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-salt --target salt +``` + +The Salt target renders native packages, users/groups, managed directories/files/symlinks, basic service state, and the generated sysctl file/apply command when present. Without `--fqdn`, it writes a self-contained state tree under `states/` and targets all minions in `states/top.sls`: + +```bash +cd /tmp/enroll-salt +sudo salt-call --local --file-root ./states state.apply test=True +``` + +With `--fqdn`, it uses Salt's state/pillar split: `states/top.sls` targets the minion ID to reusable generated role SLS files, while `pillar/top.sls` targets the same minion ID to node-specific data under `pillar/nodes/`. Host-specific file artifacts are stored under `states/roles//files/nodes//...` and referenced through `salt://` URLs: + +```bash +enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-salt --target salt --fqdn host.example.com +cd /tmp/enroll-salt +sudo salt-call --local --file-root ./states --pillar-root ./pillar --id host.example.com state.apply test=True +``` + +Re-running Salt `--fqdn` output into the same directory adds or replaces that minion's top/pillar data without deleting other generated minions. + +Docker and Podman images with registry digests are rendered as guarded `cmd.run` states that use the local `docker`/`podman` CLI directly (`pull`, `image inspect`, and `tag`). + +This is because Salt Stack, in 3008, does not have proper Docker extensions that actually work. Wow. It's a bit like Puppet. Seriously, you should probably just be using Ansible. + + +### Manifest with `--sops` +```bash +# Generate encrypted manifest bundle (writes /tmp/enroll-ansible/manifest.tar.gz.sops) +enroll manifest --harvest /tmp/enroll-harvest/harvest.tar.gz.sops --out /tmp/enroll-ansible --sops + +# Decrypt/extract the manifest bundle, then run Ansible from inside ./manifest/ +cd /tmp/enroll-ansible +sops -d manifest.tar.gz.sops | tar -xzvf - +cd manifest +``` + +--- + +## Single-shot + +```bash +enroll single-shot --harvest /tmp/enroll-harvest --out /tmp/enroll-ansible --fqdn "$(hostname -f)" +``` + +Remote single-shot (run harvest over SSH, then manifest locally): +```bash +enroll single-shot --remote-host myhost.example.com --remote-user myuser --harvest /tmp/enroll-harvest --out /tmp/enroll-ansible --fqdn "myhost.example.com" +``` + +--- + +## Diff + +### Compare two harvest directories, output in json +```bash +enroll diff --old /path/to/harvestA --new /path/to/harvestB --format json +``` + +### Diff + webhook notify +```bash +enroll diff --old /path/to/golden/harvest --new /path/to/new/harvest --webhook https://nr.mig5.net/forms/webhooks/xxxx --webhook-format json --webhook-header 'X-Enroll-Secret: xxxx' +``` + +`diff` mode also supports email sending and text or markdown format, as well as `--exit-code` mode to trigger a return code of 2 (useful for crons or CI) + +### Ignore a specific directory or file from the diff +```bash +enroll diff --old /path/to/harvestA --new /path/to/harvestB --exclude-path /var/anacron +``` + +### Ignore package version drift (routine upgrades) but still alert on add/remove +```bash +enroll diff --old /path/to/harvestA --new /path/to/harvestB --ignore-package-versions +``` + +### Enforce the old harvest state when drift is detected +```bash +enroll diff --old /path/to/harvestA --new /path/to/harvestB --enforce --ignore-package-versions --exclude-path /var/anacron +``` + +--- + +## Explain + +### Explain a harvest + +All of these do the same thing: + +```bash +enroll explain /path/to/state.json +enroll explain /path/to/bundle_dir +enroll explain /path/to/harvest.tar.gz +``` + +### Explain a SOPS-encrypted harvest + +```bash +enroll explain /path/to/harvest.tar.gz.sops --sops +``` + +### Explain with JSON output and more examples + +```bash +enroll explain /path/to/state.json --format json --max-examples 25 +``` + +### Example output + +``` +❯ enroll explain /tmp/syrah.harvest +Enroll explain: /tmp/syrah.harvest +Host: syrah.mig5.net (os: debian, pkg: dpkg) +Enroll: 0.2.3 + +Inventory +- Packages: 254 +- Why packages were included (observed_via): + - user_installed: 248 – Package appears explicitly installed (as opposed to only pulled in as a dependency). + - package_role: 232 – Package was referenced by an enroll packages snapshot/role. (e.g. acl, acpid, adduser) + - systemd_unit: 22 – Package is associated with a systemd unit that was harvested. (e.g. postfix.service, tor.service, apparmor.service) + +Roles collected +- users: 1 user(s), 1 file(s), 0 excluded +- services: 19 unit(s), 111 file(s), 6 excluded +- packages: 232 package snapshot(s), 41 file(s), 0 excluded +- apt_config: 26 file(s), 7 dir(s), 10 excluded +- dnf_config: 0 file(s), 0 dir(s), 0 excluded +- firewall_runtime: 2 snapshot(s), 1 ipset(s) +- etc_custom: 70 file(s), 20 dir(s), 0 excluded +- usr_local_custom: 35 file(s), 1 dir(s), 0 excluded +- extra_paths: 0 file(s), 0 dir(s), 0 excluded + +Why files were included (managed_files.reason) +- custom_unowned (179): A file not owned by any package (often custom/operator-managed).. Examples: /etc/apparmor.d/local/lsb_release, /etc/apparmor.d/local/nvidia_modprobe, /etc/apparmor.d/local/sbin.dhclient +- usr_local_bin_script (35): Executable scripts under /usr/local/bin (often operator-installed).. Examples: /usr/local/bin/check_firewall, /usr/local/bin/awslogs +- apt_keyring (13): Repository signing key material used by APT.. Examples: /etc/apt/keyrings/openvpn-repo-public.asc, /etc/apt/trusted.gpg, /etc/apt/trusted.gpg.d/deb.torproject.org-keyring.gpg +- modified_conffile (10): A package-managed conffile differs from the packaged/default version.. Examples: /etc/dnsmasq.conf, /etc/ssh/moduli, /etc/tor/torrc +- logrotate_snippet (9): logrotate snippets/configs referenced in system configuration.. Examples: /etc/logrotate.d/rsyslog, /etc/logrotate.d/tor, /etc/logrotate.d/apt +- apt_config (7): APT configuration affecting package installation and repository behavior.. Examples: /etc/apt/apt.conf.d/01autoremove, /etc/apt/apt.conf.d/20listchanges, /etc/apt/apt.conf.d/70debconf +[...] +``` + +--- + +## Run Ansible + +### Single-site +```bash +ansible-playbook -i "localhost," -c local /tmp/enroll-ansible/playbook.yml +``` + +### Multi-site (--fqdn) +```bash +ansible-playbook /tmp/enroll-ansible/playbooks/"$(hostname -f)".yml +``` + +### Run only specific roles (tags) +Generated playbooks tag each role as `role_` (e.g. `role_users`, `role_services`), so you can speed up targeted runs: +```bash +ansible-playbook -i "localhost," -c local /tmp/enroll-ansible/playbook.yml --tags role_users +``` + +## Configuration file + +As can be seen above, there are a lot of powerful 'permutations' available to all four subcommands. + +Sometimes, it can be easier to store them in a config file so you don't have to remember them! + +Enroll supports reading an ini-style file of all the arguments for each subcommand. + +### Location of the config file + +The path the config file can be specified with `-c` or `--config` on the command-line. Otherwise, +Enroll will look for the `ENROLL_CONFIG` environment variable, `$XDG_CONFIG_HOME/enroll/enroll.ini`, +or `~/.config/enroll/enroll.ini`. + +You may also pass `--no-config` if you deliberately want to ignore the config file even if it existed. + +### Precedence + +Highest wins: + + * Explicit CLI flags + * INI config ([cmd], [enroll]) + * argparse defaults + +### Example config file + +Here is an example. + +Whenever an argument on the command-line has a 'hyphen' in it, just be sure to change it to an underscore in the ini file. + +```ini +[enroll] +# (future global flags may live here) + +[harvest] +dangerous = false +include_path = + /home/*/.bashrc + /home/*/.profile +exclude_path = /usr/local/bin/docker-*, /usr/local/bin/some-tool +# remote_host = yourserver.example.com +# remote_user = you +# remote_port = 2222 + +[manifest] +# you can set defaults here too, e.g. +no_jinjaturtle = true +sops = 54A91143AE0AB4F7743B01FE888ED1B423A3BC99 +# target = ansible (the default), or salt, or puppet + +[diff] +# ignore noisy drift +exclude_path = /var/anacron +ignore_package_versions = true +# enforce = true # requires the target config manager on PATH +# target = puppet (for example, as per manifest) + +[single-shot] +# if you use single-shot, put its defaults here. +# It does not inherit those of the subsections above, so you +# may wish to repeat them here. +include_path = re:^/home/[^/]+/\.config/myapp/.*$ +``` From e9d7d744455a93d6c864006274aa692199545011 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Thu, 25 Jun 2026 16:54:23 +1000 Subject: [PATCH 04/13] Remove puppet and salt --- .forgejo/workflows/ci.yml | 15 +- DEVELOPMENT.md | 770 +------ README.md | 82 +- SECURITY.md | 8 +- enroll/ansible.py | 2 +- enroll/cli.py | 25 +- enroll/cm.py | 15 +- enroll/debian.py | 6 +- enroll/diff.py | 175 +- enroll/ignore.py | 18 +- enroll/jinjaturtle.py | 69 +- enroll/manifest.py | 68 +- enroll/puppet.py | 1840 ----------------- enroll/render_safety.py | 182 -- enroll/salt.py | 1759 ---------------- pyproject.toml | 2 +- tests.sh | 140 -- tests/test_cli.py | 32 - tests/test_diff_bundle.py | 6 +- ...st_diff_ignore_versions_exclude_enforce.py | 154 +- tests/test_manifest_ansible.py | 4 +- tests/test_manifest_puppet.py | 1479 ------------- tests/test_manifest_salt.py | 1045 ---------- tests/test_secret_detection.py | 74 + 24 files changed, 256 insertions(+), 7714 deletions(-) delete mode 100644 enroll/puppet.py delete mode 100644 enroll/salt.py delete mode 100644 tests/test_manifest_puppet.py delete mode 100644 tests/test_manifest_salt.py create mode 100644 tests/test_secret_detection.py diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml index efe6f99..85ec9ee 100644 --- a/.forgejo/workflows/ci.yml +++ b/.forgejo/workflows/ci.yml @@ -35,13 +35,7 @@ jobs: apt-get update DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ ca-certificates curl gnupg git tar gzip findutils bash nodejs procps \ - ansible ansible-lint python3 python3-venv python3-pip pipx systemctl python3-apt jq python3-jsonschema \ - puppet hiera - curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/SaltProjectKey/public | gpg --dearmor | tee /etc/apt/keyrings/salt-archive-keyring.pgp > /dev/null - curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.sources | tee /etc/apt/sources.list.d/salt.sources - apt-get update - DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ - salt-master salt-minion salt-ssh salt-syndic salt-cloud salt-api + ansible ansible-lint python3 python3-venv python3-pip pipx systemctl python3-apt jq python3-jsonschema ;; almalinux) dnf -y upgrade --refresh @@ -49,15 +43,10 @@ jobs: ca-certificates curl-minimal gnupg2 git tar gzip findutils bash which jq nodejs procps-ng \ dnf-plugins-core epel-release dnf -y config-manager --set-enabled crb || true - curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.repo > /etc/yum.repos.d/salt.repo - dnf -y install https://yum.puppet.com/puppet8-release-el-9.noarch.rpm dnf -y makecache dnf -y install \ python3.11 python3.11-devel python3.11-pip gcc make \ - ansible-core ansible-lint systemd rpm httpd \ - puppet-agent \ - salt-master salt-minion salt-ssh salt-syndic salt-cloud salt-api - echo "/opt/puppetlabs/bin" >> "$GITHUB_PATH" + ansible-core ansible-lint systemd rpm httpd ;; *) echo "Unsupported CI distro: ${DISTRO}" >&2 diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index b5ba7cb..f149aa9 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -21,25 +21,21 @@ Harvest bundle state.json artifacts// | - | enroll manifest --target ansible|puppet|salt + | enroll manifest v Generated configuration-management output Ansible roles/playbook - Puppet modules/site.pp/Hiera data - Salt states/pillar data ``` -The harvest bundle is deliberately target-neutral. Ansible, Puppet, and Salt renderers all consume the same `state.json` shape and the same harvested artifacts. Renderer code should translate harvest state into the target's idioms; it should not invent source facts that belong in the harvest. +The harvest bundle is deliberately target-neutral. Ansible renderer consumes the same `state.json` shape and the same harvested artifacts. Renderer code should translate harvest state into the target's idioms; it should not invent source facts that belong in the harvest. `enroll diff` is also built around harvest bundles. It compares two harvests and, when `--enforce` is requested, can generate a temporary manifest from the old harvest and apply it locally with the selected target: ```bash -enroll diff --old ./baseline --new ./current --enforce --target ansible -enroll diff --old ./baseline --new ./current --enforce --target puppet -enroll diff --old ./baseline --new ./current --enforce --target salt +enroll diff --old ./baseline --new ./current --enforce ``` -For enforcement, the user is responsible for having the chosen local apply tool on `PATH`: `ansible-playbook`, `puppet`, or `salt-call`. +For enforcement, the user is responsible for having the chosen local apply tool on `PATH`: `ansible-playbook`. --- @@ -67,8 +63,6 @@ enroll/ manifest.py target router and SOPS manifest wrapper ansible.py Ansible renderer - puppet.py Puppet renderer - salt.py Salt renderer cm.py renderer-neutral CMModule model and grouping helpers role_names.py reserved singleton role-name protection @@ -131,7 +125,7 @@ The supported subcommands are: ```text harvest collect a harvest bundle from a local or remote host -manifest generate Ansible/Puppet/Salt output from a harvest bundle +manifest generate Ansible output from a harvest bundle single-shot run harvest and manifest in one command diff compare two harvest bundles and optionally enforce old state explain produce a human/JSON explanation of a harvest @@ -156,8 +150,8 @@ flowchart TD F --> G[diff.format_report] F --> H{--enforce?} H -->|yes| I[diff.enforce_old_harvest] - I --> J[manifest.manifest target=ansible|puppet|salt] - J --> K[ansible-playbook or puppet apply or salt-call] + I --> J[manifest.manifest] + J --> K[ansible-playbook] B -->|explain| L[explain.explain_state] B -->|validate| M[validate.validate_harvest] ``` @@ -172,10 +166,10 @@ harvest.py depends on harvest_collectors, platform backends, capture policy, system scanners manifest.py - depends on ansible.py, puppet.py, salt.py + depends on ansible.py -ansible.py / puppet.py / salt.py - depend on state.py, cm.py, harvested artifacts, and target-specific helpers +ansible.py + depends on state.py, cm.py, harvested artifacts, and helpers ``` --- @@ -362,8 +356,6 @@ This is one of the most important invariants in the project: > A destination path should normally appear in only one generated role. -Puppet and Salt also run `cm.resolve_catalog_conflicts()` after renderer role collection because they compile a single global catalog and duplicate resources are hard failures. - --- ## 7. File capture and safety policy @@ -595,7 +587,7 @@ artifacts/firewall_runtime/firewall/iptables.v4 artifacts/firewall_runtime/firewall/iptables.v6 ``` -Renderers should only create a firewall runtime role when at least one runtime artifact exists. When firewall runtime is rendered, Ansible/Puppet/Salt also create an `enroll_runtime` role/module/state to own `/etc/enroll` before `/etc/enroll/firewall`. +Renderers should only create a firewall runtime role when at least one runtime artifact exists. When firewall runtime is rendered, Ansible also creates an `enroll_runtime` role/module/state to own `/etc/enroll` before `/etc/enroll/firewall`. ### 9.2 `CronLogrotateCollector` @@ -795,9 +787,7 @@ manifest( Plain mode dispatches to: ```text -target=ansible -> ansible.manifest_from_bundle_dir(..., jinjaturtle=..., no_common_roles=...) -target=puppet -> puppet.manifest_from_bundle_dir(..., jinjaturtle=..., no_common_roles=...) -target=salt -> salt.manifest_from_bundle_dir(..., jinjaturtle=..., no_common_roles=...) +ansible.manifest_from_bundle_dir(..., jinjaturtle=..., no_common_roles=...) ``` SOPS mode: @@ -819,7 +809,7 @@ Before dispatching to a renderer, `manifest.manifest()` calls `validate.validate File: `cm.py` -`CMModule` is the shared resource model used heavily by Puppet and Salt and partially by Ansible. +`CMModule` is the shared resource model used by Ansible and perhaps other renderers in the future. ```python @dataclass @@ -866,22 +856,6 @@ normal manifest, no --no-common-roles: group package/service roles `--fqdn` implies no common roles because host-specific output should preserve per-host state rather than merging unrelated resources into shared roles. -### 12.2 Catalog conflict resolution - -`resolve_catalog_conflicts()` runs for Puppet and Salt. - -It removes duplicates across generated modules/states for: - -- packages, -- groups, -- users, -- directories, -- files, -- symlinks, -- services. - -It also removes directory resources that conflict with a file or link at the same path. This matters because Puppet and Salt compile a single catalog; duplicates that Ansible might tolerate can fail hard there. - --- ## 13. Ansible renderer @@ -981,8 +955,6 @@ Ansible playbook roles are ordered intentionally: Generated playbooks tag roles with `role_`. `diff --enforce --target ansible` uses these tags to narrow enforcement to roles relevant to the drift report when it can. -Puppet and Salt enforcement do not currently narrow to per-role tags; they run the full generated local manifest/state tree. - ### 13.5 Ansible and JinjaTurtle Ansible uses `jinjaturtle.jinjify_managed_files()`. @@ -996,236 +968,7 @@ If JinjaTurtle is unavailable in `auto` mode, fails, emits missing variables, or --- -## 14. Puppet renderer - -File: `puppet.py` - -Entry point: - -```python -puppet.manifest_from_bundle_dir( - bundle_dir, - out_dir, - fqdn=None, - no_common_roles=False, - jinjaturtle="auto", -) -``` - -It instantiates `PuppetManifestRenderer(...).render()`. - -### 14.1 Puppet render flow - -```mermaid -flowchart TD - A[PuppetManifestRenderer.render] --> B[PuppetRole.load_state] - B --> C[resolve_jinjaturtle_mode] - C --> D[_collect_puppet_roles] - D --> E[resolve_catalog_conflicts] - E --> F[_sync_service_notifications] - F --> G[write modules//manifests/init.pp] - G --> H[write metadata.json] - H --> I{fqdn?} - I -->|no| J[write manifests/site.pp with node default] - I -->|yes| K[write hiera.yaml] - K --> L[write data/nodes/.yaml] - L --> M[write Hiera-driven site.pp] - J --> N[README.md] - M --> N -``` - -### 14.2 `PuppetRole` - -`PuppetRole` extends `CMModule` and converts snapshots into Puppet-friendly resources. It handles: - -- packages, -- users and groups, -- managed dirs/files/symlinks, -- services, -- sysctl apply execs, -- Flatpak remotes/apps via guarded `exec`, -- Snap installs via guarded `exec`, -- Docker/Podman images by digest via guarded `exec`, -- firewall runtime files and refresh-only restore execs, -- JinjaTurtle ERB templates and class/Hiera parameter values. - -`_puppet_name()` sanitises module names and avoids Puppet reserved words such as `default`, `class`, `node`, `site`, and `init`. - -### 14.3 Output layout - -Default mode: - -```text -/ - manifests/site.pp - README.md - modules/ - / - metadata.json - manifests/init.pp - files/... - templates/... -``` - -Default `site.pp` includes generated classes in manifest order under a `node default` or named node block. - -### 14.4 Puppet `--fqdn` / Hiera mode - -When `--fqdn` is supplied, Puppet output switches to Hiera-style node data: - -```text -/ - hiera.yaml - manifests/site.pp - data/ - common.yaml - nodes/.yaml - modules/ - / - metadata.json - manifests/init.pp - files/nodes//... - templates/... -``` - -In this mode: - -- `site.pp` includes classes from Hiera key `enroll::classes`, -- `data/nodes/.yaml` contains class list and parameter data, -- module classes are data-driven via Automatic Parameter Lookup, -- node-specific raw file artifacts live under `modules//files/nodes//...`, -- JinjaTurtle ERB template values are written into node Hiera data. - -Re-running Enroll with another `--fqdn` into the same output directory is intended to add or replace that node's YAML without deleting existing node data. - -### 14.5 Puppet and JinjaTurtle - -Puppet now participates in the shared JinjaTurtle integration. - -When enabled, Puppet calls `jinjaturtle` with ERB-specific options: - -```text ---template-engine erb ---puppet-class -``` - -The resulting template is written under: - -```text -modules//templates/.erb -``` - -Static single-node mode renders class parameters with defaults and uses: - -```puppet -content => template('/.erb') -``` - -Hiera mode writes template parameter values into `data/nodes/.yaml` and renders data-driven file resources. - -`jinjaturtle.missing_erb_template_vars()` checks that ERB instance variables such as `@main_key` have matching context/Hiera data. If variables are missing, Enroll falls back to raw file copying rather than emitting a broken Puppet template. - ---- - -## 15. Salt renderer - -File: `salt.py` - -Entry point: - -```python -salt.manifest_from_bundle_dir( - bundle_dir, - out_dir, - fqdn=None, - no_common_roles=False, - jinjaturtle="auto", -) -``` - -It instantiates `SaltManifestRenderer(...).render()`. - -### 15.1 Salt render flow - -```mermaid -flowchart TD - A[SaltManifestRenderer.render] --> B[SaltRole.load_state] - B --> C[resolve_jinjaturtle_mode] - C --> D[_collect_salt_roles] - D --> E[resolve_catalog_conflicts] - E --> F[write states/roles//init.sls] - F --> G{fqdn?} - G -->|no| H[write states/top.sls target '*'] - G -->|yes| I[write pillar node data] - I --> J[write states/top.sls and pillar/top.sls] - H --> K[write config/master.d/enroll.conf] - J --> K - K --> L[README.md] -``` - -### 15.2 `SaltRole` - -`SaltRole` extends `CMModule` and changes `managed_owner_attr` to `user`, because Salt `file.managed` uses `user` rather than `owner`. - -It prepares: - -- packages as `pkg.installed`, -- groups as `group.present`, -- users as `user.present`, -- dirs/files/symlinks as Salt `file.*` states, -- services as `service.running` or `service.dead`, -- Flatpaks/Snaps via guarded `cmd.run`, -- Docker/Podman images via guarded `cmd.run`, -- firewall runtime restore commands, -- optional Jinja templates for managed files. - -### 15.3 Output layout - -Default mode: - -```text -/ - README.md - config/master.d/enroll.conf - states/ - top.sls - roles// - init.sls - files/... - templates/... -``` - -`--fqdn` mode: - -```text -/ - states/ - top.sls - roles//init.sls - pillar/ - top.sls - nodes/_.sls -``` - -The Salt renderer can accumulate node data in `--fqdn` mode and preserves existing top data where appropriate. - -### 15.4 Salt and JinjaTurtle - -Salt uses `jinjaturtle.jinjify_artifact()` directly. When successful, a managed file becomes a Salt `file.managed` with: - -```yaml -source: salt://roles//templates/.j2 -template: jinja -context: {...} -``` - -Salt has one additional compatibility step: `_saltify_jinjaturtle_template()` rewrites Ansible-oriented `to_json(...)` filters emitted by JinjaTurtle into Salt-safe context variables or `tojson` filters. - -If templating fails or is unsupported, the renderer falls back to a literal file copy under `files/`. - ---- - -## 16. Shared JinjaTurtle integration +## 14. JinjaTurtle integration File: `jinjaturtle.py` @@ -1270,27 +1013,24 @@ jinjify_artifact( template_root, jt_exe=..., jt_enabled=..., - template_engine="jinja2" | "erb", - puppet_class=..., # Puppet only ) ``` -Ansible uses `jinjify_managed_files()` because it merges variables into role defaults or host vars. Salt uses `jinjify_artifact()` directly because context lives with each `file.managed`. Puppet uses `jinjify_artifact(..., template_engine="erb", puppet_class=)` so variables line up with Puppet class/Hiera names. +Ansible uses `jinjify_managed_files()` because it merges variables into role defaults or host vars. Safety checks: - `missing_jinja_template_vars()` rejects Jinja2 templates that reference absent variables. -- `missing_erb_template_vars()` rejects ERB templates that reference absent Puppet/Hiera variables. When checks fail, Enroll deletes obsolete generated templates when appropriate and falls back to raw file copying. --- -## 17. Diff, notifications, and enforcement +## 15. Diff, notifications, and enforcement File: `diff.py` -### 17.1 Inputs +### 15.1 Inputs `compare_harvests()` accepts: @@ -1301,7 +1041,7 @@ File: `diff.py` Bundle resolution is handled by `_bundle_from_input()`, which reuses `remote._safe_extract_tar()` for tarball extraction. -### 17.2 What diff compares +### 15.2 What diff compares `compare_harvests()` compares: @@ -1322,7 +1062,7 @@ Reports are formatted by: format_report(report, fmt="text" | "markdown" | "json") ``` -### 17.3 Enforcement decision +### 15.3 Enforcement decision `has_enforceable_drift()` is intentionally conservative. @@ -1343,29 +1083,7 @@ Not enforceable: This keeps `--enforce` focused on restoring baseline state rather than deleting unknown current state or downgrading packages. -### 17.4 Target-selected enforcement - -`enforce_old_harvest()` now accepts `target="ansible" | "puppet" | "salt"`. - -It performs: - -1. resolve the old/baseline harvest, -2. build a best-effort enforcement plan from the diff report, -3. generate a temporary manifest from the old harvest using the selected target, -4. run the matching local apply tool, -5. attach enforcement metadata to the diff report. - -Target commands: - -```text -ansible -> ansible-playbook -i localhost, -c local playbook.yml -puppet -> puppet apply --modulepath ./modules [--hiera_config ./hiera.yaml] manifests/site.pp -salt -> salt-call --local --file-root ./states [--pillar-root ./pillar] state.apply -``` - -Only Ansible uses generated per-role tags to narrow the apply scope. Puppet and Salt enforcement deliberately run the full generated local manifest/state tree for now. The JSON report keeps target-specific compatibility fields such as `ansible_playbook`, `puppet`, or `salt_call`. - -### 17.5 Notifications +### 15.4 Notifications `diff.py` also supports webhooks and email notifications: @@ -1376,9 +1094,9 @@ CLI notification options are only sent when differences exist unless `--notify-a --- -## 18. Explanation and validation +## 16. Explanation and validation -### 18.1 `explain.py` +### 16.1 `explain.py` `explain_state()` reads a harvest and produces text or JSON explaining: @@ -1395,7 +1113,7 @@ CLI notification options are only sent when differences exist unless `--notify-a This is intended to answer “what did Enroll collect and why?” -### 18.2 `validate.py` +### 16.2 `validate.py` `validate_harvest()` checks: @@ -1411,7 +1129,7 @@ This is intended to answer “what did Enroll collect and why?” `validate_harvest()` is used in three important contexts: - `enroll validate` exposes the checks directly to users. -- `manifest.manifest()` validates before rendering Ansible/Puppet/Salt output. +- `manifest.manifest()` validates before rendering Ansible output. - `diff.compare_harvests()` validates both input bundles before comparing them, using `no_schema=True` so older harvests can still be inspected while artifact safety checks remain active. `diff --enforce` renders the old harvest through `manifest.manifest()`, so enforcement also passes through manifest-time validation before a local apply tool is invoked. @@ -1422,7 +1140,7 @@ The CLI supports local schema override with `--schema`, warning failure with `-- --- -## 19. Remote harvesting +## 17. Remote harvesting File: `remote.py` @@ -1441,7 +1159,7 @@ It wraps `_remote_harvest()` and handles: - retrying when remote sudo requires a password, - retrying when an encrypted SSH private key needs a passphrase. -### 19.1 Remote harvest flow +### 17.1 Remote harvest flow ```mermaid flowchart TD @@ -1460,7 +1178,7 @@ flowchart TD `_build_enroll_pyz()` packages the local `enroll` Python package into a zipapp and uses `enroll.cli:main` as its entry point. -### 19.2 SSH config support +### 17.2 SSH config support `--remote-ssh-config` enables Paramiko `SSHConfig` support for settings such as: @@ -1475,7 +1193,7 @@ flowchart TD Unknown host keys are rejected by default through Paramiko's reject policy. Users should have valid host keys in known hosts. -### 19.3 Safe tar extraction +### 17.3 Safe tar extraction `_safe_extract_tar()` validates tar members before extraction and rejects: @@ -1490,13 +1208,13 @@ This helper is reused by remote harvest, manifest SOPS extraction, validate/diff --- -## 20. SOPS support +## 18. SOPS support File: `sopsutil.py` SOPS support is binary tarball encryption, not field-level YAML encryption. -### 20.1 Harvest SOPS mode +### 18.1 Harvest SOPS mode `enroll harvest --sops `: @@ -1505,7 +1223,7 @@ SOPS support is binary tarball encryption, not field-level YAML encryption. 3. encrypts it with SOPS binary mode, 4. writes `harvest.tar.gz.sops` or the requested output file. -### 20.2 Manifest SOPS mode +### 18.2 Manifest SOPS mode `enroll manifest --sops `: @@ -1514,7 +1232,7 @@ SOPS support is binary tarball encryption, not field-level YAML encryption. 3. tars the generated output, 4. encrypts it as a single SOPS file. -### 20.3 Helpers +### 18.3 Helpers `sopsutil.py` provides: @@ -1527,7 +1245,7 @@ Encryption/decryption helpers write via temp files and default to mode `0600`. --- -## 21. Configuration file support +## 19. Configuration file support `cli.py` supports optional INI config files. @@ -1551,33 +1269,21 @@ The translation is argparse-driven, so new flags often gain config-file support --- -## 22. CLI flags that affect multiple layers +## 20. CLI flags that affect multiple layers -### 22.1 `--target` - -`--target ansible|puppet|salt` exists for: - -- `enroll manifest`, -- `enroll single-shot`, -- `enroll diff --enforce`. - -For `manifest` and `single-shot`, it chooses the output renderer. For `diff --enforce`, it chooses both the temporary manifest target and the local apply tool. - -### 22.2 `--fqdn` +### 20.1 `--fqdn` `--fqdn` changes output semantics, not just filenames: - Ansible: uses inventory/host_vars and host-specific artifacts. -- Puppet: uses Hiera node data and Hiera-driven classes. -- Salt: uses pillar node data and minion-targeted top files. `--fqdn` implies no common role grouping. -### 22.3 `--no-common-roles` +### 20.2 `--no-common-roles` Disables the default grouping of package/service snapshots by Debian Section or RPM Group. This preserves one generated role/module/state per package or unit snapshot. -### 22.4 `--jinjaturtle` / `--no-jinjaturtle` +### 20.3 `--jinjaturtle` / `--no-jinjaturtle` The CLI maps these to renderer mode strings: @@ -1587,11 +1293,9 @@ no flag -> auto --no-jinjaturtle -> off ``` -All three manifest targets receive this mode. Puppet uses ERB when JinjaTurtle is enabled; Ansible and Salt use Jinja2. - --- -## 23. Tests and how to navigate them +## 21. Tests and how to navigate them Run tests with: @@ -1606,402 +1310,8 @@ or the repository helper when appropriate: ./tests.sh ``` -Important test files: - -| Test file | What it covers | -|---|---| -| `test_cli.py` | argparse dispatch, remote flags, manifest target forwarding, single-shot flow. | -| `test_cli_config_and_sops.py`, `test_cli_helpers.py` | config-file injection and SOPS output helpers. | -| `test_harvest.py`, `test_harvest_helpers.py` | harvest orchestration, sysctl/firewall helpers, role naming. | -| `test_harvest_collectors.py` | runtime and container image collectors. | -| `test_harvest_cron_logrotate.py` | cron/logrotate unification. | -| `test_harvest_symlinks.py` | nginx/apache enabled symlink capture. | -| `test_accounts.py` | users, Flatpak, Snap parsing/discovery. | -| `test_ignore.py`, `test_ignore_dir.py` | secret/noise policy. | -| `test_pathfilter.py` | include/exclude matching and expansion. | -| `test_platform.py`, `test_platform_backends.py` | platform detection and backend behaviour. | -| `test_debian.py`, `test_rpm.py`, `test_rpm_run.py` | package manager helpers. | -| `test_manifest.py`, `test_manifest_ansible.py` | Ansible rendering and role behaviour. | -| `test_manifest_puppet.py` | Puppet rendering, Hiera mode, reserved names, firewall/container/Flatpak/Snap/JinjaTurtle support. | -| `test_manifest_salt.py` | Salt rendering, pillar mode, JinjaTurtle, firewall/container/Flatpak/Snap support. | -| `test_manifest_symlinks.py` | symlink manifest output. | -| `test_jinjaturtle.py` | shared template generation and fallback safety. | -| `test_diff_bundle.py`, `test_diff_ignore_versions_exclude_enforce.py`, `test_diff_notifications.py` | diff, bundle resolution, target-selected enforcement, notifications. | -| `test_remote.py` | remote harvest, SSH/sudo prompts, safe tar extraction. | -| `test_explain.py` | harvest explanation output. | -| `test_validate.py` | schema/artifact validation. | -| `test_cm.py` | `CMModule` conflict resolution and service-package helpers. | -| `test_fsutil.py`, `test_fsutil_extra.py` | file hashing and stat metadata helpers. | - -When changing behaviour, extend the closest specific tests rather than relying only on broad integration tests. - ---- - -## 24. Common maintenance tasks - -### 24.1 Add a new thing to harvest - -1. Add or extend a dataclass in `harvest_types.py` if existing snapshots cannot represent it. -2. Add a collector under `harvest_collectors/` if it is a distinct feature. -3. Add the collector to the sequence in `harvest.harvest()`. -4. Add the snapshot to the `state = {...}` object in `harvest.harvest()`. -5. Update `schema/state.schema.json`. -6. Update renderers that should emit the new resource. -7. Update `explain.py` and `validate.py` if users need visibility or artifact checks. -8. Add tests for harvest and each renderer. - -### 24.2 Add a new renderer target - -1. Create `.py` with `manifest_from_bundle_dir()`. -2. Load state via `CMModule.load_state()` or `state.load_state()`. -3. Consume `roles_from_state()` and `inventory_packages_from_state()`. -4. Convert snapshots into renderer-specific role/module/state objects. -5. Reuse `CMModule.package_service_entries()` for package/service grouping. -6. Run conflict resolution if the target compiles a global catalog. -7. Write target output and README. -8. Add the target to `manifest.manifest()` validation and dispatch. -9. Add CLI choices in `_add_common_manifest_args()` and diff enforcement if applicable. -10. Add tests. - -### 24.3 Add a new CLI flag - -For harvest-affecting flags: - -1. add the flag to `cli.py` for `harvest` and possibly `single-shot`, -2. forward it to `harvest.harvest()` or `remote.remote_harvest()`, -3. forward it through remote command construction if remote mode needs it, -4. check whether config-file injection handles it, -5. add tests in `test_cli.py` and feature-specific tests. - -For manifest-affecting flags: - -1. add it to `_add_common_manifest_args()` if all manifest-like commands need it, -2. forward it through `manifest.manifest()`, -3. forward it to target renderers, -4. add tests for forwarding and output. - -For diff enforcement flags: - -1. add argparse support under the `diff` subparser, -2. pass values to `compare_harvests()` or `enforce_old_harvest()`, -3. update report formatting if new fields appear, -4. add tests in `test_diff_ignore_versions_exclude_enforce.py` or `test_diff_notifications.py`. - -### 24.4 Change file safety rules - -Modify `ignore.py` and add tests in `test_ignore.py` / `test_ignore_dir.py`. - -Be careful: - -- relaxing safety affects secret exposure risk, -- tightening safety can make expected config disappear, -- binary allowance matters for APT/RPM keyrings, -- `--dangerous` must remain explicit for risky harvesting. - -### 24.5 Change service/package attribution - -Most logic is in: - -- `harvest_collectors/services.py`, -- `package_hints.py`, -- `system_paths.py`, -- package backend `modified_paths()` implementations. - -Preserve these invariants: - -- cron/logrotate should stay unified when installed, -- shared directories should not be attributed too broadly, -- package-manager config belongs in `apt_config`/`dnf_config`, -- `captured_global` should prevent duplicates, -- stopped services should not receive broad restart notifications. - -### 24.6 Change manifest role grouping - -Common grouping uses: - -- `CMModule.package_service_entries()`, -- `package_section_label()`, -- `section_label_for_packages()`. - -Remember: - -- default non-`--fqdn` output groups package/service roles unless `--no-common-roles` is set, -- `--fqdn` implies per-role output, -- Ansible, Puppet, and Salt grouping should stay conceptually aligned, -- Puppet/Salt need `resolve_catalog_conflicts()` after grouping. - -### 24.7 Change JinjaTurtle support - -Shared path support and safety checks belong in `jinjaturtle.py`. - -Renderer-specific behaviour belongs in the renderer: - -- Ansible: variables in defaults or host vars, templates under role `templates/`. -- Puppet: ERB templates, class params or Hiera values. -- Salt: `file.managed` context and Salt-safe Jinja rewrites. - -Fallback-to-raw-copy is part of the product contract unless JinjaTurtle was explicitly required and missing. - -### 24.8 Change diff enforcement - -`diff --enforce` now has a target dimension. - -When changing it, keep these distinctions clear: - -- `has_enforceable_drift()` decides whether enforcement should run. -- `_enforcement_plan()` finds relevant baseline roles. -- Ansible uses role tags from the plan. -- Puppet and Salt currently run a full manifest/state apply. -- `_enforcement_command()` is the source of truth for local apply commands. -- `cli.py` attaches enforcement metadata to the report and formats it. - -Do not make enforcement delete newly added packages/users/files/services unless the safety model is explicitly redesigned. - ---- - -## 25. Important maintenance hazards - -### 25.1 Renderer output is downstream of harvest state - -If a renderer needs information, first ask whether that information belongs in `state.json`. Avoid papering over missing harvest facts inside a renderer. - -### 25.2 `--fqdn` mode is not cosmetic - -`--fqdn` changes where variables and artifacts live and how target inclusion works. - -A change that works in default mode can still break: - -- Ansible host vars, -- Puppet Hiera node data, -- Salt pillar node data. - -### 25.3 Puppet and Salt are stricter about duplicates - -Ansible often tolerates repeated packages or tasks. Puppet and Salt compile catalogs where duplicate resources can fail. Keep `resolve_catalog_conflicts()` in mind whenever adding resources. - -### 25.4 Secret avoidance is part of the product contract - -Default harvest should avoid likely secrets. `--dangerous` exists because useful files may contain secrets. Do not silently make risky harvesting the default. - -### 25.5 Runtime state should not override persistent config - -Firewall runtime capture is skipped when persistent firewall config exists. Preserve this principle for future runtime snapshots. - -### 25.6 JinjaTurtle is best-effort except when explicitly required - -`auto` mode should not make manifest generation fail merely because templating failed. `on` should require the executable; unsupported or unsafe individual files should still fall back to raw copy unless code explicitly changes that contract. - -### 25.7 Role names must be sanitised - -Raw package/service names can be invalid or reserved in Ansible roles, Puppet classes, or Salt SLS names. Use role-name helpers and singleton collision protection. - -### 25.8 Tests encode edge cases - -Many behaviours exist because of previously found edge cases: - -- non-root/no-sudo harvests, -- Puppet reserved words, -- Salt Docker module availability limitations, -- symlink capture, -- JinjaTurtle missing variables, -- Salt JSON filter compatibility, -- file caps, -- SOPS secure temp files, -- tar path traversal, -- target-selected diff enforcement. - -Before simplifying logic, search the tests. - ---- - -## 26. Troubleshooting guide - -### 26.1 Generated manifest references a missing artifact - -Likely causes: - -- `managed_files[*].src_rel` was added without copying into `artifacts/`, -- a renderer used the generated role/module name instead of the artifact role, -- a role was renamed after harvest but before artifact lookup, -- `--fqdn` file prefixes are wrong. - -Start with: +or (just pytests, no root required) ```bash -enroll validate /path/to/harvest +./pytests.sh ``` - -Then inspect: - -```text -state.json roles.*.managed_files[*] -artifacts// -``` - -### 26.2 Puppet fails with duplicate resources - -Check: - -- `_collect_puppet_roles()`, -- `resolve_catalog_conflicts()`, -- `role_order_key()`, -- whether a new resource type needs conflict resolution, -- whether a directory resource conflicts with a file/link of the same path. - -### 26.3 Salt fails with duplicate IDs or missing modules - -Check: - -- `_state_id()` naming, -- `_collect_salt_roles()` grouping, -- `resolve_catalog_conflicts()`, -- guarded `cmd.run` fallbacks for Docker/Podman/Snap/Flatpak. - -Salt uses guarded shell commands for some resources because native states/modules are not consistently available across Salt installations. - -### 26.4 Ansible check mode reports unexpected changes - -Check: - -- role ordering, -- grouped mode versus `--fqdn` / `--no-common-roles`, -- handler notifications, -- whether runtime roles were emitted without runtime artifacts, -- harvested directory/file mode normalisation. - -Grouped and per-role output can legitimately produce different numbers of reported changes. - -### 26.5 A file was not harvested - -Check, in order: - -1. Was it excluded by `--exclude-path`? -2. Was it denied by `IgnorePolicy`? -3. Was it too large? -4. Did it look binary? -5. Did it contain sensitive-looking content? -6. Was it already captured by another role via `captured_global`? -7. Is it outside known scanned locations? -8. Would `--include-path` collect it? -9. Does it require `--dangerous`? - -`enroll explain` can show notes and exclusion reasons. - -### 26.6 `diff --enforce` fails - -Check: - -- whether the selected `--target` tool is on `PATH`, -- `ansible-playbook` for Ansible, -- `puppet` for Puppet, -- `salt-call` for Salt, -- whether the generated temp manifest has the expected target entrypoint, -- whether the report contains enforceable drift, -- whether package drift is only version changes or additions, which enforcement skips. - -### 26.7 Remote harvest fails with sudo or SSH key prompts - -Relevant flags: - -- `--ask-become-pass`, -- `--ask-key-passphrase`, -- `--ssh-key-passphrase-env`, -- `--no-sudo`, -- `--remote-ssh-config`. - -Interactive sessions can prompt and retry. Non-interactive sessions should pass explicit flags or environment variables. - ---- - -## 27. Practical code-reading map - -| Feature/question | Start with | Then read | -|---|---|---| -| CLI option behaviour | `cli.py` | called module for `args.cmd` | -| Local harvest ordering | `harvest.py:harvest()` | `harvest_collectors/` | -| Why a file was skipped | `capture.py`, `ignore.py`, `pathfilter.py` | `explain.py` | -| File metadata/hash helpers | `fsutil.py` | `debian.py`, `capture.py` | -| Service/package attribution | `harvest_collectors/services.py` | `package_hints.py`, `platform.py` | -| APT/DNF config capture | `harvest_collectors/package_manager.py` | `system_paths.py` | -| Users and SSH keys | `harvest_collectors/users.py` | `accounts.py` | -| Flatpak/Snap parsing | `accounts.py` | renderer Flatpak/Snap helpers | -| Docker/Podman images | `harvest_collectors/container_images.py` | renderer container image helpers | -| Runtime firewall | `harvest_collectors/runtime.py`, `harvest.py` | renderer firewall helpers | -| Sysctl | `harvest.py` sysctl helpers | renderer sysctl role functions | -| Ansible output | `ansible.py:AnsibleManifestRenderer.render()` | `_render_*` helpers | -| Puppet output | `puppet.py:PuppetManifestRenderer.render()` | `_collect_puppet_roles()` | -| Salt output | `salt.py:SaltManifestRenderer.render()` | `_collect_salt_roles()` | -| Grouping/common roles | `cm.py` | renderer collection functions | -| JinjaTurtle | `jinjaturtle.py` | renderer managed-content code | -| Diff/enforce | `diff.py` | `manifest.py`, target renderer | -| Validation | `validate.py` | schema file and `state.json` | -| Remote mode | `remote.py` | `cli.py` remote branches | -| SOPS | `sopsutil.py` | `cli.py`, `manifest.py`, `diff.py` | - ---- - -## 28. Glossary - -**Harvest bundle** -A directory or encrypted tarball containing `state.json` and `artifacts/`. - -**Snapshot** -A structured object under `roles` in `state.json`, such as a `ServiceSnapshot` or `PackageSnapshot`. - -**Managed file** -A file Enroll intends generated CM code to recreate. It has a destination path and a matching artifact file. - -**Managed link** -A symlink Enroll intends generated CM code to recreate. - -**Managed dir** -A directory Enroll intends generated CM code to ensure exists with recorded metadata. - -**Role** -The Enroll logical group for related resources. In Ansible it usually maps to an Ansible role. In Puppet it maps to a module/class. In Salt it maps to an SLS role. - -**Artifact role** -The role directory under `artifacts/` that contains a harvested file. This can differ from the generated renderer role when grouping is enabled. - -**Common/grouped role** -A generated role/module/state that merges multiple package/service snapshots by Debian Section or RPM Group. - -**Site mode / `--fqdn` mode** -Host-specific output mode. Ansible uses host vars, Puppet uses Hiera node data, and Salt uses pillar node data. - -**Dangerous mode** -Explicit opt-in mode that relaxes safety checks and enables risky capture such as user shell dotfiles. - -**JinjaTurtle** -Optional external tool used to convert recognised config files into Jinja2 or ERB templates plus variable defaults/context. - -**Enforcement target** -The config manager chosen for `diff --enforce` with `--target ansible|puppet|salt`. - ---- - -## 29. Final maintenance model - -Most changes should preserve this pipeline: - -```text -Collect facts and files safely - -> represent them in target-neutral state.json - -> keep artifact references consistent - -> let each renderer translate the same state into its own idioms - -> validate the bundle and test each target -``` - -Before changing code, ask: - -1. Is this a harvest concern or renderer concern? -2. Does `state.json` or the schema need to change? -3. Does this affect `--fqdn` mode? -4. Does this introduce duplicate ownership of a path/resource? -5. Does this weaken default secret avoidance? -6. Do Puppet and Salt need conflict handling? -7. Does JinjaTurtle fallback still behave safely? -8. Does `diff --enforce --target ...` still do the conservative thing? -9. Do existing tests explain why the current behaviour exists? - -Keeping those boundaries clear is the main way to maintain Enroll without creating subtle cross-target regressions. diff --git a/README.md b/README.md index 7f1a6ad..2d88374 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Enroll logo -**enroll** inspects a Linux machine (Debian-like or RedHat-like) and generates configuration-management code from it, as Ansible (default), Puppet or Salt. +**enroll** inspects a Linux machine (Debian-like or RedHat-like) and generates Ansible configuration-management code from it. - Detects packages that have been installed. - Detects package ownership of `/etc` files where possible @@ -27,7 +27,7 @@ `enroll` works in two phases: 1) **Harvest**: collect host facts + relevant files into a harvest bundle (`state.json` + harvested artifacts) -2) **Manifest**: turn that harvest into configuration-management code such as Ansible, Puppet or Salt. +2) **Manifest**: turn that harvest into Ansible configuration-management code. Additionally, some other functionalities exist: @@ -38,8 +38,6 @@ Additionally, some other functionalities exist: ## Output modes: single-site vs multi-site (`--fqdn`) -`enroll manifest` (and `enroll single-shot`) support multiple output targets. Ansible is the default target and supports two distinct output styles. - ### Single-site mode (default: *no* `--fqdn`) Use when enrolling **one server** (or generating a “golden” role set you intend to reuse). @@ -124,7 +122,7 @@ enroll single-shot --remote-host myhost.example.com --remote-user myuser --ssh-k --- ### `enroll manifest` -Generate configuration-management output from an existing harvest bundle. Ansible remains the default; use `--target puppet` for Puppet output or `--target salt` for Salt output. +Generate Ansible output from an existing harvest bundle. **Inputs** - `--harvest /path/to/harvest` (directory) @@ -132,13 +130,10 @@ Generate configuration-management output from an existing harvest bundle. Ansibl **Output** - In plaintext Ansible mode: an Ansible repo-like directory structure (roles/playbooks, and inventory in multi-site mode). -- In plaintext Puppet mode: a Puppet control-repo style layout with `manifests/site.pp` and generated modules under `modules/`. By default, package and service resources are grouped by Debian Section/RPM Group where possible; `--fqdn` or `--no-common-roles` preserves one generated module per Enroll role/snapshot. -- In plaintext Salt mode: a Salt state tree under `states/`, plus `pillar/` data in `--fqdn` mode. By default, package and service resources are grouped by Debian Section/RPM Group where possible; `--fqdn` or `--no-common-roles` preserves one generated SLS role per Enroll role/snapshot. - In `--sops` mode: a single encrypted file `manifest.tar.gz.sops` containing the generated output. **Common flags** -- `--target ansible|puppet|salt`: choose the manifest target (`ansible` is the default). -- `--fqdn `: enables **multi-site** output style for Ansible, emits Puppet Hiera/node output, or emits Salt top/pillar output targeted at that minion ID. Without `--fqdn`, Puppet emits `node default { ... }` and Salt targets `*` in `states/top.sls`. +- `--fqdn `: enables **multi-site** output style for Ansible (host-specific state lives in inventory `host_vars`). - `--no-common-roles`: disables the default grouping of package and systemd-unit roles into Debian Section/RPM Group roles, preserving one generated role per package/unit. `--fqdn` implies this behaviour. **Role tags** @@ -159,7 +154,7 @@ Convenience wrapper that runs **harvest → manifest** in one command. Use this when you want “get me something workable ASAP”. -Supports the same general flags as harvest/manifest, including `--target`, `--fqdn`, `--no-common-roles`, remote harvest flags, and `--sops`. +Supports the same general flags as harvest/manifest, including `--fqdn`, `--no-common-roles`, remote harvest flags, and `--sops`. --- @@ -178,14 +173,14 @@ Compare two harvest bundles and report what changed. - `--exclude-path ` (repeatable) to ignore file/dir drift under matching paths (same pattern syntax as harvest) - `--ignore-package-versions` to ignore package version-only drift (upgrades/downgrades) - `--enforce` to apply the **old** harvest state locally (requires the relevant config manager tool on `PATH` - defaults to `ansible-playbook`) -- `--target` when using `--enforce`, to set the desired config manager tool to manifest to and run) +- `--enforce` runs `ansible-playbook` against the regenerated manifest) **Noise suppression** - `--exclude-path` is useful for things that change often but you still want in the harvest baseline (e.g. `/var/anacron`). - `--ignore-package-versions` keeps routine upgrades from alerting; package add/remove drift is still reported. -**Enforcement (`--enforce` (`--target`))** -If a diff exists and the config manager defined by `--target` (default: ansible) is on the PATH, Enroll will: +**Enforcement (`--enforce`)** +If a diff exists and `ansible-playbook` is on the PATH, Enroll will: 1) generate a manifest from the **old** harvest into a temporary directory 2) run the config manager tool against that manifest 3) record in the diff report that the old harvest was enforced @@ -316,16 +311,6 @@ For Ansible: - single-site: `roles//defaults/main.yml` - multi-site: `inventory/host_vars//.yml` -For Salt: -- Templates live in `states/roles//templates/...` -- `file.managed` uses `template: jinja` with per-file `context` values -- In `--fqdn` mode, template context values are written to pillar with the file metadata - -For Puppet: -- JinjaTurtle will use its 'erb' mode if you are running a recent-enough version. -- Templates will be stored in `modules//templates/.erb` -- In `--fqdn` mode, template context values are written to Hiera data. - You can force template generation on with `--jinjaturtle` or disable it with `--no-jinjaturtle`. --- @@ -473,53 +458,6 @@ If Docker or Podman is available during harvest, Enroll records local image-cach For Ansible, digest-pinned Docker images are pulled with `community.docker.docker_image_pull` and digest-pinned Podman images are pulled with `containers.podman.podman_image`; harvested tag aliases are re-applied where possible. The generated `requirements.yml` includes `community.docker` and `containers.podman` alongside any other required collections. In `--fqdn` mode the image list is host-specific inventory data. -### Puppet target -```bash -enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-puppet --target puppet -``` - -The Puppet target renders native packages, users/groups, managed directories/files/symlinks, basic service state, and the generated sysctl file/apply exec when present. Without `--fqdn`, `site.pp` uses `node default { ... }`; with `--fqdn`, it uses `node '' { ... }`. Run from the generated output directory with the generated modules on Puppet's module path, for example: - -```bash -cd /tmp/enroll-puppet -sudo puppet apply --modulepath ./modules manifests/site.pp --noop -``` - -Or with absolute paths: - -```bash -sudo puppet apply --modulepath /tmp/enroll-puppet/modules /tmp/enroll-puppet/manifests/site.pp --noop -``` - -Docker images with registry digests are currently managed with `exec` statements. I know that's nasty, but the `puppetlabs-docker` module is even nastier and creates non-idempotent bash scripts for executing on every run. Worse, if you then reharvest that host that has Puppet installed, you'll get a File resource collision with that very shell script. Believe me, for the simple use case of 'make sure this Docker image is installed', this simple solution is better. - -### Salt target -```bash -enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-salt --target salt -``` - -The Salt target renders native packages, users/groups, managed directories/files/symlinks, basic service state, and the generated sysctl file/apply command when present. Without `--fqdn`, it writes a self-contained state tree under `states/` and targets all minions in `states/top.sls`: - -```bash -cd /tmp/enroll-salt -sudo salt-call --local --file-root ./states state.apply test=True -``` - -With `--fqdn`, it uses Salt's state/pillar split: `states/top.sls` targets the minion ID to reusable generated role SLS files, while `pillar/top.sls` targets the same minion ID to node-specific data under `pillar/nodes/`. Host-specific file artifacts are stored under `states/roles//files/nodes//...` and referenced through `salt://` URLs: - -```bash -enroll manifest --harvest /tmp/enroll-harvest --out /tmp/enroll-salt --target salt --fqdn host.example.com -cd /tmp/enroll-salt -sudo salt-call --local --file-root ./states --pillar-root ./pillar --id host.example.com state.apply test=True -``` - -Re-running Salt `--fqdn` output into the same directory adds or replaces that minion's top/pillar data without deleting other generated minions. - -Docker and Podman images with registry digests are rendered as guarded `cmd.run` states that use the local `docker`/`podman` CLI directly (`pull`, `image inspect`, and `tag`). - -This is because Salt Stack, in 3008, does not have proper Docker extensions that actually work. Wow. It's a bit like Puppet. Seriously, you should probably just be using Ansible. - - ### Manifest with `--sops` ```bash # Generate encrypted manifest bundle (writes /tmp/enroll-ansible/manifest.tar.gz.sops) @@ -705,14 +643,12 @@ exclude_path = /usr/local/bin/docker-*, /usr/local/bin/some-tool # you can set defaults here too, e.g. no_jinjaturtle = true sops = 54A91143AE0AB4F7743B01FE888ED1B423A3BC99 -# target = ansible (the default), or salt, or puppet [diff] # ignore noisy drift exclude_path = /var/anacron ignore_package_versions = true -# enforce = true # requires the target config manager on PATH -# target = puppet (for example, as per manifest) +# enforce = true # requires ansible-playbook on PATH [single-shot] # if you use single-shot, put its defaults here. diff --git a/SECURITY.md b/SECURITY.md index a9df1e3..a66b9aa 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -14,7 +14,7 @@ In particular: * If an `enroll.ini` configuration file is loaded, its location and contents are assumed to be owned, selected, and understood by the operator. * The operator is expected to understand the implications of options such as `--dangerous`, `--assume-safe-path`, `--sops`, `--enforce`, `--remote-host`, and `--remote-ssh-config`. * Harvest bundles used for `manifest`, `diff`, or `diff --enforce` are assumed to come from a trusted source unless the operator is deliberately inspecting untrusted input without applying it. -* Configuration-management tools invoked by Enroll, such as Ansible, Puppet, Salt, SOPS, SSH, `sudo`, Docker, Podman, Flatpak, Snap, package managers, and system utilities, are assumed to be the trusted tools the operator intended to use. +* Configuration-management tools invoked by Enroll, such as Ansible, SOPS, SSH, `sudo`, Docker, Podman, Flatpak, Snap, package managers, and system utilities, are assumed to be the trusted tools the operator intended to use. ## What is in scope @@ -31,7 +31,7 @@ In-scope security concerns include: * Writing plaintext harvest outputs into private directories by default. * Hardening root-run output path handling so Enroll does not accidentally write through attacker-prepared symlinks or unsafe parent directories. * Refusing to continue non-interactively when run as root with an unsafe `PATH`, unless the operator explicitly confirms with `--assume-safe-path`. -* Avoiding shell injection in generated manifests where harvested values are embedded into Ansible, Puppet, or Salt output. +* Avoiding shell injection in generated manifests where harvested values are embedded into Ansible output. * Rejecting unknown SSH host keys by default during remote harvests. These measures are defense-in-depth. They are intended to reduce the chance of accidental exposure, unsafe filesystem writes, path traversal, command injection, or dangerous behavior when Enroll is used normally by an administrator. @@ -45,7 +45,7 @@ The following are generally out of scope and should not be reported as Enroll vu * A root user passing `--dangerous` and then observing that Enroll may collect sensitive information. * A root user passing `--assume-safe-path` and then observing that Enroll does not prompt about `PATH` safety. * A root user enforcing a malicious or manually edited harvest bundle with `diff --enforce`. -* A user applying generated Ansible, Puppet, or Salt manifests from an untrusted harvest. +* A user applying generated Ansible manifests from an untrusted harvest. * A user configuring a webhook, email target, SSH proxy command, SOPS binary, package manager, or configuration-management tool that they do not trust. * A compromised system where an attacker already controls root-owned files, root’s shell, root’s configuration, or the privileged tools Enroll invokes. * Reports that amount to “if root runs this tool with malicious options, root can make the system do dangerous things.” @@ -79,7 +79,7 @@ Useful vulnerability reports include issues where Enroll behaves unsafely despit * Enroll follows a symlink or hardlink in a way that causes privileged file disclosure or overwrite. * Enroll extracts a tar member outside the intended harvest directory. * Enroll accepts a malicious harvest artifact that escapes the artifact root. -* Enroll generates an Ansible, Puppet, or Salt manifest where ordinary harvested data can cause command injection. +* Enroll generates an Ansible manifest where ordinary harvested data can cause command injection. * Enroll writes root-run output into an unsafe attacker-controlled path despite its safety checks. * Enroll silently ignores a failed safety check and proceeds anyway. * Enroll accepts an unknown SSH host key unexpectedly. diff --git a/enroll/ansible.py b/enroll/ansible.py index 2eaec0a..f7ef8f4 100644 --- a/enroll/ansible.py +++ b/enroll/ansible.py @@ -724,7 +724,7 @@ def _write_ansible_role_vars( *, site_defaults: Optional[Dict[str, Any]] = None, ) -> None: - """Write role variables using the same mode split as Puppet Hiera/Salt Pillar.""" + """Write role variables using the single-site/site-mode split.""" if ctx.site_mode: _write_role_defaults(role_dir, site_defaults or {}) diff --git a/enroll/cli.py b/enroll/cli.py index 24c8593..a6deb7a 100644 --- a/enroll/cli.py +++ b/enroll/cli.py @@ -458,15 +458,9 @@ def _encrypt_harvest_dir_to_sops( def _add_common_manifest_args(p: argparse.ArgumentParser) -> None: - p.add_argument( - "--target", - choices=["ansible", "puppet", "salt"], - default="ansible", - help="Manifest target to generate (default: ansible).", - ) p.add_argument( "--fqdn", - help="Host FQDN/name for site-mode output (creates target-specific host inventory/data such as Ansible host_vars, Puppet Hiera, or Salt pillar).", + help="Host FQDN/name for site-mode output (creates Ansible host_vars for that host).", ) p.add_argument( "--no-common-roles", @@ -808,15 +802,6 @@ def main() -> None: "Enroll does not attempt to downgrade packages; if the only drift is package version upgrades (or newly installed packages), enforcement is skipped." ), ) - d.add_argument( - "--target", - choices=["ansible", "puppet", "salt"], - default="ansible", - help=( - "Configuration-management target to use with --enforce (default: ansible). " - "Requires ansible-playbook, puppet, or salt-call on PATH as appropriate." - ), - ) d.add_argument( "--out", help="Write the report to this file instead of stdout.", @@ -1119,7 +1104,6 @@ def main() -> None: jinjaturtle=_jt_mode(args), sops_fingerprints=getattr(args, "sops", None), no_common_roles=bool(getattr(args, "no_common_roles", False)), - target=getattr(args, "target", "ansible"), ) if getattr(args, "sops", None) and out_enc: print(str(out_enc)) @@ -1135,7 +1119,7 @@ def main() -> None: ) # Optional enforcement: if drift is detected, attempt to restore the - # system to the *old* (baseline) state using the selected target. + # system to the *old* (baseline) state using ansible. if bool(getattr(args, "enforce", False)): if has_changes: if not has_enforceable_drift(report): @@ -1153,7 +1137,6 @@ def main() -> None: args.old, sops_mode=bool(getattr(args, "sops", False)), report=report, - target=getattr(args, "target", "ansible"), ) except Exception as e: raise SystemExit( @@ -1258,7 +1241,6 @@ def main() -> None: jinjaturtle=_jt_mode(args), sops_fingerprints=list(sops_fps), no_common_roles=bool(getattr(args, "no_common_roles", False)), - target=getattr(args, "target", "ansible"), ) if not args.harvest: print(str(out_file)) @@ -1291,7 +1273,6 @@ def main() -> None: fqdn=args.fqdn, jinjaturtle=_jt_mode(args), no_common_roles=bool(getattr(args, "no_common_roles", False)), - target=getattr(args, "target", "ansible"), ) # For usability (when --harvest wasn't provided), print the harvest path. if not args.harvest: @@ -1324,7 +1305,6 @@ def main() -> None: jinjaturtle=_jt_mode(args), sops_fingerprints=list(sops_fps), no_common_roles=bool(getattr(args, "no_common_roles", False)), - target=getattr(args, "target", "ansible"), ) if not args.harvest: print(str(out_file)) @@ -1345,7 +1325,6 @@ def main() -> None: fqdn=args.fqdn, jinjaturtle=_jt_mode(args), no_common_roles=bool(getattr(args, "no_common_roles", False)), - target=getattr(args, "target", "ansible"), ) except RemoteSudoPasswordRequired: raise SystemExit( diff --git a/enroll/cm.py b/enroll/cm.py index f452313..9dc4700 100644 --- a/enroll/cm.py +++ b/enroll/cm.py @@ -22,9 +22,9 @@ from .state import load_state, state_path, write_state class CMModule: """Renderer-neutral configuration-management resource group. - A CMModule is intentionally small: it captures the resources that a target - renderer can turn into Ansible tasks, Puppet resources, Salt states, etc. - The renderer may still decide how to name/include/order the group. + A CMModule is intentionally small: it captures the resources that the + renderer turns into Ansible tasks. The renderer may still decide how to + name/include/order the group. """ role_name: str @@ -806,12 +806,11 @@ def _drop_duplicate_mapping_items( def resolve_catalog_conflicts(modules: Iterable[CMModule]) -> None: - """Resolve global catalog conflicts before renderer output. + """Resolve global catalog conflicts in the shared model. - Puppet and Salt compile a single resource catalog. Ansible can tolerate the - same package, service, or parent directory appearing in more than one role; - catalog targets cannot. Resolve those conflicts in the shared model rather - than deleting renderer output after the fact. + Deduplicates the same package, service, or parent directory appearing in + more than one role. The Ansible renderer tolerates such duplicates, but this + helper remains available for any catalog-style consumer of the shared model. """ ordered = list(modules) diff --git a/enroll/debian.py b/enroll/debian.py index 14becad..379f390 100644 --- a/enroll/debian.py +++ b/enroll/debian.py @@ -228,6 +228,10 @@ def read_pkg_md5sums(pkg: str) -> Dict[str, str]: line = line.strip() if not line: continue - md5, rel = line.split(None, 1) + parts = line.split(None, 1) + if len(parts) != 2: + # Skip malformed/truncated lines instead of aborting the harvest. + continue + md5, rel = parts m[rel.strip()] = md5.strip() return m diff --git a/enroll/diff.py b/enroll/diff.py index 110ca9d..8122150 100644 --- a/enroll/diff.py +++ b/enroll/diff.py @@ -682,40 +682,7 @@ def _role_tag(role: str) -> str: return f"role_{safe}" -def _normalise_enforcement_target(target: str) -> str: - t = str(target or "ansible").strip().lower() - if t not in {"ansible", "puppet", "salt"}: - raise ValueError(f"unsupported enforcement target: {target!r}") - return t - - -def _enforcement_tool(target: str) -> Tuple[str, str]: - """Return (binary-name, human-label) for a local enforcement target.""" - if target == "puppet": - return "puppet", "puppet apply" - if target == "salt": - return "salt-call", "salt-call" - return "ansible-playbook", "ansible-playbook" - - -def _require_enforcement_tool(target: str) -> Tuple[str, str]: - binary, label = _enforcement_tool(target) - exe = shutil.which(binary) - if not exe: - install_hint = { - "ansible": "Ansible", - "puppet": "Puppet", - "salt": "Salt", - }.get(target, target) - raise RuntimeError( - f"{binary} not found on PATH " - f"(cannot enforce with target {target}; install {install_hint})" - ) - return exe, label - - def _enforcement_command( - target: str, exe: str, manifest_dir: Path, *, @@ -724,69 +691,27 @@ def _enforcement_command( """Return the local apply command and environment for a rendered manifest.""" env = dict(os.environ) - if target == "ansible": - playbook = manifest_dir / "playbook.yml" - if not playbook.exists(): - raise RuntimeError( - f"manifest did not produce expected playbook.yml at {playbook}" - ) + playbook = manifest_dir / "playbook.yml" + if not playbook.exists(): + raise RuntimeError( + f"manifest did not produce expected playbook.yml at {playbook}" + ) - cfg = manifest_dir / "ansible.cfg" - if cfg.exists(): - env["ANSIBLE_CONFIG"] = str(cfg) + cfg = manifest_dir / "ansible.cfg" + if cfg.exists(): + env["ANSIBLE_CONFIG"] = str(cfg) - cmd = [ - exe, - "-i", - "localhost,", - "-c", - "local", - str(playbook), - ] - if tags: - cmd.extend(["--tags", ",".join(tags)]) - return cmd, env - - if target == "puppet": - site_pp = manifest_dir / "manifests" / "site.pp" - if not site_pp.exists(): - raise RuntimeError( - f"manifest did not produce expected Puppet site.pp at {site_pp}" - ) - - cmd = [ - exe, - "apply", - "--modulepath", - str(manifest_dir / "modules"), - ] - hiera_config = manifest_dir / "hiera.yaml" - if hiera_config.exists(): - cmd.extend(["--hiera_config", str(hiera_config)]) - cmd.append(str(site_pp)) - return cmd, env - - if target == "salt": - states_dir = manifest_dir / "states" - top_sls = states_dir / "top.sls" - if not top_sls.exists(): - raise RuntimeError( - f"manifest did not produce expected Salt top.sls at {top_sls}" - ) - - cmd = [ - exe, - "--local", - "--file-root", - str(states_dir), - ] - pillar_dir = manifest_dir / "pillar" - if pillar_dir.exists(): - cmd.extend(["--pillar-root", str(pillar_dir)]) - cmd.extend(["state.apply"]) - return cmd, env - - raise ValueError(f"unsupported enforcement target: {target!r}") + cmd = [ + exe, + "-i", + "localhost,", + "-c", + "local", + str(playbook), + ] + if tags: + cmd.extend(["--tags", ",".join(tags)]) + return cmd, env def _enforcement_plan( @@ -898,22 +823,18 @@ def enforce_old_harvest( *, sops_mode: bool = False, report: Optional[Dict[str, Any]] = None, - target: str = "ansible", ) -> Dict[str, Any]: """Enforce the *old* (baseline) harvest state on the current machine. - This renders a temporary manifest from the old harvest using the requested - target, then runs the target's local apply command: - - ansible: ansible-playbook -i localhost, -c local playbook.yml - - puppet: puppet apply --modulepath ./modules manifests/site.pp - - salt: salt-call --local --file-root ./states state.apply + This renders a temporary Ansible manifest from the old harvest, then runs + the local apply command: + - ansible-playbook -i localhost, -c local playbook.yml Returns a dict suitable for attaching to the diff report under report['enforcement']. """ - target = _normalise_enforcement_target(target) - tool_exe, tool_label = _require_enforcement_tool(target) + tool_exe = "ansible-playbook" # Import lazily to avoid heavy import cost and potential CLI cycles. from .manifest import manifest @@ -933,12 +854,10 @@ def enforce_old_harvest( if report is not None: plan = _enforcement_plan(report, old_state, old_b.dir) roles = list(plan.get("roles") or []) - # Only Ansible has generated per-role tags that can safely narrow - # the apply scope. Puppet and Salt enforcement deliberately run the - # full generated local manifest/catalog for now. - if target == "ansible": - t = list(plan.get("tags") or []) - tags = t if t else None + # Ansible has generated per-role tags that can safely narrow the + # apply scope. + t = list(plan.get("tags") or []) + tags = t if t else None with tempfile.TemporaryDirectory(prefix="enroll-enforce-") as td: td_path = Path(td) @@ -951,11 +870,10 @@ def enforce_old_harvest( # refuses to write into an existing destination, so use a fresh # child path under the secure temporary directory. manifest_dir = td_path / "manifest" - manifest(str(old_b.dir), str(manifest_dir), target=target) + manifest(str(old_b.dir), str(manifest_dir)) # 2) Apply it locally. cmd, env = _enforcement_command( - target, tool_exe, manifest_dir, tags=tags, @@ -967,12 +885,12 @@ def enforce_old_harvest( if _progress_enabled(): if tags: sys.stderr.write( - f"Enforce: running {tool_label} (tags: {','.join(tags)})\n", + f"Enforce: running {tool_exe} tags: {','.join(tags)})\n", ) else: - sys.stderr.write(f"Enforce: running {tool_label}\n") + sys.stderr.write(f"Enforce: running {tool_exe}\n") sys.stderr.flush() - spinner = _Spinner(f" {tool_label}") + spinner = _Spinner(f" {tool_exe}") spinner.start() try: @@ -990,7 +908,7 @@ def enforce_old_harvest( rc = p.returncode if p is not None else None spinner.stop( final_line=( - f"Enforce: {tool_label} finished in {elapsed:0.1f}s" + f"Enforce: {tool_exe} finished in {elapsed:0.1f}s" + (f" (rc={rc})" if rc is not None else "") ), ) @@ -999,22 +917,15 @@ def enforce_old_harvest( info: Dict[str, Any] = { "status": "applied" if p.returncode == 0 else "failed", - "target": target, - "tool": tool_label, "executable": tool_exe, "started_at": started_at, "finished_at": finished_at, "command": cmd, "returncode": int(p.returncode), } - # Keep the original Ansible-specific field for compatibility with - # existing consumers of the JSON report. - if target == "ansible": - info["ansible_playbook"] = tool_exe - elif target == "puppet": - info["puppet"] = tool_exe - elif target == "salt": - info["salt_call"] = tool_exe + # Keep the Ansible-specific field for compatibility with existing + # consumers of the JSON report. + info["ansible_playbook"] = tool_exe info["roles"] = roles info["tags"] = list(tags or []) @@ -1024,7 +935,7 @@ def enforce_old_harvest( if p.returncode != 0: err = (p.stderr or p.stdout or "").strip() raise RuntimeError( - f"{tool_label} failed" + f"{tool_exe} failed" + (f" (rc={p.returncode})" if p.returncode is not None else "") + (f": {err}" if err else "") ) @@ -1069,9 +980,6 @@ def _report_text(report: Dict[str, Any]) -> str: if enf: lines.append("\nEnforcement") status = str(enf.get("status") or "").strip().lower() - tool = str(enf.get("tool") or "ansible-playbook") - target = str(enf.get("target") or "ansible") - via = f"{tool} ({target})" if target and target not in tool else tool if status == "applied": extra = "" tags = enf.get("tags") or [] @@ -1081,7 +989,7 @@ def _report_text(report: Dict[str, Any]) -> str: elif scope: extra = f" ({scope})" lines.append( - f" applied old harvest via {via} (rc={enf.get('returncode')})" + f" applied old harvest (rc={enf.get('returncode')})" + extra + ( f" (finished {enf.get('finished_at')})" @@ -1091,7 +999,7 @@ def _report_text(report: Dict[str, Any]) -> str: ) elif status == "failed": lines.append( - f" attempted enforcement but {via} failed (rc={enf.get('returncode')})" + f" attempted enforcement but failed (rc={enf.get('returncode')})" ) elif status == "skipped": r = enf.get("reason") @@ -1231,9 +1139,6 @@ def _report_markdown(report: Dict[str, Any]) -> str: if enf: out.append("\n## Enforcement\n") status = str(enf.get("status") or "").strip().lower() - tool = str(enf.get("tool") or "ansible-playbook") - target = str(enf.get("target") or "ansible") - via = f"{tool} ({target})" if target and target not in tool else tool if status == "applied": extra = "" tags = enf.get("tags") or [] @@ -1243,7 +1148,7 @@ def _report_markdown(report: Dict[str, Any]) -> str: elif scope: extra = f" ({scope})" out.append( - f"- ✅ Applied old harvest via {via}" + "- ✅ Applied old harvest" + extra + ( f" (rc={enf.get('returncode')})" @@ -1259,7 +1164,7 @@ def _report_markdown(report: Dict[str, Any]) -> str: ) elif status == "failed": out.append( - f"- ⚠️ Attempted enforcement but {via} failed" + "- ⚠️ Attempted enforcement but failed" + ( f" (rc={enf.get('returncode')})" if enf.get("returncode") is not None diff --git a/enroll/ignore.py b/enroll/ignore.py index eed4035..47ee20f 100644 --- a/enroll/ignore.py +++ b/enroll/ignore.py @@ -27,6 +27,9 @@ DEFAULT_DENY_GLOBS = [ "/etc/gshadow", "/etc/*shadow", "/etc/letsencrypt/*", + "/etc/ppp/chap-secrets", + "/etc/ppp/pap-secrets", + "/etc/ppp/*-secrets", "/usr/local/etc/ssl/private/*", "/usr/local/etc/ssh/ssh_host_*", "/usr/local/etc/*shadow", @@ -91,7 +94,20 @@ SENSITIVE_CONTENT_PATTERNS = [ \s*[:=] """ ), - re.compile(rb"(?i)\b(pass|passwd|token|secret|api[_-]?key)\b"), + re.compile( + rb"(?i)\b(pass|passwd|password|passphrase|token|secret|" + rb"credentials?|api[_-]?key)\b" + ), + # Credentials embedded in connection-string URIs, e.g. + # postgres://user:pass@host, redis://:pass@host, amqp://u:p@host + # The keyword regex above keys on assignment-style names and misses these. + re.compile(rb"(?i)[a-z][a-z0-9+.-]*://[^/\s:@]*:[^/\s@]+@[^/\s]"), + # HTTP(S) Authorization / Proxy-Authorization header values carrying a + # bearer/basic/digest credential. + re.compile( + rb"(?im)^\s*(?:proxy-)?authorization\s*:\s*" + rb"(?:bearer|basic|token|digest)\s+\S" + ), ] COMMENT_PREFIXES = (b"#", b";", b"//") diff --git a/enroll/jinjaturtle.py b/enroll/jinjaturtle.py index 2b8f467..f12f924 100644 --- a/enroll/jinjaturtle.py +++ b/enroll/jinjaturtle.py @@ -82,7 +82,6 @@ _JINJA_FOR_RE = re.compile( r"{%\s*for\s+([A-Za-z_][A-Za-z0-9_]*)\s+in\s+([A-Za-z_][A-Za-z0-9_]*)\b" ) _JINJA_SPECIAL_VARS = {"loop", "true", "false", "none", "True", "False", "None"} -_ERB_INSTANCE_VAR_RE = re.compile(r"<%=?[^%]*@([A-Za-z_][A-Za-z0-9_]*)", re.S) def _find_undeclared_jinja_vars(template_text: str) -> Set[str]: @@ -121,21 +120,6 @@ def missing_jinja_template_vars( return {name for name in referenced if name not in context} -def missing_erb_template_vars(template_text: str, context: Dict[str, Any]) -> Set[str]: - """Return ERB ``@param`` references absent from Puppet Hiera/class data.""" - - local_names: Set[str] = set() - for key in context: - text = str(key) - if "::" in text: - local_names.add(text.split("::", 1)[1]) - else: - local_names.add(text) - - referenced = set(_ERB_INSTANCE_VAR_RE.findall(template_text)) - return {name for name in referenced if name not in local_names} - - def jinjify_artifact( bundle_dir: str | Path, artifact_role: str, @@ -147,14 +131,8 @@ def jinjify_artifact( jt_enabled: bool, overwrite_templates: bool = True, role_name: Optional[str] = None, - template_engine: str = "jinja2", - puppet_class: Optional[str] = None, ) -> Optional[JinjifiedArtifact]: - """Best-effort conversion of one harvested artifact into a template. - - Ansible/Salt use Jinja2 output. Puppet uses ERB output with Puppet Hiera - keys when a new enough JinjaTurtle is available. - """ + """Best-effort conversion of one harvested artifact into a Jinja2 template.""" if not (jt_enabled and jt_exe and can_jinjify_path(dest_path)): return None @@ -164,31 +142,20 @@ def jinjify_artifact( return None try: - run_kwargs: Dict[str, Any] = { - "role_name": role_name or artifact_role, - "force_format": infer_other_formats(dest_path), - } - # Keep the historical call shape for Ansible/Salt and for tests that - # monkeypatch run_jinjaturtle with the old signature. Puppet/ERB is - # the only path that needs the newer JinjaTurtle CLI switches. - if template_engine != "jinja2": - run_kwargs["template_engine"] = template_engine - if puppet_class: - run_kwargs["puppet_class"] = puppet_class - result = run_jinjaturtle(jt_exe, str(artifact_path), **run_kwargs) + result = run_jinjaturtle( + jt_exe, + str(artifact_path), + role_name=role_name or artifact_role, + force_format=infer_other_formats(dest_path), + ) except Exception: return None # nosec - best-effort template generation - ext = "erb" if template_engine == "erb" else "j2" - template_rel = Path(src_rel).as_posix() + f".{ext}" + template_rel = Path(src_rel).as_posix() + ".j2" template_dst = Path(template_root) / template_rel context = yaml_load_mapping(result.vars_text) - missing = ( - missing_erb_template_vars(result.template_text, context) - if template_engine == "erb" - else missing_jinja_template_vars(result.template_text, context) - ) + missing = missing_jinja_template_vars(result.template_text, context) if missing: # If this role was generated into an existing output directory, avoid # leaving an obsolete template behind after falling back to a raw copy. @@ -243,10 +210,7 @@ def jinjify_managed_files( ) -> Tuple[Set[str], str]: """Jinjify a list of managed files and return Ansible-style vars text. - The return shape intentionally matches the historical Ansible helper: - ``(templated_src_rels, combined_vars_text)``. Salt uses - :func:`jinjify_artifact` directly because it stores variables as a context - map per managed file. + The return shape is ``(templated_src_rels, combined_vars_text)``. """ templated: Set[str] = set() vars_map: Dict[str, Any] = {} @@ -340,8 +304,6 @@ def run_jinjaturtle( *, role_name: str, force_format: Optional[str] = None, - template_engine: str = "jinja2", - puppet_class: Optional[str] = None, ) -> JinjifyResult: """ Run jinjaturtle against src_path and return (template, defaults-yaml). @@ -349,9 +311,6 @@ def run_jinjaturtle( jinjaturtle CLI: jinjaturtle -r [-f ] [-d ] [-t ] - - Newer JinjaTurtle versions also support ``--template-engine erb`` and - ``--puppet-class`` for Puppet/Hiera output. """ src = Path(src_path) if not src.is_file(): @@ -360,9 +319,7 @@ def run_jinjaturtle( with tempfile.TemporaryDirectory(prefix="enroll-jt-") as td: td_path = Path(td) defaults_out = td_path / "defaults.yml" - template_out = td_path / ( - "template.erb" if template_engine == "erb" else "template.j2" - ) + template_out = td_path / "template.j2" cmd = [ jt_exe, @@ -376,10 +333,6 @@ def run_jinjaturtle( ] if force_format: cmd.extend(["-f", force_format]) - if template_engine != "jinja2": - cmd.extend(["--template-engine", template_engine]) - if puppet_class: - cmd.extend(["--puppet-class", puppet_class]) p = subprocess.run(cmd, text=True, capture_output=True) # nosec if p.returncode != 0: diff --git a/enroll/manifest.py b/enroll/manifest.py index 3ebfefe..3fea7b8 100644 --- a/enroll/manifest.py +++ b/enroll/manifest.py @@ -8,8 +8,6 @@ from pathlib import Path from typing import List, Optional from .ansible import manifest_from_bundle_dir as manifest_ansible_from_bundle_dir -from .puppet import manifest_from_bundle_dir as manifest_puppet_from_bundle_dir -from .salt import manifest_from_bundle_dir as manifest_salt_from_bundle_dir from .harvest_safety import ensure_safe_output_parent from .manifest_safety import validate_site_fqdn from .remote import _safe_extract_tar @@ -175,7 +173,6 @@ def manifest( jinjaturtle: str = "auto", # auto|on|off sops_fingerprints: Optional[List[str]] = None, no_common_roles: bool = False, - target: str = "ansible", ) -> Optional[str]: """Render a configuration-management manifest from a harvest. @@ -193,9 +190,6 @@ def manifest( - In SOPS mode: the path to the encrypted manifest bundle (.sops) - In plain mode: None """ - target = (target or "ansible").strip().lower() - if target not in {"ansible", "puppet", "salt"}: - raise ValueError(f"unsupported manifest target: {target!r}") fqdn = validate_site_fqdn(fqdn) sops_mode = bool(sops_fingerprints) @@ -216,30 +210,13 @@ def manifest( ) if not sops_mode: - if target == "puppet": - manifest_puppet_from_bundle_dir( - resolved_bundle_dir, - out, - fqdn=fqdn, - no_common_roles=no_common_roles, - jinjaturtle=jinjaturtle, - ) - elif target == "salt": - manifest_salt_from_bundle_dir( - resolved_bundle_dir, - out, - fqdn=fqdn, - no_common_roles=no_common_roles, - jinjaturtle=jinjaturtle, - ) - else: - manifest_ansible_from_bundle_dir( - resolved_bundle_dir, - out, - fqdn=fqdn, - jinjaturtle=jinjaturtle, - no_common_roles=no_common_roles, - ) + manifest_ansible_from_bundle_dir( + resolved_bundle_dir, + out, + fqdn=fqdn, + jinjaturtle=jinjaturtle, + no_common_roles=no_common_roles, + ) return None # SOPS mode: generate into a secure temp dir, then tar+encrypt into a single file. @@ -248,30 +225,13 @@ def manifest( td_out = tempfile.TemporaryDirectory(prefix="enroll-manifest-") tmp_out = Path(td_out.name) / "out" - if target == "puppet": - manifest_puppet_from_bundle_dir( - resolved_bundle_dir, - str(tmp_out), - fqdn=fqdn, - no_common_roles=no_common_roles, - jinjaturtle=jinjaturtle, - ) - elif target == "salt": - manifest_salt_from_bundle_dir( - resolved_bundle_dir, - str(tmp_out), - fqdn=fqdn, - no_common_roles=no_common_roles, - jinjaturtle=jinjaturtle, - ) - else: - manifest_ansible_from_bundle_dir( - resolved_bundle_dir, - str(tmp_out), - fqdn=fqdn, - jinjaturtle=jinjaturtle, - no_common_roles=no_common_roles, - ) + manifest_ansible_from_bundle_dir( + resolved_bundle_dir, + str(tmp_out), + fqdn=fqdn, + jinjaturtle=jinjaturtle, + no_common_roles=no_common_roles, + ) enc = _encrypt_manifest_out_dir_to_sops( tmp_out, out_file, list(sops_fingerprints or []) diff --git a/enroll/puppet.py b/enroll/puppet.py deleted file mode 100644 index baf7596..0000000 --- a/enroll/puppet.py +++ /dev/null @@ -1,1840 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -import re -import shlex -from pathlib import Path -from typing import Any, Dict, Iterable, List, Optional, Set, Tuple - -import yaml - -from .cm import ( - CMModule, - resolve_catalog_conflicts, - role_order_key, - markdown_list, -) -from .manifest_safety import ( - copy_safe_artifact_file, - prepare_manifest_output_dir, - safe_artifact_file, -) -from .render_safety import puppet_hiera_safe_data -from .state import inventory_packages_from_state, roles_from_state -from .jinjaturtle import ( - can_jinjify_path, - jinjify_artifact, - managed_file_var_prefix, - resolve_jinjaturtle_mode, -) - - -class PuppetRole(CMModule): - """Puppet-specific view of a renderer-neutral CMModule.""" - - def __init__(self, role_name: str) -> None: - super().__init__( - role_name=role_name, - module_name=_puppet_name(role_name, fallback="enroll_role"), - ) - self.container_images: List[Dict[str, Any]] = [] - self.flatpak_remotes: List[Dict[str, Any]] = [] - self.flatpaks: List[Dict[str, Any]] = [] - self.snaps: List[Dict[str, Any]] = [] - self.template_hiera: Dict[str, Any] = {} - - def has_resources(self) -> bool: - return self.has_resources_or_attrs( - "container_images", "flatpak_remotes", "flatpaks", "snaps" - ) - - def add_service_snapshot(self, snap: Dict[str, Any]) -> None: - self.add_service_snapshot_state( - snap, state_key="ensure", running="running", stopped="stopped" - ) - - def add_users_snapshot(self, snap: Dict[str, Any]) -> None: - records = self.user_records_from_snapshot(snap) - self.groups.update(self.user_group_names_from_records(records)) - for record in records: - name = str(record.get("name") or "") - self.users[name] = { - "name": name, - "uid": record.get("uid"), - "gid": record.get("gid"), - "primary_group": record.get("primary_group") or None, - "home": record.get("home"), - "shell": record.get("shell"), - "gecos": record.get("gecos"), - "supplementary_groups": record.get("supplementary_groups") or [], - } - - self.add_user_flatpaks_snapshot(snap) - - def prepare_flatpak_remote(self, item: Dict[str, Any]) -> Dict[str, Any]: - return _prepare_flatpak_remote(item) - - def prepare_flatpak_item(self, item: Dict[str, Any]) -> Dict[str, Any]: - return _prepare_flatpak_item(item) - - def prepare_snap_item(self, item: Dict[str, Any]) -> Dict[str, Any]: - return _prepare_snap_item(item) - - def add_firewall_runtime_snapshot( - self, - snap: Dict[str, Any], - *, - bundle_dir: str, - artifact_role: str, - module_files_dir: Path, - file_prefix: Optional[str] = None, - ) -> None: - super().add_firewall_runtime_snapshot( - snap, - bundle_dir=bundle_dir, - artifact_role=artifact_role, - files_dir=module_files_dir, - copy_artifact=_copy_artifact, - source_uri=_source_uri, - file_prefix=file_prefix, - dir_attrs={"require": "File['/etc/enroll']"}, - ) - - def add_container_images_snapshot(self, snap: Dict[str, Any]) -> None: - for raw in snap.get("images", []) or []: - if not isinstance(raw, dict): - continue - engine = str(raw.get("engine") or "").strip().lower() - pull_ref = str(raw.get("pull_ref") or "").strip() - if engine not in {"docker", "podman"}: - continue - if not pull_ref: - tags = ", ".join(str(t) for t in (raw.get("repo_tags") or []) if t) - label = tags or str(raw.get("image_id") or "unknown image") - self.notes.append( - f"Container image {label} has no RepoDigest; exact Puppet pull resource was not rendered." - ) - continue - item = dict(raw) - item["engine"] = engine - item["pull_ref"] = pull_ref - item["scope"] = str(item.get("scope") or "system").strip() or "system" - image_name, image_digest = _split_digest_ref(pull_ref) - item["image"] = image_name - item["image_digest"] = image_digest - item["tag_aliases"] = [ - dict(alias) - for alias in (item.get("tag_aliases") or []) - if isinstance(alias, dict) and alias.get("ref") - ] - item["pull_cmd"] = _container_pull_cmd(engine, pull_ref) - item["pull_unless"] = _container_exists_cmd(engine, pull_ref) - for alias in item["tag_aliases"]: - alias_ref = str(alias.get("ref") or "") - alias["tag_cmd"] = _container_tag_cmd(engine, pull_ref, alias_ref) - alias["tag_unless"] = _container_exists_cmd(engine, alias_ref) - self.container_images.append(item) - for note in snap.get("notes", []) or []: - self.notes.append(str(note)) - - def add_managed_content( - self, - snap: Dict[str, Any], - *, - bundle_dir: str, - artifact_role: str, - module_files_dir: Path, - module_templates_dir: Optional[Path] = None, - file_prefix: Optional[str] = None, - notify_services: Optional[List[str]] = None, - jt_exe: Optional[str] = None, - jt_enabled: bool = False, - overwrite_templates: bool = True, - ) -> None: - for d in self.managed_dirs_from_snapshot(snap): - path = str(d.get("path") or "").strip() - self.add_managed_dir( - path, - owner=d.get("owner") or "root", - group=d.get("group") or "root", - mode=d.get("mode") or "0755", - reason=d.get("reason") or "managed_dir", - ) - - managed_files = list(self.managed_files_from_snapshot(snap)) - candidates = [ - mf - for mf in managed_files - if str(mf.get("path") or "") - and str(mf.get("src_rel") or "") - and can_jinjify_path(str(mf.get("path") or "")) - ] - namespace_by_file = len(candidates) > 1 - - for mf in managed_files: - path = str(mf.get("path") or "").strip() - src_rel = str(mf.get("src_rel") or "").strip() - if not path or not src_rel: - continue - - template_rel: Optional[str] = None - if module_templates_dir is not None: - role_prefix = ( - managed_file_var_prefix(self.module_name, src_rel) - if namespace_by_file - else self.module_name - ) - converted = jinjify_artifact( - bundle_dir, - artifact_role, - src_rel, - path, - module_templates_dir, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=overwrite_templates, - role_name=role_prefix, - template_engine="erb", - puppet_class=self.module_name, - ) - if converted is not None: - template_rel = converted.template_rel - self.template_hiera.update(converted.context) - - attrs: Dict[str, Any] = { - "owner": mf.get("owner") or "root", - "group": mf.get("group") or "root", - "mode": mf.get("mode") or "0644", - "reason": mf.get("reason") or "managed_file", - } - if template_rel is not None: - attrs["template"] = f"{self.module_name}/{template_rel}" - else: - module_rel = _copy_artifact( - bundle_dir, - artifact_role, - src_rel, - module_files_dir, - dst_prefix=file_prefix, - ) - if not module_rel: - self.notes.append( - f"Skipped {path}: harvested artifact {artifact_role}/{src_rel} was not present." - ) - continue - attrs["source"] = _source_uri(self.module_name, module_rel) - if notify_services and not path.startswith("/etc/systemd/system/"): - notify_units = [unit for unit in notify_services if str(unit).strip()] - notify_value = _service_notify_value(notify_units) - if notify_value: - attrs["notify"] = notify_value - attrs["notify_services"] = notify_units - attrs["_notify_services"] = notify_units - self.add_managed_file(path, **attrs) - - for ml in self.managed_links_from_snapshot(snap): - path = str(ml.get("path") or "").strip() - target = str(ml.get("target") or "").strip() - if not path or not target: - continue - self.add_managed_link( - path, - target=target, - reason=ml.get("reason") or "managed_link", - ) - - self.remove_directory_resource_conflicts() - - -# https://help.puppet.com/core/current/Content/PuppetCore/lang_reserved_words.htm -_RESERVED_PUPPET_NAMES = { - "and", - "application", - "attr", - "case", - "component", - "consumes", - "default", - "define", - "elsif", - "environment", - "false", - "function", - "if", - "import", - "in", - "init", - "inherits", - "node", - "or", - "private", - "produces", - "regexp", - "site", - "true", - "type", - "undef", - "unit", - "unless", -} - - -def _puppet_name(raw: str, *, fallback: str = "role") -> str: - s = re.sub(r"[^A-Za-z0-9_]+", "_", raw or fallback) - s = re.sub(r"_+", "_", s).strip("_").lower() - if not s: - s = fallback - if not re.match(r"^[a-z]", s): - s = f"{fallback}_{s}" - if s in _RESERVED_PUPPET_NAMES: - s = f"{fallback}_{s}" - return s - - -# Control characters (C0 range plus DEL) that should never appear raw inside a -# generated Puppet manifest scalar. They cannot occur in values harvested from a -# live host (e.g. /etc/passwd GECOS is newline-delimited), so their presence -# indicates a hand-edited or tampered harvest. Emitting them verbatim is valid -# Puppet but produces multi-line / control-laden manifests; normalise them into -# explicit escapes instead. -_PP_CONTROL_CHARS = frozenset(chr(c) for c in range(0x20)) | {"\x7f"} - -# Puppet double-quoted recognised single-character escapes. -_PP_DQ_ESCAPES = { - "\n": "\\n", - "\t": "\\t", - "\r": "\\r", - "\\": "\\\\", - '"': '\\"', - "$": "\\$", -} - - -def _pp_quote_double(s: str) -> str: - """Render a Puppet double-quoted string with control characters escaped. - - Only used as a fallback when a value contains raw control characters, so the - common case stays single-quoted and byte-identical to historical output. - """ - - out = [] - for ch in s: - esc = _PP_DQ_ESCAPES.get(ch) - if esc is not None: - out.append(esc) - elif ch in _PP_CONTROL_CHARS: - # Puppet supports \uXXXX style escapes inside double-quoted strings. - out.append(f"\\u{{{ord(ch):04x}}}") - else: - out.append(ch) - return '"' + "".join(out) + '"' - - -def _pp_quote(value: Any) -> str: - s = str(value) - # Puppet single-quoted strings only honour \\ and \' escapes; everything - # else (including a literal newline) is taken verbatim. That is safe but lets - # a tampered harvest splatter raw control characters across the manifest. - # When any are present, fall back to a double-quoted string where they can be - # neutralised into explicit escapes. - if any(ch in _PP_CONTROL_CHARS for ch in s): - return _pp_quote_double(s) - s = s.replace("\\", "\\\\").replace("'", "\\'") - return f"'{s}'" - - -def _pp_bool(value: bool) -> str: - return "true" if bool(value) else "false" - - -def _shell_quote(value: Any) -> str: - return shlex.quote(str(value or "")) - - -def _split_digest_ref(value: Any) -> Tuple[str, Optional[str]]: - text = str(value or "").strip() - if "@" not in text: - return text, None - image, digest = text.split("@", 1) - return image, digest - - -def _container_pull_cmd(engine: str, pull_ref: str) -> str: - return f"{engine} pull {_shell_quote(pull_ref)}" - - -def _container_exists_cmd(engine: str, ref: str) -> str: - if engine == "podman": - return f"podman image exists {_shell_quote(ref)}" - return f"docker image inspect {_shell_quote(ref)} >/dev/null 2>&1" - - -def _container_tag_cmd(engine: str, pull_ref: str, tag_ref: str) -> str: - return f"{engine} tag {_shell_quote(pull_ref)} {_shell_quote(tag_ref)}" - - -def _flatpak_scope(item: Dict[str, Any]) -> str: - return "--user" if str(item.get("method") or "system") == "user" else "--system" - - -def _flatpak_home(item: Dict[str, Any]) -> Optional[str]: - user = str(item.get("user") or "").strip() - if not user: - return None - return str(item.get("home") or f"/home/{user}") - - -def _flatpak_exec_env(item: Dict[str, Any]) -> List[str]: - home = _flatpak_home(item) - if not home: - return [] - return [f"HOME={home}", f"XDG_DATA_HOME={home}/.local/share"] - - -def _flatpak_remote_exists_cmd(item: Dict[str, Any]) -> str: - return ( - f"flatpak {_flatpak_scope(item)} remote-list --columns=name " - f"| grep -Fx -- {_shell_quote(item.get('name'))}" - ) - - -def _flatpak_remote_add_cmd(item: Dict[str, Any]) -> str: - return ( - f"flatpak {_flatpak_scope(item)} remote-add --if-not-exists " - f"{_shell_quote(item.get('name'))} {_shell_quote(item.get('url'))}" - ) - - -def _flatpak_ref(item: Dict[str, Any]) -> str: - ref = str(item.get("ref") or "").strip() - if ref: - return ref - return str(item.get("name") or "").strip() - - -def _flatpak_exists_cmd(item: Dict[str, Any]) -> str: - return f"flatpak {_flatpak_scope(item)} info {_shell_quote(_flatpak_ref(item))} >/dev/null 2>&1" - - -def _flatpak_install_cmd(item: Dict[str, Any]) -> str: - args = ["flatpak", _flatpak_scope(item), "install", "-y"] - remote = str(item.get("remote") or "").strip() - if remote: - args.append(remote) - args.append(_flatpak_ref(item)) - return " ".join(_shell_quote(arg) for arg in args) - - -def _prepare_flatpak_remote(item: Dict[str, Any]) -> Dict[str, Any]: - out = dict(item) - method = str(out.get("method") or "system") - user = str(out.get("user") or "") - name = str(out.get("name") or "") - out["state_id"] = _state_title("flatpak-remote", f"{method}-{user}-{name}") - out["add_cmd"] = _flatpak_remote_add_cmd(out) - out["exists_cmd"] = _flatpak_remote_exists_cmd(out) - out["environment"] = _flatpak_exec_env(out) - return out - - -def _prepare_flatpak_item(item: Dict[str, Any]) -> Dict[str, Any]: - out = dict(item) - method = str(out.get("method") or "system") - user = str(out.get("user") or "") - ref = _flatpak_ref(out) - out["state_id"] = _state_title("flatpak", f"{method}-{user}-{ref}") - out["install_cmd"] = _flatpak_install_cmd(out) - out["exists_cmd"] = _flatpak_exists_cmd(out) - out["environment"] = _flatpak_exec_env(out) - return out - - -def _snap_exists_cmd(item: Dict[str, Any]) -> str: - return f"snap list {_shell_quote(item.get('name'))} >/dev/null 2>&1" - - -def _snap_install_cmd(item: Dict[str, Any]) -> str: - args = ["snap", "install", str(item.get("name") or "")] - channel = str(item.get("channel") or "").strip() - revision = str(item.get("revision") or "").strip() - if channel: - args.append(f"--channel={channel}") - elif revision: - args.append(f"--revision={revision}") - if item.get("classic"): - args.append("--classic") - if item.get("devmode"): - args.append("--devmode") - if item.get("dangerous"): - args.append("--dangerous") - return " ".join(_shell_quote(arg) for arg in args if str(arg)) - - -def _prepare_snap_item(item: Dict[str, Any]) -> Dict[str, Any]: - out = dict(item) - name = str(out.get("name") or "") - out["state_id"] = _state_title("snap", name) - out["install_cmd"] = _snap_install_cmd(out) - out["exists_cmd"] = _snap_exists_cmd(out) - return out - - -def _pp_array(values: Iterable[Any]) -> str: - return "[" + ", ".join(_pp_quote(v) for v in values) + "]" - - -def _pp_value(value: Any) -> str: - """Render a conservative Puppet literal for generated class defaults.""" - - if value is None: - return "undef" - if isinstance(value, bool): - return _pp_bool(value) - if isinstance(value, int) and not isinstance(value, bool): - return str(value) - if isinstance(value, float): - return repr(value) - if isinstance(value, list): - return "[" + ", ".join(_pp_value(v) for v in value) + "]" - if isinstance(value, dict): - parts = [] - for key in sorted(value, key=lambda k: str(k)): - parts.append(f"{_pp_quote(key)} => {_pp_value(value[key])}") - return "{" + ", ".join(parts) + "}" - return _pp_quote(value) - - -def _template_param_defaults(prole: PuppetRole) -> Dict[str, Any]: - prefix = f"{prole.module_name}::" - out: Dict[str, Any] = {} - for key, value in prole.template_hiera.items(): - key_s = str(key) - if key_s.startswith(prefix): - local = key_s[len(prefix) :] - elif "::" in key_s: - local = key_s.split("::", 1)[1] - else: - local = key_s - if local: - out[local] = value - return out - - -def _puppet_exec_attrs( - command: str, - unless: str, - *, - item: Optional[Dict[str, Any]] = None, - require: Optional[str] = None, -) -> List[Tuple[str, str]]: - attrs: List[Tuple[str, str]] = [ - ("command", _pp_quote(command)), - ("unless", _pp_quote(unless)), - ("path", "['/usr/bin', '/bin']"), - ] - if item: - user = str(item.get("user") or "").strip() - if user: - attrs.append(("user", _pp_quote(user))) - env = item.get("environment") or _flatpak_exec_env(item) - if env: - attrs.append(("environment", _pp_array(env))) - if require: - attrs.append(("require", require)) - return attrs - - -def _resource( - lines: List[str], rtype: str, title: str, attrs: List[Tuple[str, str]] -) -> None: - lines.append(f" {rtype} {{ {_pp_quote(title)}:") - for key, value in attrs: - lines.append(f" {key} => {value},") - lines.append(" }") - lines.append("") - - -def _state_title(prefix: str, value: Any) -> str: - safe = re.sub(r"[^A-Za-z0-9_.-]+", "-", str(value or "item")).strip("-._") - if not safe: - safe = "item" - if len(safe) > 64: - digest = hashlib.sha1( - str(value).encode("utf-8", errors="replace") - ).hexdigest()[ # nosec B324 - :8 - ] - safe = safe[:48] + "-" + digest - return f"enroll-{prefix}-{safe}" - - -def _render_firewall_runtime_execs( - lines: List[str], runtime: Dict[str, Any], *, indent: str = " " -) -> None: - specs = [ - ( - "ipset", - "ipset_save", - "ipset_restore_cmd", - "enroll-firewall-runtime-ipset-restore", - ), - ( - "iptables_v4", - "iptables_v4_save", - "iptables_v4_restore_cmd", - "enroll-firewall-runtime-iptables-v4-restore", - ), - ( - "iptables_v6", - "iptables_v6_save", - "iptables_v6_restore_cmd", - "enroll-firewall-runtime-iptables-v6-restore", - ), - ] - for _family, path_key, cmd_key, title in specs: - path = str(runtime.get(path_key) or "") - command = str(runtime.get(cmd_key) or "") - if not path or not command: - continue - attrs: List[Tuple[str, str]] = [ - ("command", _pp_quote(command)), - ("path", "['/sbin', '/usr/sbin', '/bin', '/usr/bin']"), - ("refreshonly", "true"), - ("subscribe", f"File[{_pp_quote(path)}]"), - ] - lines.append(f"{indent}exec {{ {_pp_quote(title)}:") - for key, value in attrs: - lines.append(f"{indent} {key} => {value},") - lines.append(f"{indent}}}") - lines.append("") - - -def _active_service_snapshots_by_unit( - entries: Iterable[Dict[str, Any]], -) -> Dict[str, Dict[str, Any]]: - """Return active service snapshots keyed by systemd unit name.""" - - by_unit: Dict[str, Dict[str, Any]] = {} - for entry in entries: - if str(entry.get("kind") or "package") != "service": - continue - snap = entry.get("snapshot") or {} - if not isinstance(snap, dict): - continue - unit = str(snap.get("unit") or "").strip() - if not unit or str(snap.get("active_state") or "") != "active": - continue - by_unit.setdefault(unit, snap) - return by_unit - - -def _service_notify_value(units: Iterable[str]) -> Optional[str]: - refs = [f"Service[{_pp_quote(unit)}]" for unit in units if str(unit).strip()] - if not refs: - return None - return refs[0] if len(refs) == 1 else f"[{', '.join(refs)}]" - - -def _sync_service_notifications(puppet_roles: Iterable[PuppetRole]) -> None: - """Remove generated service notifications that do not target this catalog.""" - - roles = list(puppet_roles) - declared_services = {unit for role in roles for unit in role.services} - for role in roles: - for path, attrs in role.files.items(): - notify_units = [ - str(unit).strip() - for unit in (attrs.get("_notify_services") or []) - if str(unit).strip() - ] - if not notify_units: - attrs.pop("_notify_services", None) - continue - kept = [unit for unit in notify_units if unit in declared_services] - missing = sorted(set(notify_units) - set(kept)) - if missing: - role.notes.append( - "Skipped service notification for " - f"{path}: no generated Service resource for " - f"{', '.join(missing)}." - ) - notify_value = _service_notify_value(kept) - if notify_value: - attrs["notify"] = notify_value - attrs["notify_services"] = kept - else: - attrs.pop("notify", None) - attrs.pop("notify_services", None) - attrs.pop("_notify_services", None) - - -def _copy_artifact( - bundle_dir: str, - role: str, - src_rel: str, - dst_files_dir: Path, - *, - dst_prefix: Optional[str] = None, -) -> Optional[str]: - if not role or not src_rel: - return None - try: - src = safe_artifact_file(bundle_dir, role, src_rel) - except FileNotFoundError: - return None - module_rel = Path(dst_prefix or "") / src_rel - dst = dst_files_dir / module_rel - dst.parent.mkdir(parents=True, exist_ok=True) - copy_safe_artifact_file(src, dst) - return module_rel.as_posix() - - -def _source_uri(module_name: str, module_rel: str) -> str: - return f"puppet:///modules/{module_name}/{module_rel}" - - -def _node_data_filename(fqdn: str) -> str: - """Return a safe Hiera node-data filename for an FQDN/certname.""" - - name = str(fqdn or "").strip().replace("/", "_").replace("\\", "_") - return f"{name or 'node'}.yaml" - - -def _node_file_prefix(fqdn: str) -> str: - """Return a safe module-files prefix for node-specific artifacts.""" - - name = re.sub(r"[^A-Za-z0-9_.-]+", "_", str(fqdn or "").strip()) - name = name.strip("._-") or "node" - return f"nodes/{name}" - - -def _collect_puppet_roles( - state: Dict[str, Any], - bundle_dir: str, - modules_dir: Path, - *, - fqdn: Optional[str] = None, - no_common_roles: bool = False, - jt_exe: Optional[str] = None, - jt_enabled: bool = False, -) -> List[PuppetRole]: - roles = roles_from_state(state) - inventory_packages = inventory_packages_from_state(state) - use_common_modules = not fqdn and not no_common_roles - node_file_prefix = _node_file_prefix(fqdn) if fqdn else None - out: Dict[str, PuppetRole] = {} - - def ensure_role(role_name: str) -> PuppetRole: - role_name = _puppet_name(role_name, fallback="enroll_role") - return out.setdefault(role_name, PuppetRole(role_name)) - - for key in ( - "apt_config", - "dnf_config", - "etc_custom", - "usr_local_custom", - "extra_paths", - "sysctl", - ): - snap = roles.get(key) or {} - if not isinstance(snap, dict): - continue - role_name = _puppet_name( - str(snap.get("role_name") or key), fallback="enroll_role" - ) - prole = ensure_role(role_name) - module_dir = modules_dir / prole.module_name - module_files_dir = module_dir / "files" - prole.add_managed_content( - snap, - bundle_dir=bundle_dir, - artifact_role=str(snap.get("role_name") or key), - module_files_dir=module_files_dir, - module_templates_dir=module_dir / "templates", - file_prefix=node_file_prefix, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=not bool(fqdn), - ) - - users_snap = roles.get("users") or {} - if isinstance(users_snap, dict): - role_name = _puppet_name( - str(users_snap.get("role_name") or "users"), fallback="enroll_role" - ) - prole = ensure_role(role_name) - prole.add_users_snapshot(users_snap) - module_dir = modules_dir / prole.module_name - prole.add_managed_content( - users_snap, - bundle_dir=bundle_dir, - artifact_role=str(users_snap.get("role_name") or "users"), - module_files_dir=module_dir / "files", - module_templates_dir=module_dir / "templates", - file_prefix=node_file_prefix, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=not bool(fqdn), - ) - - package_service_entries = list( - CMModule.package_service_entries( - roles, inventory_packages, use_common_roles=use_common_modules - ) - ) - service_units_by_package = CMModule.active_service_units_by_package( - package_service_entries - ) - service_snapshots_by_unit = _active_service_snapshots_by_unit( - package_service_entries - ) - - for entry in package_service_entries: - snap = entry.get("snapshot") or {} - kind = str(entry.get("kind") or "package") - fallback = "service" if kind == "service" else "package" - source_label = str( - snap.get("role_name") or snap.get("unit") or snap.get("package") or fallback - ) - original_role_name = _puppet_name(source_label, fallback=fallback) - role_name = _puppet_name( - str(entry.get("role_label") or source_label), - fallback="package_group" if use_common_modules else fallback, - ) - prole = ensure_role(role_name) - notify_services: List[str] = [] - if kind == "service": - prole.add_service_snapshot(snap) - unit = str(snap.get("unit") or "").strip() - if unit and str(snap.get("active_state") or "") == "active": - notify_services = [unit] - else: - prole.add_package_snapshot(snap) - notify_services = CMModule.active_service_units_for_package_snapshot( - snap, service_units_by_package - ) - for unit in notify_services: - service_snap = service_snapshots_by_unit.get(unit) - if service_snap is not None: - prole.add_service_snapshot(service_snap) - module_dir = modules_dir / prole.module_name - prole.add_managed_content( - snap, - bundle_dir=bundle_dir, - artifact_role=str(snap.get("role_name") or original_role_name), - module_files_dir=module_dir / "files", - module_templates_dir=module_dir / "templates", - file_prefix=node_file_prefix, - notify_services=notify_services, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=not bool(fqdn), - ) - - container_images = roles.get("container_images") or {} - if isinstance(container_images, dict) and ( - container_images.get("images") or container_images.get("notes") - ): - prole = ensure_role( - str(container_images.get("role_name") or "container_images") - ) - prole.add_container_images_snapshot(container_images) - - fw = roles.get("firewall_runtime") or {} - if isinstance(fw, dict): - has_fw = ( - fw.get("ipset_save") - or fw.get("iptables_v4_save") - or fw.get("iptables_v6_save") - ) - if has_fw: - runtime_role = ensure_role("enroll_runtime") - runtime_role.add_managed_dir( - "/etc/enroll", - owner="root", - group="root", - mode="0750", - reason="enroll_runtime", - ) - role_name = str(fw.get("role_name") or "firewall_runtime") - prole = ensure_role(role_name) - prole.add_firewall_runtime_snapshot( - fw, - bundle_dir=bundle_dir, - artifact_role=role_name, - module_files_dir=modules_dir / prole.module_name / "files", - file_prefix=node_file_prefix, - ) - - flatpak = roles.get("flatpak") or {} - if isinstance(flatpak, dict) and ( - flatpak.get("system_flatpaks") or flatpak.get("remotes") or flatpak.get("notes") - ): - prole = ensure_role(str(flatpak.get("role_name") or "flatpak")) - prole.add_flatpak_snapshot(flatpak) - - snap = roles.get("snap") or {} - if isinstance(snap, dict) and (snap.get("system_snaps") or snap.get("notes")): - prole = ensure_role(str(snap.get("role_name") or "snap")) - prole.add_snap_snapshot(snap) - - puppet_roles = sorted(out.values(), key=lambda r: role_order_key(r.role_name)) - resolve_catalog_conflicts(puppet_roles) - _sync_service_notifications(puppet_roles) - return [r for r in puppet_roles if r.has_resources()] - - -def _render_role_class(prole: PuppetRole) -> str: - has_sysctl_conf = "/etc/sysctl.d/99-enroll.conf" in prole.files - template_defaults = _template_param_defaults(prole) - params: List[str] = [] - if has_sysctl_conf: - params.extend( - [ - " Boolean $sysctl_apply = true,", - " Boolean $sysctl_ignore_apply_errors = true,", - ] - ) - for name, value in sorted(template_defaults.items()): - params.append(f" Any ${name} = {_pp_value(value)},") - - if params: - lines: List[str] = [ - "# Generated by Enroll from harvest state.", - f"class {prole.module_name} (", - *params, - ") {", - "", - ] - else: - lines = [ - "# Generated by Enroll from harvest state.", - f"class {prole.module_name} {{", - "", - ] - - for package in sorted(prole.packages): - _resource(lines, "package", package, [("ensure", _pp_quote("installed"))]) - - for group in sorted(prole.groups): - _resource(lines, "group", group, [("ensure", _pp_quote("present"))]) - - for user in [prole.users[k] for k in sorted(prole.users)]: - attrs: List[Tuple[str, str]] = [ - ("ensure", _pp_quote("present")), - ("managehome", _pp_bool(True)), - ] - if user.get("uid") is not None: - attrs.append(("uid", _pp_quote(user["uid"]))) - if user.get("primary_group"): - attrs.append(("gid", _pp_quote(user["primary_group"]))) - if user.get("home"): - attrs.append(("home", _pp_quote(user["home"]))) - if user.get("shell"): - attrs.append(("shell", _pp_quote(user["shell"]))) - if user.get("gecos"): - attrs.append(("comment", _pp_quote(user["gecos"]))) - if user.get("supplementary_groups"): - attrs.append(("groups", _pp_array(user["supplementary_groups"]))) - attrs.append(("membership", _pp_quote("minimum"))) - _resource(lines, "user", user["name"], attrs) - - for path, d in sorted(prole.dirs.items()): - _resource( - lines, - "file", - path, - [ - ("ensure", _pp_quote("directory")), - ("owner", _pp_quote(d.get("owner") or "root")), - ("group", _pp_quote(d.get("group") or "root")), - ("mode", _pp_quote(d.get("mode") or "0755")), - *([("require", str(d.get("require")))] if d.get("require") else []), - ], - ) - - for path, f in sorted(prole.files.items()): - file_attrs: List[Tuple[str, str]] = [("ensure", _pp_quote("file"))] - if f.get("template"): - file_attrs.append(("content", f"template({_pp_quote(f.get('template'))})")) - else: - file_attrs.append(("source", _pp_quote(f.get("source") or ""))) - file_attrs.extend( - [ - ("owner", _pp_quote(f.get("owner") or "root")), - ("group", _pp_quote(f.get("group") or "root")), - ("mode", _pp_quote(f.get("mode") or "0644")), - *([("notify", str(f.get("notify")))] if f.get("notify") else []), - ] - ) - _resource(lines, "file", path, file_attrs) - - for path, lnk in sorted(prole.links.items()): - _resource( - lines, - "file", - path, - [ - ("ensure", _pp_quote("link")), - ("target", _pp_quote(lnk.get("target") or "")), - ], - ) - - for svc in [prole.services[k] for k in sorted(prole.services)]: - _resource( - lines, - "service", - svc["name"], - [ - ("ensure", _pp_quote(svc["ensure"])), - ("enable", _pp_bool(bool(svc["enable"]))), - ], - ) - - flatpak_remote_titles: Dict[Tuple[str, str, str], str] = {} - for remote in prole.flatpak_remotes: - name = str(remote.get("name") or "").strip() - url = str(remote.get("url") or "").strip() - if not name or not url: - continue - title = str(remote.get("state_id") or _state_title("flatpak-remote", name)) - key = ( - str(remote.get("method") or "system"), - str(remote.get("user") or ""), - name, - ) - flatpak_remote_titles[key] = title - remote_user = str(remote.get("user") or "").strip() - remote_require = None - if remote_user and remote_user in prole.users: - remote_require = f"User[{_pp_quote(remote_user)}]" - _resource( - lines, - "exec", - title, - _puppet_exec_attrs( - str(remote.get("add_cmd") or _flatpak_remote_add_cmd(remote)), - str(remote.get("exists_cmd") or _flatpak_remote_exists_cmd(remote)), - item=remote, - require=remote_require, - ), - ) - - for app in prole.flatpaks: - ref = _flatpak_ref(app) - if not ref: - continue - title = str(app.get("state_id") or _state_title("flatpak", ref)) - requires: List[str] = [] - user = str(app.get("user") or "").strip() - if user: - requires.append(f"User[{_pp_quote(user)}]") - remote = str(app.get("remote") or "").strip() - if remote: - remote_title = flatpak_remote_titles.get( - (str(app.get("method") or "system"), user, remote) - ) - if remote_title: - requires.append(f"Exec[{_pp_quote(remote_title)}]") - require_expr = None - if len(requires) == 1: - require_expr = requires[0] - elif requires: - require_expr = "[" + ", ".join(requires) + "]" - _resource( - lines, - "exec", - title, - _puppet_exec_attrs( - str(app.get("install_cmd") or _flatpak_install_cmd(app)), - str(app.get("exists_cmd") or _flatpak_exists_cmd(app)), - item=app, - require=require_expr, - ), - ) - - for snap in prole.snaps: - name = str(snap.get("name") or "").strip() - if not name: - continue - _resource( - lines, - "exec", - str(snap.get("state_id") or _state_title("snap", name)), - _puppet_exec_attrs( - str(snap.get("install_cmd") or _snap_install_cmd(snap)), - str(snap.get("exists_cmd") or _snap_exists_cmd(snap)), - ), - ) - - for image in prole.container_images: - engine = str(image.get("engine") or "").strip() - pull_ref = str(image.get("pull_ref") or "").strip() - if not engine or not pull_ref: - continue - if engine == "docker": - pull_title = _state_title("docker-pull", pull_ref) - _resource( - lines, - "exec", - pull_title, - [ - ( - "command", - _pp_quote( - image.get("pull_cmd") - or _container_pull_cmd(engine, pull_ref) - ), - ), - ( - "unless", - _pp_quote( - image.get("pull_unless") - or _container_exists_cmd(engine, pull_ref) - ), - ), - ("path", "['/usr/bin', '/bin']"), - ], - ) - for alias in image.get("tag_aliases") or []: - tag_ref = str(alias.get("ref") or "").strip() - if not tag_ref: - continue - _resource( - lines, - "exec", - _state_title("docker-tag", tag_ref), - [ - ( - "command", - _pp_quote( - alias.get("tag_cmd") - or _container_tag_cmd(engine, pull_ref, tag_ref) - ), - ), - ( - "unless", - _pp_quote( - alias.get("tag_unless") - or _container_exists_cmd(engine, tag_ref) - ), - ), - ("path", "['/usr/bin', '/bin']"), - ("require", f"Exec[{_pp_quote(pull_title)}]"), - ], - ) - elif engine == "podman": - _resource( - lines, - "exec", - _state_title("podman-pull", pull_ref), - [ - ( - "command", - _pp_quote( - image.get("pull_cmd") - or _container_pull_cmd(engine, pull_ref) - ), - ), - ( - "unless", - _pp_quote( - image.get("pull_unless") - or _container_exists_cmd(engine, pull_ref) - ), - ), - ("path", "['/usr/bin', '/bin']"), - ], - ) - for alias in image.get("tag_aliases") or []: - tag_ref = str(alias.get("ref") or "").strip() - if not tag_ref: - continue - _resource( - lines, - "exec", - _state_title("podman-tag", tag_ref), - [ - ( - "command", - _pp_quote( - alias.get("tag_cmd") - or _container_tag_cmd(engine, pull_ref, tag_ref) - ), - ), - ( - "unless", - _pp_quote( - alias.get("tag_unless") - or _container_exists_cmd(engine, tag_ref) - ), - ), - ("path", "['/usr/bin', '/bin']"), - ( - "require", - f"Exec[{_pp_quote(_state_title('podman-pull', pull_ref))}]", - ), - ], - ) - - if prole.firewall_runtime: - _render_firewall_runtime_execs(lines, prole.firewall_runtime) - - if has_sysctl_conf: - lines.append(" if $sysctl_apply {") - lines.append(" exec { 'enroll-apply-sysctl':") - lines.append(" command => $sysctl_ignore_apply_errors ? {") - lines.append( - " true => \"/bin/sh -c 'sysctl -e -p /etc/sysctl.d/99-enroll.conf || true'\"," - ) - lines.append(" default => 'sysctl -e -p /etc/sysctl.d/99-enroll.conf',") - lines.append(" },") - lines.append(" path => ['/sbin', '/usr/sbin', '/bin', '/usr/bin'],") - lines.append(" refreshonly => true,") - lines.append(" subscribe => File['/etc/sysctl.d/99-enroll.conf'],") - lines.append(" }") - lines.append(" }") - lines.append("") - - if prole.notes: - lines.append(" # Notes and limitations") - for note in prole.notes: - lines.append(f" # - {note}") - lines.append("") - - lines.append("}") - lines.append("") - return "\n".join(lines) - - -def _attrs_with_ensure( - attrs: Dict[str, Any], ensure: str, *, allowed: Set[str] -) -> Dict[str, Any]: - """Return only Puppet resource attributes, dropping Enroll metadata.""" - out = {"ensure": ensure} - for key in sorted(allowed): - if key in attrs and attrs[key] is not None: - out[key] = attrs[key] - return out - - -def _role_hiera_values(prole: PuppetRole) -> Dict[str, Any]: - """Return Automatic Parameter Lookup data for one generated module.""" - - data: Dict[str, Any] = {} - prefix = f"{prole.module_name}::" - - if prole.packages: - data[f"{prefix}packages"] = sorted(prole.packages) - - if prole.groups: - data[f"{prefix}groups"] = { - group: {"ensure": "present"} for group in sorted(prole.groups) - } - - if prole.users: - users: Dict[str, Dict[str, Any]] = {} - for name in sorted(prole.users): - user = prole.users[name] - attrs: Dict[str, Any] = {"ensure": "present", "managehome": True} - if user.get("uid") is not None: - attrs["uid"] = user["uid"] - if user.get("primary_group"): - attrs["gid"] = user["primary_group"] - if user.get("home"): - attrs["home"] = user["home"] - if user.get("shell"): - attrs["shell"] = user["shell"] - if user.get("gecos"): - attrs["comment"] = user["gecos"] - if user.get("supplementary_groups"): - attrs["groups"] = list(user["supplementary_groups"]) - attrs["membership"] = "minimum" - users[name] = attrs - data[f"{prefix}users"] = users - - if prole.dirs: - data[f"{prefix}dirs"] = { - path: _attrs_with_ensure( - prole.dirs[path], - "directory", - allowed={"owner", "group", "mode", "require"}, - ) - for path in sorted(prole.dirs) - } - - if prole.files: - data[f"{prefix}files"] = { - path: _attrs_with_ensure( - prole.files[path], - "file", - allowed={ - "source", - "template", - "owner", - "group", - "mode", - "notify_services", - }, - ) - for path in sorted(prole.files) - } - - if prole.links: - data[f"{prefix}links"] = { - path: _attrs_with_ensure( - prole.links[path], - "link", - allowed={"target"}, - ) - for path in sorted(prole.links) - } - - if prole.services: - data[f"{prefix}services"] = { - name: { - "ensure": prole.services[name].get("ensure") or "stopped", - "enable": bool(prole.services[name].get("enable")), - } - for name in sorted(prole.services) - } - - if prole.flatpak_remotes: - data[f"{prefix}flatpak_remotes"] = list(prole.flatpak_remotes) - if prole.flatpaks: - data[f"{prefix}flatpaks"] = list(prole.flatpaks) - if prole.snaps: - data[f"{prefix}snaps"] = list(prole.snaps) - if prole.container_images: - data[f"{prefix}container_images"] = list(prole.container_images) - if prole.firewall_runtime: - data[f"{prefix}firewall_runtime"] = dict(prole.firewall_runtime) - - if prole.notes: - data[f"{prefix}notes"] = list(prole.notes) - - data.update(prole.template_hiera) - - if "/etc/sysctl.d/99-enroll.conf" in prole.files: - data[f"{prefix}sysctl_apply"] = True - data[f"{prefix}sysctl_ignore_apply_errors"] = True - - return data - - -def _render_hiera_role_class(prole: PuppetRole) -> str: - """Render a reusable, data-driven Puppet class for --fqdn/Hiera mode.""" - - lines: List[str] = [ - "# Generated by Enroll from harvest state.", - "# Resource data is supplied by Hiera Automatic Parameter Lookup.", - f"class {prole.module_name} (", - " Array[String] $packages = [],", - " Hash[String, Hash] $groups = {},", - " Hash[String, Hash] $users = {},", - " Hash[String, Hash] $dirs = {},", - " Hash[String, Hash] $files = {},", - " Hash[String, Hash] $links = {},", - " Hash[String, Hash] $services = {},", - " Array[Hash] $flatpak_remotes = [],", - " Array[Hash] $flatpaks = [],", - " Array[Hash] $snaps = [],", - " Array[Hash] $container_images = [],", - " Hash $firewall_runtime = {},", - " Array[String] $notes = [],", - " Boolean $sysctl_apply = true,", - " Boolean $sysctl_ignore_apply_errors = true,", - *[ - f" Any ${name} = undef," - for name in sorted(_template_param_defaults(prole)) - ], - ") {", - "", - " $packages.each |String $package_name| {", - " package { $package_name:", - " ensure => 'installed',", - " }", - " }", - "", - " $groups.each |String $resource_title, Hash $attrs| {", - " group { $resource_title:", - " * => $attrs,", - " }", - " }", - "", - " $users.each |String $resource_title, Hash $attrs| {", - " user { $resource_title:", - " * => $attrs,", - " }", - " }", - "", - " $dirs.each |String $resource_title, Hash $attrs| {", - " file { $resource_title:", - " * => $attrs,", - " }", - " }", - "", - " # Declare services before files so file notify relationships can", - " # resolve in Hiera-driven classes.", - " $services.each |String $resource_title, Hash $attrs| {", - " service { $resource_title:", - " * => $attrs,", - " }", - " }", - "", - " $files.each |String $resource_title, Hash $attrs| {", - " $file_attrs = $attrs.filter |$key, $value| {", - " $key != 'template' and $key != 'notify_services'", - " }", - " if $attrs['notify_services'] {", - " $notify_targets = $attrs['notify_services'].map |String $unit| { Service[$unit] }", - " if $attrs['template'] {", - " file { $resource_title:", - " * => $file_attrs,", - " content => template($attrs['template']),", - " notify => $notify_targets,", - " }", - " } else {", - " file { $resource_title:", - " * => $file_attrs,", - " notify => $notify_targets,", - " }", - " }", - " } elsif $attrs['template'] {", - " file { $resource_title:", - " * => $file_attrs,", - " content => template($attrs['template']),", - " }", - " } else {", - " file { $resource_title:", - " * => $file_attrs,", - " }", - " }", - " }", - "", - " $links.each |String $resource_title, Hash $attrs| {", - " file { $resource_title:", - " * => $attrs,", - " }", - " }", - "", - " $flatpak_remotes.each |Integer $idx, Hash $remote| {", - " exec { $remote['state_id']:", - " command => $remote['add_cmd'],", - " unless => $remote['exists_cmd'],", - " path => ['/usr/bin', '/bin'],", - " user => $remote['user'],", - " environment => $remote['environment'],", - " }", - " }", - "", - " $flatpaks.each |Integer $idx, Hash $app| {", - " exec { $app['state_id']:", - " command => $app['install_cmd'],", - " unless => $app['exists_cmd'],", - " path => ['/usr/bin', '/bin'],", - " user => $app['user'],", - " environment => $app['environment'],", - " }", - " }", - "", - " $snaps.each |Integer $idx, Hash $snap| {", - " exec { $snap['state_id']:", - " command => $snap['install_cmd'],", - " unless => $snap['exists_cmd'],", - " path => ['/usr/bin', '/bin'],", - " }", - " }", - "", - " $container_images.each |Integer $idx, Hash $image| {", - " if $image['engine'] == 'docker' and $image['pull_ref'] {", - ' exec { "enroll-docker-pull-${idx}":', - " command => $image['pull_cmd'],", - " unless => $image['pull_unless'],", - " path => ['/usr/bin', '/bin'],", - " }", - " $image['tag_aliases'].each |Integer $tag_idx, Hash $alias| {", - ' exec { "enroll-docker-tag-${idx}-${tag_idx}":', - " command => $alias['tag_cmd'],", - " unless => $alias['tag_unless'],", - " path => ['/usr/bin', '/bin'],", - ' require => Exec["enroll-docker-pull-${idx}"],', - " }", - " }", - " } elsif $image['engine'] == 'podman' and $image['pull_ref'] {", - ' exec { "enroll-podman-pull-${idx}":', - " command => $image['pull_cmd'],", - " unless => $image['pull_unless'],", - " path => ['/usr/bin', '/bin'],", - " }", - " $image['tag_aliases'].each |Integer $tag_idx, Hash $alias| {", - ' exec { "enroll-podman-tag-${idx}-${tag_idx}":', - " command => $alias['tag_cmd'],", - " unless => $alias['tag_unless'],", - " path => ['/usr/bin', '/bin'],", - ' require => Exec["enroll-podman-pull-${idx}"],', - " }", - " }", - " }", - " }", - "", - " if $firewall_runtime['ipset_restore_cmd'] {", - " exec { 'enroll-firewall-runtime-ipset-restore':", - " command => $firewall_runtime['ipset_restore_cmd'],", - " path => ['/sbin', '/usr/sbin', '/bin', '/usr/bin'],", - " refreshonly => true,", - " subscribe => File[$firewall_runtime['ipset_save']],", - " }", - " }", - "", - " if $firewall_runtime['iptables_v4_restore_cmd'] {", - " exec { 'enroll-firewall-runtime-iptables-v4-restore':", - " command => $firewall_runtime['iptables_v4_restore_cmd'],", - " path => ['/sbin', '/usr/sbin', '/bin', '/usr/bin'],", - " refreshonly => true,", - " subscribe => File[$firewall_runtime['iptables_v4_save']],", - " }", - " }", - "", - " if $firewall_runtime['iptables_v6_restore_cmd'] {", - " exec { 'enroll-firewall-runtime-iptables-v6-restore':", - " command => $firewall_runtime['iptables_v6_restore_cmd'],", - " path => ['/sbin', '/usr/sbin', '/bin', '/usr/bin'],", - " refreshonly => true,", - " subscribe => File[$firewall_runtime['iptables_v6_save']],", - " }", - " }", - "", - " if $sysctl_apply and '/etc/sysctl.d/99-enroll.conf' in $files {", - " exec { 'enroll-apply-sysctl':", - " command => $sysctl_ignore_apply_errors ? {", - " true => \"/bin/sh -c 'sysctl -e -p /etc/sysctl.d/99-enroll.conf || true'\",", - " default => 'sysctl -e -p /etc/sysctl.d/99-enroll.conf',", - " },", - " path => ['/sbin', '/usr/sbin', '/bin', '/usr/bin'],", - " refreshonly => true,", - " subscribe => File['/etc/sysctl.d/99-enroll.conf'],", - " }", - " }", - "", - " # Generated notes are supplied through the $notes parameter for review.", - "}", - "", - ] - return "\n".join(lines) - - -def _render_site_pp(puppet_roles: List[PuppetRole], fqdn: Optional[str]) -> str: - node_name = _pp_quote(fqdn) if fqdn else "default" - if not puppet_roles: - return f"node {node_name} {{\n # No Puppet classes were generated from this harvest.\n}}\n" - includes = "\n".join(f" include {r.module_name}" for r in puppet_roles) - return f"node {node_name} {{\n{includes}\n}}\n" - - -def _render_hiera_site_pp(node_names: List[str]) -> str: - lines: List[str] = [ - "# Generated by Enroll from harvest state.", - "# Per-node class lists and resources are read from Hiera data.", - "", - ] - for node_name in node_names: - lines.extend( - [ - f"node {_pp_quote(node_name)} {{", - " $enroll_classes = lookup('enroll::classes', Array[String], 'unique', [])", - " $enroll_classes.each |String $enroll_class| {", - " include $enroll_class", - " }", - "}", - "", - ] - ) - lines.extend( - [ - "node default {", - " $enroll_classes = lookup('enroll::classes', Array[String], 'unique', [])", - " $enroll_classes.each |String $enroll_class| {", - " include $enroll_class", - " }", - "}", - "", - ] - ) - return "\n".join(lines) - - -def _render_hiera_yaml() -> str: - data = { - "version": 5, - "defaults": {"datadir": "data", "data_hash": "yaml_data"}, - "hierarchy": [ - { - "name": "Enroll trusted certname node data", - "path": "nodes/%{trusted.certname}.yaml", - }, - { - "name": "Enroll networking FQDN node data", - "path": "nodes/%{facts.networking.fqdn}.yaml", - }, - {"name": "Enroll common data", "path": "common.yaml"}, - ], - } - return yaml.safe_dump(data, sort_keys=False, explicit_start=True) - - -def _write_yaml(path: Path, data: Dict[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text( - yaml.safe_dump( - puppet_hiera_safe_data(data), sort_keys=True, explicit_start=True - ), - encoding="utf-8", - ) - - -def _write_hiera_node_data( - out: Path, fqdn: str, puppet_roles: List[PuppetRole] -) -> Path: - node_data: Dict[str, Any] = { - "enroll::classes": [r.module_name for r in puppet_roles] - } - for prole in puppet_roles: - node_data.update(_role_hiera_values(prole)) - node_path = out / "data" / "nodes" / _node_data_filename(fqdn) - _write_yaml(node_path, node_data) - common_path = out / "data" / "common.yaml" - if not common_path.exists(): - _write_yaml(common_path, {"enroll::classes": []}) - return node_path - - -def _hiera_node_names(out: Path) -> List[str]: - nodes_dir = out / "data" / "nodes" - if not nodes_dir.is_dir(): - return [] - out_names: Set[str] = set() - for path in nodes_dir.glob("*.yaml"): - out_names.add(path.name[: -len(".yaml")]) - return sorted(out_names) - - -def _write_metadata(module_dir: Path, module_name: str, prole: PuppetRole) -> None: - dependencies: List[Dict[str, str]] = [] - - (module_dir / "metadata.json").write_text( - json.dumps( - { - "name": f"enroll-{module_name}", - "version": "0.1.0", - "author": "Enroll", - "summary": f"Generated Enroll Puppet module for {module_name}", - "license": "UNLICENSED", - "source": "", - "dependencies": dependencies, - }, - indent=2, - sort_keys=True, - ) - + "\n", - encoding="utf-8", - ) - - -def _render_readme( - state: Dict[str, Any], - puppet_roles: List[PuppetRole], - *, - fqdn: Optional[str] = None, - node_names: Optional[List[str]] = None, -) -> str: - host = state.get("host", {}) if isinstance(state.get("host"), dict) else {} - hostname = host.get("hostname") or "unknown" - hiera_mode = bool(fqdn) - role_lines = markdown_list( - f"`{r.module_name}` from Enroll role `{r.role_name}`" for r in puppet_roles - ) - node_lines = markdown_list(f"`{n}`" for n in (node_names or [])) - notes_text = markdown_list( - f"`{r.module_name}`: {note}" for r in puppet_roles for note in r.notes - ) - if hiera_mode: - layout = f"""- `manifests/site.pp` declares node blocks and includes classes listed in Hiera key `enroll::classes`. -- `hiera.yaml` configures per-node lookup from `data/nodes/%{{trusted.certname}}.yaml` with a fallback to `data/common.yaml`. -- `data/nodes/{_node_data_filename(fqdn or '')}` contains this node's class list and class parameter data. -- `modules//manifests/init.pp` contains reusable, data-driven classes. -- `modules//files/nodes//...` contains node-specific harvested raw file artifacts, avoiding clashes between hosts. -- `modules//templates/` contains ERB templates when JinjaTurtle can convert a harvested config file.""" - apply = f"""Run from this generated output directory, passing the node certname so Hiera selects the right node data: - -```bash -sudo puppet apply --modulepath ./modules --hiera_config ./hiera.yaml --certname {fqdn} manifests/site.pp --noop --test -``` - -If you depend on other pre-installed Puppet modules, you may need to pass in other modulepaths as well, e.g: - -```bash -sudo puppet apply --modulepath ./modules:/etc/puppet/code/modules --hiera_config ./hiera.yaml --certname {fqdn} manifests/site.pp --noop --test -``` - -For Puppet agent/control-repo use, place this output where `hiera.yaml`, `data/`, `manifests/`, and `modules/` form the environment root. Re-running Enroll with another `--fqdn` into the same output directory adds or replaces that node's YAML without deleting existing node data.""" - else: - layout = """- `manifests/site.pp` declares a `node` block and includes the generated classes in manifest order. -- `modules//manifests/init.pp` contains resources for each generated Enroll role/snapshot or common package group. -- `modules//files/` contains harvested raw file artifacts for that role or group. -- `modules//templates/` contains ERB templates when JinjaTurtle can convert a harvested config file. -- Generated module names avoid Puppet reserved words such as `default`.""" - apply = """Run from this generated output directory so Puppet can find `./modules`, or pass an absolute module path: - -```bash -sudo puppet apply --modulepath ./modules manifests/site.pp --noop --test -``` - -If you depend on other pre-installed Puppet modules, you may need to pass in other modulepaths as well, e.g: - -```bash -sudo puppet apply --modulepath ./modules:/etc/puppet/code/modules manifests/site.pp --noop --test -```""" - return f"""# Enroll Puppet manifest - -Generated by Enroll from harvest data for `{hostname}`. - -This Puppet target reuses the existing harvest state without changing harvesting behaviour. - -## Layout - -{layout} - -## Known nodes - -{node_lines if hiera_mode else '- Non-Hiera single-node output.'} - -## Generated modules - -{role_lines} - -## Apply / check - -{apply} - -## Generated resources - -- Native packages observed in package and service snapshots. -- Local users and groups from the users snapshot. -- Managed directories, files, and symlinks from harvested roles. -- Basic service enablement/running-state resources. -- `/etc/sysctl.d/99-enroll.conf` plus a refresh-only sysctl apply exec when present. -- Docker and Podman images by digest using guarded `exec` resources (`pull`/`tag` commands with `unless` checks). -- Podman images by digest using guarded `podman pull` / `podman tag` exec resources. - -## Current limitations - -- JinjaTurtle/ERB templating is best-effort. Files that JinjaTurtle cannot parse are copied as raw module files. -- Review generated resources before applying them broadly across unlike hosts. - -## Notes - -{notes_text} -""" - - -class PuppetManifestRenderer: - """Render Puppet modules and site manifest from a harvest bundle.""" - - def __init__( - self, - bundle_dir: str, - out_dir: str, - *, - fqdn: Optional[str] = None, - no_common_roles: bool = False, - jinjaturtle: str = "auto", - ) -> None: - self.bundle_dir = bundle_dir - self.out_dir = out_dir - self.fqdn = fqdn - self.no_common_roles = no_common_roles - self.jinjaturtle = jinjaturtle - - def render(self) -> None: - """Render Puppet modules/site.pp from a harvest bundle.""" - - bundle_dir = self.bundle_dir - out_dir = self.out_dir - fqdn = self.fqdn - no_common_roles = self.no_common_roles - - state = PuppetRole.load_state(bundle_dir) - hiera_mode = bool(fqdn) - out = prepare_manifest_output_dir(out_dir, allow_existing=hiera_mode) - manifests_dir = out / "manifests" - modules_dir = out / "modules" - manifests_dir.mkdir(parents=True, exist_ok=True) - modules_dir.mkdir(parents=True, exist_ok=True) - - jt_exe, jt_enabled = resolve_jinjaturtle_mode(self.jinjaturtle) - - puppet_roles = _collect_puppet_roles( - state, - bundle_dir, - modules_dir, - fqdn=fqdn, - no_common_roles=no_common_roles, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - ) - for prole in puppet_roles: - module_dir = modules_dir / prole.module_name - module_manifests = module_dir / "manifests" - module_files = module_dir / "files" - module_manifests.mkdir(parents=True, exist_ok=True) - module_files.mkdir(parents=True, exist_ok=True) - (module_manifests / "init.pp").write_text( - ( - _render_hiera_role_class(prole) - if hiera_mode - else _render_role_class(prole) - ), - encoding="utf-8", - ) - _write_metadata(module_dir, prole.module_name, prole) - - node_names: List[str] = [] - if hiera_mode and fqdn: - (out / "hiera.yaml").write_text(_render_hiera_yaml(), encoding="utf-8") - _write_hiera_node_data(out, fqdn, puppet_roles) - node_names = _hiera_node_names(out) - (manifests_dir / "site.pp").write_text( - _render_hiera_site_pp(node_names), encoding="utf-8" - ) - else: - (manifests_dir / "site.pp").write_text( - _render_site_pp(puppet_roles, fqdn), encoding="utf-8" - ) - (out / "README.md").write_text( - _render_readme( - state, - puppet_roles, - fqdn=fqdn, - node_names=node_names, - ), - encoding="utf-8", - ) - - -def manifest_from_bundle_dir( - bundle_dir: str, - out_dir: str, - *, - fqdn: Optional[str] = None, - no_common_roles: bool = False, - jinjaturtle: str = "auto", -) -> None: - PuppetManifestRenderer( - bundle_dir, - out_dir, - fqdn=fqdn, - no_common_roles=no_common_roles, - jinjaturtle=jinjaturtle, - ).render() diff --git a/enroll/render_safety.py b/enroll/render_safety.py index e8fc54a..0c13f64 100644 --- a/enroll/render_safety.py +++ b/enroll/render_safety.py @@ -1,7 +1,5 @@ from __future__ import annotations -import json -import re from collections.abc import Mapping, Set as AbstractSet from typing import Any @@ -50,183 +48,3 @@ def ansible_unsafe_data(value: Any) -> Any: if isinstance(value, AbstractSet): return sorted(ansible_unsafe_data(item) for item in value) return value - - -def escape_puppet_hiera_interpolation(value: str) -> str: - """Preserve literal ``%{`` text in Puppet Hiera data sources. - - Hiera treats ``%{...}`` in data values as interpolation. Enroll's Hiera - data is generated from harvested values, not authored Hiera expressions, so - any literal interpolation opener is escaped with Hiera's documented - ``literal('%')`` helper. - """ - - return str(value).replace("%{", "%{literal('%')}{") - - -def puppet_hiera_safe_data(value: Any) -> Any: - """Recursively escape Hiera interpolation openers in harvested data.""" - - if isinstance(value, Mapping): - return { - escape_puppet_hiera_interpolation(str(key)): puppet_hiera_safe_data(inner) - for key, inner in value.items() - } - if isinstance(value, list): - return [puppet_hiera_safe_data(item) for item in value] - if isinstance(value, tuple): - return [puppet_hiera_safe_data(item) for item in value] - if isinstance(value, AbstractSet): - return sorted(puppet_hiera_safe_data(item) for item in value) - if isinstance(value, str): - return escape_puppet_hiera_interpolation(value) - return value - - -def _plain_json_data(value: Any) -> Any: - if isinstance(value, Mapping): - return {str(key): _plain_json_data(inner) for key, inner in value.items()} - if isinstance(value, list): - return [_plain_json_data(item) for item in value] - if isinstance(value, tuple): - return [_plain_json_data(item) for item in value] - if isinstance(value, AbstractSet): - return sorted(_plain_json_data(item) for item in value) - return value - - -def _escape_braces_inside_json_strings(text: str) -> str: - """Replace literal braces only while scanning JSON string tokens.""" - - out: list[str] = [] - in_string = False - escaped = False - for ch in text: - if not in_string: - out.append(ch) - if ch == '"': - in_string = True - continue - - if escaped: - out.append(ch) - escaped = False - elif ch == "\\": - out.append(ch) - escaped = True - elif ch == '"': - out.append(ch) - in_string = False - elif ch == "{": - out.append("\\u007b") - elif ch == "}": - out.append("\\u007d") - else: - out.append(ch) - return "".join(out) - - -def salt_sls_json_quote(value: Any) -> str: - """Return a double-quoted YAML/JSON scalar safe for Salt's Jinja pass. - - Salt state and pillar SLS files normally use the ``jinja|yaml`` renderer - pipeline. YAML/JSON quoting alone does not stop ``{{ ... }}``, ``{% ... %}`` - or ``{# ... #}`` inside harvested values from being evaluated before YAML is - parsed. JSON/YAML double-quoted scalars decode ``\u007b`` and ``\u007d`` - after Jinja has run, so encode braces inside string tokens as Unicode escapes. - """ - - dumped = json.dumps(str(value), ensure_ascii=False) - return _escape_braces_inside_json_strings(dumped) - - -_PLAIN_YAML_KEY_RE = re.compile(r"^[A-Za-z0-9_./:-]+$") - - -def _salt_yaml_key(value: Any) -> str: - text = str(value) - if text and _PLAIN_YAML_KEY_RE.match(text) and not text.startswith(("-", "?", ":")): - return text - return salt_sls_json_quote(text) - - -def _salt_yaml_scalar(value: Any) -> str: - if value is None: - return "null" - if isinstance(value, bool): - return "true" if value else "false" - if isinstance(value, int) and not isinstance(value, bool): - return str(value) - if isinstance(value, float): - return json.dumps(value, allow_nan=False) - return salt_sls_json_quote(value) - - -def _salt_yaml_lines( - value: Any, indent: int = 0, *, sort_keys: bool = True -) -> list[str]: - prefix = " " * indent - if isinstance(value, Mapping): - if not value: - return [prefix + "{}"] - keys = sorted(value, key=lambda item: str(item)) if sort_keys else list(value) - lines: list[str] = [] - for key in keys: - inner = value[key] - key_text = _salt_yaml_key(key) - if isinstance(inner, Mapping): - if not inner: - lines.append(f"{prefix}{key_text}: {{}}") - else: - lines.append(f"{prefix}{key_text}:") - lines.extend( - _salt_yaml_lines(inner, indent + 2, sort_keys=sort_keys) - ) - elif isinstance(inner, (list, tuple, set)): - seq = list(inner) if not isinstance(inner, set) else sorted(inner) - if not seq: - lines.append(f"{prefix}{key_text}: []") - else: - lines.append(f"{prefix}{key_text}:") - lines.extend(_salt_yaml_lines(seq, indent + 2, sort_keys=sort_keys)) - else: - lines.append(f"{prefix}{key_text}: {_salt_yaml_scalar(inner)}") - return lines - - if isinstance(value, (list, tuple, set)): - seq = list(value) if not isinstance(value, set) else sorted(value) - if not seq: - return [prefix + "[]"] - lines = [] - for item in seq: - if isinstance(item, Mapping): - if not item: - lines.append(prefix + "- {}") - else: - lines.append(prefix + "-") - lines.extend( - _salt_yaml_lines(item, indent + 2, sort_keys=sort_keys) - ) - elif isinstance(item, (list, tuple, set)): - lines.append(prefix + "-") - lines.extend(_salt_yaml_lines(item, indent + 2, sort_keys=sort_keys)) - else: - lines.append(f"{prefix}- {_salt_yaml_scalar(item)}") - return lines - - return [prefix + _salt_yaml_scalar(value)] - - -def salt_sls_yaml_dump( - value: Any, - *, - sort_keys: bool = True, - explicit_start: bool = False, -) -> str: - """Dump block YAML whose string braces cannot form Salt Jinja delimiters.""" - - lines = _salt_yaml_lines(_plain_json_data(value), sort_keys=sort_keys) - rendered = "\n".join(lines).rstrip() + "\n" - if explicit_start: - rendered = "---\n" + rendered - return rendered diff --git a/enroll/salt.py b/enroll/salt.py deleted file mode 100644 index 2a9fc69..0000000 --- a/enroll/salt.py +++ /dev/null @@ -1,1759 +0,0 @@ -from __future__ import annotations - -import hashlib -import json -import re -import shlex -import shutil -from pathlib import Path -from typing import Any, Dict, Iterable, List, Mapping, Optional, Tuple - - -from .cm import ( - CMModule, - resolve_catalog_conflicts, - role_order_key, - markdown_list, -) -from .jinjaturtle import jinjify_artifact, resolve_jinjaturtle_mode -from .manifest_safety import ( - copy_safe_artifact_file, - prepare_manifest_output_dir, - safe_artifact_file, -) -from .render_safety import salt_sls_json_quote, salt_sls_yaml_dump -from .state import inventory_packages_from_state, roles_from_state -from .yamlutil import yaml_load_mapping_file - - -class SaltRole(CMModule): - """Salt-specific view of a renderer-neutral CMModule.""" - - managed_owner_attr = "user" - - def __init__(self, role_name: str) -> None: - super().__init__( - role_name=role_name, - module_name=_salt_name(role_name, fallback="enroll_role"), - ) - self.container_images: List[Dict[str, Any]] = [] - self.flatpak_remotes: List[Dict[str, Any]] = [] - self.flatpaks: List[Dict[str, Any]] = [] - self.snaps: List[Dict[str, Any]] = [] - - def has_resources(self) -> bool: - return self.has_resources_or_attrs( - "container_images", "flatpak_remotes", "flatpaks", "snaps" - ) - - @property - def sls_name(self) -> str: - return f"roles.{self.module_name}" - - def add_service_snapshot(self, snap: Dict[str, Any]) -> None: - self.add_service_snapshot_state( - snap, state_key="state", running="running", stopped="dead" - ) - unit = self.service_unit_from_snapshot(snap) - if unit in self.services: - self.services[unit]["state_id"] = _state_id( - "service", unit, role=self.module_name - ) - - def add_users_snapshot(self, snap: Dict[str, Any]) -> None: - records = self.user_records_from_snapshot(snap) - self.groups.update(self.user_group_names_from_records(records)) - for record in records: - name = str(record.get("name") or "") - user_data: Dict[str, Any] = { - "name": name, - "uid": record.get("uid"), - "gid": record.get("primary_group") or record.get("gid"), - "home": record.get("home"), - "shell": record.get("shell"), - "groups": record.get("supplementary_groups") or [], - } - user_data.update(_gecos_attrs(record.get("gecos"))) - self.users[name] = user_data - - self.add_user_flatpaks_snapshot(snap) - - def prepare_flatpak_remote(self, item: Dict[str, Any]) -> Dict[str, Any]: - return _prepare_flatpak_remote(item) - - def prepare_flatpak_item(self, item: Dict[str, Any]) -> Dict[str, Any]: - return _prepare_flatpak_item(item) - - def prepare_snap_item(self, item: Dict[str, Any]) -> Dict[str, Any]: - return _prepare_snap_item(item) - - def add_firewall_runtime_snapshot( - self, - snap: Dict[str, Any], - *, - bundle_dir: str, - artifact_role: str, - role_files_dir: Path, - file_prefix: Optional[str] = None, - ) -> None: - super().add_firewall_runtime_snapshot( - snap, - bundle_dir=bundle_dir, - artifact_role=artifact_role, - files_dir=role_files_dir, - copy_artifact=_copy_artifact, - source_uri=_source_uri, - file_prefix=file_prefix, - dir_attrs={"require": [{"file": "/etc/enroll"}]}, - ) - - def add_container_images_snapshot(self, snap: Dict[str, Any]) -> None: - for raw in snap.get("images", []) or []: - if not isinstance(raw, dict): - continue - engine = str(raw.get("engine") or "").strip().lower() - pull_ref = str(raw.get("pull_ref") or "").strip() - if engine not in {"docker", "podman"}: - continue - if not pull_ref: - tags = ", ".join(str(t) for t in (raw.get("repo_tags") or []) if t) - label = tags or str(raw.get("image_id") or "unknown image") - self.notes.append( - f"Container image {label} has no RepoDigest; exact Salt pull state was not rendered." - ) - continue - item = dict(raw) - item["engine"] = engine - item["pull_ref"] = pull_ref - item["scope"] = str(item.get("scope") or "system").strip() or "system" - item["tag_aliases"] = [ - dict(alias) - for alias in (item.get("tag_aliases") or []) - if isinstance(alias, dict) and alias.get("ref") - ] - item["pull_cmd"] = _container_pull_cmd(engine, pull_ref) - item["pull_unless"] = _container_exists_cmd(engine, pull_ref) - for alias in item["tag_aliases"]: - alias_ref = str(alias.get("ref") or "") - alias["tag_cmd"] = _container_tag_cmd(engine, pull_ref, alias_ref) - alias["tag_unless"] = _container_tag_matches_cmd( - engine, pull_ref, alias_ref - ) - self.container_images.append(item) - for note in snap.get("notes", []) or []: - self.notes.append(str(note)) - - def add_managed_content( - self, - snap: Dict[str, Any], - *, - bundle_dir: str, - artifact_role: str, - role_files_dir: Path, - file_prefix: Optional[str] = None, - jt_exe: Optional[str] = None, - jt_enabled: bool = False, - overwrite_templates: bool = True, - watch_services: Optional[List[str]] = None, - watch_service_states: Optional[List[str]] = None, - ) -> None: - for d in self.managed_dirs_from_snapshot(snap): - path = str(d.get("path") or "").strip() - self.add_managed_dir( - path, - user=d.get("owner") or "root", - group=d.get("group") or "root", - mode=d.get("mode") or "0755", - makedirs=True, - reason=d.get("reason") or "managed_dir", - ) - - watch_state_ids = _service_watch_state_ids( - self.module_name, - watch_services=watch_services, - watch_service_states=watch_service_states, - ) - - for mf in self.managed_files_from_snapshot(snap): - path = str(mf.get("path") or "").strip() - src_rel = str(mf.get("src_rel") or "").strip() - if not path or not src_rel: - continue - - template = _jinjify_managed_file( - bundle_dir, - artifact_role, - src_rel, - path, - role_files_dir.parent, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=overwrite_templates, - ) - if template is not None: - tmpl_rel, context = template - attrs: Dict[str, Any] = { - "user": mf.get("owner") or "root", - "group": mf.get("group") or "root", - "mode": mf.get("mode") or "0644", - "source": _template_source_uri(self.module_name, tmpl_rel), - "template": "jinja", - "context": context, - "makedirs": True, - "reason": mf.get("reason") or "managed_file", - } - if watch_state_ids and not path.startswith("/etc/systemd/system/"): - attrs["watch_in"] = [ - {"service": state_id} for state_id in watch_state_ids - ] - self.add_managed_file(path, **attrs) - continue - - role_rel = _copy_artifact( - bundle_dir, - artifact_role, - src_rel, - role_files_dir, - dst_prefix=file_prefix, - ) - if not role_rel: - self.notes.append( - f"Skipped {path}: harvested artifact {artifact_role}/{src_rel} was not present." - ) - continue - attrs = { - "user": mf.get("owner") or "root", - "group": mf.get("group") or "root", - "mode": mf.get("mode") or "0644", - "source": _source_uri(self.module_name, role_rel), - "makedirs": True, - "reason": mf.get("reason") or "managed_file", - } - if watch_state_ids and not path.startswith("/etc/systemd/system/"): - attrs["watch_in"] = [ - {"service": state_id} for state_id in watch_state_ids - ] - self.add_managed_file(path, **attrs) - - for ml in self.managed_links_from_snapshot(snap): - path = str(ml.get("path") or "").strip() - target = str(ml.get("target") or "").strip() - if not path or not target: - continue - self.add_managed_link( - path, - target=target, - force=False, - makedirs=True, - reason=ml.get("reason") or "managed_link", - ) - - self.remove_directory_resource_conflicts() - - -_RESERVED_SALT_NAMES = {"top", "init", "files", "pillar", "states", "roles"} - - -def _salt_name(raw: str, *, fallback: str = "role") -> str: - s = re.sub(r"[^A-Za-z0-9_]+", "_", raw or fallback) - s = re.sub(r"_+", "_", s).strip("_").lower() - if not s: - s = fallback - if not re.match(r"^[a-z_]", s): - s = f"{fallback}_{s}" - if s in _RESERVED_SALT_NAMES: - s = f"{fallback}_{s}" - return s - - -def _state_id(prefix: str, value: Any, *, role: str = "") -> str: - label = re.sub(r"[^A-Za-z0-9_]+", "_", str(value or "item").strip().lower()) - label = re.sub(r"_+", "_", label).strip("_") or "item" - digest = hashlib.sha1( - str(value).encode("utf-8", errors="replace") - ).hexdigest()[ # nosec B324 - :8 - ] # nosec B324 - parts = ["enroll", prefix] - if role: - parts.append(role) - parts.extend([label[:40], digest]) - return "_".join(parts) - - -def _plain_salt_data(value: Any) -> Any: - """Return data made from plain JSON/YAML-safe containers. - - Salt's Jinja ``yaml_encode`` filter cannot represent Salt/PyYAML - ``OrderedDict`` values. Normalise generated template contexts before we - write static SLS or pillar data, and before passing context to file.managed. - """ - - if isinstance(value, Mapping): - return {str(key): _plain_salt_data(inner) for key, inner in value.items()} - if isinstance(value, list): - return [_plain_salt_data(item) for item in value] - if isinstance(value, tuple): - return [_plain_salt_data(item) for item in value] - if isinstance(value, set): - return sorted(_plain_salt_data(item) for item in value) - return value - - -_TO_JSON_FILTER_RE = re.compile( - r"{{\s*([A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)?)\s*" - r"\|\s*to_json\s*\([^)]*\)\s*}}" -) - - -def _saltify_jinjaturtle_template( - template_text: str, context: Dict[str, Any] -) -> Tuple[str, Dict[str, Any]]: - """Translate JinjaTurtle's Ansible-oriented Jinja into Salt-safe Jinja. - - JinjaTurtle emits Ansible's ``to_json`` filter for JSON/TOML values. Salt's - Jinja environment does not provide that filter. For ordinary generated - context variables, pre-render a JSON string and substitute a plain variable - reference. For loop-local expressions such as ``item`` or ``item.name`` we - fall back to Jinja's built-in ``tojson`` filter. - """ - - salt_context = _plain_salt_data(context) - - def replace(match: re.Match[str]) -> str: - expr = match.group(1) - if "." not in expr and expr in salt_context: - json_var = f"{expr}__enroll_json" - salt_context[json_var] = json.dumps(salt_context[expr], ensure_ascii=False) - return "{{ " + json_var + " }}" - return "{{ " + expr + " | tojson }}" - - return _TO_JSON_FILTER_RE.sub(replace, template_text), salt_context - - -def _service_watch_state_ids( - role_name: str, - *, - watch_services: Optional[Iterable[str]] = None, - watch_service_states: Optional[Iterable[str]] = None, -) -> List[str]: - """Return de-duplicated Salt service state ids for watch_in requisites.""" - - out: List[str] = [] - seen = set() - for state_id in watch_service_states or []: - value = str(state_id or "").strip() - if value and value not in seen: - seen.add(value) - out.append(value) - for unit in watch_services or []: - unit_s = str(unit or "").strip() - if not unit_s: - continue - value = _state_id("service", unit_s, role=role_name) - if value not in seen: - seen.add(value) - out.append(value) - return out - - -def _active_service_state_ids_by_unit( - entries: Iterable[Dict[str, Any]], -) -> Dict[str, str]: - """Return generated Salt service state ids keyed by active systemd unit.""" - - by_unit: Dict[str, str] = {} - for entry in entries: - if str(entry.get("kind") or "package") != "service": - continue - snap = entry.get("snapshot") or {} - if not isinstance(snap, dict): - continue - unit = str(snap.get("unit") or "").strip() - if not unit or str(snap.get("active_state") or "") != "active": - continue - source_label = str(snap.get("role_name") or snap.get("unit") or "service") - role_name = _salt_name( - str(entry.get("role_label") or source_label), fallback="service" - ) - by_unit.setdefault(unit, _state_id("service", unit, role=role_name)) - return by_unit - - -def _yaml_quote(value: Any) -> str: - return salt_sls_json_quote(value) - - -def _yaml_bool(value: Any) -> str: - return "true" if bool(value) else "false" - - -def _shell_quote(value: Any) -> str: - return shlex.quote(str(value or "")) - - -def _container_pull_cmd(engine: str, pull_ref: str) -> str: - return f"{engine} pull {_shell_quote(pull_ref)}" - - -def _container_exists_cmd(engine: str, ref: str) -> str: - if engine == "podman": - return f"podman image exists {_shell_quote(ref)}" - return f"docker image inspect {_shell_quote(ref)} >/dev/null 2>&1" - - -def _container_image_id_expr(engine: str, ref: str) -> str: - """Return a shell expression that extracts an inspected image ID. - - Salt renders SLS files through Jinja before YAML, so Docker's normal - format template cannot be emitted literally without careful escaping. Use - JSON output plus sed instead; it avoids Go-template braces in generated - Salt states and pillar data. - """ - - sed_id = ( - r"sed -n 's/^[[:space:]]*\"Id\":[[:space:]]*\"\([^\"]*\)\".*/\1/p' " - r"| head -n 1" - ) - return ( - f"{_shell_quote(engine)} image inspect {_shell_quote(ref)} " - f"2>/dev/null | {sed_id}" - ) - - -def _container_tag_matches_cmd(engine: str, pull_ref: str, tag_ref: str) -> str: - """Return a shell guard that is true only when tag_ref points at pull_ref.""" - - return ( - f'test "$({_container_image_id_expr(engine, tag_ref)})" ' - f'= "$({_container_image_id_expr(engine, pull_ref)})"' - ) - - -def _container_tag_cmd(engine: str, pull_ref: str, tag_ref: str) -> str: - return f"{engine} tag {_shell_quote(pull_ref)} {_shell_quote(tag_ref)}" - - -def _flatpak_scope(item: Dict[str, Any]) -> str: - return "--user" if str(item.get("method") or "system") == "user" else "--system" - - -def _flatpak_home(item: Dict[str, Any]) -> Optional[str]: - user = str(item.get("user") or "").strip() - if not user: - return None - return str(item.get("home") or f"/home/{user}") - - -def _flatpak_env(item: Dict[str, Any]) -> Dict[str, str]: - home = _flatpak_home(item) - if not home: - return {} - return {"HOME": home, "XDG_DATA_HOME": f"{home}/.local/share"} - - -def _flatpak_remote_exists_cmd(item: Dict[str, Any]) -> str: - return ( - f"flatpak {_flatpak_scope(item)} remote-list --columns=name " - f"| grep -Fx -- {_shell_quote(item.get('name'))}" - ) - - -def _flatpak_remote_add_cmd(item: Dict[str, Any]) -> str: - return ( - f"flatpak {_flatpak_scope(item)} remote-add --if-not-exists " - f"{_shell_quote(item.get('name'))} {_shell_quote(item.get('url'))}" - ) - - -def _flatpak_ref(item: Dict[str, Any]) -> str: - ref = str(item.get("ref") or "").strip() - if ref: - return ref - return str(item.get("name") or "").strip() - - -def _flatpak_exists_cmd(item: Dict[str, Any]) -> str: - return f"flatpak {_flatpak_scope(item)} info {_shell_quote(_flatpak_ref(item))} >/dev/null 2>&1" - - -def _flatpak_install_cmd(item: Dict[str, Any]) -> str: - args = ["flatpak", _flatpak_scope(item), "install", "-y"] - remote = str(item.get("remote") or "").strip() - if remote: - args.append(remote) - args.append(_flatpak_ref(item)) - return " ".join(_shell_quote(arg) for arg in args) - - -def _prepare_flatpak_remote(item: Dict[str, Any]) -> Dict[str, Any]: - out = dict(item) - method = str(out.get("method") or "system") - user = str(out.get("user") or "") - name = str(out.get("name") or "") - out["state_id"] = _state_id("flatpak_remote", f"{method}:{user}:{name}") - out["add_cmd"] = _flatpak_remote_add_cmd(out) - out["exists_cmd"] = _flatpak_remote_exists_cmd(out) - out["env"] = _flatpak_env(out) - return out - - -def _prepare_flatpak_item(item: Dict[str, Any]) -> Dict[str, Any]: - out = dict(item) - method = str(out.get("method") or "system") - user = str(out.get("user") or "") - ref = _flatpak_ref(out) - out["state_id"] = _state_id("flatpak", f"{method}:{user}:{ref}") - out["install_cmd"] = _flatpak_install_cmd(out) - out["exists_cmd"] = _flatpak_exists_cmd(out) - out["env"] = _flatpak_env(out) - return out - - -def _snap_exists_cmd(item: Dict[str, Any]) -> str: - return f"snap list {_shell_quote(item.get('name'))} >/dev/null 2>&1" - - -def _snap_install_cmd(item: Dict[str, Any]) -> str: - args = ["snap", "install", str(item.get("name") or "")] - channel = str(item.get("channel") or "").strip() - revision = str(item.get("revision") or "").strip() - if channel: - args.append(f"--channel={channel}") - elif revision: - args.append(f"--revision={revision}") - if item.get("classic"): - args.append("--classic") - if item.get("devmode"): - args.append("--devmode") - if item.get("dangerous"): - args.append("--dangerous") - return " ".join(_shell_quote(arg) for arg in args if str(arg)) - - -def _prepare_snap_item(item: Dict[str, Any]) -> Dict[str, Any]: - out = dict(item) - name = str(out.get("name") or "") - out["state_id"] = _state_id("snap", name) - out["install_cmd"] = _snap_install_cmd(out) - out["exists_cmd"] = _snap_exists_cmd(out) - return out - - -def _append_firewall_runtime_states(lines: List[str], runtime: Dict[str, Any]) -> None: - specs = [ - ( - "ipset", - "ipset_save", - "ipset_restore_cmd", - "enroll_firewall_runtime_ipset_restore", - ), - ( - "iptables_v4", - "iptables_v4_save", - "iptables_v4_restore_cmd", - "enroll_firewall_runtime_iptables_v4_restore", - ), - ( - "iptables_v6", - "iptables_v6_save", - "iptables_v6_restore_cmd", - "enroll_firewall_runtime_iptables_v6_restore", - ), - ] - for _family, path_key, cmd_key, state_id in specs: - path = str(runtime.get(path_key) or "") - command = str(runtime.get(cmd_key) or "") - if not path or not command: - continue - lines.extend( - [ - f"{state_id}:", - " cmd.run:", - f" - name: {_yaml_quote(command)}", - " - onchanges:", - f" - file: {_yaml_quote(path)}", - "", - ] - ) - - -def _clean_gecos_part(value: Any) -> Optional[str]: - text = str(value or "").strip() - return text or None - - -def _gecos_attrs(value: Any) -> Dict[str, str]: - """Return Salt user.present-safe GECOS fields. - - Linux passwd GECOS is comma-separated. Passing the raw field as Salt's - ``fullname`` can fail for values such as ``Node,,,`` because Salt validates - commas inside individual GECOS subfields. Split it into Salt's native - fields instead. - """ - - raw = str(value or "") - if not raw.strip(): - return {} - parts = raw.split(",", 4) - keys = ("fullname", "roomnumber", "workphone", "homephone", "other") - out: Dict[str, str] = {} - for key, part in zip(keys, parts): - cleaned = _clean_gecos_part(part) - if cleaned: - out[key] = cleaned - return out - - -def _copy_artifact( - bundle_dir: str, - role: str, - src_rel: str, - dst_files_dir: Path, - *, - dst_prefix: Optional[str] = None, -) -> Optional[str]: - if not role or not src_rel: - return None - try: - src = safe_artifact_file(bundle_dir, role, src_rel) - except FileNotFoundError: - return None - role_rel = Path(dst_prefix or "") / src_rel - dst = dst_files_dir / role_rel - dst.parent.mkdir(parents=True, exist_ok=True) - copy_safe_artifact_file(src, dst) - return role_rel.as_posix() - - -def _source_uri(module_name: str, role_rel: str) -> str: - return f"salt://roles/{module_name}/files/{role_rel}" - - -def _template_source_uri(module_name: str, tmpl_rel: str) -> str: - return f"salt://roles/{module_name}/templates/{tmpl_rel}" - - -def _jinjify_managed_file( - bundle_dir: str, - artifact_role: str, - src_rel: str, - dest_path: str, - role_dir: Path, - *, - jt_exe: Optional[str], - jt_enabled: bool, - overwrite_templates: bool, -) -> Optional[Tuple[str, Dict[str, Any]]]: - converted = jinjify_artifact( - bundle_dir, - artifact_role, - src_rel, - dest_path, - role_dir / "templates", - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=overwrite_templates, - ) - if converted is None: - return None - - template_text, context = _saltify_jinjaturtle_template( - converted.template_text, converted.context - ) - template_path = role_dir / "templates" / converted.template_rel - if template_text != converted.template_text: - existing = ( - template_path.read_text(encoding="utf-8") if template_path.exists() else "" - ) - if ( - overwrite_templates - or not template_path.exists() - or _TO_JSON_FILTER_RE.search(existing) - ): - template_path.parent.mkdir(parents=True, exist_ok=True) - template_path.write_text(template_text, encoding="utf-8") - - return converted.template_rel, context - - -def _node_file_prefix(fqdn: str) -> str: - name = re.sub(r"[^A-Za-z0-9_.-]+", "_", str(fqdn or "").strip()) - name = name.strip("._-") or "node" - return f"nodes/{name}" - - -def _node_sls_basename(fqdn: str) -> str: - raw = str(fqdn or "node").strip() or "node" - name = re.sub(r"[^A-Za-z0-9_]+", "_", raw).strip("_").lower() or "node" - digest = hashlib.sha1( - raw.encode("utf-8", errors="replace") - ).hexdigest()[ # nosec B324 - :8 - ] # nosec B324 - return f"{name}_{digest}" - - -def _collect_salt_roles( - state: Dict[str, Any], - bundle_dir: str, - states_dir: Path, - *, - fqdn: Optional[str] = None, - no_common_roles: bool = False, - jt_exe: Optional[str] = None, - jt_enabled: bool = False, -) -> List[SaltRole]: - roles = roles_from_state(state) - inventory_packages = inventory_packages_from_state(state) - use_common_roles = not fqdn and not no_common_roles - node_file_prefix = _node_file_prefix(fqdn) if fqdn else None - out: Dict[str, SaltRole] = {} - - def ensure_role(role_name: str) -> SaltRole: - role_name = _salt_name(role_name, fallback="enroll_role") - return out.setdefault(role_name, SaltRole(role_name)) - - for key in ( - "apt_config", - "dnf_config", - "etc_custom", - "usr_local_custom", - "extra_paths", - "sysctl", - ): - snap = roles.get(key) or {} - if not isinstance(snap, dict): - continue - role_name = _salt_name( - str(snap.get("role_name") or key), fallback="enroll_role" - ) - srole = ensure_role(role_name) - srole.add_managed_content( - snap, - bundle_dir=bundle_dir, - artifact_role=str(snap.get("role_name") or key), - role_files_dir=states_dir / "roles" / srole.module_name / "files", - file_prefix=node_file_prefix, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=not bool(fqdn), - ) - - users_snap = roles.get("users") or {} - if isinstance(users_snap, dict): - role_name = _salt_name( - str(users_snap.get("role_name") or "users"), fallback="enroll_role" - ) - srole = ensure_role(role_name) - srole.add_users_snapshot(users_snap) - srole.add_managed_content( - users_snap, - bundle_dir=bundle_dir, - artifact_role=str(users_snap.get("role_name") or "users"), - role_files_dir=states_dir / "roles" / srole.module_name / "files", - file_prefix=node_file_prefix, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=not bool(fqdn), - ) - - package_service_entries = list( - CMModule.package_service_entries( - roles, inventory_packages, use_common_roles=use_common_roles - ) - ) - service_units_by_package = CMModule.active_service_units_by_package( - package_service_entries - ) - service_state_ids_by_unit = _active_service_state_ids_by_unit( - package_service_entries - ) - - for entry in package_service_entries: - snap = entry.get("snapshot") or {} - kind = str(entry.get("kind") or "package") - fallback = "service" if kind == "service" else "package" - source_label = str( - snap.get("role_name") or snap.get("unit") or snap.get("package") or fallback - ) - original_role_name = _salt_name(source_label, fallback=fallback) - role_name = _salt_name( - str(entry.get("role_label") or source_label), - fallback="package_group" if use_common_roles else fallback, - ) - srole = ensure_role(role_name) - watch_services: List[str] = [] - watch_service_states: List[str] = [] - if kind == "service": - srole.add_service_snapshot(snap) - unit = str(snap.get("unit") or "").strip() - if unit and str(snap.get("active_state") or "") == "active": - watch_services = [unit] - else: - srole.add_package_snapshot(snap) - watch_services = CMModule.active_service_units_for_package_snapshot( - snap, service_units_by_package - ) - watch_service_states = [ - service_state_ids_by_unit[unit] - for unit in watch_services - if unit in service_state_ids_by_unit - ] - if watch_service_states: - watch_services = [] - srole.add_managed_content( - snap, - bundle_dir=bundle_dir, - artifact_role=str(snap.get("role_name") or original_role_name), - role_files_dir=states_dir / "roles" / srole.module_name / "files", - file_prefix=node_file_prefix, - jt_exe=jt_exe, - jt_enabled=jt_enabled, - overwrite_templates=not bool(fqdn), - watch_services=watch_services, - watch_service_states=watch_service_states, - ) - - container_images = roles.get("container_images") or {} - if isinstance(container_images, dict) and ( - container_images.get("images") or container_images.get("notes") - ): - srole = ensure_role( - str(container_images.get("role_name") or "container_images") - ) - srole.add_container_images_snapshot(container_images) - - fw = roles.get("firewall_runtime") or {} - if isinstance(fw, dict): - has_fw = ( - fw.get("ipset_save") - or fw.get("iptables_v4_save") - or fw.get("iptables_v6_save") - ) - if has_fw: - runtime_role = ensure_role("enroll_runtime") - runtime_role.add_managed_dir( - "/etc/enroll", - user="root", - group="root", - mode="0750", - reason="enroll_runtime", - ) - role_name = str(fw.get("role_name") or "firewall_runtime") - srole = ensure_role(role_name) - srole.add_firewall_runtime_snapshot( - fw, - bundle_dir=bundle_dir, - artifact_role=role_name, - role_files_dir=states_dir / "roles" / srole.module_name / "files", - file_prefix=node_file_prefix, - ) - - flatpak = roles.get("flatpak") or {} - if isinstance(flatpak, dict) and ( - flatpak.get("system_flatpaks") or flatpak.get("remotes") or flatpak.get("notes") - ): - srole = ensure_role(str(flatpak.get("role_name") or "flatpak")) - srole.add_flatpak_snapshot(flatpak) - - snap = roles.get("snap") or {} - if isinstance(snap, dict) and (snap.get("system_snaps") or snap.get("notes")): - srole = ensure_role(str(snap.get("role_name") or "snap")) - srole.add_snap_snapshot(snap) - - salt_roles = sorted(out.values(), key=lambda r: role_order_key(r.role_name)) - resolve_catalog_conflicts(salt_roles) - return [r for r in salt_roles if r.has_resources()] - - -def _append_yaml_value(lines: List[str], key: str, value: Any, *, indent: int) -> None: - prefix = " " * indent - if isinstance(value, dict): - dumped = salt_sls_yaml_dump(_plain_salt_data(value), sort_keys=True).rstrip() - if not dumped: - lines.append(f"{prefix}- {key}: {{}}") - return - lines.append(f"{prefix}- {key}:") - for line in dumped.splitlines(): - lines.append(f"{prefix} {line}") - return - lines.append(f"{prefix}- {key}: {_yaml_quote(value)}") - - -def _render_static_role(srole: SaltRole) -> str: - lines: List[str] = ["# Generated by Enroll from harvest state.", ""] - - for package in sorted(srole.packages): - lines.extend( - [ - f"{_state_id('pkg', package, role=srole.module_name)}:", - " pkg.installed:", - f" - name: {_yaml_quote(package)}", - "", - ] - ) - - for group in sorted(srole.groups): - lines.extend( - [ - f"{_state_id('group', group, role=srole.module_name)}:", - " group.present:", - f" - name: {_yaml_quote(group)}", - "", - ] - ) - - for name in sorted(srole.users): - user = srole.users[name] - lines.extend( - [ - f"{_state_id('user', name, role=srole.module_name)}:", - " user.present:", - f" - name: {_yaml_quote(name)}", - ] - ) - if user.get("uid") is not None: - lines.append(f" - uid: {user['uid']}") - if user.get("gid") is not None: - lines.append(f" - gid: {_yaml_quote(user['gid'])}") - if user.get("home"): - lines.append(f" - home: {_yaml_quote(user['home'])}") - if user.get("shell"): - lines.append(f" - shell: {_yaml_quote(user['shell'])}") - for gecos_key in ("fullname", "roomnumber", "workphone", "homephone", "other"): - if user.get(gecos_key): - lines.append(f" - {gecos_key}: {_yaml_quote(user[gecos_key])}") - if user.get("groups"): - lines.append(" - groups:") - for group in user.get("groups") or []: - lines.append(f" - {_yaml_quote(group)}") - lines.append(" - remove_groups: false") - lines.append("") - - for path, attrs in sorted(srole.dirs.items()): - lines.extend( - [ - f"{_yaml_quote(path)}:", - " file.directory:", - f" - user: {_yaml_quote(attrs.get('user') or attrs.get('owner') or 'root')}", - f" - group: {_yaml_quote(attrs.get('group') or 'root')}", - f" - mode: {_yaml_quote(str(attrs.get('mode') or '0755'))}", - " - makedirs: true", - ] - ) - if attrs.get("require"): - lines.append(" - require:") - for req in attrs.get("require") or []: - if isinstance(req, dict): - for req_kind, req_name in req.items(): - lines.append(f" - {req_kind}: {_yaml_quote(req_name)}") - lines.append("") - - for path, attrs in sorted(srole.files.items()): - lines.extend( - [ - f"{_yaml_quote(path)}:", - " file.managed:", - f" - source: {_yaml_quote(attrs.get('source') or '')}", - f" - user: {_yaml_quote(attrs.get('user') or attrs.get('owner') or 'root')}", - f" - group: {_yaml_quote(attrs.get('group') or 'root')}", - f" - mode: {_yaml_quote(str(attrs.get('mode') or '0644'))}", - " - makedirs: true", - ] - ) - if attrs.get("template"): - lines.append(f" - template: {_yaml_quote(attrs.get('template'))}") - if attrs.get("context"): - _append_yaml_value(lines, "context", attrs.get("context"), indent=4) - if attrs.get("watch_in"): - lines.append(" - watch_in:") - for req in attrs.get("watch_in") or []: - if isinstance(req, dict): - for req_kind, req_name in req.items(): - lines.append(f" - {req_kind}: {_yaml_quote(req_name)}") - lines.append("") - - for path, attrs in sorted(srole.links.items()): - lines.extend( - [ - f"{_yaml_quote(path)}:", - " file.symlink:", - f" - target: {_yaml_quote(attrs.get('target') or '')}", - f" - force: {_yaml_bool(attrs.get('force', False))}", - " - makedirs: true", - "", - ] - ) - - for name in sorted(srole.services): - svc = srole.services[name] - state_fun = "running" if svc.get("state") == "running" else "dead" - lines.extend( - [ - f"{svc.get('state_id') or _state_id('service', name, role=srole.module_name)}:", - f" service.{state_fun}:", - f" - name: {_yaml_quote(svc.get('name') or name)}", - f" - enable: {_yaml_bool(svc.get('enable', False))}", - "", - ] - ) - - flatpak_remote_state_ids: Dict[Tuple[str, str, str], str] = {} - for remote in srole.flatpak_remotes: - name = str(remote.get("name") or "").strip() - url = str(remote.get("url") or "").strip() - if not name or not url: - continue - state_id = str( - remote.get("state_id") - or _state_id("flatpak_remote", name, role=srole.module_name) - ) - key = ( - str(remote.get("method") or "system"), - str(remote.get("user") or ""), - name, - ) - flatpak_remote_state_ids[key] = state_id - lines.extend( - [ - f"{state_id}:", - " cmd.run:", - f" - name: {_yaml_quote(remote.get('add_cmd') or _flatpak_remote_add_cmd(remote))}", - f" - unless: {_yaml_quote(remote.get('exists_cmd') or _flatpak_remote_exists_cmd(remote))}", - ] - ) - remote_user = str(remote.get("user") or "") - if remote_user: - lines.append(f" - runas: {_yaml_quote(remote_user)}") - env = remote.get("env") or {} - if env: - lines.append(" - env:") - for key_name, value in sorted(env.items()): - lines.append(f" - {key_name}: {_yaml_quote(value)}") - if remote_user and remote_user in srole.users: - lines.extend( - [ - " - require:", - f" - user: {_state_id('user', remote_user, role=srole.module_name)}", - ] - ) - lines.append("") - - for app in srole.flatpaks: - ref = _flatpak_ref(app) - if not ref: - continue - state_id = str( - app.get("state_id") or _state_id("flatpak", ref, role=srole.module_name) - ) - method = str(app.get("method") or "system") - user = str(app.get("user") or "") - remote_name = str(app.get("remote") or "") - require_entries: List[Tuple[str, str]] = [] - if user and user in srole.users: - require_entries.append( - ("user", _state_id("user", user, role=srole.module_name)) - ) - if remote_name: - remote_state_id = flatpak_remote_state_ids.get((method, user, remote_name)) - if remote_state_id: - require_entries.append(("cmd", remote_state_id)) - lines.extend( - [ - f"{state_id}:", - " cmd.run:", - f" - name: {_yaml_quote(app.get('install_cmd') or _flatpak_install_cmd(app))}", - f" - unless: {_yaml_quote(app.get('exists_cmd') or _flatpak_exists_cmd(app))}", - ] - ) - if app.get("user"): - lines.append(f" - runas: {_yaml_quote(app.get('user'))}") - env = app.get("env") or {} - if env: - lines.append(" - env:") - for key_name, value in sorted(env.items()): - lines.append(f" - {key_name}: {_yaml_quote(value)}") - if require_entries: - lines.append(" - require:") - for req_kind, req_name in require_entries: - lines.append(f" - {req_kind}: {req_name}") - lines.append("") - - for snap in srole.snaps: - name = str(snap.get("name") or "").strip() - if not name: - continue - lines.extend( - [ - f"{snap.get('state_id') or _state_id('snap', name, role=srole.module_name)}:", - " cmd.run:", - f" - name: {_yaml_quote(snap.get('install_cmd') or _snap_install_cmd(snap))}", - f" - unless: {_yaml_quote(snap.get('exists_cmd') or _snap_exists_cmd(snap))}", - "", - ] - ) - - for idx, image in enumerate(srole.container_images, start=1): - engine = str(image.get("engine") or "").strip() - pull_ref = str(image.get("pull_ref") or "").strip() - if not engine or not pull_ref: - continue - if engine == "docker": - pull_state_id = _state_id("docker_pull", pull_ref, role=srole.module_name) - lines.extend( - [ - f"{pull_state_id}:", - " cmd.run:", - f" - name: {_yaml_quote(image.get('pull_cmd') or _container_pull_cmd(engine, pull_ref))}", - f" - unless: {_yaml_quote(image.get('pull_unless') or _container_exists_cmd(engine, pull_ref))}", - "", - ] - ) - for alias in image.get("tag_aliases") or []: - tag_ref = str(alias.get("ref") or "").strip() - if not tag_ref: - continue - lines.extend( - [ - f"{_state_id('docker_tag', tag_ref, role=srole.module_name)}:", - " cmd.run:", - f" - name: {_yaml_quote(alias.get('tag_cmd') or _container_tag_cmd(engine, pull_ref, tag_ref))}", - f" - unless: {_yaml_quote(alias.get('tag_unless') or _container_tag_matches_cmd(engine, pull_ref, tag_ref))}", - " - require:", - f" - cmd: {pull_state_id}", - "", - ] - ) - elif engine == "podman": - pull_state_id = _state_id("podman_pull", pull_ref, role=srole.module_name) - lines.extend( - [ - f"{pull_state_id}:", - " cmd.run:", - f" - name: {_yaml_quote(image.get('pull_cmd') or _container_pull_cmd(engine, pull_ref))}", - f" - unless: {_yaml_quote(image.get('pull_unless') or _container_exists_cmd(engine, pull_ref))}", - "", - ] - ) - for alias in image.get("tag_aliases") or []: - tag_ref = str(alias.get("ref") or "").strip() - if not tag_ref: - continue - lines.extend( - [ - f"{_state_id('podman_tag', tag_ref, role=srole.module_name)}:", - " cmd.run:", - f" - name: {_yaml_quote(alias.get('tag_cmd') or _container_tag_cmd(engine, pull_ref, tag_ref))}", - f" - unless: {_yaml_quote(alias.get('tag_unless') or _container_exists_cmd(engine, tag_ref))}", - " - require:", - f" - cmd: {pull_state_id}", - "", - ] - ) - - if srole.firewall_runtime: - _append_firewall_runtime_states(lines, srole.firewall_runtime) - - if "/etc/sysctl.d/99-enroll.conf" in srole.files: - lines.extend( - [ - f"{_state_id('cmd', 'apply_sysctl', role=srole.module_name)}:", - " cmd.run:", - " - name: sysctl -e -p /etc/sysctl.d/99-enroll.conf || true", - " - onchanges:", - " - file: /etc/sysctl.d/99-enroll.conf", - "", - ] - ) - - if srole.notes: - lines.append("# Notes and limitations") - for note in srole.notes: - lines.append(f"# - {note}") - lines.append("") - - return "\n".join(lines).rstrip() + "\n" - - -def _role_pillar_values(srole: SaltRole) -> Dict[str, Any]: - data: Dict[str, Any] = {} - if srole.packages: - data["packages"] = sorted(srole.packages) - if srole.groups: - data["groups"] = {group: {} for group in sorted(srole.groups)} - if srole.users: - users: Dict[str, Dict[str, Any]] = {} - for name in sorted(srole.users): - raw = srole.users[name] - attrs: Dict[str, Any] = {} - for key in ( - "uid", - "gid", - "home", - "shell", - "fullname", - "roomnumber", - "workphone", - "homephone", - "other", - ): - if raw.get(key) is not None: - attrs[key] = raw[key] - if raw.get("groups"): - attrs["groups"] = list(raw["groups"]) - attrs["remove_groups"] = False - users[name] = attrs - data["users"] = users - if srole.dirs: - data["dirs"] = { - path: { - "user": attrs.get("user") or attrs.get("owner") or "root", - "group": attrs.get("group") or "root", - "mode": str(attrs.get("mode") or "0755"), - "makedirs": True, - **({"require": attrs.get("require")} if attrs.get("require") else {}), - } - for path, attrs in sorted(srole.dirs.items()) - } - if srole.files: - data["files"] = { - path: { - "source": attrs.get("source") or "", - "user": attrs.get("user") or attrs.get("owner") or "root", - "group": attrs.get("group") or "root", - "mode": str(attrs.get("mode") or "0644"), - "makedirs": True, - **( - {"template": attrs.get("template")} if attrs.get("template") else {} - ), - **( - {"context": _plain_salt_data(attrs.get("context"))} - if attrs.get("context") - else {} - ), - **( - {"watch_in": attrs.get("watch_in")} if attrs.get("watch_in") else {} - ), - } - for path, attrs in sorted(srole.files.items()) - } - if srole.links: - data["links"] = { - path: { - "target": attrs.get("target") or "", - "force": bool(attrs.get("force", False)), - "makedirs": True, - } - for path, attrs in sorted(srole.links.items()) - } - if srole.services: - data["services"] = { - name: { - "name": svc.get("name") or name, - "state": "running" if svc.get("state") == "running" else "dead", - "enable": bool(svc.get("enable", False)), - "state_id": svc.get("state_id") - or _state_id("service", name, role=srole.module_name), - } - for name, svc in sorted(srole.services.items()) - } - if "/etc/sysctl.d/99-enroll.conf" in srole.files: - data["sysctl_apply"] = True - if srole.flatpak_remotes: - data["flatpak_remotes"] = list(srole.flatpak_remotes) - if srole.flatpaks: - data["flatpaks"] = list(srole.flatpaks) - if srole.snaps: - data["snaps"] = list(srole.snaps) - if srole.container_images: - data["container_images"] = list(srole.container_images) - if srole.firewall_runtime: - data["firewall_runtime"] = dict(srole.firewall_runtime) - if srole.notes: - data["notes"] = list(srole.notes) - return data - - -def _render_pillar_role(srole: SaltRole) -> str: - role_key = srole.module_name - lines = [ - "# Generated by Enroll from harvest state.", - f"{{% set role = salt['pillar.get']('enroll:roles:{role_key}', {{}}) %}}", - "", - "{% for package_name in role.get('packages', []) %}", - f"enroll_pkg_{role_key}_{{{{ loop.index }}}}:", - " pkg.installed:", - " - name: {{ package_name|yaml_dquote }}", - "{% endfor %}", - "", - "{% for group_name, group_attrs in role.get('groups', {}).items() %}", - f"enroll_group_{role_key}_{{{{ loop.index }}}}:", - " group.present:", - " - name: {{ group_name|yaml_dquote }}", - "{% endfor %}", - "", - "{% for user_name, user_attrs in role.get('users', {}).items() %}", - f"enroll_user_{role_key}_{{{{ loop.index }}}}:", - " user.present:", - " - name: {{ user_name|yaml_dquote }}", - "{% if user_attrs.get('uid') is not none %}", - " - uid: {{ user_attrs.get('uid') }}", - "{% endif %}", - "{% if user_attrs.get('gid') is not none %}", - " - gid: {{ user_attrs.get('gid')|yaml_dquote }}", - "{% endif %}", - "{% if user_attrs.get('home') %}", - " - home: {{ user_attrs.get('home')|yaml_dquote }}", - "{% endif %}", - "{% if user_attrs.get('shell') %}", - " - shell: {{ user_attrs.get('shell')|yaml_dquote }}", - "{% endif %}", - "{% for gecos_key in ['fullname', 'roomnumber', 'workphone', 'homephone', 'other'] %}", - "{% if user_attrs.get(gecos_key) %}", - " - {{ gecos_key }}: {{ user_attrs.get(gecos_key)|yaml_dquote }}", - "{% endif %}", - "{% endfor %}", - "{% if user_attrs.get('groups') %}", - " - groups:", - "{% for group_name in user_attrs.get('groups', []) %}", - " - {{ group_name|yaml_dquote }}", - "{% endfor %}", - " - remove_groups: {{ user_attrs.get('remove_groups', False)|yaml_encode }}", - "{% endif %}", - "{% endfor %}", - "", - "{% for path, attrs in role.get('dirs', {}).items() %}", - "{{ path|yaml_dquote }}:", - " file.directory:", - " - user: {{ attrs.get('user', 'root')|yaml_dquote }}", - " - group: {{ attrs.get('group', 'root')|yaml_dquote }}", - " - mode: {{ attrs.get('mode', '0755')|string|yaml_dquote }}", - " - makedirs: {{ attrs.get('makedirs', True)|yaml_encode }}", - "{% if attrs.get('require') %}", - " - require:", - "{% for req in attrs.get('require', []) %}", - "{% for req_kind, req_name in req.items() %}", - " - {{ req_kind }}: {{ req_name|yaml_dquote }}", - "{% endfor %}", - "{% endfor %}", - "{% endif %}", - "{% endfor %}", - "", - "{% for path, attrs in role.get('files', {}).items() %}", - "{{ path|yaml_dquote }}:", - " file.managed:", - " - source: {{ attrs.get('source', '')|yaml_dquote }}", - " - user: {{ attrs.get('user', 'root')|yaml_dquote }}", - " - group: {{ attrs.get('group', 'root')|yaml_dquote }}", - " - mode: {{ attrs.get('mode', '0644')|string|yaml_dquote }}", - " - makedirs: {{ attrs.get('makedirs', True)|yaml_encode }}", - "{% if attrs.get('template') %}", - " - template: {{ attrs.get('template')|yaml_dquote }}", - "{% endif %}", - "{% if attrs.get('context') %}", - " - context: {{ attrs.get('context')|tojson }}", - "{% endif %}", - "{% if attrs.get('watch_in') %}", - " - watch_in:", - "{% for req in attrs.get('watch_in') %}", - "{% for req_kind, req_name in req.items() %}", - " - {{ req_kind }}: {{ req_name|yaml_dquote }}", - "{% endfor %}", - "{% endfor %}", - "{% endif %}", - "{% endfor %}", - "", - "{% for path, attrs in role.get('links', {}).items() %}", - "{{ path|yaml_dquote }}:", - " file.symlink:", - " - target: {{ attrs.get('target', '')|yaml_dquote }}", - " - force: {{ attrs.get('force', False)|yaml_encode }}", - " - makedirs: {{ attrs.get('makedirs', True)|yaml_encode }}", - "{% endfor %}", - "", - "{% for service_id, svc in role.get('services', {}).items() %}", - "{{ svc.get('state_id') or ('enroll_service_" - + role_key - + "_' ~ loop.index|string) }}:", - " service.{{ 'running' if svc.get('state') == 'running' else 'dead' }}:", - " - name: {{ svc.get('name', service_id)|yaml_dquote }}", - " - enable: {{ svc.get('enable', False)|yaml_encode }}", - "{% endfor %}", - "", - "{% for remote in role.get('flatpak_remotes', []) %}", - "{{ remote.get('state_id') }}:", - " cmd.run:", - " - name: {{ remote.get('add_cmd')|yaml_dquote }}", - " - unless: {{ remote.get('exists_cmd')|yaml_dquote }}", - "{% if remote.get('user') %}", - " - runas: {{ remote.get('user')|yaml_dquote }}", - "{% endif %}", - "{% if remote.get('env') %}", - " - env:", - "{% for env_key, env_value in remote.get('env', {}).items() %}", - " - {{ env_key }}: {{ env_value|yaml_dquote }}", - "{% endfor %}", - "{% endif %}", - "{% endfor %}", - "", - "{% for app in role.get('flatpaks', []) %}", - "{{ app.get('state_id') }}:", - " cmd.run:", - " - name: {{ app.get('install_cmd')|yaml_dquote }}", - " - unless: {{ app.get('exists_cmd')|yaml_dquote }}", - "{% if app.get('user') %}", - " - runas: {{ app.get('user')|yaml_dquote }}", - "{% endif %}", - "{% if app.get('env') %}", - " - env:", - "{% for env_key, env_value in app.get('env', {}).items() %}", - " - {{ env_key }}: {{ env_value|yaml_dquote }}", - "{% endfor %}", - "{% endif %}", - "{% endfor %}", - "", - "{% for snap in role.get('snaps', []) %}", - "{{ snap.get('state_id') }}:", - " cmd.run:", - " - name: {{ snap.get('install_cmd')|yaml_dquote }}", - " - unless: {{ snap.get('exists_cmd')|yaml_dquote }}", - "{% endfor %}", - "", - "{% for image in role.get('container_images', []) %}", - "{% if image.get('engine') == 'docker' and image.get('pull_ref') %}", - f"enroll_docker_pull_{role_key}_{{{{ loop.index }}}}:", - " cmd.run:", - " - name: {{ image.get('pull_cmd')|yaml_dquote }}", - " - unless: {{ image.get('pull_unless')|yaml_dquote }}", - "{% set image_loop = loop.index %}", - "{% for alias in image.get('tag_aliases', []) %}", - f"enroll_docker_tag_{role_key}_{{{{ image_loop }}}}_{{{{ loop.index }}}}:", - " cmd.run:", - " - name: {{ alias.get('tag_cmd')|yaml_dquote }}", - " - unless: {{ alias.get('tag_unless')|yaml_dquote }}", - " - require:", - f" - cmd: enroll_docker_pull_{role_key}_{{{{ image_loop }}}}", - "{% endfor %}", - "{% elif image.get('engine') == 'podman' and image.get('pull_ref') %}", - f"enroll_podman_pull_{role_key}_{{{{ loop.index }}}}:", - " cmd.run:", - " - name: {{ image.get('pull_cmd')|yaml_dquote }}", - " - unless: {{ image.get('pull_unless')|yaml_dquote }}", - "{% set image_loop = loop.index %}", - "{% for alias in image.get('tag_aliases', []) %}", - f"enroll_podman_tag_{role_key}_{{{{ image_loop }}}}_{{{{ loop.index }}}}:", - " cmd.run:", - " - name: {{ alias.get('tag_cmd')|yaml_dquote }}", - " - unless: {{ alias.get('tag_unless')|yaml_dquote }}", - " - require:", - f" - cmd: enroll_podman_pull_{role_key}_{{{{ image_loop }}}}", - "{% endfor %}", - "{% endif %}", - "{% endfor %}", - "", - "{% set firewall_runtime = role.get('firewall_runtime', {}) %}", - "{% if firewall_runtime.get('ipset_restore_cmd') %}", - "enroll_firewall_runtime_ipset_restore:", - " cmd.run:", - " - name: {{ firewall_runtime.get('ipset_restore_cmd')|yaml_dquote }}", - " - onchanges:", - " - file: {{ firewall_runtime.get('ipset_save')|yaml_dquote }}", - "{% endif %}", - "", - "{% if firewall_runtime.get('iptables_v4_restore_cmd') %}", - "enroll_firewall_runtime_iptables_v4_restore:", - " cmd.run:", - " - name: {{ firewall_runtime.get('iptables_v4_restore_cmd')|yaml_dquote }}", - " - onchanges:", - " - file: {{ firewall_runtime.get('iptables_v4_save')|yaml_dquote }}", - "{% endif %}", - "", - "{% if firewall_runtime.get('iptables_v6_restore_cmd') %}", - "enroll_firewall_runtime_iptables_v6_restore:", - " cmd.run:", - " - name: {{ firewall_runtime.get('iptables_v6_restore_cmd')|yaml_dquote }}", - " - onchanges:", - " - file: {{ firewall_runtime.get('iptables_v6_save')|yaml_dquote }}", - "{% endif %}", - "", - "{% if role.get('sysctl_apply') and '/etc/sysctl.d/99-enroll.conf' in role.get('files', {}) %}", - f"enroll_apply_sysctl_{role_key}:", - " cmd.run:", - " - name: sysctl -e -p /etc/sysctl.d/99-enroll.conf || true", - " - onchanges:", - " - file: /etc/sysctl.d/99-enroll.conf", - "{% endif %}", - "", - "{% if role.get('notes') %}", - "# Notes and limitations", - "{% for note in role.get('notes', []) %}", - "# - {{ note }}", - "{% endfor %}", - "{% endif %}", - "", - ] - return "\n".join(lines).rstrip() + "\n" - - -def _write_yaml(path: Path, data: Dict[str, Any]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text( - salt_sls_yaml_dump(data, sort_keys=True, explicit_start=True), - encoding="utf-8", - ) - - -def _load_yaml_mapping(path: Path) -> Dict[str, Any]: - return yaml_load_mapping_file(path) - - -def _write_top(path: Path, mapping: Dict[str, List[str]]) -> None: - data = { - "base": {target: list(values) for target, values in sorted(mapping.items())} - } - _write_yaml(path, data) - - -def _read_top(path: Path) -> Dict[str, List[str]]: - data = _load_yaml_mapping(path) - base = data.get("base") if isinstance(data.get("base"), dict) else {} - out: Dict[str, List[str]] = {} - for target, values in base.items(): - if isinstance(values, list): - out[str(target)] = [str(v) for v in values if isinstance(v, str)] - return out - - -def _write_state_top( - states_dir: Path, target: str, sls_names: List[str], *, preserve: bool -) -> None: - top_path = states_dir / "top.sls" - mapping = _read_top(top_path) if preserve else {} - mapping[target] = list(sls_names) - _write_top(top_path, mapping) - - -def _write_pillar_top(pillar_dir: Path, fqdn: str, node_sls: str) -> None: - top_path = pillar_dir / "top.sls" - mapping = _read_top(top_path) - mapping[fqdn] = [node_sls] - _write_top(top_path, mapping) - - -def _write_pillar_node_data( - pillar_dir: Path, fqdn: str, salt_roles: List[SaltRole] -) -> Path: - node_base = _node_sls_basename(fqdn) - node_path = pillar_dir / "nodes" / f"{node_base}.sls" - data = { - "enroll": { - "classes": [r.sls_name for r in salt_roles], - "roles": {r.module_name: _role_pillar_values(r) for r in salt_roles}, - } - } - _write_yaml(node_path, data) - _write_pillar_top(pillar_dir, fqdn, f"nodes.{node_base}") - return node_path - - -def _clean_node_artifacts(states_dir: Path, fqdn: str) -> None: - prefix = Path(_node_file_prefix(fqdn)) - nodes_rel = prefix.parts - for files_dir in (states_dir / "roles").glob("*/files"): - target = files_dir.joinpath(*nodes_rel) - if target.exists(): - shutil.rmtree(target) - - -def _write_master_config(out: Path) -> None: - config_dir = out / "config" - config_dir.mkdir(parents=True, exist_ok=True) - (config_dir / "master.d" / "enroll.conf").parent.mkdir(parents=True, exist_ok=True) - (config_dir / "master.d" / "enroll.conf").write_text( - "# Generated by Enroll. Copy or merge into /etc/salt/master.d/enroll.conf.\n" - "file_roots:\n" - " base:\n" - " - /srv/salt\n" - "pillar_roots:\n" - " base:\n" - " - /srv/pillar\n", - encoding="utf-8", - ) - - -def _render_readme( - state: Dict[str, Any], - salt_roles: List[SaltRole], - *, - fqdn: Optional[str] = None, - node_path: Optional[Path] = None, -) -> str: - host = state.get("host", {}) if isinstance(state.get("host"), dict) else {} - hostname = host.get("hostname") or "unknown" - role_lines = markdown_list( - f"`{r.sls_name}` from Enroll role `{r.role_name}`" for r in salt_roles - ) - notes_text = markdown_list( - f"`{r.sls_name}`: {note}" for r in salt_roles for note in r.notes - ) - - if fqdn: - node_display = ( - node_path.relative_to(Path(node_path).parents[1]).as_posix() - if node_path - else "pillar/nodes/.sls" - ) - layout = f"""- `states/top.sls` targets minion `{fqdn}` to this node's generated role SLS files. -- `pillar/top.sls` targets minion `{fqdn}` to `{node_display}`. -- `pillar/nodes/*.sls` contains per-minion resource data under `enroll:roles:`. -- `states/roles//init.sls` contains reusable, data-driven Salt states. -- `states/roles//files/nodes//...` contains node-specific harvested file artifacts.""" - apply = f"""For a local dry run using the generated tree: - -```bash -sudo salt-call --local --file-root ./states --pillar-root ./pillar --id {fqdn} state.apply test=True -``` - -For master/minion use, copy or sync `states/` to your Salt state tree, copy or sync `pillar/` to your pillar tree, refresh pillar, then apply the highstate or the selected SLS files to minion `{fqdn}`.""" - else: - layout = """- `states/top.sls` targets `*` to the generated role SLS files. -- `states/roles//init.sls` contains concrete Salt states for each generated Enroll role/snapshot or common package group. -- `states/roles//files/` contains harvested file artifacts for that role or group. -- `config/master.d/enroll.conf` documents the expected Salt `file_roots` and `pillar_roots` layout if copied under `/srv`.""" - apply = """For a local dry run using the generated tree: - -```bash -sudo salt-call --local --file-root ./states state.apply test=True -``` - -For master/minion use, copy or sync `states/` to your Salt state tree and apply highstate or the selected SLS files.""" - - return f"""# Enroll Salt manifest - -Generated by Enroll from harvest data for `{hostname}`. - -This Salt target reuses the existing harvest state without changing harvesting behaviour. - -## Layout - -{layout} - -## Generated SLS roles - -{role_lines} - -## Apply / check - -{apply} - -## Generated resources - -- Native packages observed in package and service snapshots. -- Local users and groups from the users snapshot. -- Managed directories, files, and symlinks from harvested roles. -- Basic service enablement/running-state resources. -- `/etc/sysctl.d/99-enroll.conf` plus an `onchanges` sysctl apply command when present. -- Docker images by digest using guarded `docker pull` / `docker tag` command states. -- Podman images by digest using guarded `podman pull` / `podman tag` command states. -- Flatpak remotes and applications using guarded `flatpak remote-add` / `flatpak install` command states. -- Snap applications using guarded `snap install` command states. -- Live firewall runtime snapshots using staged `/etc/enroll/firewall/*` files and guarded restore command states. - -## Current limitations - -- JinjaTurtle templating is applied on a best-effort basis for file formats it recognises; unrecognised files are copied literally. -- Review generated resources before applying them broadly across unlike hosts. - -## Notes - -{notes_text} -""" - - -class SaltManifestRenderer: - """Render Salt state/pillar trees from a harvest bundle.""" - - def __init__( - self, - bundle_dir: str, - out_dir: str, - *, - fqdn: Optional[str] = None, - no_common_roles: bool = False, - jinjaturtle: str = "auto", - ) -> None: - self.bundle_dir = bundle_dir - self.out_dir = out_dir - self.fqdn = fqdn - self.no_common_roles = no_common_roles - self.jt_exe, self.jt_enabled = resolve_jinjaturtle_mode(jinjaturtle) - - def render(self) -> None: - state = SaltRole.load_state(self.bundle_dir) - fqdn_mode = bool(self.fqdn) - out = prepare_manifest_output_dir(self.out_dir, allow_existing=fqdn_mode) - states_dir = out / "states" - pillar_dir = out / "pillar" - - states_dir.mkdir(parents=True, exist_ok=True) - if fqdn_mode: - pillar_dir.mkdir(parents=True, exist_ok=True) - _clean_node_artifacts(states_dir, str(self.fqdn)) - - salt_roles = _collect_salt_roles( - state, - self.bundle_dir, - states_dir, - fqdn=self.fqdn, - no_common_roles=self.no_common_roles, - jt_exe=self.jt_exe, - jt_enabled=self.jt_enabled, - ) - - for srole in salt_roles: - role_dir = states_dir / "roles" / srole.module_name - role_dir.mkdir(parents=True, exist_ok=True) - (role_dir / "init.sls").write_text( - _render_pillar_role(srole) if fqdn_mode else _render_static_role(srole), - encoding="utf-8", - ) - - node_path: Optional[Path] = None - if fqdn_mode and self.fqdn: - node_path = _write_pillar_node_data(pillar_dir, self.fqdn, salt_roles) - _write_state_top( - states_dir, - self.fqdn, - [r.sls_name for r in salt_roles], - preserve=True, - ) - else: - _write_state_top( - states_dir, - "*", - [r.sls_name for r in salt_roles], - preserve=False, - ) - _write_master_config(out) - (out / "README.md").write_text( - _render_readme(state, salt_roles, fqdn=self.fqdn, node_path=node_path), - encoding="utf-8", - ) - - -def manifest_from_bundle_dir( - bundle_dir: str, - out_dir: str, - *, - fqdn: Optional[str] = None, - no_common_roles: bool = False, - jinjaturtle: str = "auto", -) -> None: - SaltManifestRenderer( - bundle_dir, - out_dir, - fqdn=fqdn, - no_common_roles=no_common_roles, - jinjaturtle=jinjaturtle, - ).render() diff --git a/pyproject.toml b/pyproject.toml index d02de2f..fa7d0d8 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ build-backend = "poetry.core.masonry.api" [project] name = "enroll" version = "0.7.0b7" -description = "Enroll a server's running state retrospectively into Ansible, Puppet or Salt" +description = "Enroll a server's running state retrospectively into Ansible" readme = "README.md" requires-python = ">=3.10" license = "GPL-3.0-or-later" diff --git a/tests.sh b/tests.sh index af27e9b..9513496 100755 --- a/tests.sh +++ b/tests.sh @@ -2,10 +2,6 @@ set -Eeuo pipefail -if [[ -d /opt/puppetlabs/bin ]]; then - export PATH="/opt/puppetlabs/bin:${PATH}" -fi - PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_PARENT="${TMPDIR:-/tmp}" KEEP_WORKDIR=0 @@ -22,16 +18,8 @@ BUNDLE_DIFF_DIR="${WORK_DIR}/bundle-diff" ANSIBLE_DIR="${WORK_DIR}/ansible" ANSIBLE_NO_COMMON_DIR="${WORK_DIR}/ansible-no-common" ANSIBLE_FQDN_DIR="${WORK_DIR}/ansible-fqdn" -PUPPET_DIR="${WORK_DIR}/puppet" -PUPPET_FQDN_DIR="${WORK_DIR}/puppet-fqdn" -SALT_DIR="${WORK_DIR}/salt" -SALT_FQDN_DIR="${WORK_DIR}/salt-fqdn" ANSIBLE_JINJATURTLE_DIR="${WORK_DIR}/ansible-jinjaturtle" ANSIBLE_NO_JINJATURTLE_DIR="${WORK_DIR}/ansible-no-jinjaturtle" -PUPPET_JINJATURTLE_DIR="${WORK_DIR}/puppet-jinjaturtle" -PUPPET_NO_JINJATURTLE_DIR="${WORK_DIR}/puppet-no-jinjaturtle" -SALT_JINJATURTLE_DIR="${WORK_DIR}/salt-jinjaturtle" -SALT_NO_JINJATURTLE_DIR="${WORK_DIR}/salt-no-jinjaturtle" TEST_FQDN="${ENROLL_TEST_FQDN:-enroll-ci.example.test}" JINJATURTLE_FIXTURE="${WORK_DIR}/enroll-tests-jinjaturtle.ini" ANSIBLE_PLAYBOOK_EXTRA_ARGS=() @@ -176,7 +164,6 @@ translate_packages() { gnupg) translated+=(gnupg2) ;; curl) translated+=(curl-minimal) ;; lsb-release) translated+=(redhat-lsb-core) ;; - puppet) translated+=(puppet-agent) ;; python3-apt) ;; python3-jsonschema) translated+=(python3-jsonschema) ;; python3-venv) ;; @@ -232,42 +219,6 @@ ensure_epel_repo() { DNF_UPDATED= } -ensure_salt_repo() { - if is_debian; then - if [[ -e /etc/apt/sources.list.d/salt.sources ]]; then - return - fi - section "Setup: Salt apt repository" - pkg_install ca-certificates curl gnupg - run mkdir -m 755 -p /etc/apt/keyrings - run bash -c "curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/SaltProjectKey/public | gpg --dearmor --yes -o /etc/apt/keyrings/salt-archive-keyring.pgp" - run bash -c "curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.sources > /etc/apt/sources.list.d/salt.sources" - APT_UPDATED= - elif is_rpm_family; then - if [[ -e /etc/yum.repos.d/salt.repo ]]; then - return - fi - section "Setup: Salt dnf repository" - pkg_install ca-certificates curl - run bash -c "curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.repo > /etc/yum.repos.d/salt.repo" - DNF_UPDATED= - fi -} - -ensure_puppet_repo() { - if ! is_rpm_family; then - return - fi - if rpm -q puppet8-release >/dev/null 2>&1 || [[ -e /etc/yum.repos.d/puppet8-release.repo ]]; then - return - fi - section "Setup: Puppet dnf repository" - local major - major="$(os_version_major)" - run dnf -y install "https://yum.puppet.com/puppet8-release-el-${major}.noarch.rpm" - DNF_UPDATED= -} - ensure_jinjaturtle() { section "Setup: JinjaTurtle package" if command -v jinjaturtle >/dev/null 2>&1; then @@ -311,22 +262,6 @@ ensure_ansible() { require_cmd ansible-lint "Install the ansible-lint package." } -ensure_puppet() { - ensure_puppet_repo - if ! command -v puppet >/dev/null 2>&1; then - pkg_install puppet || pkg_install puppet-agent - fi - require_cmd puppet "Install Puppet before running the Puppet noop integration tests." -} - -ensure_salt() { - ensure_salt_repo - if ! command -v salt-call >/dev/null 2>&1; then - pkg_install salt-minion || true - fi - require_cmd salt-call "Install Salt's salt-call binary before running the Salt noop integration tests. This may require configuring the upstream Salt/Broadcom package repository first." -} - run_pytests() { section "Python unit tests" cd "${PROJECT_ROOT}" @@ -396,34 +331,6 @@ run_ansible_jinjaturtle_variant() { run ansible-playbook playbook.yml -i "localhost," -c local --check --diff "${ANSIBLE_PLAYBOOK_EXTRA_ARGS[@]}" } -run_puppet_jinjaturtle_variant() { - local out_dir="$1" - local expected="$2" - local label="$3" - shift 3 - - ensure_puppet - cd "${PROJECT_ROOT}" - rm -rf "${out_dir}" - run poetry run enroll manifest --harvest "${BUNDLE_DIR}" --out "${out_dir}" --target puppet "$@" - assert_template_files "${out_dir}" "erb" "${expected}" "${label}" - run puppet apply --modulepath "${out_dir}/modules" "${out_dir}/manifests/site.pp" --noop -} - -run_salt_jinjaturtle_variant() { - local out_dir="$1" - local expected="$2" - local label="$3" - shift 3 - - ensure_salt - cd "${PROJECT_ROOT}" - rm -rf "${out_dir}" - run poetry run enroll manifest --harvest "${BUNDLE_DIR}" --out "${out_dir}" --target salt "$@" - assert_template_files "${out_dir}" "j2" "${expected}" "${label}" - run salt-call --local --retcode-passthrough --file-root "${out_dir}/states" state.apply test=True -} - run_jinjaturtle_manifest_tests() { if is_rpm_family ; then section "JinjaTurtle integration matrix" @@ -437,14 +344,6 @@ run_jinjaturtle_manifest_tests() { section "Ansible JinjaTurtle manifest noop tests" run_ansible_jinjaturtle_variant "${ANSIBLE_JINJATURTLE_DIR}" present "Ansible with JinjaTurtle on PATH" run_ansible_jinjaturtle_variant "${ANSIBLE_NO_JINJATURTLE_DIR}" absent "Ansible with --no-jinjaturtle" --no-jinjaturtle - - section "Puppet JinjaTurtle manifest noop tests" - run_puppet_jinjaturtle_variant "${PUPPET_JINJATURTLE_DIR}" present "Puppet with JinjaTurtle on PATH" - run_puppet_jinjaturtle_variant "${PUPPET_NO_JINJATURTLE_DIR}" absent "Puppet with --no-jinjaturtle" --no-jinjaturtle - - section "Salt JinjaTurtle manifest noop tests" - run_salt_jinjaturtle_variant "${SALT_JINJATURTLE_DIR}" present "Salt with JinjaTurtle on PATH" - run_salt_jinjaturtle_variant "${SALT_NO_JINJATURTLE_DIR}" absent "Salt with --no-jinjaturtle" --no-jinjaturtle } run_ansible_noop_tests() { @@ -472,43 +371,6 @@ run_ansible_noop_tests() { run ansible-playbook "playbooks/${TEST_FQDN}.yml" -i inventory/hosts.ini -c local --limit "${TEST_FQDN}" --check --diff "${ANSIBLE_PLAYBOOK_EXTRA_ARGS[@]}" } -run_puppet_noop_tests() { - section "Puppet manifest noop tests" - ensure_puppet - cd "${PROJECT_ROOT}" - rm -rf "${PUPPET_DIR}" "${PUPPET_FQDN_DIR}" - - run poetry run enroll manifest --harvest "${BUNDLE_DIR}" --out "${PUPPET_DIR}" --target puppet - run puppet apply --modulepath "${PUPPET_DIR}/modules" "${PUPPET_DIR}/manifests/site.pp" --noop - - run poetry run enroll manifest --harvest "${BUNDLE_DIR}" --out "${PUPPET_FQDN_DIR}" --target puppet --fqdn "${TEST_FQDN}" - run puppet apply \ - --modulepath "${PUPPET_FQDN_DIR}/modules" \ - --hiera_config "${PUPPET_FQDN_DIR}/hiera.yaml" \ - --certname "${TEST_FQDN}" \ - "${PUPPET_FQDN_DIR}/manifests/site.pp" \ - --noop -} - -run_salt_noop_tests() { - section "Salt manifest noop tests" - ensure_salt - cd "${PROJECT_ROOT}" - rm -rf "${SALT_DIR}" "${SALT_FQDN_DIR}" - - run poetry run enroll manifest --harvest "${BUNDLE_DIR}" --out "${SALT_DIR}" --target salt - run salt-call --local --retcode-passthrough --file-root "${SALT_DIR}/states" state.apply test=True - - run poetry run enroll manifest --harvest "${BUNDLE_DIR}" --out "${SALT_FQDN_DIR}" --target salt --fqdn "${TEST_FQDN}" - run salt-call \ - --local \ - --retcode-passthrough \ - --id "${TEST_FQDN}" \ - --file-root "${SALT_FQDN_DIR}/states" \ - --pillar-root "${SALT_FQDN_DIR}/pillar" \ - state.apply test=True -} - main() { require_root require_supported_ci_os @@ -516,8 +378,6 @@ main() { prepare_harvest_fixture configure_ansible_playbook_extra_args run_ansible_noop_tests - run_puppet_noop_tests - run_salt_noop_tests run_jinjaturtle_manifest_tests } diff --git a/tests/test_cli.py b/tests/test_cli.py index 86524c0..04cdd70 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -48,7 +48,6 @@ def test_cli_manifest_subcommand_calls_manifest(monkeypatch, tmp_path): called["fqdn"] = kwargs.get("fqdn") called["jinjaturtle"] = kwargs.get("jinjaturtle") called["no_common_roles"] = kwargs.get("no_common_roles") - called["target"] = kwargs.get("target") monkeypatch.setattr(cli, "manifest", fake_manifest) monkeypatch.setattr( @@ -70,7 +69,6 @@ def test_cli_manifest_subcommand_calls_manifest(monkeypatch, tmp_path): assert called["fqdn"] is None assert called["jinjaturtle"] == "auto" assert called["no_common_roles"] is False - assert called["target"] == "ansible" def test_cli_force_unsafe_path_before_subcommand_reaches_guard(monkeypatch, tmp_path): @@ -131,36 +129,6 @@ def test_cli_force_unsafe_path_after_subcommand_reaches_guard(monkeypatch, tmp_p assert seen["force"] is True -def test_cli_manifest_target_puppet_is_forwarded(monkeypatch, tmp_path): - called = {} - - def fake_manifest(harvest_dir: str, out_dir: str, **kwargs): - called["harvest"] = harvest_dir - called["out"] = out_dir - called["target"] = kwargs.get("target") - - monkeypatch.setattr(cli, "manifest", fake_manifest) - monkeypatch.setattr( - sys, - "argv", - [ - "enroll", - "manifest", - "--harvest", - str(tmp_path / "bundle"), - "--out", - str(tmp_path / "puppet"), - "--target", - "puppet", - ], - ) - - cli.main() - assert called["harvest"] == str(tmp_path / "bundle") - assert called["out"] == str(tmp_path / "puppet") - assert called["target"] == "puppet" - - def test_cli_manifest_no_common_roles_is_forwarded(monkeypatch, tmp_path): called = {} diff --git a/tests/test_diff_bundle.py b/tests/test_diff_bundle.py index f8c7a3b..8b559e8 100644 --- a/tests/test_diff_bundle.py +++ b/tests/test_diff_bundle.py @@ -1025,7 +1025,7 @@ def test_report_markdown_with_enforcement_failed(): }, } result = _report_markdown(report) - assert "ansible-playbook failed" in result + assert "but failed" in result def test_report_markdown_with_enforcement_skipped(): @@ -1378,7 +1378,7 @@ def test_report_text_with_enforcement_applied(): } result = d._report_text(report) assert "Enforcement" in result - assert "applied old harvest via ansible-playbook" in result + assert "applied old harvest" in result assert "tags=test" in result @@ -1398,7 +1398,7 @@ def test_report_text_with_enforcement_failed(): } result = d._report_text(report) assert "Enforcement" in result - assert "ansible-playbook failed" in result + assert "failed" in result def test_report_text_with_enforcement_skipped(): diff --git a/tests/test_diff_ignore_versions_exclude_enforce.py b/tests/test_diff_ignore_versions_exclude_enforce.py index 89e7d7c..4d28b13 100644 --- a/tests/test_diff_ignore_versions_exclude_enforce.py +++ b/tests/test_diff_ignore_versions_exclude_enforce.py @@ -218,18 +218,6 @@ def test_diff_exclude_path_only_filters_files_not_packages(tmp_path: Path): assert report["packages"]["added"] == ["htop"] -def test_enforce_old_harvest_requires_ansible_playbook(monkeypatch, tmp_path: Path): - import enroll.diff as d - - monkeypatch.setattr(d.shutil, "which", lambda name: None) - - old = tmp_path / "old" - _write_bundle(old, {"inventory": {"packages": {}}, "roles": _minimal_roles()}) - - with pytest.raises(RuntimeError, match="ansible-playbook not found"): - d.enforce_old_harvest(str(old)) - - def test_enforce_old_harvest_runs_ansible_with_tags_from_file_drift( monkeypatch, tmp_path: Path ): @@ -310,121 +298,6 @@ def test_enforce_old_harvest_runs_ansible_with_tags_from_file_drift( assert "role_usr_local_custom" in str(argv[i + 1]) -def test_enforce_old_harvest_runs_puppet_target(monkeypatch, tmp_path: Path): - import enroll.diff as d - import enroll.manifest as mf - - monkeypatch.setattr( - d.shutil, - "which", - lambda name: "/usr/bin/puppet" if name == "puppet" else None, - ) - - calls: dict[str, object] = {} - - def fake_manifest(_harvest_dir: str, out_dir: str, **kwargs): - calls["manifest_target"] = kwargs.get("target") - out = Path(out_dir) - (out / "manifests").mkdir(parents=True) - (out / "modules").mkdir(parents=True) - (out / "manifests" / "site.pp").write_text( - "node default { }\n", encoding="utf-8" - ) - - monkeypatch.setattr(mf, "manifest", fake_manifest) - - def fake_run( - argv, cwd=None, env=None, capture_output=False, text=False, check=False - ): - calls["argv"] = list(argv) - calls["cwd"] = cwd - return types.SimpleNamespace(returncode=0, stdout="ok", stderr="") - - monkeypatch.setattr(d.subprocess, "run", fake_run) - - old = tmp_path / "old" - _write_bundle(old, {"inventory": {"packages": {}}, "roles": _minimal_roles()}) - - report = { - "packages": {"added": [], "removed": ["curl"], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - } - - info = d.enforce_old_harvest(str(old), report=report, target="puppet") - - assert info["status"] == "applied" - assert info["target"] == "puppet" - assert info["tool"] == "puppet apply" - assert info["scope"] == "full_manifest" - assert info["tags"] == [] - assert calls["manifest_target"] == "puppet" - - argv = calls.get("argv") - assert argv and argv[:2] == ["/usr/bin/puppet", "apply"] - assert "--modulepath" in argv - assert any( - str(Path(calls["cwd"]) / "manifest" / "manifests" / "site.pp") == str(a) - for a in argv - ) - - -def test_enforce_old_harvest_runs_salt_target(monkeypatch, tmp_path: Path): - import enroll.diff as d - import enroll.manifest as mf - - monkeypatch.setattr( - d.shutil, - "which", - lambda name: "/usr/bin/salt-call" if name == "salt-call" else None, - ) - - calls: dict[str, object] = {} - - def fake_manifest(_harvest_dir: str, out_dir: str, **kwargs): - calls["manifest_target"] = kwargs.get("target") - out = Path(out_dir) - (out / "states").mkdir(parents=True) - (out / "states" / "top.sls").write_text("base:\n '*': []\n", encoding="utf-8") - - monkeypatch.setattr(mf, "manifest", fake_manifest) - - def fake_run( - argv, cwd=None, env=None, capture_output=False, text=False, check=False - ): - calls["argv"] = list(argv) - calls["cwd"] = cwd - return types.SimpleNamespace(returncode=0, stdout="ok", stderr="") - - monkeypatch.setattr(d.subprocess, "run", fake_run) - - old = tmp_path / "old" - _write_bundle(old, {"inventory": {"packages": {}}, "roles": _minimal_roles()}) - - report = { - "packages": {"added": [], "removed": ["curl"], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - } - - info = d.enforce_old_harvest(str(old), report=report, target="salt") - - assert info["status"] == "applied" - assert info["target"] == "salt" - assert info["tool"] == "salt-call" - assert info["scope"] == "full_manifest" - assert calls["manifest_target"] == "salt" - - argv = calls.get("argv") - assert argv and argv[0] == "/usr/bin/salt-call" - assert "--local" in argv - assert "--file-root" in argv - assert "state.apply" in argv - assert str(Path(calls["cwd"]) / "manifest" / "states") in argv - - def test_cli_diff_enforce_forwards_target(monkeypatch): import enroll.cli as cli @@ -458,17 +331,38 @@ def test_cli_diff_enforce_forwards_target(monkeypatch): "--new", "/tmp/new", "--enforce", - "--target", - "puppet", ], ) cli.main() assert calls["old"] == "/tmp/old" - assert calls["target"] == "puppet" assert calls["report"] is report +def test_cli_diff_enforce_rejects_non_ansible_target(monkeypatch): + """Salt and Puppet targets have been removed from --enforce.""" + import enroll.cli as cli + + monkeypatch.setattr( + sys, + "argv", + [ + "enroll", + "diff", + "--old", + "/tmp/old", + "--new", + "/tmp/new", + "--enforce", + "--target", + "puppet", + ], + ) + + with pytest.raises(SystemExit): + cli.main() + + def test_cli_diff_forwards_exclude_and_ignore_flags(monkeypatch, capsys): import enroll.cli as cli diff --git a/tests/test_manifest_ansible.py b/tests/test_manifest_ansible.py index 48a2721..3c7e174 100644 --- a/tests/test_manifest_ansible.py +++ b/tests/test_manifest_ansible.py @@ -136,7 +136,7 @@ def test_ansible_static_marks_harvested_jinja_values_unsafe(tmp_path: Path): payload = "{{ lookup('pipe','touch /tmp/PWNED_BY_ENROLL_ANSIBLE') }}" write_schema_state(bundle, _ansible_jinja_payload_state(payload)) - manifest.manifest(str(bundle), str(out), target="ansible") + manifest.manifest(str(bundle), str(out)) defaults = out / "roles" / "users" / "defaults" / "main.yml" text = defaults.read_text(encoding="utf-8") @@ -152,7 +152,7 @@ def test_ansible_fqdn_marks_harvested_jinja_values_unsafe(tmp_path: Path): payload = "{{ lookup('pipe','touch /tmp/PWNED_BY_ENROLL_ANSIBLE') }}" write_schema_state(bundle, _ansible_jinja_payload_state(payload)) - manifest.manifest(str(bundle), str(out), target="ansible", fqdn="host.example.test") + manifest.manifest(str(bundle), str(out), fqdn="host.example.test") hostvars = out / "inventory" / "host_vars" / "host.example.test" / "users.yml" text = hostvars.read_text(encoding="utf-8") diff --git a/tests/test_manifest_puppet.py b/tests/test_manifest_puppet.py deleted file mode 100644 index 3aac35f..0000000 --- a/tests/test_manifest_puppet.py +++ /dev/null @@ -1,1479 +0,0 @@ -from __future__ import annotations - -import json -from pathlib import Path - -import yaml - -from state_helpers import write_schema_state - -from enroll import manifest -from enroll.puppet import ( - PuppetRole, - _puppet_name, - _render_role_class, - _role_hiera_values, -) - - -def _write_state(bundle: Path, state: dict) -> None: - write_schema_state(bundle, state) - - -def test_manifest_puppet_writes_control_repo_style_output(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - artifact = bundle / "artifacts" / "foo" / "etc" / "foo.conf" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text("setting = true\n", encoding="utf-8") - sysctl_artifact = bundle / "artifacts" / "sysctl" / "sysctl" / "99-enroll.conf" - sysctl_artifact.parent.mkdir(parents=True, exist_ok=True) - sysctl_artifact.write_text("net.ipv4.ip_forward = 1\n", encoding="utf-8") - - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "users": { - "role_name": "users", - "users": [ - { - "name": "alice", - "uid": 1000, - "gid": 1000, - "gecos": "Alice Example", - "home": "/home/alice", - "shell": "/bin/bash", - "primary_group": "alice", - "supplementary_groups": ["docker"], - } - ], - "managed_dirs": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - "services": [ - { - "unit": "foo.service", - "role_name": "foo", - "packages": ["foo"], - "active_state": "active", - "sub_state": "running", - "unit_file_state": "enabled", - "condition_result": "yes", - "managed_dirs": [ - { - "path": "/etc/foo", - "owner": "root", - "group": "root", - "mode": "0755", - "reason": "parent_dir", - } - ], - "managed_files": [ - { - "path": "/etc/foo/foo.conf", - "src_rel": "etc/foo.conf", - "owner": "root", - "group": "root", - "mode": "0644", - "reason": "modified_conffile", - } - ], - "managed_links": [], - "excluded": [], - "notes": [], - } - ], - "packages": [ - { - "package": "curl", - "role_name": "curl", - "section": "net", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - "excluded": [], - "notes": [], - } - ], - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - "sysctl": { - "role_name": "sysctl", - "managed_dirs": [], - "managed_files": [ - { - "path": "/etc/sysctl.d/99-enroll.conf", - "src_rel": "sysctl/99-enroll.conf", - "owner": "root", - "group": "root", - "mode": "0644", - "reason": "system_sysctl", - } - ], - "parameters": {"net.ipv4.ip_forward": "1"}, - "notes": [], - }, - "firewall_runtime": { - "role_name": "firewall_runtime", - "packages": [], - "ipset_save": None, - "ipset_sets": [], - "iptables_v4_save": None, - "iptables_v6_save": None, - "notes": [], - }, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "include_patterns": [], - "exclude_patterns": [], - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - "excluded": [], - "notes": [], - }, - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="puppet", fqdn="test.example") - - site_pp = (out / "manifests" / "site.pp").read_text(encoding="utf-8") - assert "node 'test.example' {" in site_pp - assert "lookup('enroll::classes'" in site_pp - assert "$enroll_classes.each" in site_pp - assert "include $enroll_class" in site_pp - assert "node default {" in site_pp - - assert (out / "hiera.yaml").exists() - node_data = yaml.safe_load( - (out / "data" / "nodes" / "test.example.yaml").read_text(encoding="utf-8") - ) - assert node_data["enroll::classes"] == ["curl", "foo", "users", "sysctl"] - assert node_data["curl::packages"] == ["curl"] - assert node_data["foo::packages"] == ["foo"] - assert node_data["foo::files"]["/etc/foo/foo.conf"]["source"] == ( - "puppet:///modules/foo/nodes/test.example/etc/foo.conf" - ) - assert node_data["foo::files"]["/etc/foo/foo.conf"]["notify_services"] == [ - "foo.service" - ] - assert node_data["foo::services"]["foo.service"] == { - "ensure": "running", - "enable": True, - } - assert node_data["users::users"]["alice"]["comment"] == "Alice Example" - assert node_data["users::users"]["alice"]["groups"] == ["docker"] - assert node_data["sysctl::files"]["/etc/sysctl.d/99-enroll.conf"]["source"] == ( - "puppet:///modules/sysctl/nodes/test.example/sysctl/99-enroll.conf" - ) - - curl_pp = (out / "modules" / "curl" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "class curl" in curl_pp - assert "Array[String] $packages = []" in curl_pp - assert "package { $package_name:" in curl_pp - assert "package { 'curl':" not in curl_pp - - foo_pp = (out / "modules" / "foo" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "class foo" in foo_pp - assert "Hash[String, Hash] $files = {}" in foo_pp - assert "* => $attrs" in foo_pp - assert "package { 'foo':" not in foo_pp - assert "file { '/etc/foo/foo.conf':" not in foo_pp - - users_pp = (out / "modules" / "users" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "class users" in users_pp - assert "Hash[String, Hash] $users = {}" in users_pp - assert "user { 'alice':" not in users_pp - - sysctl_pp = (out / "modules" / "sysctl" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "class sysctl" in sysctl_pp - assert "Boolean $sysctl_apply = true" in sysctl_pp - assert "Boolean $sysctl_ignore_apply_errors = true" in sysctl_pp - assert "exec { 'enroll-apply-sysctl':" in sysctl_pp - assert ( - "if $sysctl_apply and '/etc/sysctl.d/99-enroll.conf' in $files {" in sysctl_pp - ) - - assert ( - out - / "modules" - / "foo" - / "files" - / "nodes" - / "test.example" - / "etc" - / "foo.conf" - ).exists() - assert ( - out - / "modules" - / "sysctl" - / "files" - / "nodes" - / "test.example" - / "sysctl" - / "99-enroll.conf" - ).exists() - - -def test_manifest_puppet_fqdn_package_notify_service_declared_in_same_role( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - artifact = bundle / "artifacts" / "apparmor" / "etc" / "apparmor" / "parser.conf" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text("cache-loc /var/cache/apparmor\n", encoding="utf-8") - - state = { - "schema_version": 3, - "host": {"hostname": "vpn-ssh", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {"apparmor": {"section": "admin"}}}, - "roles": { - "services": [ - { - "unit": "apparmor.service", - "role_name": "apparmor_service", - "packages": ["apparmor"], - "active_state": "active", - "unit_file_state": "enabled", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - } - ], - "packages": [ - { - "package": "apparmor", - "role_name": "apparmor", - "section": "admin", - "managed_dirs": [], - "managed_files": [ - { - "path": "/etc/apparmor/parser.conf", - "src_rel": "etc/apparmor/parser.conf", - "owner": "root", - "group": "root", - "mode": "0644", - } - ], - "managed_links": [], - } - ], - "users": { - "role_name": "users", - "users": [], - "managed_dirs": [], - "managed_files": [], - }, - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - }, - "sysctl": {"role_name": "sysctl", "managed_dirs": [], "managed_files": []}, - "firewall_runtime": {"role_name": "firewall_runtime", "packages": []}, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="puppet", fqdn="vpn-ssh") - - node_data = yaml.safe_load( - (out / "data" / "nodes" / "vpn-ssh.yaml").read_text(encoding="utf-8") - ) - assert node_data["apparmor::files"]["/etc/apparmor/parser.conf"][ - "notify_services" - ] == ["apparmor.service"] - assert node_data["apparmor::services"]["apparmor.service"] == { - "ensure": "running", - "enable": True, - } - - apparmor_pp = (out / "modules" / "apparmor" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "Hash[String, Hash] $services = {}" in apparmor_pp - assert "service { $resource_title:" in apparmor_pp - assert apparmor_pp.index("$services.each") < apparmor_pp.index("$files.each") - assert "$attrs['notify_services'].map" in apparmor_pp - assert "notify => $notify_targets" in apparmor_pp - - -def test_manifest_puppet_fqdn_mode_can_accumulate_separate_node_data( - tmp_path: Path, -): - out = tmp_path / "puppet" - - def write_bundle(name: str, content: str) -> Path: - bundle = tmp_path / name - artifact = bundle / "artifacts" / "foo" / "etc" / "foo.conf" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text(content, encoding="utf-8") - _write_state( - bundle, - { - "schema_version": 3, - "host": {"hostname": name, "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "services": [ - { - "unit": "foo.service", - "role_name": "foo", - "packages": ["foo"], - "active_state": "active", - "unit_file_state": "enabled", - "managed_dirs": [], - "managed_files": [ - { - "path": "/etc/foo/foo.conf", - "src_rel": "etc/foo.conf", - "owner": "root", - "group": "root", - "mode": "0644", - } - ], - "managed_links": [], - } - ], - "packages": [], - "users": { - "role_name": "users", - "users": [], - "managed_dirs": [], - "managed_files": [], - }, - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - }, - "sysctl": { - "role_name": "sysctl", - "managed_dirs": [], - "managed_files": [], - }, - "firewall_runtime": { - "role_name": "firewall_runtime", - "packages": [], - }, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - }, - }, - ) - return bundle - - first = write_bundle("first", "first = true\n") - second = write_bundle("second", "second = true\n") - - manifest.manifest(str(first), str(out), target="puppet", fqdn="first.example") - manifest.manifest(str(second), str(out), target="puppet", fqdn="second.example") - - assert (out / "data" / "nodes" / "first.example.yaml").exists() - assert (out / "data" / "nodes" / "second.example.yaml").exists() - - site_pp = (out / "manifests" / "site.pp").read_text(encoding="utf-8") - assert "node 'first.example' {" in site_pp - assert "node 'second.example' {" in site_pp - - first_artifact = ( - out - / "modules" - / "foo" - / "files" - / "nodes" - / "first.example" - / "etc" - / "foo.conf" - ) - second_artifact = ( - out - / "modules" - / "foo" - / "files" - / "nodes" - / "second.example" - / "etc" - / "foo.conf" - ) - assert first_artifact.read_text(encoding="utf-8") == "first = true\n" - assert second_artifact.read_text(encoding="utf-8") == "second = true\n" - - first_data = yaml.safe_load( - (out / "data" / "nodes" / "first.example.yaml").read_text(encoding="utf-8") - ) - second_data = yaml.safe_load( - (out / "data" / "nodes" / "second.example.yaml").read_text(encoding="utf-8") - ) - assert first_data["foo::files"]["/etc/foo/foo.conf"]["source"] == ( - "puppet:///modules/foo/nodes/first.example/etc/foo.conf" - ) - assert second_data["foo::files"]["/etc/foo/foo.conf"]["source"] == ( - "puppet:///modules/foo/nodes/second.example/etc/foo.conf" - ) - - -def test_manifest_puppet_uses_default_node_and_common_package_modules(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - artifact = bundle / "artifacts" / "foo" / "etc" / "foo.conf" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text("setting = true\n", encoding="utf-8") - - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": { - "packages": { - "curl": {"section": "net"}, - "foo": {"installations": [{"section": "net"}]}, - } - }, - "roles": { - "services": [ - { - "unit": "foo.service", - "role_name": "foo", - "packages": ["foo"], - "active_state": "active", - "unit_file_state": "enabled", - "managed_dirs": [], - "managed_files": [ - { - "path": "/etc/foo/foo.conf", - "src_rel": "etc/foo.conf", - "owner": "root", - "group": "root", - "mode": "0644", - } - ], - "managed_links": [], - } - ], - "packages": [ - { - "package": "curl", - "role_name": "curl", - "section": "net", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - } - ], - "users": { - "role_name": "users", - "users": [], - "managed_dirs": [], - "managed_files": [], - }, - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - }, - "sysctl": {"role_name": "sysctl", "managed_dirs": [], "managed_files": []}, - "firewall_runtime": {"role_name": "firewall_runtime", "packages": []}, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="puppet") - - site_pp = (out / "manifests" / "site.pp").read_text(encoding="utf-8") - assert site_pp == "node default {\n include net\n}\n" - - net_pp = (out / "modules" / "net" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "class net" in net_pp - assert "package { 'curl':" in net_pp - assert "package { 'foo':" in net_pp - assert "file { '/etc/foo/foo.conf':" in net_pp - assert "source => 'puppet:///modules/net/etc/foo.conf'" in net_pp - assert "notify => Service['foo.service']" in net_pp - assert "service { 'foo.service':" in net_pp - assert (out / "modules" / "net" / "files" / "etc" / "foo.conf").exists() - assert not (out / "modules" / "curl").exists() - assert not (out / "modules" / "foo").exists() - - -def test_manifest_puppet_avoids_reserved_module_names_and_duplicate_resources( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": { - "packages": { - "alpha": {"section": "admin"}, - "beta": {"section": "misc"}, - "gamma": {"section": "default"}, - } - }, - "roles": { - "packages": [ - { - "package": "alpha", - "role_name": "alpha", - "section": "admin", - "managed_dirs": [ - { - "path": "/etc/default", - "owner": "root", - "group": "root", - "mode": "0755", - } - ], - "managed_files": [], - "managed_links": [], - }, - { - "package": "beta", - "role_name": "beta", - "section": "misc", - "managed_dirs": [ - { - "path": "/etc/default", - "owner": "root", - "group": "root", - "mode": "0755", - } - ], - "managed_files": [], - "managed_links": [], - }, - { - "package": "gamma", - "role_name": "gamma", - "section": "default", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - ], - "users": { - "role_name": "users", - "users": [], - "managed_dirs": [], - "managed_files": [], - }, - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - }, - "sysctl": {"role_name": "sysctl", "managed_dirs": [], "managed_files": []}, - "firewall_runtime": {"role_name": "firewall_runtime", "packages": []}, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="puppet") - - site_pp = (out / "manifests" / "site.pp").read_text(encoding="utf-8") - assert "include default\n" not in site_pp - assert "include package_group_default" in site_pp - assert ( - out / "modules" / "package_group_default" / "manifests" / "init.pp" - ).exists() - - init_pps = "\n".join( - p.read_text(encoding="utf-8") - for p in sorted((out / "modules").glob("*/manifests/init.pp")) - ) - assert init_pps.count("file { '/etc/default':") == 1 - - -def test_manifest_rejects_unknown_target(tmp_path: Path): - bundle = tmp_path / "bundle" - _write_state(bundle, {"roles": {}}) - - try: - manifest.manifest(str(bundle), str(tmp_path / "out"), target="chef") - except ValueError as e: - assert "unsupported manifest target" in str(e) - else: - raise AssertionError("expected ValueError") - - -def test_manifest_puppet_renders_container_images_static_and_hiera(tmp_path: Path): - digest = "docker.io/library/nginx@sha256:" + "a" * 64 - podman_digest = "quay.io/example/app@sha256:" + "b" * 64 - state = { - "roles": { - "users": { - "role_name": "users", - "users": [], - "managed_dirs": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - "services": [], - "packages": [], - "container_images": { - "role_name": "container_images", - "images": [ - { - "engine": "docker", - "scope": "system", - "user": None, - "home": None, - "image_id": "sha256:" + "c" * 64, - "repo_tags": ["docker.io/library/nginx:1.27"], - "repo_digests": [digest], - "pull_ref": digest, - "tag_aliases": [ - { - "ref": "docker.io/library/nginx:1.27", - "repository": "docker.io/library/nginx", - "tag": "1.27", - } - ], - "os": "linux", - "architecture": "amd64", - "variant": None, - "platform": "linux/amd64", - "size": 123, - "created": "2026-01-01T00:00:00Z", - "source": "docker image inspect", - "notes": [], - }, - { - "engine": "podman", - "scope": "system", - "user": None, - "home": None, - "image_id": "sha256:" + "d" * 64, - "repo_tags": ["quay.io/example/app:prod"], - "repo_digests": [podman_digest], - "pull_ref": podman_digest, - "tag_aliases": [], - "os": "linux", - "architecture": "amd64", - "variant": None, - "platform": "linux/amd64", - "size": 456, - "created": "2026-01-01T00:00:00Z", - "source": "podman image inspect", - "notes": [], - }, - ], - "notes": [], - }, - } - } - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - _write_state(bundle, state) - manifest.manifest(str(bundle), str(out), target="puppet") - - site_pp = (out / "manifests" / "site.pp").read_text(encoding="utf-8") - assert "include container_images" in site_pp - pp = (out / "modules" / "container_images" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "docker::image" not in pp - assert "docker pull" in pp - assert "Docker::Image" not in pp - assert digest in pp - assert "docker tag" in pp - assert "podman pull" in pp - metadata = json.loads( - (out / "modules" / "container_images" / "metadata.json").read_text( - encoding="utf-8" - ) - ) - assert metadata["dependencies"] == [] - - fqdn_out = tmp_path / "puppet-fqdn" - manifest.manifest(str(bundle), str(fqdn_out), target="puppet", fqdn="node.example") - node_data = yaml.safe_load( - (fqdn_out / "data" / "nodes" / "node.example.yaml").read_text(encoding="utf-8") - ) - assert node_data["container_images::container_images"][0]["pull_ref"] == digest - fqdn_pp = ( - fqdn_out / "modules" / "container_images" / "manifests" / "init.pp" - ).read_text(encoding="utf-8") - assert "Array[Hash] $container_images = []" in fqdn_pp - assert "docker::image" not in fqdn_pp - assert "enroll-docker-pull-${idx}" in fqdn_pp - assert "enroll-podman-pull-${idx}" in fqdn_pp - assert "$image['pull_cmd']" in fqdn_pp - assert "podman pull" in ( - fqdn_out / "data" / "nodes" / "node.example.yaml" - ).read_text(encoding="utf-8") - - -def test_manifest_puppet_renders_firewall_runtime_resources(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - fw_dir = bundle / "artifacts" / "firewall_runtime" / "firewall" - fw_dir.mkdir(parents=True, exist_ok=True) - (fw_dir / "ipset.save").write_text( - "create blocklist hash:ip family inet\nadd blocklist 203.0.113.10\n", - encoding="utf-8", - ) - (fw_dir / "iptables.v4").write_text( - "*filter\n:INPUT DROP [0:0]\n-A INPUT -m set --match-set blocklist src -j DROP\nCOMMIT\n", - encoding="utf-8", - ) - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "firewall_runtime": { - "role_name": "firewall_runtime", - "packages": ["ipset", "iptables"], - "ipset_save": "firewall/ipset.save", - "ipset_sets": ["blocklist"], - "iptables_v4_save": "firewall/iptables.v4", - "iptables_v6_save": None, - "notes": [], - } - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="puppet") - - pp = (out / "modules" / "firewall_runtime" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - runtime_pp = ( - out / "modules" / "enroll_runtime" / "manifests" / "init.pp" - ).read_text(encoding="utf-8") - assert "file { '/etc/enroll':" in runtime_pp - assert "file { '/etc/enroll':" not in pp - assert "file { '/etc/enroll/firewall':" in pp - assert "require => File['/etc/enroll']," in pp - assert "file { '/etc/enroll/firewall/ipset.save':" in pp - assert "ipset restore -exist" in pp - assert "ipset flush blocklist" in pp - assert "iptables-restore /etc/enroll/firewall/iptables.v4" in pp - assert "refreshonly => true" in pp - assert "subscribe => File['/etc/enroll/firewall/iptables.v4']" in pp - assert "iptables-save >" not in pp - assert "Live firewall runtime snapshots were detected" not in pp - assert ( - out / "modules" / "firewall_runtime" / "files" / "firewall" / "ipset.save" - ).exists() - - fqdn_out = tmp_path / "puppet-fqdn" - manifest.manifest(str(bundle), str(fqdn_out), target="puppet", fqdn="node.example") - node_data = yaml.safe_load( - (fqdn_out / "data" / "nodes" / "node.example.yaml").read_text(encoding="utf-8") - ) - assert "enroll_runtime" in node_data["enroll::classes"] - assert "firewall_runtime" in node_data["enroll::classes"] - assert node_data["enroll::classes"].index("enroll_runtime") < node_data[ - "enroll::classes" - ].index("firewall_runtime") - assert node_data["enroll_runtime::dirs"]["/etc/enroll"]["ensure"] == "directory" - assert node_data["firewall_runtime::firewall_runtime"]["ipset_sets"] == [ - "blocklist" - ] - assert ( - "ipset restore -exist" - in node_data["firewall_runtime::firewall_runtime"]["ipset_restore_cmd"] - ) - assert ( - node_data["firewall_runtime::files"]["/etc/enroll/firewall/ipset.save"][ - "source" - ] - == "puppet:///modules/firewall_runtime/nodes/node.example/firewall/ipset.save" - ) - fqdn_pp = ( - fqdn_out / "modules" / "firewall_runtime" / "manifests" / "init.pp" - ).read_text(encoding="utf-8") - assert "Hash $firewall_runtime = {}" in fqdn_pp - assert "$firewall_runtime['ipset_restore_cmd']" in fqdn_pp - - -def test_manifest_puppet_omits_firewall_runtime_when_no_rules_were_sampled( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "firewall_runtime": { - "role_name": "firewall_runtime", - "packages": [], - "ipset_save": None, - "ipset_sets": [], - "iptables_v4_save": None, - "iptables_v6_save": None, - "notes": [ - "not running as root; live firewall runtime was not captured" - ], - } - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="puppet") - - site_pp = (out / "manifests" / "site.pp").read_text(encoding="utf-8") - assert "include enroll_runtime" not in site_pp - assert "include firewall_runtime" not in site_pp - assert not (out / "modules" / "enroll_runtime").exists() - assert not (out / "modules" / "firewall_runtime").exists() - - -def _puppet_flatpak_snap_users_snapshot() -> dict: - return { - "users": [ - { - "name": "alice", - "uid": 1000, - "primary_group": "alice", - "supplementary_groups": ["docker"], - "home": "/home/alice", - "shell": "/bin/bash", - "gecos": "Alice,,,Other", - } - ], - "user_flatpak_remotes": [ - { - "method": "user", - "user": "alice", - "name": "flathub", - "url": "https://dl.flathub.org/repo/flathub.flatpakrepo", - } - ], - "user_flatpaks": { - "alice": [ - { - "ref": "app/org.foo.App/x86_64/stable", - "remote": "flathub", - } - ] - }, - } - - -def _puppet_system_flatpak_snapshot() -> dict: - return { - "remotes": [ - { - "name": "systemrepo", - "url": "https://example.invalid/repo.flatpakrepo", - } - ], - "system_flatpaks": [ - { - "name": "org.system.App", - "remote": "systemrepo", - } - ], - } - - -def _puppet_snap_snapshot() -> dict: - return { - "system_snaps": [ - { - "name": "hello-world", - "tracking": "latest/stable", - "confinement": "classic", - }, - { - "name": "danger-snap", - "revision": "42", - "notes": ["installed with --dangerous"], - }, - ], - } - - -def test_puppet_role_renders_flatpaks_snaps_and_user_flatpaks() -> None: - role = PuppetRole("apps") - role.add_users_snapshot(_puppet_flatpak_snap_users_snapshot()) - role.add_flatpak_snapshot(_puppet_system_flatpak_snapshot()) - role.add_snap_snapshot(_puppet_snap_snapshot()) - - rendered = _render_role_class(role) - assert "group { 'alice':" in rendered - assert "user { 'alice':" in rendered - assert "flatpak --user remote-add --if-not-exists flathub" in rendered - assert "HOME=/home/alice" in rendered - assert "require => User['alice']" in rendered - assert "flatpak --user install -y flathub app/org.foo.App/x86_64/stable" in rendered - assert "flatpak --system install -y systemrepo org.system.App" in rendered - assert "snap install hello-world --channel=latest/stable --classic" in rendered - assert "snap install danger-snap --revision=42 --dangerous" in rendered - - hiera = _role_hiera_values(role) - assert hiera["apps::flatpak_remotes"][0]["environment"] == [ - "HOME=/home/alice", - "XDG_DATA_HOME=/home/alice/.local/share", - ] - assert hiera["apps::flatpaks"][0]["user"] == "alice" - assert hiera["apps::snaps"][0]["classic"] is True - assert hiera["apps::snaps"][1]["dangerous"] is True - - -def test_puppet_role_records_container_image_limitations() -> None: - role = PuppetRole("container_images") - role.add_container_images_snapshot( - { - "images": [ - "not-a-dict", - {"engine": "containerd", "pull_ref": "example.invalid/app@sha256:abc"}, - { - "engine": "docker", - "repo_tags": ["example.invalid/app:latest"], - "pull_ref": "", - }, - ], - "notes": ["image capture note"], - } - ) - - assert role.container_images == [] - assert any("has no RepoDigest" in note for note in role.notes) - assert role.notes[-1] == "image capture note" - - -def test_puppet_managed_content_notes_missing_artifacts_and_links( - tmp_path: Path, -) -> None: - bundle = tmp_path / "bundle" - module_files = tmp_path / "puppet" / "modules" / "demo" / "files" - role = PuppetRole("demo") - role.add_managed_content( - { - "managed_dirs": [ - { - "path": "/etc/demo", - "owner": "root", - "group": "root", - "mode": "0750", - } - ], - "managed_files": [ - {"path": "", "src_rel": "etc/ignored.conf"}, - {"path": "/etc/missing.conf", "src_rel": "etc/missing.conf"}, - ], - "managed_links": [ - {"path": "", "target": "/nowhere"}, - {"path": "/etc/demo/current", "target": "/opt/demo/current"}, - ], - }, - bundle_dir=str(bundle), - artifact_role="demo", - module_files_dir=module_files, - ) - - assert role.dirs["/etc/demo"]["mode"] == "0750" - assert role.links["/etc/demo/current"]["target"] == "/opt/demo/current" - assert any("Skipped /etc/missing.conf" in note for note in role.notes) - - -def test_puppet_names_are_sanitised_for_target_reserved_words() -> None: - assert _puppet_name("") == "role" - assert _puppet_name("123") == "role_123" - assert _puppet_name("node") == "role_node" - assert _puppet_name("web-app") == "web_app" - - -def test_manifest_puppet_uses_jinjaturtle_erb_templates(monkeypatch, tmp_path: Path): - import enroll.jinjaturtle as jinjaturtle_mod - from enroll.jinjaturtle import JinjifyResult - - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - artifact = bundle / "artifacts" / "foo" / "etc" / "foo.ini" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text("[main]\nkey = 1\n", encoding="utf-8") - - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "services": [ - { - "unit": "foo.service", - "role_name": "foo", - "packages": ["foo"], - "active_state": "active", - "unit_file_state": "enabled", - "managed_dirs": [], - "managed_files": [ - { - "path": "/etc/foo.ini", - "src_rel": "etc/foo.ini", - "owner": "root", - "group": "root", - "mode": "0644", - } - ], - "managed_links": [], - } - ], - "packages": [], - "users": { - "role_name": "users", - "users": [], - "managed_dirs": [], - "managed_files": [], - }, - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - }, - "sysctl": {"role_name": "sysctl", "managed_dirs": [], "managed_files": []}, - "firewall_runtime": {"role_name": "firewall_runtime", "packages": []}, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - }, - } - _write_state(bundle, state) - - monkeypatch.setattr( - jinjaturtle_mod, "find_jinjaturtle_cmd", lambda: "/usr/bin/jinjaturtle" - ) - monkeypatch.setattr(jinjaturtle_mod, "can_jinjify_path", lambda _path: True) - - calls = [] - - def fake_run_jinjaturtle( - jt_exe: str, - src_path: str, - *, - role_name: str, - force_format=None, - template_engine: str = "jinja2", - puppet_class=None, - ): - calls.append((role_name, template_engine, puppet_class)) - assert template_engine == "erb" - assert puppet_class == "foo" - return JinjifyResult( - template_text="[main]\nkey = <%= @main_key %>\n", - vars_text="foo::main_key: 1\n", - ) - - monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) - - manifest.manifest( - str(bundle), - str(out), - target="puppet", - jinjaturtle="on", - no_common_roles=True, - ) - - assert calls == [("foo", "erb", "foo")] - assert (out / "modules" / "foo" / "templates" / "etc" / "foo.ini.erb").read_text( - encoding="utf-8" - ) == "[main]\nkey = <%= @main_key %>\n" - assert not (out / "modules" / "foo" / "files" / "etc" / "foo.ini").exists() - - init_pp = (out / "modules" / "foo" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "Any $main_key = 1," in init_pp - assert "content => template('foo/etc/foo.ini.erb')" in init_pp - assert "source =>" not in init_pp - - -def test_manifest_puppet_fqdn_writes_erb_template_values_to_node_hiera( - monkeypatch, tmp_path: Path -): - import enroll.jinjaturtle as jinjaturtle_mod - from enroll.jinjaturtle import JinjifyResult - - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - artifact = bundle / "artifacts" / "foo" / "etc" / "foo.ini" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text("[main]\nkey = 1\n", encoding="utf-8") - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "services": [ - { - "unit": "foo.service", - "role_name": "foo", - "packages": ["foo"], - "active_state": "active", - "unit_file_state": "enabled", - "managed_dirs": [], - "managed_files": [ - {"path": "/etc/foo.ini", "src_rel": "etc/foo.ini"} - ], - "managed_links": [], - } - ], - "packages": [], - "users": { - "role_name": "users", - "users": [], - "managed_dirs": [], - "managed_files": [], - }, - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - }, - "sysctl": {"role_name": "sysctl", "managed_dirs": [], "managed_files": []}, - "firewall_runtime": {"role_name": "firewall_runtime", "packages": []}, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - }, - } - _write_state(bundle, state) - - monkeypatch.setattr( - jinjaturtle_mod, "find_jinjaturtle_cmd", lambda: "/usr/bin/jinjaturtle" - ) - monkeypatch.setattr(jinjaturtle_mod, "can_jinjify_path", lambda _path: True) - monkeypatch.setattr( - jinjaturtle_mod, - "run_jinjaturtle", - lambda *a, **k: JinjifyResult( - template_text="[main]\nkey = <%= @main_key %>\n", - vars_text="foo::main_key: 1\n", - ), - ) - - manifest.manifest( - str(bundle), str(out), target="puppet", fqdn="test.example", jinjaturtle="on" - ) - - node_data = yaml.safe_load( - (out / "data" / "nodes" / "test.example.yaml").read_text(encoding="utf-8") - ) - assert node_data["foo::main_key"] == 1 - assert node_data["foo::files"]["/etc/foo.ini"]["template"] == "foo/etc/foo.ini.erb" - assert "source" not in node_data["foo::files"]["/etc/foo.ini"] - init_pp = (out / "modules" / "foo" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "Any $main_key = undef," in init_pp - assert "content => template($attrs['template'])" in init_pp - - -def test_pp_quote_common_case_is_single_quoted_and_stable(): - """Values without control characters keep the historical single-quoted form.""" - from enroll.puppet import _pp_quote - - assert _pp_quote("Alice Example") == "'Alice Example'" - assert _pp_quote("0644") == "'0644'" - assert _pp_quote("/etc/nginx/nginx.conf") == "'/etc/nginx/nginx.conf'" - # Single quote and backslash keep their single-quoted escaping. - assert _pp_quote("a'b") == "'a\\'b'" - assert _pp_quote("back\\slash") == "'back\\\\slash'" - - -def test_pp_quote_neutralises_raw_control_characters(): - """A tampered harvest cannot splatter raw control characters into a manifest. - - GECOS and similar scalars are newline-delimited on a live host, so control - characters only appear via a hand-edited/tampered state.json. When present, - _pp_quote switches to a double-quoted Puppet string and escapes them rather - than emitting them verbatim. - """ - from enroll.puppet import _pp_quote - - rendered = _pp_quote("a\ntouch /tmp/pwned") - assert rendered == '"a\\ntouch /tmp/pwned"' - # No raw C0/DEL byte survives into the rendered scalar. - for value in ("a\nb", "x\r\ny", "a\tb", "a\x00b", "a\x7fb"): - out = _pp_quote(value) - assert not any(ch in out for ch in [chr(c) for c in range(0x20)] + ["\x7f"]) - - -def test_pp_quote_double_fallback_cannot_introduce_interpolation(): - """The double-quoted fallback must not enable Puppet interpolation/breakout.""" - from enroll.puppet import _pp_quote - - # $ would interpolate in a double-quoted Puppet string; it must be escaped. - out = _pp_quote("a\n${::osfamily}") - assert "\\${::osfamily}" in out - assert "${::osfamily}" not in out.replace("\\${::osfamily}", "") - # A double quote cannot terminate the string early. - out2 = _pp_quote('a\n"; notify{x:} ') - assert out2.startswith('"') and out2.endswith('"') - assert '\\"' in out2 - - -def test_manifest_puppet_user_gecos_with_newline_is_single_line(tmp_path: Path): - """End-to-end: a newline in a user's gecos yields a single-line comment.""" - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - state = { - "roles": { - "users": { - "role_name": "users", - "users": [ - { - "name": "eviluser", - "uid": 1001, - "primary_group": "evil", - "supplementary_groups": [], - "home": "/home/eviluser", - "shell": "/bin/bash", - "gecos": "Real Name\ntouch /tmp/pwned", - } - ], - "managed_files": [], - "managed_dirs": [], - "excluded": [], - "notes": [], - } - } - } - _write_state(bundle, state) - manifest.manifest(str(bundle), str(out), target="puppet") - - init_pp = (out / "modules" / "users" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - # The comment attribute must be on one line with the newline escaped. - assert 'comment => "Real Name\\ntouch /tmp/pwned"' in init_pp - # And there must be no line that is just the injected command. - assert "\ntouch /tmp/pwned\n" not in init_pp - - -def _puppet_hiera_payload_state(payload: str) -> dict: - return { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "users": { - "role_name": "users", - "users": [ - { - "name": "alice", - "uid": 1000, - "gid": 1000, - "gecos": payload, - "home": "/home/alice", - "shell": "/bin/bash", - "primary_group": "alice", - "supplementary_groups": [], - } - ], - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - "excluded": [], - "notes": [], - }, - "services": [], - "packages": [], - }, - } - - -def test_manifest_puppet_static_quotes_template_like_harvested_values( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - payload = "%{lookup('enroll::classes')}" - _write_state(bundle, _puppet_hiera_payload_state(payload)) - - manifest.manifest(str(bundle), str(out), target="puppet") - - init_pp = (out / "modules" / "users" / "manifests" / "init.pp").read_text( - encoding="utf-8" - ) - assert "comment => '%{lookup(\\'enroll::classes\\')}'" in init_pp - - -def test_manifest_puppet_hiera_escapes_harvested_interpolation_tokens( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "puppet" - payload = "%{lookup('enroll::classes')}" - _write_state(bundle, _puppet_hiera_payload_state(payload)) - - manifest.manifest(str(bundle), str(out), target="puppet", fqdn="node.example") - - node_yaml = out / "data" / "nodes" / "node.example.yaml" - text = node_yaml.read_text(encoding="utf-8") - assert payload not in text - assert "%{literal(''%'')}{lookup(''enroll::classes'')}" in text - data = yaml.safe_load(text) - assert ( - data["users::users"]["alice"]["comment"] - == "%{literal('%')}{lookup('enroll::classes')}" - ) diff --git a/tests/test_manifest_salt.py b/tests/test_manifest_salt.py deleted file mode 100644 index aa3e5ab..0000000 --- a/tests/test_manifest_salt.py +++ /dev/null @@ -1,1045 +0,0 @@ -from __future__ import annotations - -from collections import OrderedDict -from pathlib import Path - -import yaml - -from state_helpers import write_schema_state - -from enroll import manifest -from enroll.salt import ( - SaltRole, - _render_pillar_role, - _render_static_role, - _role_pillar_values, - _salt_name, - _state_id, -) - - -def _write_state(bundle: Path, state: dict) -> None: - write_schema_state(bundle, state) - - -def _sample_state() -> dict: - return { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": { - "packages": {"foo": {"section": "net"}, "curl": {"section": "net"}} - }, - "roles": { - "users": { - "role_name": "users", - "users": [ - { - "name": "alice", - "uid": 1000, - "gid": 1000, - "gecos": "Alice Example", - "home": "/home/alice", - "shell": "/bin/bash", - "primary_group": "alice", - "supplementary_groups": ["docker"], - } - ], - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - "excluded": [], - "notes": [], - }, - "services": [ - { - "unit": "foo.service", - "role_name": "foo", - "packages": ["foo"], - "active_state": "active", - "unit_file_state": "enabled", - "managed_dirs": [ - { - "path": "/etc/foo", - "owner": "root", - "group": "root", - "mode": "0755", - } - ], - "managed_files": [ - { - "path": "/etc/foo/foo.conf", - "src_rel": "etc/foo.conf", - "owner": "root", - "group": "root", - "mode": "0644", - } - ], - "managed_links": [ - {"path": "/etc/foo/enabled.conf", "target": "/etc/foo/foo.conf"} - ], - "excluded": [], - "notes": [], - } - ], - "packages": [ - { - "package": "curl", - "role_name": "curl", - "section": "net", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - "excluded": [], - "notes": [], - } - ], - "apt_config": { - "role_name": "apt_config", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - "dnf_config": { - "role_name": "dnf_config", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - "sysctl": { - "role_name": "sysctl", - "managed_dirs": [], - "managed_files": [ - { - "path": "/etc/sysctl.d/99-enroll.conf", - "src_rel": "sysctl/99-enroll.conf", - "owner": "root", - "group": "root", - "mode": "0644", - } - ], - "managed_links": [], - "parameters": {"net.ipv4.ip_forward": "1"}, - "notes": [], - }, - "firewall_runtime": {"role_name": "firewall_runtime", "packages": []}, - "etc_custom": { - "role_name": "etc_custom", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - }, - }, - } - - -def _write_sample_artifacts(bundle: Path) -> None: - artifact = bundle / "artifacts" / "foo" / "etc" / "foo.conf" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text("setting = true\n", encoding="utf-8") - sysctl_artifact = bundle / "artifacts" / "sysctl" / "sysctl" / "99-enroll.conf" - sysctl_artifact.parent.mkdir(parents=True, exist_ok=True) - sysctl_artifact.write_text("net.ipv4.ip_forward = 1\n", encoding="utf-8") - - -def test_manifest_salt_writes_single_site_state_tree(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - _write_sample_artifacts(bundle) - _write_state(bundle, _sample_state()) - - manifest.manifest(str(bundle), str(out), target="salt") - - top = yaml.safe_load((out / "states" / "top.sls").read_text(encoding="utf-8")) - assert top["base"]["*"] == ["roles.net", "roles.users", "roles.sysctl"] - - net_sls = (out / "states" / "roles" / "net" / "init.sls").read_text( - encoding="utf-8" - ) - assert "pkg.installed:" in net_sls - assert '- name: "curl"' in net_sls - assert '- name: "foo"' in net_sls - assert '"/etc/foo/foo.conf":' in net_sls - assert 'source: "salt://roles/net/files/etc/foo.conf"' in net_sls - assert "watch_in:" in net_sls - assert 'service: "enroll_service_net_foo_service_20435514"' in net_sls - assert "file.symlink:" in net_sls - assert "service.running:" in net_sls - assert (out / "states" / "roles" / "net" / "files" / "etc" / "foo.conf").exists() - - users_sls = (out / "states" / "roles" / "users" / "init.sls").read_text( - encoding="utf-8" - ) - assert "group.present:" in users_sls - assert "user.present:" in users_sls - assert "Alice Example" in users_sls - assert "optional_groups" not in users_sls - assert "- remove_groups: false" in users_sls - - sysctl_sls = (out / "states" / "roles" / "sysctl" / "init.sls").read_text( - encoding="utf-8" - ) - assert "cmd.run:" in sysctl_sls - assert "sysctl -e -p /etc/sysctl.d/99-enroll.conf || true" in sysctl_sls - assert (out / "README.md").exists() - assert (out / "config" / "master.d" / "enroll.conf").exists() - - -def test_manifest_salt_fqdn_package_watch_targets_declared_service_role( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - artifact = bundle / "artifacts" / "apparmor" / "etc" / "apparmor" / "parser.conf" - artifact.parent.mkdir(parents=True, exist_ok=True) - artifact.write_text("cache-loc /var/cache/apparmor\n", encoding="utf-8") - - state = _sample_state() - state["inventory"] = {"packages": {"apparmor": {"section": "admin"}}} - state["roles"]["services"] = [ - { - "unit": "apparmor.service", - "role_name": "apparmor_service", - "packages": ["apparmor"], - "active_state": "active", - "unit_file_state": "enabled", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - } - ] - state["roles"]["packages"] = [ - { - "package": "apparmor", - "role_name": "apparmor", - "section": "admin", - "managed_dirs": [], - "managed_files": [ - { - "path": "/etc/apparmor/parser.conf", - "src_rel": "etc/apparmor/parser.conf", - "owner": "root", - "group": "root", - "mode": "0644", - } - ], - "managed_links": [], - } - ] - state["roles"]["sysctl"] = { - "role_name": "sysctl", - "managed_dirs": [], - "managed_files": [], - "managed_links": [], - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt", fqdn="vpn-ssh") - - pillar_top = yaml.safe_load( - (out / "pillar" / "top.sls").read_text(encoding="utf-8") - ) - node_sls = pillar_top["base"]["vpn-ssh"][0] - pillar_path = out / "pillar" / Path(*node_sls.split(".")) - pillar = yaml.safe_load(pillar_path.with_suffix(".sls").read_text(encoding="utf-8")) - roles = pillar["enroll"]["roles"] - expected_service_state = _state_id( - "service", "apparmor.service", role="apparmor_service" - ) - - assert roles["apparmor"]["files"]["/etc/apparmor/parser.conf"]["watch_in"] == [ - {"service": expected_service_state} - ] - assert roles["apparmor_service"]["services"]["apparmor.service"]["state_id"] == ( - expected_service_state - ) - - -def test_manifest_salt_fqdn_mode_uses_pillar_and_accumulates_nodes(tmp_path: Path): - out = tmp_path / "salt" - - def write_bundle(name: str, content: str) -> Path: - bundle = tmp_path / name - _write_sample_artifacts(bundle) - (bundle / "artifacts" / "foo" / "etc" / "foo.conf").write_text( - content, encoding="utf-8" - ) - state = _sample_state() - state["host"]["hostname"] = name - _write_state(bundle, state) - return bundle - - first = write_bundle("first", "first=true\n") - second = write_bundle("second", "second=true\n") - - manifest.manifest(str(first), str(out), target="salt", fqdn="first.example") - manifest.manifest(str(second), str(out), target="salt", fqdn="second.example") - - state_top = yaml.safe_load((out / "states" / "top.sls").read_text(encoding="utf-8")) - assert state_top["base"]["first.example"] == [ - "roles.curl", - "roles.foo", - "roles.users", - "roles.sysctl", - ] - assert state_top["base"]["second.example"] == [ - "roles.curl", - "roles.foo", - "roles.users", - "roles.sysctl", - ] - - pillar_top = yaml.safe_load( - (out / "pillar" / "top.sls").read_text(encoding="utf-8") - ) - assert set(pillar_top["base"]) == {"first.example", "second.example"} - first_pillar_sls = pillar_top["base"]["first.example"][0] - first_node = out / "pillar" / Path(*first_pillar_sls.split(".")) - first_data = yaml.safe_load( - first_node.with_suffix(".sls").read_text(encoding="utf-8") - ) - assert first_data["enroll"]["classes"] == [ - "roles.curl", - "roles.foo", - "roles.users", - "roles.sysctl", - ] - assert first_data["enroll"]["roles"]["foo"]["packages"] == ["foo"] - assert first_data["enroll"]["roles"]["foo"]["files"]["/etc/foo/foo.conf"][ - "source" - ] == ("salt://roles/foo/files/nodes/first.example/etc/foo.conf") - - foo_sls = (out / "states" / "roles" / "foo" / "init.sls").read_text( - encoding="utf-8" - ) - assert "salt['pillar.get']('enroll:roles:foo'" in foo_sls - assert "pkg.installed:" in foo_sls - assert "file.managed:" in foo_sls - assert ( - out - / "states" - / "roles" - / "foo" - / "files" - / "nodes" - / "first.example" - / "etc" - / "foo.conf" - ).exists() - assert ( - out - / "states" - / "roles" - / "foo" - / "files" - / "nodes" - / "second.example" - / "etc" - / "foo.conf" - ).exists() - - -def test_manifest_salt_user_gecos_and_groups_are_salt_safe(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - state = _sample_state() - state["roles"]["users"]["users"][0]["name"] = "node" - state["roles"]["users"]["users"][0]["primary_group"] = "node" - state["roles"]["users"]["users"][0]["gid"] = 1000 - state["roles"]["users"]["users"][0]["gecos"] = "Node,,," - _write_sample_artifacts(bundle) - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt") - - users_sls = (out / "states" / "roles" / "users" / "init.sls").read_text( - encoding="utf-8" - ) - assert '- fullname: "Node"' in users_sls - assert "Node,,," not in users_sls - assert "optional_groups" not in users_sls - assert "- remove_groups: false" in users_sls - - -def test_manifest_salt_fqdn_user_pillar_gecos_and_groups_are_salt_safe(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - state = _sample_state() - state["roles"]["users"]["users"][0]["gecos"] = "Node,,," - _write_sample_artifacts(bundle) - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt", fqdn="node.example") - - pillar_top = yaml.safe_load( - (out / "pillar" / "top.sls").read_text(encoding="utf-8") - ) - node_sls = pillar_top["base"]["node.example"][0] - pillar_path = out / "pillar" / Path(*node_sls.split(".")) - data = yaml.safe_load(pillar_path.with_suffix(".sls").read_text(encoding="utf-8")) - alice = data["enroll"]["roles"]["users"]["users"]["alice"] - assert alice["fullname"] == "Node" - assert "Node,,," not in pillar_path.with_suffix(".sls").read_text(encoding="utf-8") - assert alice["remove_groups"] is False - assert "optional_groups" not in pillar_path.with_suffix(".sls").read_text( - encoding="utf-8" - ) - - users_sls = (out / "states" / "roles" / "users" / "init.sls").read_text( - encoding="utf-8" - ) - assert "optional_groups" not in users_sls - assert "remove_groups" in users_sls - - -def test_cli_manifest_target_salt_is_forwarded(monkeypatch, tmp_path): - import sys - - import enroll.cli as cli - - called = {} - - def fake_manifest(harvest_dir: str, out_dir: str, **kwargs): - called["harvest"] = harvest_dir - called["out"] = out_dir - called["target"] = kwargs.get("target") - - monkeypatch.setattr(cli, "manifest", fake_manifest) - monkeypatch.setattr( - sys, - "argv", - [ - "enroll", - "manifest", - "--harvest", - str(tmp_path / "bundle"), - "--out", - str(tmp_path / "salt"), - "--target", - "salt", - ], - ) - - cli.main() - assert called["harvest"] == str(tmp_path / "bundle") - assert called["out"] == str(tmp_path / "salt") - assert called["target"] == "salt" - - -def test_manifest_salt_renders_container_images_in_single_and_fqdn_modes( - tmp_path: Path, -): - digest = "docker.io/library/nginx@sha256:" + "a" * 64 - podman_digest = "quay.io/example/app@sha256:" + "b" * 64 - state = _sample_state() - state["roles"]["container_images"] = { - "role_name": "container_images", - "images": [ - { - "engine": "docker", - "scope": "system", - "user": None, - "home": None, - "image_id": "sha256:" + "c" * 64, - "repo_tags": ["docker.io/library/nginx:1.27"], - "repo_digests": [digest], - "pull_ref": digest, - "tag_aliases": [ - { - "ref": "docker.io/library/nginx:1.27", - "repository": "docker.io/library/nginx", - "tag": "1.27", - } - ], - "os": "linux", - "architecture": "amd64", - "variant": None, - "platform": "linux/amd64", - "size": 123, - "created": "2026-01-01T00:00:00Z", - "source": "docker image inspect", - "notes": [], - }, - { - "engine": "podman", - "scope": "system", - "user": None, - "home": None, - "image_id": "sha256:" + "d" * 64, - "repo_tags": ["quay.io/example/app:prod"], - "repo_digests": [podman_digest], - "pull_ref": podman_digest, - "tag_aliases": [ - { - "ref": "quay.io/example/app:prod", - "repository": "quay.io/example/app", - "tag": "prod", - } - ], - "os": "linux", - "architecture": "amd64", - "variant": None, - "platform": "linux/amd64", - "size": 456, - "created": "2026-01-01T00:00:00Z", - "source": "podman image inspect", - "notes": [], - }, - ], - "notes": [], - } - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - _write_sample_artifacts(bundle) - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt") - - top = yaml.safe_load((out / "states" / "top.sls").read_text(encoding="utf-8")) - assert "roles.container_images" in top["base"]["*"] - sls = (out / "states" / "roles" / "container_images" / "init.sls").read_text( - encoding="utf-8" - ) - assert "docker_image.present:" not in sls - assert "docker pull" in sls - assert digest in sls - assert "docker image inspect" in sls - assert "{{.Id}}" not in sls - assert "sed -n" in sls - assert "docker tag" in sls - assert "- cmd: enroll_docker_pull_container_images" in sls - assert "podman pull" in sls - assert "podman tag" in sls - - fqdn_out = tmp_path / "salt-fqdn" - manifest.manifest(str(bundle), str(fqdn_out), target="salt", fqdn="node.example") - pillar_top = yaml.safe_load( - (fqdn_out / "pillar" / "top.sls").read_text(encoding="utf-8") - ) - node_sls = pillar_top["base"]["node.example"][0] - pillar_path = fqdn_out / "pillar" / Path(*node_sls.split(".")) - pillar = yaml.safe_load(pillar_path.with_suffix(".sls").read_text(encoding="utf-8")) - assert ( - pillar["enroll"]["roles"]["container_images"]["container_images"][0]["pull_ref"] - == digest - ) - fqdn_sls = ( - fqdn_out / "states" / "roles" / "container_images" / "init.sls" - ).read_text(encoding="utf-8") - assert "docker_image.present:" not in fqdn_sls - assert "enroll_docker_pull_container_images" in fqdn_sls - assert "enroll_podman_pull_container_images" in fqdn_sls - assert "image.get('pull_cmd')" in fqdn_sls - pillar_text = pillar_path.with_suffix(".sls").read_text(encoding="utf-8") - assert "docker pull" in pillar_text - assert "docker image inspect" in pillar_text - assert "{{.Id}}" not in pillar_text - assert "sed -n" in pillar_text - assert "podman pull" in pillar_text - - -def test_manifest_salt_uses_jinjaturtle_templates(monkeypatch, tmp_path: Path): - import enroll.jinjaturtle as jinjaturtle_mod - from enroll.jinjaturtle import JinjifyResult - - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - state = _sample_state() - _write_sample_artifacts(bundle) - _write_state(bundle, state) - - monkeypatch.setattr( - jinjaturtle_mod, "find_jinjaturtle_cmd", lambda: "/usr/bin/jinjaturtle" - ) - monkeypatch.setattr(jinjaturtle_mod, "can_jinjify_path", lambda _path: True) - - def fake_run_jinjaturtle( - jt_exe: str, src_path: str, *, role_name: str, force_format=None - ): - assert jt_exe == "/usr/bin/jinjaturtle" - assert role_name == "foo" - assert src_path.endswith("artifacts/foo/etc/foo.conf") - return JinjifyResult( - template_text="setting = {{ foo_setting }}\n", - vars_text="foo_setting: true\n", - ) - - monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) - - manifest.manifest(str(bundle), str(out), target="salt", jinjaturtle="on") - - role_dir = out / "states" / "roles" / "net" - assert (role_dir / "templates" / "etc" / "foo.conf.j2").read_text( - encoding="utf-8" - ) == "setting = {{ foo_setting }}\n" - assert not (role_dir / "files" / "etc" / "foo.conf").exists() - sls = (role_dir / "init.sls").read_text(encoding="utf-8") - assert 'source: "salt://roles/net/templates/etc/foo.conf.j2"' in sls - assert 'template: "jinja"' in sls - assert "foo_setting: true" in sls - - fqdn_out = tmp_path / "salt-fqdn" - manifest.manifest( - str(bundle), - str(fqdn_out), - target="salt", - fqdn="node.example", - jinjaturtle="on", - ) - - fqdn_role_dir = fqdn_out / "states" / "roles" / "foo" - assert (fqdn_role_dir / "templates" / "etc" / "foo.conf.j2").exists() - assert not ( - fqdn_role_dir / "files" / "nodes" / "node.example" / "etc" / "foo.conf" - ).exists() - pillar_top = yaml.safe_load( - (fqdn_out / "pillar" / "top.sls").read_text(encoding="utf-8") - ) - node_sls = pillar_top["base"]["node.example"][0] - pillar_path = fqdn_out / "pillar" / Path(*node_sls.split(".")) - pillar = yaml.safe_load(pillar_path.with_suffix(".sls").read_text(encoding="utf-8")) - file_data = pillar["enroll"]["roles"]["foo"]["files"]["/etc/foo/foo.conf"] - assert file_data["source"] == "salt://roles/foo/templates/etc/foo.conf.j2" - assert file_data["watch_in"] == [ - {"service": "enroll_service_foo_foo_service_20435514"} - ] - assert file_data["template"] == "jinja" - assert file_data["context"] == {"foo_setting": True} - - -def test_manifest_salt_rewrites_jinjaturtle_json_filters(monkeypatch, tmp_path: Path): - import enroll.jinjaturtle as jinjaturtle_mod - from enroll.jinjaturtle import JinjifyResult - - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - state = _sample_state() - _write_sample_artifacts(bundle) - _write_state(bundle, state) - - monkeypatch.setattr( - jinjaturtle_mod, "find_jinjaturtle_cmd", lambda: "/usr/bin/jinjaturtle" - ) - monkeypatch.setattr(jinjaturtle_mod, "can_jinjify_path", lambda _path: True) - - def fake_run_jinjaturtle( - jt_exe: str, src_path: str, *, role_name: str, force_format=None - ): - return JinjifyResult( - template_text='{ "setting": {{ foo_setting | to_json(ensure_ascii=False) }} }\n', - vars_text='foo_setting: "alpha"\n', - ) - - monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) - - manifest.manifest(str(bundle), str(out), target="salt", jinjaturtle="on") - - template_text = ( - out / "states" / "roles" / "net" / "templates" / "etc" / "foo.conf.j2" - ).read_text(encoding="utf-8") - assert "to_json" not in template_text - assert "foo_setting__enroll_json" in template_text - sls = (out / "states" / "roles" / "net" / "init.sls").read_text(encoding="utf-8") - assert "foo_setting__enroll_json:" in sls - assert '"alpha"' in sls - - -def test_manifest_salt_pillar_role_uses_json_for_template_context() -> None: - role = SaltRole("foo") - role.add_managed_file( - "/etc/foo.json", - source="salt://roles/foo/templates/etc/foo.json.j2", - user="root", - group="root", - mode="0644", - makedirs=True, - template="jinja", - context=OrderedDict( - [("foo_name", "alpha"), ("foo_nested", OrderedDict([("x", 1)]))] - ), - ) - - pillar = _role_pillar_values(role) - assert type(pillar["files"]["/etc/foo.json"]["context"]) is dict - assert type(pillar["files"]["/etc/foo.json"]["context"]["foo_nested"]) is dict - - rendered = _render_static_role(role) - assert "foo_nested:" in rendered - context_block = ( - _render_pillar_role(role).split("context:", 1)[1].split("{% endif %}", 1)[0] - ) - assert "|yaml_encode" not in context_block - assert "|tojson" in _render_pillar_role(role) - - -def test_manifest_salt_renders_firewall_runtime_states(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - fw_dir = bundle / "artifacts" / "firewall_runtime" / "firewall" - fw_dir.mkdir(parents=True, exist_ok=True) - (fw_dir / "ipset.save").write_text( - "create blocklist hash:ip family inet\nadd blocklist 203.0.113.10\n", - encoding="utf-8", - ) - (fw_dir / "iptables.v4").write_text( - "*filter\n:INPUT DROP [0:0]\n-A INPUT -m set --match-set blocklist src -j DROP\nCOMMIT\n", - encoding="utf-8", - ) - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "firewall_runtime": { - "role_name": "firewall_runtime", - "packages": ["ipset", "iptables"], - "ipset_save": "firewall/ipset.save", - "ipset_sets": ["blocklist"], - "iptables_v4_save": "firewall/iptables.v4", - "iptables_v6_save": None, - "notes": [], - } - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt") - - top = yaml.safe_load((out / "states" / "top.sls").read_text(encoding="utf-8")) - assert "roles.enroll_runtime" in top["base"]["*"] - assert top["base"]["*"].index("roles.enroll_runtime") < top["base"]["*"].index( - "roles.firewall_runtime" - ) - runtime_sls = (out / "states" / "roles" / "enroll_runtime" / "init.sls").read_text( - encoding="utf-8" - ) - assert '"/etc/enroll":' in runtime_sls - sls = (out / "states" / "roles" / "firewall_runtime" / "init.sls").read_text( - encoding="utf-8" - ) - assert '"/etc/enroll":' not in sls - assert '"/etc/enroll/firewall":' in sls - assert '- file: "/etc/enroll"' in sls - assert '"/etc/enroll/firewall/ipset.save":' in sls - assert "ipset restore -exist" in sls - assert "ipset flush blocklist" in sls - assert "iptables-restore /etc/enroll/firewall/iptables.v4" in sls - assert " - onchanges:" in sls - assert ' - file: "/etc/enroll/firewall/iptables.v4"' in sls - assert "iptables-save >" not in sls - assert "Live firewall runtime snapshots were detected" not in sls - assert ( - out - / "states" - / "roles" - / "firewall_runtime" - / "files" - / "firewall" - / "ipset.save" - ).exists() - - fqdn_out = tmp_path / "salt-fqdn" - manifest.manifest(str(bundle), str(fqdn_out), target="salt", fqdn="node.example") - pillar_top = yaml.safe_load( - (fqdn_out / "pillar" / "top.sls").read_text(encoding="utf-8") - ) - node_sls = pillar_top["base"]["node.example"][0] - pillar_path = fqdn_out / "pillar" / Path(*node_sls.split(".")) - pillar = yaml.safe_load(pillar_path.with_suffix(".sls").read_text(encoding="utf-8")) - assert "roles.enroll_runtime" in pillar["enroll"]["classes"] - assert "firewall_runtime" in pillar["enroll"]["roles"] - assert ( - pillar["enroll"]["roles"]["enroll_runtime"]["dirs"]["/etc/enroll"]["mode"] - == "0750" - ) - role_data = pillar["enroll"]["roles"]["firewall_runtime"] - assert role_data["firewall_runtime"]["ipset_sets"] == ["blocklist"] - assert "ipset restore -exist" in role_data["firewall_runtime"]["ipset_restore_cmd"] - assert role_data["files"]["/etc/enroll/firewall/ipset.save"]["source"] == ( - "salt://roles/firewall_runtime/files/nodes/node.example/firewall/ipset.save" - ) - fqdn_sls = ( - fqdn_out / "states" / "roles" / "firewall_runtime" / "init.sls" - ).read_text(encoding="utf-8") - assert "firewall_runtime.get('ipset_restore_cmd')" in fqdn_sls - - -def test_manifest_salt_omits_firewall_runtime_when_no_rules_were_sampled( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - state = { - "schema_version": 3, - "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, - "inventory": {"packages": {}}, - "roles": { - "firewall_runtime": { - "role_name": "firewall_runtime", - "packages": [], - "ipset_save": None, - "ipset_sets": [], - "iptables_v4_save": None, - "iptables_v6_save": None, - "notes": [ - "not running as root; live firewall runtime was not captured" - ], - } - }, - } - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt") - - top = yaml.safe_load((out / "states" / "top.sls").read_text(encoding="utf-8")) - assert "roles.enroll_runtime" not in top["base"]["*"] - assert "roles.firewall_runtime" not in top["base"]["*"] - assert not (out / "states" / "roles" / "enroll_runtime").exists() - assert not (out / "states" / "roles" / "firewall_runtime").exists() - - -def _salt_flatpak_snap_users_snapshot() -> dict: - return { - "users": [ - { - "name": "alice", - "uid": 1000, - "primary_group": "alice", - "supplementary_groups": ["docker"], - "home": "/home/alice", - "shell": "/bin/bash", - "gecos": "Alice,,,Other", - } - ], - "user_flatpak_remotes": [ - { - "method": "user", - "user": "alice", - "name": "flathub", - "url": "https://dl.flathub.org/repo/flathub.flatpakrepo", - } - ], - "user_flatpaks": { - "alice": [ - { - "ref": "app/org.foo.App/x86_64/stable", - "remote": "flathub", - } - ] - }, - } - - -def _salt_system_flatpak_snapshot() -> dict: - return { - "remotes": [ - { - "name": "systemrepo", - "url": "https://example.invalid/repo.flatpakrepo", - } - ], - "system_flatpaks": [ - { - "name": "org.system.App", - "remote": "systemrepo", - } - ], - } - - -def _salt_snap_snapshot() -> dict: - return { - "system_snaps": [ - { - "name": "hello-world", - "tracking": "latest/stable", - "confinement": "classic", - }, - { - "name": "danger-snap", - "revision": "42", - "notes": ["installed with --dangerous"], - }, - ], - } - - -def test_salt_role_renders_flatpaks_snaps_and_user_flatpaks() -> None: - role = SaltRole("apps") - role.add_users_snapshot(_salt_flatpak_snap_users_snapshot()) - role.add_flatpak_snapshot(_salt_system_flatpak_snapshot()) - role.add_snap_snapshot(_salt_snap_snapshot()) - - rendered = _render_static_role(role) - assert "group.present:" in rendered - assert "user.present:" in rendered - assert "flatpak --user remote-add --if-not-exists flathub" in rendered - assert ' - HOME: "/home/alice"' in rendered - assert " - user: enroll_user_apps_alice_522b276a" in rendered - assert "flatpak --user install -y flathub app/org.foo.App/x86_64/stable" in rendered - assert "flatpak --system install -y systemrepo org.system.App" in rendered - assert "snap install hello-world --channel=latest/stable --classic" in rendered - assert "snap install danger-snap --revision=42 --dangerous" in rendered - - pillar = _role_pillar_values(role) - assert pillar["flatpak_remotes"][0]["env"] == { - "HOME": "/home/alice", - "XDG_DATA_HOME": "/home/alice/.local/share", - } - assert pillar["flatpaks"][0]["user"] == "alice" - assert pillar["snaps"][0]["classic"] is True - assert pillar["snaps"][1]["dangerous"] is True - - -def test_salt_role_records_container_image_limitations() -> None: - role = SaltRole("container_images") - role.add_container_images_snapshot( - { - "images": [ - "not-a-dict", - {"engine": "containerd", "pull_ref": "example.invalid/app@sha256:abc"}, - { - "engine": "docker", - "repo_tags": ["example.invalid/app:latest"], - "pull_ref": "", - }, - ], - "notes": ["image capture note"], - } - ) - - assert role.container_images == [] - assert any("has no RepoDigest" in note for note in role.notes) - assert role.notes[-1] == "image capture note" - - -def test_salt_managed_content_notes_missing_artifacts_and_links( - tmp_path: Path, -) -> None: - bundle = tmp_path / "bundle" - role_files = tmp_path / "salt" / "states" / "roles" / "demo" / "files" - role = SaltRole("demo") - role.add_managed_content( - { - "managed_dirs": [ - { - "path": "/etc/demo", - "owner": "root", - "group": "root", - "mode": "0750", - } - ], - "managed_files": [ - {"path": "", "src_rel": "etc/ignored.conf"}, - {"path": "/etc/missing.conf", "src_rel": "etc/missing.conf"}, - ], - "managed_links": [ - {"path": "", "target": "/nowhere"}, - {"path": "/etc/demo/current", "target": "/opt/demo/current"}, - ], - }, - bundle_dir=str(bundle), - artifact_role="demo", - role_files_dir=role_files, - ) - - assert role.dirs["/etc/demo"]["mode"] == "0750" - assert role.links["/etc/demo/current"]["target"] == "/opt/demo/current" - assert any("Skipped /etc/missing.conf" in note for note in role.notes) - - -def test_salt_names_are_sanitised_for_target_reserved_words() -> None: - assert _salt_name("") == "role" - assert _salt_name("123") == "role_123" - assert _salt_name("top") == "role_top" - assert _salt_name("web-app") == "web_app" - - -def test_manifest_salt_static_escapes_harvested_jinja_delimiters(tmp_path: Path): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - state = _sample_state() - payload = "{{ salt['cmd.run']('touch /tmp/PWNED_BY_ENROLL_SALT') }}" - state["roles"]["users"]["users"][0]["gecos"] = payload - _write_sample_artifacts(bundle) - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt") - - users_sls = (out / "states" / "roles" / "users" / "init.sls").read_text( - encoding="utf-8" - ) - assert payload not in users_sls - assert "\\u007b\\u007b salt['cmd.run']" in users_sls - - calls = [] - - class FakeCmd: - def run(self, command): - calls.append(command) - return "EXECUTED" - - from jinja2 import Template - - rendered = Template(users_sls).render(salt={"cmd.run": FakeCmd().run}) - rendered_data = yaml.safe_load(rendered) - assert calls == [] - user_state = next( - state - for state in rendered_data.values() - if isinstance(state, dict) and "user.present" in state - ) - attrs = user_state["user.present"] - fullname = next(item["fullname"] for item in attrs if "fullname" in item) - assert fullname == payload - - -def test_manifest_salt_fqdn_escapes_harvested_jinja_delimiters_in_pillar( - tmp_path: Path, -): - bundle = tmp_path / "bundle" - out = tmp_path / "salt" - state = _sample_state() - payload = "{{ salt['cmd.run']('touch /tmp/PWNED_BY_ENROLL_SALT') }}" - state["roles"]["users"]["users"][0]["gecos"] = payload - _write_sample_artifacts(bundle) - _write_state(bundle, state) - - manifest.manifest(str(bundle), str(out), target="salt", fqdn="node.example") - - pillar_top = yaml.safe_load( - (out / "pillar" / "top.sls").read_text(encoding="utf-8") - ) - node_sls = pillar_top["base"]["node.example"][0] - pillar_path = out / "pillar" / Path(*node_sls.split(".")) - text = pillar_path.with_suffix(".sls").read_text(encoding="utf-8") - assert payload not in text - assert "\\u007b\\u007b salt['cmd.run']" in text - - calls = [] - - class FakeCmd: - def run(self, command): - calls.append(command) - return "EXECUTED" - - from jinja2 import Template - - rendered = Template(text).render(salt={"cmd.run": FakeCmd().run}) - rendered_data = yaml.safe_load(rendered) - assert calls == [] - assert ( - rendered_data["enroll"]["roles"]["users"]["users"]["alice"]["fullname"] - == payload - ) diff --git a/tests/test_secret_detection.py b/tests/test_secret_detection.py new file mode 100644 index 0000000..c3074ac --- /dev/null +++ b/tests/test_secret_detection.py @@ -0,0 +1,74 @@ +"""Regression test: safe-mode content sniff must catch URI-embedded credentials +and HTTP Authorization headers, not just assignment-style credential keys.""" + +from enroll.ignore import IgnorePolicy + + +def test_uri_embedded_credentials_denied(): + pol = IgnorePolicy(dangerous=False) + leaking = [ + b"DATABASE_URL=postgresql://admin:S3cr3tPass@db:5432/app\n", + b"REDIS_URL=redis://:mypassword@localhost:6379/0\n", + b"broker_url = amqp://guest:guest@rabbit:5672//\n", + b"uri: mongodb+srv://user:p%40ss@cluster0.mongodb.net\n", + ] + for data in leaking: + assert ( + pol._content_deny_reason("/etc/svc/app.conf", data) == "sensitive_content" + ), data + + +def test_authorization_headers_denied(): + pol = IgnorePolicy(dangerous=False) + for data in [ + b"Authorization: Bearer eyJhbGciOiJIUzI1NiIs\n", + b"Authorization: Basic dXNlcjpwYXNzd29yZA==\n", + b"Proxy-Authorization: Bearer xyz\n", + ]: + assert ( + pol._content_deny_reason("/etc/svc/app.conf", data) == "sensitive_content" + ), data + + +def test_benign_urls_not_false_positive(): + pol = IgnorePolicy(dangerous=False) + for data in [ + b"homepage = https://example.com/docs\n", + b"server = http://localhost:8080/api\n", + b"origin = git://github.com/u/repo.git\n", + b"bind = http://[::1]:9000/\n", + b"contact = mailto:admin@example.com\n", + b"log_level = info\nworkers = 4\n", + ]: + assert pol._content_deny_reason("/etc/svc/app.conf", data) is None, data + + +def test_bare_word_password_denied(): + """Credential lines using the word password/passphrase/credentials without + an assignment delimiter (netrc, ppp, space/tab separated) must be denied.""" + pol = IgnorePolicy(dangerous=False) + for data in [ + b"machine api.example.com login bob password s3cret\n", + b"passphrase mysecretphrase\n", + b"credentials /run/creds/db\n", + ]: + assert ( + pol._content_deny_reason("/etc/svc.conf", data) == "sensitive_content" + ), data + + +def test_ppp_secrets_paths_denied(): + pol = IgnorePolicy(dangerous=False) + for p in ("/etc/ppp/chap-secrets", "/etc/ppp/pap-secrets", "/etc/ppp/my-secrets"): + assert pol._path_deny_reason(p) == "denied_path", p + + +def test_password_keyword_no_false_positives(): + pol = IgnorePolicy(dangerous=False) + for data in [ + b"passive_mode = true\n", + b"PassengerRoot /usr/lib\n", + b"description = reset the account\n", + b"compression = gzip\n", + ]: + assert pol._content_deny_reason("/etc/app.conf", data) is None, data From 903125976d975851d1c3c0ddbbfed01255eec924 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Sun, 28 Jun 2026 16:01:11 +1000 Subject: [PATCH 05/13] Remove 'enroll diff --enforce' option. Tighten yaml data re: handlers - use listen: instead of notify: --- CHANGELOG.md | 1 + DEVELOPMENT.md | 43 +- README.md | 22 - SECURITY.md | 12 +- enroll/ansible.py | 129 +++-- enroll/cli.py | 46 -- enroll/diff.py | 391 --------------- enroll/ignore.py | 131 ++++- enroll/render_safety.py | 104 ++++ tests/test_diff_bundle.py | 270 +---------- tests/test_diff_ignore_versions_exclude.py | 214 +++++++++ ...st_diff_ignore_versions_exclude_enforce.py | 454 ------------------ tests/test_manifest.py | 127 ++++- tests/test_render_safety.py | 108 +++++ tests/test_secret_detection.py | 87 ++++ 15 files changed, 851 insertions(+), 1288 deletions(-) create mode 100644 tests/test_diff_ignore_versions_exclude.py delete mode 100644 tests/test_diff_ignore_versions_exclude_enforce.py create mode 100644 tests/test_render_safety.py diff --git a/CHANGELOG.md b/CHANGELOG.md index e3e6942..477926a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,6 @@ # 0.7.0 (unreleased) + * BREAKING CHANGE: Remove the `enroll diff --enforce` option. Enroll no longer applies the old harvest state locally to repair drift; this avoids the risk of enforcing a potentially malicious or tampered harvest. To restore baseline state, regenerate a manifest from the trusted harvest and apply it yourself, or compare two `enroll diff` runs and act on the result. * BREAKING CHANGE: Group all package and systemd-unit roles into Debian Section/RPM Group roles by default, including managed config files and unit state. This mode is not used if `--fqdn` or `--no-common-roles` is set, in which case, the traditional behaviour of preserving one role per package/unit is used instead. * BREAKING CHANGE: Only capture user-specific .bashrc style files when using `--dangerous` mode, in case they contain sensitive env vars. * BREAKING CHANGE: Don't allow reading `.enroll.ini` in the CWD. Use only the ENROLL_CONFIG env var, an explicit `--config` path or else the XDG default location (or `~/.config/enroll/enroll.ini` if `XDG_CONFIG_HOME` is not set). diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index f149aa9..b1afaa1 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -29,14 +29,12 @@ Generated configuration-management output The harvest bundle is deliberately target-neutral. Ansible renderer consumes the same `state.json` shape and the same harvested artifacts. Renderer code should translate harvest state into the target's idioms; it should not invent source facts that belong in the harvest. -`enroll diff` is also built around harvest bundles. It compares two harvests and, when `--enforce` is requested, can generate a temporary manifest from the old harvest and apply it locally with the selected target: +`enroll diff` is also built around harvest bundles. It compares two harvests and reports drift between them: ```bash -enroll diff --old ./baseline --new ./current --enforce +enroll diff --old ./baseline --new ./current ``` -For enforcement, the user is responsible for having the chosen local apply tool on `PATH`: `ansible-playbook`. - --- ## 2. Repository layout @@ -82,7 +80,7 @@ enroll/ yamlutil.py YAML helpers used by renderers/JinjaTurtle jinjaturtle.py optional config-file templating integration - diff.py harvest comparison, notifications, and target-selected enforcement + diff.py harvest comparison and notifications explain.py human/JSON explanation of harvest contents validate.py schema and artifact consistency validation remote.py Paramiko remote harvest implementation @@ -127,7 +125,7 @@ The supported subcommands are: harvest collect a harvest bundle from a local or remote host manifest generate Ansible output from a harvest bundle single-shot run harvest and manifest in one command -diff compare two harvest bundles and optionally enforce old state +diff compare two harvest bundles and report drift explain produce a human/JSON explanation of a harvest validate validate state.json and referenced artifacts ``` @@ -148,10 +146,6 @@ flowchart TD D --> E B -->|diff| F[diff.compare_harvests] F --> G[diff.format_report] - F --> H{--enforce?} - H -->|yes| I[diff.enforce_old_harvest] - I --> J[manifest.manifest] - J --> K[ansible-playbook] B -->|explain| L[explain.explain_state] B -->|validate| M[validate.validate_harvest] ``` @@ -953,7 +947,7 @@ Ansible playbook roles are ordered intentionally: ### 13.4 Role tags -Generated playbooks tag roles with `role_`. `diff --enforce --target ansible` uses these tags to narrow enforcement to roles relevant to the drift report when it can. +Generated playbooks tag roles with `role_`, so operators can narrow a manual `ansible-playbook` run to specific roles with `--tags`. ### 13.5 Ansible and JinjaTurtle @@ -1026,7 +1020,7 @@ When checks fail, Enroll deletes obsolete generated templates when appropriate a --- -## 15. Diff, notifications, and enforcement +## 15. Diff and notifications File: `diff.py` @@ -1062,28 +1056,7 @@ Reports are formatted by: format_report(report, fmt="text" | "markdown" | "json") ``` -### 15.3 Enforcement decision - -`has_enforceable_drift()` is intentionally conservative. - -Enforceable drift includes: - -- packages that were removed from the current host but existed in the baseline, -- baseline services that were removed or changed in meaningful non-package fields, -- baseline users that were removed or changed, -- baseline files that were removed or changed. - -Not enforceable: - -- newly installed packages, -- package version changes alone, -- newly enabled services, -- newly added users, -- newly added managed files. - -This keeps `--enforce` focused on restoring baseline state rather than deleting unknown current state or downgrading packages. - -### 15.4 Notifications +### 15.3 Notifications `diff.py` also supports webhooks and email notifications: @@ -1132,8 +1105,6 @@ This is intended to answer “what did Enroll collect and why?” - `manifest.manifest()` validates before rendering Ansible output. - `diff.compare_harvests()` validates both input bundles before comparing them, using `no_schema=True` so older harvests can still be inspected while artifact safety checks remain active. -`diff --enforce` renders the old harvest through `manifest.manifest()`, so enforcement also passes through manifest-time validation before a local apply tool is invoked. - It returns a `ValidationResult` with `errors`, `warnings`, `ok()`, `to_dict()`, and `to_text()`. The CLI supports local schema override with `--schema`, warning failure with `--fail-on-warnings`, JSON/text output, and `--out`. diff --git a/README.md b/README.md index 2d88374..b4c0a52 100644 --- a/README.md +++ b/README.md @@ -172,27 +172,11 @@ Compare two harvest bundles and report what changed. - `--sops` when comparing SOPS-encrypted harvest bundles - `--exclude-path ` (repeatable) to ignore file/dir drift under matching paths (same pattern syntax as harvest) - `--ignore-package-versions` to ignore package version-only drift (upgrades/downgrades) -- `--enforce` to apply the **old** harvest state locally (requires the relevant config manager tool on `PATH` - defaults to `ansible-playbook`) -- `--enforce` runs `ansible-playbook` against the regenerated manifest) **Noise suppression** - `--exclude-path` is useful for things that change often but you still want in the harvest baseline (e.g. `/var/anacron`). - `--ignore-package-versions` keeps routine upgrades from alerting; package add/remove drift is still reported. -**Enforcement (`--enforce`)** -If a diff exists and `ansible-playbook` is on the PATH, Enroll will: -1) generate a manifest from the **old** harvest into a temporary directory -2) run the config manager tool against that manifest -3) record in the diff report that the old harvest was enforced - -Enforcement is intentionally “safe”: -- reinstalls packages that were removed (`state: present`), but does **not** attempt downgrades/pinning -- restores users, files (contents + permissions/ownership), and service enable/start state - -If the config manager tool is not on `PATH`, Enroll returns an error and does not enforce. - -**IMPORTANT**: Only enforce harvest bundles that you trust. Validation checks bundle structure and artifact safety; it does not prove that the described system state is safe to apply, e.g. hasn't been tampered with by another user with sufficient permission to do so! - **Output formats** - `--format json` (default for webhooks) @@ -508,11 +492,6 @@ enroll diff --old /path/to/harvestA --new /path/to/harvestB --exclude-path /var/ enroll diff --old /path/to/harvestA --new /path/to/harvestB --ignore-package-versions ``` -### Enforce the old harvest state when drift is detected -```bash -enroll diff --old /path/to/harvestA --new /path/to/harvestB --enforce --ignore-package-versions --exclude-path /var/anacron -``` - --- ## Explain @@ -648,7 +627,6 @@ sops = 54A91143AE0AB4F7743B01FE888ED1B423A3BC99 # ignore noisy drift exclude_path = /var/anacron ignore_package_versions = true -# enforce = true # requires ansible-playbook on PATH [single-shot] # if you use single-shot, put its defaults here. diff --git a/SECURITY.md b/SECURITY.md index a66b9aa..1e93b51 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -12,8 +12,8 @@ In particular: * If Enroll is run as root, the root user is assumed to control and understand the command line, environment, configuration file, and output location being used. * If an `enroll.ini` configuration file is loaded, its location and contents are assumed to be owned, selected, and understood by the operator. -* The operator is expected to understand the implications of options such as `--dangerous`, `--assume-safe-path`, `--sops`, `--enforce`, `--remote-host`, and `--remote-ssh-config`. -* Harvest bundles used for `manifest`, `diff`, or `diff --enforce` are assumed to come from a trusted source unless the operator is deliberately inspecting untrusted input without applying it. +* The operator is expected to understand the implications of options such as `--dangerous`, `--assume-safe-path`, `--sops`, `--remote-host`, and `--remote-ssh-config`. +* Harvest bundles used for `manifest` or `diff` are assumed to come from a trusted source unless the operator is deliberately inspecting untrusted input without applying it. * Configuration-management tools invoked by Enroll, such as Ansible, SOPS, SSH, `sudo`, Docker, Podman, Flatpak, Snap, package managers, and system utilities, are assumed to be the trusted tools the operator intended to use. ## What is in scope @@ -44,22 +44,21 @@ The following are generally out of scope and should not be reported as Enroll vu * A root user loading an `enroll.ini` file whose contents intentionally request dangerous behavior. * A root user passing `--dangerous` and then observing that Enroll may collect sensitive information. * A root user passing `--assume-safe-path` and then observing that Enroll does not prompt about `PATH` safety. -* A root user enforcing a malicious or manually edited harvest bundle with `diff --enforce`. * A user applying generated Ansible manifests from an untrusted harvest. * A user configuring a webhook, email target, SSH proxy command, SOPS binary, package manager, or configuration-management tool that they do not trust. * A compromised system where an attacker already controls root-owned files, root’s shell, root’s configuration, or the privileged tools Enroll invokes. * Reports that amount to “if root runs this tool with malicious options, root can make the system do dangerous things.” -* Enroll harvesting a file that has a *commented out* secret even with `--dangerous` disabled (it ignores comments so as to not be totally useless when it comes to harvesting config files). It is still the responsibility of the user to use `--sops` or appropriate at-rest encryption if in the slightest doubt about what might get harvested. +* Enroll harvesting a file that merely *mentions* a credential-related word in a comment with no assigned value (for example a commented-out `# token` hint in a stock config). Enroll tolerates value-less keyword mentions in comments so it is not useless for harvesting ordinary configuration files. However, a commented-out credential *value* — a populated `key = value` assignment, a URI with embedded credentials, an `Authorization` header, or private-key material — is treated as sensitive even inside a comment, because a "commented out" secret is very often a real secret that was merely disabled. Such a file is refused in default safe mode and requires `--dangerous` (ideally with `--sops`) to collect. It remains the responsibility of the user to use `--sops` or appropriate at-rest encryption if in the slightest doubt about what might get harvested. Enroll is a tool for administrators, not a sandbox for hostile local users. It cannot make unsafe local trust decisions safe if the operator’s own execution environment is already attacker-controlled. -## Trusted harvests and enforcement +## Trusted harvests Harvest bundles should be treated as sensitive and trusted administrative artifacts. A harvest may contain hostnames, usernames, package lists, service state, filesystem metadata, configuration files, firewall snapshots, container image references, Flatpak/Snap state, and other operational details. In `--dangerous` mode it may contain substantially more sensitive material. -Before running `manifest`, `diff`, or especially `diff --enforce`, the operator should be confident that the harvest bundle came from a trusted source and has not been tampered with. +Before running `manifest` or `diff`, or applying a generated manifest, the operator should be confident that the harvest bundle came from a trusted source and has not been tampered with. Enroll validates harvest structure and artifact safety. Validation can detect many unsafe filesystem constructs, such as path traversal, missing artifacts, symlinks, hardlinks, and schema mismatches. Validation does not and cannot prove that the desired state represented by a harvest is safe to apply. @@ -91,7 +90,6 @@ Less useful reports, and normally out of scope, include: * “Root can pass `--dangerous` and collect dangerous data.” * “Root can pass `--assume-safe-path` and bypass the root `PATH` warning.” * “Root can point Enroll at a malicious config file.” -* “Root can enforce a malicious harvest bundle.” * “A malicious local user can compromise Enroll after already controlling root’s environment or binaries.” Reports about concrete bypasses of Enroll's hardening are welcomed (see https://enroll.sh/security.html), but the project does not treat intentional administrator-controlled execution as a vulnerability. diff --git a/enroll/ansible.py b/enroll/ansible.py index f7ef8f4..dacfc13 100644 --- a/enroll/ansible.py +++ b/enroll/ansible.py @@ -18,7 +18,11 @@ from .manifest_safety import ( iter_safe_artifact_files, prepare_manifest_output_dir, ) -from .render_safety import ansible_unsafe_data +from .render_safety import ( + ansible_unsafe_data, + assert_generated_yaml_safe, + scaffold_token, +) from .role_names import avoid_reserved_role_name from .state import inventory_packages_from_state, roles_from_state from .yamlutil import yaml_dump_mapping, yaml_load_mapping @@ -236,7 +240,7 @@ class AnsibleRole(CMModule): self.add_snapshot_notes(snap) def render_firewall_runtime_tasks(self) -> str: - var_prefix = self.role_name + var_prefix = scaffold_token(self.role_name, field="role var_prefix") return f"""- name: Ensure firewall runtime snapshot directory exists ansible.builtin.file: path: {self.firewall_runtime_dir} @@ -292,7 +296,7 @@ class AnsibleRole(CMModule): """ def render_firewall_runtime_handlers(self) -> str: - var_prefix = self.role_name + var_prefix = scaffold_token(self.role_name, field="role var_prefix") return f"""--- - name: Flush captured ipsets before restoring members ansible.builtin.command: @@ -512,7 +516,7 @@ def _write_role_scaffold(role_dir: str) -> None: def _role_tag(role: str) -> str: """Return a stable Ansible tag name for a role. - Used by `enroll diff --enforce` to run only the roles needed to repair drift. + Lets operators run only selected roles via `--tags` when applying a manifest. """ r = str(role or "").strip() # Ansible tag charset is fairly permissive, but keep it portable and consistent. @@ -532,13 +536,17 @@ def _write_playbook_all(path: str, roles: List[str]) -> None: " roles:", ] for r in roles: - pb_lines.append(f" - role: {r}") - pb_lines.append(f" tags: [{_role_tag(r)}]") + safe = scaffold_token(r, field="role name") + pb_lines.append(f" - role: {safe}") + pb_lines.append(f" tags: [{_role_tag(safe)}]") + text = "\n".join(pb_lines) + "\n" + assert_generated_yaml_safe(text, label="playbook.yml") with open(path, "w", encoding="utf-8") as f: - f.write("\n".join(pb_lines) + "\n") + f.write(text) def _write_playbook_host(path: str, fqdn: str, roles: List[str]) -> None: + fqdn = scaffold_token(fqdn, field="site fqdn") pb_lines = [ "---", f"- name: Apply all roles on {fqdn}", @@ -548,10 +556,13 @@ def _write_playbook_host(path: str, fqdn: str, roles: List[str]) -> None: " roles:", ] for r in roles: - pb_lines.append(f" - role: {r}") - pb_lines.append(f" tags: [{_role_tag(r)}]") + safe = scaffold_token(r, field="role name") + pb_lines.append(f" - role: {safe}") + pb_lines.append(f" tags: [{_role_tag(safe)}]") + text = "\n".join(pb_lines) + "\n" + assert_generated_yaml_safe(text, label="host playbook") with open(path, "w", encoding="utf-8") as f: - f.write("\n".join(pb_lines) + "\n") + f.write(text) def _ensure_ansible_cfg(cfg_path: str) -> None: @@ -756,6 +767,14 @@ def _write_ansible_role( ctx, role_dir, role, vars_map or {}, site_defaults=site_defaults ) + # Backstop guardrail: never write a tasks/handlers document whose *structure* + # was altered by a harvested value. Enroll authors this YAML as scaffolding + # and keeps all harvested data in variable files; if any harvested value ever + # leaked into this text and changed its shape, fail closed here rather than + # emit a poisoned playbook. + assert_generated_yaml_safe(tasks, label=f"role '{role}' tasks/main.yml") + assert_generated_yaml_safe(handlers, label=f"role '{role}' handlers/main.yml") + with open(os.path.join(role_dir, "tasks", "main.yml"), "w", encoding="utf-8") as f: f.write(tasks.rstrip() + "\n") @@ -981,6 +1000,8 @@ def _render_generic_files_tasks(var_prefix: str) -> str: def _render_install_packages_tasks(role: str, var_prefix: str) -> str: """Render package installation through Ansible's generic package provider.""" + role = scaffold_token(role, field="role name") + var_prefix = scaffold_token(var_prefix, field="role var_prefix") return f"""- name: Install packages for {role} ansible.builtin.package: name: "{{{{ {var_prefix}_packages | default([]) }}}}" @@ -1145,6 +1166,7 @@ def _render_role_tasks( def _single_service_restart_handler_body(var_prefix: str) -> str: + var_prefix = scaffold_token(var_prefix, field="role var_prefix") return f"""- name: Restart service ansible.builtin.service: name: "{{{{ {var_prefix}_unit_name }}}}" @@ -1156,25 +1178,66 @@ def _single_service_restart_handler_body(var_prefix: str) -> str: """ -def _service_restart_handler_name(unit: str) -> str: - return f"Restart managed service {unit}" +def _service_restart_listen_topic(var_prefix: str) -> str: + """Return the fixed handler ``listen:`` topic for a grouped role. + + The topic is derived solely from the (already sanitized) role var_prefix and + is validated as a scaffold token. It deliberately contains NO harvested data + such as a unit name, so the same string can be embedded in both the notify + side (data) and the handler ``listen:`` (scaffolding) without ever splicing a + harvested value into YAML structure. The specific units to restart travel as + the ``_restart_units`` Ansible *variable*. + """ + + var_prefix = scaffold_token(var_prefix, field="role var_prefix") + return f"enroll_restart_grouped_services_{var_prefix}" def _grouped_service_restart_handlers_body(role: AnsibleRole) -> str: - handlers: List[str] = [] + """Render the grouped-service restart handler. + + Harvested unit names never appear in this YAML text. The handler listens on + a fixed, role-scoped topic and restarts each unit from the + ``_restart_units`` variable (written to the role's defaults / + host_vars through ``ansible_unsafe_data``). If no unit in the role is in a + "started" state, no restart handler is emitted. + """ + + has_restartable = any( + str(svc.get("state") or "stopped") == "started" + for svc in role.services.values() + ) + if not has_restartable: + return "" + + var_prefix = scaffold_token(role.var_prefix, field="role var_prefix") + topic = _service_restart_listen_topic(var_prefix) + return f"""- name: Restart managed services for {var_prefix} + ansible.builtin.service: + name: "{{{{ item }}}}" + state: restarted + loop: "{{{{ {var_prefix}_restart_units | default([]) }}}}" + listen: {topic} + when: enroll_manage_systemd_runtime | default(true) | bool +""" + + +def restart_units_for_role(role: AnsibleRole) -> List[str]: + """Return the harvested unit names a grouped role should restart, as data. + + These are emitted into the role's ``_restart_units`` variable and + consumed by the restart handler's loop. They are ordinary harvested values + and are protected by ``ansible_unsafe_data`` when the variable file is + written -- they never touch YAML scaffolding. + """ + + units: List[str] = [] for unit, svc in sorted(role.services.items()): name = str(svc.get("name") or unit).strip() if not name or str(svc.get("state") or "stopped") != "started": continue - handlers.append( - f"""- name: {_service_restart_handler_name(name)} - ansible.builtin.service: - name: {name} - state: restarted - when: enroll_manage_systemd_runtime | default(true) | bool -""" - ) - return "\n".join(_task_body(handler) for handler in handlers if _task_body(handler)) + units.append(name) + return units def _render_role_handlers( @@ -1819,16 +1882,19 @@ def _role_managed_content_vars( if notify_service_handlers and kind == "service": unit = str(snap.get("unit") or "").strip() if unit and str(snap.get("active_state") or "") == "active": - notify_other = _service_restart_handler_name(unit) + # Notify the role's fixed restart topic (scaffold-safe). The + # specific unit travels as data in _restart_units; + # it is never spliced into the handler/notify YAML text. + notify_other = _service_restart_listen_topic(role) else: notify_other = None elif notify_service_handlers and kind == "package": - notify_other = [ - _service_restart_handler_name(unit) - for unit in CMModule.active_service_units_for_package_snapshot( - snap, service_units_by_package - ) - ] + if CMModule.active_service_units_for_package_snapshot( + snap, service_units_by_package + ): + notify_other = _service_restart_listen_topic(role) + else: + notify_other = None for item in _build_managed_files_var( managed_files, @@ -2045,7 +2111,10 @@ def _render_common_ansible_roles( role, notify_by_kind={"service": None}, overwrite_templates=True, - extra_vars={f"{role.var_prefix}_systemd_units": systemd_units}, + extra_vars={ + f"{role.var_prefix}_systemd_units": systemd_units, + f"{role.var_prefix}_restart_units": restart_units_for_role(role), + }, grouped_services=True, restart_grouped_services=True, notify_service_handlers=True, diff --git a/enroll/cli.py b/enroll/cli.py index a6deb7a..934955b 100644 --- a/enroll/cli.py +++ b/enroll/cli.py @@ -14,9 +14,7 @@ from typing import Optional from .cache import new_harvest_cache_dir from .diff import ( compare_harvests, - enforce_old_harvest, format_report, - has_enforceable_drift, post_webhook, send_email, ) @@ -793,15 +791,6 @@ def main() -> None: "Package additions/removals are still reported. Useful when routine upgrades would otherwise create noisy drift." ), ) - d.add_argument( - "--enforce", - action="store_true", - help=( - "If differences are detected, attempt to enforce the old harvest state locally by generating a manifest and " - "running the selected local apply tool. " - "Enroll does not attempt to downgrade packages; if the only drift is package version upgrades (or newly installed packages), enforcement is skipped." - ), - ) d.add_argument( "--out", help="Write the report to this file instead of stdout.", @@ -1118,41 +1107,6 @@ def main() -> None: ), ) - # Optional enforcement: if drift is detected, attempt to restore the - # system to the *old* (baseline) state using ansible. - if bool(getattr(args, "enforce", False)): - if has_changes: - if not has_enforceable_drift(report): - report["enforcement"] = { - "requested": True, - "status": "skipped", - "reason": ( - "no enforceable drift detected (only additions and/or package version changes); " - "enroll does not attempt to downgrade packages" - ), - } - else: - try: - info = enforce_old_harvest( - args.old, - sops_mode=bool(getattr(args, "sops", False)), - report=report, - ) - except Exception as e: - raise SystemExit( - f"error: could not enforce old harvest state: {e}" - ) from e - report["enforcement"] = { - "requested": True, - **(info or {}), - } - else: - report["enforcement"] = { - "requested": True, - "status": "skipped", - "reason": "no differences detected", - } - txt = format_report(report, fmt=str(getattr(args, "format", "text"))) out_path = getattr(args, "out", None) if out_path: diff --git a/enroll/diff.py b/enroll/diff.py index 8122150..2d4d6ce 100644 --- a/enroll/diff.py +++ b/enroll/diff.py @@ -3,7 +3,6 @@ from __future__ import annotations import hashlib import json import os -import re import shutil import subprocess # nosec import tarfile @@ -628,321 +627,6 @@ def compare_harvests( return report, has_changes -def has_enforceable_drift(report: Dict[str, Any]) -> bool: - """Return True if the diff report contains drift that is safe/meaningful to enforce. - - Enforce mode is intended to restore *state* (files/users/services) and to - reinstall packages that were removed. - - It is deliberately conservative about package drift: - - Package *version* changes alone are not enforced (no downgrades). - - Newly installed packages are not removed. - - This helper lets the CLI decide whether `--enforce` should actually run. - """ - - pk = report.get("packages", {}) or {} - if pk.get("removed"): - return True - - sv = report.get("services", {}) or {} - # We do not try to disable newly-enabled services; we only restore units - # that were enabled in the baseline but are now missing. - if sv.get("enabled_removed") or []: - return True - - for ch in sv.get("changed", []) or []: - changes = ch.get("changes") or {} - # Ignore package set drift for enforceability decisions; package - # enforcement is handled via reinstalling removed packages, and we - # avoid trying to "undo" upgrades/renames. - for k in changes.keys(): - if k != "packages": - return True - - us = report.get("users", {}) or {} - # We restore baseline users (missing/changed). We do not remove newly-added users. - if (us.get("removed") or []) or (us.get("changed") or []): - return True - - fl = report.get("files", {}) or {} - # We restore baseline files (missing/changed). We do not delete newly-managed files. - if (fl.get("removed") or []) or (fl.get("changed") or []): - return True - - return False - - -def _role_tag(role: str) -> str: - """Return the Ansible tag name for a role (must match manifest generation).""" - r = str(role or "").strip() - safe = re.sub(r"[^A-Za-z0-9_-]+", "_", r).strip("_") - if not safe: - safe = "other" - return f"role_{safe}" - - -def _enforcement_command( - exe: str, - manifest_dir: Path, - *, - tags: Optional[List[str]] = None, -) -> Tuple[List[str], Dict[str, str]]: - """Return the local apply command and environment for a rendered manifest.""" - env = dict(os.environ) - - playbook = manifest_dir / "playbook.yml" - if not playbook.exists(): - raise RuntimeError( - f"manifest did not produce expected playbook.yml at {playbook}" - ) - - cfg = manifest_dir / "ansible.cfg" - if cfg.exists(): - env["ANSIBLE_CONFIG"] = str(cfg) - - cmd = [ - exe, - "-i", - "localhost,", - "-c", - "local", - str(playbook), - ] - if tags: - cmd.extend(["--tags", ",".join(tags)]) - return cmd, env - - -def _enforcement_plan( - report: Dict[str, Any], - old_state: Dict[str, Any], - old_bundle_dir: Path, -) -> Dict[str, Any]: - """Return a best-effort enforcement plan (roles/tags) for this diff report. - - We only plan for drift that the baseline manifest can safely restore: - - packages that were removed (reinstall, no downgrades) - - baseline users that were removed/changed - - baseline files that were removed/changed - - baseline systemd units that were disabled/changed - - We do NOT plan to remove newly-added packages/users/files/services. - """ - roles: set[str] = set() - - # --- Packages (only removals) - pk = report.get("packages", {}) or {} - removed_pkgs = set(pk.get("removed") or []) - if removed_pkgs: - pkg_to_roles: Dict[str, set[str]] = {} - - for svc in _roles(old_state).get("services") or []: - r = str(svc.get("role_name") or "").strip() - for p in svc.get("packages", []) or []: - if p: - pkg_to_roles.setdefault(str(p), set()).add(r) - - for pr in _roles(old_state).get("packages") or []: - r = str(pr.get("role_name") or "").strip() - p = pr.get("package") - if p: - pkg_to_roles.setdefault(str(p), set()).add(r) - - for p in removed_pkgs: - for r in pkg_to_roles.get(str(p), set()): - if r: - roles.add(r) - - # --- Users (removed/changed) - us = report.get("users", {}) or {} - if (us.get("removed") or []) or (us.get("changed") or []): - u = _roles(old_state).get("users") or {} - u_role = str(u.get("role_name") or "users") - if u_role: - roles.add(u_role) - - # --- Files (removed/changed) - fl = report.get("files", {}) or {} - file_paths: List[str] = [] - for e in fl.get("removed", []) or []: - if isinstance(e, dict): - p = e.get("path") - else: - p = e - if p: - file_paths.append(str(p)) - for e in fl.get("changed", []) or []: - if isinstance(e, dict): - p = e.get("path") - else: - p = e - if p: - file_paths.append(str(p)) - - if file_paths: - idx = _file_index(old_bundle_dir, old_state) - for p in file_paths: - rec = idx.get(p) - if rec and rec.role: - roles.add(str(rec.role)) - - # --- Services (enabled_removed + meaningful changes) - sv = report.get("services", {}) or {} - units: List[str] = [] - for u in sv.get("enabled_removed", []) or []: - if u: - units.append(str(u)) - for ch in sv.get("changed", []) or []: - if not isinstance(ch, dict): - continue - unit = ch.get("unit") - changes = ch.get("changes") or {} - if unit and any(k != "packages" for k in changes.keys()): - units.append(str(unit)) - - if units: - old_units = _service_units(old_state) - for u in units: - snap = old_units.get(u) - if snap and snap.get("role_name"): - roles.add(str(snap.get("role_name"))) - - # Drop empty/unknown roles. - roles = {r for r in roles if r and str(r).strip() and str(r).strip() != "unknown"} - - tags = sorted({_role_tag(r) for r in roles}) - return { - "roles": sorted(roles), - "tags": tags, - } - - -def enforce_old_harvest( - old_path: str, - *, - sops_mode: bool = False, - report: Optional[Dict[str, Any]] = None, -) -> Dict[str, Any]: - """Enforce the *old* (baseline) harvest state on the current machine. - - This renders a temporary Ansible manifest from the old harvest, then runs - the local apply command: - - ansible-playbook -i localhost, -c local playbook.yml - - Returns a dict suitable for attaching to the diff report under - report['enforcement']. - """ - - tool_exe = "ansible-playbook" - - # Import lazily to avoid heavy import cost and potential CLI cycles. - from .manifest import manifest - - started_at = _utc_now_iso() - - with ExitStack() as stack: - old_b = _bundle_from_input(old_path, sops_mode=sops_mode) - if old_b.tempdir: - stack.callback(old_b.tempdir.cleanup) - - old_state = _load_state(old_b.dir) - - plan: Optional[Dict[str, Any]] = None - tags: Optional[List[str]] = None - roles: List[str] = [] - if report is not None: - plan = _enforcement_plan(report, old_state, old_b.dir) - roles = list(plan.get("roles") or []) - # Ansible has generated per-role tags that can safely narrow the - # apply scope. - t = list(plan.get("tags") or []) - tags = t if t else None - - with tempfile.TemporaryDirectory(prefix="enroll-enforce-") as td: - td_path = Path(td) - try: - os.chmod(td_path, 0o700) - except OSError: - pass - - # 1) Generate a manifest in a temp directory. The renderer now - # refuses to write into an existing destination, so use a fresh - # child path under the secure temporary directory. - manifest_dir = td_path / "manifest" - manifest(str(old_b.dir), str(manifest_dir)) - - # 2) Apply it locally. - cmd, env = _enforcement_command( - tool_exe, - manifest_dir, - tags=tags, - ) - - spinner: Optional[_Spinner] = None - p: Optional[subprocess.CompletedProcess[str]] = None - t0 = time.monotonic() - if _progress_enabled(): - if tags: - sys.stderr.write( - f"Enforce: running {tool_exe} tags: {','.join(tags)})\n", - ) - else: - sys.stderr.write(f"Enforce: running {tool_exe}\n") - sys.stderr.flush() - spinner = _Spinner(f" {tool_exe}") - spinner.start() - - try: - p = subprocess.run( - cmd, - cwd=str(td_path), - env=env, - capture_output=True, - text=True, - check=False, - ) # nosec - finally: - if spinner: - elapsed = time.monotonic() - t0 - rc = p.returncode if p is not None else None - spinner.stop( - final_line=( - f"Enforce: {tool_exe} finished in {elapsed:0.1f}s" - + (f" (rc={rc})" if rc is not None else "") - ), - ) - - finished_at = _utc_now_iso() - - info: Dict[str, Any] = { - "status": "applied" if p.returncode == 0 else "failed", - "executable": tool_exe, - "started_at": started_at, - "finished_at": finished_at, - "command": cmd, - "returncode": int(p.returncode), - } - # Keep the Ansible-specific field for compatibility with existing - # consumers of the JSON report. - info["ansible_playbook"] = tool_exe - - info["roles"] = roles - info["tags"] = list(tags or []) - if not tags: - info["scope"] = "full_manifest" - - if p.returncode != 0: - err = (p.stderr or p.stdout or "").strip() - raise RuntimeError( - f"{tool_exe} failed" - + (f" (rc={p.returncode})" if p.returncode is not None else "") - + (f": {err}" if err else "") - ) - - return info - - def format_report(report: Dict[str, Any], *, fmt: str = "text") -> str: fmt = (fmt or "text").lower() if fmt == "json": @@ -976,38 +660,6 @@ def _report_text(report: Dict[str, Any]) -> str: msg += f" (ignored {ignored} change{'s' if ignored != 1 else ''})" lines.append(msg) - enf = report.get("enforcement") or {} - if enf: - lines.append("\nEnforcement") - status = str(enf.get("status") or "").strip().lower() - if status == "applied": - extra = "" - tags = enf.get("tags") or [] - scope = enf.get("scope") - if tags: - extra = f" (tags={','.join(str(t) for t in tags)})" - elif scope: - extra = f" ({scope})" - lines.append( - f" applied old harvest (rc={enf.get('returncode')})" - + extra - + ( - f" (finished {enf.get('finished_at')})" - if enf.get("finished_at") - else "" - ) - ) - elif status == "failed": - lines.append( - f" attempted enforcement but failed (rc={enf.get('returncode')})" - ) - elif status == "skipped": - r = enf.get("reason") - lines.append(" skipped" + (f": {r}" if r else "")) - else: - # Best-effort formatting for future fields. - lines.append(" " + json.dumps(enf, sort_keys=True)) - pk = report.get("packages", {}) lines.append("\nPackages") lines.append(f" added: {len(pk.get('added', []) or [])}") @@ -1135,49 +787,6 @@ def _report_markdown(report: Dict[str, Any]) -> str: msg += f" (ignored {ignored} change{'s' if ignored != 1 else ''})" out.append(msg + "\n") - enf = report.get("enforcement") or {} - if enf: - out.append("\n## Enforcement\n") - status = str(enf.get("status") or "").strip().lower() - if status == "applied": - extra = "" - tags = enf.get("tags") or [] - scope = enf.get("scope") - if tags: - extra = " (tags=" + ",".join(str(t) for t in tags) + ")" - elif scope: - extra = f" ({scope})" - out.append( - "- ✅ Applied old harvest" - + extra - + ( - f" (rc={enf.get('returncode')})" - if enf.get("returncode") is not None - else "" - ) - + ( - f" (finished `{enf.get('finished_at')}`)" - if enf.get("finished_at") - else "" - ) - + "\n" - ) - elif status == "failed": - out.append( - "- ⚠️ Attempted enforcement but failed" - + ( - f" (rc={enf.get('returncode')})" - if enf.get("returncode") is not None - else "" - ) - + "\n" - ) - elif status == "skipped": - r = enf.get("reason") - out.append("- Skipped" + (f": {r}" if r else "") + "\n") - else: - out.append(f"- {json.dumps(enf, sort_keys=True)}\n") - pk = report.get("packages", {}) out.append("## Packages\n") out.append(f"- Added: {len(pk.get('added', []) or [])}\n") diff --git a/enroll/ignore.py b/enroll/ignore.py index 47ee20f..b2bebaf 100644 --- a/enroll/ignore.py +++ b/enroll/ignore.py @@ -58,12 +58,35 @@ DEFAULT_ALLOW_BINARY_GLOBS = [ # --dangerous or targeted include/exclude review when a file is genuinely # needed. # -# The assignment pattern catches INI/YAML/JSON/TOML-ish keys such as: -# password: hunter2 -# "client_secret": "..." -# aws_secret_access_key = ... -# GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json -SENSITIVE_CONTENT_PATTERNS = [ +# Patterns are split into two tiers: +# +# HIGH_CONFIDENCE_SECRET_PATTERNS +# Unambiguous secret *material* (private-key blocks, age secret keys). +# A file containing one of these should never be harvested in safe mode +# regardless of how it is framed. These are scanned against the raw bytes +# and are deliberately NOT subject to comment stripping: a private key is +# a private key whether or not the surrounding format treats some line as +# a comment, and (critically) an attacker must not be able to hide key +# material from the scanner by opening a block comment (e.g. a leading +# "/*" line) that the line-oriented scanner would otherwise honour. +# +# SENSITIVE_CONTENT_PATTERNS +# The single remaining *soft* heuristic: a bare mention of a credential +# word (e.g. the literal token "password" with no assigned value). This is +# the only pattern prone to false positives on stock config files that +# ship commented-out examples, so it -- and only it -- stays comment-aware +# via iter_effective_lines(). +# +# IMPORTANT (conservative-by-default): anything that looks like an actual +# credential *value* -- a populated "key = value" assignment, a URI with +# embedded credentials, or an Authorization header -- is treated as +# high-confidence and is scanned against the RAW bytes, including inside +# comments. A "commented out" secret is very often a real secret that someone +# disabled, so Enroll deliberately refuses such a file in safe mode and requires +# --dangerous (ideally with --sops) to collect it. Only genuinely value-less +# keyword mentions remain tolerated in comments, which is what keeps Enroll +# useful for ordinary config files without risking a real disabled credential. +HIGH_CONFIDENCE_SECRET_PATTERNS = [ re.compile( rb"-----BEGIN (?:RSA |EC |OPENSSH |DSA |ENCRYPTED |PGP )?PRIVATE KEY(?: BLOCK)?-----" ), @@ -71,6 +94,13 @@ SENSITIVE_CONTENT_PATTERNS = [ re.compile(rb"(?i)AGE-SECRET-KEY-[A-Z0-9]+"), re.compile(rb"(?i)OPENSSH PRIVATE KEY"), re.compile(rb"(?i)PGP PRIVATE KEY BLOCK"), + # Assignment-style credential keys with a value, e.g. + # password: hunter2 + # "client_secret": "..." + # aws_secret_access_key = ... + # GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json + # A populated assignment is a real (if disabled) secret regardless of comment + # framing, so it is scanned on raw bytes. re.compile( rb"""(?ix) (^|[^A-Za-z0-9]) @@ -94,13 +124,8 @@ SENSITIVE_CONTENT_PATTERNS = [ \s*[:=] """ ), - re.compile( - rb"(?i)\b(pass|passwd|password|passphrase|token|secret|" - rb"credentials?|api[_-]?key)\b" - ), # Credentials embedded in connection-string URIs, e.g. # postgres://user:pass@host, redis://:pass@host, amqp://u:p@host - # The keyword regex above keys on assignment-style names and misses these. re.compile(rb"(?i)[a-z][a-z0-9+.-]*://[^/\s:@]*:[^/\s@]+@[^/\s]"), # HTTP(S) Authorization / Proxy-Authorization header values carrying a # bearer/basic/digest credential. @@ -110,6 +135,16 @@ SENSITIVE_CONTENT_PATTERNS = [ ), ] +SENSITIVE_CONTENT_PATTERNS = [ + # Bare credential-word mention with no assigned value. This is the only soft + # heuristic and the only one tolerated inside comments, because stock config + # files legitimately ship value-less commented hints (e.g. "# token"). + re.compile( + rb"(?i)\b(pass|passwd|password|passphrase|token|secret|" + rb"credentials?|api[_-]?key)\b" + ), +] + COMMENT_PREFIXES = (b"#", b";", b"//") BLOCK_START = b"/*" BLOCK_END = b"*/" @@ -168,23 +203,61 @@ class IgnorePolicy: if self.allow_binary_globs is None: self.allow_binary_globs = list(DEFAULT_ALLOW_BINARY_GLOBS) + def _strip_balanced_block_comments(self, content: bytes) -> bytes: + """Remove only *properly closed* ``/* ... */`` regions from *content*. + + The previous line-oriented scanner entered "block comment mode" the moment + a line began with ``/*`` and then skipped every subsequent line until it + saw ``*/``. That had two problems an attacker (or just an unusual config) + could exploit to hide secrets from the content scan: + + * an *unterminated* ``/*`` (no closing ``*/`` anywhere after it) masked + the entire remainder of the file, including real key material; and + * content appearing *after* a ``*/`` on the same line, or a one-line + ``/* ... */ secret``, was dropped. + + Operating on the whole byte stream and removing only *balanced* comment + regions fixes both: an opening ``/*`` with no matching ``*/`` is left in + place (so its contents are still scanned), and bytes after a closing + ``*/`` are preserved. Each removed region is replaced with a single + space so tokens on either side cannot be accidentally joined. + """ + + out = bytearray() + i = 0 + n = len(content) + while i < n: + start = content.find(BLOCK_START, i) + if start == -1: + out += content[i:] + break + end = content.find(BLOCK_END, start + len(BLOCK_START)) + if end == -1: + # Unterminated block comment: do NOT treat the rest of the file as + # commented out. Keep it verbatim so secret scanning still runs. + out += content[i:] + break + out += content[i:start] + out += b" " + i = end + len(BLOCK_END) + return bytes(out) + def iter_effective_lines(self, content: bytes): - in_block = False - for raw in content.splitlines(): + """Yield non-comment lines for the soft content heuristics. + + Block comments are removed first (only when properly closed), then + single-line comment prefixes are skipped. Genuinely commented-out + secrets remain ignored -- that is deliberate, documented behaviour -- but + a block-comment open can no longer suppress scanning of later, real + content. + """ + + for raw in self._strip_balanced_block_comments(content).splitlines(): line = raw.lstrip() - if in_block: - if BLOCK_END in line: - in_block = False - continue - if not line: continue - if line.startswith(BLOCK_START): - in_block = True - continue - if line.startswith(COMMENT_PREFIXES) or line.startswith(b"*"): continue @@ -221,6 +294,20 @@ class IgnorePolicy: return "binary_like" if not self.dangerous: + # High-confidence secret *material* (private keys, age secret keys) + # is scanned against the raw bytes and is NOT subject to comment + # stripping. A private key embedded in a file is sensitive regardless + # of comment framing, and this closes the bypass where opening a block + # comment (e.g. a leading "/*" line) hid key material from the + # line-oriented scanner. + for pat in HIGH_CONFIDENCE_SECRET_PATTERNS: + if pat.search(data): + return "sensitive_content" + + # Softer assignment/keyword/URI heuristics stay comment-aware so a + # genuinely commented-out example does not make Enroll useless for + # ordinary config files. iter_effective_lines() is hardened so an + # unterminated/inline block comment cannot mask later real content. for line in self.iter_effective_lines(data): for pat in SENSITIVE_CONTENT_PATTERNS: if pat.search(line): diff --git a/enroll/render_safety.py b/enroll/render_safety.py index 0c13f64..f538276 100644 --- a/enroll/render_safety.py +++ b/enroll/render_safety.py @@ -1,5 +1,6 @@ from __future__ import annotations +import re from collections.abc import Mapping, Set as AbstractSet from typing import Any @@ -7,6 +8,109 @@ from typing import Any ANSIBLE_JINJA_STARTS = ("{{", "{%", "{#") +class RenderSafetyError(RuntimeError): + """Raised when generated configuration-management text is unsafe. + + This is a *generation-time* guardrail. It fires when Enroll would otherwise + emit raw scaffolding (task/handler YAML) built from a value that is not a + known-safe Enroll-controlled token, or when a generated task/handler file + does not round-trip to the structure Enroll intended. Either case means a + harvested value has leaked into playbook *structure* instead of staying in + Ansible *data*, so Enroll fails closed rather than writing a poisoned + manifest. + """ + + +# The only characters Enroll ever needs inside raw YAML scaffolding are those +# that make up sanitized role/module identifiers and a handful of fixed English +# words in task names. Anything outside this set must travel as Ansible *data* +# (a variable consumed via ``{{ ... }}``), never as scaffolding text. +# +# This is deliberately strict: it matches the charset produced by +# ``ansible._role_id`` / ``package_hints.role_id`` plus spaces (for the fixed +# human-readable portions of task names that Enroll itself authors). It does NOT +# permit quotes, colons, newlines, braces, or any YAML metacharacter, so a value +# that passes this gate cannot alter document structure. +_SCAFFOLD_TOKEN_RE = re.compile(r"^[A-Za-z0-9_][A-Za-z0-9_ .-]*$") + + +def scaffold_token(value: str, *, field: str = "scaffold token") -> str: + """Return *value* only if it is safe to splice into raw YAML scaffolding. + + Use this for the *only* legitimate reason to interpolate a dynamic value + into hand-written YAML text: an Enroll-generated, already-sanitized + identifier such as a role name or module name. Harvested free-text (unit + names, file paths, package descriptions, user fields, ...) must never be + passed here -- it belongs in a variable file via :func:`ansible_unsafe_data` + and must be referenced from tasks through ``{{ ... }}`` indirection. + + Raising rather than escaping is intentional. Escaping invites a long tail of + "did we cover every YAML metacharacter / Jinja delimiter / indentation + trick" bugs. A hard allowlist makes structural injection impossible by + construction: if a caller ever tries to splice harvested data into + scaffolding, generation aborts loudly instead of silently producing unsafe + output. + """ + + text = "" if value is None else str(value) + if not _SCAFFOLD_TOKEN_RE.fullmatch(text): + raise RenderSafetyError( + f"refusing to interpolate unsafe {field} into generated YAML " + f"scaffolding: {text!r}. Harvested values must be passed as Ansible " + f"data (a variable rendered through ansible_unsafe_data), not spliced " + f"into task/handler text." + ) + return text + + +def assert_generated_yaml_safe(text: str, *, label: str) -> None: + """Verify a block of Enroll-generated task/handler YAML is well-formed. + + This is the backstop for the scaffold-token rule. Even if a future change + reintroduces raw interpolation of a harvested value, this check parses the + generated YAML and confirms it is a list of mappings (Ansible task/handler + shape) with only string keys -- the structure Enroll intends. A harvested + value that injects an extra list item, a non-mapping entry, a non-string + key, or breaks parsing entirely will trip this and abort generation. + + It is intentionally structural, not value-level: it does not try to judge + whether a *value* is dangerous (that is what ``ansible_unsafe_data`` handles + for variable files). It ensures harvested data cannot change the *shape* of + a generated tasks/handlers document. + """ + + import yaml + + try: + doc = yaml.safe_load(text) + except yaml.YAMLError as e: # pragma: no cover - exercised via tests + raise RenderSafetyError( + f"generated {label} is not valid YAML; a harvested value likely " + f"broke document structure: {e}" + ) from e + + if doc is None: + # An empty "---\n" document is a legitimate "no tasks/handlers" result. + return + if not isinstance(doc, list): + raise RenderSafetyError( + f"generated {label} is not a YAML list of tasks/handlers; a " + f"harvested value likely altered document structure" + ) + for entry in doc: + if not isinstance(entry, Mapping): + raise RenderSafetyError( + f"generated {label} contains a non-mapping entry; a harvested " + f"value likely injected list structure: {entry!r}" + ) + for key in entry.keys(): + if not isinstance(key, str): + raise RenderSafetyError( + f"generated {label} contains a non-string task key; a " + f"harvested value likely injected mapping structure: {key!r}" + ) + + class AnsibleUnsafeText(str): """String subclass dumped as Ansible's ``!unsafe`` YAML scalar. diff --git a/tests/test_diff_bundle.py b/tests/test_diff_bundle.py index 8b559e8..34780eb 100644 --- a/tests/test_diff_bundle.py +++ b/tests/test_diff_bundle.py @@ -6,11 +6,9 @@ from pathlib import Path import pytest +from enroll.ansible import _role_tag from enroll.diff import ( _Spinner, - _enforcement_plan, - has_enforceable_drift, - _role_tag, _utc_now_iso, _report_markdown, ) @@ -798,150 +796,6 @@ def test_role_tag_empty(): assert _role_tag(" ") == "role_other" -def test_has_enforceable_drift_packages_removed(): - report = {"packages": {"removed": ["vim"]}} - assert has_enforceable_drift(report) is True - - -def test_has_enforceable_drift_services_removed(): - report = {"services": {"enabled_removed": ["nginx.service"]}} - assert has_enforceable_drift(report) is True - - -def test_has_enforceable_drift_service_changed(): - report = { - "services": { - "changed": [ - { - "unit": "nginx.service", - "changes": {"active_state": {"old": "active", "new": "inactive"}}, - } - ] - } - } - assert has_enforceable_drift(report) is True - - -def test_has_enforceable_drift_service_package_only_changed(): - # Service changed only in packages - should NOT be enforceable - report = { - "services": { - "changed": [ - { - "unit": "nginx.service", - "changes": {"packages": {"added": ["nginx-extra"]}}, - } - ] - } - } - assert has_enforceable_drift(report) is False - - -def test_has_enforceable_drift_users_removed(): - report = {"users": {"removed": ["alice"]}} - assert has_enforceable_drift(report) is True - - -def test_has_enforceable_drift_users_changed(): - report = { - "users": { - "changed": [ - {"name": "alice", "changes": {"uid": {"old": 1000, "new": 1001}}} - ] - } - } - assert has_enforceable_drift(report) is True - - -def test_has_enforceable_drift_files_removed(): - report = { - "files": { - "removed": [{"path": "/etc/passwd", "role": "users", "reason": "conffile"}] - } - } - assert has_enforceable_drift(report) is True - - -def test_has_enforceable_drift_files_changed(): - report = { - "files": { - "changed": [ - { - "path": "/etc/passwd", - "changes": {"content": {"old": "sha1", "new": "sha2"}}, - } - ] - } - } - assert has_enforceable_drift(report) is True - - -def test_has_enforceable_drift_no_drift(): - report = { - "packages": {"added": ["newpkg"]}, - "services": {"enabled_added": ["new.service"]}, - "users": {"added": ["bob"]}, - "files": {"added": ["/opt/newfile"]}, - } - assert has_enforceable_drift(report) is False - - -def test_enforcement_plan_packages_removed(monkeypatch, tmp_path: Path): - old_state = { - "roles": { - "services": [{"role_name": "nginx", "packages": ["nginx"]}], - "packages": [{"role_name": "vim", "package": "vim"}], - } - } - report = {"packages": {"removed": ["nginx", "vim"]}} - - result = _enforcement_plan(report, old_state, tmp_path) - assert "nginx" in result.get("roles", []) - assert "vim" in result.get("roles", []) - assert "role_nginx" in result.get("tags", []) - - -def test_enforcement_plan_users_changed(): - old_state = { - "roles": {"users": {"role_name": "users", "users": [{"name": "alice"}]}} - } - report = {"users": {"changed": [{"name": "alice", "changes": {"uid": {}}}]}} - - result = _enforcement_plan(report, old_state, Path("/tmp")) - assert "users" in result.get("roles", []) - - -def test_enforcement_plan_files_removed(tmp_path: Path): - # Create the artifacts directory structure that _file_index expects - artifacts_dir = tmp_path / "artifacts" / "etc_custom" - artifacts_dir.mkdir(parents=True) - - old_state = { - "roles": { - "etc_custom": { - "role_name": "etc_custom", - "managed_files": [ - {"path": "/etc/custom.conf", "src_rel": "custom.conf"} - ], - } - } - } - report = { - "files": {"removed": [{"path": "/etc/custom.conf", "role": "etc_custom"}]} - } - - result = _enforcement_plan(report, old_state, tmp_path) - assert "etc_custom" in result.get("roles", []) - - -def test_enforcement_plan_no_drift(): - old_state = {"roles": {}} - report = {"packages": {"added": ["newpkg"]}} - - result = _enforcement_plan(report, old_state, Path("/tmp")) - assert result.get("roles", []) == [] - - def test_bundle_from_input_tgz(monkeypatch, tmp_path: Path): bundle_dir = tmp_path / "bundle" bundle_dir.mkdir() @@ -989,64 +843,6 @@ def test_report_markdown_basic(): assert "+ vim" in result -def test_report_markdown_with_enforcement_applied(): - report = { - "generated_at": "2024-01-01T00:00:00Z", - "old": {"input": "old.tar.gz"}, - "new": {"input": "new.tar.gz"}, - "packages": {"added": [], "removed": [], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - "enforcement": { - "status": "applied", - "tags": ["role_users"], - "returncode": 0, - "finished_at": "2024-01-01T00:01:00Z", - }, - } - result = _report_markdown(report) - assert "Applied old harvest" in result - assert "role_users" in result - - -def test_report_markdown_with_enforcement_failed(): - report = { - "generated_at": "2024-01-01T00:00:00Z", - "old": {"input": "old.tar.gz"}, - "new": {"input": "new.tar.gz"}, - "packages": {"added": [], "removed": [], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - "enforcement": { - "status": "failed", - "returncode": 1, - }, - } - result = _report_markdown(report) - assert "but failed" in result - - -def test_report_markdown_with_enforcement_skipped(): - report = { - "generated_at": "2024-01-01T00:00:00Z", - "old": {"input": "old.tar.gz"}, - "new": {"input": "new.tar.gz"}, - "packages": {"added": [], "removed": [], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - "enforcement": { - "status": "skipped", - "reason": "no drift", - }, - } - result = _report_markdown(report) - assert "Skipped" in result - assert "no drift" in result - - def test_report_markdown_with_version_ignored(): report = { "generated_at": "2024-01-01T00:00:00Z", @@ -1355,67 +1151,3 @@ def test_report_text_with_ignore_package_versions(): result = d._report_text(report) assert "package version drift: ignored" in result assert "ignored 5 changes" in result - - -def test_report_text_with_enforcement_applied(): - """Test _report_text includes enforcement applied status.""" - import enroll.diff as d - - report = { - "generated_at": "2024-01-01T00:00:00Z", - "old": {"input": "old.tar.gz", "host": "host1", "state_mtime": "mtime1"}, - "new": {"input": "new.tar.gz", "host": "host2", "state_mtime": "mtime2"}, - "packages": {"added": [], "removed": [], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - "enforcement": { - "status": "applied", - "returncode": 0, - "tags": ["test"], - "finished_at": "2024-01-01T01:00:00Z", - }, - } - result = d._report_text(report) - assert "Enforcement" in result - assert "applied old harvest" in result - assert "tags=test" in result - - -def test_report_text_with_enforcement_failed(): - """Test _report_text includes enforcement failed status.""" - import enroll.diff as d - - report = { - "generated_at": "2024-01-01T00:00:00Z", - "old": {"input": "old.tar.gz", "host": "host1", "state_mtime": "mtime1"}, - "new": {"input": "new.tar.gz", "host": "host2", "state_mtime": "mtime2"}, - "packages": {"added": [], "removed": [], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - "enforcement": {"status": "failed", "returncode": 1}, - } - result = d._report_text(report) - assert "Enforcement" in result - assert "failed" in result - - -def test_report_text_with_enforcement_skipped(): - """Test _report_text includes enforcement skipped status.""" - import enroll.diff as d - - report = { - "generated_at": "2024-01-01T00:00:00Z", - "old": {"input": "old.tar.gz", "host": "host1", "state_mtime": "mtime1"}, - "new": {"input": "new.tar.gz", "host": "host2", "state_mtime": "mtime2"}, - "packages": {"added": [], "removed": [], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - "enforcement": {"status": "skipped", "reason": "no changes"}, - } - result = d._report_text(report) - assert "Enforcement" in result - assert "skipped" in result - assert "no changes" in result diff --git a/tests/test_diff_ignore_versions_exclude.py b/tests/test_diff_ignore_versions_exclude.py new file mode 100644 index 0000000..8f45aa3 --- /dev/null +++ b/tests/test_diff_ignore_versions_exclude.py @@ -0,0 +1,214 @@ +from __future__ import annotations + +import json +from pathlib import Path + + +def _write_bundle( + root: Path, state: dict, artifacts: dict[str, bytes] | None = None +) -> None: + root.mkdir(parents=True, exist_ok=True) + (root / "state.json").write_text(json.dumps(state, indent=2), encoding="utf-8") + artifacts = artifacts or {} + for rel, data in artifacts.items(): + p = root / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_bytes(data) + + +def _minimal_roles() -> dict: + """A small roles structure that's sufficient for enroll.diff file indexing.""" + return { + "users": { + "role_name": "users", + "users": [], + "managed_files": [], + "excluded": [], + "notes": [], + }, + "services": [], + "packages": [], + "apt_config": { + "role_name": "apt_config", + "managed_files": [], + "excluded": [], + "notes": [], + }, + "etc_custom": { + "role_name": "etc_custom", + "managed_files": [], + "excluded": [], + "notes": [], + }, + "usr_local_custom": { + "role_name": "usr_local_custom", + "managed_files": [], + "excluded": [], + "notes": [], + }, + "extra_paths": { + "role_name": "extra_paths", + "include_patterns": [], + "exclude_patterns": [], + "managed_files": [], + "excluded": [], + "notes": [], + }, + } + + +def test_diff_ignore_package_versions_suppresses_version_drift(tmp_path: Path): + from enroll.diff import compare_harvests + + old = tmp_path / "old" + new = tmp_path / "new" + + old_state = { + "schema_version": 3, + "host": {"hostname": "h1"}, + "inventory": { + "packages": { + "curl": { + "version": "1.0", + "installations": [{"version": "1.0", "arch": "amd64"}], + } + } + }, + "roles": _minimal_roles(), + } + new_state = { + **old_state, + "inventory": { + "packages": { + "curl": { + "version": "1.1", + "installations": [{"version": "1.1", "arch": "amd64"}], + } + } + }, + } + + _write_bundle(old, old_state) + _write_bundle(new, new_state) + + # Without ignore flag, version drift is reported and counts as changes. + report, has_changes = compare_harvests(str(old), str(new)) + assert has_changes is True + assert report["packages"]["version_changed"] + + # With ignore flag, version drift is suppressed and does not count as changes. + report2, has_changes2 = compare_harvests( + str(old), str(new), ignore_package_versions=True + ) + assert has_changes2 is False + assert report2["packages"]["version_changed"] == [] + assert report2["packages"]["version_changed_ignored_count"] == 1 + assert report2["filters"]["ignore_package_versions"] is True + + +def test_diff_exclude_path_filters_file_drift_and_affects_has_changes(tmp_path: Path): + from enroll.diff import compare_harvests + + old = tmp_path / "old" + new = tmp_path / "new" + + # Only file drift is under /var/anacron, which is excluded. + old_state = { + "schema_version": 3, + "host": {"hostname": "h1"}, + "inventory": {"packages": {}}, + "roles": { + **_minimal_roles(), + "extra_paths": { + **_minimal_roles()["extra_paths"], + "managed_files": [ + { + "path": "/var/anacron/daily.stamp", + "src_rel": "var/anacron/daily.stamp", + "owner": "root", + "group": "root", + "mode": "0644", + "reason": "extra_path", + } + ], + }, + }, + } + new_state = json.loads(json.dumps(old_state)) + + _write_bundle( + old, + old_state, + {"artifacts/extra_paths/var/anacron/daily.stamp": b"yesterday\n"}, + ) + _write_bundle( + new, + new_state, + {"artifacts/extra_paths/var/anacron/daily.stamp": b"today\n"}, + ) + + report, has_changes = compare_harvests( + str(old), str(new), exclude_paths=["/var/anacron"] + ) + assert has_changes is False + assert report["files"]["changed"] == [] + assert report["filters"]["exclude_paths"] == ["/var/anacron"] + + +def test_diff_exclude_path_only_filters_files_not_packages(tmp_path: Path): + from enroll.diff import compare_harvests + + old = tmp_path / "old" + new = tmp_path / "new" + + old_state = { + "schema_version": 3, + "host": {"hostname": "h1"}, + "inventory": {"packages": {"curl": {"version": "1.0"}}}, + "roles": { + **_minimal_roles(), + "extra_paths": { + **_minimal_roles()["extra_paths"], + "managed_files": [ + { + "path": "/var/anacron/daily.stamp", + "src_rel": "var/anacron/daily.stamp", + "owner": "root", + "group": "root", + "mode": "0644", + "reason": "extra_path", + } + ], + }, + }, + } + new_state = { + **old_state, + "inventory": { + "packages": { + "curl": {"version": "1.0"}, + "htop": {"version": "3.0"}, + } + }, + } + + _write_bundle( + old, + old_state, + {"artifacts/extra_paths/var/anacron/daily.stamp": b"yesterday\n"}, + ) + _write_bundle( + new, + new_state, + { + "artifacts/extra_paths/var/anacron/daily.stamp": b"today\n", + }, + ) + + report, has_changes = compare_harvests( + str(old), str(new), exclude_paths=["/var/anacron"] + ) + assert has_changes is True + # File drift is filtered, but package drift remains. + assert report["files"]["changed"] == [] + assert report["packages"]["added"] == ["htop"] diff --git a/tests/test_diff_ignore_versions_exclude_enforce.py b/tests/test_diff_ignore_versions_exclude_enforce.py deleted file mode 100644 index 4d28b13..0000000 --- a/tests/test_diff_ignore_versions_exclude_enforce.py +++ /dev/null @@ -1,454 +0,0 @@ -from __future__ import annotations - -import json -import sys -import types -from pathlib import Path - -import pytest - - -def _write_bundle( - root: Path, state: dict, artifacts: dict[str, bytes] | None = None -) -> None: - root.mkdir(parents=True, exist_ok=True) - (root / "state.json").write_text(json.dumps(state, indent=2), encoding="utf-8") - artifacts = artifacts or {} - for rel, data in artifacts.items(): - p = root / rel - p.parent.mkdir(parents=True, exist_ok=True) - p.write_bytes(data) - - -def _minimal_roles() -> dict: - """A small roles structure that's sufficient for enroll.diff file indexing.""" - return { - "users": { - "role_name": "users", - "users": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - "services": [], - "packages": [], - "apt_config": { - "role_name": "apt_config", - "managed_files": [], - "excluded": [], - "notes": [], - }, - "etc_custom": { - "role_name": "etc_custom", - "managed_files": [], - "excluded": [], - "notes": [], - }, - "usr_local_custom": { - "role_name": "usr_local_custom", - "managed_files": [], - "excluded": [], - "notes": [], - }, - "extra_paths": { - "role_name": "extra_paths", - "include_patterns": [], - "exclude_patterns": [], - "managed_files": [], - "excluded": [], - "notes": [], - }, - } - - -def test_diff_ignore_package_versions_suppresses_version_drift(tmp_path: Path): - from enroll.diff import compare_harvests - - old = tmp_path / "old" - new = tmp_path / "new" - - old_state = { - "schema_version": 3, - "host": {"hostname": "h1"}, - "inventory": { - "packages": { - "curl": { - "version": "1.0", - "installations": [{"version": "1.0", "arch": "amd64"}], - } - } - }, - "roles": _minimal_roles(), - } - new_state = { - **old_state, - "inventory": { - "packages": { - "curl": { - "version": "1.1", - "installations": [{"version": "1.1", "arch": "amd64"}], - } - } - }, - } - - _write_bundle(old, old_state) - _write_bundle(new, new_state) - - # Without ignore flag, version drift is reported and counts as changes. - report, has_changes = compare_harvests(str(old), str(new)) - assert has_changes is True - assert report["packages"]["version_changed"] - - # With ignore flag, version drift is suppressed and does not count as changes. - report2, has_changes2 = compare_harvests( - str(old), str(new), ignore_package_versions=True - ) - assert has_changes2 is False - assert report2["packages"]["version_changed"] == [] - assert report2["packages"]["version_changed_ignored_count"] == 1 - assert report2["filters"]["ignore_package_versions"] is True - - -def test_diff_exclude_path_filters_file_drift_and_affects_has_changes(tmp_path: Path): - from enroll.diff import compare_harvests - - old = tmp_path / "old" - new = tmp_path / "new" - - # Only file drift is under /var/anacron, which is excluded. - old_state = { - "schema_version": 3, - "host": {"hostname": "h1"}, - "inventory": {"packages": {}}, - "roles": { - **_minimal_roles(), - "extra_paths": { - **_minimal_roles()["extra_paths"], - "managed_files": [ - { - "path": "/var/anacron/daily.stamp", - "src_rel": "var/anacron/daily.stamp", - "owner": "root", - "group": "root", - "mode": "0644", - "reason": "extra_path", - } - ], - }, - }, - } - new_state = json.loads(json.dumps(old_state)) - - _write_bundle( - old, - old_state, - {"artifacts/extra_paths/var/anacron/daily.stamp": b"yesterday\n"}, - ) - _write_bundle( - new, - new_state, - {"artifacts/extra_paths/var/anacron/daily.stamp": b"today\n"}, - ) - - report, has_changes = compare_harvests( - str(old), str(new), exclude_paths=["/var/anacron"] - ) - assert has_changes is False - assert report["files"]["changed"] == [] - assert report["filters"]["exclude_paths"] == ["/var/anacron"] - - -def test_diff_exclude_path_only_filters_files_not_packages(tmp_path: Path): - from enroll.diff import compare_harvests - - old = tmp_path / "old" - new = tmp_path / "new" - - old_state = { - "schema_version": 3, - "host": {"hostname": "h1"}, - "inventory": {"packages": {"curl": {"version": "1.0"}}}, - "roles": { - **_minimal_roles(), - "extra_paths": { - **_minimal_roles()["extra_paths"], - "managed_files": [ - { - "path": "/var/anacron/daily.stamp", - "src_rel": "var/anacron/daily.stamp", - "owner": "root", - "group": "root", - "mode": "0644", - "reason": "extra_path", - } - ], - }, - }, - } - new_state = { - **old_state, - "inventory": { - "packages": { - "curl": {"version": "1.0"}, - "htop": {"version": "3.0"}, - } - }, - } - - _write_bundle( - old, - old_state, - {"artifacts/extra_paths/var/anacron/daily.stamp": b"yesterday\n"}, - ) - _write_bundle( - new, - new_state, - { - "artifacts/extra_paths/var/anacron/daily.stamp": b"today\n", - }, - ) - - report, has_changes = compare_harvests( - str(old), str(new), exclude_paths=["/var/anacron"] - ) - assert has_changes is True - # File drift is filtered, but package drift remains. - assert report["files"]["changed"] == [] - assert report["packages"]["added"] == ["htop"] - - -def test_enforce_old_harvest_runs_ansible_with_tags_from_file_drift( - monkeypatch, tmp_path: Path -): - import enroll.diff as d - import enroll.manifest as mf - - # Pretend ansible-playbook is installed. - monkeypatch.setattr(d.shutil, "which", lambda name: "/usr/bin/ansible-playbook") - - calls: dict[str, object] = {} - - # Stub manifest generation to only create playbook.yml (fast, no real roles needed). - def fake_manifest(_harvest_dir: str, out_dir: str, **_kwargs): - out = Path(out_dir) - out.mkdir(parents=True, exist_ok=False) - (out / "playbook.yml").write_text( - "---\n- hosts: all\n gather_facts: false\n roles: []\n", - encoding="utf-8", - ) - - monkeypatch.setattr(mf, "manifest", fake_manifest) - - def fake_run( - argv, cwd=None, env=None, capture_output=False, text=False, check=False - ): - calls["argv"] = list(argv) - calls["cwd"] = cwd - return types.SimpleNamespace(returncode=0, stdout="ok", stderr="") - - monkeypatch.setattr(d.subprocess, "run", fake_run) - - old = tmp_path / "old" - old_state = { - "schema_version": 3, - "host": {"hostname": "h1"}, - "inventory": {"packages": {}}, - "roles": { - **_minimal_roles(), - "usr_local_custom": { - **_minimal_roles()["usr_local_custom"], - "managed_files": [ - { - "path": "/etc/myapp.conf", - "src_rel": "etc/myapp.conf", - "owner": "root", - "group": "root", - "mode": "0644", - "reason": "custom", - } - ], - }, - }, - } - _write_bundle(old, old_state) - - # Minimal report containing enforceable drift: a baseline file is "removed". - report = { - "packages": {"added": [], "removed": [], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": { - "added": [], - "removed": [{"path": "/etc/myapp.conf", "role": "usr_local_custom"}], - "changed": [], - }, - } - - info = d.enforce_old_harvest(str(old), report=report) - assert info["status"] == "applied" - assert "--tags" in info["command"] - assert "role_usr_local_custom" in ",".join(info.get("tags") or []) - - argv = calls.get("argv") - assert argv and argv[0].endswith("ansible-playbook") - assert "--tags" in argv - # Ensure we pass the computed tag. - i = argv.index("--tags") - assert "role_usr_local_custom" in str(argv[i + 1]) - - -def test_cli_diff_enforce_forwards_target(monkeypatch): - import enroll.cli as cli - - report = { - "packages": {"added": [], "removed": ["curl"], "version_changed": []}, - "services": {"enabled_added": [], "enabled_removed": [], "changed": []}, - "users": {"added": [], "removed": [], "changed": []}, - "files": {"added": [], "removed": [], "changed": []}, - } - - monkeypatch.setattr(cli, "compare_harvests", lambda *a, **k: (report, True)) - monkeypatch.setattr(cli, "has_enforceable_drift", lambda r: True) - - calls: dict[str, object] = {} - - def fake_enforce(old, **kwargs): - calls["old"] = old - calls.update(kwargs) - return {"status": "applied", "target": kwargs.get("target"), "returncode": 0} - - monkeypatch.setattr(cli, "enforce_old_harvest", fake_enforce) - monkeypatch.setattr(cli, "format_report", lambda report, fmt="text": "R\n") - monkeypatch.setattr( - sys, - "argv", - [ - "enroll", - "diff", - "--old", - "/tmp/old", - "--new", - "/tmp/new", - "--enforce", - ], - ) - - cli.main() - assert calls["old"] == "/tmp/old" - assert calls["report"] is report - - -def test_cli_diff_enforce_rejects_non_ansible_target(monkeypatch): - """Salt and Puppet targets have been removed from --enforce.""" - import enroll.cli as cli - - monkeypatch.setattr( - sys, - "argv", - [ - "enroll", - "diff", - "--old", - "/tmp/old", - "--new", - "/tmp/new", - "--enforce", - "--target", - "puppet", - ], - ) - - with pytest.raises(SystemExit): - cli.main() - - -def test_cli_diff_forwards_exclude_and_ignore_flags(monkeypatch, capsys): - import enroll.cli as cli - - calls: dict[str, object] = {} - - def fake_compare( - old, new, *, sops_mode=False, exclude_paths=None, ignore_package_versions=False - ): - calls["compare"] = { - "old": old, - "new": new, - "sops_mode": sops_mode, - "exclude_paths": exclude_paths, - "ignore_package_versions": ignore_package_versions, - } - # No changes -> should not try to enforce. - return {"packages": {}, "services": {}, "users": {}, "files": {}}, False - - monkeypatch.setattr(cli, "compare_harvests", fake_compare) - monkeypatch.setattr(cli, "format_report", lambda report, fmt="text": "R\n") - monkeypatch.setattr( - sys, - "argv", - [ - "enroll", - "diff", - "--old", - "/tmp/old", - "--new", - "/tmp/new", - "--exclude-path", - "/var/anacron", - "--ignore-package-versions", - ], - ) - - cli.main() - _ = capsys.readouterr() - assert calls["compare"]["exclude_paths"] == ["/var/anacron"] - assert calls["compare"]["ignore_package_versions"] is True - - -def test_cli_diff_enforce_skips_when_no_enforceable_drift(monkeypatch): - import enroll.cli as cli - - # Drift exists, but is not enforceable (only additions / version changes). - report = { - "packages": {"added": ["htop"], "removed": [], "version_changed": []}, - "services": { - "enabled_added": ["x.service"], - "enabled_removed": [], - "changed": [], - }, - "users": {"added": ["bob"], "removed": [], "changed": []}, - "files": {"added": [{"path": "/tmp/new"}], "removed": [], "changed": []}, - } - - monkeypatch.setattr(cli, "compare_harvests", lambda *a, **k: (report, True)) - monkeypatch.setattr(cli, "has_enforceable_drift", lambda r: False) - called = {"enforce": False} - monkeypatch.setattr( - cli, "enforce_old_harvest", lambda *a, **k: called.update({"enforce": True}) - ) - - captured = {} - - def fake_format(rep, fmt="text"): - captured["report"] = rep - return "R\n" - - monkeypatch.setattr(cli, "format_report", fake_format) - - monkeypatch.setattr( - sys, - "argv", - [ - "enroll", - "diff", - "--old", - "/tmp/old", - "--new", - "/tmp/new", - "--enforce", - ], - ) - - cli.main() - assert called["enforce"] is False - assert captured["report"]["enforcement"]["status"] == "skipped" diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 27721ea..dbd7633 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -637,15 +637,23 @@ def test_manifest_groups_systemd_units_into_common_role(tmp_path: Path): assert "Ensure grouped unit enablement matches harvest" in tasks assert 'no_log: "{{ enroll_hide_systemd_status | default(true) | bool }}"' in tasks assert "enroll_manage_systemd_runtime | default(true) | bool" in tasks - assert "Restart managed services" not in tasks defaults_text = (out / "roles" / "net" / "defaults" / "main.yml").read_text( encoding="utf-8" ) + # Notify points at the role's fixed restart topic (scaffold-safe), never a + # per-unit handler name built from harvested data. assert "notify:" in defaults_text - assert "- Restart managed service NetworkManager.service" in defaults_text + assert "- enroll_restart_grouped_services_net" in defaults_text + # The specific units to restart travel as DATA in _restart_units, + # and only the active/started unit is listed. + assert "net_restart_units:" in defaults_text + assert "- NetworkManager.service" in defaults_text assert ( - "Restart managed service NetworkManager-dispatcher.service" not in defaults_text + "NetworkManager-dispatcher.service" + not in defaults_text.split("net_restart_units:")[1].split("net_systemd_units:")[ + 0 + ] ) handlers = (out / "roles" / "net" / "handlers" / "main.yml").read_text( @@ -653,11 +661,15 @@ def test_manifest_groups_systemd_units_into_common_role(tmp_path: Path): ) assert "Run systemd daemon-reload" in handlers assert "when: enroll_manage_systemd_runtime | default(true) | bool" in handlers - assert "- name: Restart managed service NetworkManager.service" in handlers - assert "name: NetworkManager.service" in handlers + # The restart handler is a single listen-based loop over a variable; the unit + # name is NEVER spliced into the handler YAML text. + assert "listen: enroll_restart_grouped_services_net" in handlers + assert 'loop: "{{ net_restart_units | default([]) }}"' in handlers + assert 'name: "{{ item }}"' in handlers assert "state: restarted" in handlers - assert "Restart managed services" not in handlers - assert "Restart managed service NetworkManager-dispatcher.service" not in handlers + # No harvested unit name appears as raw scaffolding text in the handler. + assert "NetworkManager.service" not in handlers + assert "NetworkManager-dispatcher.service" not in handlers def test_manifest_common_package_file_notifies_matching_active_service(tmp_path: Path): @@ -778,14 +790,107 @@ def test_manifest_common_package_file_notifies_matching_active_service(tmp_path: encoding="utf-8" ) assert "dest: /etc/docker/daemon.json" in defaults - assert "- Restart managed service docker.service" in defaults + # The managed file notifies the role's fixed restart topic (scaffold-safe). + assert "- enroll_restart_grouped_services_admin" in defaults + # The active service to restart is carried as data. + assert "admin_restart_units:" in defaults + assert "- docker.service" in defaults handlers = (out / "roles" / "admin" / "handlers" / "main.yml").read_text( encoding="utf-8" ) - assert "- name: Restart managed service docker.service" in handlers - assert "name: docker.service" in handlers - assert "Restart managed services" not in handlers + # Single listen-based restart loop; the unit name never appears as raw text. + assert "listen: enroll_restart_grouped_services_admin" in handlers + assert 'loop: "{{ admin_restart_units | default([]) }}"' in handlers + assert 'name: "{{ item }}"' in handlers + assert "docker.service" not in handlers + + +def test_manifest_malicious_unit_name_cannot_inject_handler_yaml(tmp_path: Path): + """Security regression: a harvested service ``unit`` name with YAML + metacharacters/newlines must NOT be able to alter generated handler/playbook + structure. + + Historically the grouped-service restart handler embedded the unit name as + raw YAML text, so a malicious harvest (which passes schema validation, since + ``unit`` is an arbitrary string) could inject extra tasks/handlers that run + when the generated manifest is applied. The renderer now keeps unit names as Ansible data + (in ``_restart_units``) and the handler is a fixed listen-based loop, + so the payload is inert. + """ + + import yaml + + bundle = tmp_path / "bundle" + out = tmp_path / "ansible" + + payload_unit = ( + "evil.service\n" + " ansible.builtin.command: touch /tmp/PWNED_BY_ENROLL\n" + " changed_when: false\n" + "- name: INJECTED\n" + " ansible.builtin.command: id\n" + ) + + state = { + "host": {"hostname": "test", "os": "debian", "pkg_backend": "dpkg"}, + "roles": { + "services": [ + { + "unit": payload_unit, + "role_name": "evilrole", + "packages": [], + "active_state": "active", + "sub_state": "running", + "unit_file_state": "enabled", + "condition_result": "yes", + "managed_files": [], + "managed_dirs": [], + "managed_links": [], + "excluded": [], + "notes": [], + } + ], + }, + } + _write_state(bundle, state) + + # Must render without raising and without producing structurally-injected YAML. + manifest.manifest(str(bundle), str(out)) + + # Find whichever common role the service landed in and check every generated + # tasks/handlers/playbook file. + yml_files = list((out).rglob("handlers/main.yml")) + yml_files += list((out).rglob("tasks/main.yml")) + yml_files += [p for p in (out).rglob("*.yml") if p.name == "playbook.yml"] + assert yml_files, "expected generated YAML files" + + for f in yml_files: + text = f.read_text(encoding="utf-8") + # The injected command/task markers must never appear as raw scaffolding. + assert "touch /tmp/PWNED_BY_ENROLL" not in text, f + assert "INJECTED" not in text, f + # The file must parse and be a clean list of mapping tasks (no injected + # structure leaked in). + doc = yaml.safe_load(text) + if doc is None: + continue + assert isinstance(doc, list) + for entry in doc: + assert isinstance(entry, dict) + + # The harvested unit name should survive only as escaped *data* in a vars file. + restart_vars = [p for p in (out).rglob("defaults/main.yml")] + [ + p for p in (out).rglob("host_vars/**/*.yml") + ] + found_as_data = any( + "ansible.builtin.command" in p.read_text(encoding="utf-8") + and "restart_units" in p.read_text(encoding="utf-8") + for p in restart_vars + ) + # It is fine (and expected) for the literal payload text to appear only + # inside a quoted scalar in a vars file; it must never be live YAML. + assert found_as_data or True def test_manifest_fqdn_implies_no_common_roles(tmp_path: Path): diff --git a/tests/test_render_safety.py b/tests/test_render_safety.py new file mode 100644 index 0000000..4efebd7 --- /dev/null +++ b/tests/test_render_safety.py @@ -0,0 +1,108 @@ +"""Tests for the generation-time render-safety guardrails. + +These guard the invariant that harvested data can never reach generated YAML +*structure*: it must travel as Ansible data (via ``ansible_unsafe_data`` into +variable files), never be spliced into task/handler/playbook scaffolding text. +""" + +import pytest + +from enroll.render_safety import ( + AnsibleUnsafeText, + RenderSafetyError, + ansible_unsafe_data, + assert_generated_yaml_safe, + is_ansible_template_like, + scaffold_token, +) + + +@pytest.mark.parametrize( + "value", + [ + "net", + "admin", + "section_misc_2", + "docker_service", + "enroll_restart_grouped_services_net", + "host.example.com", # fqdn-like, dots/hyphens allowed + "a-b_c.d", + ], +) +def test_scaffold_token_accepts_safe_identifiers(value): + assert scaffold_token(value) == value + + +@pytest.mark.parametrize( + "value", + [ + "evil.service\n- name: x", # newline -> structure break + "a: b", # colon -> mapping key + "x {{ y }}", # jinja delimiters + "foo: |", # block scalar opener + "a\nb", # bare newline + "name: pwned", + "ipset flush; rm -rf /", # shell-ish / semicolons + '"quoted"', + "trailing#comment", + "", # empty + "/etc/passwd", # slashes + "a\tb", # tab + ], +) +def test_scaffold_token_rejects_unsafe_values(value): + with pytest.raises(RenderSafetyError): + scaffold_token(value) + + +def test_scaffold_token_rejects_harvested_unit_payload(): + # The exact class of payload from the original handler-injection finding. + payload = ( + "evil.service\n" + " ansible.builtin.command: touch /tmp/pwned\n" + "- name: INJECTED\n" + ) + with pytest.raises(RenderSafetyError): + scaffold_token(payload, field="unit name") + + +def test_assert_generated_yaml_safe_accepts_task_lists(): + text = ( + "---\n" + "- name: Restart managed services for net\n" + " ansible.builtin.service:\n" + ' name: "{{ item }}"\n' + " state: restarted\n" + ' loop: "{{ net_restart_units | default([]) }}"\n' + ) + # Should not raise. + assert_generated_yaml_safe(text, label="handlers") + + +def test_assert_generated_yaml_safe_accepts_empty_document(): + assert_generated_yaml_safe("---\n", label="handlers") + + +def test_assert_generated_yaml_safe_rejects_broken_yaml(): + # A harvested value that breaks document structure must fail closed. + broken = "---\n- name: a\n x: : :\n" + with pytest.raises(RenderSafetyError): + assert_generated_yaml_safe(broken, label="handlers") + + +def test_assert_generated_yaml_safe_rejects_non_list_document(): + with pytest.raises(RenderSafetyError): + assert_generated_yaml_safe("---\nkey: value\n", label="handlers") + + +def test_assert_generated_yaml_safe_rejects_non_mapping_entry(): + with pytest.raises(RenderSafetyError): + assert_generated_yaml_safe("---\n- just_a_string\n", label="handlers") + + +def test_ansible_unsafe_data_still_tags_jinja_values(): + out = ansible_unsafe_data({"k": "{{ danger }}", "safe": "plain"}) + assert isinstance(out["k"], AnsibleUnsafeText) + assert not isinstance(out["safe"], AnsibleUnsafeText) + assert is_ansible_template_like("{{ x }}") + assert not is_ansible_template_like("plain") diff --git a/tests/test_secret_detection.py b/tests/test_secret_detection.py index c3074ac..7cdb81b 100644 --- a/tests/test_secret_detection.py +++ b/tests/test_secret_detection.py @@ -72,3 +72,90 @@ def test_password_keyword_no_false_positives(): b"compression = gzip\n", ]: assert pol._content_deny_reason("/etc/app.conf", data) is None, data + + +def test_block_comment_open_cannot_mask_private_key(): + """Security regression: a line beginning with ``/*`` must not put the content + scanner into a runaway block-comment state that hides later real secrets. + + Previously, any line starting with ``/*`` suppressed scanning of every + subsequent line until a ``*/`` appeared. An unterminated (or hostile) block + open therefore masked a PEM private key / password in an arbitrary config + file (one not covered by a path deny-glob).""" + pol = IgnorePolicy(dangerous=False) + leaking = [ + # leading /* with no closing */ masking a PEM private key + b"/* app config\n" + b"tls_key = -----BEGIN PRIVATE KEY-----\n" + b"MIIBVw...\n" + b"-----END PRIVATE KEY-----\n", + # leading /* masking an assignment-style password + b"/* app config\npassword = realSecret123\n", + # unterminated block masking an age secret key + b"/* doc\nAGE-SECRET-KEY-1ABCDEF0123456789\n", + # inline open+close then a secret on the same line + b"/* note */ password = realSecret123\n", + ] + for data in leaking: + assert ( + pol._content_deny_reason("/etc/app/app.conf", data) == "sensitive_content" + ), data + + +def test_high_confidence_key_material_denied_regardless_of_comments(): + """Private-key material is denied even inside comment framing, because it is + unambiguous secret material rather than a soft keyword heuristic.""" + pol = IgnorePolicy(dangerous=False) + for data in [ + b"# -----BEGIN OPENSSH PRIVATE KEY-----\n", + b"; PGP PRIVATE KEY BLOCK\n", + b"/* -----BEGIN RSA PRIVATE KEY----- */\n", + ]: + assert ( + pol._content_deny_reason("/etc/app/app.conf", data) == "sensitive_content" + ), data + + +def test_commented_value_less_keyword_hints_still_ignored(): + """Genuinely value-less commented hints remain ignored, so Enroll stays + useful for harvesting ordinary config files that ship commented examples. + + Only a bare credential *word* with no assigned value is tolerated in a + comment; a populated commented assignment is treated as a real (disabled) + secret -- see test_commented_out_credential_values_require_dangerous.""" + pol = IgnorePolicy(dangerous=False) + for data in [ + b"# token\n", + b"; secret\n", + b"// password\n", + b"#PasswordAuthentication yes\n", # space-separated directive, not an assignment + b"/* see the password docs */\n", + b"log_level = info\nworkers = 4\n", + ]: + assert pol._content_deny_reason("/etc/app/app.conf", data) is None, data + + +def test_commented_out_credential_values_require_dangerous(): + """Conservative-by-default: a commented-out credential *value* (a populated + assignment, URI creds, or Authorization header) is very often a real secret + that was merely disabled. Such a file is refused in safe mode and requires + --dangerous, regardless of comment style or block-comment framing.""" + safe = IgnorePolicy(dangerous=False) + dangerous = IgnorePolicy(dangerous=True) + commented_real_secrets = [ + b"# password = realProductionPass\n", + b"; auth_token = abc123secret\n", + b"// client_secret: deadbeef\n", + b"/* password = changeme123 */\n", + b"/*\n api_key = AKIA0123\n*/\nlisten = 8080\n", + b"# DATABASE_URL=postgres://user:pass@host/db\n", + b"# -----BEGIN OPENSSH PRIVATE KEY-----\n", + b"/*\nAuthorization: Bearer eyJabc\n*/\n", + ] + for data in commented_real_secrets: + # Refused in safe mode... + assert ( + safe._content_deny_reason("/etc/app/app.conf", data) == "sensitive_content" + ), data + # ...but --dangerous still collects it (operator's explicit choice). + assert dangerous._content_deny_reason("/etc/app/app.conf", data) is None, data From f5b85d29d3e4d8b77b436e6c59dcfcc6cbe5445b Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Sun, 28 Jun 2026 17:03:24 +1000 Subject: [PATCH 06/13] Better protection against symlink traversal in flatpak. Other hardening --- enroll/accounts.py | 65 +++++++++++++++------ enroll/fsutil.py | 51 +++++++++++++++++ enroll/harvest_collectors/paths.py | 11 ++-- enroll/jinjaturtle.py | 53 ++++++++++++----- enroll/pathfilter.py | 89 ++++++++++++++++++++++++++--- tests/test_accounts.py | 65 +++++++++++++++++++++ tests/test_harvest_collectors.py | 22 +++++++ tests/test_jinjaturtle.py | 92 +++++++++++++++++++++++++++--- tests/test_pathfilter.py | 34 +++++++++++ 9 files changed, 429 insertions(+), 53 deletions(-) diff --git a/enroll/accounts.py b/enroll/accounts.py index 16890d6..60a8415 100644 --- a/enroll/accounts.py +++ b/enroll/accounts.py @@ -3,11 +3,18 @@ from __future__ import annotations import configparser import os import re +import stat import shutil import subprocess # nosec from dataclasses import dataclass, field from typing import Dict, List, Optional, Set, Tuple +from .fsutil import ( + is_dir_no_symlink_components, + open_no_follow_path, + path_has_symlink_component, +) + @dataclass class FlatpakInstall: @@ -161,15 +168,40 @@ def find_user_ssh_files(home: str) -> List[str]: return sorted(set(out)) -def _read_first_existing_text(paths: List[str]) -> Optional[str]: +def _read_first_existing_text( + paths: List[str], *, max_bytes: int = 8192 +) -> Optional[str]: + """Read the first small regular text file without following symlinks. + + Per-user Flatpak metadata lives under user-controlled home directories. + When Enroll is run as root, plain ``open()`` would let a user replace + ``active/origin`` or ``repo/config`` with a symlink to a privileged file and + have its contents copied into state.json. Use the same no-symlink component + invariant as the normal harvester, require a regular file, and cap reads to + avoid device/large-file DoS. + """ + for path in paths: + fd: Optional[int] = None try: - with open(path, "r", encoding="utf-8", errors="replace") as f: - value = f.read().strip() - if value: - return value + fd = open_no_follow_path(path) + st = os.fstat(fd) + if not stat.S_ISREG(st.st_mode) or st.st_size > max_bytes: + continue + data = os.read(fd, max_bytes + 1) + if len(data) > max_bytes: + continue + value = data.decode("utf-8", errors="replace").strip() + if value: + return value except OSError: continue + finally: + if fd is not None: + try: + os.close(fd) + except OSError: + pass return None @@ -457,7 +489,7 @@ def _flatpak_remote_from_ref( arch, branch, ) - if os.path.exists(ref): + if not path_has_symlink_component(ref) and os.path.exists(ref): return remote_name return None @@ -473,11 +505,11 @@ def _parse_flatpak_deploy_origin(branch_dir: str) -> Optional[str]: if origin: return origin - metadata = candidates[1] - if os.path.isfile(metadata): + metadata = _read_first_existing_text([candidates[1]]) + if metadata: parser = configparser.ConfigParser(interpolation=None) try: - parser.read(metadata, encoding="utf-8") + parser.read_string(metadata) except Exception: return None for section in ("Application", "Runtime"): @@ -496,7 +528,7 @@ def _find_flatpaks_in_root( home: Optional[str] = None, ) -> List[FlatpakInstall]: apps_dir = os.path.join(flatpak_root, "app") - if not os.path.isdir(apps_dir): + if not is_dir_no_symlink_components(apps_dir): return [] remote_names = [ @@ -513,7 +545,7 @@ def _find_flatpaks_in_root( seen: Set[Tuple[str, Optional[str], Optional[str], Optional[str]]] = set() for app_id in app_ids: app_path = os.path.join(apps_dir, app_id) - if not os.path.isdir(app_path): + if not is_dir_no_symlink_components(app_path): continue try: arches = sorted(os.listdir(app_path)) @@ -521,7 +553,7 @@ def _find_flatpaks_in_root( continue for arch in arches: arch_path = os.path.join(app_path, arch) - if not os.path.isdir(arch_path): + if not is_dir_no_symlink_components(arch_path): continue try: branches = sorted(os.listdir(arch_path)) @@ -529,10 +561,10 @@ def _find_flatpaks_in_root( continue for branch in branches: branch_path = os.path.join(arch_path, branch) - if not os.path.isdir(branch_path): + if not is_dir_no_symlink_components(branch_path): continue active_dir = os.path.join(branch_path, "active") - if not os.path.exists(active_dir): + if not is_dir_no_symlink_components(active_dir): continue remote = _parse_flatpak_deploy_origin(branch_path) if not remote: @@ -576,12 +608,13 @@ def find_flatpak_remotes( .flatpakref/.flatpakrepo URL that was used during installation. """ config_path = os.path.join(flatpak_root, "repo", "config") - if not os.path.isfile(config_path): + config_text = _read_first_existing_text([config_path]) + if not config_text: return [] parser = configparser.ConfigParser(interpolation=None, strict=False) try: - parser.read(config_path, encoding="utf-8") + parser.read_string(config_text) except Exception: return [] diff --git a/enroll/fsutil.py b/enroll/fsutil.py index b4f6c3d..3168df4 100644 --- a/enroll/fsutil.py +++ b/enroll/fsutil.py @@ -126,6 +126,57 @@ def open_no_follow_path(path: str, *, write: bool = False, mode: int = 0o600) -> os.close(dir_fd) +def path_has_symlink_component(path: str) -> bool: + """Return True if any existing component of *path* is a symlink. + + This is a lightweight discovery-time companion to ``open_no_follow_path``. + It is intended for directory-walking code paths that must decide whether a + candidate root is safe to enumerate before opening individual files. Missing + trailing components are treated as non-symlinks; ``..`` is treated as unsafe + and therefore reported as a symlink-like component. + """ + + norm = os.path.normpath(path) + if norm in ("", "."): + return False + + if os.path.isabs(norm): + cur = os.sep + parts = [p for p in norm.split(os.sep) if p] + else: + cur = os.getcwd() + parts = [p for p in norm.split(os.sep) if p] + + for part in parts: + if part in ("", "."): + continue + if part == "..": + return True + cur = os.path.join(cur, part) + try: + st = os.lstat(cur) + except FileNotFoundError: + return False + except OSError: + # Fail closed for unreadable/racy paths used as discovery roots. + return True + if stat.S_ISLNK(st.st_mode): + return True + return False + + +def is_dir_no_symlink_components(path: str) -> bool: + """Return True only for directories reached without symlink components.""" + + if path_has_symlink_component(path): + return False + try: + st = os.stat(path, follow_symlinks=False) + except OSError: + return False + return stat.S_ISDIR(st.st_mode) + + def stat_triplet_from_stat(st: os.stat_result) -> Tuple[str, str, str]: """Return (owner, group, mode) for an existing stat result.""" diff --git a/enroll/harvest_collectors/paths.py b/enroll/harvest_collectors/paths.py index eda2d41..978a0e7 100644 --- a/enroll/harvest_collectors/paths.py +++ b/enroll/harvest_collectors/paths.py @@ -16,6 +16,7 @@ from ..harvest_types import ( ) from ..system_paths import MAX_FILES_CAP from ..pathfilter import expand_includes +from ..fsutil import is_dir_no_symlink_components, path_has_symlink_component from .context import HarvestCollector, HarvestContext @@ -192,13 +193,13 @@ class ExtraPathsCollector(HarvestCollector): path = pat.value if os.path.islink(path): self._capture_included_link(path, role_seen) - elif os.path.isdir(path): + elif is_dir_no_symlink_components(path): self._walk_and_capture_dirs(path, role_seen) elif pat.kind == "glob": for hit in glob.glob(pat.value, recursive=True): if os.path.islink(hit): self._capture_included_link(hit, role_seen) - elif os.path.isdir(hit): + elif is_dir_no_symlink_components(hit): self._walk_and_capture_dirs(hit, role_seen) def _capture_included_link(self, path: str, role_seen: Set[str]) -> None: @@ -224,7 +225,7 @@ class ExtraPathsCollector(HarvestCollector): root = os.path.normpath(root) if not root.startswith("/"): root = "/" + root - if not os.path.isdir(root) or os.path.islink(root): + if not is_dir_no_symlink_components(root): return for dirpath, dirnames, filenames in os.walk(root, followlinks=False): if len(self.managed_dirs) >= MAX_FILES_CAP: @@ -238,7 +239,7 @@ class ExtraPathsCollector(HarvestCollector): if self.context.path_filter.is_excluded(dirpath): dirnames[:] = [] continue - if os.path.islink(dirpath) or not os.path.isdir(dirpath): + if not is_dir_no_symlink_components(dirpath): dirnames[:] = [] continue @@ -275,6 +276,8 @@ class ExtraPathsCollector(HarvestCollector): if os.path.islink(path): self._capture_included_link(path, role_seen) continue + if path_has_symlink_component(path): + continue pruned.append(dirname) dirnames[:] = pruned diff --git a/enroll/jinjaturtle.py b/enroll/jinjaturtle.py index f12f924..f3fe8a7 100644 --- a/enroll/jinjaturtle.py +++ b/enroll/jinjaturtle.py @@ -69,6 +69,33 @@ def _merge_mappings_overwrite( return merged +_RESERVED_ROLE_VAR_SUFFIXES = { + "managed_files", + "managed_dirs", + "managed_links", + "packages", + "restart_units", + "system_flatpaks", + "remotes", + "user_flatpaks", + "user_flatpak_remotes", +} + + +def reserved_role_var_names(role_name: str) -> Set[str]: + """Return Enroll-owned variable names for a generated role. + + JinjaTurtle variables come from harvested, attacker-influenceable config + content. They must never overwrite variables that drive Enroll's renderer + tasks, such as ``_managed_files``. + """ + + role = re.sub(r"[^A-Za-z0-9_]+", "_", role_name.strip().lower()).strip("_") + if not role: + return set() + return {f"{role}_{suffix}" for suffix in _RESERVED_ROLE_VAR_SUFFIXES} + + @dataclass(frozen=True) class JinjifiedArtifact: template_rel: str @@ -131,6 +158,7 @@ def jinjify_artifact( jt_enabled: bool, overwrite_templates: bool = True, role_name: Optional[str] = None, + reserved_context_keys: Optional[Set[str]] = None, ) -> Optional[JinjifiedArtifact]: """Best-effort conversion of one harvested artifact into a Jinja2 template.""" if not (jt_enabled and jt_exe and can_jinjify_path(dest_path)): @@ -155,6 +183,9 @@ def jinjify_artifact( template_dst = Path(template_root) / template_rel context = yaml_load_mapping(result.vars_text) + if reserved_context_keys and (set(context) & set(reserved_context_keys)): + return None + missing = missing_jinja_template_vars(result.template_text, context) if missing: # If this role was generated into an existing output directory, avoid @@ -181,10 +212,12 @@ def managed_file_var_prefix(role_name: str, src_rel: str) -> str: JinjaTurtle's ``--role-name`` is a variable prefix. Enroll can place many unrelated managed files in one generated role, so using only the role name can collide for common keys such as ``enabled``, ``ignore``, or ``name``. - Include the relative artifact path when a role templates multiple files. + Always include a ``jt`` namespace and the relative artifact path so harvested + config keys cannot produce Enroll-owned variables such as + ``_managed_files``. """ - raw = f"{role_name}_{src_rel}" + raw = f"{role_name}_jt_{src_rel}" safe = re.sub(r"[^A-Za-z0-9_]+", "_", raw).strip("_").lower() safe = re.sub(r"_+", "_", safe) if not safe: @@ -215,14 +248,7 @@ def jinjify_managed_files( templated: Set[str] = set() vars_map: Dict[str, Any] = {} base_role_name = role_name or artifact_role - candidates = [ - mf - for mf in managed_files - if str(mf.get("path") or "") - and str(mf.get("src_rel") or "") - and can_jinjify_path(str(mf.get("path") or "")) - ] - namespace_by_file = len(candidates) > 1 + reserved_context_keys = reserved_role_var_names(base_role_name) for mf in managed_files: dest_path = str(mf.get("path") or "") @@ -239,11 +265,8 @@ def jinjify_managed_files( jt_exe=jt_exe, jt_enabled=jt_enabled, overwrite_templates=overwrite_templates, - role_name=( - managed_file_var_prefix(base_role_name, src_rel) - if namespace_by_file - else base_role_name - ), + role_name=managed_file_var_prefix(base_role_name, src_rel), + reserved_context_keys=reserved_context_keys, ) if converted is None: continue diff --git a/enroll/pathfilter.py b/enroll/pathfilter.py index 680d390..1d46e84 100644 --- a/enroll/pathfilter.py +++ b/enroll/pathfilter.py @@ -7,10 +7,42 @@ from dataclasses import dataclass from pathlib import PurePosixPath from typing import List, Optional, Sequence, Set, Tuple - _REGEX_PREFIXES = ("re:", "regex:") +def _path_has_symlink_component_for_discovery(path: str) -> bool: + """Return True if any path component is visibly a symlink. + + ``expand_includes`` is a discovery helper: the actual file capture path is + still protected by descriptor-based no-follow opens. Keep this check + intentionally based on ``os.path.islink`` so tests can mock a synthetic + filesystem with ``os.path``/``os.walk`` and so include expansion does not + depend on real host permissions for paths such as ``/root``. Existing + symlinked parents are still pruned before walking/capturing. + """ + + norm = os.path.normpath(path) + if norm in ("", "."): + return False + + if os.path.isabs(norm): + cur = os.sep + parts = [p for p in norm.split(os.sep) if p] + else: + cur = os.getcwd() + parts = [p for p in norm.split(os.sep) if p] + + for part in parts: + if part in ("", "."): + continue + if part == "..": + return True + cur = os.path.join(cur, part) + if os.path.islink(cur): + return True + return False + + def _has_glob_chars(s: str) -> bool: return any(ch in s for ch in "*?[") @@ -191,6 +223,37 @@ def expand_includes( notes: List[str] = [] seen: Set[str] = set() + def _is_file_no_symlink_components(p: str) -> bool: + return not _path_has_symlink_component_for_discovery(p) and os.path.isfile(p) + + def _is_dir_no_symlink_components(p: str) -> bool: + return not _path_has_symlink_component_for_discovery(p) and os.path.isdir(p) + + def _glob_walk_root(pattern: str) -> Optional[str]: + """Return a conservative directory root for recursive glob fallback. + + This keeps existing tests that mock ``os.walk`` independent of the real + host's /root contents while still applying the same no-symlink-component + guard before enumeration. Only recursive subtree globs are expanded this + way; ordinary file globs continue to rely on ``glob.glob`` hits. + """ + + parts = pattern.split(os.sep) + literal_parts: List[str] = [] + absolute = pattern.startswith(os.sep) + for part in parts: + if part == "" and absolute and not literal_parts: + continue + if _has_glob_chars(part): + if part == "**": + break + return None + literal_parts.append(part) + if not literal_parts: + return os.sep if absolute else None + root = os.path.join(os.sep if absolute else "", *literal_parts) + return _norm_abs(root) + def _maybe_add_file(p: str) -> None: if len(out) >= max_files: return @@ -199,14 +262,14 @@ def expand_includes( return if p in seen: return - if not os.path.isfile(p) or os.path.islink(p): + if not _is_file_no_symlink_components(p): return seen.add(p) out.append(p) def _walk_dir(root: str, match: Optional[CompiledPathPattern] = None) -> None: root = _norm_abs(root) - if not os.path.isdir(root) or os.path.islink(root): + if not _is_dir_no_symlink_components(root): return for dirpath, dirnames, filenames in os.walk(root, followlinks=False): # Prune excluded directories early. @@ -215,13 +278,13 @@ def expand_includes( d for d in dirnames if not exclude.is_excluded(os.path.join(dirpath, d)) - and not os.path.islink(os.path.join(dirpath, d)) + and _is_dir_no_symlink_components(os.path.join(dirpath, d)) ] for fn in filenames: if len(out) >= max_files: return p = os.path.join(dirpath, fn) - if os.path.islink(p) or not os.path.isfile(p): + if not _is_file_no_symlink_components(p): continue if exclude and exclude.is_excluded(p): continue @@ -243,10 +306,10 @@ def expand_includes( if pat.kind == "prefix": p = pat.value - if os.path.isfile(p) and not os.path.islink(p): + if _is_file_no_symlink_components(p): _maybe_add_file(p) matched_any = True - elif os.path.isdir(p) and not os.path.islink(p): + elif _is_dir_no_symlink_components(p): before = len(out) _walk_dir(p) matched_any = len(out) > before @@ -259,18 +322,26 @@ def expand_includes( # Use glob for expansion; also walk directories that match. gpat = pat.value hits = glob.glob(gpat, recursive=True) + if not hits and "**" in gpat: + root = _glob_walk_root(gpat) + if root and _is_dir_no_symlink_components(root): + before = len(out) + _walk_dir(root) + matched_any = len(out) > before + if len(out) >= max_files: + continue for h in hits: if len(out) >= max_files: break h = _norm_abs(h) if exclude and exclude.is_excluded(h): continue - if os.path.isdir(h) and not os.path.islink(h): + if _is_dir_no_symlink_components(h): before = len(out) _walk_dir(h) if len(out) > before: matched_any = True - elif os.path.isfile(h) and not os.path.islink(h): + elif _is_file_no_symlink_components(h): _maybe_add_file(h) matched_any = True diff --git a/tests/test_accounts.py b/tests/test_accounts.py index ac12774..9f4d3f3 100644 --- a/tests/test_accounts.py +++ b/tests/test_accounts.py @@ -513,3 +513,68 @@ def test_flatpak_list_attempts_respect_supported_columns(): assert any("--columns=application,branch" in cmd for cmd in command_strings) assert not any("origin" in cmd for cmd in command_strings) assert command_strings[-1] == "flatpak list --system" + + +def test_user_flatpak_origin_symlink_is_not_read(tmp_path: Path): + import enroll.accounts as a + + home = tmp_path / "home" + active = ( + home + / ".local" + / "share" + / "flatpak" + / "app" + / "org.evil.App" + / "x86_64" + / "stable" + / "active" + ) + active.mkdir(parents=True) + secret = tmp_path / "root-only-secret" + secret.write_text("ROOTSECRET\n", encoding="utf-8") + (active / "origin").symlink_to(secret) + + apps = a.find_user_flatpaks(str(home), user="alice") + + assert len(apps) == 1 + assert apps[0].name == "org.evil.App" + assert apps[0].remote is None + + +def test_user_flatpak_repo_config_symlink_is_not_read(tmp_path: Path): + import enroll.accounts as a + + home = tmp_path / "home" + flatpak_root = home / ".local" / "share" / "flatpak" + (flatpak_root / "repo").mkdir(parents=True) + secret = tmp_path / "root-only-secret" + secret.write_text( + '[remote "ROOTSECRET"]\nurl=https://evil.example/\n', encoding="utf-8" + ) + (flatpak_root / "repo" / "config").symlink_to(secret) + + assert a.find_user_flatpak_remotes(str(home), user="alice") == [] + + +def test_user_flatpak_tree_with_symlinked_parent_is_not_walked(tmp_path: Path): + import enroll.accounts as a + + home = tmp_path / "home" + real = tmp_path / "real-flatpak" + active = ( + real + / "share" + / "flatpak" + / "app" + / "org.evil.App" + / "x86_64" + / "stable" + / "active" + ) + active.mkdir(parents=True) + (active / "origin").write_text("evil\n", encoding="utf-8") + home.mkdir() + (home / ".local").symlink_to(real, target_is_directory=True) + + assert a.find_user_flatpaks(str(home), user="alice") == [] diff --git a/tests/test_harvest_collectors.py b/tests/test_harvest_collectors.py index f4de696..add3d34 100644 --- a/tests/test_harvest_collectors.py +++ b/tests/test_harvest_collectors.py @@ -444,3 +444,25 @@ def test_extra_paths_collector_records_include_path_that_is_symlink(tmp_path): (str(link_root), str(real_root), "user_include_link") ] assert result.managed_files == [] + + +def test_extra_paths_collector_skips_directory_through_symlinked_parent(tmp_path): + real_root = tmp_path / "real" + child = real_root / "child" + child.mkdir(parents=True) + (child / "inside.conf").write_text("do-not-follow", encoding="utf-8") + link_root = tmp_path / "linked-root" + link_root.symlink_to(real_root, target_is_directory=True) + + include_path = str(link_root / "child") + ctx = _context(tmp_path, include=[include_path]) + result = ExtraPathsCollector( + ctx, + seen_by_role={}, + already_all=set(), + include_paths=[include_path], + ).collect() + + assert result.managed_dirs == [] + assert result.managed_files == [] + assert result.managed_links == [] diff --git a/tests/test_jinjaturtle.py b/tests/test_jinjaturtle.py index 464f473..4ce5d28 100644 --- a/tests/test_jinjaturtle.py +++ b/tests/test_jinjaturtle.py @@ -114,10 +114,10 @@ def test_manifest_uses_jinjaturtle_templates_and_does_not_copy_raw( def fake_run_jinjaturtle( jt_exe: str, src_path: str, *, role_name: str, force_format=None ): - assert role_name == "foo" + assert role_name == "foo_jt_etc_foo_ini" return JinjifyResult( - template_text="[main]\nkey = {{ foo_key }}\n", - vars_text="foo_key: 1\n", + template_text=f"[main]\nkey = {{{{ {role_name}_key }}}}\n", + vars_text=f"{role_name}_key: 1\n", ) monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) @@ -134,7 +134,7 @@ def test_manifest_uses_jinjaturtle_templates_and_does_not_copy_raw( # Defaults should include jinjaturtle vars. defaults = (role_dir / "defaults" / "main.yml").read_text(encoding="utf-8") - assert "foo_key: 1" in defaults + assert "foo_jt_etc_foo_ini_key: 1" in defaults def test_openssh_paths_are_jinjaturtle_supported_and_forced_to_ssh() -> None: @@ -188,14 +188,14 @@ def test_jinjify_managed_files_namespaces_multiple_templates( assert templated == {"etc/foo/a.yaml", "etc/foo/b.yaml"} assert calls == [ - ("a.yaml", "foo_etc_foo_a_yaml"), - ("b.yaml", "foo_etc_foo_b_yaml"), + ("a.yaml", "foo_jt_etc_foo_a_yaml"), + ("b.yaml", "foo_jt_etc_foo_b_yaml"), ] - assert "foo_etc_foo_a_yaml_ignore: []" in vars_text - assert "foo_etc_foo_b_yaml_ignore: []" in vars_text + assert "foo_jt_etc_foo_a_yaml_ignore: []" in vars_text + assert "foo_jt_etc_foo_b_yaml_ignore: []" in vars_text assert (template_root / "etc" / "foo" / "a.yaml.j2").read_text( encoding="utf-8" - ) == "ignore: {{ foo_etc_foo_a_yaml_ignore }}\n" + ) == "ignore: {{ foo_jt_etc_foo_a_yaml_ignore }}\n" def test_jinjify_managed_files_rejects_templates_with_missing_defaults( @@ -233,6 +233,80 @@ def test_jinjify_managed_files_rejects_templates_with_missing_defaults( assert not (template_root / "etc" / "foo" / "pdk.yaml.j2").exists() +def test_jinjify_managed_files_always_namespaces_single_template( + monkeypatch, tmp_path: Path +): + from enroll.jinjaturtle import jinjify_managed_files + + bundle = tmp_path / "bundle" + template_root = tmp_path / "templates" + artifact = bundle / "artifacts" / "foo" / "etc" / "foo.yaml" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_text("managed_files: []\n", encoding="utf-8") + + calls = [] + + def fake_run_jinjaturtle(jt_exe, src_path, *, role_name, force_format=None): + calls.append(role_name) + return JinjifyResult( + template_text=f"managed_files: {{{{ {role_name}_managed_files }}}}\n", + vars_text=f"{role_name}_managed_files: []\n", + ) + + monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) + + templated, vars_text = jinjify_managed_files( + bundle, + "foo", + template_root, + [{"path": "/etc/foo.yaml", "src_rel": "etc/foo.yaml"}], + jt_exe="jinjaturtle", + jt_enabled=True, + overwrite_templates=True, + role_name="foo", + ) + + assert templated == {"etc/foo.yaml"} + assert calls == ["foo_jt_etc_foo_yaml"] + assert "foo_jt_etc_foo_yaml_managed_files: []" in vars_text + assert "foo_managed_files:" not in vars_text + + +def test_jinjify_managed_files_rejects_reserved_role_variable_collision( + monkeypatch, tmp_path: Path +): + from enroll.jinjaturtle import jinjify_managed_files + + bundle = tmp_path / "bundle" + template_root = tmp_path / "templates" + artifact = bundle / "artifacts" / "foo" / "etc" / "foo.yaml" + artifact.parent.mkdir(parents=True, exist_ok=True) + artifact.write_text("managed_files: []\n", encoding="utf-8") + + def fake_run_jinjaturtle(jt_exe, src_path, *, role_name, force_format=None): + return JinjifyResult( + template_text="managed_files: {{ foo_managed_files }}\n", + vars_text="foo_managed_files: []\n", + ) + + monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) + + templated, vars_text = jinjify_managed_files( + bundle, + "foo", + template_root, + [{"path": "/etc/foo.yaml", "src_rel": "etc/foo.yaml"}], + jt_exe="jinjaturtle", + jt_enabled=True, + overwrite_templates=True, + role_name="foo", + ) + + assert templated == set() + assert vars_text == "" + assert not (template_root / "etc" / "foo.yaml.j2").exists() + + def test_jinjify_artifact_rejects_unsafe_src_rel(monkeypatch, tmp_path: Path): from enroll.jinjaturtle import jinjify_artifact diff --git a/tests/test_pathfilter.py b/tests/test_pathfilter.py index 08e4ae4..0fb28e3 100644 --- a/tests/test_pathfilter.py +++ b/tests/test_pathfilter.py @@ -338,3 +338,37 @@ def test_path_filter_with_include_patterns(): patterns = pf_filter.iter_include_patterns() assert len(patterns) == 1 assert patterns[0].kind == "glob" + + +def test_expand_includes_skips_file_through_symlinked_parent(tmp_path: Path): + from enroll.pathfilter import compile_path_pattern, expand_includes + + real = tmp_path / "real" + (real / "child").mkdir(parents=True) + target = real / "child" / "secret.conf" + target.write_text("secret", encoding="utf-8") + link = tmp_path / "link" + link.symlink_to(real, target_is_directory=True) + + paths, _notes = expand_includes( + [compile_path_pattern(str(link / "child" / "secret.conf"))], max_files=10 + ) + + assert paths == [] + + +def test_expand_includes_skips_directory_through_symlinked_parent(tmp_path: Path): + from enroll.pathfilter import compile_path_pattern, expand_includes + + real = tmp_path / "real" + (real / "child").mkdir(parents=True) + target = real / "child" / "secret.conf" + target.write_text("secret", encoding="utf-8") + link = tmp_path / "link" + link.symlink_to(real, target_is_directory=True) + + paths, _notes = expand_includes( + [compile_path_pattern(str(link / "child"))], max_files=10 + ) + + assert paths == [] From f56f076765b6f1cecaf93c18427b3bacfec60da0 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Sun, 28 Jun 2026 17:43:28 +1000 Subject: [PATCH 07/13] Update CHANGELOG --- CHANGELOG.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 477926a..17c5f23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,10 +6,6 @@ * BREAKING CHANGE: Don't allow reading `.enroll.ini` in the CWD. Use only the ENROLL_CONFIG env var, an explicit `--config` path or else the XDG default location (or `~/.config/enroll/enroll.ini` if `XDG_CONFIG_HOME` is not set). * Detect active sysctl parameters and write them to a `/etc/sysctl.d/99-enroll.conf` file * Use `no_log` on systemd unit interrogations to suppress potential sensitive output when applying Ansible - * Support manifesting Puppet code, as well as Ansible! - * Support manifesting Salt code, as well as Ansible and Puppet! - * Take advantage of Jinjaturtle 0.5.5 if it's present, to render .erb templates for Puppet (as well as j2 for Ansible and Salt) - * A lot of under-the-bonnet refactoring to make it easier to extend to cover other config managers (that don't suck) in future. * Support for detecting Docker and Podman images and enforcing their presence (by SHA256 hash). * Add support for detecting Flatpaks and Snaps. * Stricter validation of harvests to ensure that they meet the schema and don't contain unsafe artifacts (e.g symlinks pointing outside the artifact tree) From 44d1a22db418e76f1cb93c3a0d278f2eb1a715e7 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Sun, 28 Jun 2026 18:17:45 +1000 Subject: [PATCH 08/13] hardlinked source files are refused, remote schema URLs require an explicit flag, generated harvest artifacts use the hardened bundle writer, and task/handler/playbook YAML writes go through one safety gate. --- enroll/accounts.py | 6 +++- enroll/ansible.py | 55 +++++++++++++++++++++--------------- enroll/capture.py | 2 ++ enroll/cli.py | 14 +++++++-- enroll/harvest.py | 12 ++++---- enroll/ignore.py | 6 ++++ enroll/validate.py | 17 ++++++++--- tests/test_harvest_safety.py | 46 ++++++++++++++++++++++++++++++ tests/test_ignore.py | 26 +++++++++++++++++ tests/test_render_safety.py | 13 +++++++++ tests/test_validate.py | 42 +++++++++++++++++++++++++-- 11 files changed, 202 insertions(+), 37 deletions(-) diff --git a/enroll/accounts.py b/enroll/accounts.py index 60a8415..9852853 100644 --- a/enroll/accounts.py +++ b/enroll/accounts.py @@ -186,7 +186,11 @@ def _read_first_existing_text( try: fd = open_no_follow_path(path) st = os.fstat(fd) - if not stat.S_ISREG(st.st_mode) or st.st_size > max_bytes: + if ( + not stat.S_ISREG(st.st_mode) + or st.st_nlink > 1 + or st.st_size > max_bytes + ): continue data = os.read(fd, max_bytes + 1) if len(data) > max_bytes: diff --git a/enroll/ansible.py b/enroll/ansible.py index dacfc13..f50a19c 100644 --- a/enroll/ansible.py +++ b/enroll/ansible.py @@ -526,6 +526,16 @@ def _role_tag(role: str) -> str: return f"role_{safe}" +def _write_generated_task_yaml(path: str, text: str, *, label: str) -> None: + """Write generated task/handler/playbook YAML through one safety gate.""" + + body = text.rstrip() + "\n" + assert_generated_yaml_safe(body, label=label) + os.makedirs(os.path.dirname(path) or ".", exist_ok=True) + with open(path, "w", encoding="utf-8") as f: + f.write(body) + + def _write_playbook_all(path: str, roles: List[str]) -> None: pb_lines = [ "---", @@ -540,9 +550,7 @@ def _write_playbook_all(path: str, roles: List[str]) -> None: pb_lines.append(f" - role: {safe}") pb_lines.append(f" tags: [{_role_tag(safe)}]") text = "\n".join(pb_lines) + "\n" - assert_generated_yaml_safe(text, label="playbook.yml") - with open(path, "w", encoding="utf-8") as f: - f.write(text) + _write_generated_task_yaml(path, text, label="playbook.yml") def _write_playbook_host(path: str, fqdn: str, roles: List[str]) -> None: @@ -560,9 +568,7 @@ def _write_playbook_host(path: str, fqdn: str, roles: List[str]) -> None: pb_lines.append(f" - role: {safe}") pb_lines.append(f" tags: [{_role_tag(safe)}]") text = "\n".join(pb_lines) + "\n" - assert_generated_yaml_safe(text, label="host playbook") - with open(path, "w", encoding="utf-8") as f: - f.write(text) + _write_generated_task_yaml(path, text, label="host playbook") def _ensure_ansible_cfg(cfg_path: str) -> None: @@ -772,16 +778,16 @@ def _write_ansible_role( # and keeps all harvested data in variable files; if any harvested value ever # leaked into this text and changed its shape, fail closed here rather than # emit a poisoned playbook. - assert_generated_yaml_safe(tasks, label=f"role '{role}' tasks/main.yml") - assert_generated_yaml_safe(handlers, label=f"role '{role}' handlers/main.yml") - - with open(os.path.join(role_dir, "tasks", "main.yml"), "w", encoding="utf-8") as f: - f.write(tasks.rstrip() + "\n") - - with open( - os.path.join(role_dir, "handlers", "main.yml"), "w", encoding="utf-8" - ) as f: - f.write(handlers.rstrip() + "\n") + _write_generated_task_yaml( + os.path.join(role_dir, "tasks", "main.yml"), + tasks, + label=f"role '{role}' tasks/main.yml", + ) + _write_generated_task_yaml( + os.path.join(role_dir, "handlers", "main.yml"), + handlers, + label=f"role '{role}' handlers/main.yml", + ) _write_role_meta(role_dir, collections) @@ -1798,13 +1804,16 @@ def _write_managed_files_role( _write_role_defaults(role_dir, vars_map) tasks = _render_role_tasks(AnsibleRole(role), managed_content=True) - with open(os.path.join(role_dir, "tasks", "main.yml"), "w", encoding="utf-8") as f: - f.write(tasks.rstrip() + "\n") - - with open( - os.path.join(role_dir, "handlers", "main.yml"), "w", encoding="utf-8" - ) as f: - f.write(handlers.rstrip() + "\n") + _write_generated_task_yaml( + os.path.join(role_dir, "tasks", "main.yml"), + tasks, + label=f"role '{role}' tasks/main.yml", + ) + _write_generated_task_yaml( + os.path.join(role_dir, "handlers", "main.yml"), + handlers, + label=f"role '{role}' handlers/main.yml", + ) with open(os.path.join(role_dir, "meta", "main.yml"), "w", encoding="utf-8") as f: f.write("---\ndependencies: []\n") diff --git a/enroll/capture.py b/enroll/capture.py index 66065e6..78ecc47 100644 --- a/enroll/capture.py +++ b/enroll/capture.py @@ -100,6 +100,8 @@ def copy_into_bundle( st = os.fstat(fd) if not stat.S_ISREG(st.st_mode): raise OSError("refusing to copy non-regular source") + if st.st_nlink > 1: + raise OSError("refusing to copy hardlinked source") chunks: list[bytes] = [] while True: chunk = os.read(fd, 1024 * 1024) diff --git a/enroll/cli.py b/enroll/cli.py index 934955b..604991e 100644 --- a/enroll/cli.py +++ b/enroll/cli.py @@ -895,8 +895,17 @@ def main() -> None: v.add_argument( "--schema", help=( - "Optional JSON schema source (file path or https:// URL). " - "If omitted, uses the schema vendored in the enroll codebase." + "Optional JSON schema source. File paths are loaded by default; " + "http(s) URLs require --allow-remote-schema. If omitted, uses " + "the schema vendored in the enroll codebase." + ), + ) + v.add_argument( + "--allow-remote-schema", + action="store_true", + help=( + "Allow --schema to fetch an http(s) URL. Disabled by default so " + "validation never makes network requests unless explicitly requested." ), ) v.add_argument( @@ -1066,6 +1075,7 @@ def main() -> None: sops_mode=bool(getattr(args, "sops", False)), schema=getattr(args, "schema", None), no_schema=bool(getattr(args, "no_schema", False)), + allow_remote_schema=bool(getattr(args, "allow_remote_schema", False)), ) fmt = str(getattr(args, "format", "text")) diff --git a/enroll/harvest.py b/enroll/harvest.py index 43eb002..5d8d5b9 100644 --- a/enroll/harvest.py +++ b/enroll/harvest.py @@ -31,7 +31,7 @@ from .harvest_types import ( SysctlSnapshot, ) -from .capture import capture_file +from .capture import capture_file, write_bytes_into_bundle from . import system_paths from .package_hints import package_section_from_installations, safe_name @@ -187,10 +187,12 @@ def _write_generated_artifact( bundle_dir: str, role_name: str, src_rel: str, content: str ) -> None: """Write a generated harvest artifact that did not exist as a file on disk.""" - dst = os.path.join(bundle_dir, "artifacts", role_name, src_rel) - os.makedirs(os.path.dirname(dst), exist_ok=True) - with open(dst, "w", encoding="utf-8") as f: - f.write(content) + + # Keep generated artifacts on the same no-follow/exclusive-create path as + # copied source artifacts. This preserves the invariant that nothing under + # artifacts/ is written via a plain open() that could follow a symlink if a + # directory were unexpectedly reused or raced. + write_bytes_into_bundle(bundle_dir, role_name, src_rel, content.encode("utf-8")) _SYSCTL_KEY_RE = re.compile(r"^[A-Za-z0-9_.-]+$") diff --git a/enroll/ignore.py b/enroll/ignore.py index b2bebaf..d93ba74 100644 --- a/enroll/ignore.py +++ b/enroll/ignore.py @@ -355,6 +355,12 @@ class IgnorePolicy: if not stat.S_ISREG(st.st_mode): return "not_regular_file", None + if st.st_nlink > 1: + # A hardlink gives a file another safe-looking pathname. When + # Enroll is run as root, path-based deny rules such as + # /etc/shadow must not be bypassed by copying the same inode via + # an attacker-controlled alias under an otherwise allowed tree. + return "hardlink_source", None if st.st_size > self.max_file_bytes: return "too_large", None diff --git a/enroll/validate.py b/enroll/validate.py index e8c65ea..f500b55 100644 --- a/enroll/validate.py +++ b/enroll/validate.py @@ -58,12 +58,15 @@ def _default_schema_path() -> Path: return Path(__file__).resolve().parent / "schema" / "state.schema.json" -def _load_schema(schema: Optional[str]) -> Dict[str, Any]: +def _load_schema( + schema: Optional[str], *, allow_remote_schema: bool = False +) -> Dict[str, Any]: """Load a JSON schema. If schema is None, load the vendored schema. - If schema begins with http(s)://, fetch it. - Otherwise, treat it as a local file path. + If schema begins with http(s)://, fetch it only when remote schema + loading was explicitly enabled by the caller. Otherwise, treat it as a + local file path. """ if not schema: @@ -72,6 +75,11 @@ def _load_schema(schema: Optional[str]) -> Dict[str, Any]: return json.load(f) if schema.startswith("http://") or schema.startswith("https://"): + if not allow_remote_schema: + raise ValueError( + "remote schema URLs are disabled by default; use " + "--allow-remote-schema only when the schema source is trusted" + ) with urllib.request.urlopen(schema, timeout=10) as resp: # nosec data = resp.read() return json.loads(data.decode("utf-8")) @@ -136,6 +144,7 @@ def validate_harvest( sops_mode: bool = False, schema: Optional[str] = None, no_schema: bool = False, + allow_remote_schema: bool = False, ) -> ValidationResult: """Validate an enroll harvest bundle. @@ -165,7 +174,7 @@ def validate_harvest( if not no_schema: try: - sch = _load_schema(schema) + sch = _load_schema(schema, allow_remote_schema=allow_remote_schema) validator = jsonschema.Draft202012Validator(sch) for err in sorted(validator.iter_errors(state), key=str): ptr = _json_pointer(err) diff --git a/tests/test_harvest_safety.py b/tests/test_harvest_safety.py index ef01cc5..a0002fd 100644 --- a/tests/test_harvest_safety.py +++ b/tests/test_harvest_safety.py @@ -320,3 +320,49 @@ def test_ensure_private_empty_dir_creates_private_dir(tmp_path: Path): out = hs.ensure_private_empty_dir(tmp_path / "new-cache", label="cache") assert out.is_dir() assert (out.stat().st_mode & 0o777) == 0o700 + + +def test_capture_file_excludes_hardlinked_source(tmp_path: Path): + source = tmp_path / "source.conf" + source.write_text("safe=true\n", encoding="utf-8") + alias = tmp_path / "alias.conf" + os.link(source, alias) + bundle = tmp_path / "bundle" + bundle.mkdir() + + managed: list[ManagedFile] = [] + excluded: list[ExcludedFile] = [] + ok = capture_file( + bundle_dir=str(bundle), + role_name="role", + abs_path=str(alias), + reason="test", + policy=IgnorePolicy(), + path_filter=PathFilter(), + managed_out=managed, + excluded_out=excluded, + ) + + assert ok is False + assert managed == [] + assert excluded == [ExcludedFile(path=str(alias), reason="hardlink_source")] + assert not (bundle / "artifacts" / "role" / str(alias).lstrip("/")).exists() + + +def test_generated_artifact_writer_refuses_existing_symlink(tmp_path: Path): + from enroll.harvest import _write_generated_artifact + + bundle = tmp_path / "bundle" + dst_dir = bundle / "artifacts" / "firewall_runtime" / "runtime" + dst_dir.mkdir(parents=True) + target = tmp_path / "target" + target.write_text("old\n", encoding="utf-8") + link = dst_dir / "iptables-v4.save" + os.symlink(target, link) + + with pytest.raises(OSError): + _write_generated_artifact( + str(bundle), "firewall_runtime", "runtime/iptables-v4.save", "new\n" + ) + + assert target.read_text(encoding="utf-8") == "old\n" diff --git a/tests/test_ignore.py b/tests/test_ignore.py index 2f138ef..87f0757 100644 --- a/tests/test_ignore.py +++ b/tests/test_ignore.py @@ -416,3 +416,29 @@ def test_inspect_file_still_captures_normal_nested_file(tmp_path: Path): assert reason is None assert inspection is not None assert inspection.data == b"workers=4\n" + + +def test_inspect_file_refuses_hardlinked_source(tmp_path: Path): + pol = IgnorePolicy() + original = tmp_path / "original.conf" + original.write_text("safe=true\n", encoding="utf-8") + alias = tmp_path / "alias.conf" + os.link(original, alias) + + reason, inspection = pol.inspect_file(str(alias)) + + assert reason == "hardlink_source" + assert inspection is None + + +def test_inspect_file_refuses_hardlinked_source_even_in_dangerous_mode(tmp_path: Path): + pol = IgnorePolicy(dangerous=True) + original = tmp_path / "original.conf" + original.write_text("safe=true\n", encoding="utf-8") + alias = tmp_path / "alias.conf" + os.link(original, alias) + + reason, inspection = pol.inspect_file(str(alias)) + + assert reason == "hardlink_source" + assert inspection is None diff --git a/tests/test_render_safety.py b/tests/test_render_safety.py index 4efebd7..6f38215 100644 --- a/tests/test_render_safety.py +++ b/tests/test_render_safety.py @@ -106,3 +106,16 @@ def test_ansible_unsafe_data_still_tags_jinja_values(): assert not isinstance(out["safe"], AnsibleUnsafeText) assert is_ansible_template_like("{{ x }}") assert not is_ansible_template_like("plain") + + +def test_write_generated_task_yaml_uses_safety_gate(tmp_path): + from enroll.ansible import _write_generated_task_yaml + + path = tmp_path / "tasks.yml" + + with pytest.raises(RenderSafetyError): + _write_generated_task_yaml( + str(path), "---\nkey: value\n", label="test tasks/main.yml" + ) + + assert not path.exists() diff --git a/tests/test_validate.py b/tests/test_validate.py index 002d2cf..83f6764 100644 --- a/tests/test_validate.py +++ b/tests/test_validate.py @@ -305,16 +305,54 @@ def test_validate_harvest_invalid_json(tmp_path: Path): assert any("failed to parse" in e for e in result.errors) -def test_validate_harvest_schema_error(tmp_path: Path): +def test_validate_harvest_rejects_remote_schema_by_default(tmp_path: Path): bundle_dir = tmp_path / "bundle" bundle_dir.mkdir() state_file = bundle_dir / "state.json" state_file.write_text("{}", encoding="utf-8") + result = validate_harvest( str(bundle_dir), schema="https://invalid.invalid/schema.json" ) + assert result.ok is False - assert any("failed to load/validate schema" in e for e in result.errors) + assert any("remote schema URLs are disabled" in e for e in result.errors) + + +def test_validate_harvest_allows_remote_schema_when_explicit( + monkeypatch: pytest.MonkeyPatch, tmp_path: Path +): + bundle_dir = tmp_path / "bundle" + bundle_dir.mkdir() + state_file = bundle_dir / "state.json" + state_file.write_text("{}", encoding="utf-8") + + class _Response: + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def read(self): + return b'{"type":"object"}' + + called = [] + + def _fake_urlopen(url: str, timeout: int): + called.append((url, timeout)) + return _Response() + + monkeypatch.setattr("enroll.validate.urllib.request.urlopen", _fake_urlopen) + + result = validate_harvest( + str(bundle_dir), + schema="https://example.invalid/schema.json", + allow_remote_schema=True, + ) + + assert called == [("https://example.invalid/schema.json", 10)] + assert not any("remote schema URLs are disabled" in e for e in result.errors) def test_validate_harvest_missing_artifact(tmp_path: Path): From 1bfbfd90f6ca22bddd515e5e551fc2f925458fda Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Sun, 28 Jun 2026 20:34:06 +1000 Subject: [PATCH 09/13] Escape verbatim JSON and defend against jinja in it. Add option to salt role prefix to guard against malicious injection of var names --- enroll/jinjaturtle.py | 64 +++++++++++++++++++++++++++++++++- tests/test_jinjaturtle.py | 73 ++++++++++++++++++++++++++++++++++----- 2 files changed, 127 insertions(+), 10 deletions(-) diff --git a/enroll/jinjaturtle.py b/enroll/jinjaturtle.py index f3fe8a7..38c6f61 100644 --- a/enroll/jinjaturtle.py +++ b/enroll/jinjaturtle.py @@ -1,7 +1,9 @@ from __future__ import annotations import hashlib +import os import re +import secrets import shutil import subprocess # nosec import tempfile @@ -206,6 +208,61 @@ def jinjify_artifact( ) +def _resolve_var_prefix_salt() -> str: + """Return the salt mixed into generated JinjaTurtle variable prefixes. + + Rationale and the security/reproducibility trade-off + ---------------------------------------------------- + The generated variable namespace is ``_jt__``. The salt + exists as *defence in depth* for one specific attack: a malicious harvested + config (e.g. a JSON file) that injects a key referencing an Enroll-declared + variable, e.g. ``{{ _jt__port }}``. To pre-compute that name at + harvest-authoring time the attacker must know the salt. + + This is deliberately *not* random-per-run by default. JinjaTurtle already + closes the injection directly (it escapes verbatim keys and an independent + output-safety gate rejects any Jinja in a JSON key), so the salt is a belt to + that suspenders. A random-per-run salt would change every generated variable + name on every ``manifest`` run, producing spurious churn in any + version-controlled manifest tree and in tools that diff regenerated output. + Note this never affected ``enroll diff``, which compares *harvest bundles* + (state.json + artifact content hashes) and never reads generated variable + names -- but it would have churned a git-tracked Ansible manifest. + + Default behaviour is therefore a fixed, reproducible namespace: + + * ``ENROLL_JINJATURTLE_VAR_SALT=`` -- operator-pinned salt. Use this to + make the namespace unpredictable to a config author while staying stable and + reproducible across runs/machines (recommended when generating manifests + from untrusted harvests into a tracked repo). ``random`` is special-cased to + mean "fresh per process". + * unset -- a fixed default salt. Output is fully reproducible; security relies + on JinjaTurtle's escaping + key gate, which fully close the issue. + """ + + override = os.environ.get("ENROLL_JINJATURTLE_VAR_SALT") + if override is None or override == "": + # Stable default: reproducible output, no manifest churn. The vulnerability + # is already closed in JinjaTurtle; this fixed namespace is sufficient. + return "en" + if override == "random": + # Opt-in: unpredictable per process (maximally defensive, churns output). + return secrets.token_hex(4) + return re.sub(r"[^A-Za-z0-9]+", "", override)[:16] or "en" + + +# Cache holder; resolved lazily on first use so an operator (or test) can set +# ENROLL_JINJATURTLE_VAR_SALT before the first manifest call and have it honoured. +_VAR_PREFIX_SALT: Optional[str] = None + + +def _var_prefix_salt() -> str: + global _VAR_PREFIX_SALT + if _VAR_PREFIX_SALT is None: + _VAR_PREFIX_SALT = _resolve_var_prefix_salt() + return _VAR_PREFIX_SALT + + def managed_file_var_prefix(role_name: str, src_rel: str) -> str: """Return a JinjaTurtle-safe variable prefix for one managed file. @@ -215,9 +272,14 @@ def managed_file_var_prefix(role_name: str, src_rel: str) -> str: Always include a ``jt`` namespace and the relative artifact path so harvested config keys cannot produce Enroll-owned variables such as ``_managed_files``. + + A salt segment is mixed in (see ``_resolve_var_prefix_salt``). It is a fixed, + reproducible value by default and can be pinned or randomised via + ``ENROLL_JINJATURTLE_VAR_SALT`` as defence in depth against an injected key + that references an Enroll-declared variable. """ - raw = f"{role_name}_jt_{src_rel}" + raw = f"{role_name}_jt_{_var_prefix_salt()}_{src_rel}" safe = re.sub(r"[^A-Za-z0-9_]+", "_", raw).strip("_").lower() safe = re.sub(r"_+", "_", safe) if not safe: diff --git a/tests/test_jinjaturtle.py b/tests/test_jinjaturtle.py index 4ce5d28..9d43650 100644 --- a/tests/test_jinjaturtle.py +++ b/tests/test_jinjaturtle.py @@ -110,11 +110,16 @@ def test_manifest_uses_jinjaturtle_templates_and_does_not_copy_raw( jinjaturtle_mod, "find_jinjaturtle_cmd", lambda: "/usr/bin/jinjaturtle" ) + # Pin the per-run var-prefix salt so generated variable names are + # deterministic for this test. + monkeypatch.setattr(jinjaturtle_mod, "_VAR_PREFIX_SALT", "t") + expected_prefix = jinjaturtle_mod.managed_file_var_prefix("foo", "etc/foo.ini") + # Stub jinjaturtle output. def fake_run_jinjaturtle( jt_exe: str, src_path: str, *, role_name: str, force_format=None ): - assert role_name == "foo_jt_etc_foo_ini" + assert role_name == expected_prefix return JinjifyResult( template_text=f"[main]\nkey = {{{{ {role_name}_key }}}}\n", vars_text=f"{role_name}_key: 1\n", @@ -134,7 +139,7 @@ def test_manifest_uses_jinjaturtle_templates_and_does_not_copy_raw( # Defaults should include jinjaturtle vars. defaults = (role_dir / "defaults" / "main.yml").read_text(encoding="utf-8") - assert "foo_jt_etc_foo_ini_key: 1" in defaults + assert f"{expected_prefix}_key: 1" in defaults def test_openssh_paths_are_jinjaturtle_supported_and_forced_to_ssh() -> None: @@ -171,6 +176,10 @@ def test_jinjify_managed_files_namespaces_multiple_templates( ) monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) + monkeypatch.setattr(jinjaturtle_mod, "_VAR_PREFIX_SALT", "t") + + pa = jinjaturtle_mod.managed_file_var_prefix("foo", "etc/foo/a.yaml") + pb = jinjaturtle_mod.managed_file_var_prefix("foo", "etc/foo/b.yaml") templated, vars_text = jinjify_managed_files( bundle, @@ -188,14 +197,14 @@ def test_jinjify_managed_files_namespaces_multiple_templates( assert templated == {"etc/foo/a.yaml", "etc/foo/b.yaml"} assert calls == [ - ("a.yaml", "foo_jt_etc_foo_a_yaml"), - ("b.yaml", "foo_jt_etc_foo_b_yaml"), + ("a.yaml", pa), + ("b.yaml", pb), ] - assert "foo_jt_etc_foo_a_yaml_ignore: []" in vars_text - assert "foo_jt_etc_foo_b_yaml_ignore: []" in vars_text + assert f"{pa}_ignore: []" in vars_text + assert f"{pb}_ignore: []" in vars_text assert (template_root / "etc" / "foo" / "a.yaml.j2").read_text( encoding="utf-8" - ) == "ignore: {{ foo_jt_etc_foo_a_yaml_ignore }}\n" + ) == f"ignore: {{{{ {pa}_ignore }}}}\n" def test_jinjify_managed_files_rejects_templates_with_missing_defaults( @@ -254,6 +263,8 @@ def test_jinjify_managed_files_always_namespaces_single_template( ) monkeypatch.setattr(jinjaturtle_mod, "run_jinjaturtle", fake_run_jinjaturtle) + monkeypatch.setattr(jinjaturtle_mod, "_VAR_PREFIX_SALT", "t") + prefix = jinjaturtle_mod.managed_file_var_prefix("foo", "etc/foo.yaml") templated, vars_text = jinjify_managed_files( bundle, @@ -267,8 +278,8 @@ def test_jinjify_managed_files_always_namespaces_single_template( ) assert templated == {"etc/foo.yaml"} - assert calls == ["foo_jt_etc_foo_yaml"] - assert "foo_jt_etc_foo_yaml_managed_files: []" in vars_text + assert calls == [prefix] + assert f"{prefix}_managed_files: []" in vars_text assert "foo_managed_files:" not in vars_text @@ -307,6 +318,50 @@ def test_jinjify_managed_files_rejects_reserved_role_variable_collision( assert not (template_root / "etc" / "foo.yaml.j2").exists() +def test_malicious_json_key_falls_back_to_raw_copy(monkeypatch, tmp_path: Path): + """A harvested JSON config that injects a Jinja construct in an object key + must not produce a template. The real JinjaTurtle output-safety gate refuses + it (exit 2), so Enroll falls back to copying the raw file verbatim. + + Uses the real jinjaturtle binary when present; skipped otherwise. + """ + import shutil as _shutil + + import pytest + + from enroll.jinjaturtle import jinjify_managed_files + + jt_exe = _shutil.which("jinjaturtle") + if not jt_exe: + pytest.skip("jinjaturtle binary not on PATH") + + bundle = tmp_path / "bundle" + template_root = tmp_path / "templates" + artifact = bundle / "artifacts" / "foo" / "etc" / "app.json" + artifact.parent.mkdir(parents=True, exist_ok=True) + # Injected key references a (predictable) Enroll-declared variable name; the + # JSON-key gate must still refuse it regardless of the salt. + artifact.write_text( + '{ "{{ ansible_hostname }}": "x", "port": 8080 }', encoding="utf-8" + ) + + templated, vars_text = jinjify_managed_files( + bundle, + "foo", + template_root, + [{"path": "/etc/app.json", "src_rel": "etc/app.json"}], + jt_exe=jt_exe, + jt_enabled=True, + overwrite_templates=True, + role_name="foo", + ) + + # No template emitted; the raw file is left to be copied verbatim downstream. + assert templated == set() + assert vars_text == "" + assert not (template_root / "etc" / "app.json.j2").exists() + + def test_jinjify_artifact_rejects_unsafe_src_rel(monkeypatch, tmp_path: Path): from enroll.jinjaturtle import jinjify_artifact From a2dc054882b6b094ec6bbbec7cedfb1f5ecea995 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Sun, 28 Jun 2026 21:07:49 +1000 Subject: [PATCH 10/13] Close TOCTOU gap by freezing directory harvest bundles into a private 0700 copy before the two consumers that actually read artifacts (manifest and diff) touch them. --- enroll/ansible.py | 15 ---- enroll/diff.py | 20 ++++- enroll/ignore.py | 1 - enroll/manifest.py | 12 ++- enroll/manifest_safety.py | 143 +++++++++++++++++++++++++++++++++- tests/test_diff_bundle.py | 91 +++++++++++++++++++++- tests/test_manifest_safety.py | 89 +++++++++++++++++++++ 7 files changed, 347 insertions(+), 24 deletions(-) diff --git a/enroll/ansible.py b/enroll/ansible.py index f50a19c..75e077a 100644 --- a/enroll/ansible.py +++ b/enroll/ansible.py @@ -1275,21 +1275,6 @@ def _render_role_handlers( return "---\n" + (body.rstrip() + "\n" if body else "") -# --- Ansible variable builders --- -def _normalise_flatpak_item( - item: Any, - *, - method: str, - user: Optional[str] = None, - home: Optional[str] = None, -) -> Dict[str, Any]: - return CMModule.normalise_flatpak_item(item, method=method, user=user, home=home) - - -def _normalise_flatpak_remote(item: Any) -> Dict[str, Any]: - return CMModule.normalise_flatpak_remote(item) - - def _normalise_snap_item(item: Any) -> Dict[str, Any]: out = CMModule.normalise_snap_item(item) diff --git a/enroll/diff.py b/enroll/diff.py index 2d4d6ce..8882d31 100644 --- a/enroll/diff.py +++ b/enroll/diff.py @@ -28,6 +28,7 @@ from .state import ( ) from .pathfilter import PathFilter from .sopsutil import decrypt_file_binary_to, require_sops_cmd +from .manifest_safety import freeze_directory_bundle def _validate_diff_bundle(label: str, bundle_dir: Path) -> None: @@ -145,7 +146,9 @@ class BundleRef: return state_path(self.dir) -def _bundle_from_input(path: str, *, sops_mode: bool) -> BundleRef: +def _bundle_from_input( + path: str, *, sops_mode: bool, freeze: bool = False +) -> BundleRef: """Resolve a user-supplied path to a harvest bundle directory. Accepts: @@ -153,6 +156,14 @@ def _bundle_from_input(path: str, *, sops_mode: bool) -> BundleRef: - a path to state.json inside a bundle directory - (sops mode or .sops) a SOPS-encrypted tar.gz bundle - a plain tar.gz/tgz bundle + + When ``freeze`` is True, a plain *directory* input is copied into a private + 0700 temp directory (no-follow, regular-files-only) before being returned, so + a later consumer cannot be raced by an unprivileged owner mutating the source + directory after validation. Tar/SOPS inputs are always extracted into a + private temp directory and so are inherently frozen. ``freeze`` is left False + for purely diagnostic callers (e.g. ``validate``) that should report on the + exact directory the operator named rather than on a copy of it. """ p = Path(path).expanduser() @@ -162,6 +173,9 @@ def _bundle_from_input(path: str, *, sops_mode: bool) -> BundleRef: p = p.parent if p.is_dir(): + if freeze: + frozen_dir, td_frozen = freeze_directory_bundle(p, label="harvest bundle") + return BundleRef(dir=Path(frozen_dir), tempdir=td_frozen) return BundleRef(dir=p) if not p.exists(): @@ -384,8 +398,8 @@ def compare_harvests( Returns (report, has_changes). """ with ExitStack() as stack: - old_b = _bundle_from_input(old_path, sops_mode=sops_mode) - new_b = _bundle_from_input(new_path, sops_mode=sops_mode) + old_b = _bundle_from_input(old_path, sops_mode=sops_mode, freeze=True) + new_b = _bundle_from_input(new_path, sops_mode=sops_mode, freeze=True) if old_b.tempdir: stack.callback(old_b.tempdir.cleanup) if new_b.tempdir: diff --git a/enroll/ignore.py b/enroll/ignore.py index d93ba74..53529be 100644 --- a/enroll/ignore.py +++ b/enroll/ignore.py @@ -191,7 +191,6 @@ class IgnorePolicy: deny_globs: Optional[list[str]] = None allow_binary_globs: Optional[list[str]] = None max_file_bytes: int = 256_000 - sample_bytes: int = 64_000 # If True, be much less conservative about collecting potentially # sensitive files. This disables deny globs (e.g. /etc/shadow, # /etc/ssl/private/*) and skips heuristic content scanning. diff --git a/enroll/manifest.py b/enroll/manifest.py index 3fea7b8..0d97330 100644 --- a/enroll/manifest.py +++ b/enroll/manifest.py @@ -9,7 +9,7 @@ from typing import List, Optional from .ansible import manifest_from_bundle_dir as manifest_ansible_from_bundle_dir from .harvest_safety import ensure_safe_output_parent -from .manifest_safety import validate_site_fqdn +from .manifest_safety import freeze_directory_bundle, validate_site_fqdn from .remote import _safe_extract_tar from .sopsutil import ( decrypt_file_binary_to, @@ -34,7 +34,15 @@ def _prepare_bundle_dir( p = Path(bundle).expanduser() if p.is_dir(): - return str(p), None + # A directory bundle may still be writable by an unprivileged user (e.g. + # root running `manifest` against /tmp/some-harvest). Validation is a + # point-in-time check, but the renderer/JinjaTurtle re-open artifacts by + # path afterwards, so a mutable source could be raced. Freeze the bundle + # into a private 0700 temp copy (no-follow, regular-files-only) and + # consume that immutable copy instead. Tar/SOPS inputs already get an + # equivalent private extraction below. + frozen_dir, td_frozen = freeze_directory_bundle(p, label="harvest bundle") + return frozen_dir, td_frozen if not sops_mode: raise RuntimeError(f"Harvest path is not a directory: {p}") diff --git a/enroll/manifest_safety.py b/enroll/manifest_safety.py index c7e64e1..a060e81 100644 --- a/enroll/manifest_safety.py +++ b/enroll/manifest_safety.py @@ -1,12 +1,15 @@ from __future__ import annotations +import errno import os import re import shutil import stat +import tempfile from pathlib import Path -from typing import Iterator, Tuple +from typing import Iterator, Optional, Tuple +from .fsutil import open_no_follow_path from .harvest_safety import ( OutputSafetyError, ensure_safe_output_parent, @@ -256,3 +259,141 @@ def copy_safe_artifact_file(src: str | Path, dst: str | Path) -> None: """Copy an already validated artifact file without following symlinks.""" shutil.copy2(src, dst, follow_symlinks=False) + + +_FREEZE_MAX_FILE_BYTES = 64 * 1024 * 1024 +_FREEZE_MAX_FILES = 200_000 + + +def _read_all_no_follow(abs_path: str) -> bytes: + """Read a regular file's bytes via a no-follow, non-hardlinked open. + + Mirrors the harvest-side capture discipline: open every path component + without following symlinks, fstat the resulting descriptor, and refuse + anything that is not a single-linked regular file. This is what makes the + frozen copy independent of later mutation of the source tree -- the bytes + are taken from the descriptor we validated, not re-resolved by path. + """ + + fd: Optional[int] = None + try: + try: + fd = open_no_follow_path(abs_path) + except OSError as e: + if e.errno == errno.ELOOP: + raise ArtifactSafetyError( + f"bundle path contains a symlink component: {abs_path}" + ) from e + if e.errno == errno.ENOTDIR: + raise ArtifactSafetyError( + f"bundle path has a non-directory component: {abs_path}" + ) from e + raise ArtifactSafetyError(f"unable to read bundle file: {abs_path}") from e + + st = os.fstat(fd) + if not stat.S_ISREG(st.st_mode): + raise ArtifactSafetyError(f"bundle file is not a regular file: {abs_path}") + if st.st_nlink > 1: + raise ArtifactSafetyError(f"bundle file is hardlinked: {abs_path}") + if st.st_size > _FREEZE_MAX_FILE_BYTES: + raise ArtifactSafetyError(f"bundle file is too large to freeze: {abs_path}") + + chunks: list[bytes] = [] + remaining = int(st.st_size) + while remaining > 0: + chunk = os.read(fd, min(1024 * 1024, remaining)) + if not chunk: + break + chunks.append(chunk) + remaining -= len(chunk) + return b"".join(chunks) + finally: + if fd is not None: + try: + os.close(fd) + except OSError: + pass + + +def freeze_directory_bundle( + bundle_dir: str | Path, *, label: str = "harvest bundle" +) -> Tuple[str, tempfile.TemporaryDirectory]: + """Copy a directory harvest bundle into a private, immutable-by-attacker tree. + + Validation of a harvest bundle (schema + artifact safety) is point-in-time: + it proves the tree is safe *at the moment it is checked*. When the bundle + directory is still writable by an unprivileged user, an attacker can race the + later consumers (``manifest``/``diff``/``explain`` re-open artifacts by path + to copy, hash, or feed to JinjaTurtle) and swap a validated regular file for a + symlink or different file after the check. + + Tar and SOPS inputs already avoid this: Enroll extracts them into a private + ``0700`` temp directory before validating. Plain *directory* inputs were used + in place. This helper closes that gap by giving directory inputs the same + treatment: it copies the bundle into a fresh ``0700`` temp directory using + no-follow traversal, taking each file's bytes from a validated descriptor + (no symlinks, no hardlinks, regular files only). The returned copy is what the + caller should validate and render from, so a subsequent mutation of the + original directory cannot affect the consumed bundle. + + Returns ``(frozen_bundle_dir, tempdir)``. The caller must keep ``tempdir`` + alive for the lifetime of the bundle use; it cleans up on close. + """ + + src_root = Path(bundle_dir).expanduser() + if not src_root.is_dir(): + raise ArtifactSafetyError(f"{label} is not a directory: {src_root}") + + td = tempfile.TemporaryDirectory(prefix="enroll-frozen-bundle-") + try: + dst_root = Path(td.name) / "bundle" + dst_root.mkdir(mode=0o700, parents=True, exist_ok=False) + try: + os.chmod(dst_root, 0o700) + except OSError: + pass + + file_count = 0 + # followlinks=False: do not descend into symlinked directories. Each + # discovered file is independently re-opened no-follow before copying, so + # a symlinked directory cannot smuggle content into the frozen tree even + # if it is swapped in mid-walk. + for cur, dirs, files in os.walk(src_root, followlinks=False): + cur_p = Path(cur) + + # Refuse symlinked subdirectories rather than silently skipping them, + # so a tampered bundle fails loudly instead of producing a partial + # frozen copy that later diverges from the operator's expectation. + for dname in list(dirs): + dp = cur_p / dname + try: + dst = dp.lstat() + except FileNotFoundError: + dirs.remove(dname) + continue + if stat.S_ISLNK(dst.st_mode): + raise ArtifactSafetyError( + f"{label} contains a symlinked directory: {dp}" + ) + + rel_dir = cur_p.relative_to(src_root) + target_dir = dst_root / rel_dir + target_dir.mkdir(mode=0o700, parents=True, exist_ok=True) + + for fname in files: + file_count += 1 + if file_count > _FREEZE_MAX_FILES: + raise ArtifactSafetyError( + f"{label} has too many files to freeze safely" + ) + src_file = cur_p / fname + data = _read_all_no_follow(str(src_file)) + dst_file = target_dir / fname + fd = open_no_follow_path(str(dst_file), write=True, mode=0o600) + with os.fdopen(fd, "wb") as fh: + fh.write(data) + + return str(dst_root), td + except BaseException: + td.cleanup() + raise diff --git a/tests/test_diff_bundle.py b/tests/test_diff_bundle.py index 34780eb..3b37e19 100644 --- a/tests/test_diff_bundle.py +++ b/tests/test_diff_bundle.py @@ -7,6 +7,7 @@ from pathlib import Path import pytest from enroll.ansible import _role_tag +from enroll.manifest_safety import freeze_directory_bundle from enroll.diff import ( _Spinner, _utc_now_iso, @@ -34,6 +35,8 @@ def test_bundle_from_directory_and_statejson_path(tmp_path: Path): b = _make_bundle_dir(tmp_path) + # Plain resolution (no freeze) returns the directory in place. Freezing is + # opt-in and exercised by the consumer-level tests below. br1 = d._bundle_from_input(str(b), sops_mode=False) assert br1.dir == b assert br1.state_path.exists() @@ -42,6 +45,31 @@ def test_bundle_from_directory_and_statejson_path(tmp_path: Path): assert br2.dir == b +def test_bundle_from_input_directory_freeze_copies_and_protects(tmp_path: Path): + import enroll.diff as d + + b = _make_bundle_dir(tmp_path) + art = b / "artifacts" / "app" + art.mkdir(parents=True) + (art / "f.conf").write_text("orig\n", encoding="utf-8") + + # With freeze=True the bundle is copied into a private temp dir, and mutating + # the source afterwards does not change the frozen copy. + result = d._bundle_from_input(str(b), sops_mode=False, freeze=True) + try: + assert result.dir != b + assert result.tempdir is not None + frozen_f = result.dir / "artifacts" / "app" / "f.conf" + assert frozen_f.read_text(encoding="utf-8") == "orig\n" + + # Attacker rewrites the source after the freeze; frozen copy is unaffected. + (art / "f.conf").write_text("tampered\n", encoding="utf-8") + assert frozen_f.read_text(encoding="utf-8") == "orig\n" + finally: + if result.tempdir: + result.tempdir.cleanup() + + def test_bundle_from_tarball_extracts(tmp_path: Path): import enroll.diff as d @@ -742,9 +770,68 @@ def test_compare_harvests_rejects_unsafe_artifact_symlink(tmp_path: Path): with pytest.raises(RuntimeError) as exc_info: compare_harvests(str(old_bundle), str(new_bundle)) + # The unsafe symlinked artifact is now rejected when the directory bundle is + # frozen into a private copy (no-follow traversal), which happens before + # validation. The diff is still refused; the rejection just fires earlier. msg = str(exc_info.value) - assert "old harvest failed validation" in msg - assert "artifact is a symlink" in msg + assert "symlink" in msg.lower() + + +def test_compare_harvests_freezes_directory_bundles_against_source_mutation( + tmp_path: Path, +): + """End-to-end: a directory diff must read frozen copies, so mutating a source + bundle's artifact after compare_harvests has frozen it cannot change the + result or redirect a read to a secret.""" + + def _bundle(name: str, content: str) -> Path: + b = tmp_path / name + art = b / "artifacts" / "users" + art.mkdir(parents=True) + (art / "passwd").write_text(content, encoding="utf-8") + (b / "state.json").write_text( + json.dumps( + { + "inventory": {"packages": {}}, + "roles": { + "users": { + "managed_files": [ + {"path": "/etc/passwd", "src_rel": "passwd"} + ] + } + }, + } + ), + encoding="utf-8", + ) + return b + + old_bundle = _bundle("old", "same\n") + new_bundle = _bundle("new", "same\n") + + # Identical content -> no file content drift. + report, _ = compare_harvests(str(old_bundle), str(new_bundle)) + changed_paths = [f["path"] for f in report["files"]["changed"]] + assert "/etc/passwd" not in changed_paths + + # The freeze already happened inside compare_harvests above and was torn down, + # so to prove immunity during a single call we instead confirm that the frozen + # copy is what gets hashed: rewrite the source between freeze and hash is not + # observable to the operator because the consumed tree is a private copy. + # Confirm at the unit level that a post-freeze swap to a secret symlink is not + # followed by the hashing path. + secret = tmp_path / "secret" + secret.write_text("TOP-SECRET\n", encoding="utf-8") + frozen_dir, td = freeze_directory_bundle(old_bundle) + try: + src = old_bundle / "artifacts" / "users" / "passwd" + src.unlink() + src.symlink_to(secret) + frozen = Path(frozen_dir) / "artifacts" / "users" / "passwd" + assert not frozen.is_symlink() + assert frozen.read_text(encoding="utf-8") == "same\n" + finally: + td.cleanup() def test_utc_now_iso(): diff --git a/tests/test_manifest_safety.py b/tests/test_manifest_safety.py index deb764d..b379941 100644 --- a/tests/test_manifest_safety.py +++ b/tests/test_manifest_safety.py @@ -9,6 +9,7 @@ from enroll.manifest_safety import ( ArtifactSafetyError, ManifestOutputError, copy_safe_artifact_file, + freeze_directory_bundle, iter_safe_artifact_files, prepare_manifest_output_dir, safe_artifact_file, @@ -124,3 +125,91 @@ def test_iter_safe_artifact_files_rejects_symlink_subdir(tmp_path: Path): with pytest.raises(ArtifactSafetyError, match="directory is a symlink"): list(iter_safe_artifact_files(bundle, "role")) + + +def _write_bundle(tmp_path: Path) -> Path: + bundle = tmp_path / "bundle" + art = bundle / "artifacts" / "app" / "etc" / "app" + art.mkdir(parents=True) + (art / "app.conf").write_text("original-content\n", encoding="utf-8") + (bundle / "state.json").write_text("{}\n", encoding="utf-8") + return bundle + + +def test_freeze_directory_bundle_copies_content(tmp_path: Path): + bundle = _write_bundle(tmp_path) + frozen_dir, td = freeze_directory_bundle(bundle) + try: + frozen = Path(frozen_dir) + assert frozen != bundle + assert (frozen / "state.json").read_text(encoding="utf-8") == "{}\n" + assert (frozen / "artifacts" / "app" / "etc" / "app" / "app.conf").read_text( + encoding="utf-8" + ) == "original-content\n" + # The frozen copy is a private directory. + mode = (Path(frozen_dir)).stat().st_mode & 0o777 + assert mode == 0o700 + finally: + td.cleanup() + + +def test_freeze_directory_bundle_is_immutable_after_source_swap(tmp_path: Path): + """The core TOCTOU regression: mutating the source after freezing must not + change what a consumer reads, and must not let a post-freeze symlink redirect + reads to a secret.""" + bundle = _write_bundle(tmp_path) + art_file = bundle / "artifacts" / "app" / "etc" / "app" / "app.conf" + + frozen_dir, td = freeze_directory_bundle(bundle) + try: + secret = tmp_path / "secret" + secret.write_text("TOP-SECRET\n", encoding="utf-8") + + # Attacker swaps the validated regular file for a symlink to a secret and + # also rewrites another path's content. None of this may affect the frozen + # copy the consumer actually uses. + art_file.unlink() + art_file.symlink_to(secret) + + frozen_file = ( + Path(frozen_dir) / "artifacts" / "app" / "etc" / "app" / "app.conf" + ) + assert not frozen_file.is_symlink() + assert frozen_file.read_text(encoding="utf-8") == "original-content\n" + assert "TOP-SECRET" not in frozen_file.read_text(encoding="utf-8") + finally: + td.cleanup() + + +def test_freeze_directory_bundle_rejects_symlinked_file(tmp_path: Path): + bundle = _write_bundle(tmp_path) + secret = tmp_path / "secret" + secret.write_text("secret\n", encoding="utf-8") + # A bundle that already contains a symlinked artifact at freeze time is + # rejected outright rather than silently following it. + link = bundle / "artifacts" / "app" / "etc" / "app" / "link.conf" + link.symlink_to(secret) + with pytest.raises(ArtifactSafetyError, match="symlink"): + freeze_directory_bundle(bundle) + + +def test_freeze_directory_bundle_rejects_symlinked_subdir(tmp_path: Path): + bundle = _write_bundle(tmp_path) + outside = tmp_path / "outside" + outside.mkdir() + (outside / "x.conf").write_text("x\n", encoding="utf-8") + (bundle / "artifacts" / "app" / "linkdir").symlink_to( + outside, target_is_directory=True + ) + with pytest.raises(ArtifactSafetyError, match="symlink"): + freeze_directory_bundle(bundle) + + +def test_freeze_directory_bundle_rejects_hardlinked_file(tmp_path: Path): + bundle = _write_bundle(tmp_path) + secret = tmp_path / "secret" + secret.write_text("secret\n", encoding="utf-8") + hard = bundle / "artifacts" / "app" / "etc" / "app" / "hard.conf" + os.link(secret, hard) + with pytest.raises(ArtifactSafetyError, match="hardlink"): + freeze_directory_bundle(bundle) From ddb18403c836a64f562e03ec9a3299582c08b377 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Mon, 29 Jun 2026 08:56:06 +1000 Subject: [PATCH 11/13] fail loudly on an empty/truncated frozen harvest bundle. Tighten as much as we can the remote harvesting --- enroll/manifest_safety.py | 23 +++- enroll/remote.py | 158 +++++++++++++++++++++++++-- tests/test_manifest_safety.py | 22 ++++ tests/test_remote.py | 195 ++++++++++++++++++++++++++++------ 4 files changed, 358 insertions(+), 40 deletions(-) diff --git a/enroll/manifest_safety.py b/enroll/manifest_safety.py index a060e81..414adff 100644 --- a/enroll/manifest_safety.py +++ b/enroll/manifest_safety.py @@ -354,11 +354,32 @@ def freeze_directory_bundle( pass file_count = 0 + + def _on_walk_error(exc: OSError) -> None: + # os.walk() defaults to *silently swallowing* directory-listing + # errors (e.g. an unreadable subdirectory raises os.scandir() -> + # PermissionError, which os.walk would otherwise drop). A swallowed + # error produces a partial frozen tree with no indication that + # content was omitted, which is exactly the "fail loudly instead of + # producing a partial frozen copy" guarantee this helper is meant to + # provide. Re-raise as an ArtifactSafetyError so the whole freeze + # aborts rather than returning a silently-truncated bundle. + raise ArtifactSafetyError( + f"{label} could not be fully read while freezing " + f"({exc.__class__.__name__}: {exc}); refusing to produce a " + f"partial frozen copy" + ) + # followlinks=False: do not descend into symlinked directories. Each # discovered file is independently re-opened no-follow before copying, so # a symlinked directory cannot smuggle content into the frozen tree even # if it is swapped in mid-walk. - for cur, dirs, files in os.walk(src_root, followlinks=False): + # + # onerror=_on_walk_error: fail closed on any unreadable directory rather + # than silently skipping it (see callback above). + for cur, dirs, files in os.walk( + src_root, followlinks=False, onerror=_on_walk_error + ): cur_p = Path(cur) # Refuse symlinked subdirectories rather than silently skipping them, diff --git a/enroll/remote.py b/enroll/remote.py index ecb1c27..78549e6 100644 --- a/enroll/remote.py +++ b/enroll/remote.py @@ -1,6 +1,7 @@ from __future__ import annotations import getpass +import hashlib import os import shlex import shutil @@ -232,11 +233,21 @@ def _safe_extract_tar(tar: tarfile.TarFile, dest: Path) -> None: tar.extract(m, path=dest) -def _build_enroll_pyz(tmpdir: Path) -> Path: +def _build_enroll_pyz(tmpdir: Path) -> tuple[Path, str]: """Build a self-contained enroll zipapp (pyz) on the local machine. The resulting file is stdlib-only and can be executed on the remote host as long as it has Python 3 available. + + Returns ``(pyz_path, sha256_hex)``. The digest is computed on the exact + bytes written locally so the caller can verify, on the remote side, that the + file that is about to be executed as root is byte-for-byte the one we built + (see ``_remote_verify_pyz_sha256``). This is transport/staging integrity + defence-in-depth: it detects a swap of the staged file between upload and + execution by anyone who gained write access to the staging directory. It is + NOT a defence against a remote host that is already root-compromised -- such + a host can subvert the interpreter regardless, and is out of scope per + SECURITY.md. """ import enroll as pkg @@ -244,15 +255,58 @@ def _build_enroll_pyz(tmpdir: Path) -> Path: stage = tmpdir / "stage" (stage / "enroll").mkdir(parents=True, exist_ok=True) - def _ignore(d: str, names: list[str]) -> set[str]: - return { - n - for n in names - if n in {"__pycache__", ".pytest_cache"} or n.endswith(".pyc") - } + # Names that must never end up in the remote zipapp. The remote only ever + # runs ``harvest``; test suites, caches, editor/VCS scratch, and compiled + # artifacts are never needed at runtime and should not be shipped to (or + # executed on) a harvested host. Excluding them keeps the payload minimal + # and avoids transferring irrelevant code to every target. + _ignore_dirs = { + "__pycache__", + ".pytest_cache", + ".mypy_cache", + ".ruff_cache", + ".git", + ".hg", + ".svn", + "tests", + "test", + } + _ignore_suffixes = (".pyc", ".pyo", ".orig", ".rej", ".bak") + + def _ignore(directory: str, names: list[str]) -> set[str]: + dropped: set[str] = set() + for n in names: + if n in _ignore_dirs: + dropped.add(n) + continue + if n.endswith(_ignore_suffixes): + dropped.add(n) + continue + # Defensive: never ship test modules even if they are colocated in + # the package directory by a future packaging change. + if n.startswith("test_") and n.endswith(".py"): + dropped.add(n) + continue + if n == "conftest.py": + dropped.add(n) + continue + return dropped shutil.copytree(pkg_dir, stage / "enroll", dirs_exist_ok=True, ignore=_ignore) + # The JSON Schema is a required runtime data file for ``validate``/``manifest`` + # consumers of the harvest; the remote harvest itself does not validate, but + # the bundle it produces is validated locally, and the schema travels with + # the package. Fail loudly if a future ignore rule ever drops it rather than + # silently shipping a package that cannot self-validate. + staged_schema = stage / "enroll" / "schema" / "state.schema.json" + if not staged_schema.is_file(): + raise RuntimeError( + "internal error: enroll.pyz staging is missing the bundled JSON " + "schema (schema/state.schema.json); refusing to build an " + "incomplete remote payload" + ) + pyz_path = tmpdir / "enroll.pyz" zipapp.create_archive( stage, @@ -260,7 +314,83 @@ def _build_enroll_pyz(tmpdir: Path) -> Path: main="enroll.cli:main", compressed=True, ) - return pyz_path + + sha256_hex = _sha256_file(pyz_path) + return pyz_path, sha256_hex + + +def _sha256_file(path: Path) -> str: + """Return the hex SHA-256 of a file, read in chunks.""" + h = hashlib.sha256() + with open(path, "rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + h.update(chunk) + return h.hexdigest() + + +def _remote_verify_pyz_sha256( + ssh, + remote_pyz_path: str, + expected_sha256: str, + *, + remote_python: str, +) -> None: + """Verify the uploaded zipapp's SHA-256 on the remote before executing it. + + This is transport/staging integrity defence-in-depth. The check runs on the + remote, immediately before the (root) execution of the zipapp, and fails + closed if the digest does not match the bytes we built locally. It shrinks + the window in which a *non-root* tamperer who somehow gained write access to + the staging directory could swap the file between upload and execution. + + It deliberately does NOT establish trust in a root-compromised remote: a + host that is already root can forge any check it runs about itself. Per + SECURITY.md, such a host is outside Enroll's threat model. The value here is + catching accidental corruption and unprivileged-local-user staging races, + not defeating a compromised root. + + The hashing is done with Python's hashlib (already required to run the + zipapp) rather than a ``sha256sum`` binary, so it does not depend on + coreutils being present or on PATH resolution of a hashing tool. + """ + + # Hash the staged file using the same interpreter that will execute it. + # ``-I`` isolates the interpreter from site/user customisation; the script + # prints only the lowercase hex digest. + hash_script = ( + "import hashlib,sys;" + "h=hashlib.sha256();" + "f=open(sys.argv[1],'rb');" + "[h.update(c) for c in iter(lambda:f.read(1048576),b'')];" + "sys.stdout.write(h.hexdigest())" + ) + cmd = " ".join( + shlex.quote(tok) + for tok in (remote_python, "-I", "-c", hash_script, remote_pyz_path) + ) + rc, out, err = _ssh_run(ssh, cmd, get_pty=False) + if rc != 0: + raise RuntimeError( + "Failed to verify the integrity of the uploaded enroll.pyz on the " + f"remote host.\nCommand: {cmd}\nExit code: {rc}\nStderr: {err.strip()}" + ) + remote_digest = out.strip().lower() + expected = expected_sha256.strip().lower() + if not remote_digest: + raise RuntimeError( + "Remote integrity check returned an empty SHA-256 for enroll.pyz; " + "refusing to execute it." + ) + if remote_digest != expected: + raise RuntimeError( + "Integrity check failed for the uploaded enroll.pyz: the staged " + "file's SHA-256 on the remote does not match the locally built " + "payload. Refusing to execute it as root.\n" + f" expected: {expected}\n" + f" remote: {remote_digest}\n" + "This can indicate the staging directory was tampered with between " + "upload and execution, or transfer corruption." + ) def _ssh_run( @@ -444,7 +574,7 @@ def _remote_harvest( # Build a zipapp locally and upload it to the remote. with tempfile.TemporaryDirectory(prefix="enroll-remote-") as td: td_path = Path(td) - pyz = _build_enroll_pyz(td_path) + pyz, pyz_sha256 = _build_enroll_pyz(td_path) local_tgz = td_path / "bundle.tgz" ssh = paramiko.SSHClient() @@ -595,6 +725,16 @@ def _remote_harvest( rapp = f"{rtmp}/enroll.pyz" sftp.put(str(pyz), rapp) + # Before executing the uploaded zipapp (as root, under sudo), verify + # on the remote that the staged bytes match what we built locally. + # This is staging/transport integrity defence-in-depth: it fails + # closed if the file was swapped or corrupted between upload and + # execution. It does not (and cannot) defend against a remote that + # is already root-compromised; see _remote_verify_pyz_sha256. + _remote_verify_pyz_sha256( + ssh, rapp, pyz_sha256, remote_python=remote_python + ) + if not no_sudo: # The remote zipapp is staged as the SSH user, but the harvest # itself runs as root. Root must not write its bundle under the diff --git a/tests/test_manifest_safety.py b/tests/test_manifest_safety.py index b379941..fa43490 100644 --- a/tests/test_manifest_safety.py +++ b/tests/test_manifest_safety.py @@ -213,3 +213,25 @@ def test_freeze_directory_bundle_rejects_hardlinked_file(tmp_path: Path): os.link(secret, hard) with pytest.raises(ArtifactSafetyError, match="hardlink"): freeze_directory_bundle(bundle) + + +def test_freeze_directory_bundle_fails_loudly_on_unreadable_subdir(tmp_path: Path): + """An unreadable subtree must abort the freeze, not silently truncate it. + + Regression test: os.walk() defaults to swallowing directory-listing errors, + which previously produced a partial frozen bundle with no error. The freeze + now passes onerror= so a PermissionError aborts with ArtifactSafetyError. + """ + if os.geteuid() == 0: + pytest.skip("root can read 0000 directories; cannot simulate the case") + + bundle = _write_bundle(tmp_path) + locked = bundle / "artifacts" / "app" / "etc" / "app" + assert locked.is_dir() + os.chmod(locked, 0o000) + try: + with pytest.raises(ArtifactSafetyError, match="could not be fully read"): + freeze_directory_bundle(bundle) + finally: + # Restore perms so tmp_path cleanup can remove the tree. + os.chmod(locked, 0o755) diff --git a/tests/test_remote.py b/tests/test_remote.py index fe54d33..dc486ac 100644 --- a/tests/test_remote.py +++ b/tests/test_remote.py @@ -8,6 +8,30 @@ from pathlib import Path import pytest +# The remote harvest now SHA-256-verifies the uploaded zipapp on the remote +# before executing it. Tests mock _build_enroll_pyz to write these fixed bytes +# and return (path, _FAKE_PYZ_SHA256); the fake SSH routers below answer the +# verification command with the same digest so the happy paths proceed. +_FAKE_PYZ_BYTES = b"PYZ" +_FAKE_PYZ_SHA256 = "d6f4e1dbf7ba69af6c798c6f6f67383c978e68f4201bf31902275ef37e6263e1" + + +def _fake_build_enroll_pyz(td) -> tuple[Path, str]: + """Stand-in for remote._build_enroll_pyz used across remote tests. + + Writes a tiny fixed payload instead of a real zipapp and returns the + (path, sha256) tuple the production builder now returns. + """ + p = Path(td) / "enroll.pyz" + p.write_bytes(_FAKE_PYZ_BYTES) + return p, _FAKE_PYZ_SHA256 + + +def _is_pyz_verify_cmd(cmd: str) -> bool: + """True if *cmd* is the remote pyz SHA-256 integrity check.""" + return "hashlib.sha256" in cmd and "enroll.pyz" in cmd + + def _make_tgz_bytes(files: dict[str, bytes]) -> bytes: bio = io.BytesIO() with tarfile.open(fileobj=bio, mode="w:gz") as tf: @@ -56,12 +80,7 @@ def test_remote_harvest_happy_path(tmp_path: Path, monkeypatch): import enroll.remote as r # Avoid building a real zipapp; just create a file. - def fake_build(_td: Path) -> Path: - p = _td / "enroll.pyz" - p.write_bytes(b"PYZ") - return p - - monkeypatch.setattr(r, "_build_enroll_pyz", fake_build) + monkeypatch.setattr(r, "_build_enroll_pyz", _fake_build_enroll_pyz) # Prepare a tiny harvest bundle tar stream from the "remote". tgz = _make_tgz_bytes({"state.json": b'{"ok": true}\n'}) @@ -156,6 +175,9 @@ def test_remote_harvest_happy_path(tmp_path: Path, monkeypatch): return self._sftp def exec_command(self, cmd: str, *, get_pty: bool = False, **_kwargs): + # Integrity check of the uploaded pyz (added with SHA-256 verify). + if _is_pyz_verify_cmd(cmd): + return (None, _Stdout(_FAKE_PYZ_SHA256.encode()), _Stderr()) calls.append((cmd, bool(get_pty))) # The tar stream uses exec_command directly. if cmd.startswith("tar -cz -C"): @@ -264,12 +286,7 @@ def test_remote_harvest_no_sudo_does_not_request_pty_or_chown( import enroll.remote as r - monkeypatch.setattr( - r, - "_build_enroll_pyz", - lambda td: (Path(td) / "enroll.pyz").write_bytes(b"PYZ") - or (Path(td) / "enroll.pyz"), - ) + monkeypatch.setattr(r, "_build_enroll_pyz", _fake_build_enroll_pyz) tgz = _make_tgz_bytes({"state.json": b"{}"}) calls: list[tuple[str, bool]] = [] @@ -357,6 +374,9 @@ def test_remote_harvest_no_sudo_does_not_request_pty_or_chown( return self._sftp def exec_command(self, cmd: str, *, get_pty: bool = False, **_kwargs): + # Integrity check of the uploaded pyz (added with SHA-256 verify). + if _is_pyz_verify_cmd(cmd): + return (None, _Stdout(_FAKE_PYZ_SHA256.encode()), _Stderr()) calls.append((cmd, bool(get_pty))) if cmd == "mktemp -d": return (None, _Stdout(b"/tmp/enroll-remote-456\n"), _Stderr()) @@ -409,12 +429,7 @@ def test_remote_harvest_sudo_password_retry_uses_sudo_s_and_writes_password( import enroll.remote as r # Avoid building a real zipapp; just create a file. - monkeypatch.setattr( - r, - "_build_enroll_pyz", - lambda td: (Path(td) / "enroll.pyz").write_bytes(b"PYZ") - or (Path(td) / "enroll.pyz"), - ) + monkeypatch.setattr(r, "_build_enroll_pyz", _fake_build_enroll_pyz) tgz = _make_tgz_bytes({"state.json": b'{"ok": true}\n'}) calls: list[tuple[str, bool]] = [] @@ -514,6 +529,9 @@ def test_remote_harvest_sudo_password_retry_uses_sudo_s_and_writes_password( return self._sftp def exec_command(self, cmd: str, *, get_pty: bool = False, **_kwargs): + # Integrity check of the uploaded pyz (added with SHA-256 verify). + if _is_pyz_verify_cmd(cmd): + return (None, _Stdout(_FAKE_PYZ_SHA256.encode()), _Stderr()) calls.append((cmd, bool(get_pty))) # Tar stream @@ -800,12 +818,7 @@ def test_remote_harvest_ssh_key_passphrase_retry(monkeypatch, tmp_path: Path): import enroll.remote as r - monkeypatch.setattr( - r, - "_build_enroll_pyz", - lambda td: (Path(td) / "enroll.pyz").write_bytes(b"PYZ") - or (Path(td) / "enroll.pyz"), - ) + monkeypatch.setattr(r, "_build_enroll_pyz", _fake_build_enroll_pyz) tgz = _make_tgz_bytes({"state.json": b'{"ok": true}\n'}) @@ -901,6 +914,9 @@ def test_remote_harvest_ssh_key_passphrase_retry(monkeypatch, tmp_path: Path): return self._sftp def exec_command(self, cmd: str, *, get_pty: bool = False, **_kwargs): + # Integrity check of the uploaded pyz (added with SHA-256 verify). + if _is_pyz_verify_cmd(cmd): + return (None, _Stdout(_FAKE_PYZ_SHA256.encode()), _Stderr()) if cmd.startswith("tar -cz -C"): return (_Stdin(cmd), _Stdout(tgz, rc=0), _Stderr(b"")) if cmd == "mktemp -d": @@ -952,12 +968,7 @@ def test_remote_harvest_ssh_key_passphrase_raises_when_not_interactive( import enroll.remote as r - monkeypatch.setattr( - r, - "_build_enroll_pyz", - lambda td: (Path(td) / "enroll.pyz").write_bytes(b"PYZ") - or (Path(td) / "enroll.pyz"), - ) + monkeypatch.setattr(r, "_build_enroll_pyz", _fake_build_enroll_pyz) class _Chan: def __init__(self): @@ -1022,6 +1033,9 @@ def test_remote_harvest_ssh_key_passphrase_raises_when_not_interactive( return self._sftp def exec_command(self, cmd: str, **_kwargs): + # Integrity check of the uploaded pyz (added with SHA-256 verify). + if _is_pyz_verify_cmd(cmd): + return (None, _Stdout(_FAKE_PYZ_SHA256.encode()), _Stderr()) return (_Stdout(), _Stdout(), _Stderr()) def close(self): @@ -1048,3 +1062,124 @@ def test_remote_harvest_ssh_key_passphrase_raises_when_not_interactive( remote_host="example.com", stdin=io.StringIO(), ) + + +# --------------------------------------------------------------------------- # +# Integrity of the uploaded zipapp before remote (root) execution. +# --------------------------------------------------------------------------- # + + +class _VerifyChan: + """Minimal Paramiko-channel stand-in for _ssh_run consumption.""" + + def __init__(self, out: bytes = b"", err: bytes = b"", rc: int = 0): + self._out, self._err, self._rc = out, err, rc + self._oi = self._ei = 0 + self._closed = False + + def recv_ready(self) -> bool: + return (not self._closed) and self._oi < len(self._out) + + def recv(self, n: int) -> bytes: + chunk = self._out[self._oi : self._oi + n] + self._oi += len(chunk) + return chunk + + def recv_stderr_ready(self) -> bool: + return (not self._closed) and self._ei < len(self._err) + + def recv_stderr(self, n: int) -> bytes: + chunk = self._err[self._ei : self._ei + n] + self._ei += len(chunk) + return chunk + + def exit_status_ready(self) -> bool: + return self._closed or ( + self._oi >= len(self._out) and self._ei >= len(self._err) + ) + + def recv_exit_status(self) -> int: + return self._rc + + def shutdown_write(self) -> None: + return + + def close(self) -> None: + self._closed = True + + +class _VerifyStdio: + def __init__(self, chan: _VerifyChan): + self.channel = chan + + +def _verify_ssh(out: bytes, *, rc: int = 0): + chan = _VerifyChan(out=out, rc=rc) + + class FakeSSH: + def exec_command(self, cmd, *, get_pty=False, **_kw): + assert "hashlib.sha256" in cmd # this is the verify command + return (io.StringIO(), _VerifyStdio(chan), _VerifyStdio(chan)) + + return FakeSSH() + + +def test_remote_verify_pyz_sha256_rejects_mismatch(): + import enroll.remote as r + + ssh = _verify_ssh(b"0" * 64) + with pytest.raises(RuntimeError, match="Integrity check failed"): + r._remote_verify_pyz_sha256( + ssh, "/tmp/x/enroll.pyz", "a" * 64, remote_python="python3" + ) + + +def test_remote_verify_pyz_sha256_accepts_match(): + import enroll.remote as r + + expected = "a" * 64 + ssh = _verify_ssh(expected.encode()) + # Should not raise. + r._remote_verify_pyz_sha256( + ssh, "/tmp/x/enroll.pyz", expected, remote_python="python3" + ) + + +def test_remote_verify_pyz_sha256_rejects_empty(): + import enroll.remote as r + + ssh = _verify_ssh(b"") + with pytest.raises(RuntimeError, match="empty SHA-256"): + r._remote_verify_pyz_sha256( + ssh, "/tmp/x/enroll.pyz", "a" * 64, remote_python="python3" + ) + + +def test_remote_verify_pyz_sha256_rejects_nonzero_rc(): + import enroll.remote as r + + ssh = _verify_ssh(b"", rc=2) + with pytest.raises(RuntimeError, match="verify the integrity"): + r._remote_verify_pyz_sha256( + ssh, "/tmp/x/enroll.pyz", "a" * 64, remote_python="python3" + ) + + +def test_build_enroll_pyz_excludes_tests_and_caches_and_returns_sha(tmp_path: Path): + import zipfile + + import enroll.remote as r + + pyz, sha = r._build_enroll_pyz(tmp_path) + assert isinstance(sha, str) and len(sha) == 64 + assert sha == r._sha256_file(pyz) + + names = zipfile.ZipFile(pyz).namelist() + assert any(n.endswith("schema/state.schema.json") for n in names) + for n in names: + leaf = n.rsplit("/", 1)[-1] + assert "__pycache__" not in n + assert not n.endswith((".pyc", ".pyo")) + assert "/tests/" not in n + assert not leaf.startswith("test_") + assert leaf != "conftest.py" From b3d4adf7d44fdcdd913ce3d97cd881d81a235482 Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Mon, 29 Jun 2026 12:25:35 +1000 Subject: [PATCH 12/13] Show a warning when manifesting from a harvest as reminder it should be trusted first --- enroll/manifest.py | 13 +++++++++++++ tests/test_manifest.py | 18 ++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/enroll/manifest.py b/enroll/manifest.py index 0d97330..b607020 100644 --- a/enroll/manifest.py +++ b/enroll/manifest.py @@ -2,6 +2,7 @@ from __future__ import annotations import os import shutil +import sys import tarfile import tempfile from pathlib import Path @@ -19,6 +20,16 @@ from .sopsutil import ( from .validate import validate_harvest +_SEMANTIC_SAFETY_WARNING = ( + "This harvest is structurally valid, but Enroll cannot prove it is semantically safe.\n" + "Only apply manifests generated from harvests whose provenance you trust.\n" +) + + +def _warn_semantic_safety() -> None: + sys.stderr.write(_SEMANTIC_SAFETY_WARNING) + + def _prepare_bundle_dir( bundle: str, *, @@ -217,6 +228,8 @@ def manifest( + validation.to_text().strip() ) + _warn_semantic_safety() + if not sops_mode: manifest_ansible_from_bundle_dir( resolved_bundle_dir, diff --git a/tests/test_manifest.py b/tests/test_manifest.py index dbd7633..e72ee2e 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -87,6 +87,24 @@ def _write_state(bundle: Path, state: dict) -> None: write_schema_state(bundle, state) +def test_manifest_warns_that_validation_is_not_semantic_safety(tmp_path: Path, capsys): + bundle = tmp_path / "bundle" + out = tmp_path / "ansible" + _write_state(bundle, _minimal_package_state([])) + + manifest.manifest(str(bundle), str(out)) + + captured = capsys.readouterr() + assert ( + "This harvest is structurally valid, but Enroll cannot prove it is semantically safe." + in captured.err + ) + assert ( + "Only apply manifests generated from harvests whose provenance you trust." + in captured.err + ) + + def test_manifest_writes_roles_and_playbook_with_clean_when(tmp_path: Path): bundle = tmp_path / "bundle" out = tmp_path / "ansible" From 7519adc705ae2d85923f85034fcfde419404e69d Mon Sep 17 00:00:00 2001 From: Miguel Jacq Date: Mon, 29 Jun 2026 13:13:33 +1000 Subject: [PATCH 13/13] Update docs --- DEVELOPMENT.md | 53 ++++++++++++++++++++++++++++++++++++++------------ SECURITY.md | 4 +++- 2 files changed, 44 insertions(+), 13 deletions(-) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index b1afaa1..674bac7 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -73,13 +73,17 @@ enroll/ package_hints.py service/package name and config attribution helpers capture.py safe file/symlink capture into artifacts/ - fsutil.py file md5 + owner/group/mode helpers + fsutil.py file md5 + owner/group/mode helpers, open_no_follow_path ignore.py secret/noise avoidance policy pathfilter.py --include-path / --exclude-path matching and expansion state.py state.json load/write helpers yamlutil.py YAML helpers used by renderers/JinjaTurtle jinjaturtle.py optional config-file templating integration + harvest_safety.py private output dirs, safe output parents, output path hardening + manifest_safety.py artifact-file safety, manifest output dirs, directory-bundle freezing + render_safety.py scaffold-token allowlist + !unsafe data wrapping for generated YAML + diff.py harvest comparison and notifications explain.py human/JSON explanation of harvest contents validate.py schema and artifact consistency validation @@ -456,8 +460,10 @@ Several safety helpers protect privileged runs from following attacker-controlle - `cache.new_harvest_cache_dir()` creates unpredictable per-harvest cache directories beneath the hardened cache root with `mkdtemp()` and private permissions. - `manifest_safety.safe_artifact_file()` validates referenced harvested artifacts before renderers copy them. It rejects absolute or `..` paths, symlinks, non-regular files, hardlinks, and paths that resolve outside the artifact root. - `manifest_safety.prepare_manifest_output_dir()` refuses unsafe manifest output paths. In `--fqdn` site mode, where an existing tree is intentionally reused, it walks the existing output tree and refuses symlinks before merging generated files. +- `manifest_safety.freeze_directory_bundle()` copies a *directory* harvest bundle into a fresh private `0700` temp tree before it is validated and consumed. Validation is point-in-time, but `manifest`/`diff` later re-open artifacts by path (to copy, hash, or feed to JinjaTurtle), so a still-writable source directory could be raced by its unprivileged owner — a validated regular file swapped for a symlink or different file after the check. Freezing uses no-follow traversal and takes each file's bytes from a validated descriptor (regular files only, no symlinks, no hardlinks), so a later mutation of the original directory cannot affect the consumed copy. Tar and SOPS inputs are already extracted into a private temp directory and are inherently frozen; this helper gives plain directory inputs the same guarantee. See sections 11 and 15.1 for where it is wired in. +- `render_safety.scaffold_token()` / `render_safety.ansible_unsafe_data()` keep harvested data out of generated playbook *structure*: the former is a strict allowlist for the only values ever spliced into raw task/handler YAML (already-sanitized role/var-prefix identifiers), and the latter recursively tags template-looking harvested strings as Ansible `!unsafe` data so they cannot be re-evaluated as Jinja at apply time. See section 13.6. -When adding a new code path that writes plaintext host state, prefer these helpers over raw `mkdir(parents=True)`, `open()`, `shutil.copy*()`, or `tar.extract*()`. +When adding a new code path that writes plaintext host state, prefer these helpers over raw `mkdir(parents=True)`, `open()`, `shutil.copy*()`, or `tar.extract*()`. When a new consumer re-opens artifacts from a directory bundle, route the bundle through `freeze_directory_bundle()` first. --- @@ -795,7 +801,14 @@ SOPS mode: The renderers do not know about SOPS. -Before dispatching to a renderer, `manifest.manifest()` calls `validate.validate_harvest()` with normal schema validation enabled. That means generated configuration-management code is only rendered after Enroll has checked the bundle schema, referenced artifact existence, and artifact safety. If validation fails, manifest generation stops rather than trying to produce best-effort output from a malformed or tampered bundle. +Bundle preparation (`_prepare_bundle_dir()`) runs before validation in both modes: + +- A *directory* bundle is frozen with `manifest_safety.freeze_directory_bundle()` into a private `0700` temp copy, and the frozen copy is what gets validated and rendered (see section 7.6). This holds even in SOPS mode when the input is an already-decrypted directory. +- A SOPS-encrypted tarball is decrypted and extracted into a private temp directory with `remote._safe_extract_tar()`. + +Either way the bundle Enroll consumes is an immutable-by-attacker private copy, not the path the operator named. + +Before dispatching to a renderer, `manifest.manifest()` calls `validate.validate_harvest()` on the prepared (frozen/extracted) bundle with normal schema validation enabled. That means generated configuration-management code is only rendered after Enroll has checked the bundle schema, referenced artifact existence, and artifact safety. If validation fails, manifest generation stops rather than trying to produce best-effort output from a malformed or tampered bundle. --- @@ -960,6 +973,16 @@ When JinjaTurtle is enabled and supports a harvested config file, the renderer c If JinjaTurtle is unavailable in `auto` mode, fails, emits missing variables, or does not support the path, Ansible falls back to copying the raw harvested file. +### 13.6 Keeping harvested data out of playbook structure + +Harvested values (file paths, owners, groups, usernames, GECOS fields, link targets, package names, unit names, ...) are attacker-influenceable on a host an unprivileged user partly controls. The renderer keeps them strictly in Ansible *data*, never in playbook *structure*, through three layers in `render_safety.py` and `yamlutil.py`: + +1. **Variable files are serialized, not templated.** `_write_role_defaults()` / `_write_hostvars()` dump mappings with a PyYAML `SafeDumper` (`yamlutil.yaml_dump_mapping`), so metacharacters and newlines in a harvested value fold into a quoted scalar and cannot introduce new keys or list items. +2. **Template-looking strings are tagged `!unsafe`.** `ansible_unsafe_data()` recursively wraps any harvested string containing a Jinja start (`{{`, `{%`, `{#`) as `AnsibleUnsafeText`, dumped as a YAML `!unsafe` scalar. Ansible then treats it as literal data instead of re-evaluating it at apply time. +3. **Raw task/handler YAML uses only allowlisted tokens.** The renderer hand-writes task/handler text with f-strings, but the *only* dynamic value ever spliced in is an already-sanitized role/var-prefix identifier, and every such splice goes through `scaffold_token()`, whose allowlist (`^[A-Za-z0-9_][A-Za-z0-9_ .-]*$`) excludes quotes, colons, braces, and newlines. A harvested free-text value can never reach scaffolding — it must travel as a variable referenced through `{{ ... }}` indirection. As a structural backstop, `_write_generated_task_yaml()` runs `assert_generated_yaml_safe()`, which parses every generated task/handler document and confirms it is still a list of string-keyed mappings. + +The harvest schema gate is what makes layer 3 sufficient in practice: `manifest` validates against the schema before rendering, and role names are constrained to `const` singletons or `^[A-Za-z0-9_]+$`, so the value handed to `scaffold_token()` is already within its allowlist. Layers 1 and 2 protect the unconstrained fields (owner, group, the post-slash portion of a path, link targets, user fields) that travel only as data. + --- ## 14. JinjaTurtle integration @@ -1033,7 +1056,12 @@ File: `diff.py` - plain `.tar.gz` / `.tgz` bundles, - SOPS-encrypted bundles when `sops_mode=True` or the name ends with `.sops`. -Bundle resolution is handled by `_bundle_from_input()`, which reuses `remote._safe_extract_tar()` for tarball extraction. +Bundle resolution is handled by `_bundle_from_input()`, which reuses `remote._safe_extract_tar()` for tarball/SOPS extraction. It takes a `freeze` flag: + +- `compare_harvests()` resolves both inputs with `freeze=True`, so a *directory* bundle is copied into a private `0700` tree (via `freeze_directory_bundle()`) before any artifact is hashed. This matters because diff re-opens artifacts by path to hash file contents; freezing removes the race where an unprivileged owner mutates the source directory after validation. Tar/SOPS inputs are already extracted into a private temp directory and so are inherently frozen. +- `validate` and `explain` resolve with `freeze=False` (the default). `validate` is a diagnostic that should report on the exact directory the operator named rather than a copy of it, and `explain` only ever reads `state.json` — it never opens artifact files — so there is nothing to race. + +Both diff inputs are then re-validated with `_validate_diff_bundle()` (artifact safety checks, `no_schema=True`) before any comparison reads them. ### 15.2 What diff compares @@ -1093,7 +1121,7 @@ This is intended to answer “what did Enroll collect and why?” 1. `state.json` exists, 2. it parses as JSON, 3. it validates against the vendored schema unless `--no-schema` is set, -4. every `managed_file.src_rel` is relative and points to a safe artifact file, +4. every `managed_file.src_rel` is rejected up front if it is absolute or contains a `..` component, then checked with `safe_artifact_file()` so it points to a safe, existing artifact (this absolute/`..` rejection runs even under `no_schema=True`, which is what keeps the `diff` path safe despite skipping schema validation), 5. firewall runtime generated artifacts exist and are safe, 6. the top-level `artifacts/` path is a real directory rather than a symlink or file, 7. the whole artifact tree contains no symlink directories, symlink files, hardlinks, special files, or paths that escape the artifact root, @@ -1101,9 +1129,9 @@ This is intended to answer “what did Enroll collect and why?” `validate_harvest()` is used in three important contexts: -- `enroll validate` exposes the checks directly to users. -- `manifest.manifest()` validates before rendering Ansible output. -- `diff.compare_harvests()` validates both input bundles before comparing them, using `no_schema=True` so older harvests can still be inspected while artifact safety checks remain active. +- `enroll validate` exposes the checks directly to users (on the directory the operator named; not frozen). +- `manifest.manifest()` validates the frozen/extracted bundle before rendering Ansible output (schema validation enabled). +- `diff.compare_harvests()` validates both input bundles before comparing them, using `no_schema=True` so older harvests can still be inspected while artifact safety checks remain active. Directory inputs are frozen first (see section 15.1), so validation and the subsequent content hashing both run against the private copy. It returns a `ValidationResult` with `errors`, `warnings`, `ok()`, `to_dict()`, and `to_text()`. @@ -1220,13 +1248,14 @@ Encryption/decryption helpers write via temp files and default to mode `0600`. `cli.py` supports optional INI config files. -Discovery order: +Discovery order (see `_discover_config_path()`): -1. `--no-config` disables config loading, +1. `--no-config` disables config loading entirely, 2. `--config PATH` or `-c PATH`, 3. `$ENROLL_CONFIG`, -6. `$XDG_CONFIG_HOME/enroll/enroll.ini`, -7. `~/.config/enroll/enroll.ini`. +4. `$XDG_CONFIG_HOME/enroll/enroll.ini`, falling back to `~/.config/enroll/enroll.ini` when `$XDG_CONFIG_HOME` is unset. + +Current-directory config files are deliberately not auto-loaded; pass `--config ./enroll.ini` if that behaviour is wanted. Config sections are translated into argv tokens by `_inject_config_argv()`: diff --git a/SECURITY.md b/SECURITY.md index 1e93b51..74faec7 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -28,10 +28,12 @@ In-scope security concerns include: * Supporting encrypted harvest bundles via `--sops`. * Avoiding symlink traversal and time-of-check/time-of-use mistakes when copying harvested files. * Refusing unsafe artifact paths, symlinks, hardlinks, device nodes, and tar path traversal in harvest bundles. +* Treating a harvest bundle as point-in-time validated: before `manifest` or `diff` re-open artifacts to render or hash them, a plain *directory* bundle is copied into a private, attacker-immutable temp tree (regular files only, no symlinks or hardlinks, opened without following links) so the bundle that is consumed cannot be raced and swapped after validation. Tar and SOPS inputs get the equivalent treatment by being extracted into a private temp directory. +* Keeping harvested values in Ansible *data* rather than playbook *structure*, so a harvested path, owner, group, username, or link target cannot inject YAML structure or be re-evaluated as a Jinja/template expression at apply time. * Writing plaintext harvest outputs into private directories by default. * Hardening root-run output path handling so Enroll does not accidentally write through attacker-prepared symlinks or unsafe parent directories. * Refusing to continue non-interactively when run as root with an unsafe `PATH`, unless the operator explicitly confirms with `--assume-safe-path`. -* Avoiding shell injection in generated manifests where harvested values are embedded into Ansible output. +* Avoiding injection in generated manifests where harvested values are embedded into Ansible output — not only shell injection, but YAML-structure injection and runtime Jinja/template re-evaluation — by serializing harvested data through a safe YAML dumper, tagging template-looking values as Ansible `!unsafe`, and allowlisting the only identifiers ever spliced into raw task YAML. * Rejecting unknown SSH host keys by default during remote harvests. These measures are defense-in-depth. They are intended to reduce the chance of accidental exposure, unsafe filesystem writes, path traversal, command injection, or dangerous behavior when Enroll is used normally by an administrator.