Table of Contents

What does diff do?
How do I run it?
Output formats
--exit-code mode
Webhooks
Email
Example: Running the diff on a systemd timer

Create script
Environment file
Systemd service
Systemd timer
Enable + test

Full usage info
Example Node-RED webhook receiver flow

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

The Enroll diff subcommand will compare two 'harvests' and see if there are any differences.

What does diff do?

These differences include:

Packages added/removed
Files added/removed/changed
Users added/removed/changed
Services added/removed/changed

The diff command can also understand SOPS-encrypted harvests, assuming the user executing the command is able to decrypt the SOPS files.

How do I run it?

Assuming you have two harvests on your filesystem:

enroll diff \
  --old harvest.v1 \
  --new harvest.v2

Output formats

The default output is just a plain text format, which is the same as passing --format text.

You can pass --format json or --format markdown to get structured formats instead.

--exit-code mode

You can pass --exit-code as an argument, which will return a return code of 2 if there was any 'diff'.

You can use this to script any further action your cron service might take, such as sending e-mail output, or rely on it to trigger failures in CI builds.

Webhooks

You can also send the output to a webhook under your control. Here is an example:

enroll diff \
  --old harvest.v1 \
  --new harvest.v2 \
  --webhook https://example.com/some/endpoint \
  --webhook-format json \
  --webhook-header 'X-Enroll-Secret: Some-Secret-Here'

You can pass --webhook-format markdown if you wanted the Markdown format in the payload, but most people probably want JSON structured data for consumption at the other end.

Use the --webhook-header to send some sort of authentication/authorization attribute (e.g an OAuth2.0 bearer token, static secret, etc) to help lock down your webhook from unauthorised requests. You can send multiple headers.

Email

You can send the output via email using email and smtp arguments. See 'Full usage info' below for more information.

Example: Running the diff on a systemd timer

In the example below, we set up a systemd timer to run on a periodic basis and trigger the webhook.

It will set up a 'golden' harvest if it didn't already exist, then periodically run a second harvest to compare against it.

You can optionally promote the new harvest to become the new 'golden' harvest for future invocations if you want a continuous, rolling 'diff' scenario.

Create script

Store in /usr/local/bin/enroll-harvest-diff.sh and make it chmod 0700

#!/usr/bin/env bash
set -euo pipefail

# Required env
: "${WEBHOOK_URL:?Set WEBHOOK_URL in /etc/enroll/enroll-harvest-diff}"
: "${ENROLL_SECRET:?Set ENROLL_SECRET in /etc/enroll/enroll-harvest-diff}"

# Optional env
STATE_DIR="${ENROLL_STATE_DIR:-/var/lib/enroll}"
GOLDEN_DIR="${STATE_DIR}/golden"
PROMOTE_NEW="${PROMOTE_NEW:-1}"          # 1=promote new->golden; 0=keep golden fixed
KEEP_BACKUPS="${KEEP_BACKUPS:-7}"        # only used if PROMOTE_NEW=1
LOCKFILE="${STATE_DIR}/.enroll-harvest-diff.lock"

mkdir -p "${STATE_DIR}"
chmod 700 "${STATE_DIR}" || true

# single-instance lock (avoid overlapping timer runs)
exec 9>"${LOCKFILE}"
flock -n 9 || exit 0

tmp_new=""
cleanup() {
  if [[ -n "${tmp_new}" && -d "${tmp_new}" ]]; then
    rm -rf "${tmp_new}"
  fi
}
trap cleanup EXIT

make_tmp_dir() {
  mktemp -d "${STATE_DIR}/.harvest.XXXXXX"
}

run_harvest() {
  local out_dir="$1"
  rm -rf "${out_dir}"
  mkdir -p "${out_dir}"
  chmod 700 "${out_dir}" || true
  enroll harvest --out "${out_dir}" >/dev/null
}

# A) create golden if missing
if [[ ! -f "${GOLDEN_DIR}/state.json" ]]; then
  tmp="$(make_tmp_dir)"
  run_harvest "${tmp}"
  rm -rf "${GOLDEN_DIR}"
  mv "${tmp}" "${GOLDEN_DIR}"
  echo "Golden harvest created at ${GOLDEN_DIR}"
  exit 0
fi

# B) create new harvest
tmp_new="$(make_tmp_dir)"
run_harvest "${tmp_new}"

# C) diff + webhook notify
enroll diff \
  --old "${GOLDEN_DIR}" \
  --new "${tmp_new}" \
  --webhook "${WEBHOOK_URL}" \
  --webhook-format json \
  --webhook-header "X-Enroll-Secret: ${ENROLL_SECRET}"

# Promote or discard new harvest
if [[ "${PROMOTE_NEW}" == "1" || "${PROMOTE_NEW,,}" == "true" || "${PROMOTE_NEW}" == "yes" ]]; then
  ts="$(date -u +%Y%m%d-%H%M%S)"
  backup="${STATE_DIR}/golden.prev.${ts}"
  mv "${GOLDEN_DIR}" "${backup}"
  mv "${tmp_new}" "${GOLDEN_DIR}"
  tmp_new=""  # don't delete it in trap

  # Keep only latest N backups
  if [[ "${KEEP_BACKUPS}" =~ ^[0-9]+$ ]] && (( KEEP_BACKUPS > 0 )); then
    ls -1dt "${STATE_DIR}"/golden.prev.* 2>/dev/null | tail -n +"$((KEEP_BACKUPS+1))" | xargs -r rm -rf
  fi

  echo "Diff complete; baseline updated."
else
  # tmp_new will be deleted by trap
  echo "Diff complete; baseline unchanged (PROMOTE_NEW=${PROMOTE_NEW})."
fi

Environment file

Save as: /etc/enroll/enroll-harvest-diff and chmod 0600 it.

# Where to store golden + temp harvests
ENROLL_STATE_DIR=/var/lib/enroll

# 1 = each run becomes new baseline ("since last harvest")
# 0 = compare against a fixed baseline ("since golden")
PROMOTE_NEW=1

# If PROMOTE_NEW=1, keep this many old baselines
KEEP_BACKUPS=7

WEBHOOK_URL=https://example.com/webhook/xxxxxxxx
ENROLL_SECRET=xxxxxxxxxxxxxxxxxxxx

Systemd service

/etc/systemd/system/enroll-harvest-diff.service:

[Unit]
Description=Enroll harvest + diff + webhook notify
Wants=network-online.target
After=network-online.target
ConditionPathExists=/etc/enroll/enroll-harvest-diff

[Service]
Type=oneshot
EnvironmentFile=/etc/enroll/enroll-harvest-diff
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
UMask=0077

# Create /var/lib/enroll automatically
StateDirectory=enroll

ExecStart=/usr/local/bin/enroll-harvest-diff.sh

Systemd timer

/etc/systemd/system/enroll-harvest-diff.timer

[Unit]
Description=Run Enroll harvest diff hourly

[Timer]
OnCalendar=hourly
RandomizedDelaySec=10m
Persistent=true

[Install]
WantedBy=timers.target

Enable + test

sudo systemctl daemon-reload
sudo systemctl enable --now enroll-harvest-diff.timer

# run once now
sudo systemctl start enroll-harvest-diff.service
sudo journalctl -u enroll-harvest-diff.service -n 200 --no-pager

If you want the “fixed baseline” behavior (always diff against the original golden snapshot), set PROMOTE_NEW=0 and you're done.

Full usage info

usage: enroll diff [-h] --old OLD --new NEW [--sops] [--format {text,markdown,json}] [--out OUT] [--exit-code] [--notify-always] [--webhook WEBHOOK]
                   [--webhook-format {json,text,markdown}] [--webhook-header K:V] [--email-to EMAIL_TO] [--email-from EMAIL_FROM] [--email-subject EMAIL_SUBJECT] [--smtp SMTP]
                   [--smtp-user SMTP_USER] [--smtp-password-env SMTP_PASSWORD_ENV]

options:
  -h, --help            show this help message and exit
  --old OLD             Old/baseline harvest (directory, a path to state.json, a tarball, or a SOPS-encrypted bundle).
  --new NEW             New/current harvest (directory, a path to state.json, a tarball, or a SOPS-encrypted bundle).
  --sops                Allow SOPS-encrypted harvest bundle inputs (requires `sops` on PATH).
  --format {text,markdown,json}
                        Report output format (default: text).
  --out OUT             Write the report to this file instead of stdout.
  --exit-code           Exit with status 2 if differences are detected.
  --notify-always       Send webhook/email even when there are no differences.
  --webhook WEBHOOK     POST the report to this URL (only when differences are detected, unless --notify-always).
  --webhook-format {json,text,markdown}
                        Payload format for --webhook (default: json).
  --webhook-header K:V  Extra HTTP header for --webhook (repeatable), e.g. 'Authorization: Bearer ...'.
  --email-to EMAIL_TO   Email the report to this address (repeatable; only when differences are detected unless --notify-always).
  --email-from EMAIL_FROM
                        From address for --email-to (default: enroll@<hostname>).
  --email-subject EMAIL_SUBJECT
                        Subject for --email-to (default: 'enroll diff report').
  --smtp SMTP           SMTP server host[:port] for --email-to. If omitted, uses local sendmail.
  --smtp-user SMTP_USER
                        SMTP username (optional).
  --smtp-password-env SMTP_PASSWORD_ENV
                        Environment variable containing SMTP password (optional).

Example Node-RED webhook receiver flow

Here is an example Node-RED flow that could act as your webhook receiver. It has been preconfigured to parse the Enroll Diff payload.

You would just need to add a downstream node to send the notification on to a service of your choosing (e.g Slack, Signal, Google Chat, a ticket system/API, etc)

[
    {
        "id": "d84e1772e5d4cb01",
        "type": "tab",
        "label": "Enroll Diff Webhook",
        "disabled": false,
        "info": "",
        "env": []
    },
    {
        "id": "90a4098e08d188cc",
        "type": "group",
        "z": "d84e1772e5d4cb01",
        "style": {
            "stroke": "#999999",
            "stroke-opacity": "1",
            "fill": "none",
            "fill-opacity": "1",
            "label": true,
            "label-position": "nw",
            "color": "#a4a4a4"
        },
        "nodes": [
            "ce827f8bd5838225",
            "04768c32a4657ddf",
            "6bb5aa95e1791737",
            "fbfd6dacc8a6339d"
        ],
        "x": 214,
        "y": 199,
        "w": 932,
        "h": 182
    },
    {
        "id": "ce827f8bd5838225",
        "type": "http in",
        "z": "d84e1772e5d4cb01",
        "g": "90a4098e08d188cc",
        "name": "Webhook HTTP in",
        "url": "/some/random/webhook/path",
        "method": "post",
        "upload": false,
        "skipBodyParsing": false,
        "swaggerDoc": "",
        "x": 330,
        "y": 300,
        "wires": [
            [
                "04768c32a4657ddf"
            ]
        ]
    },
    {
        "id": "04768c32a4657ddf",
        "type": "function",
        "z": "d84e1772e5d4cb01",
        "g": "90a4098e08d188cc",
        "name": "Parse",
        "func": "const r = msg.payload || {};\nconst MAX_CHARS = 2000;\nconst MAX_ITEMS = 25;\n\nfunction arr(x) { return Array.isArray(x) ? x : []; }\nfunction safe(s) { return (s === null || s === undefined) ? \"\" : String(s); }\n\nfunction formatList(title, items, prefix = \"• \") {\n    items = arr(items).map(safe).filter(Boolean);\n    if (items.length === 0) return \"\";\n\n    const shown = items.slice(0, MAX_ITEMS);\n    const more = items.length - shown.length;\n\n    let line = `${title}: ${shown.join(\", \")}`;\n    if (more > 0) line += ` … (+${more} more)`;\n    return line;\n}\n\nfunction formatChangedUsers(changed) {\n    changed = arr(changed);\n    if (changed.length === 0) return \"\";\n\n    const shown = changed.slice(0, MAX_ITEMS).map(u => {\n        const name = safe(u.name);\n        const ch = u.changes || {};\n        const keys = Object.keys(ch);\n\n        // make a short per-user change summary\n        const parts = keys.map(k => {\n            const v = ch[k];\n            if (v && typeof v === \"object\" && (\"old\" in v || \"new\" in v)) return k;\n            if (v && typeof v === \"object\" && (\"added\" in v || \"removed\" in v)) return k;\n            return k;\n        });\n\n        return parts.length ? `${name} (${parts.join(\", \")})` : name;\n    });\n\n    const more = changed.length - shown.length;\n    return `Users changed: ${shown.join(\"; \")}${more > 0 ? ` … (+${more} more)` : \"\"}`;\n}\n\nfunction formatChangedServices(changed) {\n    changed = arr(changed);\n    if (changed.length === 0) return \"\";\n\n    const shown = changed.slice(0, MAX_ITEMS).map(svc => {\n        const unit = safe(svc.unit);\n        const changes = svc.changes || {};\n        const keys = Object.keys(changes);\n        return keys.length ? `${unit} (${keys.join(\", \")})` : unit;\n    });\n\n    const more = changed.length - shown.length;\n    return `Services changed: ${shown.join(\"; \")}${more > 0 ? ` … (+${more} more)` : \"\"}`;\n}\n\nfunction formatPkgVersionChanged(changed) {\n    changed = arr(changed);\n    if (changed.length === 0) return \"\";\n\n    const shown = changed.slice(0, MAX_ITEMS).map(x => {\n        const pkg = safe(x.package);\n        const oldv = safe(x.old);\n        const newv = safe(x.new);\n        if (!pkg) return \"\";\n        if (oldv || newv) return `${pkg} (${oldv} → ${newv})`;\n        return pkg;\n    }).filter(Boolean);\n\n    const more = changed.length - shown.length;\n    return `Packages upgraded/downgraded: ${shown.join(\"; \")}${more > 0 ? ` … (+${more} more)` : \"\"}`;\n}\n\nfunction metaInline(f) {\n    const role = safe(f.role);\n    const reason = safe(f.reason);\n    const parts = [];\n    if (reason) parts.push(reason);\n    if (role) parts.push(role);\n    return parts.length ? ` (${parts.join(\", \")})` : \"\";\n}\n\nfunction changeInline(f) {\n    const ch = f.changes || {};\n    const parts = [];\n\n    // show role/reason changes explicitly when present\n    if (ch.role && ((\"old\" in ch.role) || (\"new\" in ch.role))) {\n        parts.push(`role: ${safe(ch.role.old)} → ${safe(ch.role.new)}`);\n    }\n    if (ch.reason && ((\"old\" in ch.reason) || (\"new\" in ch.reason))) {\n        parts.push(`reason: ${safe(ch.reason.old)} → ${safe(ch.reason.new)}`);\n    }\n\n    // keep your existing keys list, but don’t duplicate role/reason\n    const keys = Object.keys(ch).filter(k => k !== \"role\" && k !== \"reason\");\n    if (keys.length) parts.push(keys.map(k => (k === \"content\" ? \"content\" : k)).join(\", \"));\n\n    return parts.length ? ` (${parts.join(\"; \")})` : \"\";\n}\n\nfunction formatFiles(list, label) {\n    list = arr(list);\n    if (list.length === 0) return \"\";\n\n    const shown = list.slice(0, MAX_ITEMS).map(f => {\n        const path = safe(f.path);\n        return path ? `${path}${metaInline(f)}` : \"\";\n    }).filter(Boolean);\n\n    const more = list.length - shown.length;\n    let line = `${label}: ${shown.join(\", \")}`;\n    if (more > 0) line += ` … (+${more} more)`;\n    return line;\n}\n\nfunction formatFilesChanged(changed) {\n    changed = arr(changed);\n    if (changed.length === 0) return \"\";\n\n    const shown = changed.slice(0, MAX_ITEMS).map(f => {\n        const path = safe(f.path);\n        return path ? `${path}${changeInline(f)}` : \"\";\n    }).filter(Boolean);\n\n    const more = changed.length - shown.length;\n    return `Files changed: ${shown.join(\"; \")}${more > 0 ? ` … (+${more} more)` : \"\"}`;\n}\n\nfunction trimToMax(text) {\n    if (text.length <= MAX_CHARS) return text;\n    return text.slice(0, MAX_CHARS - 20).trimEnd() + \"\\n…(truncated)\";\n}\n\n// Header bits (host/time if present)\nconst host = r.new?.host || r.old?.host || \"\";\nconst ts = r.generated_at || \"\";\nconst header = `enroll diff${host ? \" @ \" + host : \"\"}${ts ? \" (\" + ts + \")\" : \"\"}`;\n\n// Build message lines\nconst lines = [header];\n\n// Packages\nlines.push(formatList(\"Packages added\", r.packages?.added));\nlines.push(formatList(\"Packages removed\", r.packages?.removed));\nlines.push(formatPkgVersionChanged(r.packages?.version_changed));\n\n// Services\nlines.push(formatList(\"Services enabled +\", r.services?.enabled_added));\nlines.push(formatList(\"Services enabled -\", r.services?.enabled_removed));\nlines.push(formatChangedServices(r.services?.changed));\n\n// Users\nlines.push(formatList(\"Users added\", r.users?.added));\nlines.push(formatList(\"Users removed\", r.users?.removed));\nlines.push(formatChangedUsers(r.users?.changed));\n\n// Files\nlines.push(formatFiles(r.files?.added, \"Files added\"));\nlines.push(formatFiles(r.files?.removed, \"Files removed\"));\nlines.push(formatFilesChanged(r.files?.changed));\n\n// Clean + join\nconst msgText = lines.filter(s => safe(s).trim().length > 0).join(\"\\n\");\n\n// Set payload for downstream node\nmsg.payload = trimToMax(msgText);\nreturn msg;",
        "outputs": 1,
        "timeout": 0,
        "noerr": 0,
        "initialize": "",
        "finalize": "",
        "libs": [],
        "x": 630,
        "y": 300,
        "wires": [
            [
                "6bb5aa95e1791737",
                "fbfd6dacc8a6339d"
            ]
        ]
    },
    {
        "id": "6bb5aa95e1791737",
        "type": "http response",
        "z": "d84e1772e5d4cb01",
        "g": "90a4098e08d188cc",
        "name": "Response",
        "statusCode": "201",
        "headers": {},
        "x": 820,
        "y": 240,
        "wires": []
    },
    {
        "id": "5778bcd1e4ff8e60",
        "type": "comment",
        "z": "d84e1772e5d4cb01",
        "name": "Remember to set X-Enroll-Secret as an  environment variable in your flow, with a secret value that matches what enroll sends with --webhook-header !",
        "info": "",
        "x": 710,
        "y": 140,
        "wires": []
    },
    {
        "id": "fbfd6dacc8a6339d",
        "type": "debug",
        "z": "d84e1772e5d4cb01",
        "g": "90a4098e08d188cc",
        "name": "Debug or send onward to Slack, Signal, etc",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "payload",
        "targetType": "msg",
        "statusVal": "",
        "statusType": "auto",
        "x": 930,
        "y": 340,
        "wires": []
    }
]