Add optional healthcheck heartbeat

The check is silent by design on a healthy host, so an absent alert is
indistinguishable from a cron that stopped running. Ping HEALTHCHECK_URL
on a clean run and the /fail endpoint when the check alerts or cannot
complete. Empty by default, which disables the pings.

.gitattributes

@@ -0,0 +1,2 @@
+* text=auto eol=lf
+*.sh text eol=lf

README.md

@@ -12,5 +12,111 @@ maintenance checks can be added here over time.
 
 ## SSL Certificate Expiry Alerts
 
-`scripts/check-cert-expiry.sh` checks Let's Encrypt certificates and alerts when
-any certificate expires in less than 14 days.
+`scripts/check-cert-expiry.sh` checks every Let's Encrypt certificate under
+`/etc/letsencrypt/live` and sends an ntfy alert when any certificate expires in
+less than 25 days.
+
+The threshold sits just below certbot's own renewal window. Certbot renews at 30
+days remaining, so a healthy certificate never drops below that. A 25 day
+threshold fires roughly five days after renewal first fails and still leaves 25
+days to fix it. A threshold below the renewal window means renewal has to stay
+broken for weeks before the alert trips, which is too late to be useful.
+
+### Install
+
+```bash
+sudo install -m 0755 scripts/check-cert-expiry.sh /opt/scripts/check-cert-expiry.sh
+```
+
+### Root cron
+
+```cron
+15 8 * * * /opt/scripts/check-cert-expiry.sh >>/var/log/cert-expiry-check.log 2>&1
+```
+
+### Defaults
+
+```bash
+CERT_DIR=/etc/letsencrypt/live
+EXPIRY_DAYS=25
+NTFY_URL=http://127.0.0.1:2586/certbot
+ALERT_ON_NO_CERTS=true
+CHECK_SERVED=true
+SERVED_ADDR=127.0.0.1:443
+SERVED_TIMEOUT=10
+HEALTHCHECK_URL=
+```
+
+### Served certificate check
+
+Certbot writes a renewed certificate to disk, but nginx keeps serving the old
+one from memory until it is reloaded. A check that only reads
+`/etc/letsencrypt/live` reports everything as healthy while browsers are being
+handed an expired certificate.
+
+With `CHECK_SERVED=true` the script also opens a TLS connection to
+`SERVED_ADDR` using each lineage's name as the SNI hostname and compares the
+served expiry against the file on disk. It warns only when the served
+certificate expires earlier than the one on disk, which is the signature of a
+renewal hook that stopped firing.
+
+nginx completes a handshake with a fallback vhost certificate when SNI matches
+no server block, so the comparison is skipped unless the served certificate's
+SAN list actually covers that domain. A lineage that nginx no longer serves is
+therefore skipped rather than compared against an unrelated certificate. Set
+`CHECK_SERVED=false` to disable the check.
+
+### Overrides
+
+```bash
+EXPIRY_DAYS=40 /opt/scripts/check-cert-expiry.sh
+```
+
+If the ntfy topic is protected with an access token:
+
+```bash
+NTFY_TOKEN=your-token /opt/scripts/check-cert-expiry.sh
+```
+
+### Manual test
+
+```bash
+sudo /opt/scripts/check-cert-expiry.sh
+```
+
+The script exits `0` when all certificates are healthy and `1` when it sends an
+alert or cannot run the check correctly.
+
+## Notification Notes
+
+ntfy is a good default for this server because it is already self-hosted, simple
+to call from shell scripts, and supports useful alert metadata such as title,
+priority, and tags.
+
+Alerts go to ntfy over `http://127.0.0.1:2586` rather than the public
+`https://ntfy.silverwal.com` URL. The public URL is served by the same nginx
+whose certificates this script watches, so an expired or broken certificate on
+this host would fail curl's TLS verification and silently drop the very alert
+that reports it. The loopback address removes DNS, nginx, and TLS from the
+alerting path.
+
+For jobs where silence is also a failure, pair ntfy with a dead man's switch such
+as Healthchecks. ntfy tells you what the script found; Healthchecks tells you when
+the script did not run at all.
+
+Set `HEALTHCHECK_URL` to enable it. The script pings that URL after a clean run
+and `$HEALTHCHECK_URL/fail` when it alerts or cannot complete the check, so a
+dead cron, a bad chmod or a host that never came back up stops looking like a
+healthy fleet:
+
+```cron
+15 8 * * * HEALTHCHECK_URL=https://hc-ping.com/<uuid> /opt/scripts/check-cert-expiry.sh >>/var/log/cert-expiry-check.log 2>&1
+```
+
+Leaving `HEALTHCHECK_URL` empty disables the pings entirely.
+
+This matters more than it looks for a certificate check specifically. The script
+is designed to stay silent on a healthy host: certbot renews at 30 days and the
+threshold is 25, so a correctly working fleet produces no notifications, ever.
+Without a heartbeat, "no alert" and "the check has not run since March" are the
+same observation.

scripts/check-cert-expiry.sh

@@ -3,13 +3,17 @@
 set -uo pipefail
 
 CERT_DIR="${CERT_DIR:-/etc/letsencrypt/live}"
-EXPIRY_DAYS="${EXPIRY_DAYS:-14}"
-NTFY_URL="${NTFY_URL:-https://ntfy.silverwal.com/certbot}"
+EXPIRY_DAYS="${EXPIRY_DAYS:-25}"
+NTFY_URL="${NTFY_URL:-http://127.0.0.1:2586/certbot}"
 NTFY_TITLE="${NTFY_TITLE:-SSL certificate warning}"
 NTFY_PRIORITY="${NTFY_PRIORITY:-high}"
 NTFY_TAGS="${NTFY_TAGS:-warning,lock}"
 NTFY_TOKEN="${NTFY_TOKEN:-}"
 ALERT_ON_NO_CERTS="${ALERT_ON_NO_CERTS:-true}"
+CHECK_SERVED="${CHECK_SERVED:-true}"
+SERVED_ADDR="${SERVED_ADDR:-127.0.0.1:443}"
+SERVED_TIMEOUT="${SERVED_TIMEOUT:-10}"
+HEALTHCHECK_URL="${HEALTHCHECK_URL:-}"
 
 host="$(hostname -f 2>/dev/null || hostname 2>/dev/null || echo "unknown-host")"
 now_epoch="$(date +%s)"
@@ -31,9 +35,47 @@ send_alert() {
   curl "${curl_args[@]}" --data-binary "${message}" "${NTFY_URL}" >/dev/null
 }
 
+# Expiry of the certificate nginx actually serves for a domain, via SNI on the
+# loopback listener. stdin is fed from echo so that s_client closes the
+# connection and, more importantly, does not consume the cert list the main
+# loop is reading from its own stdin.
+#
+# nginx completes the TLS handshake with a fallback vhost certificate when SNI
+# matches no server block, so the served certificate is only trustworthy for
+# this comparison when its SAN list actually covers the domain. Anything else
+# is reported as "not served" rather than compared against the wrong lineage.
+served_expiry_epoch() {
+  local domain="$1" pem sans enddate
+
+  pem="$(echo | timeout "${SERVED_TIMEOUT}" openssl s_client \
+    -connect "${SERVED_ADDR}" -servername "${domain}" 2>/dev/null \
+    | openssl x509 2>/dev/null)"
+  [[ -z "${pem}" ]] && return 1
+
+  sans="$(printf '%s\n' "${pem}" | openssl x509 -noout -ext subjectAltName 2>/dev/null \
+    | tr ',' '\n' | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')"
+  grep -Fxq "DNS:${domain}" <<<"${sans}" || return 1
+
+  enddate="$(printf '%s\n' "${pem}" | openssl x509 -noout -enddate 2>/dev/null)"
+  date -d "${enddate#notAfter=}" +%s 2>/dev/null
+}
+
+# Dead man's switch. ntfy reports what the check found; this reports that the
+# check ran at all. Without it, a dead cron, a failed boot or a bad chmod all
+# look exactly like a healthy fleet.
+ping_healthcheck() {
+  local suffix="${1:-}"
+
+  [[ -z "${HEALTHCHECK_URL}" ]] && return 0
+
+  curl -fsS -m 10 -o /dev/null "${HEALTHCHECK_URL}${suffix}" \
+    || echo "Failed to ping healthcheck at ${HEALTHCHECK_URL}${suffix}" >&2
+}
+
 fail() {
   local message="$1"
   send_alert "${message}" || echo "Failed to send ntfy alert to ${NTFY_URL}" >&2
+  ping_healthcheck "/fail"
   echo "${message}" >&2
   exit 1
 }
@@ -74,6 +116,18 @@ while IFS= read -r -d '' cert_path; do
   seconds_left=$((expiry_epoch - now_epoch))
   days_left=$((seconds_left / 86400))
 
+  # certbot writes the new file, but nginx keeps serving the old certificate
+  # from memory until it is reloaded. Comparing disk against what is served is
+  # the only way to catch a renewal hook that stopped firing.
+  if [[ "${CHECK_SERVED}" == "true" ]]; then
+    if served_epoch="$(served_expiry_epoch "${cert_name}")" && [[ -n "${served_epoch}" ]]; then
+      if (( served_epoch < expiry_epoch )); then
+        served_days=$(( (served_epoch - now_epoch) / 86400 ))
+        warnings+=("${cert_name}: renewed on disk but nginx is still serving the previous certificate (served copy expires in ${served_days} day(s)). Reload nginx.")
+      fi
+    fi
+  fi
+
   if (( seconds_left < 0 )); then
     warnings+=("${cert_name}: EXPIRED on ${not_after}")
   elif ! openssl x509 -checkend "${threshold_seconds}" -noout -in "${cert_path}" >/dev/null 2>&1; then
@@ -87,6 +141,7 @@ if (( cert_count == 0 )); then
   fi
 
   echo "No certificates found under ${CERT_DIR}; no alert sent."
+  ping_healthcheck
   exit 0
 fi
 
@@ -96,8 +151,10 @@ if (( ${#warnings[@]} > 0 )); then
 $(printf '%s\n' "${warnings[@]}")"
 
   send_alert "${message}" || echo "Failed to send ntfy alert to ${NTFY_URL}" >&2
+  ping_healthcheck "/fail"
   echo "${message}" >&2
   exit 1
 fi
 
 echo "All ${cert_count} certificate(s) are valid for more than ${EXPIRY_DAYS} day(s)."
+ping_healthcheck