3b797bb4e5
The check is silent by design on a healthy host, so an absent alert is indistinguishable from a cron that stopped running. Ping HEALTHCHECK_URL on a clean run and the /fail endpoint when the check alerts or cannot complete. Empty by default, which disables the pings.
4.2 KiB
Silver Server Ops
Server-wide maintenance scripts and operational notes for the Silver Cloud Hetzner host.
This repo is for host-level improvements that are not tied to one specific app or service. SSL expiry alerts are the first check in the repo; more server-wide maintenance checks can be added here over time.
Implemented Checks
- SSL certificate expiry alerts
SSL Certificate Expiry Alerts
scripts/check-cert-expiry.sh checks every Let's Encrypt certificate under
/etc/letsencrypt/live and sends an ntfy alert when any certificate expires in
less than 25 days.
The threshold sits just below certbot's own renewal window. Certbot renews at 30 days remaining, so a healthy certificate never drops below that. A 25 day threshold fires roughly five days after renewal first fails and still leaves 25 days to fix it. A threshold below the renewal window means renewal has to stay broken for weeks before the alert trips, which is too late to be useful.
Install
sudo install -m 0755 scripts/check-cert-expiry.sh /opt/scripts/check-cert-expiry.sh
Root cron
15 8 * * * /opt/scripts/check-cert-expiry.sh >>/var/log/cert-expiry-check.log 2>&1
Defaults
CERT_DIR=/etc/letsencrypt/live
EXPIRY_DAYS=25
NTFY_URL=http://127.0.0.1:2586/certbot
ALERT_ON_NO_CERTS=true
CHECK_SERVED=true
SERVED_ADDR=127.0.0.1:443
SERVED_TIMEOUT=10
HEALTHCHECK_URL=
Served certificate check
Certbot writes a renewed certificate to disk, but nginx keeps serving the old
one from memory until it is reloaded. A check that only reads
/etc/letsencrypt/live reports everything as healthy while browsers are being
handed an expired certificate.
With CHECK_SERVED=true the script also opens a TLS connection to
SERVED_ADDR using each lineage's name as the SNI hostname and compares the
served expiry against the file on disk. It warns only when the served
certificate expires earlier than the one on disk, which is the signature of a
renewal hook that stopped firing.
nginx completes a handshake with a fallback vhost certificate when SNI matches
no server block, so the comparison is skipped unless the served certificate's
SAN list actually covers that domain. A lineage that nginx no longer serves is
therefore skipped rather than compared against an unrelated certificate. Set
CHECK_SERVED=false to disable the check.
Overrides
EXPIRY_DAYS=40 /opt/scripts/check-cert-expiry.sh
If the ntfy topic is protected with an access token:
NTFY_TOKEN=your-token /opt/scripts/check-cert-expiry.sh
Manual test
sudo /opt/scripts/check-cert-expiry.sh
The script exits 0 when all certificates are healthy and 1 when it sends an
alert or cannot run the check correctly.
Notification Notes
ntfy is a good default for this server because it is already self-hosted, simple to call from shell scripts, and supports useful alert metadata such as title, priority, and tags.
Alerts go to ntfy over http://127.0.0.1:2586 rather than the public
https://ntfy.silverwal.com URL. The public URL is served by the same nginx
whose certificates this script watches, so an expired or broken certificate on
this host would fail curl's TLS verification and silently drop the very alert
that reports it. The loopback address removes DNS, nginx, and TLS from the
alerting path.
For jobs where silence is also a failure, pair ntfy with a dead man's switch such as Healthchecks. ntfy tells you what the script found; Healthchecks tells you when the script did not run at all.
Set HEALTHCHECK_URL to enable it. The script pings that URL after a clean run
and $HEALTHCHECK_URL/fail when it alerts or cannot complete the check, so a
dead cron, a bad chmod or a host that never came back up stops looking like a
healthy fleet:
15 8 * * * HEALTHCHECK_URL=https://hc-ping.com/<uuid> /opt/scripts/check-cert-expiry.sh >>/var/log/cert-expiry-check.log 2>&1
Leaving HEALTHCHECK_URL empty disables the pings entirely.
This matters more than it looks for a certificate check specifically. The script is designed to stay silent on a healthy host: certbot renews at 30 days and the threshold is 25, so a correctly working fleet produces no notifications, ever. Without a heartbeat, "no alert" and "the check has not run since March" are the same observation.