3b797bb4e5
The check is silent by design on a healthy host, so an absent alert is indistinguishable from a cron that stopped running. Ping HEALTHCHECK_URL on a clean run and the /fail endpoint when the check alerts or cannot complete. Empty by default, which disables the pings.
123 lines
4.2 KiB
Markdown
123 lines
4.2 KiB
Markdown
# Silver Server Ops
|
|
|
|
Server-wide maintenance scripts and operational notes for the Silver Cloud Hetzner host.
|
|
|
|
This repo is for host-level improvements that are not tied to one specific app or
|
|
service. SSL expiry alerts are the first check in the repo; more server-wide
|
|
maintenance checks can be added here over time.
|
|
|
|
## Implemented Checks
|
|
|
|
- SSL certificate expiry alerts
|
|
|
|
## SSL Certificate Expiry Alerts
|
|
|
|
`scripts/check-cert-expiry.sh` checks every Let's Encrypt certificate under
|
|
`/etc/letsencrypt/live` and sends an ntfy alert when any certificate expires in
|
|
less than 25 days.
|
|
|
|
The threshold sits just below certbot's own renewal window. Certbot renews at 30
|
|
days remaining, so a healthy certificate never drops below that. A 25 day
|
|
threshold fires roughly five days after renewal first fails and still leaves 25
|
|
days to fix it. A threshold below the renewal window means renewal has to stay
|
|
broken for weeks before the alert trips, which is too late to be useful.
|
|
|
|
### Install
|
|
|
|
```bash
|
|
sudo install -m 0755 scripts/check-cert-expiry.sh /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
### Root cron
|
|
|
|
```cron
|
|
15 8 * * * /opt/scripts/check-cert-expiry.sh >>/var/log/cert-expiry-check.log 2>&1
|
|
```
|
|
|
|
### Defaults
|
|
|
|
```bash
|
|
CERT_DIR=/etc/letsencrypt/live
|
|
EXPIRY_DAYS=25
|
|
NTFY_URL=http://127.0.0.1:2586/certbot
|
|
ALERT_ON_NO_CERTS=true
|
|
CHECK_SERVED=true
|
|
SERVED_ADDR=127.0.0.1:443
|
|
SERVED_TIMEOUT=10
|
|
HEALTHCHECK_URL=
|
|
```
|
|
|
|
### Served certificate check
|
|
|
|
Certbot writes a renewed certificate to disk, but nginx keeps serving the old
|
|
one from memory until it is reloaded. A check that only reads
|
|
`/etc/letsencrypt/live` reports everything as healthy while browsers are being
|
|
handed an expired certificate.
|
|
|
|
With `CHECK_SERVED=true` the script also opens a TLS connection to
|
|
`SERVED_ADDR` using each lineage's name as the SNI hostname and compares the
|
|
served expiry against the file on disk. It warns only when the served
|
|
certificate expires earlier than the one on disk, which is the signature of a
|
|
renewal hook that stopped firing.
|
|
|
|
nginx completes a handshake with a fallback vhost certificate when SNI matches
|
|
no server block, so the comparison is skipped unless the served certificate's
|
|
SAN list actually covers that domain. A lineage that nginx no longer serves is
|
|
therefore skipped rather than compared against an unrelated certificate. Set
|
|
`CHECK_SERVED=false` to disable the check.
|
|
|
|
### Overrides
|
|
|
|
```bash
|
|
EXPIRY_DAYS=40 /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
If the ntfy topic is protected with an access token:
|
|
|
|
```bash
|
|
NTFY_TOKEN=your-token /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
### Manual test
|
|
|
|
```bash
|
|
sudo /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
The script exits `0` when all certificates are healthy and `1` when it sends an
|
|
alert or cannot run the check correctly.
|
|
|
|
## Notification Notes
|
|
|
|
ntfy is a good default for this server because it is already self-hosted, simple
|
|
to call from shell scripts, and supports useful alert metadata such as title,
|
|
priority, and tags.
|
|
|
|
Alerts go to ntfy over `http://127.0.0.1:2586` rather than the public
|
|
`https://ntfy.silverwal.com` URL. The public URL is served by the same nginx
|
|
whose certificates this script watches, so an expired or broken certificate on
|
|
this host would fail curl's TLS verification and silently drop the very alert
|
|
that reports it. The loopback address removes DNS, nginx, and TLS from the
|
|
alerting path.
|
|
|
|
For jobs where silence is also a failure, pair ntfy with a dead man's switch such
|
|
as Healthchecks. ntfy tells you what the script found; Healthchecks tells you when
|
|
the script did not run at all.
|
|
|
|
Set `HEALTHCHECK_URL` to enable it. The script pings that URL after a clean run
|
|
and `$HEALTHCHECK_URL/fail` when it alerts or cannot complete the check, so a
|
|
dead cron, a bad chmod or a host that never came back up stops looking like a
|
|
healthy fleet:
|
|
|
|
```cron
|
|
15 8 * * * HEALTHCHECK_URL=https://hc-ping.com/<uuid> /opt/scripts/check-cert-expiry.sh >>/var/log/cert-expiry-check.log 2>&1
|
|
```
|
|
|
|
Leaving `HEALTHCHECK_URL` empty disables the pings entirely.
|
|
|
|
This matters more than it looks for a certificate check specifically. The script
|
|
is designed to stay silent on a healthy host: certbot renews at 30 days and the
|
|
threshold is 25, so a correctly working fleet produces no notifications, ever.
|
|
Without a heartbeat, "no alert" and "the check has not run since March" are the
|
|
same observation.
|