c16d6dcf75
Certbot renews at 30 days remaining, so a healthy certificate never falls below it and the old 14 day threshold could only fire after renewal had been broken for 16 straight days, leaving 14 days to react. 25 days fires about five days after the first failed renewal.
83 lines
2.5 KiB
Markdown
83 lines
2.5 KiB
Markdown
# Silver Server Ops
|
|
|
|
Server-wide maintenance scripts and operational notes for the Silver Cloud Hetzner host.
|
|
|
|
This repo is for host-level improvements that are not tied to one specific app or
|
|
service. SSL expiry alerts are the first check in the repo; more server-wide
|
|
maintenance checks can be added here over time.
|
|
|
|
## Implemented Checks
|
|
|
|
- SSL certificate expiry alerts
|
|
|
|
## SSL Certificate Expiry Alerts
|
|
|
|
`scripts/check-cert-expiry.sh` checks every Let's Encrypt certificate under
|
|
`/etc/letsencrypt/live` and sends an ntfy alert when any certificate expires in
|
|
less than 25 days.
|
|
|
|
The threshold sits just below certbot's own renewal window. Certbot renews at 30
|
|
days remaining, so a healthy certificate never drops below that. A 25 day
|
|
threshold fires roughly five days after renewal first fails and still leaves 25
|
|
days to fix it. A threshold below the renewal window means renewal has to stay
|
|
broken for weeks before the alert trips, which is too late to be useful.
|
|
|
|
### Install
|
|
|
|
```bash
|
|
sudo install -m 0755 scripts/check-cert-expiry.sh /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
### Root cron
|
|
|
|
```cron
|
|
15 8 * * * /opt/scripts/check-cert-expiry.sh >>/var/log/cert-expiry-check.log 2>&1
|
|
```
|
|
|
|
### Defaults
|
|
|
|
```bash
|
|
CERT_DIR=/etc/letsencrypt/live
|
|
EXPIRY_DAYS=25
|
|
NTFY_URL=http://127.0.0.1:2586/certbot
|
|
ALERT_ON_NO_CERTS=true
|
|
```
|
|
|
|
### Overrides
|
|
|
|
```bash
|
|
EXPIRY_DAYS=40 /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
If the ntfy topic is protected with an access token:
|
|
|
|
```bash
|
|
NTFY_TOKEN=your-token /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
### Manual test
|
|
|
|
```bash
|
|
sudo /opt/scripts/check-cert-expiry.sh
|
|
```
|
|
|
|
The script exits `0` when all certificates are healthy and `1` when it sends an
|
|
alert or cannot run the check correctly.
|
|
|
|
## Notification Notes
|
|
|
|
ntfy is a good default for this server because it is already self-hosted, simple
|
|
to call from shell scripts, and supports useful alert metadata such as title,
|
|
priority, and tags.
|
|
|
|
Alerts go to ntfy over `http://127.0.0.1:2586` rather than the public
|
|
`https://ntfy.silverwal.com` URL. The public URL is served by the same nginx
|
|
whose certificates this script watches, so an expired or broken certificate on
|
|
this host would fail curl's TLS verification and silently drop the very alert
|
|
that reports it. The loopback address removes DNS, nginx, and TLS from the
|
|
alerting path.
|
|
|
|
For jobs where silence is also a failure, pair ntfy with a dead man's switch such
|
|
as Healthchecks. ntfy tells you what the script found; Healthchecks tells you when
|
|
the script did not run at all.
|