Linux Server Automation with Native Tools: cron, systemd, SSH

Every fleet older than three years has a cron job graveyard: scripts that have run nightly since 2019, written by an engineer who left in 2021, cleaning up a directory that moved in 2023. Nobody dares delete them. That graveyard is what server automation turns into when it's built without structure, and it's avoidable.

Part one catalogued the manual-SSH toil that eats ops time (log hunts, disk cleanups, service restarts, cert and package checks) and the safety rules any server automation agent has to respect: key-based auth, restricted keys, trusted hosts, audit trails. This part is the DIY build. Everything below uses tools already installed on every mainstream distro: systemd timers, cron, OpenSSH, bash, logrotate and unattended-upgrades. An afternoon covers the first host, and a second one gets you to fleet-wide rollout.

The walkthrough goes tool by tool, starting with the scheduler and ending with where this approach stops scaling.

01Schedule

systemd timers

Recurring checks with journal output, catch-up runs and jitter.

02Transport

Hardened SSH

Key-only auth, restricted keys, pinned host keys, a bastion.

03Fan-out

Fleet loops

One read-only script across every host, in parallel, logged.

04Prevent

logrotate, upgrades

Retire the disk pages and patch audits at the source.

Everything here ships with a mainstream distro. Nothing to buy, nothing new to install.

Tool 1: cron and systemd timers, the scheduler

Recurring checks are the easiest wins: disk usage, service health, cert expiry. Classic cron still works for a single check:

# crontab -e
# cron mails any output to the crontab owner: print filesystems above 85%
*/30 * * * * df -P | awk 'NR>1 && $5+0 > 85'

For anything you care about, prefer systemd timers. Output lands in the journal, Persistent=true catches runs missed while a host was down, and RandomizedDelaySec stops 200 hosts from hammering your package mirror in the same second (systemd.timer docs).

The check script itself, written to fail loudly:

#!/usr/bin/env bash
# /usr/local/sbin/disk-check.sh: exit non-zero if any filesystem is at or above 85%
set -euo pipefail
over=$(df -P -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 >= 85 {print $6" "$5}')
if [ -n "$over" ]; then
  echo "Filesystems over threshold: $over"
  exit 1
fi

A service and timer pair to run it:

# /etc/systemd/system/disk-check.service
[Unit]
Description=Alert on filesystems above 85%

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/disk-check.sh

# /etc/systemd/system/disk-check.timer
[Unit]
Description=Run disk check every 30 minutes

[Timer]
OnCalendar=*:0/30
RandomizedDelaySec=120
Persistent=true

[Install]
WantedBy=timers.target

Enable and verify:

sudo systemctl daemon-reload
sudo systemctl enable --now disk-check.timer
systemctl list-timers disk-check.timer
journalctl -u disk-check.service --since today

Because the script exits non-zero, a breach puts disk-check.service into a failed state, so systemctl --failed on the host shows it, and an OnFailure= unit can send it somewhere a person will see. That last journalctl command is the quiet advantage over cron: every run's output, timestamped and queryable on the box. When a check misbehaves at 3 a.m., the journal answers in seconds.

Tool 2: hardened SSH, the transport layer

Fleet automation means a key that can log into many machines, which makes that key your biggest liability. OpenSSH gives you three layers of containment. Use all three.

Layer 1: key-based auth only, on trusted hosts only

On every managed host, in a drop-in such as /etc/ssh/sshd_config.d/10-automation.conf:

PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
PermitRootLogin prohibit-password

For most sshd options the first value read wins, and the stock sshd_config on current Debian, Ubuntu and RHEL includes sshd_config.d/*.conf near the top, so a drop-in overrides the main file. Validate before you reload, and confirm the effective values:

sudo sshd -t && sudo sshd -T | grep -Ei 'passwordauth|kbdinteractive|permitrootlogin'
sudo systemctl reload ssh    # Debian and Ubuntu; the unit is sshd on RHEL-family

Keep an active session open while you test. Locking yourself out of a fleet is a rite of passage you can skip. And keep a curated known_hosts for the automation user rather than StrictHostKeyChecking no: the whole point of trusted hosts is that an unexpected key change stops the run.

Layer 2: restricted keys with command= and from=

The automation key shouldn't be a general-purpose login. In the target host's ~/.ssh/authorized_keys, prefix the key with restrictions (sshd authorized_keys format):

command="/usr/local/sbin/healthcheck.sh",from="10.0.4.10",no-pty,no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAA... automation@bastion

Now this key can do exactly one thing (run your wrapper script), from one source IP, with no interactive terminal and no forwarding. If it leaks, the blast radius is one read-only script. For automation that needs several verbs, make the command= target a dispatcher that reads $SSH_ORIGINAL_COMMAND and allowlists what it will run:

#!/usr/bin/env bash
# /usr/local/sbin/healthcheck.sh: allowlist dispatcher for the read-only key
case "$SSH_ORIGINAL_COMMAND" in
  "disk")     df -P -x tmpfs -x devtmpfs ;;
  "failed")   systemctl --failed --no-pager ;;
  "errors")   journalctl -p err --since "1 hour ago" --no-pager ;;
  *)          echo "denied: $SSH_ORIGINAL_COMMAND" >&2; exit 1 ;;
esac

Match on exact strings, never pass $SSH_ORIGINAL_COMMAND to a shell. Use one restricted key per privilege tier, not one god key.

Layer 3: client-side Match blocks and ProxyJump

On the control machine, structure ~/.ssh/config so automation traffic is explicit and routed through your bastion (ssh_config docs):

Host bastion
    HostName bastion.example.com
    User automation
    IdentityFile ~/.ssh/automation_ed25519
    IdentitiesOnly yes

Match host "prod-*" user automation
    ProxyJump bastion
    IdentityFile ~/.ssh/automation_ed25519
    IdentitiesOnly yes
    BatchMode yes
    ConnectTimeout 10
    StrictHostKeyChecking yes

ProxyJump tunnels through the bastion without landing keys on it. BatchMode yes makes failures fail fast instead of hanging on a password prompt, and IdentitiesOnly stops ssh from offering every key in your agent to every host, which both leaks key fingerprints and trips MaxAuthTries.

Tool 3: fleet loops with ssh and bash

With hardened transport, a fleet-wide check is a loop over a hosts file:

#!/usr/bin/env bash
# fleet-check.sh: read-only inspection across the fleet
set -euo pipefail
mkdir -p logs
while read -r host; do
  echo "== ${host} =="
  ssh -n -o BatchMode=yes -o ConnectTimeout=10 "${host}" \
    'df -P /var; systemctl is-active nginx postgresql; uptime' \
    2>&1 | tee -a "logs/${host}.$(date +%F).log" || true
done < hosts-prod.txt

Two details that bite people: ssh -n stops ssh from swallowing the rest of hosts-prod.txt from stdin (without it the loop quietly stops after the first host), and || true keeps one unreachable host from killing the whole run under set -e.

Serial loops crawl past 20 hosts. Parallelize with xargs:

xargs -a hosts-prod.txt -P 10 -I {} \
  ssh -n -o BatchMode=yes -o ConnectTimeout=10 {} 'echo "$(hostname) $(df -P / | tail -1)"' \
  2>&1 | tee "fleet-disk.$(date +%F).log"

Two rules keep this safe. First, separate read scripts from write scripts: different files, different restricted keys, so a typo in a report loop can never restart a service. Second, tee everything to dated logs. Those files are your only audit trail, so treat them as append-only and rotate them. Anything that mutates (restarts, cleanups, upgrades) gets a dry-run mode that only prints, and a person reads the output before the real pass.

Tool 4: logrotate, to stop disk pages at the source

A large share of the disk alerts from part one are unrotated application logs. The native fix takes five minutes (logrotate man page):

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Validate without waiting a day:

sudo logrotate --debug /etc/logrotate.d/myapp
sudo logrotate --force /etc/logrotate.d/myapp

Where the app supports it, have it reopen its log file on a signal (a postrotate script sending kill -USR1 or systemctl kill -s HUP) instead of copytruncate, which can drop lines written during the truncate window. On modern distros logrotate already runs from logrotate.timer, so there's nothing to schedule. Cap the journal too while you're here: SystemMaxUse=1G in /etc/systemd/journald.conf, then sudo systemctl restart systemd-journald.

Tool 5: unattended-upgrades, patching on autopilot

The recurring "are we patched?" check from part one automates natively. On Debian and Ubuntu (Debian wiki):

sudo apt-get install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Then scope it to security updates in /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
    "${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Mail "ops@example.com";

Do a dry run to see what it would install: sudo unattended-upgrade --dry-run --debug.

RHEL-family hosts get the same behavior from dnf-automatic: install it, set upgrade_type = security and apply_updates = yes under [commands] in /etc/dnf/automatic.conf, then sudo systemctl enable --now dnf-automatic.timer. Either way, keep reboots manual and let your fleet loop report /var/run/reboot-required (or needs-restarting -r on RHEL). Patching is safe to automate. Reboot timing usually isn't.

Where scripts stop scaling

Everything above is worth building. It'll retire real toil this week. Run it for a year across 50+ hosts, though, and the same failure pattern shows up: scripts execute, they don't observe, decide or explain.

01Observe

Runs blind

A timer fires whether or not the thing it checks still matters.

02Decide

One branch

Threshold crossed, run the cleanup. No weighing of a deploy in progress.

03Explain

A pile of logs

What ran, yes. Why, for whom and with what outcome, no.

Scripts execute. These three are what they leave to a person.

They don't observe. A timer fires whether or not the thing it checks still matters. Nothing notices that the disk check has been failing silently since a hostname changed, or that a "temporary" cleanup job is now deleting a directory another team depends on. Cron mails root, and nobody reads root's mail. That's how the graveyard forms.

They don't decide. A script encodes one branch: threshold exceeded, run the cleanup. It can't weigh context, like the disk filling because a deploy is in progress, or the host being mid-incident, or tonight needing a different action from last night. Every judgment call still pages a person, usually the one who wrote the script.

They don't explain. When a fleet loop changes 50 hosts, your audit trail is a pile of dated tee logs: what ran, but not why, for whom, or with what outcome. Reconstructing that after an incident is an evening's work, and after the author leaves it's archaeology.

The engineer-hours don't disappear either. They move. You stop typing df -h and start maintaining hosts files, rotating restricted keys and debugging the automation itself. It's useful, and it's a floor, not a ceiling.

Where this series goes next

The gap the scripts leave (observing, deciding, explaining) is what part three covers: CloudThinker working over the same hardened SSH you just built, with a dedicated key to trusted hosts only, read-only inspection by default, investigation before any action, and anything that changes a host waiting for your approval, with every command in an audit trail.