A Safe Server Automation Agent Over SSH: From Inspection to Action

Think about what you'd let a new junior engineer run on production in their first week.

df -h, journalctl, systemctl status, ps aux: read all day, no supervision needed. systemctl restart nginx? Only with someone senior watching. Deleting files under /var, editing a unit file, rebooting a box? Not in week one. Trust on a production fleet is earned by showing good judgment on low-stakes work first.

A server automation agent should earn trust the same way, and that's where the last part of this series lands. Part one catalogued the recurring shell toil (log hunts, disk-full cleanups, service restarts, cert and package checks) and the risk ladder any automation has to respect. Part two built the DIY version with systemd timers, hardened SSH config, fleet loops, logrotate and unattended-upgrades, and ended on the limit: scripts execute, but they don't observe, decide or explain.

This part covers the half the scripts leave behind: how CloudThinker works over SSH with a dedicated key to trusted hosts only, read-only inspection by default, investigation before any action, and changes to a host only under rules you set, with every command in an audit trail.

Why scripts and runbooks stop at the first finding

The part-two setup tells you /var is at 91% on worker-03. What it can't do is the next twenty minutes, and the reasons are structural.

A check encodes one question. disk-check.sh asks "is any filesystem over 85%?" The useful questions come after a yes, and which one comes next depends on the last answer.

The cause isn't where the symptom is. A full disk might be a log, the journal, a deleted file still held open, container layers, or inodes. Each needs a different command, and only one of them is right on this host tonight.

Runbooks freeze judgment at writing time. "Clear logs older than 7 days" was right for the service that existed when it was written. It's wrong for the one that now writes its audit trail to the same directory.

Fleet loops don't remember. The loop ran yesterday and found 88%. Today it finds 91%. Nothing connects those two numbers into "growing about 1.5 GB a day since Thursday's deploy".

So the on-call still SSHes in. The agent's job is to be the one who does.

Connecting: a dedicated key, trusted hosts, about five minutes

01Your fleet

Linux hosts

Enrolled hosts onlyNo discovery, no network scanning
Unprivileged userLog reads via adm and systemd-journal groups
Pinned host keysAn unexpected key change stops the session

Dedicated ed25519 key

02Investigation

CloudThinker

InspectDisk, logs, services, processes, certs, packages
InvestigateFollows the evidence to a cause
ProposeThe exact command, its effect, its rollback

Every command logged

03Where the team works

Chat and on-call

Slack or TeamsFindings and approvals in the thread
PagerDutySame escalation path, evidence attached
JiraFollow-ups for the service owner

Nothing new to watch

The agent holds one capability: a shell, as an unprivileged user, on hosts you picked.

CloudThinker connects to your servers the way part two taught you to let anything connect: key-based auth, never passwords, with a key you can identify and revoke independently of every other credential on the box.

Generate a dedicated keypair for the agent. Don't reuse a person's key:

ssh-keygen -t ed25519 -f ~/.ssh/cloudthinker_agent -C "cloudthinker-agent"

Create a non-root user on only the hosts you choose and add the public key. There's no discovery or network scanning: the agent reaches exactly the hosts you enrolled. To let it read system logs without sudo, add the user to the groups that already grant log access:

sudo useradd --create-home --shell /bin/bash cloudthinker
sudo usermod -aG systemd-journal cloudthinker   # read the journal
sudo usermod -aG adm cloudthinker               # read /var/log on Debian and Ubuntu

Optionally apply the authorized_keys restrictions from part two, so the key is useless from anywhere but CloudThinker's egress:

from="203.0.113.10",no-agent-forwarding,no-port-forwarding,no-X11-forwarding ssh-ed25519 AAAA... cloudthinker-agent

Two inspection commands see more as root: du in directories the user can't read, and lsof for processes owned by other users. If you want them, allow exactly those two binaries and nothing else:

# /etc/sudoers.d/cloudthinker (edit with visudo -f)
cloudthinker ALL=(root) NOPASSWD: /usr/bin/du, /usr/bin/lsof

Check the paths with command -v du lsof first, since some distributions install lsof under /usr/sbin. Sudoers matches the full path, so a wrong one silently grants nothing.

Paste the host details into the SSH connection form and run the connection test.

The SSH connection guide has the exact steps and the host-key verification flow. It takes about five minutes per environment, and at the end the agent holds precisely one capability: a shell, as an unprivileged user, on hosts you picked.

Read-only by default: the week-one rulebook

Out of the box, CloudThinker treats your servers the way you'd want that junior engineer to: look, don't touch. The default is inspection, the same commands you run by hand today, on a schedule or on request:

Disk and filesystem. df -h, du -xh --max-depth=2 /var | sort -rh | head -20, inode usage, and growth over time rather than a single snapshot.
Logs. journalctl -p err --since "24 hours ago", per-unit log pulls, error spikes across hosts.
Services. systemctl --failed, unit states, restart counts, dependency failures.
Processes and memory. ps aux --sort=-%mem | head -15, runaway CPU, RSS growth on long-lived processes.
Certificates and packages. TLS expiry dates, pending security updates, hosts that need a reboot for an already-installed kernel.

None of that changes anything. If the enrolled user has no write access outside its home directory and no sudo beyond those two read-only binaries, the blast radius of the default mode is what a cat can do. You can take that to a security review as it is.

One disk alert, end to end

01Signal

/var at 91%

A scheduled inspection or an alert flags the mount.

02Walk

Where it went

du on one filesystem, then the journal, deleted-but-open files, images.

03Attribute

Who wrote it

The file, the process holding it, the unit and its log level.

04Explain

Why now

Missing logrotate stanza, debug logging since a deploy six days ago.

05Output

Proposal

The stanza to add and the backlog to compress, waiting for approval.

A script stops at step one. Steps two to four are the SSH session you used to run by hand.

Here's the 91% on worker-03, worked the way you would. The host, sizes and dates are illustrative.

1. Name the possible readings. Four usual suspects, each predicting something different:

Reading	What it predicts	The check
A log file growing unrotated	`du` finds a large, recent file under `/var/log`	`du -xh /var --max-depth=3`
The systemd journal	`/var/log/journal` is large	`journalctl --disk-usage`
A deleted file still held open	`df` says full, `du` can't account for it	`lsof -nP +L1`
Container images and layers	`/var/lib/docker` or `/var/lib/containerd` is large	`docker system df`

2. Run the cheap checks first.

df -h /var
sudo du -xh /var --max-depth=3 2>/dev/null | sort -rh | head -n 10
journalctl --disk-usage

du accounts for nearly all the usage, so the deleted-file reading is out. The journal is 400 MB, so that's not it either. /var/log/app is 9.2 GB, and almost all of it is one file.

3. Attribute it.

ls -lh --time-style=long-iso /var/log/app/
sudo lsof /var/log/app/worker.log
systemctl cat app-worker | grep -E 'ExecStart|Environment'
ls /etc/logrotate.d/

worker.log is 9 GB and was last rotated never. The process holding it is app-worker. Its unit sets LOG_LEVEL=debug, and there's no stanza for /var/log/app in /etc/logrotate.d/.

4. Explain why now. The journal shows the unit was restarted six days ago, and the deploy event in the same minute is the one that changed the log level. Growth since then is about 1.5 GB a day. At that rate /var fills in under two days.

5. Propose, with rollback. The finding in the thread: what's full, what's writing it, why it started, and how long until it fails. Proposed: add a logrotate stanza for /var/log/app/*.log (shown in full), force one rotation to compress the backlog, and flag the debug log level to the service owner instead of changing it. Each step has its expected effect and how to undo it. Nothing runs until someone approves it, and every command above, with its output, is already in the record.

The same pattern covers a flapping service (pull the journal around each restart, check for OOM kills or a failing dependency), a load spike (which process, since when, next to which deploy or cron entry), and a full inode table (which directory is breeding small files).

Changes run only inside the limits you set

CloudThinker acts within Auto Mode, so you decide what runs on its own and what waits for approval. Every command lands in an audit trail: host, timestamp, the literal command line, its output, the finding it belonged to, and who approved it. When someone asks "what has the agent done to prod this month?", the answer is a filterable log.

What a first pass over a typical fleet turns up

The numbers below are illustrative, a composite of what the first day of inspection tends to surface on a mid-market fleet of 30 to 50 Linux hosts. Your fleet will differ.

Finding	Detail	Proposed next step
Disk pressure	3 hosts above 85% on `/var`; one app log unrotated, 9 GB	Add logrotate stanza, compress backlog
Restart loop	1 unit cycling every 4 minutes for 11 days, hidden by `Restart=on-failure`	Fix missing env file, then restart cleanly
Certificate expiry	2 TLS certs expiring in under 21 days, no renewal timer	Enable the `certbot renew` timer
Pending security updates	17 hosts with outstanding security patches; 6 need a reboot	Schedule a staged upgrade window
Memory growth	One JVM at 88% RSS, growing ~2% per day since last deploy	Flag to service owner before OOM
SSH hygiene drift	4 hosts still accepting password auth, against the baseline	Set `PasswordAuthentication no`, reload
Time drift	1 host 40s off; `chronyd` dead after a partial migration	Re-enable chrony, verify sources

None of it is exotic. It's part one's toil list, found on a schedule instead of during the next incident. And every row was found, investigated and explained without a single command that changed a host.

Why this beats the traditional approach

	Scripts and runbooks	With CloudThinker over SSH
A disk alert	A percentage	The file, the process, the cause and the time left
Next question	Whatever the script author predicted	Chosen from what the last check found
Context	None between runs	Trends, deploys and past findings
Changes	Whatever the cron job was told	Proposed with rollback, gated by Auto Mode
Audit	Dated `tee` logs	Every command, its output, the finding and the approver

The deeper difference is where the investigation lives. With scripts it lives in the on-call's SSH session and disappears when they log out. With an agent it's written down, command by command, and the next person starts from it.

Getting started

Three steps. None of them needs write access.

1. Enroll a few staging hosts with the dedicated key and an unprivileged user, following the SSH connection guide.

2. Ask for a read-only inspection. Nothing changes on the hosts. From chat:

"Check disk usage across all connected hosts and flag anything over 80%. Tell me what's actually growing and why, not just the percentage. Propose only, change nothing."

3. Point it at something that already happened.

"Why did nginx restart on web-03 last night? Pull the journal entries around it and give me the sequence of events."

"Which hosts have pending security updates, and which of those need a reboot? Group them by how safe a staged rollout would be."

Each answer cites the commands it ran and their output, so checking it takes seconds: the same evidence you'd have gathered yourself in part two, without the twenty SSH sessions.

What the agents won't do

"AI with a shell on prod" should get exactly the scrutiny it sounds like it deserves:

Nothing that changes a host runs unchecked. Every command that mutates state goes through the Auto Mode rules you set.
No host it wasn't given. No scanning, no lateral movement, no reaching hosts beyond the ones you enrolled with the dedicated key.
No privilege it wasn't granted. The agent runs as the user you created. If that user can't sudo, neither can the agent, and you enforce that in sshd_config and sudoers, outside the agent's reach.
No unlogged action. If a command isn't in the audit trail, it didn't run.

The right mental model is the one this post started with: a capable new engineer who reads everything, explains their reasoning, asks before touching production, and earns trust by being visibly right, repeatedly.

Earning back the hours

Part one measured the toil. Part two automated the executing half with native tools. The half that remained (observing, deciding, explaining) is what an agent adds, and on a 30 to 50 host fleet that's typically several engineer-hours a week, plus the incidents that don't happen because the disk never quite filled up.

Try CloudThinker free with 100 premium credits and no card required, and follow the SSH connection guide to get your first fleet findings, read-only, within the hour.