A Safe Server Automation Agent Over SSH: From Inspection to Action

What would you let a new junior engineer run on production in their first week?

df -h, journalctl, systemctl status, ps aux — read all day, no supervision needed. systemctl restart nginx? Only with someone senior watching. Deleting files under /var, editing a unit file, rebooting a box? Not in week one. Trust on a production fleet is earned by demonstrating good judgment on low-stakes work first, and it's granted per action, not all at once.

A server automation agent should earn trust exactly the same way — and that intuition is where part three of our Linux server automation series lands. Part one catalogued the recurring shell toil — log hunts, disk-full cleanups, service restarts, cert and package checks — and the risk ladder any automation has to respect. Part two built the DIY version with cron, systemd timers, hardened SSH config, and fleet loops, and ended on an honest note: scripts execute, but they don't observe, decide, or explain.

This article covers the half the scripts leave behind: how CloudThinker agents operate over SSH — a dedicated key to trusted hosts only, read-only inspection by default, investigation before any action, and mutating commands only under graduated autonomy with an approval gate and a full audit trail of every command run.

Connecting: a dedicated key, trusted hosts, about five minutes

CloudThinker connects to your servers the way part two taught you to let anything connect: key-based auth, never passwords, with a key you can identify and revoke independently of every other credential on the box.

The setup is deliberately boring:

1. Generate a dedicated keypair for the agent — don't reuse a human's key:

ssh-keygen -t ed25519 -f ~/.ssh/cloudthinker_agent -C "cloudthinker-agent"

1. Add the public key to a non-root user on only the hosts you choose. There is no discovery or network scanning — the agent can reach exactly the hosts you enrolled, nothing else.

2. Optionally apply the authorized_keys restrictions from part two, so the key is useless from anywhere but CloudThinker's egress:

from="203.0.113.10",no-agent-forwarding,no-X11-forwarding ssh-ed25519 AAAA... cloudthinker-agent

1. Paste the host details into the SSH connection form and run the connection test.

The SSH connection guide has the exact steps and the host-key verification flow. The whole thing takes about five minutes per environment, and at the end of it the agent holds precisely one capability: a shell, as an unprivileged user, on hosts you picked.

Read-only by default: the week-one rulebook

Out of the box, CloudThinker agents treat your servers the way you'd want that junior engineer to: look, don't touch. The default operating mode is inspection — the same commands you run by hand today, executed on a schedule or on request:

• Disk and filesystem — df -h, du -xh --max-depth=2 /var | sort -rh | head -20, inode usage, growth over time rather than a single snapshot
• Logs — journalctl -p err --since "24 hours ago", per-unit log pulls, pattern spikes across hosts
• Services — systemctl --failed, unit states, restart loops, dependency failures
• Processes and memory — ps aux --sort=-%mem | head -15, runaway CPU, RSS growth on long-lived processes
• Certificates and packages — TLS expiry dates, pending security updates, hosts that need a reboot for an already-installed kernel

None of that mutates anything. If your enrolled user has no sudo and no write access outside its home directory, the blast radius of the default mode is what a cat command can do — which is the point. You can put this in front of a security review as-is.

Investigation before action

Here's where an agent stops being a fancier cron job. The scripts from part two can tell you /var is at 91%. They cannot tell you why, and 3 a.m. you has to SSH in and find out.

When a CloudThinker agent finds disk pressure, it investigates the way you would: check which mount is full, walk the largest directories, identify the file that's actually growing, check which process holds it open, read the relevant unit and logrotate config, and then present a conclusion — "/var/log/app/worker.log has grown 9 GB in six days; the app-worker service logs at debug level and this path is missing from /etc/logrotate.d/" — with the evidence attached. Every command it ran to get there is in the record, so you're reviewing a reasoned finding, not trusting a verdict.

The same pattern applies to a flapping service (pull the journal around each restart, correlate with OOM kills or a dependency), a load spike (which process, since when, correlated with what deploy or cron entry), or a full inode table (which directory is breeding small files).

Graduated autonomy: mutating commands are earned, not assumed

Anything that changes a server sits behind an autonomy level you set per action type and per environment:

• Notify — the agent reports the finding and its root cause. Nothing else. This is the default for everything.
• Suggest — the agent proposes the exact command or change — the logrotate stanza to add, the specific systemctl restart, the apt-get upgrade list — with expected effect and rollback notes.
• Approve — the agent stages the action and executes only after a named human approves it in the app. This is where most teams keep restarts and cleanups for production.
• Autonomous — the agent executes and reports. Sensible only for provably reversible actions on non-critical hosts — say, compressing rotated logs on a staging box — and only after weeks of watching it be right at the Approve level.

The trust ladder maps cleanly onto the junior-engineer intuition: reads are free, writes are supervised, and unsupervised writes are a privilege granted narrowly after a track record — never a default.

Every command at every level lands in an audit trail: host, timestamp, the literal command line, its output, the finding it belonged to, and who approved it. When someone asks "what has the agent actually done to prod this month," the answer is a filterable log, not a shrug.

What a first pass over a typical fleet turns up

Illustrative numbers — a composite of what the first day of inspection tends to surface on a mid-market fleet of 30–50 Linux hosts. Your fleet will differ.

Two things about this table. First, none of it is exotic — it's part one's toil list, found automatically instead of during the next incident. Second, every row arrived at Notify level: the agent found, investigated, and explained all of it without having run a single mutating command.

Prompts to try in your first session

You talk to the agents in plain language, the way you'd brief a colleague:

"Check disk usage across all connected hosts and flag anything over 80% — and tell me what's actually growing, not just the percentage."

"Why did nginx restart on web-03 last night? Pull the journal entries around it and give me the sequence of events."

"Which hosts have pending security updates, and which of those need a reboot? Group them by how safe a staged rollout would be."

Each answer cites the commands it ran and their output, so verifying an answer takes seconds — the same evidence you'd have gathered yourself in part two, minus the twenty SSH sessions.

What the agents will not do

Because "AI with a shell on prod" should trigger exactly the scrutiny it sounds like it should:

• No mutating command runs at the default settings. Everything ships at Notify. Escalation to Suggest, Approve, or Autonomous is an explicit choice you make per action type, per environment.
• No host it wasn't given. There's no scanning, no lateral movement, no reaching hosts beyond the ones you enrolled with the dedicated key.
• No privilege it wasn't granted. The agent runs as the user you created. If that user can't sudo, neither can the agent — and you can enforce that in sshd_config and sudoers, outside the agent's reach.
• No unlogged action, ever. If a command isn't in the audit trail, it didn't run. That property holds at every autonomy level, including Autonomous.

The right mental model is the one you started with: a capable new engineer who reads everything, explains their reasoning, asks before touching production — and only gets broader access after being visibly right, repeatedly.

Earning back the hours

Part one measured the toil; part two automated the executing half with native tools. The half that remained — observing, deciding, explaining — is what an agent adds, and on a 30–50 host fleet that's typically several engineer-hours a week plus the incidents that no longer happen because the disk never quite filled up.

Try CloudThinker free — 100 premium credits, no card required — and follow the SSH connection guide to get your first fleet findings, read-only, within the hour.