Everything so far has been about the Mac in front of you. This part of the manual widens the lens to the whole estate: the Linux boxes in a rack, a vintage Xserve still running macOS Server, a Raspberry Pi on a shelf, a cloud VPS, even a Windows host. You link each one once over SSH, and from then on FrontierStack treats them as a single fleet — running diagnostics, fanning commands out, deploying sites, and watching their health. It also keeps track of where this Mac is on the network, so a laptop that moves between home, office and a café doesn't fire false alarms about servers it simply can't see right now.

This chapter covers linking and running the fleet. Chapter 9 goes deeper on the network and perimeter, Chapter 10 on hardening and the security audits, and Chapter 11 on continuous monitoring and alerts. The AI Administrator can drive almost everything here over the same tools — see Chapter 13.

8.1Locations & Places

Open Locations & Places ▸ Locations. A location is simply a named network — Home, Office, On VPN — recognised from several signals: the Wi-Fi SSID, the router (gateway) MAC address, the Wi-Fi BSSID, the subnet prefix of your IP address (e.g. 192.168.1.), your public IP block, and whether a VPN is up. The header shows where you are right now (Now: Office), with the live snapshot beneath it: Wi-Fi name, IP, gateway and VPN state. Detection runs every 30 seconds.

When FrontierStack sees a network it doesn't recognise it offers a New network detected banner — click Add… to save it. You can also press Add Current Network as Location, then refine the criteria: a location matches when all the set criteria hold (with SSID and router MAC treated as any-of), so leaving a field blank simply ignores it. When several locations match, the most specific one wins rather than the first in the list. Use the Mode picker to pin a location manually instead of auto-detecting — handy when the signals are ambiguous.

Not all signals are equally trustworthy, and this matters more than it sounds. A subnet prefix is not an identity: home routers, cafés and hotels hand out 192.168.1.× everywhere, so a location defined only by its subnet would happily match a completely different building. The router MAC is the strongest signal — it is unique to one piece of hardware and readable without any special permission — which is why Update all criteria from current network is the one-click fix for a location that was saved before router matching existed.

8.12.1Pinning a place with GPS

For the awkward case — two sites that genuinely share a subnet and you would rather not fuss with MAC addresses — a location can also carry a GPS pin. Stand in the place, open the location and press Pin This Spot (GPS); a Radius slider sets how far counts as "here" (150 m by default, because indoor positioning is coarse and a site is a building, not a point).

The pin is a cross-reference, not a requirement. It does two things: it lets a weak match be confirmed, and — more usefully — it lets a wrong one be vetoed, so Home stops claiming a café that happens to use the same subnet. To keep that from ever making detection worse, a location identified only by shared criteria (a subnet, a VPN toggle) is no longer reported as confirmed when a position is available and contradicts it; you get Away / Unknown instead of a confident wrong answer.

Devices belong to Locations too. In Device Discovery, a monitored device's ⋯ menu ▸ Assign to Location places it at a specific site — or leave it Automatic, where it falls into whichever location owns its subnet. You can also set it from the device's own pane, and each location's row lists the devices assigned to it. When the Mac arrives at a recognised location, FrontierStack checks those devices (skipping any seen recently, e.g. over VPN) and shows each one's status: green reachable, red if it should be here but isn't answering, and orange when it's on a Wi-Fi segment this Mac can't reach — usually a different access point or band, such as a 2.4 GHz-only sensor while the Mac is on 5 GHz. Orange means “can't verify from here”, not “gone”.

The point of all this is the Location-Dependent Monitoring section. Mark a device, Wi-Fi network, Cloudflare zone or watch as critical in Overview ▸ Alerts, then scope it here to Only at Office. That check is paused everywhere else: your NAS won't ring as DOWN from a hotel, and your home Wi-Fi won't alert as lost while you're at work. Paused items resume silently when you return. When the location itself changes, FrontierStack can post a local notification, send a "📍 Now at: Office" alert through your Messaging Gateways, or both.

8.2Linking a server over SSH

A server joins the fleet through the Link a Server sheet — reachable from a device's pane, from Device Discovery, or from the Cloud Servers and Remote Tools panes. Enter a name, the host or IP, the port, an SSH username (root, ubuntu, ec2-user…) and the login password. That password is used exactly once: the app installs this Mac's managed SSH key into the server's authorized_keys, then connects key-only from then on. Nothing is stored except the server's address — the password is discarded.

For a key-only server with no password login, expand Can't use a password? Install the key manually, copy the shown public key into the server's authorized_keys yourself, then link. Tick Also install the monitoring helper to set up the host monitor in the same step (covered below).

Linked servers carry a small green or red status dot for reachability. If it goes red with Permission denied (publickey) — typically after a server is reinstalled, or the key was never installed — the AI tool repair_ssh_access clears the stale host key and re-installs the managed key, using a one-time login password you keep as a vault secret. The password is read locally and re-installs key access; on success the dot turns green again.

8.3The managed key and the saved sudo password

Key access lets the app connect, but many useful operations — reading /var/log, editing Apache config, reloading a firewall — need root on the server. FrontierStack handles this with a per-host sudo password, saved in the macOS Keychain (never in plain files). It is captured automatically when you link a server, and you can set or change it later in the host's pane or in the Remote Tools target section (the Sudo password field).

When a root-only command runs, the helper primes sudo's credential cache with that saved password over the SSH session, then runs the command. If no password is saved, root operations fall back to passwordless sudo -n and simply report "needs sudo" when that isn't configured. Because secrets are referenced by name, you can keep several — for example a distinct MySQL password per server in the .env vault.

8.4Fleet Run — one command across every node

Fleet & Remote ▸ Fleet Run fans a single operation out across all linked servers at once, or a selected subset, with a live per-host result. Pick your Targets (toggle hosts; the reachability dot shows which are up), choose an Operation, and press Run on N Server(s) — everything runs concurrently over SSH and each host reports back below, expandable to its full output.

Operations are OS-aware: a package update maps to apt, dnf, pacman, zypper, apk or brew depending on the host; a service restart tries systemd, then Homebrew. You never write per-OS commands by hand.

Fleet Run is the fastest way to spot drift: run Disk usage or Uptime & load to compare hosts at a glance, or use the file-checksum and health-check tools in Remote Tools (next) to find a config that's diverged across the fleet.

8.5Remote Tools — per-node diagnostics

Fleet & Remote ▸ Remote Tools is a suite of SSH-driven diagnostics, distilled from real fleet-debugging work. Choose the target first — a pinned server, or Other (IP / host)… for an ad-hoc address that isn't a saved server (its sudo password persists in the Keychain too) — then the tool and its parameters. Read-only checks need no root; ones that read protected logs or config use the saved sudo password through the helper, and a few are marked mutating.

Some tools work across several selected nodes at once — File Diff, Healthcheck, TLS Expiry and Serving Path Diagnosis (who's up, which public IPs each node binds, per-process CPU saturation and stopped services, cross-checked against every pinned Cloudflare zone's origin IP) are fleet-wide by design — which is exactly how you confirm every web node is serving the same content with a valid certificate. The bottom of the pane also offers safe over-SSH editing of /etc/hosts (root, through the helper) and the SSH user's ~/.ssh/known_hosts, each backed up to .fsbak first.

8.6Recovering a wedged server — and a safe power reset without SSH

A server can stop answering SSH while its services keep running — overloaded, not down. FrontierStack shows this: when SSH is unreachable but a service port still answers (say MySQL on 3306), the server's dot turns orange, not red, and its pane reads "SSH is wedged, but the server is still serving" and lists what's answering, with the database's version banner. The machine is alive — only its management channel is stuck — so don't assume it's dead or blindly cut the power.

Try a normal Reboot first. After Reboot, the dot flashes while FrontierStack waits out a grace period and then polls, refreshing the instant the host returns — no need to keep pressing Refresh.

Make a power reset safe — without SSH. When a database is still answering, the pane shows a red-outlined EMERGENCY box, Prepare databases for a safe power reset. Each step runs over the database's own connection (no SSH), using the credentials saved in Database Health, top to bottom: Stop accepting writes (reads still work, reversible), Quiesce & flush to disk, then Clean-shutdown. After a clean shutdown a hard power-cycle is safe, because nothing dirty is left in memory to corrupt. Coverage differs by engine — MySQL, Redis and MongoDB can be cleanly stopped over their protocol; PostgreSQL can be quiesced and flushed but not stopped over SQL, so you flush then power-cycle. The clean-shutdown button is red and confirms first.

Power-cycle, then bring it back. The hardware power controls (SwitchBot plug, smart powerboard/PDU, KVM) each offer Shut Down Databases First in their confirmation, so you can flush and stop the databases before cutting power. And if the shutdown relieved the pressure and SSH recovers, a Start button starts the database again over SSH — so the ideal path is often: clean-shutdown the runaway database, let the box recover, and Start it, with no power-cycle at all.

8.7The Host Monitor (remote agent)

SSH diagnostics run on demand. For continuous visibility, install the Host Monitor — a tiny, read-only Go helper (fsagent) that lives on the server and reports its own health, with no per-cycle SSH. From a linked host's pane, choose to install it (or tick the box in Link a Server); over the existing key the app pushes the right binary for the host's OS and architecture, sets up the platform service (systemd, launchd or BSD rc.d), and pins its TLS certificate. The AI tool install_monitor does the same on request.

Once enrolled, the monitor streams live metrics: CPU, memory, per-mount disk, network interfaces up, detected services, firewall and fail2ban state — and on GPU hosts, per-GPU temperature and utilisation, so a mining or ML rig surfaces thermals. list_monitors reports the fleet's helpers, their versions and latest metrics. Crucially, the monitor keeps watching and can alert even when the Mac app is offline, notifying directly through its own channels and replaying anything it buffered when the app returns. Logs windows prefer the monitor too — it runs as root on the server, so it reads privileged logs without needing the sudo password at all.

An enrolled host's pane also keeps a Behavior & History chart — CPU, memory and disk at five-minute resolution for up to 30 days, against a learned baseline that flags unusual behaviour. Vertical lines place incidents on the same timeline: a solid red line is a server reboot (derived from the reported uptime), a dashed orange line is a watchdog issue — a forced service restart, a database-corruption warning or a reboot escalation. If the OS crash reporter starts using CPU (a process crashing repeatedly), an orange Crashes curve joins the chart, so "MySQL was force-restarted twice last night and something was crash-looping" is visible at a glance.

The monitor is read-only by default. Turning on Allow actions in its Host Monitor pane unlocks a small, fixed set of control verbs — never arbitrary shell — driven by the gated monitor_action tool: restart/reload/start/stop an allowlisted service, flush DNS, reload the firewall or fail2ban, or reboot. Each action is logged on the server. Targets cover Linux (amd64/arm64/arm — covering every Raspberry Pi), macOS (including legacy Intel builds back to OS X 10.11 El Capitan), FreeBSD (pfSense/OPNsense/TrueNAS) and Windows.

Keeping a monitor healthy over time is handled for you. Rotate monitor credential replaces its bearer credential in place: the new secret is generated on the Mac, sent over your approved FS1 signing key, and activated on the server without ever being shown or handed to an AI. With Allow self-update enabled, pushing a newer build is safe by construction — the candidate binary's release signature is verified, it must pass a compatibility self-test before it is given root, and the previous binary is kept as a rollback; if the new one fails to answer its health check, FrontierStack restores the old one automatically. Pull-mode updates fetch the binary through a short-lived, single-use grant rather than a durable token. When a monitor is verifying FS1-signed requests, its pane shows a FS1 signed badge.

8.8Cloud Servers, Server Clone & Remote Apache

Alongside the compute panes, AWS ▸ DynamoDB lists the tables in the selected region with their status, item count, size and partition/sort key. It leans on the two things that actually cost you: the capacity mode capsule separates provisioned tables — which bill for their read and write units continuously, whether or not anything touches them — from on-demand tables that bill per request; and a no PITR flag marks any table without point-in-time recovery, which is off by default on every new table and is the only way to rewind after a bad write. Point-in-time recovery can be switched on or off from each row's menu. DynamoDB has no bulk describe call, so each table is queried individually and the list is capped at 40 per region.

Fleet & Remote ▸ Cloud Servers rolls up one inventory across every cloud provider that has a configured API token — DigitalOcean, Vultr, Linode, Hetzner, Sakura, Contabo — plus AWS EC2, Lightsail and RDS. It is a read-only roll-up (power control stays in each provider's own pane). Each running instance with an IP offers a Services button to discover what's listening, and a + Helper button that links the instance over SSH and installs the monitor in one move — just set the right username and key in the sheet.

Server Clone is a guided wizard that stands a fleet server up to match this Mac: it installs the detected Homebrew stack, copies configs and site files to their original paths, pushes your domain configs and reloads the web server, and can clone MySQL databases (local mysqldump piped into mysql on the target over SSH). Every step runs over key-based SSH and is safe to re-run; existing data isn't deleted.

When a single site is ready to go live, the Promote to Production push (from a domain's context menu) sends its vhost config to the same path it uses locally, optionally copies the document root and database, points DNS at the target via Cloudflare, and can issue a real Let's Encrypt certificate with certbot — files, database, vhost, DNS and HTTPS in one push.

You can also manage a remote server's Apache directly. The Apache pane gains a host selector (This Mac / each linked server) when any remote host exists; pick a server and FrontierStack discovers its Apache over SSH — version, config layout, and every active vhost with its real log paths — and lets you edit vhosts, modules, MIME types, ports and WebDAV, with each change validated (httpd -t) and gracefully reloaded before it sticks, rolling back if the config is invalid. It even understands the legacy macOS Server.app Apache tree. From the Sites list, Push to Server… renders a local site's vhost onto a monitor-linked server with its document root swapped in. Chapter 7 covers the local Apache and Sites workflow these mirror.

8.9KVM-over-IP: console and power, lights-out

SSH and Screen Sharing need the machine to be up and on the network. When it isn't — a kernel panic, a BIOS/firmware screen, a network stack that never came up — you need out-of-band access: a KVM-over-IP appliance that captures the real HDMI output and injects USB keyboard/mouse, independent of the operating system. The Remote KVM pane registers as many of these as you own — PiKVM, JetKVM, TinyPilot, NanoKVM, GL.iNet Comet, or any generic unit — each pinnable, each one click from its web console down to the BIOS.

Units with power control close the loop. A PiKVM with an ATX board, a network PDU, or GL.iNet's Comet Pro (GL-RM10) — which is a KVM and a smart power plug in one device — can switch mains power, so you can hard power-cycle a truly frozen box. Link such a unit as a server's power source in its Power section (beside SwitchBot, UPS and PDU commands) for On/Off/Cycle. The AI Administrator can drive it too: list_kvms, kvm_power, and server_power — the last reaches a host's linked plug/PDU/KVM to restart a hung server that won't answer reboot_host.

8.10Old Macs, Raspberry Pis & Windows hosts

Part of the point of a fleet is keeping older or odd hardware useful. A vintage Xserve or Mac mini still running macOS Server links like any other host; the Remote Apache discovery understands Server.app's own Apache tree and its internal ports, so its websites show up and can be edited, and the legacy Intel builds of the monitor run as far back as OS X 10.11 El Capitan. A shelf of Raspberry Pis links as ordinary Linux boxes — the monitor's arm build covers every Pi including the Zero — so you watch their CPU, disk and services right alongside everything else.

Windows hosts join over Microsoft's OpenSSH server: link them like a Linux box for SSH diagnostics, and install the monitor as a native Windows Service (because Windows services don't reliably inherit machine environment, its config is read from a file beside the executable). Windows monitor installs are done manually with the supplied PowerShell installer from the Host Monitor pane, rather than auto-pushed.

For interactive control beyond diagnostics, the catalog also carries the remote-access tools — RustDesk, MeshCentral, Apache Guacamole, Windows Remote Desktop and WinRM — but those are services you run, not the SSH fleet itself. The fleet's strength is uniform, scriptable, low-overhead management of everything from one window — and, with Chapter 13's run_script targeting any node by name, an AI that can diagnose and fix a remote server two steps at a time: a read-only test, then a minimal fix, re-run to confirm.

8.11Migration wizards: moving a stack onto a new device

Retiring an old server rarely means "reinstall everything by hand." FrontierStack ships guided migration wizards that lift a stack off a legacy machine and stand it up on a destination you choose — this Mac, another Mac, or a Linux box. The source is only ever read; everything is written to the target.

The Migrate Setups pane handles a MAMP, XAMPP or Apple Server.app web stack. It auto-detects the source, then lets you tick exactly what to carry: the Homebrew tools, web files, MySQL databases, the sites (vhosts), MIME overrides, Apache modules, PHP settings, WordPress wp-config.php fix-ups and Git repositories. A Set up on picker chooses the destination: leave it on This Mac for the full in-app migration, or pick a linked host to install the stack on that new device over SSH, copying the web files (rsync) and databases (dump over SSH) across — with a Linux toggle to use apt/dnf instead of Homebrew.

The Apple Server Migration pane is a one-click wizard for a whole Server.app machine. Point it at the source (this Mac or a remote one) and it inventories every service with serveradmin — Websites, Mail, Calendar and Contacts, Messages (XMPP), VPN, DNS, DHCP, NetInstall, Open Directory, File Sharing, Time Machine, Profile Manager and more — and anything else it finds running, such as a git server, databases, Docker or a standalone Nginx. Each service shows a status badge and a modern equivalent; where there's more than one sensible replacement (Calendar → Radicale / SOGo / Baïkal, VPN → WireGuard / strongSwan, DHCP → dnsmasq / Kea) you choose which one. Pick a target machine and OS, and the wizard installs the equivalents and prints a precise carry-over checklist. Services with dedicated importers — Websites (full vhost + file copy) and Open Directory (users and groups) — hand off to their own panes; a detected git server hands off to the Git Server migration.

8.12The Git Server: Gitea & repository migration

The Git Server pane stands up a self-hosted Gitea (behind Caddy for automatic HTTPS) on any target — this Mac, a linked server, a NAS, a Docker host or a Linux box — detecting the platform and installing the right way. Once it's running, connect with an access token for a live monitor: version, repository count and backup-staleness alerting when a mirror falls behind its source.

The same pane migrates or mirrors another Git server into Gitea. Point it at on-disk bare repos — the legacy macOS app Simple Git Server, Xcode Server, Gitolite, a plain git-daemon, or any folder — or at a running remote server via any git://, https:// or ssh:// clone base (enumerate the repo names over SSH, or paste them). Scan lists the repositories; Migrate mirror-clones each one with full history, every branch and tag, and pushes it into Gitea, creating the destination repo through the API. The copy runs on this Mac.