Monitoring & Alerts

A control panel that only lets you change things is half a tool. The other half is knowing when something has gone wrong — ideally before it becomes an outage. FrontierStack runs a continuous monitor sweep across everything you have told it to watch and turns the first sign of trouble into a message on whatever channel you choose. This chapter covers the Local Health board that summarises it all, the Alerts pane that raises and delivers warnings, the messaging gateways that carry them, and the specialised monitors for cloud services, project tools, power and storage.

11.1The Local Health board

Open Overview ▸ Local Health for the single screen that answers "is anything broken right now?" Every monitored service, device and subsystem appears as a row marked UP or DOWN, grouped by area — web and database services, the fleet, sites and certificates, disks, UPS and SNMP devices, connected cloud accounts. At the top sits a one-line summary in the form X up / Y down, so you can take in fleet-wide status in a glance without reading every row.

Groups are collapsible, but any group containing a failed item opens automatically. Select the history button on a row for its last 30 checks, measured response-time sparkline, exact rolling observed uptime over 24 hours, 7 days and 30 days, and recent transitions. Coverage sits beside each uptime value: app-off, paused and out-of-network time is unknown, never guessed as healthy. Fleet-wide Events retains at most 1,000 first observations, outages and recoveries for 31 days without storing endpoints, credentials or response bodies.

The board is the same data the AI Administrator reads through its get_server_health tool: ask it "what's down?" and it reports exactly what this screen shows, then offers to investigate before suggesting a fix (see Chapter 13). The sweep runs about every sixty seconds; each DOWN item is also attributed to the pane that owns it, which is why a red count capsule can appear on a sidebar row and on the Dock icon — you are told where the trouble is, not just that there is some.

11.2The Alerts pane: what is watched

The Local Health board shows status; the Alerts pane (Overview ▸ Alerts) decides what is worth a message and sends it. Switch on "Monitor critical services and alert me" and the sweep starts raising alerts. What it watches:

• Site & service health — HTTP/loopback checks, TCP ports, and deeper protocol probes (an SMTP/IMAP login, a database SELECT 1, an LDAP bind) so a service that is listening but actually broken still trips. The MySQL probe additionally reads the connection pool (current use vs max_connections and the "too many connections" refusal counter): it alerts when the pool passes 90% or when clients were refused between sweeps, and every sweep is charted as Connection Health History in the Database Health pane — pool % over time with red lines marking refusal bursts and outages.
• Linked servers — every fleet host's SSH endpoint is reachability-checked each sweep (a bare TCP connect, no login attempt), so a server that is completely off raises a plain "Server down" alert — and a "Server recovered" when it returns. This is separate from the SSH cool-down notice, which fires only when a host is up but rate-limiting connections.
• Certificates & domains — TLS expiry across the whole fleet, plus registrar / DNS / SSL checks that warn you days ahead rather than on the morning a domain lapses (Chapter 7).
• Disks filling up — a volume crossing its free-space threshold, surfaced from the Disk Health monitor below.
• Security exposure — a new open port or unexpected listener, a VPN tunnel drop, or a critical SSH-hardening failure while SSH is reachable from the network.
• Failed payments & billing — connected paid services (OpenAI, Anthropic, Cloudflare, Google Cloud) returning insufficient-quota, low-balance or auth errors — the symptom of a lapsed card.
• Connected cloud services — live monitors for Stripe, Shopify, Freshdesk, SendGrid and more, firing when a metric you chose crosses its threshold.

Most checks are threshold-based, so you decide what "trouble" means: alert when a certificate is within N days of expiry, when open tickets pass a number, when a disk crosses a percentage. Conservative thresholds turn a post-mortem into a heads-up. The pane also offers "Alert on intrusions" — new fail2ban/CrowdSec bans, high-severity Suricata signatures and notable Wazuh alerts (level ≥ 7) are messaged as they happen, and existing history is never re-alerted.

SwitchBot devices. Pinning a SwitchBot device watches it through the SwitchBot cloud. Plugs are read every 2 minutes and get the Wi-Fi-unstable warning; locks, sensors, curtains, Bots, lights, Hub 2/3 and other devices that report status are read every 10 minutes and alert when SwitchBot says the device, or the hub it connects through, is offline. Cameras and Hub Mini report no status and are never shown as offline. To stay inside SwitchBot's 10,000-calls-a-day quota the intervals lengthen automatically when many devices are pinned; the quota is per account, so each account is budgeted separately. A pinned SwitchBot device's dot is green only after its own status check has answered, and stays grey while it is unchecked or has no status. Turn on Receive SwitchBot webhooks in the SwitchBot pane and give it Internet access (Cloudflare Tunnel, Tailscale Funnel or a router forward), and Indoor Cam and Pan/Tilt Cam report each motion detection, shown as last active. Pan/Tilt Cam 2K and the Plus models send no webhooks. A quiet camera is never reported offline.

Several SwitchBot accounts. A token only lists the devices added to its own account, so a home someone else owns and shares with you is reached through their token. Add it under Other Accounts in the SwitchBot pane. Commands and checks for each device use the account that lists it. SwitchBot's API has no rooms, so Group by sorts devices by the hub they connect through, by the Place you pinned them to, or by account.

11.3Messaging gateways

An alert is only useful if it reaches you. The Alerts pane lets you enable several gateways at once and list multiple recipients per channel, so the right people are reached on the app they already have open. Paste a bot token or webhook URL, send a test, and you are live. With KeyRotate paired, the Telegram bot token and the LINE channel access token stay in step with it: a token rotated in KeyRotate is used for the very next alert, and KeyRotate can issue LINE's 30-day tokens by itself. The AI Administrator can enumerate the enabled gateways with get_messaging_channels and send through them with send_notification (Chapter 13).

11.18.1Paging the on-call (PagerDuty, Opsgenie, urgent ntfy)

The message gateways above are fire-and-forget text. The paging channels are different in kind: they are stateful. When a monitored item goes down, FrontierStack triggers an incident keyed to that item — PagerDuty (an Events API v2 routing key) or Opsgenie (a GenieKey) then runs your escalation policy, pushing, texting or phoning whoever is on call until someone acknowledges. When the item recovers, FrontierStack resolves the same incident automatically. Because the incident is keyed per item, a flapping service updates one incident rather than paging the rotation over and over, and nobody is woken for an outage that has already ended. Without an on-call service, the ntfy gateway's Page on down toggle sends "down" alerts at ntfy's maximum priority — a louder, repeating tone that overrides many phones' quiet settings — while recoveries and reports stay at normal priority. Each paging channel has a Send Test Page button that triggers a real incident and auto-resolves it about ten seconds later, proving the whole escalation path end to end.

11.18.2Telling someone else: contacts, presets and escalation

The channels above reach you. Many incidents also need someone else: a supervisor who wants an email, a second support person, or the technician who has to go to the site. Alerts ▸ Contacts & Escalation handles this in three parts.

For example, the preset Email the supervisor, page the on-site tech sends the supervisor an email and gives the technician a text and a phone call that reads the alert aloud. Tech now, supervisor if not fixed in 30 min is an escalation: the supervisor's step runs only if the item is still down 30 minutes later and nobody has pressed Acknowledge. Pending escalations are listed in the section and in the Alerts palette (Ack), and each one is cancelled automatically when the item recovers. People who were told something broke are told when it is fixed; nobody gets a phone call for good news, so an all-clear goes by text instead. Choose New Preset ▸ Start from a template to use a ready-made preset. It adds a blank contact for each role you don't have yet. Test sends every step at once, marked TEST, with delays ignored. On a pinned camera, router or other device, the offline-alert switch has a Who else gets this alert… button: it lists who besides you is told, and opens this section with a rule for that device already filled in, so you only pick the preset.

Contacts use the gateways you have already set up, with their own address in place of yours. Emailing a supervisor uses your SMTP account, and texts and calls use your Twilio number, which must be able to make voice calls. Those gateways can stay switched off for your own alerts. Each message states the contact's role, the location, and the preset's instructions, so the person understands why they received it. A missing address or an unconfigured gateway appears in Delivery Errors as, for example, SMS → Alex (On-site technician). A network failure is retried twice, two minutes apart, before it is reported.

IFTTT. Turn on IFTTT (Webhooks) in Messaging Channels to fire an IFTTT applet on every alert: value1 is the title, value2 the message and value3 is critical for a down alert or info otherwise. Give down alerts their own event name to trigger something louder, such as flashing lights, than a recovery does. A contact can also have their own Webhooks key, so a preset step fires an applet on their account. IFTTT's Webhooks service needs an IFTTT Pro plan.

Tokens and webhooks live in the macOS Keychain and never leave your Mac. You can also send a scheduled AI Evaluation Report: the AI Administrator composes a summary from your live monitored health (running read-only diagnostics) and emails it daily or weekly, or pushes it to all enabled channels — with an optional fuller report whenever two or more critical items are down at once.

11.4Delivery errors and how to fix them

A monitor is worthless if its alerts silently fail to send. FrontierStack verifies every send: when a gateway fails, it is recorded in the Delivery Errors section of the Alerts pane with plain-English fix-it guidance, the raw error underneath, and a Fix button that scrolls straight to the misconfigured channel's settings. A red badge appears on the sidebar's Alerts row and on the Dock icon so you actually notice, and it stays until you clear it. The AI Administrator reads the same list through get_alert_errors, so you can simply ask "why aren't my alerts arriving?"

The common causes are mundane and quick to fix:

11.5The on-host Service Watchdog & database health

The monitors above run on your Mac. For a critical server you also want a watchdog that lives on the box itself, so it keeps working when your Mac is asleep, offline, or simply not the machine that failed. FrontierStack's Service Watchdog is deployed with the host monitor (Chapter 8): it health-checks a service on a short interval, restarts it when it fails, and — as a last resort, and only when you have allowed it — reboots the machine, with a minimum-interval guard so a still-broken service can't cause a boot loop. Each watch has a check type: a process check, an HTTP or TCP probe, or a purpose-built MySQL or PostgreSQL check. When the whole server wedges while each service still looks fine — SSH stops answering or the network is lost — Lockup reboot in the same panel reboots it from the inside; see Chapter 8.

11.18.3A recovery ladder for pinned servers

The pinned server's Service Watchdog panel can add two opt-in steps after ordinary on-host service recovery. Reboot from FrontierStack lets the Mac issue one verified reboot after a new Recovery exhausted event. You choose the delay. If a controllable outlet is linked in Power Control, the next step can power-cycle the server after it remains completely unresponsive for the chosen period. FrontierStack waits 90 seconds after the reboot attempt before judging the host, and the power timer has a five-minute minimum.

The hard-power test is deliberately stricter than a failed SSH login. The Mac must have a healthy network path, and the server must have no SSH response, no healthy monitor report, no ping or ARP evidence, and no response on the common service ports. Any sign of life cancels that incident. A power cycle runs once, then the ladder stops. Old watchdog events are marked handled when you enable the policy, and watchdog_reboot_suppressed remains final, so the app cannot defeat the helper's boot-loop protection.

11.18.4Keep watching while the Mac is closed

A wedged server cannot reliably operate its own smart plug. Choose a different enrolled server under Recovery observer to put the last step on that machine's helper. The observer keeps probing the target by private IP, ping, and several TCP ports even when FrontierStack is closed. It first confirms that the private power controller answers its status request, persists an anti-loop receipt, cuts power for the configured dwell, restores power, and performs no second cycle until it has seen the target alive again. A persisted 30-minute cooldown also survives an observer restart.

The database Heartbeats catch failures a process/port check misses. A MySQL Heartbeat asks for SELECT 1 without a password. Success or “access denied” proves that MySQL answered; a refused, timed-out or saturated connection is unhealthy. No MySQL password is stored and no tables are read. A blank probe name becomes the non-root fs_watchdog, which need not be a real account. A PostgreSQL Heartbeat uses pg_isready without a database username or password. With error-log early-warning switched on, the watchdog also tails the database's own error log and raises a critical event the first time it sees a corruption signature. Because all of this runs on the server, it can alert you autonomously over ntfy or a webhook even while your Mac is off; turn that on with Alert me directly from this server in the host's Service Watchdog panel.

To create a check-only account, replace your_database and the example password:

CREATE USER 'frontierstack_health'@'localhost' IDENTIFIED BY 'use-a-unique-random-password';
GRANT SELECT ON `your_database`.* TO 'frontierstack_health'@'localhost';

Save that login in Database Health. Leave auto-repair off until both a manual check and verified backup succeed. If a complete backup or repair needs another privilege, add only the schema-scoped privilege MySQL names; never use *.*. For a monitor connecting from another machine, replace localhost with that one exact private address.

The pinned server's Services row shows a green DB checker: On dot while scheduled table scans or server self-heal are active, Paused when a target exists but neither is running, and an orange Needs login instead of green when a selected MySQL scan has no database password. Database Health and the Heartbeat are complementary: the former checks tables on a slower schedule, while the latter detects outages quickly and can restart MySQL on the server.

When a database isn't down but is slow or stuck, the Why slow/stuck? button on each target in the Database Health pane answers it on demand: it reads the server's live activity (SHOW FULL PROCESSLIST on MySQL, pg_stat_activity on PostgreSQL), groups the running queries by shape so a burst of the same query collapses to one line with a count, and states the verdict — a single query hammered in a burst, a long-runner blocking others, connections waiting on a lock, oversized result sets streaming to clients (the bandwidth cost), or connections near the ceiling. A useful subtlety: if the load is bursty it may read idle between spikes, and the tool says so and tells you to re-run during one, rather than falsely reporting all-clear. Remote targets are probed over SSH, so no database port need be exposed.

11.18.5Database replicas (slaves)

If a pinned server's database copies its data from another server, Database Health watches the copy. Each database has a Replication section, and each MySQL/PostgreSQL row on a pinned server has a Replication menu. Choose Detect automatically (the default, checked hourly), Replica (slave) of another server, or Not a replica. A replica is checked every two minutes: MySQL with SHOW REPLICA STATUS (the login needs REPLICATION CLIENT), PostgreSQL through its WAL receiver and logical subscriptions, and Redis with INFO replication. When a replica stops, can't reach its source, or falls further behind than its limit (300 seconds by default), a Replication alert goes out on your messaging channels. In the sidebar, the replica's database icon becomes a stack with a dotted copy, and turns orange when the replica is out of sync and red when it is broken. Hover over any icon on a server row to see what it is.

You don't have to create the MySQL login yourself. When none is saved and FrontierStack can be root on the server, it creates a read-only account, frontierstack_health, with a random password kept in the Keychain. The latest result appears under the database on the pinned server's page. If a replica stopped with Temporary file write failure, a big ALTER ran out of room in a small /tmp: Fix Temp Space… opens a script that moves InnoDB's temporary files to the data disk and restarts the replica.

11.18.6When this Mac has no internet at all

A Mac that wakes from sleep before Wi-Fi has rejoined, or that has simply lost its connection, would otherwise see every device, server and site fail at once and raise an alert for each. Before every sweep FrontierStack confirms that the internet answers, by checking that well-known hosts respond on TCP 443 and retrying once a few seconds later. If nothing answers, it shows a single No internet connection notification and pauses all device and service checks. It re-checks every 30 seconds and resumes on its own when the connection is back, then removes that notification rather than posting another. Device states are held as they were, so nothing flips to down and back. The pause and resume are written to the alert log.

11.18.7Pausing checks for maintenance

A watchdog that restarts things is exactly what you don't want while you are deliberately taking a service down. Rather than switching protection off — and relying on yourself to remember to switch it back on — use Pause Checks… in the Guardian's header, or the pause button on an individual service row. Choose a window from 15 minutes to a day.

While paused, the Guardian stops health-checking and stops restarting whatever you paused, so your work doesn't fight it. Nothing is disabled and no setting is lost. Keep Alive, Auto Recover, thresholds and intervals are all left exactly as they are, and checking resumes on its own when the window ends — or immediately, if you press Resume Now. A pause never survives its own window, even if you quit and reopen the app.

11.6Case study: the High Sierra Server.app serviceproxy wedge

Old Macs still running macOS Server (Server.app) on High Sierra have a well-known failure: the web front proxy — the launchd job com.apple.serviceproxy, which binds ports 80/443 in front of the real backend — occasionally wedges. It keeps accepting TCP connections but never answers them, so every website behind it goes dark while every process check still looks healthy. There is no fix from Apple; the stack is end-of-life. The Service Watchdog was built with exactly this case in mind.

How to set the watch up. On the server's pane, add a watchdog entry for the service name serviceproxy and — this is the important part — give it an HTTP check with the target http://127.0.0.1/, not a process check. The HTTP probe is the only check that sees the real failure (connections accepted, no answer); on failure the watchdog heals it with launchctl kickstart, which is precisely the manual fix an admin would type. Any HTTP status below 500 counts as alive (a 403/404 from the proxy still proves it is answering); each probe times out after 6 seconds, so a wedged proxy fails the check by timing out. The backend behind the proxy can be watched the same way as server-httpd.

Timings that work well. Check every 30 seconds; restart after 3 consecutive failed checks; try up to 3 restarts; leave the post-restart grace at its default (~20 s, generous enough for Apache on old hardware). That confirms a wedge for ~90 seconds before acting — long enough that one slow response or a momentary load spike never triggers a kickstart, short enough that sites are back about two minutes after a real wedge, hands-off. If the sites are critical, tighten to a 15–20 s interval with 2 failures (≈40 s to confirm) — going tighter than that mostly buys false restarts, because a genuinely slow old box can take a few seconds to answer under load. Leave “Reboot the server if recovery still fails” off unless the machine is truly unattended: a kickstart resolves the wedge in practice, and reboot-as-last-resort on a box like this mostly adds downtime (the anti-boot-loop guard enforces at least 30 minutes between watchdog reboots regardless).

Why not a process check? There is no process named serviceproxy — the job runs as httpd with a special config — and monitors older than v1.6.1 read a healthy proxy as permanently “down” because of that, restarting it over and over and, if reboot escalation was allowed, rebooting a healthy machine on a schedule. From v1.6.1 the monitor asks launchd for the job's real state instead, so a process check now tells the truth; the HTTP check remains the one that catches the actual wedge. If an older server of yours has been rebooting with no visible cause, check its monitor version first — and note that “Why did it reboot?” attributes watchdog-initiated reboots explicitly (with the reason, on v1.6+ monitors; older monitors leave a stamp that is still reported).

One guarantee closes the loop on delivery: an alert is never silent. On top of the messaging gateways, every state-change alert also raises a native macOS notification and push, independent of the gateways — so even with email off and every channel disabled, a service going down still reaches you on the Mac. Database recovery — running mysqlcheck/pg_amcheck and the AI-guided rebuild — is covered in Chapter 5.

11.7Incidents, on-call and status pages

For teams that already run formal incident response, FrontierStack connects to the tools you use rather than replacing them. Each has a live pane reached from its catalog entry: paste a read-only API key and it shows current state.

• Incident management & on-call — PagerDuty (open and high-urgency incidents, who is on-call, services), Opsgenie (open/unacknowledged alerts, with a US/EU region toggle), incident.io and Rootly (active incidents).
• Status pages — Statuspage (unresolved incidents and components that are down) and Better Stack (monitors up / down / paused).

Set up the key in each pane (for example PagerDuty ▸ Integrations ▸ API Access Keys) and the pane confirms the connection. These feed the same monitoring picture, so an open PagerDuty incident or a down status-page component shows alongside your own health.

11.8SaaS live monitors & project-management panes

FrontierStack does not stop at infrastructure. A config-driven SaaS live monitor watches connected cloud accounts and refreshes every few minutes. Each service has a credentials form, live metric tiles, an "Alert me" toggle and a threshold; alarmable services raise a DOWN alert when the metric crosses the line, and the rest are watched for reachability. Live monitors include Stripe (disputes needing response, balance), Shopify and WooCommerce (open orders), Adobe Commerce and Magento — on-premises, on Cloud Infrastructure and as a Cloud Service, all through the same V1 REST orders API (orders awaiting action; the Cloud Service authenticates through Adobe IMS) — Freshdesk and Zammad (open/pending/overdue tickets), SendGrid, Mailgun and Postmark (bounces and blocks), GitHub Copilot (inactive seats), and identity providers Okta and Microsoft Entra ID. Pin a service to make it always-on-Local-Health regardless of its alert setting.

Engineering and compliance services are covered the same way: Sentry (new unresolved issues in the last 24 hours), Grafana and Grafana Cloud (firing alert rules), Honeycomb (triggered triggers), New Relic (active issues), Make (invalid scenarios and incomplete executions), MongoDB Atlas (open alerts), Vanta and Drata (failing compliance tests), CircleCI and Buildkite (failed builds in the last 24 hours), and Netlify and Render (a site or service whose latest deploy failed). Each needs only a read key; the pane says which permission to give it.

Ahrefs (API v3, Lite plan or higher) is watched without spending your allowance: the monitor reads only the free usage endpoint and alerts when this month’s API units pass 90%, and fetches Domain Rating once a day if you give it a domain.

The GMO group’s services have their own monitors. Color Me Shop (カラーミーショップ) reads today’s, this week’s and this month’s sales with an OAuth token from an app you register at developer.shop-pro.jp, feeds the Sales Channels dashboard, and alerts when orders pile up waiting to ship. GMO Trust Login is checked through its SCIM endpoint — give the monitor its own SCIM IDP entry, because generating a credential on one that Entra ID already uses breaks that sync. GMO Aozora Net Bank shows account balances and can alert below a yen floor; with the OAuth client ID, secret and refresh token it renews its 30-day access token itself, and the free sunabar sandbox lets you try it first. GlobalSign needs no key: it watches the Atlas ACME directory, GlobalSign’s status page and a domain’s live certificate.

Microsoft and Veeam on your own network. The Active Directory monitor watches on-premises domain controllers without an agent. It finds them in DNS and checks their Kerberos, LDAP and SMB ports, their sync state, the clock difference from this Mac and their LDAPS certificates. With a plain domain user it also reads each DC’s inbound replication and counts locked-out accounts, over LDAPS or StartTLS only. Microsoft Entra ID alerts when Entra Connect stops syncing, when an app registration’s secrets are about to expire, when a subscription lapses and on Microsoft 365 service interruptions; the same app registration imports users in the Accounts pane. Veeam Backup & Replication, Veeam Backup for Microsoft 365 and the Veeam Service Provider Console show job results and alert on failed jobs (and, for Backup & Replication, repositories under 10% free). A self-signed Veeam certificate is refused, and nothing is sent, until you approve its fingerprint.

The dedicated GitHub pane adds account and repository totals, Actions-related work, PRs, requested reviews, assigned issues, unread notifications, API requests remaining and release-asset download counts from the five most recently updated repositories. Its Installed app permissions section watches the GitHub Apps you name (GitHub won’t let a token list them) and raises a Security alert when one asks for more access than you last reviewed — a new permission, read-only raised to read & write, or a new owner. It is read-only: accepting or declining stays on github.com. Anthropic account/model monitoring and GitHub Copilot seat usage need only their API credentials; no host or port is invented for a cloud-only service.

AppSignal has a dedicated read-only live connection. Enter the application ID and a personal API token; the token is stored in Keychain and FrontierStack uses AppSignal’s GraphQL API to show open error incidents, active uptime alerts, the latest deploy and available last-hour error/latency metrics. The alert threshold combines open incidents and uptime alerts. AppSignal requires its API token in the GraphQL URL, so FrontierStack constructs the URL privately and never writes the authenticated URL to logs. For deeper investigation, add the AppSignal preset in MCP Servers: it opens AppSignal’s hosted MCP through a version-pinned OAuth bridge, without copying the monitoring token into an AI client. Keep AppSignal access read-only until you have reviewed any write-capable tool.

Paessler PRTG has a dedicated live connection for both a self-hosted PRTG core and PRTG Hosted Monitor. Open the PRTG service, enter its HTTPS base URL, and create a Read access scripting key under Setup ▸ Account Settings ▸ API Keys. FrontierStack stores the key in Keychain, reads a bounded sensor-status table, and shows Up, Down, Warning, Paused and other sensor totals. Turn on Alert me when this needs attention to feed Down sensors and sensors without a connected probe into Alerts. The integration never acknowledges alarms or changes devices, sensors or probes. PRTG Network Monitor and Enterprise Monitor run their core service on Windows; use the linked Windows server and vendor installer, or use Hosted Monitor. Remote Windows probes and multi-platform probes extend collection into the rest of the fleet.

NinjaOne, Atera and Site24x7 also have dedicated read-only live connections. NinjaOne uses a Client Credentials application with Monitoring scope to summarise devices, offline endpoints, active alerts and running jobs. Atera uses a targeted API key limited to Agents and Alerts. Site24x7 exchanges a read-scope Zoho refresh token in the account's regional data centre and reads current monitor states. Their secrets remain in Keychain, responses are size-bounded, and none of these connections patches endpoints, runs jobs, closes alerts, starts maintenance or changes vendor configuration.

The same pattern gives project-management tools live token-login panes: Jira (open / unassigned / blocked / in-sprint counts), Linear, monday.com, OpenProject, Plane and Taiga. These are dashboards rather than alert sources, but they put your team's workload in the same window as the servers that run it. Email delivery has its own unified Email Delivery pane that combines transactional-provider metrics with SPF/DKIM/DMARC checks (Chapter 7).

11.9Queue Operations

Queue Operations is a read-only dashboard for RabbitMQ, Kafka, NATS/JetStream, Redpanda, AWS SQS, Azure Service Bus, Google Pub/Sub, Celery/Flower, Redis Streams, Apache Pulsar and Apache RocketMQ. Add one monitor for each broker, namespace or cloud account. FrontierStack shows queue or consumer-group names, ready and in-flight counts, consumer counts, lag, dead-letter counts and oldest-message age when the provider exposes them. It never reads or stores message bodies.

Client and administration ports are kept separate. For example, RabbitMQ clients use AMQP on 5672 while its management API normally uses 15672; NATS clients use 4222 while monitoring normally uses 8222; Pulsar clients use 6650 while its HTTP admin API uses 8080. Keep monitoring endpoints private. Use TLS and a monitoring-only account whenever the dashboard is not reached through the on-host FrontierStack monitor.

For a linked server, select its installed monitor and FrontierStack asks the credential-free helper for bounded summaries over the signed connection. Cloud providers use their normal local CLI profiles or a scoped credential saved in Keychain. Critical thresholds join the Alerts sweep, the same sanitised summaries are available in the iPhone/iPad app, and the AI Administrator can answer queue-health questions through the read-only get_server_health tool. The dashboard has no purge, delete, publish, acknowledge or replay action.

11.10UPS monitoring and SNMP devices

Open Power Control and use its UPS section for battery-backup monitoring. FrontierStack reads USB UPS devices directly from macOS power sources, queries NUT with upsc, and queries APC devices through apcupsd with apcaccess. These paths cover APC, Eaton, CyberPower and Vertiv, including Liebert models, as well as other devices supported by macOS or NUT. NUT can also expose serial and network UPS devices.

NUT is free, open-source software and requires no subscription. Status reads need no API key, so the NUT service pane intentionally has no API-credential or Plan & Cost card. A dedicated NUT command login is only needed when you opt into reviewed UPS controls; paid support or hosted dashboards are separate products.

The same pane supports switchable Raritan / Legrand Xerus PDUs. Add the private HTTPS address, the number printed on the outlet, and a dedicated account limited to outlet control; its password stays in Keychain. FrontierStack reads the exact outlet state and genuine active-power sensor when present, supports On and Off, and uses Xerus's atomic cyclePowerState method for Cycle. It refuses public addresses, embedded credentials, redirects and arbitrary JSON-RPC methods. Off and Cycle show a fresh destructive-action confirmation, and the AI Harness receives the same confirmation gate.

Aviosys IP Power units (9258 family, 9820, 9850, 9858, 9255) ship with Windows-only utilities, but every model answers the same local web commands those utilities send, so FrontierStack drives them from the Mac with nothing to install. Add one as Aviosys IP Power with its address, the outlet number printed on it and its login; the password stays in Keychain and is sent in a header, never in a URL. You get status, On, Off and the unit's own timed power cycle, which restores the outlet even if it powers the network path back to this Mac. The original 9258 firmware numbers its outlets from P60; FrontierStack recognises it from the unit's reply and switches the right outlet. Change the factory login (admin / 12345678) — the unit's web commands accept whoever knows it.

Each row shows the readings supplied by that model: charge, estimated battery runtime, load, real power in watts, input and output voltage, battery voltage, frequency and temperature. A watt value reported by the UPS is shown directly. If the device reports rated real power and load percentage but not current watts, FrontierStack shows a value marked estimated. It does not treat volt-amperes as watts.

Turn on Alerts, then set the charge, runtime and load thresholds. Utility-power loss, low runtime, low charge, high load, a battery-service warning, forced shutdown and communication loss are tracked separately. This means a falling-runtime alert can arrive after the first on-battery alert instead of being hidden by the existing outage. A disconnected unit remains visible as Comms lost. Pin a UPS to keep its overall health in Local Health and on the Places map. The NUT and apcupsd install buttons add those optional tools when needed.

Battery shutdown rules can now carry out the whole orderly sequence: wait until the UPS has been on battery for a chosen time or falls below a charge threshold, stop named managed services and VM services in order, verify each stop, and shut down the protected host last. A failed verification halts the ladder with the host still running; returning utility power cancels the unfinished stages. If PowerChute Network Shutdown should be running on the host, FrontierStack can detect common service names or accept a manual declaration, alert separately when it disappears, and either stop or continue the configured fallback.

A scan waits for NUT and apcupsd to answer. While it is running, Stop abandons the wait, keeps readings already found and pauses automatic refresh until you press Refresh.

A server or ordinary pinned device has two separate setup dialogs in its Power section. Add power control… is for the switchable path: a SwitchBot Plug, a configured Home Assistant, Shelly, Tasmota, private HTTP, Raritan PDU, Anker SOLIX, TP-Link Tapo / Kasa, Meross or Smart Life / Tuya outlet, a command-based PDU, or a power-capable Remote KVM. The dialog also links directly to the SwitchBot, Home Assistant, Anker SOLIX and TP-Link Kasa/Tapo provider panes; providers configured natively in Power Control open there. Add backup power… is for a pinned monitored UPS, Mini DC UPS, or Unmanaged / dumb UPS. Treat a PDU as a controller: assign every physical outlet its printed number in Power Control, then select that named, numbered outlet for the server or device. A pinned PDU is also offered for command-based control, whose exact outlet number and commands are entered directly in the server's Power section. SwitchBot remotes and sensors are excluded. A protected device may have both a smart plug and a dumb UPS. A device marked as a Smart Plug is itself a power controller, so its pane omits the entire Power section rather than offering another controller. A linked monitored UPS contributes real battery, runtime and warning status. Unmanaged choices take maker and model notes and remain inventory-only, so FrontierStack and the AI Harness expose no invented readings or controls. TREEDIX is available for its 5 V Raspberry Pi UPS controller as a Mini DC UPS.

Smart-plug brands. TP-Link Tapo and Kasa plugs and power strips are added in Power Control as TP-Link Tapo / Kasa and switched on the local network: enter the plug's address and the TP-Link ID it was set up with, then choose Find outlets — each strip outlet is its own entry. TESSAN, BN-LINK, Edison Smart, OHMAX and MEAPUXIN plugs and strips are white-label Tuya devices: connect the Smart Life pane, then add a Smart Life / Tuya outlet and pick the device and outlet (plugs set up in a brand's own app must first be re-added in Smart Life). A Smart Life power-cycle also starts the outlet's own countdown, so it powers back on even if it feeds the router. Onvis and Matter Amazon Basics plugs are reached by sharing them to Home Assistant's Matter integration and adding the Home Assistant switch. Shelly plugs of every generation (Gen 1 relay API and Gen 2/3/4 RPC) switch directly.

Meross plugs and strips. Add an outlet of type Meross with the plug's private IP address. Meross signs every local request with the account's device key, so enter the Meross account email and password and choose Get key from Meross account once: FrontierStack signs in to the Meross cloud, keeps the key in Keychain and signs out again, and the password (and any verification code) is never saved. You can paste a key you already have instead. Then choose Find outlets. MSS110, MSS210, MSS305, MSS310 and MSS315 plugs and MSS420, MSS425 and MSS620 strips are switched on the local network — on, off and power-cycle, watts on the metering MSS305, MSS310 and MSS315, and one outlet or a whole strip (strip watts are for the whole strip). Device Discovery recognises Meross plugs; the plug's device pane has Add to Power Control.

Shelly relays, plugs, strips and meters. Add an outlet of type Shelly with the device's private IP address. Every generation works: Gen 1 (Shelly 1/1PM/2.5, Plug S, EM, 3EM) and Gen 2, 3 and 4 (Plus, Pro, Mini, Plug S Gen3, Plug US, Power Strip, Pro 3EM). If the device has a password, enter it — Gen 1 also takes its login name, Gen 2+ always use admin — and it stays in Keychain; Gen 2+ authenticate with HTTP Digest, so the password itself never crosses the network. Find channels reads the model, generation, firmware, each channel's name and whether Matter is on; channels count from 0. A meter without a switch (Pro 3EM, PM Mini) is watched for watts but not switched. Power-cycles arm the device's own flip-back timer with the “off”, so power returns even when the outlet feeds the router this Mac reaches it through. A Gen 4 Shelly can be in Apple Home through Matter at the same time; FrontierStack keeps using the local interface and both stay in step. A Shelly with no password gets a warning. Device Discovery recognises Shellys by MAC block, _shelly._tcp or name and offers Add to Power Control.

Battery alerts. SwitchBot locks, keypads and sensors, Nuki locks with their keypads and door sensors, Seam locks and Zigbee devices report their battery. Each one shows the level and alerts twice: when the battery gets low (20 % by default) and again when it is almost dead (5 %). Set both levels in Alerts ▸ Device Batteries, which also lists every level FrontierStack has read. A pinned SwitchBot Hub Mini, Hub or Hub Plus alerts when it goes offline, checked through a device that connects through it.

Outlet reachability. In Power Control, edit an outlet and turn on Alert when this outlet is unreachable. FrontierStack checks the outlet every 2 minutes with a read-only probe and alerts after 2 missed checks in a row. It does this only while this Mac is on the same physical network, identified by the router's hardware address, where the outlet last answered. A laptop away from home, or on a network that reuses the same addresses, is never told its plugs are dead. For plugs this Mac can't reach, use the Shelly Cloud pane (IoT). Paste the authorization cloud key and server address from the Shelly app, add each device ID, and turn on the bell for offline alerts. When a Shelly is watched both ways, the local check wins while you are on its network.

11.18.8Link a device to a numbered PDU outlet

1. In Power Control, add one outlet record for each physical PDU socket you want to use. Give it a useful name and enter the number printed beside that socket; for example, Rack server · Outlet 4.
2. Open the pinned server or device, expand Power, and choose Add power control…. Configured outlets appear by name and physical number. Select the exact outlet feeding that device.
3. If no configured outlet is available, choose Add a smart plug or PDU in Power Control…. FrontierStack opens the setup pane so you can supply the controller's private IP address or URL and add its outlets; it does not create an empty PDU link.
4. If a PDU with an address is already pinned but has no native FrontierStack outlet integration, select it under Command-based PDU / powerboard. Enter that device's outlet number plus the reviewed On and Off commands in the server's Power section. The commands are configured there, not in the selection menu.
5. Choose Add backup power… separately for a UPS. A pinned UPS appears there because it describes the device's backup supply; it is not the switchable PDU outlet.

Do not link the server to the PDU as a whole when individual outlets can be identified. FrontierStack includes the outlet number in the row label and in destructive-action confirmations, so the operator can verify the physical socket before an Off or Cycle action.

11.11Remote KVM platforms and power

The Remote KVM pane records multiple PiKVM, JetKVM, TinyPilot, NanoKVM, Raritan Dominion, ATEN, Vertiv Avocent, Lantronix Spider/SpiderDuo, Adder, GL.iNet, AWERAY and generic KVM-over-IP appliances. Each record keeps its exact model or firmware, can be pinned, and opens its private browser console; AWERAY records launch the selected companion app.

Power control is exposed only when a reviewed path is configured. PiKVM uses its documented local ATX API and checks that ATX is enabled, idle and in the expected current state before sending a hard action. JetKVM uses the shared MQTT connection with one exact base topic: it reads the retained ATX state before a short or long press, while its DC extension uses explicit ON and OFF messages; commands use QoS 1 and are never retained. TinyPilot and NanoKVM power hardware, and the enterprise vendors' PDU integrations, differ by model and firmware, so those records remain console/inventory-only unless an explicit private endpoint is supplied. A Raritan PDU associated with a Dominion target should use the reviewed Xerus entry in Power Control. KVM Off and Cycle require fresh confirmation in both the pane and AI Harness.

11.12Home Assistant entity monitors (Beta)

If Home Assistant already runs your building, FrontierStack can watch what it sees. Open the Home Assistant service pane, enter its URL and a long-lived access token, and the Entity Monitors section appears. Press Load Entities from HA, pick an entity, and give it a rule: numeric (is above / is below) for a temperature, humidity or power reading, or text (equals / is not) for a binary sensor's on/off, wet/dry or home/not_home state. Each watch becomes an ordinary Alert, checked on the usual sweep.

This is the right home for the sensors that can never be pinned as devices. A Zigbee leak sensor under the server-room floor, a Z-Wave door contact on the rack, a Bluetooth thermometer — none of them has an IP address, so none can be pinged. Home Assistant already speaks their protocols, so their readings arrive through it. Home Assistant 2026.9's shared Modbus bus access extends the same idea to solar inverters, PDUs and energy meters: HA polls the bus and FrontierStack reads the named values, rather than the two products fighting over one serial connection.

Two deliberate behaviours. An entity reporting unavailable or unknown raises its alert instead of being skipped — a leak sensor that has gone quiet is exactly the one you need to hear about. And the monitor is read-only by construction: it never calls Home Assistant's service API, so it can read your building but cannot switch a light, unlock a door or open a valve. Switching stays in Power Control, where every action is explicit and confirmed.

Unavailable devices. Rather than watching entities one by one, turn on Alert when any device becomes unavailable. Once a minute FrontierStack reads every entity in the domains you choose and alerts on any that Home Assistant marks unavailable. An entity that is merely unknown (no value yet) doesn't count, and nothing is reported offline while Home Assistant itself is unreachable. Alert after (default 10 minutes) keeps short blips and HA restarts quiet. If many entities fail together (more than 30% or more than 50), you get one alert, "Home Assistant: N devices unavailable", instead of dozens, because the cause is usually the same: HA restarting, a Zigbee/Z-Wave bridge down, or the network. Use Exclude (or the button on each row) for retired devices you haven't deleted from HA.

The same connection powers Import from Home Assistant in Device Discovery: the network devices HA tracks, listed with the names you gave them there, pinned in one click — individually or all at once. Imported devices behave exactly like scan results, with health checks, Places and licence limits applying as normal, and anything already pinned filtered out. Both features are marked Beta while Home Assistant 2026.9's device-registry API settles; importing full registry detail (areas, models) waits for that.

For anything else that speaks SNMP — managed switches, printers, network UPSes, NAS units — the SNMP / OIDs section of a device's detail pane queries it directly. Choose legacy SNMPv2c or SNMPv3; the recommended v3 profile uses SHA authentication with AES privacy. Passphrases remain in Keychain and are passed to net-snmp through an owner-only temporary configuration file, never command arguments. Pick a built-in template (System, Host Resources, Interfaces, Printer RFC 3805, UPS RFC 1628, Synology, QNAP, APC PowerNet) and press Query for a label-and-value readout, or fetch a single custom OID by hand. You can also watch an OID: set a comparison and threshold and FrontierStack polls it and raises an alert when the condition is met.

11.13Eve accessories

Eve (Eve Systems, formerly Elgato Eve) makes the Eve Energy smart plug and sensors such as Eve Room, Weather, Degree, Door & Window, Motion and Water Guard. Eve has no cloud or network API, and macOS keeps HomeKit to Apple's own apps, so FrontierStack reads Eve accessories through the same Home Assistant connection. Add them to Home Assistant with the HomeKit Device integration, or share them from the Eve or Home app as Matter devices; they stay in Apple Home either way. The Eve service pane then lists every Home Assistant device made by Eve, with on/off and watts, temperature, humidity, air quality, door, motion, water and battery.

Add to Power Control turns an Eve Energy into a Home Assistant outlet. You can switch it (with confirmation), see its watts, and link it to the server it powers for power-cycle recovery. The alert-rule menu on each accessory adds one-click rules, such as above 35 °C, water detected or battery below 15 %, as entity monitors. The bell raises an alert when the accessory drops off the network or its battery runs low. Reading Eve devices uses Home Assistant's template API, which needs a token from an administrator account.

11.14Zigbee and Z-Wave devices

Offline alerts. On the Zigbee pane, click the bell on a device (or Alert on All) to be alerted when it drops off the mesh. With Zigbee2MQTT this needs availability turned on in Zigbee2MQTT's settings. Without it the pane shows an orange note, because FrontierStack can't tell a quiet device from a dead one, and it never guesses. deCONZ reports reachability for lights. Watched devices are checked every minute, even with the pane closed.

The Zigbee pane also watches Z-Wave nodes driven by Z-Wave JS UI. In Z-Wave JS UI, enable the MQTT gateway (Settings ▸ MQTT) on the same broker Zigbee2MQTT uses and keep Retain on. Then turn on Z-Wave JS UI via MQTT in the pane's Z-Wave section and enter the same prefix (default zwave). Each node shows as alive, awake, asleep or dead; ring its bell to be alerted when it goes dead. A sleeping battery node is normal and counts as up. If the gateway is offline, has not reported, or the last read is more than five minutes old, every node shows as unknown and no alert fires — FrontierStack never reports an unknown state as offline.

The same SNMP profile is used by Device Discovery, Network Path and PoE monitoring. PoE writes use SNMPv3 when the configured user has write permission; SNMPv2c keeps its separate per-switch write community. The AI Harness can inspect the non-secret profile, read numeric OIDs from pinned devices and list PoE state. Its write tool has no arbitrary OID operation: it is limited to reviewed PoE on/off/cycle actions, needs changes enabled and pauses for a fresh visible manual confirmation.

SMLIGHT coordinators. Network Zigbee/Thread coordinators from SMLIGHT (SLZB-06, SLZB-07, the SLZB-MR series) run SLZB-OS on an ESP32 and can't host FrontierStack's monitor, so the Zigbee pane reads them over the LAN instead: add the coordinator's IP address or .local name. Every two minutes it reads the model, firmware, radio role, connection, temperatures and uptime, and alerts when the coordinator stops answering, when it runs hot, or — the failure that matters most — when no Zigbee2MQTT, ZHA or Thread border router is connected to its radio socket, the state in which the box looks fine but every sensor in the house is about to go unavailable. SLZB-OS has no HTTPS, so it is only read on private addresses.

11.15Music Assistant

Music Assistant is the Home Assistant team's music server: streaming services and local files in one library, played on AirPlay, Google Cast, Sonos, Squeezebox, Snapcast and DLNA speakers. When it fails, nothing crashes — a streaming login expires, an upgrade breaks a speaker integration, a speaker drops off the network — and nobody notices until the music doesn't start. Open the Music Assistant service pane (Home Automation), enter the server URL (usually port 8095) and a long-lived access token from Music Assistant ▸ Settings ▸ Profile.

On every Alerts sweep the monitor checks three things: that the server answers and accepts the token; that no music or player provider has failed to load or needs signing in again; and that the speakers you switched to Alert when offline are still available. Speakers are opt-in because phones and laptops come and go all day. The monitor only sends list commands, so it can never start playback, change volume or edit settings.

Service Scan and Discover Services recognise a Music Assistant server on port 8095 from its /info response and show its version, rather than guessing from the port alone.

11.16Disk, RAID and SMART health

Drives fail with warning if you are listening for it. The Disk Health pane (a built-in monitoring tool) combines several layers. Volume space and Time Machine status are always shown. With smartmontools installed (a one-click Install button), each physical drive is enriched with deep SMART attributes — health, temperature, power-on hours, reallocated and pending sectors, and predicted-failure flags — visible per drive with a Check button and a Details sheet for the raw report. A drive trips an alert on a SMART failure or on concerning attributes (reallocated/pending sectors or a predict-fail), not only on outright death.

Software AppleRAID sets (mirror, stripe, concat) appear in their own section with level, status and per-member state; an alert fires when a set is degraded or a member drops offline. Under Alerts & Thresholds, switch on disk alerts and they route to your channels exactly like every other monitor.

Taken together, these monitors give you one board to glance at, one pane to tune, and one set of channels to reach you on — and an AI Administrator (Chapter 13) that reads the same health and sends the same notifications on your behalf. The host monitors that feed fleet-wide CPU, memory and GPU metrics into this picture are covered in Chapter 8.

11.17Packages & containers that misbehave

Package managers install things that then run on their own — a Homebrew service, a global npm tool, a pip-installed server, a Docker container — and nothing tells you when one of them starts eating a core or keeps falling over. FrontierStack watches them on this Mac every 30 seconds. It works out which package each busy process belongs to from where its program lives (Homebrew's Cellar, node_modules, site-packages, gems, ~/.cargo/bin), reads docker stats and each container's last exit, checks macOS crash reports and brew services, and flags:

• Too much CPU or memory — over your thresholds (CPU counts like Activity Monitor, 100% = one core, summed over all of a package's processes), or a container at 90% of its memory limit;
• Crashes recorded in the last 24 hours, and containers killed for running out of memory or exiting with an error;
• Restart loops, failing health checks, and brew services stuck in an error state.

Problems appear in a Needs Attention section at the top of the Docker, Homebrew, Node.js, Python, Ruby and Rust panes, with a badge on the package's own row, and in the Server Activity window on This Mac's Overview, where the tile also tints. The Activity window's Processes tab names the package each process belongs to, and its Docker tab flags troubled containers on every server, not just this Mac. Ignore stops watching one package.

To be told rather than having to look, turn on Alerts ▸ Packages & Containers. Choose the CPU and memory thresholds, how many minutes high use must last before it pages (a build pegging a core for a minute is normal), and whether resource use, failures or both should alert. Alerts then travel through your channels, contacts and escalation rules like any other, appear in Local Health under Packages & Containers, and badge the pane that owns the package.

11.18The Server Activity window

When you want to watch every box at once rather than configure one, open Monitors ▸ Server Activity Window (⇧⌘A, or the gauge button in the toolbar). It is a separate window holding only your pinned servers and the pinned devices that can report live state — routers and firewalls, UPS units, PoE switches, mining rigs, KVM consoles and smart plugs — and none of the rest of the app. Each tile shows the metrics you asked for with a sparkline and a status dot, and turns orange or red when something needs attention.

Layouts decide what is on the board: the built-ins are All Servers, Macs, a compact Cluster grid sorted by CPU, Mining Rigs & GPUs, Network & Power, AI Agents, Durable Objects and Everything. Customise a copy or make your own — kinds of target, a server filter, individual members, the metrics and their order, tile size, grouping, sort and the sampling interval. Readings come from the FrontierStack monitor where one is installed and otherwise from one SSH round-trip; sampling runs only while the window is open.

The window is meant to be driven from the keyboard. Arrow keys move the selection through the board or the list, with up and down stepping a whole row of tiles; ⌘↑ and ⌘↓ jump to the first or last target; ⇧⌘0 returns to the board; and esc steps back out — detail, then board, then closing the window. ⌘1 to ⌘9 switch straight to a layout, ⌥⌘← and ⌥⌘→ cycle through them, ⌘E edits the current one and ⌘⌫ hides the selected target from it. ⌘F focuses the filter, ⌘R samples now, ⌥⇥ moves between a server’s tabs, ⌘T opens a terminal window, ⇧⌘O opens the selection in the main window, ⇧⌘K runs one command on every server, and ⌃⌘S shows or hides the list. Press ⌘/ for the full map at any time.

Run on All… runs one command across every server in the layout. Devices have their own detail: gateways and IDS alerts for a router, battery and voltages for a UPS, per-port control for a PoE switch, hashrate and shares for a rig, console and power for a KVM, state and watts for a plug.

A server’s Web Traffic tab, next to Overview, is the Web Traffic pane for that one server: FrontierStack finds its Apache or Nginx access logs over SSH and shows requests, visitors, error rate, top pages, referrers and bots for the range you pick. The same view is in the main window — choose the server in the Web Traffic pane’s Host picker, or press Traffic on the Apache or Nginx row of a server’s Services. Root-only logs need the server’s password saved under SSH & root access.

The AI Agents layout puts every agent runtime you've set up that has a status API on the board — T3 Code, OpenClaw, Hermes, Paperclip, Octop, LangGraph and others — with agents working, and tokens and cost today. T3 Code, once connected read-only, adds the threads waiting for you (the tile turns orange until they're answered) and the CPU and memory of its whole process tree, as a share of this Mac. Its detail lists the threads waiting, working and failed, usage by model and the busiest agent processes. T3 Code workers that the fleet check finds on servers appear as presence-only tiles.

The Durable Objects layout shows celld — the Deno team's self-hosted runtime for Cloudflare Workers and Durable Objects — with one tile per fleet node added on the celld pane and per celld dev server running on this Mac. Each tile shows resident and owned cells, CPU and memory; it turns orange when the node signals it needs capacity (memory pressure, shedding or cells waiting for residency) and red when its operator API stops answering or its health route reports it draining. The detail lists WebSockets, the restore backlog, the deployment version, cell phases and the resident cells, with Open Worker, Refresh and, for a server on this Mac, Stop.

A connected Proxmox VE cluster adds one tile per node to the All Servers, Cluster and Everything layouts, with CPU, memory, disk and uptime sparklines and a count of running VMs and containers. Its detail lists every guest with start, shut-down, reboot, stop and reset controls, the node's storage and the cluster's quorum, and offers Boot an ISO… into the Boot Media pane. Connecting a cluster is covered in Chapter 8.

The AWS layout (and All Servers) needs nothing set up in FrontierStack: if the aws CLI has credentials, the window scans every profile in ~/.aws and every Region the account has enabled, and shows each EC2 and Lightsail instance as a tile with CloudWatch CPU and network (5-minute points) and its uptime. Two profiles for the same account are scanned once, GovCloud and China accounts work, and a stopped instance is shown neutral rather than as an outage. The detail page starts, stops and reboots it (stop and reboot ask first), opens it in the AWS console, and either opens the server it already is or offers Add to Servers — filled in with its address, the login Lightsail set up (or the usual one for the image), and the key pair’s file in ~/.ssh when it is there.

For an instance that is one of your servers, the Disk growth box (also on the server’s own page) handles a filling disk. When the root disk passes the Alerts threshold, FrontierStack works out the grow — the new size, by a step you set, never beyond your cap, and the extra monthly cost — and either asks you (the default) or, if you chose Grow automatically for that server, goes ahead: an optional EBS snapshot, modify-volume, then growpart and resize2fs or xfs_growfs over SSH, with no reboot. Grow Disk… does the same on demand. EBS volumes can only grow and AWS allows one change per volume every six hours, so the cap matters; LVM, Windows and Lightsail get instructions rather than automation.

Three controls sit in the window's own titlebar. Appearance makes this window alone dark or light — useful for a monitoring board you want dark while the rest of the app stays light — cycling follow-the-app, dark, light. Tab layout moves a selected server's tabs between a strip above the content and a vertical list beside it, which suits a tall window and shows every tab at once. Keyboard opens the shortcut list. Both toggles are also in the layout editor, and on ⇧⌘D and ⇧⌘L.