Monitoring & Alerts

A control panel that only lets you change things is half a tool. The other half is knowing when something has gone wrong — ideally before it becomes an outage. FrontierStack runs a continuous monitor sweep across everything you have told it to watch and turns the first sign of trouble into a message on whatever channel you choose. This chapter covers the Local Health board that summarises it all, the Alerts pane that raises and delivers warnings, the messaging gateways that carry them, and the specialised monitors for cloud services, project tools, power and storage.

11.1The Local Health board

Open Overview ▸ Local Health for the single screen that answers "is anything broken right now?" Every monitored service, device and subsystem appears as a row marked UP or DOWN, grouped by area — web and database services, the fleet, sites and certificates, disks, UPS and SNMP devices, connected cloud accounts. At the top sits a one-line summary in the form X up / Y down, so you can take in fleet-wide status in a glance without reading every row.

The board is the same data the AI Administrator reads through its get_server_health tool: ask it "what's down?" and it reports exactly what this screen shows, then offers to investigate before suggesting a fix (see Chapter 13). The sweep runs about every sixty seconds; each DOWN item is also attributed to the pane that owns it, which is why a red count capsule can appear on a sidebar row and on the Dock icon — you are told where the trouble is, not just that there is some.

11.2The Alerts pane: what is watched

The Local Health board shows status; the Alerts pane (Overview ▸ Alerts) decides what is worth a message and sends it. Switch on "Monitor critical services and alert me" and the sweep starts raising alerts. What it watches:

• Site & service health — HTTP/loopback checks, TCP ports, and deeper protocol probes (an SMTP/IMAP login, a database SELECT 1, an LDAP bind) so a service that is listening but actually broken still trips. The MySQL probe additionally reads the connection pool (current use vs max_connections and the "too many connections" refusal counter): it alerts when the pool passes 90% or when clients were refused between sweeps, and every sweep is charted as Connection Health History in the Database Health pane — pool % over time with red lines marking refusal bursts and outages.
• Linked servers — every fleet host's SSH endpoint is reachability-checked each sweep (a bare TCP connect, no login attempt), so a server that is completely off raises a plain "Server down" alert — and a "Server recovered" when it returns. This is separate from the SSH cool-down notice, which fires only when a host is up but rate-limiting connections.
• Certificates & domains — TLS expiry across the whole fleet, plus registrar / DNS / SSL checks that warn you days ahead rather than on the morning a domain lapses (Chapter 7).
• Disks filling up — a volume crossing its free-space threshold, surfaced from the Disk Health monitor below.
• Security exposure — a new open port or unexpected listener, a VPN tunnel drop, or a critical SSH-hardening failure while SSH is reachable from the network.
• Failed payments & billing — connected paid services (OpenAI, Anthropic, Cloudflare, Google Cloud) returning insufficient-quota, low-balance or auth errors — the symptom of a lapsed card.
• Connected cloud services — live monitors for Stripe, Shopify, Freshdesk, SendGrid and more, firing when a metric you chose crosses its threshold.

Most checks are threshold-based, so you decide what "trouble" means: alert when a certificate is within N days of expiry, when open tickets pass a number, when a disk crosses a percentage. Conservative thresholds turn a post-mortem into a heads-up. The pane also offers "Alert on intrusions" — new fail2ban/CrowdSec bans, high-severity Suricata signatures and notable Wazuh alerts (level ≥ 7) are messaged as they happen, and existing history is never re-alerted.

11.3Messaging gateways

An alert is only useful if it reaches you. The Alerts pane lets you enable several gateways at once and list multiple recipients per channel, so the right people are reached on the app they already have open. Paste a bot token or webhook URL, send a test, and you are live. The AI Administrator can enumerate the enabled gateways with get_messaging_channels and send through them with send_notification (Chapter 13).

11.11.1Paging the on-call (PagerDuty, Opsgenie, urgent ntfy)

The message gateways above are fire-and-forget text. The paging channels are different in kind: they are stateful. When a monitored item goes down, FrontierStack triggers an incident keyed to that item — PagerDuty (an Events API v2 routing key) or Opsgenie (a GenieKey) then runs your escalation policy, pushing, texting or phoning whoever is on call until someone acknowledges. When the item recovers, FrontierStack resolves the same incident automatically. Because the incident is keyed per item, a flapping service updates one incident rather than paging the rotation over and over, and nobody is woken for an outage that has already ended. Without an on-call service, the ntfy gateway's Page on down toggle sends "down" alerts at ntfy's maximum priority — a louder, repeating tone that overrides many phones' quiet settings — while recoveries and reports stay at normal priority. Each paging channel has a Send Test Page button that triggers a real incident and auto-resolves it about ten seconds later, proving the whole escalation path end to end.

Tokens and webhooks live in the macOS Keychain and never leave your Mac. You can also send a scheduled AI Evaluation Report: the AI Administrator composes a summary from your live monitored health (running read-only diagnostics) and emails it daily or weekly, or pushes it to all enabled channels — with an optional fuller report whenever two or more critical items are down at once.

11.4Delivery errors and how to fix them

A monitor is worthless if its alerts silently fail to send. FrontierStack verifies every send: when a gateway fails, it is recorded in the Delivery Errors section of the Alerts pane with plain-English fix-it guidance, the raw error underneath, and a Fix button that scrolls straight to the misconfigured channel's settings. A red badge appears on the sidebar's Alerts row and on the Dock icon so you actually notice, and it stays until you clear it. The AI Administrator reads the same list through get_alert_errors, so you can simply ask "why aren't my alerts arriving?"

The common causes are mundane and quick to fix:

11.5The on-host Service Watchdog & database health

The monitors above run on your Mac. For a critical server you also want a watchdog that lives on the box itself, so it keeps working when your Mac is asleep, offline, or simply not the machine that failed. FrontierStack's Service Watchdog is deployed with the host monitor (Chapter 8): it health-checks a service on a short interval, restarts it when it fails, and — as a last resort, and only when you have allowed it — reboots the machine, with a minimum-interval guard so a still-broken service can't cause a boot loop. Each watch has a check type: a process check, an HTTP or TCP probe, or a purpose-built MySQL or PostgreSQL check.

The database Heartbeats catch failures a process/port check misses. A MySQL Heartbeat asks for SELECT 1 without a password. Success or “access denied” proves that MySQL answered; a refused, timed-out or saturated connection is unhealthy. No MySQL password is stored and no tables are read. A blank probe name becomes the non-root fs_watchdog, which need not be a real account. A PostgreSQL Heartbeat uses pg_isready without a database username or password. With error-log early-warning switched on, the watchdog also tails the database's own error log and raises a critical event the first time it sees a corruption signature. Because all of this runs on the server, it can alert you autonomously over ntfy or a webhook even while your Mac is off; turn that on with Alert me directly from this server in the host's Service Watchdog panel.

To create a check-only account, replace your_database and the example password:

CREATE USER 'frontierstack_health'@'localhost' IDENTIFIED BY 'use-a-unique-random-password';
GRANT SELECT ON `your_database`.* TO 'frontierstack_health'@'localhost';

Save that login in Database Health. Leave auto-repair off until both a manual check and verified backup succeed. If a complete backup or repair needs another privilege, add only the schema-scoped privilege MySQL names; never use *.*. For a monitor connecting from another machine, replace localhost with that one exact private address.

The pinned server's Services row shows a green DB checker: On dot while scheduled table scans or server self-heal are active, Paused when a target exists but neither is running, and an orange Needs login instead of green when a selected MySQL scan has no database password. Database Health and the Heartbeat are complementary: the former checks tables on a slower schedule, while the latter detects outages quickly and can restart MySQL on the server.

When a database isn't down but is slow or stuck, the Why slow/stuck? button on each target in the Database Health pane answers it on demand: it reads the server's live activity (SHOW FULL PROCESSLIST on MySQL, pg_stat_activity on PostgreSQL), groups the running queries by shape so a burst of the same query collapses to one line with a count, and states the verdict — a single query hammered in a burst, a long-runner blocking others, connections waiting on a lock, oversized result sets streaming to clients (the bandwidth cost), or connections near the ceiling. A useful subtlety: if the load is bursty it may read idle between spikes, and the tool says so and tells you to re-run during one, rather than falsely reporting all-clear. Remote targets are probed over SSH, so no database port need be exposed.

11.11.2Pausing checks for maintenance

A watchdog that restarts things is exactly what you don't want while you are deliberately taking a service down. Rather than switching protection off — and relying on yourself to remember to switch it back on — use Pause Checks… in the Guardian's header, or the pause button on an individual service row. Choose a window from 15 minutes to a day.

While paused, the Guardian stops health-checking and stops restarting whatever you paused, so your work doesn't fight it. Nothing is disabled and no setting is lost. Keep Alive, Auto Recover, thresholds and intervals are all left exactly as they are, and checking resumes on its own when the window ends — or immediately, if you press Resume Now. A pause never survives its own window, even if you quit and reopen the app.

11.6Case study: the High Sierra Server.app serviceproxy wedge

Old Macs still running macOS Server (Server.app) on High Sierra have a well-known failure: the web front proxy — the launchd job com.apple.serviceproxy, which binds ports 80/443 in front of the real backend — occasionally wedges. It keeps accepting TCP connections but never answers them, so every website behind it goes dark while every process check still looks healthy. There is no fix from Apple; the stack is end-of-life. The Service Watchdog was built with exactly this case in mind.

How to set the watch up. On the server's pane, add a watchdog entry for the service name serviceproxy and — this is the important part — give it an HTTP check with the target http://127.0.0.1/, not a process check. The HTTP probe is the only check that sees the real failure (connections accepted, no answer); on failure the watchdog heals it with launchctl kickstart, which is precisely the manual fix an admin would type. Any HTTP status below 500 counts as alive (a 403/404 from the proxy still proves it is answering); each probe times out after 6 seconds, so a wedged proxy fails the check by timing out. The backend behind the proxy can be watched the same way as server-httpd.

Timings that work well. Check every 30 seconds; restart after 3 consecutive failed checks; try up to 3 restarts; leave the post-restart grace at its default (~20 s, generous enough for Apache on old hardware). That confirms a wedge for ~90 seconds before acting — long enough that one slow response or a momentary load spike never triggers a kickstart, short enough that sites are back about two minutes after a real wedge, hands-off. If the sites are critical, tighten to a 15–20 s interval with 2 failures (≈40 s to confirm) — going tighter than that mostly buys false restarts, because a genuinely slow old box can take a few seconds to answer under load. Leave “Reboot the server if recovery still fails” off unless the machine is truly unattended: a kickstart resolves the wedge in practice, and reboot-as-last-resort on a box like this mostly adds downtime (the anti-boot-loop guard enforces at least 30 minutes between watchdog reboots regardless).

Why not a process check? There is no process named serviceproxy — the job runs as httpd with a special config — and monitors older than v1.6.1 read a healthy proxy as permanently “down” because of that, restarting it over and over and, if reboot escalation was allowed, rebooting a healthy machine on a schedule. From v1.6.1 the monitor asks launchd for the job's real state instead, so a process check now tells the truth; the HTTP check remains the one that catches the actual wedge. If an older server of yours has been rebooting with no visible cause, check its monitor version first — and note that “Why did it reboot?” attributes watchdog-initiated reboots explicitly (with the reason, on v1.6+ monitors; older monitors leave a stamp that is still reported).

One guarantee closes the loop on delivery: an alert is never silent. On top of the messaging gateways, every state-change alert also raises a native macOS notification and push, independent of the gateways — so even with email off and every channel disabled, a service going down still reaches you on the Mac. Database recovery — running mysqlcheck/pg_amcheck and the AI-guided rebuild — is covered in Chapter 5.

11.7Incidents, on-call and status pages

For teams that already run formal incident response, FrontierStack connects to the tools you use rather than replacing them. Each has a live pane reached from its catalog entry: paste a read-only API key and it shows current state.

• Incident management & on-call — PagerDuty (open and high-urgency incidents, who is on-call, services), Opsgenie (open/unacknowledged alerts, with a US/EU region toggle), incident.io and Rootly (active incidents).
• Status pages — Statuspage (unresolved incidents and components that are down) and Better Stack (monitors up / down / paused).

Set up the key in each pane (for example PagerDuty ▸ Integrations ▸ API Access Keys) and the pane confirms the connection. These feed the same monitoring picture, so an open PagerDuty incident or a down status-page component shows alongside your own health.

11.8SaaS live monitors & project-management panes

FrontierStack does not stop at infrastructure. A config-driven SaaS live monitor watches connected cloud accounts and refreshes every few minutes. Each service has a credentials form, live metric tiles, an "Alert me" toggle and a threshold; alarmable services raise a DOWN alert when the metric crosses the line, and the rest are watched for reachability. Live monitors include Stripe (disputes needing response, balance), Shopify and WooCommerce (open orders), Freshdesk and Zammad (open/pending/overdue tickets), SendGrid, Mailgun and Postmark (bounces and blocks), GitHub Copilot (inactive seats), and identity providers Okta and Microsoft Entra ID. Pin a service to make it always-on-Local-Health regardless of its alert setting.

The same pattern gives project-management tools live token-login panes: Jira (open / unassigned / blocked / in-sprint counts), Linear, monday.com, OpenProject, Plane and Taiga. These are dashboards rather than alert sources, but they put your team's workload in the same window as the servers that run it. Email delivery has its own unified Email Delivery pane that combines transactional-provider metrics with SPF/DKIM/DMARC checks (Chapter 7).

11.9Queue Operations

Queue Operations is a read-only dashboard for RabbitMQ, Kafka, NATS/JetStream, Redpanda, AWS SQS, Azure Service Bus, Google Pub/Sub, Celery/Flower, Redis Streams, Apache Pulsar and Apache RocketMQ. Add one monitor for each broker, namespace or cloud account. FrontierStack shows queue or consumer-group names, ready and in-flight counts, consumer counts, lag, dead-letter counts and oldest-message age when the provider exposes them. It never reads or stores message bodies.

Client and administration ports are kept separate. For example, RabbitMQ clients use AMQP on 5672 while its management API normally uses 15672; NATS clients use 4222 while monitoring normally uses 8222; Pulsar clients use 6650 while its HTTP admin API uses 8080. Keep monitoring endpoints private. Use TLS and a monitoring-only account whenever the dashboard is not reached through the on-host FrontierStack monitor.

For a linked server, select its installed monitor and FrontierStack asks the credential-free helper for bounded summaries over the signed connection. Cloud providers use their normal local CLI profiles or a scoped credential saved in Keychain. Critical thresholds join the Alerts sweep, the same sanitised summaries are available in the iPhone/iPad app, and the AI Administrator can answer queue-health questions through the read-only get_server_health tool. The dashboard has no purge, delete, publish, acknowledge or replay action.

11.10UPS monitoring and SNMP devices

Power is the failure mode people forget until the lights go out. FrontierStack discovers UPS units three ways — macOS power sources (pmset -g ps), the NUT daemon (upsc), and apcupsd (apcaccess) — and shows charge, runtime, load and input in the UPS section of Power Control. Turn on alerts and you are paged when a unit goes on battery, hits a low-charge threshold you set, or loses comms with its controller. Pin a UPS to keep it on Local Health and on the Places map; one-click buttons install NUT or apcupsd if they are missing. Because a scan waits on those daemons, a Stop button appears while one is running — if a NUT server has wedged, stop the scan instead of waiting; readings already found are kept and automatic refresh pauses until you press Refresh.

For anything else that speaks SNMP — managed switches, printers, network UPSes, NAS units — the SNMP / OIDs section of a device's detail pane queries it directly. Pick a built-in template (System, Host Resources, Interfaces, Printer RFC 3805, UPS RFC 1628, Synology, QNAP, APC PowerNet) and press Query for a label-and-value readout, or fetch a single custom OID by hand. You can also watch an OID: set a comparison and threshold (greater-than, equals, contains, or simply "alert on any change") and FrontierStack polls it and raises an alert when the condition is met. It uses net-snmp, installable in one click.

11.11Disk, RAID and SMART health

Drives fail with warning if you are listening for it. The Disk Health pane (a built-in monitoring tool) combines several layers. Volume space and Time Machine status are always shown. With smartmontools installed (a one-click Install button), each physical drive is enriched with deep SMART attributes — health, temperature, power-on hours, reallocated and pending sectors, and predicted-failure flags — visible per drive with a Check button and a Details sheet for the raw report. A drive trips an alert on a SMART failure or on concerning attributes (reallocated/pending sectors or a predict-fail), not only on outright death.

Software AppleRAID sets (mirror, stripe, concat) appear in their own section with level, status and per-member state; an alert fires when a set is degraded or a member drops offline. Under Alerts & Thresholds, switch on disk alerts and they route to your channels exactly like every other monitor.

Taken together, these monitors give you one board to glance at, one pane to tune, and one set of channels to reach you on — and an AI Administrator (Chapter 13) that reads the same health and sends the same notifications on your behalf. The host monitors that feed fleet-wide CPU, memory and GPU metrics into this picture are covered in Chapter 8.