Home › Services › Data Pipelines & ETL

Data Pipelines & ETL: 9 solutions compared

Which Data Pipelines & ETL tools can you self-host, run on a Mac or connect to? FrontierStack's catalogue covers 9 Data Pipelines & ETL options: Airbyte, Dagster, dbt, Kestra, Meltano, NATS JetStream Pipelines and others. For 8 of them FrontierStack installs the service on a Mac or server, or connects to one you already run, and checks status, ports and certificates where they apply; 1 is a hosted service FrontierStack connects to with an API token and monitors.

NameTypeWhat FrontierStack does
AirbyteSelf-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
DagsterSelf-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
dbtSelf-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
KestraSelf-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
MeltanoSelf-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
NATS JetStream PipelinesSelf-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
PrefectSelf-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
Redpanda Connect (Benthos)Self-hosted or connectedInstall or connect; status, ports and certificate checks where they apply
SwitchboardHosted serviceConnect with an API token; reachability and status alerts

Airbyte

Open-source ELT with 300+ connectors (self-hosted / cloud)

Airbyte is an open-source data-integration (ELT) platform; 300+ source/destination connectors move data from APIs, files and databases into your warehouse/lake, with incremental sync, schema propagation and the Connector Builder for custom sources. Self-host with abctl (Docker/Kubernetes; web UI on :8000) or use Airbyte Cloud; a REST/Config API and Terraform provider automate connections. Pairs with dbt for the transform step.

Install or connect Airbyte; then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

Dagster

Asset-oriented data orchestrator (self-hosted / cloud)

Dagster is a modern data orchestrator built around software-defined assets; declare the tables/files/ML models you want and how to compute them, and it tracks lineage, freshness, partitions and backfills. Run the webserver with pip install dagster dagster-webserver then dagster dev (UI on :3000); the GraphQL API drives runs. Integrates dbt, Airbyte, Spark and the warehouses; Dagster+ is the hosted tier.

Install or connect Dagster; then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

dbt

SQL-based data transformation & modeling (self-hosted / cloud)

dbt (data build tool) is the transform step of the modern data stack; define models, tests, snapshots and docs as SQL + Jinja, then dbt run / dbt test to build them in your warehouse with full lineage. Install dbt Core with pip install dbt-core dbt-<adapter> (postgres, snowflake, bigquery, …); dbt docs serve hosts the lineage site. dbt Cloud adds a hosted IDE, scheduler and API. Pairs with Airbyte/Meltano (EL) and Dagster/Prefect (orchestration).

Install or connect dbt; then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

Kestra

Event-driven orchestration & scheduling, YAML flows (self-hosted)

Kestra is a declarative, event-driven orchestrator; define flows in YAML with 600+ plugins (databases, cloud, dbt, scripts), trigger on schedules, events or webhooks, and watch executions in a rich web UI. Self-host with Docker/Docker Compose (UI + REST API on :8080; needs PostgreSQL for production). Language-agnostic, so it schedules SQL, Python, shell and container tasks alike.

Install or connect Kestra; then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

Meltano

Open-source ELT built on Singer taps & targets (CLI)

Meltano is a code-first, Git-friendly ELT tool built around the Singer spec; compose 600+ Singer taps (sources) and targets (destinations) in meltano.yml, run meltano run tap-… target-…, and add dbt for transforms and Airflow/Dagster for scheduling. pipx install meltano; a local web UI is available. Everything-as-code, so pipelines live in version control.

Install or connect Meltano; then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

NATS JetStream Pipelines

Persistent streams & consumers for event pipelines (self-hosted)

JetStream is NATS' built-in persistence layer; durable streams, replay, at-least/exactly-once consumers and work-queue/retention policies that turn NATS into the backbone of event-driven data pipelines. Run nats-server -js (monitoring on :8222, client on :4222) and manage streams/consumers with the nats CLI. Pair with Redpanda Connect or your own subscribers to build ingest → process → store flows. (For general pub/sub, see NATS under Message Queues & Streaming.)

Install or connect NATS JetStream Pipelines; then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

Prefect

Modern Python workflow orchestration (self-hosted / cloud)

Prefect orchestrates Python data workflows with a dynamic, Pythonic API; flows and tasks with retries, scheduling, caching and observability. Run your own server (pip install prefect then prefect server start); UI and REST API on :4200. The API lets other services trigger flow runs and read state; Prefect Cloud is also available.

Install or connect Prefect; then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

Redpanda Connect (Benthos)

Declarative stream-processing & connectors (self-hosted CLI)

Redpanda Connect (formerly Benthos) is a single-binary, declarative stream processor; wire inputs → processors → outputs in YAML with rich connectors (Kafka, NATS, HTTP, S3, SQL, …), the Bloblang mapping language, and built-in delivery guarantees. brew install redpanda-data/tap/redpanda-connect (or run the benthos/rpk connect binary); great for glue ETL, enrichment and routing between systems with no JVM.

Install or connect Redpanda Connect (Benthos); then check its status, ports and certificate from FrontierStack. The same screen links to firewall checks and backups where they apply.

Switchboard

Managed DataOps: ad, revenue & finance data into BigQuery (cloud)

Switchboard (Switchboard Software, Inc.) is a hosted data-integration and DataOps platform built by former Google BigQuery engineers for publishers, media, ad-tech and marketing teams. 200+ pre-built connectors (Google Ad Manager, Meta, Snapchat, TikTok, SSPs, OMS/ERP and custom APIs) pull advertising, monetisation, e-commerce and finance data, normalise and reconcile it, and load it into your cloud warehouse (BigQuery; Snowflake and Redshift also mentioned) or export it as gzip CSV, Parquet or Avro. It handles schema detection, retries, automatic backfills and 30-day lookbacks, and alerts by Slack or email when a source API breaks or a credential is about to expire. A Command Centre dashboard monitors pipeline health, version-controlled branches are checked by the Switchboard Compiler before publishing, and Intelligent Alerts watches revenue. Governed data can be reached from Claude, Gemini or any MCP client through native BigQuery access with IAM roles. SOC 2 Type II. Pricing is by quote, and it can be run by your own team or as a managed service. There is no public API or status page, so FrontierStack keeps the sign-in details here and links out; watch the BigQuery side with your warehouse tools.

Switchboard is hosted, so FrontierStack does not try to install it. Connect an API token to check reachability and published status beside the services you host yourself; FrontierStack alerts you if it stops answering. Credentials stay in Keychain.

All service genres →

Run it from your Mac.

FrontierStack installs, monitors and secures services on this Mac and on linked servers; and keeps your passwords and keys out of any AI's sight.

Download FrontierStack

Apple notarized · Safe & secure · macOS 13 Ventura+