Data engineering, priced honestly

ingestion, transformation, warehousing and orchestration

The modern data stack is a stack of bills. Ingestion, storage, transformation and orchestration are four separate line items from four separate vendors, and each one prices on a different metric — rows synced, credits consumed, active rows, compute seconds. The result is that a data platform's cost is genuinely difficult to forecast, and the surprises arrive after the architecture is committed.

This site profiles the components of that stack and describes how each one's pricing actually behaves at scale. Fivetran's active-row metric and Snowflake's per-start minimum are both reasonable designs that produce unreasonable bills when the workload does not match the assumption behind them.

Each profile covers what the tool does, the pricing mechanic that drives cost, where it stops being the right choice, and what migrating away would involve. The goal is to make the trade-offs visible before the contract rather than after the first invoice.

Profiles

• Airbyte ingestion
• Apache Airflow orchestration

  the task orchestrator, its depth and its scheduling latency

• ClickHouse analytics
• Confluent streaming platform

  managed Kafka, and why egress decides the invoice

• Dagster orchestration
• Databricks lakehouse

  the lakehouse, DBUs and the cost of an idle cluster

• dbt transformation
• DuckDB embedded analytics

  analytics in your own process, faster than a cluster

• Fivetran managed pipelines
• Apache Iceberg table format

  an open table format and the catalogue that makes it usable

• Materialize streaming SQL

  SQL views maintained incrementally as data arrives

• Redpanda streaming platform

  Kafka without the JVM, and what that changes

• Snowflake warehouse
• Trino query engine

  federated SQL across systems that do not know about each other