Data engineering, priced honestly

ingestion, transformation, warehousing and orchestration

The modern data stack is a stack of bills. Ingestion, storage, transformation and orchestration are four separate line items from four separate vendors, and each one prices on a different metric — rows synced, credits consumed, active rows, compute seconds. The result is that a data platform's cost is genuinely difficult to forecast, and the surprises arrive after the architecture is committed.

This site profiles the components of that stack and describes how each one's pricing actually behaves at scale. Fivetran's active-row metric and Snowflake's per-start minimum are both reasonable designs that produce unreasonable bills when the workload does not match the assumption behind them.

Each profile covers what the tool does, the pricing mechanic that drives cost, where it stops being the right choice, and what migrating away would involve. The goal is to make the trade-offs visible before the contract rather than after the first invoice.

Profiles

• Airbyte ingestion

  Airbyte is a ingestion product that connectors for hundreds of sources. This profile covers what it does, how it is priced, and where it stops being the right choice.

• ClickHouse analytics

  ClickHouse is a analytics product that columnar store built for aggregation. This profile covers what it does, how it is priced, and where it stops being the right choice.

• dbt transformation

  dbt is a transformation product that version-controls SQL models. This profile covers what it does, how it is priced, and where it stops being the right choice.

• Snowflake warehouse

  Snowflake is a warehouse product that separates storage from compute. This profile covers what it does, how it is priced, and where it stops being the right choice.

• Fivetran managed pipelines

  Fivetran is a managed pipelines product that keeps warehouse tables in sync. This profile covers what it does, how it is priced, and where it stops being the right choice.

• Dagster orchestration

  Dagster is a orchestration product that asset-based scheduling for pipelines. This profile covers what it does, how it is priced, and where it stops being the right choice.