PostgreSQL → Snowflake: 89× faster, 1B rows

So we did our first benchmark and we are thrilled to share the results. We chose to compare our benchmark with Airbyte since that is one of the most recognized data migration enterprise software out there.

This was inspired from a benchmark that Airbyte published - https://airbyte.com/blog/postgres-replication-performance-benchmark-airbyte-vs-fivetran.

Benchmark Details

Table

As you can see we used the same table that Airbyte used. It has a numeric primary key column, some number columns, some string columns and some date columns. This fits nicely with most databases.

CREATE TABLE firenibble
(
    f0 BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    f1 BIGINT,
    f2 BIGINT,
    f3 INTEGER,
    f4 DOUBLE PRECISION,
    f5 DOUBLE PRECISION,
    f6 DOUBLE PRECISION,
    f7 DOUBLE PRECISION,
    f8 VARCHAR COLLATE pg_catalog."default",
    f9 VARCHAR COLLATE pg_catalog."default",
    f10 DATE,
    f11 DATE,
    f12 DATE,
    f13 VARCHAR COLLATE pg_catalog."default",
    f14 VARCHAR COLLATE pg_catalog."default",
    f15 VARCHAR COLLATE pg_catalog."default"
) tablespace benchmark ;

And here’s the 1 billion rows. The query to do a count takes a very long time.

wirekite=# select count(*) from firenibble;
   count
------------
 1000000000
(1 row)

Machine Configurations

For Postgres we created an n2-standard-64 GCP instance with all the standard configurations - 64 CPU’s and 256 GB RAM in the us-central region.

For Snowflake we created a Large size warehouse with Standard Type also in the us-central1 region to minimize the network distance.

Airbyte Run

We created the firenibble table with various sizes to get the sense of how airbyte functions. We did Full Refresh Overwrite and made sure the table did not exist in the target.

We tried a bunch of runs. Some failed for various reasons. It is frustrating when we saw the run failed at 975 million rows.

But then it succeeded !!

1 billion rows migrated in 12 hours 6 minutes and 29 seconds.

Execution logs and details as reported by Airbyte.

2025-03-08 19:21:20 info sync summary: {
  "status" : "completed",
  "recordsSynced" : 1000000000,
  "bytesSynced" : 439710415164,
  "startTime" : 1741418118670,
  "endTime" : 1741461680232,
  "totalStats" : {
    "bytesCommitted" : 439710415164,
    "bytesEmitted" : 439710415164,
    "destinationStateMessagesEmitted" : 100000,
    "destinationWriteEndTime" : 1741461680072,
    "destinationWriteStartTime" : 1741418118737,
    "meanSecondsBeforeSourceStateMessageEmitted" : 3,
    "maxSecondsBeforeSourceStateMessageEmitted" : 100000,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 68,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 48,
    "recordsEmitted" : 1000000000,
    "recordsCommitted" : 1000000000,
    "recordsFilteredOut" : 0,
    "bytesFilteredOut" : 0,
    "replicationEndTime" : 1741461680087,
    "replicationStartTime" : 1741418118670,
    "sourceReadEndTime" : 1741460800423,
    "sourceReadStartTime" : 1741418118738,
    "sourceStateMessagesEmitted" : 100000
  }

Wirekite Run

Same Postgres instance. Same Snowflake instance. Same table. Same data.

1 billion rows were migrated in 8 minutes 9 seconds.

# cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Fri Mar 14 03:54:04 UTC 2025
FINISH: Fri Mar 14 04:02:13 UTC 2025
ELAPSED: 0 hrs 8 min 9 sec

And here’s the table in Snowflake.