SQL Server → Snowflake: 1B rows in 10m 52s

So one more benchmark except this time with a really expensive enterprise grade database.

We could not compare this benchmark with Airbyte since Airbyte does not allow an open source connector for SQL Server. Entirely understandable. When we tried to configure the connector after spending a long time installing Airbyte we got the following message.

Benchmark Details

As usual we will use our firenibble table with 1 Billion rows. That’s a lot of rows.

Since we could not do an Airbyte benchmark we decided to play a little bit with SQL Server and see how far we can scale our benchmarks as we increased the source SQL Server machine configuration, the target Snowflake configurations and the number of wirekite threads.

Table

As usual we used our standard firenibble table.

The table has 16 columns, one of them is a numeric primary key, few columns are numbers of various shapes and sizes, few strings and few dates. This makes the table pretty simple to be absorbed by most databases.

CREATE TABLE dbo.firenibble (
    f0 bigint NOT NULL,
    f1 bigint DEFAULT NULL,
    f2 bigint DEFAULT NULL,
    f3 int DEFAULT NULL,
    f4 float DEFAULT NULL,
    f5 float DEFAULT NULL,
    f6 float DEFAULT NULL,
    f7 float DEFAULT NULL,
    f8 varchar(128) DEFAULT NULL,
    f9 varchar(128) DEFAULT NULL,
    f10 date DEFAULT NULL,
    f11 date DEFAULT NULL,
    f12 date DEFAULT NULL,
    f13 varchar(128) DEFAULT NULL,
    f14 varchar(128) DEFAULT NULL,
    f15 varchar(128) DEFAULT NULL,
    PRIMARY KEY (f0)
  );

The table also has 1 Billion rows.

=> select count(*) from dbo.firenibble1;
1: : 1000000000
=>

Loading Data in SQL Server

We used the following command to load the 1 Billion rows in the firenibble table. It took sometime but it worked.

BULK INSERT dbo.firenibble FROM '/mnt/spinning/csv/firenibble.csv' WITH (FIELDTERMINATOR = '\t', ROWTERMINATOR = '\n', FIRSTROW = 1, BATCHSIZE=5000);

Wirekite Run 1

We had a standard 128 CPU GCP instance with 512 GB RAM. The host was Linux Ubuntu 20.04 with SQL Server version 2022. The machine lived in us-central1 region. We rebooted instances between runs.

For Snowflake we created a Large size warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We ran with 96 wirekite threads.

And it took about 10 minutes and 55 seconds.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Mon Jun 16 23:58:11 UTC 2025
FINISH: Tue Jun 17 00:09:06 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 10 min 55 sec

Wirekite Run 2

Ran with same machine configurations but with 90 threads and got this - 10 minutes and 58 seconds. Oops the time increased.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Tue Jun 17 00:11:11 UTC 2025
FINISH: Tue Jun 17 00:22:09 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 10 min 58 sec

Wirekite Run 3

This time we ran with 84 threads and got our best time.

Moved 1 billion rows from SQL Server to Snowflake in 10 minutes and 52 seconds.

cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Mon Jun 16 23:44:02 UTC 2025
FINISH: Mon Jun 16 23:54:54 UTC 2025
TOTAL ELAPSED TIME: 0 hrs 10 min 52 sec

I think there is still scope for improvement.