MySQL → Snowflake: 1B rows in 8m 56s

So we did another benchmark this time with MySQL to Snowflake to demonstrate how Wirekite sizes up in the enterprise data migration space. Again we decided to compare the benchmark with Airbyte since it is one of the most recognized data migration enterprise software out there.

Benchmark Details

Table

First of all lets talk about the table structure. We have used the same table that we used in other benchmarks. The table has 16 columns, one of them is a numeric primary key, few columns are numbers of various types, few strings and few dates. This makes the table pretty simple to be absorbed by most databases.

The table also has 1 Billion rows.

We want to make it easy and standardized across various other benchmarks and databases.

CREATE TABLE `firenibble` (
  `f0` bigint NOT NULL,
  `f1` bigint DEFAULT NULL,
  `f2` bigint DEFAULT NULL,
  `f3` int DEFAULT NULL,
  `f4` float DEFAULT NULL,
  `f5` float DEFAULT NULL,
  `f6` float DEFAULT NULL,
  `f7` float DEFAULT NULL,
  `f8` varchar(256) DEFAULT NULL,
  `f9` varchar(256) DEFAULT NULL,
  `f10` date DEFAULT NULL,
  `f11` date DEFAULT NULL,
  `f12` date DEFAULT NULL,
  `f13` varchar(256) DEFAULT NULL,
  `f14` varchar(256) DEFAULT NULL,
  `f15` varchar(256) DEFAULT NULL,
  PRIMARY KEY (`f0`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci DATA DIRECTORY='/mnt/benchmark/mysqldata/'

And here’s the 1 billion rows.

mysql> select count(*) from firenibble;
+------------+
| count(*)   |
+------------+
| 1000000000 |
+------------+
1 row in set (13 min 4.70 sec)

mysql>

Machine Configuration

For MySQL we created an n2-standard-64 GCP instance with all the standard configurations - 64 CPU’s and 256 GB RAM in the us-central region. We installed Ubuntu version 24 and MySQL version 8.

For Snowflake we created a Large size warehouse with Standard Type also in the us-central1 region to minimize the network distance.

We have been using the same configurations for all our benchmarks so that we are comparing apples and apples.

Airbyte Run

We created the firenibble table with Full Refresh Overwrite and made sure the table did not exist in the target, thus forcing a full sync of the whole table.

And then we ran the benchmark only to get the following message after waiting for a long time.

After waiting and watching for almost 18 hours and transferring 103 million records, we did not have the courage to run the benchmark again. Turns out the snowflake free trial that we used to do the benchmark had expired. So we decided to accept this as a technically invalid benchmark, but since the rate of transfer was not changing drastically, we decided to publish it.

The logs

2025-03-10 23:46:48 info sync summary: {
  "status" : "failed",
  "recordsSynced" : 103123546,
  "bytesSynced" : 41680061203,
  "startTime" : 1741644491842,
  "endTime" : 1741650408147,
  "totalStats" : {
    "bytesCommitted" : 41680061203,
    "bytesEmitted" : 43332279893,
    "destinationStateMessagesEmitted" : 21,
    "destinationWriteEndTime" : 0,
    "destinationWriteStartTime" : 1741644491938,
    "meanSecondsBeforeSourceStateMessageEmitted" : 376,
    "maxSecondsBeforeSourceStateMessageEmitted" : 21,
    "maxSecondsBetweenStateMessageEmittedandCommitted" : 73,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 59,
    "recordsEmitted" : 107200542,
    "recordsCommitted" : 103123546,
    "recordsFilteredOut" : 0,
    "bytesFilteredOut" : 0,
    "replicationEndTime" : 1741650408144,
    "replicationStartTime" : 1741644491842,
    "sourceReadEndTime" : 0,
    "sourceReadStartTime" : 1741644491939,
    "sourceStateMessagesEmitted" : 21
  }

But our favorite message in the logs was the following !

"internalMessage" : "net.snowflake.client.jdbc.SnowflakeSQLException: Your account is suspended due to lack of payment method.",

Stack Trace: net.snowflake.client.jdbc.SnowflakeSQLException: Your free trial has ended and all of your virtual warehouses have been suspended. Add billing information in the Snowflake web UI to continue using the full set of Snowflake features.

You are poor !! Yes we are.

Wirekite Run

After rebooting the MySQL GCP instance to flush the buffer pool and any other remnants of the previous run we ran the same benchmark.

Everything else being same we got the following results.

1 billion rows moved under 9 minutes !!

# cat wirekite.log | grep -E 'START|FINISH|ELAPSED'
START: Mon Mar 10 03:06:20 UTC 2025
FINISH: Mon Mar 10 03:15:16 UTC 2025
ELAPSED: 0 hrs 8 min 56 sec

The billion rows in Snowflake.

#COMPUTE_WH@TEST.APP_DB>select count(*) from FIRENIBBLE;
+------------+
|   COUNT(*) |
|------------|
| 1000000000 |
+------------+
1 Row(s) produced. Time Elapsed: 0.242s
#COMPUTE_WH@TEST.APP_DB>
Goodbye!