This article is the first in a planned series on what it’s like to have Snowflake as your data warehouse/data lake. In teaching “Zero 2 Snowflake” workshops, I have found that people have difficulty wrapping their heads around what it means to have a groundbreaking technology that is truly “designed for the cloud” database.
It’s easy to get involved in the technical details of the architecture and miss the proverbial forest for the trees. This post describes how Snowflake’s architecture spells an end to run ETL at night and on the weekends and how it drastically reduces the time taken for ETL processes to run.
The Snowflake features that help achieve all these are:
Immutable data storage
Pointers to the data
On-demand compute clusters that you only pay for as you use
Separation of data from computing
Performance scales near linearly by increasing the size of your cluster
Why have companies been running ETL processes at night and on the weekend for all these years? The ETL process would run on the same computers running the reports. Also, for traditional databases, writes and reads contend with each other. Nobody wants the ETL process to bog down the system.
Snowflake data is stored in an immutable database. There are only writes. Here, updates are new data that is written while pointers to this data are changed. This enables another great Snowflake feature, time travel, that won’t be addressed in this blog. From an ETL perspective, the lack of contention between the writes and updates allows ETL to be run at any time.
The ability to spin up or resume a compute cluster (Snowflake calls them compute warehouses) at any time and the fact that compute scales independently of storage means that “regular business use” of the data is handled by different compute clusters than ETL. When it’s time to run ETL, all you have to do is spin up a cluster just for the ETL. This takes only seconds, and Snowflake doesn’t charge when you aren’t using a compute cluster.
If an ETL process would take ten hours with a 1cpu “cluster,” it should take five hours with a 2-node cluster, two-and-a-half hours with a 4-node cluster, and so on. The cost of the cluster for use doubles each time as well. But you are only paying for the cluster when you are using it. So, running a ten-hour process on a 1cpu “cluster” costs the same as running a five-minute process on a 128-node cluster. During development, you can test to find the “point of no further advantage” or at what size the cluster is optimum. Though, one can scale without paying for 365x24 costs for that capacity.
Who in ETL hasn’t come to work in the morning only to find that a six-hour ETL processed failed during the night? The task of fixing the bug and then rerunning the load without impacting the business might be impossible. There isn’t time to rerun before the users come in, expecting to run their reports. With Snowflake, you could shorten the time a rerun would take and rerun the process without impacting the users running reports.
This radical ability to scale performance and only pay for what you use has made Snowflake data warehouse “the” place to do all the processing that can be done. In essence, ETL has become ELT – Extract, Load, and then Transform.
You can move your data into Snowflake, then transform it with the massive scale and cost advantages of the Snowflake cloud database. But that’s a story for another blog. For now, living with Snowflake means sparing the ETL developer from working so many nights and weekends.
Contact Us Today
Cisco’s AppDynamics unit is releasing new updates to their signature Experience Journey Map (ExJM) feature to allow user data segmentation by location, device, OS and browser. The release is a strong signal that the company is intent on planting its lead firmly in the application management space and extending the kind of key integrations that Cisco hoped would pay off from the 2017 acquisition.Explore
By 2010—three years after the iPhone 1 took the market by storm, the app revolution was in full swing, with smartphones outselling personal computers for the first time. The “App” was voted Word of the Year by the American Dialect Society, while more than 100 million smartphones were shipped in the fourth quarter. Companies were moving swiftly to be among the first in their industry to hit the market with proprietary apps. They were ready to pay millions of dollars and devote thousands of man-hours to develop a minimum viable product worthy of their user base’s expectations.Explore
The latest market trends indicate that enterprises will invest the most on digital transformation technologies that support operating model innovations. These investments will help enterprises focus on making business operations more responsive and effective by leveraging digitally-connected products/services, assets, people, and trading partners.Explore
Your workforce’s ability to collaborate, to access vital information and to disseminate that information to stakeholders is a major factor in its potential for innovation. Over the past several years, ServiceNow has expanded its offering of data management services, enabling automation at every level of your business. Trianz is a proud ServiceNow partner, and we specialize in creating the custom solutions you need to drive the rapid progress you expect.Explore
Would you like to speak with an expert?x