This article is the first in a planned series on what it’s like to have Snowflake as your data warehouse/data lake. In teaching “Zero 2 Snowflake” workshops, I have found that people have difficulty wrapping their heads around what it means to have a groundbreaking technology that is truly “designed for the cloud” database.
It’s easy to get involved in the technical details of the architecture and miss the proverbial forest for the trees. This post describes how Snowflake’s architecture spells an end to run ETL at night and on the weekends and how it drastically reduces the time taken for ETL processes to run.
The Snowflake features that help achieve all these are:
Immutable data storage
Pointers to the data
On-demand compute clusters that you only pay for as you use
Separation of data from computing
Performance scales near linearly by increasing the size of your cluster
Why have companies been running ETL processes at night and on the weekend for all these years? The ETL process would run on the same computers running the reports. Also, for traditional databases, writes and reads contend with each other. Nobody wants the ETL process to bog down the system.
Snowflake data is stored in an immutable database. There are only writes. Here, updates are new data that is written while pointers to this data are changed. This enables another great Snowflake feature, time travel, that won’t be addressed in this blog. From an ETL perspective, the lack of contention between the writes and updates allows ETL to be run at any time.
The ability to spin up or resume a compute cluster (Snowflake calls them compute warehouses) at any time and the fact that compute scales independently of storage means that “regular business use” of the data is handled by different compute clusters than ETL. When it’s time to run ETL, all you have to do is spin up a cluster just for the ETL. This takes only seconds, and Snowflake doesn’t charge when you aren’t using a compute cluster.
If an ETL process would take ten hours with a 1cpu “cluster,” it should take five hours with a 2-node cluster, two-and-a-half hours with a 4-node cluster, and so on. The cost of the cluster for use doubles each time as well. But you are only paying for the cluster when you are using it. So, running a ten-hour process on a 1cpu “cluster” costs the same as running a five-minute process on a 128-node cluster. During development, you can test to find the “point of no further advantage” or at what size the cluster is optimum. Though, one can scale without paying for 365x24 costs for that capacity.
Who in ETL hasn’t come to work in the morning only to find that a six-hour ETL processed failed during the night? The task of fixing the bug and then rerunning the load without impacting the business might be impossible. There isn’t time to rerun before the users come in, expecting to run their reports. With Snowflake, you could shorten the time a rerun would take and rerun the process without impacting the users running reports.
This radical ability to scale performance and only pay for what you use has made Snowflake data warehouse “the” place to do all the processing that can be done. In essence, ETL has become ELT – Extract, Load, and then Transform.
You can move your data into Snowflake, then transform it with the massive scale and cost advantages of the Snowflake cloud database. But that’s a story for another blog. For now, living with Snowflake means sparing the ETL developer from working so many nights and weekends.
Contact Us Today
Cisco’s AppDynamics unit is releasing new updates to their signature Experience Journey Map (ExJM) feature to allow user data segmentation by location, device, OS and browser. The release is a strong signal that the company is intent on planting its lead firmly in the application management space and extending the kind of key integrations that Cisco hoped would pay off from the 2017 acquisition.Explore
By 2010—three years after the iPhone 1 took the market by storm, the app revolution was in full swing, with smartphones outselling personal computers for the first time. The “App” was voted Word of the Year by the American Dialect Society, while more than 100 million smartphones were shipped in the fourth quarter. Companies were moving swiftly to be among the first in their industry to hit the market with proprietary apps. They were ready to pay millions of dollars and devote thousands of man-hours to develop a minimum viable product worthy of their user base’s expectations.Explore
Data sharing has evolved a lot over the past few decades. The starting point was sending attachments via email, which allowed the sharing of data at the cost of manageability. Hundreds of emails with individual attachments are an administrative nightmare, and the introduction of the cloud has overcome this issue.Explore
When you think of enterprise IT, your mind may picture massive data centers and server infrastructure. While servers are the backbone of enterprise IT, you also need to consider the numerous endpoint assets assigned to your employees. These individual assets may include desktop PCs, laptops, mobile phones, and tablet devices. Less obvious examples could also include printers and fax machines, along with software assets.Explore
HR management is one of the most critical aspects of enterprise management. Your human resources are the driving force for workflows and projects, helping you to achieve business goals. Without these people, productivity will grind to a halt, leaving your business in stasis. The best way to keep these people engaged and committed to the company is through exemplary HR management. This could include measures such as tracking employee satisfaction levels, simplifying holiday request provision, and providing salary data visibility for employees.Explore
Enterprise IT performance is a crucial indicator of the quality of your internal and external services. Internal services are used by employees, where reliability and responsiveness contribute towards better productivity and reduced stress. It also indirectly benefits your IT department, as fewer performance-related IT service requests will reach your service desk. External services are even more critical as they serve your customer base. When these customers experience slowdowns, service disruption, or complete outages, they may decide to vote with their wallet and move elsewhere.Explore