As I mentioned in my Data Mesh, Data Fabric, Data Lakehouse presentation, the data lakehouse architecture, where you use a data lake with delta lake as a software layer and skip using a relational data warehouse, is becoming more and … Continue reading →
A question I get asked frequently from customers when discussing Data lake architecture is “Should I use one data lake for all my data, or multiple lakes?”. Ideally, you would use just one data lake, but I have seen many … Continue reading →
Another buzzword that you may have been hearing a lot about lately is Data Fabric. In short, a data fabric is a single environment consisting of a unified architecture with services and technologies running on it that architecture that helps … Continue reading →
The two latest trends in emerging data platform architectures are the Data Lakehouse (the subject of my last blog Data Lakehouse defined), and the Data Mesh, the subject of this blog. Data Mesh was first introduced by ThoughtWorks via the … Continue reading →
As a follow-up to my blog Data Lakehouse & Synapse, I wanted to talk about the various definitions I am seeing about what a data lakehouse is, including a recent paper by Databricks. Databricks uses the term “Lakehouse” in their … Continue reading →
A question that I have been hearing recently from customers using Azure Synapse Analytics (the public preview version) is what is the difference between using an external table versus a T-SQL view on a file in a data lake? Note … Continue reading →
As I was finishing up a demo script for my presentation at the SQL PASS Virtual Summit on 11/13 (details on my session here), I wanted to blog about part of the demo that shows a feature in the public … Continue reading →
I am starting to see this relatively new phrase, “Data Lakehouse”, being used in the data platform world. It’s the combination of “Data Lake” and “Data Warehouse”. In this post I’ll give my thoughts on it, and how the next … Continue reading →
On the heels of More Azure Blob Storage enhancements come more enhancements for Azure Data Lake Store Gen2 (ADLS Gen2): Archive tier is now GA: The archive tier provides an ultra-low cost tier for long term retention of data while … Continue reading →
I discover a small new feature the other day, but a very useful one. Previously, there was no way to upload files to an ADLS Gen2 storage account via the Azure portal. You had to use Azure Storage Explorer – … Continue reading →
Just announced is Query Acceleration for Azure Data Lake Storage Gen2 (ADLS) as well as Blob Storage. This is a new capability for ADLS that enables applications and analytics frameworks to dramatically optimize data processing by retrieving only the data … Continue reading →
As a followup to my blog post Azure Data Lake Store Gen2 is GA, I wanted to give some pointers when using ADLS Gen2 as well as blob storage, as it can get a bit confusing with all the options … Continue reading →
With data lakes becoming popular, and Azure Data Lake Store (ADLS) Gen2 being used for many of them, a common question I am asked about is “How can I access data in ADLS Gen2 instead of a copy of the … Continue reading →
The Common Data Model (CDM) is a shared data model that is a place to keep all common data to be shared between applications and data sources. Another way to think of it is is a way to organize data … Continue reading →
As a follow-up to my blogs What product to use to transform my data? and Should I load structured data into my data lake?, I wanted to talk about where you should you clean your data when building a modern data warehouse … Continue reading →
Azure Data Lake Store (ADLS) Gen2 was made generally available on February 7th. In short, ADLS Gen2 is the best of the previous version of ADLS (now called ADLS Gen1) and Azure Blob Storage. ADLS Gen2 is built on Blob storage … Continue reading →
Big news! The next generation of Azure Data Lake Store (ADLS) has arrived. See the official announcement. In short, ADLS Gen2 is the combination of the current ADLS (now called Gen1) and Blob storage. Gen2 is built on Blob storage. By … Continue reading →
There are a tremendous amount of Microsoft products that are cloud-based for building big data solutions. It’s great that there are so many products to choose from, but it does lead to confusion on what are the best products to … Continue reading →
I have blogged before about data lakes (see What is a data lake? and Why use a data lake?), and wanted to provide more details on this popular technology, some of which I cover in my presentation “Big data architectures and the data lake“. … Continue reading →
Azure Data Lake Analytics (ADLA) is a distributed analytics service built on Apache YARN that allows developers to be productive immediately on big data. This is accomplished by submitting a job to the service where the service will automatically run it … Continue reading →
