With data lakes becoming popular, and Azure Data Lake Store (ADLS) Gen2 being used for many of them, a common question I am asked about is “How can I access data in ADLS Gen2 instead of a copy of the … Continue reading →
As a follow-up to my blogs What product to use to transform my data? and Should I load structured data into my data lake?, I wanted to talk about where you should you clean your data when building a modern data warehouse … Continue reading →
If you are building a big data solution in the cloud, you will likely be landing most of the source data into a data lake. And much of this data will need to be transformed (i.e. cleaned and joined together … Continue reading →
At the Microsoft Ignite conference, Microsoft announced that SQL Server 2019 is now in preview and that SQL Server 2019 will include Apache Spark and Hadoop Distributed File System (HDFS) for scalable compute and storage. This new architecture that combines together the … Continue reading →
Azure Data Lake Analytics (ADLA) is a distributed analytics service built on Apache YARN that allows developers to be productive immediately on big data. This is accomplished by submitting a job to the service where the service will automatically run it … Continue reading →
I previously talked about PolyBase and its enhancements (see PASS Summit Announcements: PolyBase enhancements). There is some confusion on PolyBase use cases as they are different depending on whether you are using PolyBase with Azure SQL Data Warehouse (SQL DW) or … Continue reading →
Below I have attempted to list the various options for reporting off of Hadoop (HDInsight, HDP, Cloudera) using Power BI Desktop and Excel. Some of the data sources prompt you to choose the Data Connectivity mode of either Import or … Continue reading →
Microsoft usually has some interesting announcements at the PASS Summit, and this year was no exception. I’m writing a set of blogs covering the major announcements. Next up is the PolyBase enhancements. PolyBase is a technology that accesses and combines both … Continue reading →
In a previous blog I talked about copying on-prem data to Azure Blob Storage (Getting data into Azure Blob Storage). Let’s say you have copied the data and it is sitting in Azure Blob Storage (or an Azure Data Lake) … Continue reading →
If you have on-prem data and want to copy it to Azure Blob Storage in the cloud, what are all the possible ways to do it? There are many, and here is a quick review of them: AzCopy: A popular command-line … Continue reading →
Analytics Platform System (APS) is Microsoft’s massively parallel processing (MPP) data warehouse technology. This has only been available as an on-prem solution (see video Overview of Microsoft Analytics Platform System). Until now. At the recent Microsoft Build Developer Conference, Executive Vice … Continue reading →
I see a lot of confusion when it comes to Hadoop and its role in a data warehouse solution. Hadoop should not be a replacement for a data warehouse, but rather should augment/complement a data warehouse. Hadoop and a data warehouse … Continue reading →
PolyBase is a new technology that integrates Microsoft’s MPP product, SQL Server Parallel Data Warehouse (PDW), with Hadoop. It is designed to enable queries across relational data stored in PDW and in non-relational Hadoop data that is stored in the Hadoop Distributed File … Continue reading →