Big Data Workshop
A challenge I have with customers who want to get hands-on experience with the Azure products that are found in a modern data warehouse architecture is finding a workshop that covers many of those products. To the rescue is a workshop created by my Microsoft colleagues Fabio Braga and Rod Colledge, explained in their blog post Azure Data Platform End2End with the GitHub located here. This is an on-demand workshop with labs that you can run at any time.
The idea of this workshop is to give experienced BI professionals (but new to Azure) a view of the variety of data services available and the role they play in the overall architecture. Most professionals never had a chance before to use a Spark cluster, or a NoSQL database, so the workshop aims to fill this gap. It’s true that similar outcomes can also be achieve with other services/features (this workshop uses only a subset of a much larger family of Azure services), but there is only so much that can be covered in a 2-day workshop. So keep in mind the architecture used in this workshop is only one of many possibilities for building a modern data warehouse solution. The lab will be updated as new products and features are released (i.e. ADF Mapping Data Flow when it GA’s).
A description of the workshop:
In this 2-day workshop you will learn about the main concepts related to advanced analytics and Big Data processing and how Azure Data Services can be used to implement a modern data warehouse architecture. You will understand what Azure services you can leverage to establish a solid data platform to quickly ingest, process and visualize data from a large variety of data sources. The reference architecture you will build as part of this exercise has been proven to give you the flexibility and scalability to grow and handle large volumes of data and keep an optimal level of performance. In the exercises in this lab you will build data pipelines using data related to New York City. The workshop was designed to progressively implement an extended modern data platform architecture starting from a traditional relational data pipeline. Then we introduce big data scenarios with large files and distributed computing. We add non-structured data and AI into the mix and finish with real-time streaming analytics. You will have done all of that by the end of the workshop. The workshop include a series of five labs with a discussion of concepts in-between each lab.
Technologies you will use: SQL Data Warehouse, SQL Server in a VM, Azure Data Factory, Databricks (w/Spark), Cognitive Services (w/computer vision), Event Hub, Stream Analytics, PolyBase, Power BI, Blob Storage, Cosmos DB, Logic App
Hi james,
Thanks for the learning opportunity we know the workshop date?
Thanks
Sathwik
Hi Sathwik,
This is an on-demand workshop with labs that you can run at any time.
Hi James… I couldn’t figure out when this workshop is happening.. is it an on-demand event? Please let me know.
Thanks
Mohamed Ahsan
Hi Mohamad,
This is an on-demand workshop with labs that you can run at any time.
Hi James,
Thanks for the information. It appears that Data bricks and Azure SQL DW is not among the ‘free’ Azure services that are included here –
https://azure.microsoft.com/en-us/free/
Am i correct in reading that? the tutorial seems to mention the use of those two services.
Thanks for your response.
Jude
Hi Jude,
When you setup a trial Azure subscription you get $200 credit to use with *any* Azure service plus the ones considered free from the list you highlighted. This credit should be enough for you to execute the 5 labs in the workshop.
Let us know if you find any issues with that approach. Thank you.
Hi James,
Thanks to Fabio Draga for setting up this workshop. I was unable to use the template provided. This may be due to some Azure services not available for free. Can you please confirm?
Thanks,
Shahid
Hi Shahid,
I am assuming you are using a trial subscription, correct? With a newly setup trial subscription you receive $200 credit that can be used with *any* Azure service and that should be enough to execute the 5 labs in the workshop. Could you please provide more details about the error message you received when you tried to deploy the Azure services? Thank you.
Hi Fabio,
Thanks for your response. I got the following error when trying to deploy:
“InvalidTemplateDeployment: The template deployment ‘MDW-Lab’ is not valid according to the validation procedure. The tracking id is ’02bf982d-e1e6-4725-9f2a-6da86a8443ea’. See inner errors for details. Please see https://aka.ms/arm-deploy for usage details.”
Shahid
Hi Shahid,
I’ve done some tests here and unfortunately free trial subscriptions have restricted quotas that will prevent the lab template from being deployed. You will need to upgrade your subscription to Pay-As-You-Go if you want to proceed and the estimated consumption cost for the execution of the labs (1 day) is around $100, as long as you decommission services off after you finish the labs.
Thanks Fabio. Appreciate you response.
Pingback:Top Modern Data Warehouse questions | James Serra's Blog