Data Lakehouse, Data Mesh, and Data Fabric
(NOTE: I have returned to Microsoft and am working as a Solution Architect in Microsoft Industry Solutions, formally known as Microsoft Consulting Services (MCS), where I help customers build solutions on Azure. Contact your Microsoft account executive for more info. That being said: the views and opinions in this blog are mine and not that of Microsoft).
There certainly has been a lot of discussion lately on the topic of Data Lakehouse, Data Mesh, and Data Fabric, and how they compare to the Modern Data Warehouse. There is no clear definition of all these data architectures, and I have created a presentation using my own take that I have been presenting frequently internally at Microsoft and externally to customers and at conferences. Hopefully these presentations, blog posts, and videos can help clarify all these data architectures for you:
- Videos of me presenting on “Data Lakehouse, Data Mesh, and Data Fabric (the alphabet soup of data architectures)” can be found in three different lengths: DataMinutes (recording – 10 minutes), Data Agility Day (recording – 30 minutes), and India Azure Community Conference 2021 (recording – 1 hour). I will also be presenting it at: SQLBits on 3/10/22 (info) and Data Summit 2022 on 5/17/22 (info). Abstract of the presentation is below
- The Data Lakehouse, Data Mesh, and Data Fabric presentation slides can be found here
- I did a 20-minute video explaining the Modern Data Warehouse that you can view here
- These are my blog posts on the subject matter: Data Lakehouse defined, Data Fabric defined, Data Mesh defined, Data Mesh: Centralized vs decentralized data architecture, Data Mesh: Centralized ownership vs decentralized ownership
- A 30-minute video at the Hevo Cloud Data Warehousing Summit: Why Modern Enterprises Need a Cloud Data Warehouse
- Check out the SaxonGlobal Data Story Podcast Series that covers all the architectures as well as common data models in four episodes
Look for a blog post of mine in a couple months that will cover Microsoft’s vision and technology solution of a data mesh.
Presentation abstract:
Data Lakehouse, Data Mesh, and Data Fabric (the alphabet soup of data architectures)
So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a modern data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. They all may sound great in theory, but I’ll dig into the concerns you need to be aware of before taking the plunge. I’ll also include use cases so you can see what approach will work best for your big data needs. And I’ll discuss Microsoft version of the data mesh.
Pingback:Lakehouse, Mesh, and Fabric – Curated SQL
So you couldn’t stay away for long?! Sounds like a good spot for you. Thanks as always for the insights.
Did you see my blog on data mesh where I add comments from your blog? I’m very skeptical, too. https://www.eckerson.com/articles/rethinking-the-data-mesh-apply-it-piecemeal
Great article Wayne! I’m very skeptical too. The more I read about data mesh, the more concerns I have
I’ve coined the term “Data Architecture as a Service” or DaaS to describe an emerging set of products that enable data architects to embed “guardrails” into self-service data integration or transformation tools so business users can create architecturally compliant data silos (i.e., non-siloed data silos– we used to call these data marts!) A new company called Coalesce comes closest to doing this. This would be key if the data mesh has any chance of succeeding. In fact, it’s one of the technologies Zhamak Dheghani said needs to be invented.
Here’s my write-up of DaaS. https://www.eckerson.com/articles/data-architecture-as-a-service-liberation-for-data-users
Glad you’re back, James!
Kind of a shame to lump lakehouse – an actual, concrete, easy to understand architectural design pattern – with two Gartner buzzwords.
Protip: if it’s not easy to describe the tradeoffs of something, it’s not actually an architecture.
I can explain the pros and cons of a lakehouse and how such a thing came to be to an experienced data engineer in 60 seconds.
Nobody can explain a data fabric or data mesh without sounding full of hot air and hype (your slides are great proof of this by the way)
Kyle, I love this quote:
“Nobody can explain a data fabric or data mesh without sounding full of hot air and hype (your slides are great proof of this by the way)”
hi, the link to the slides does not work. Could you please update the link? Thanks.
I placed the deck here: https://serrapublic.blob.core.windows.net/presentations/Data%20Lakehouse%2C%20Data%20Mesh%2C%20and%20Data%20Fabric%20-%20Public.pdf