HomeAzure Data LakeData Mesh defined

Comments

Data Mesh defined — 28 Comments

  1. Excellent view on a strategy to shorten the gap between corporate and self-service BI as two ends of a spectrum. I like to suggest CDM as one major vehicle to enable easy reuse of published data. As you pointed out data domain knowledge might be low and additional meta is in result crucial for usage of data on high quality level.

  2. Pingback:The Data Mesh – Curated SQL

  3. Very enlighting article James, thanks for that. My thoughts are that Data Governance aims to achieve exactly the same goals, be it without prescribing the architecture but by creating the same responsibilities for data (domains), treating data as an asset (sort of product), training staff and ministration and administration of the data assets.

    So if you enter in Data Mesh Architecture, you automatically have half of your Data Governance effort and if you start a Data Governance program, you are likely to end up with a structure like Data Mesh.

  4. Pingback:Pragmatism: A guide to unlocking data’s strategic value | Intellect

    • Hi Doug, I have not heard the term “data mesh” before, but certainly the concepts have been around a while (i.e. data marts with Kimball). Would be interested to see if you have heard the data mesh term before.

      • Hi James, Data marts are very different. Data mesh seems to be a much more academic/theoretical concept, rather than an actual architecture, that looks a lot like pseudo-controlled data chaos–at least how ThoughtWorks has attempted to redefine it. I just watched Zhamak Dehghani’s presentation and have read and re-read her posts on the topic. Either she’s making a simple concept much more convoluted and buzzword-laden than necessary (likely to claim propriety), or the concept itself is way over-baked and riddled with impracticality holes. I’ve made some notes with the intent to post a blog sometime soon. Stay tuned. -Doug

          • As I’ve looked into Data Mesh more, including talking deeply with key folks at one of its biggest proponents, Starburst, I’ve come to realize it’s an age-old approach with newer technology (i.e., old wine in new wineskins). Call it VDW, EII, or data virtualization. Guess who Starburst competes with? Denodo and Dremio, both DV vendors, although Dremio has moved away from that term because it’s “old school”. Starburst is getting a lot of traction with the new term but it’s actual customer use cases are all DV ones.

              • Data mesh is driving a lot of leads to DV companies. It’s good marketecture. I like what Nikos says below — companies that do DG, DW, and DL right already federate a lot of capabilities, knowledge, and standards. How could you not? I think Occam’s razor applies here: the simplest solution is the best.

  5. I am happy that these ideas are being grouped together and re-branded as “Data Mesh” because I think there’s merit to them. They are not new ideas, e.g. aligning domains to business capabilities, cross-functional teams, achieving data interoperability via semantic metadata and master data management, centralised vs decentralised operating models for data governance & stewardship, etc. Neither are Domain-Driven Design ideas new, e.g. many of us already didn’t use a straw-man monolithic Enterprise Data Model, but a loosely coupled collection of subject area (domain) models, with controlled vocabularies and glossaries, concept models, business rules, cross-domain translation dictionaries and even upper ontologies to go with.

    Some of the potential issues with Data Mesh I see are:

    1. Convincing operational dev teams that they need to do their own data quality, stewardship, metadata and master data management, on top of their main day-to-day activities.
    2. Getting product owners to prioritise data management tasks over customer-focused user stories on any sprint. Expect regular and unpredictable lags on data governance across the business.
    3. Augmenting 2-pizza teams already staffed predominantly with “imperative-mindset” software developers (not being polemical) with a minority of “data-mindset” owners and stewards risks creating the known tensions between developer and DBA at distributed scale.
    4. Metadata and master/reference data management cannot necessarily happen in isolation within each domain.
    5. Proliferation of transformations across consuming domains will create inconsistent views in absence of precise data element semantics and enforcement thereof (may or may not be an issue).
    6. Business capability-aligned domains are, in essence, silos. The value stream that serves an end customer combines capabilities and those need to work flawlessly to achieve successful and efficient execution of the value stream. In terms of data, this means that unless perfect interoperability is achieved between Data Mesh “data products”, the combination of those to deliver end customer value can easily be disrupted and require manual intervention.

    • Nikos, you said it, thanks! “Business capability-aligned domains are, in essence, silos.”

      Yes, data mesh panders to the lowest common denominator, reinforcing an organization’s worst tendencies to silo data. However, data mesh technology (i.e., data virtualization) is a critical element of any data architecture, creating a data service that gives users and applications transparent access to distributed data. It federates data to support a holistic data environment.

  6. Pingback:Data Mesh: Centralized ownership vs decentralized ownership | James Serra's Blog

  7. Pingback:Data Mesh: Centralized vs decentralized data architecture | James Serra's Blog

  8. Pingback:The Future of the Modern Data Stack in 2022 – Data Science Austria

  9. Pingback:Data Mesh, towards a new paradigm in Data? - SogetiLabs

  10. Pingback:The Way forward for the Trendy Knowledge Stack in 2022 - Atlan - hapidzfadli

  11. Pingback:The Future of the Modern Data Stack in 2022 - Atlan - CW ORG

  12. Pingback:The Way forward for the Fashionable Information Stack in 2022 - Atlan - Lecheyre.ch

  13. Pingback:The Way forward for the Trendy Information Stack in 2022 - Atlan -

  14. Pingback:The Future of the Modern Data Stack in 2022 – Atlan – Apk Know

  15. Pingback:The Future of the Modern Data Stack in 2022 – Atlan | Technophoney

  16. Pingback:The Way forward for the Trendy Knowledge Stack in 2022 – Atlan – Floryup

  17. Pingback:The Future of the Modern Data Stack in 2022 – RadioStudio

Leave a Reply

Your email address will not be published. Required fields are marked *

HTML tags allowed in your comment: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>