HomeData LakeShould I load structured data into my data lake?

Comments

Should I load structured data into my data lake? — 4 Comments

  1. Pingback:When Structured Data Makes Sense In A Data Lake – Curated SQL

  2. Hi James,
    Great discussion within this article.
    We started a project a year ago with the idea of a data lake to first park data from source systems (source system zone) and second, replace the staging area of a new DWH.
    We still have discussions on a daily basis about the architecture of every new data source to be integrated for several reasons, i.e.
    – If a use case clearly needs a data set, it turns out to be better to import it directly into the DWH and if storage is a constraint move old data into HDFS. With polybase or similar technologies data is still available in DWH views.
    – If Data Scientists want to have a look first and find value in data then first in HDFS.
    – If resolution of data sets is too high, i.e. metering data with multiple values every second, depending of the use case put the raw data in HDFS and use for instance HBase or Spark if analysis on this level needs to be done. If other use cases need hourly or daily values, then aggregate these time series with something powerful like spark and store the results in DWH.

    The big efforts are :
    – Maintenance of several different components and technologies.
    – Skill Sets and workforce
    – Know how
    – Maintenance of multiple and heterogeneous ETL/ELT Interfaces
    – User management, multi tenancy and roles concept.
    – Data governance: specially in data lake and tools like notebooks.

    Data Solution Architecture became an extensive and interesting topic 😊

  3. Pingback:Gestructureerde data: direct naar DW of via Data Lake? - BITrainer.nl

Leave a Reply

Your email address will not be published. Required fields are marked *

HTML tags allowed in your comment: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>