Data Lake

datalake

Welcome to HackerNoon’s interview series. This section is curated by our Editorial team to spotlight bold and disruptive startups across the globe. If you’ve been nominated, and answer these questions Startups-Of-The-Year create a brand profile here. Hey Hackers, EU-funded startup Data Lake has been nominated in HackerNoon's annual Startup of the Year awards in Warsaw, Poland! We would be beyond grateful if you could vote for us here: https://hackernoon.com/startups/europe/europe-warsaw-poland Read more about us below to understand the amazing things we’re working on, and why we believe we deserve your vote. Meet Data Lake Founded by two medical doctors, is an EU-funded startup working to solve two of the biggest issues in medical research today: access to - and bias in - medical data. Data Lake Humanity is on the brink of some incredible developments in medical technologies and treatments - from AI algorithms detecting cancer to nano-robotics and smart sensors, we are looking at a future in which patient outcomes are hugely impacted by medical technology. Having access to high-quality and representative data, however, is of vital importance to the design, training, and implementation of these technologies. Up until recently, however, there was no way for researchers to legally and ethically access the datasets needed to build these incredible developments. Thanks to new EU legislation and our tools, we have created a way for researchers to do just that! Based on ethically-gathered digital consents, Data Lake enables patients to provide access to their medical data to the research entities and startups who are building our medical future. Along with the and more than 40 patient and healthcare organizations, we have that will soon allow anyone in the world to donate their medical data to science. Donate Your Data Foundation launched a global medical data donation platform My Role My name is Wojciech Sierocki, co-founder and CEO of Data Lake. I am a medical doctor by training but have previously founded several startups including ‘Treatment with a Mission’, at the time one of Poland’s largest NGOs operating in Africa. From the start of the COVID pandemic, I worked as a physician in the Emergency Department of the HCP Hospital in Poznan, Poland. While being able to make an immediate impact in someone’s life as a doctor was something I found immensely satisfying, seeing the impact of data on COVID outcomes I soon realized that my greater potential resided in building systems that could impact hundreds of thousands - if not millions - of people. Thus, along with my co-founder Dr. Ligia Kornowska, the idea for an international medical data donation program and Data Lake was born. How We're Disrupting the Medical Research Industry We have built an ethical medical data access layer that allows medical researchers (who are performing pro-patient research) to study large amounts of medical data that has previously never been studied. Based on the informed consent of patients who have the right to revoke it at any time, we can connect researchers to multi-source data sets that can be incrementally and regularly updated. This can take various forms, including the creation of privacy-focused patient registries that allow researchers to connect with patients suffering from ultra-rare diseases, as well as enabling a new era of digital clinical trials. Standing Out from The Crowd Securing access to medical data has been attempted - and failed - before. This has primarily been due to two factors - first, previous attempts have required patients and the general public to gain access to and then upload their medical data. For anyone who has ever tried to get their medical files, they’ll know just how difficult a proposition that is. Additionally, previous attempts have not sufficiently taken into account the aspects of privacy and transparency, often asking people to trust them regarding how and when their data is being used. We have solved both of these issues, using technology that allows for direct integration with hospitals for the transfer of medical records based on the express consent of patients, given digitally and recorded with distributed ledger technology. In short, we do all the heavy lifting by taking on both all of the legal and operational burdens. What really makes our approach different, however, is the focus on openness and transparency. We don’t want patients to have to trust that their consent (or revocation of consent) has been received, and therefore we chose blockchain technology to ensure a verifiable and trust-less system. By recording operations of consent on a public blockchain - without personally-identifiable information of course - we have removed the need for trust from the system entirely, allowing patients to know beyond a shadow of a doubt what their consent status is, and whether their medical data is being used for research or not. Our system not only respects patient agency and privacy, but it reduces the risk for researchers who know absolutely whether they have permission to study a dataset or not. Our Predictions/Thoughts on the HealthTech Industry in 2023 As with many industries, the healthcare industry will become increasingly reliant upon and driven by data in the coming years. A.I. algorithms, machine learning, and smart sensors are already online and assisting in the diagnosis and treatment of diseases, and with continuous and parallel innovations in nanorobotics and 3D printing, we are looking at a Science Fiction level of healthcare being possible within the next few years. Our main prediction - and something we are working to ensure - is that the ethical sourcing, storage, and analysis of data will become an increasing trend. Before we started and before key EU legislation was put in place, the idea of “consent-based data donation” was foreign to many people in the healthcare industry. Thanks in part to our efforts, we are hearing it talked about more and more at conferences and in the media. We believe that engaging people in what happens with their medical data and ensuring they are stakeholders is key to unlocking a new wave of medical innovation. We are convinced that you can have both privacy and innovation, and we believe the coming year will prove us right. What word defines the state of HealthTech in 2023? It’s hard to define in just one word, but we’ll give you three: “exciting, but critical.” The technological innovations we’ve talked about previously are incredibly exciting when you consider their potential for improving quality of life, patient outcomes, and ultimately, saving lives. We’re not exaggerating when we say we’re on the brink of being able to cure and treat many diseases previously considered unbeatable. But how we build these technologies and algorithms - particularly when it comes to the data we are using to create and train them - is of crucial importance. It’s why we’re working not just on the technological side of things, but also engaging on the ethics and rules for practical implementation of these exciting developments, such as my co-founder being a leading contributor to the . Whitepaper on AI in Clinical Practice Why we decided to participate in HackerNoon's Startup of the Year awards To be honest, one of our community members must have signed us up because it was a surprise to us! But a very pleasant one, because we believe that our technology and the idea of medical data donation have the power to save lives, and we want as many people out there to be aware of this growing movement. Final Thoughts We’re very thankful for the nomination and would like to thank everyone who has voted for us, and we hope that if you haven’t already you will consider doing so! We would also love for you to check out as well and sign up for the waiting list to become a medical data donor! Best of luck to everyone! https://donateyourdata.io Click here: Vote for us today!

Startups of the Year 2023: Data Lake - The Startup That Is Improving Access to Medical Research

The idea of an AI-driven robot taking over our healthcare – one of the most “human” fields that we have – is not a new one. Until recently, however, the idea remained relatively firmly in the realm of science fiction books and movies. But with the viral success of ChatGPT, the discussion of what this new generation of AI will be doing for us in the future has started to come up in the healthcare field. Now we’re not saying that you’ll be talking to a robot doctor at your next annual checkup any time soon, but we have to ask the question… Could ChatGPT be your doctor in the future? How Does ChatGPT Work? Without getting too deep into AI and Machine Learning, let’s define ChatGPT. The initials “GPT” stand for “Generative Pre-training Transformer,” three fundamental properties of the world’s most famous (currently) Artificial Intelligence that already provide a description of how it works: : More specifically than an AI, ChatGPT is a generative language model, a program that has been trained to hold natural conversations with humans. Generative : ChatGPT was pre-trained using the largest text dataset ever used for this purpose. Furthermore, the company behind ChatGPT, OpenAI, also trained its product through human feedback. Pre-Training : ChatGPT is built on transformers, which are a type of artificial neural network architecture that allows it to elaborate an input with the information it has been trained with and produce an output. Transformer How is ChatGPT Similar to Doctors? Simply put, ChatGPT must be trained with datasets in order to process inputs and generate answers that are understandable and accurate to humans. Sounds familiar? This process is similar to the path our society has outlined for anyone seeking to become a professional – in our case, a medical doctor. Doctors are among the professions that require the most years of university education, studying from books and through lessons (datasets) before they can finally talk to patients (receive inputs), provide their diagnosis, and recommend specific treatments (generate outputs). Only in its infancy, ChatGPT is already a remarkable product, but it is still far from being sufficiently advanced to trust with healthcare decisions or diagnoses. AIs require training on specialized data sets in order to perform specialized functions, and ChatGPT just isn’t trained for healthcare purposes. In fact, when we asked ChatGPT if it could provide medical diagnoses, it responded: “ .” No, I’m sorry, but as an AI language model, I am not qualified to make medical diagnoses. My programming and training are focused on understanding and processing natural language to provide information, answer questions, and assist with a wide range of tasks. However, I can provide you with general information on medical conditions and their symptoms, as well as advice on seeking professional medical help. If you have a specific medical concern, it’s always best to consult with a licensed medical professional who can properly diagnose and treat your condition For now, doctors all over the world can breathe easily. However, if a new version of ChatGPT was developed – one fed with a dataset of scientific information about healthcare and medicine that is thousands of times larger than any amount of material humans can effectively learn – it could mean a drastic and possibly rapid shift in how we receive our healthcare. If functionality included voice recognition and medical imaging machine learning, such a tool could not only perform initial patient assessments and triage but could potentially become a primary tool - or at least a supplementary tool - for diagnosis in hospitals and clinics. Human doctors will always have their advantages. Being people, they always have the capacity for empathy and provide the crucial human, emotional and psychological support that patients need to recuperate. Humans are also better at innovative ideas and improvisation, both of which are crucial in ERs, operating theaters, and many other areas of healthcare. But AI models like ChatGPT may soon be able to provide critical support to their work and may even be a potential solution to over-stressed healthcare networks and national services. The Shift to AI May Not be Seamless Nevertheless, the implementation of AI in our daily lives may not be as smooth as one might expect. For example, in order to have a trustworthy product, AIs must be trained with reliable medical data. Furthermore, several paradoxes may emerge from a new collaboration between humans and robots. For example, the AI may use its sophisticated algorithms and data analysis to come up with a diagnosis, whereas the medical doctor may rely on their professional experience and intuition to make a different diagnosis. Which diagnosis should take precedence if the doctor and the AI arrive at opposing conclusions? What happens if a doctor and AI team disagree with the diagnosis of another doctor and AI team? Even without a deep dive into the ethics and implementation, it is obvious that - before any AI integration occurs - experts and governments must discuss a robust framework to regulate scenarios like this one. For these and other reasons, such a change may appear radical or even dangerous to many, but when one zooms out and considers the incredible advancements in medicine over the last century, as well as the incredibly fast rate at which things continue to change today, we think it’s safe to assume that future generations will have AI robots like ChatGPT covering some diagnoses and taking care of their health on a daily basis. In less than 100 years, we’ve gone from the discovery of penicillin to custom-printed drugs created through genetics-powered personalized healthcare. In the last few decades alone, we’ve doubled cancer survival rates for almost every type of cancer. Nearly every discipline of our healthcare is seeing benefits from technology, as well as a shift towards data-driven, patient-centric medicine. AI will play a major part in the fourth industrial revolution, and both medical practice and research are already beginning to be more and more driven by these advancements. Multnomah County Hospital, 1948 ( ) vs. Modern Medical Facility ( ) Source Source Conclusion History repeats itself: as it happened with the introduction of computers and of the internet, AI and similar other disruptive technologies are inevitably going to cause rapid and profound changes in every aspect of our society. AI technology is already being used in many healthcare sectors to diagnose and treat illnesses, automate medical processes, to discover and print new drugs, and further advancements are likely to have a major impact on the way medical care is provided. is an EU-funded startup aiming to solve the issues of access to and bias in data in order to ensure these technological developments are equitable and trustworthy, and board chair Dr. Ligia Kornowska is also the leader of the “AI in Healthcare” Coalition. She was instrumental in the creation of the , discussing the benefits, concerns, and ethical guidelines required when it comes to the use of AI in healthcare. It will take many similar efforts from governmental institutions and citizen organizations to ensure AI is properly and ethically implemented, with sufficient oversight and a patient-centric focus. Data Lake “AI in Clinical Practice” whitepaper This is an exciting time for healthcare, and as we move towards a digitalized future, it is our duty as a society to prepare for the impact of these technologies and begin a discussion on how they can be useful for humankind in a safe manner. Could ChatGPT replace your doctor? Well, it’s certainly not going to happen today or any time soon. However, as the introduction of computers and the internet brought profound changes to the way we live, work, and interact, the introduction of AI is certain to have a similarly dramatic impact on medicine and healthcare. This article was written by the staff of Data Lake, an EU-funded startup working to provide ethical access to medical data for pro-patient research and development purposes. Also published . here

ChatGPT Could Have a Profound Impact on the Healthcare of the Future

Architecting a Modern Data Lake in a Post-Hadoop World

Mä Moderno Datos Quta Arquitectura ukax mä Post-Hadoop Mundo ukan luratawa

Post-Hadoop Dünyasında Müasir Məlumat Gölünün Memarlığı

Arhitektura modernog jezera podataka u post-Hadoop svijetu

Arkitektering af en moderne datasø i en post-hadoop verden

Arquitectura de un lago de datos moderno en un mundo post-Hadoop

ポストHadoopの世界における最新のデータレイクの設計

თანამედროვე მონაცემთა ტბის არქიტექტურა პოსტჰადოპის სამყაროში

Хадоопын дараах ертөнцөд орчин үеийн мэдээллийн нуурыг барих нь

Go aga Letsha la Datha la Sebjalebjale Lefaseng la Ka morago ga Hadoop

Kubaka Ikiyaga kigezweho muri nyuma yisi ya Hadoop

Arkitektimi i një liqeni modern të të dhënave në një botë post-Hadoop

Pag-arkitekto ng Modernong Data Lake sa isang Post-Hadoop World

An Architect’s Guide to the Top 10 Tools Needed to Build the Modern Data Lake

আধুনিক ডেটা লেক তৈরির জন্য প্রয়োজনীয় শীর্ষ 10টি সরঞ্জামের জন্য একজন স্থপতির গাইড

Ein Architektenhandbuch zu den 10 wichtigsten Tools zum Aufbau eines modernen Data Lake

Guía para arquitectos sobre las 10 herramientas principales necesarias para crear un lago de datos moderno

Guide de l’architecte sur les 10 principaux outils nécessaires à la création du lac de données moderne

आधुनिक डेटा लेक के निर्माण के लिए आवश्यक शीर्ष 10 उपकरणों के लिए एक आर्किटेक्ट गाइड

現代のデータレイクを構築するために必要なトップ 10 ツールに関するアーキテクト ガイド

Um guia de arquiteto para as 10 principais ferramentas necessárias para construir o moderno Data Lake

Руководство архитектора по 10 лучшим инструментам, необходимым для создания современного озера данных

Modern Veri Gölünü Oluşturmak İçin Gereken En İyi 10 Araca Yönelik Bir Mimarın Rehberi

Hướng dẫn của Kiến trúc sư về 10 Công cụ hàng đầu cần thiết để Xây dựng Hồ dữ liệu hiện đại

架构师指南：构建现代数据湖所需的十大工具

Build a Data Warehouse Not to Drown in Your Data Lake

An Architect’s Guide to Building Reference Architecture for an AI/ML Datalake

এআই/এমএল ডাটালেকের জন্য রেফারেন্স আর্কিটেকচার তৈরির জন্য একজন স্থপতির গাইড

Ein Architektenhandbuch zum Erstellen einer Referenzarchitektur für einen KI/ML-Datalake

Una guía del arquitecto para crear una arquitectura de referencia para un lago de datos de IA/ML

Guide de l'architecte pour créer une architecture de référence pour un datalake IA/ML

AI/ML डेटालेक के लिए संदर्भ आर्किटेक्चर बनाने के लिए आर्किटेक्ट की मार्गदर्शिका

AI/ML データレイクのリファレンス アーキテクチャを構築するためのアーキテクト ガイド

AI/ML 데이터레이크용 참조 아키텍처 구축을 위한 설계자 가이드

Guia do arquiteto para construir arquitetura de referência para um Datalake de IA/ML

Руководство архитектора по созданию эталонной архитектуры для озера данных AI/ML

AI/ML Datalake için Referans Mimarisi Oluşturmaya Yönelik Mimar Kılavuzu

Hướng dẫn của Kiến trúc sư về Xây dựng Kiến trúc Tham chiếu cho Datalake AI/ML

架构师指南：构建 AI/ML 数据湖参考架构

The Architect’s Guide to A Modern Datalake Reference Architecture

Developing Next-Gen Data Solutions: SingleStore, MinIO, and the Modern Datalake Stack

নেক্সট-জেন ডেটা সলিউশন তৈরি করা: একক স্টোর, মিনিও এবং আধুনিক ডেটালেক স্ট্যাক

Entwicklung von Datenlösungen der nächsten Generation: SingleStore, MinIO und der moderne Datalake-Stack

Desarrollo de soluciones de datos de próxima generación: SingleStore, MinIO y Modern Datalake Stack

Développer des solutions de données de nouvelle génération : SingleStore, MinIO et la pile Datalake moderne

अगली पीढ़ी के डेटा समाधान विकसित करना: सिंगलस्टोर, मिनियो और आधुनिक डेटालेक स्टैक

次世代データソリューションの開発: SingleStore、MinIO、最新のデータレイク スタック

차세대 데이터 솔루션 개발: SingleStore, MinIO 및 최신 Datalake 스택

Desenvolvendo soluções de dados de última geração: SingleStore, MinIO e Modern Datalake Stack

Разработка решений для обработки данных следующего поколения: SingleStore, MinIO и современный стек Datalake

Yeni Nesil Veri Çözümlerinin Geliştirilmesi: SingleStore, MinIO ve Modern Datalake Stack

Phát triển các giải pháp dữ liệu thế hệ tiếp theo: SingleStore, MinIO và Datalake Stack hiện đại

开发下一代数据解决方案：SingleStore、MinIO 和现代 Datalake 堆栈

Many of us have made changes to data based on requirements that later evolve. By the time we realize it, it becomes impossible to roll back. Change isn't just a constant; it's an essential part of managing data that demands a sophisticated approach. Enter the Git-like functionality of Dremio's data catalog Nessie. Just as Git has become fundamental to software development, data engineers need similar tools to work in parallel, compare data versions, promote changes to production and roll back data when needed. Nessie provides data engineers with a Git-like version control system for managing data versions, branches, merges, and commits. This can be very helpful when multiple data engineers are working with and transforming data at the same time. Nessie allows each engineer to work in separate branches while maintaining a single source of truth in the form of the main branch. This functionality empowers data engineering teams to maintain data quality collaboratively in the face of unrelenting change. This article provides a step-by-step guide demonstrating how Nessie, Dremio and MinIO work together to enhance data quality and collaboration in your data engineering workflows. Whether you're a data engineer, ML engineer, or just a modern data lake enthusiast, this blog equips you with the knowledge and tools needed to effectively enhance your data versioning practices. Understanding the Basics: Nessie Workloads Nessie allows for Git-like workloads that let you test, develop and push to production. Let's break down some key concepts: Just like in Git, branches in Nessie allow you to work on different data engineering tasks concurrently. For example, you might have a branch for feature development, data cleaning and data transformation. Each branch can have its own set of data changes. Branches: In Nessie, a commit represents a snapshot of the data at a specific point in time. When you make changes to your data, you create a new commit, which records those changes. Commits are linked to a specific branch. When you need to revert your data to a stable or known-good state, Nessie makes it easy to select a specific commit and roll back to that data version. This ensures data quality and consistency. Commits: : Nessie allows you to merge the changes from one branch into another. This is similar to merging code changes in Git. When you merge a branch, the data changes made in that branch become part of the target branch. Merges The Components is a distributed analytics engine that operates as an open-source platform, offering an intuitive self-service interface for data exploration, transformation and collaborative work. Its design is grounded in a high-speed columnar memory format. Dremio Apache Arrow, We’ve already explored how to deploy Dremio with Kubernetes, and also how to . For additional information about Dremio, consult . query Iceberg tables on MinIO with Dremio Dremio Resources is high-performance object storage. Renowned for its exceptional and , MinIO serves as a pivotal component in building and maintaining modern data lake infrastructures. MinIO empowers architects to efficiently manage and store massive volumes of data on-prem, on bare metal, on the edge, or on any of the public clouds. MinIO speed scalability is an open table format suited for managing large volumes of data in data lakes. Unique features like time travel, dynamic schema evolution, and Partition Evolution make it a game-changer, allowing query engines to work concurrently on the same data safely and efficiently. See for more information about Iceberg features. Apache Iceberg MinIO The Definitive Guide to Lakehouse Architecture with Iceberg and Getting Started You’ll need the and for this tutorial. The easiest way to get both if you don’t already have them is to install Docker Desktop. Docker Engine Docker Compose This part of the tutorial is based on Dremio’s . You can get the repo for this project . blog post here To begin, open a terminal and navigate to the folder where you cloned/downloaded the repo, and run the command below to start up Dremio. docker-compose up dremio Wait a few moments and then navigate to to access Dremio. Update the requested fields and then click Next. http://localhost:9047 Next, run the following command to start MinIO. docker-compose up minio The final command below will start up Nessie. docker-compose up docker-compose up nessie Navigate to to log in to MinIO with the username and password of . You’ll be prompted to create a bucket. http://localhost:9001 minioadmin:minioadmin Make one named . iceberg-datalake Then head back over to Dremio at and click on Add Source and select Nessie. http://localhost:9047 Set the Name to nessie Set the endpoint URL to http://nessie:19120/api/v2 Set the authentication to none Don’t click Save yet. Instead, in the navigation panel on the left, click Storage. MinIO is S3-API compatible object storage and can use the same connection pathways as AWS S3. For your access key, set minioadmin For your secret key, set minioadmin Set root path to /iceberg-datalake Scroll down for the next set of instructions. Click on the button under to create and configure the following properties. Add Property Connection Properties to fs.s3a.path.style.access true to fs.s3a.endpoint minio:9000 to dremio.s3.compat true Uncheck Encrypt connection Then click Save. You should now see Nessie Catalogs in your data sources. Create a Data Source In Dremio navigate to SQL Runner on the left-hand side. Make sure that in the upper right area of the text editor is set to our Nessie source. Otherwise, you will have to reference the context like instead of just to run this query. Copy and paste the SQL below and run. Context nessie.SalesData SalesData CREATE TABLE SalesData (
    id INT,
    product_name VARCHAR,
    sales_amount DECIMAL,
    transaction_date DATE
) PARTITION BY (transaction_date); Run the query below to insert data into the table you just created. INSERT INTO SalesData (id, product_name, sales_amount, transaction_date)
VALUES
    (1, 'ProductA', 1500.00, '2023-10-15'),
    (2, 'ProductB', 2000.00, '2023-10-15'),
    (3, 'ProductA', 1200.00, '2023-10-16'),
    (4, 'ProductC', 1800.00, '2023-10-16'),
    (5, 'ProductB', 2200.00, '2023-10-17'); Navigate back to MinIO to see that your data lake has been populated with the Iceberg tables. Branching and Merging with Nessie Return to Dremio at . Begin by querying a table on the main branch using the syntax: http://localhost:9047 AT BRANCH SELECT * FROM  nessie.SalesData AT BRANCH main; Create an ETL (Extract Transform and Load) branch to allow you to play around with and transform data without impacting production. CREATE BRANCH etl_06092023 in nessie Within the ETL branch, insert new data into the table: USE BRANCH etl_06092023 in nessie;
INSERT INTO nessie.SalesData (id, product_name, sales_amount, transaction_date) VALUES
(6, 'ProductC', 1400.00, '2023-10-18'); Confirm the immediate availability of the new data within the ETL branch: SELECT * FROM nessie.SalesData AT BRANCH etl_06092023; Note the isolation of changes from users on the main branch: SELECT * FROM nessie.SalesData AT BRANCH main; Merge the changes from the ETL branch back into the main branch: MERGE BRANCH etl_06092023 INTO main in nessie; Select the main branch again to see that the changes have indeed been merged. SELECT * FROM nessie.SalesData AT BRANCH main This branching strategy enables data engineers to independently handle numerous transactions across multiple tables. When they are ready, data engineers can merge these transactions into a single, comprehensive multi-table transaction within the main branch. Conclusion This blog post delved into the power of Git-like version control in data engineering, emphasizing how seamlessly manages data versions, branches and merges. This step-by-step guide demonstrates how Nessie, in collaboration with Dremio and MinIO, as the object storage foundation, enhances data quality and collaboration in data engineering workflows. Nessie Let us know what your data lakehouse looks like at or on . hello@minio.io our Slack channel Also appears . here

Data Lake Mysteries Revealed: Nessie, Dremio, and MinIO Make Waves

ডেটা লেক রহস্য প্রকাশিত হয়েছে: নেসি, ড্রেমিও এবং মিনিও মেক ওয়েভস

Data-Lake-Rätsel aufgedeckt: Nessie, Dremio und MinIO sorgen für Aufsehen

Misterios del lago de datos revelados: Nessie, Dremio y MinIO hacen olas

Les mystères du Data Lake révélés : Nessie, Dremio et MinIO font des vagues

डेटा लेक रहस्य का खुलासा: नेस्सी, ड्रेमियो और मिनिओ मेक वेव्स

データレイクの謎が明らかに: Nessie、Dremio、MinIO が波紋を起こす

데이터 레이크의 미스터리 공개: Nessie, Dremio 및 MinIO가 물결을 일으키다

Mistérios do Data Lake revelados: Nessie, Dremio e MinIO fazem ondas

Тайны озера данных раскрыты: Несси, Дремио и MinIO вызывают фурор

Data Lake'in Gizemleri Ortaya Çıktı: Nessie, Dremio ve MinIO Dalga Yaratıyor

Những bí ẩn về hồ dữ liệu được tiết lộ: Nessie, Dremio và MinIO tạo nên làn sóng

数据湖谜团揭开：Nessie、Dremio 和 MinIO 掀起波澜

Amid the fervor to adopt AI is a critical and often overlooked truth -  the success of any AI initiative is intrinsically tied to the quality, reliability and performance of the underlying data infrastructure. If you don't have the proper foundation, you are limited in what you can build and therefore what you can achieve. Your data infrastructure is the bedrock upon which your entire AI infrastructure is built. It's where data is collected, stored, processed, and transformed. Training models using supervised, unsupervised, and reinforcement learning require storage solutions that can handle structured data - like a Data Warehouse. On the other hand, if you are training Large Language Models (LLMs), you must manage unstructured data - documents in their raw and processed form. A Modern Datalake, or Lakehouse, is the foundation for both these different flavors of AI. A Modern Datalake is one-half Data Warehouse and one-half Data Lake and uses object storage for everything. More recently, we have seen the rise of open table formats. Open Table Formats (OTFs) like Apache Iceberg, Apache Hudi, and Delta Lake make it seamless for object storage to be used within a data warehouse. The rest of this article will examine how to leverage the characteristics of the Modern Datalake that differentiate it from conventional solutions like proprietary Data Warehouses and appliances. To build a foundation for AI infrastructure you need the following: Disaggregation of Compute and Storage Scale Out (Not Up) Software Defined Cloud Native Commodity Hardware If we agree on the above, a series of best practices emerge that focus on two areas of performance. If incorporated, the modern Datalake will be both fast and scalable. These best practices include: Optimizing Drive Price and Performance Incorporate a High-Speed Network Disaggregating Compute and Storage Disaggregating compute and storage within your data infrastructure means that separate resources are used for compute and storage. This contrasts conventional storage solutions, where everything comes packaged in a single server or, worse yet, an appliance. However, Modern Datalakes take disaggregation to another level. If the Data Lake and the Data Warehouse had completely different storage requirements, we could use two separate instances of an object store, as shown below. Additionally, If the Data Warehouse needs to support workloads that require conflicting configurations, then you could use more than one processing engine. This is shown below. A composable infrastructure allows you to scale your compute and storage resources independently. This means you can allocate more resources to the part of your infrastructure that needs it most rather than upgrading both compute and storage together. It results in cost-effective scaling as you only invest in the required resources. Scale-Out Not Up AI workloads are data intensive, are often distributed across multiple CPUs or GPUs, use copious compute power for training, and necessitate real-time inference. Scaling out, not up, helps optimize performance and accommodate high-speed networks. Scaling out and scaling up are two different approaches to increasing the capacity and performance of your data infrastructure. However, scaling out is proving to be the more viable approach as advances are made in clustering platforms like Kubernetes and more and more solutions strive to be cloud native. Scaling out in a disaggregated infrastructure provides: - If one node is busy, another node can take on a new request, reducing wait times and increasing throughput. If one node fails, the workload can be shifted to other nodes, reducing downtime and ensuring continuity. High Availability and Fault Tolerance - Scaling out can provide better performance by distributing workloads across multiple nodes or servers to handle larger volumes of data and more concurrent requests. Scaling out is also more flexible because you can add or remove nodes as needed, making it easier to adjust to fluctuating workloads or to accommodate seasonal variations. Performance and Flexibility - Maintenance and upgrades are simplified when you scale out. Instead of taking a critical system offline for upgrades, you can perform maintenance on individual storage or compute nodes without disrupting the entire infrastructure. Operationally and Resource Efficient Cloud Native + Software Defined The last component of leveraging the Modern Datalake to build a strong foundation for AI is taking a cloud-native, software-defined approach. Containers like Docker and container orchestration tools, like Kubernetes, make cloud-native architectures possible. All the components of a Modern Datalake run in containers that run in Kubernetes. Therefore, a Modern Datalake is cloud native. "Software-defined" refers to an approach in which software controls and manages the configuration, functionality, and behavior of hardware components, often in the context of computer systems and networking. This is the building block of the infrastructure as code movement where the emphasis is on smart software and dumb fast hardware. Software-defined storage abstracts and manages the storage resources through software, making allocating and managing storage capacity across different devices and storage media easier. Built for Speed: NVMe and 100GbE To take full advantage of your commodity hardware and software-defined architecture - you need two more key pieces. The first is NVMe drives. Modern, performance oriented workloads, the random nature of read/write, the rise of small objects and falling SSD pricing all favor a . Do that math, the upfront may be higher, the TCO will be lower. NVMe centric architecture The second component is 100GbE networking. In a software-defined world, the network turns out to be the bottleneck in many setups even at 100GbE. Here are some of those scenarios: - AI workloads often process massive datasets, such as images, videos, natural language text, and sensor data. High-speed networks can quickly transfer these large datasets between storage and processing units, reducing data transfer bottlenecks. Data Intensive - Many AI tasks involve distributed computing across multiple CPUs or GPUs. High-speed networks enable efficient communication and data exchange between these devices, ensuring the computing clusters work in parallel effectively. Distributed Computing - Training deep learning models, especially LLMs like transformers or convolutional neural networks, requires a lot of data and computational power. A high-speed network allows faster data loading and synchronization between distributed GPUs, which can significantly speed up training times. Model Training - Low-latency and high-throughput networks are essential for responsive applications that incorporate AI. A high-speed network ensures minimal delay between a user request and a response from a model. Real-time Inference Foundational Concepts By adhering to these principles: the disaggregation of compute and storage, scale-out, not up, dumb, fast hardware and smart cloud native software the enterprise can construct a Modern Datalake that has the right foundation for meeting these requirements and pushing your AI initiatives forward. You cannot construct a building on a poor foundation, just ask the ancient Egyptians. The AI game is about performance at scale, and this requires the right foundation. To skimp on the foundation is to accumulate technical debt that will topple your Jenga tower after a few minutes. Build smart, put the foundation in place.

The Principles to Keep In Mind When Building a Modern Datalake for Your AI Infrastructure

আপনার AI পরিকাঠামোর জন্য একটি আধুনিক ডেটালেক তৈরি করার সময় যে নীতিগুলি মাথায় রাখতে হবে

Die Grundsätze, die Sie beim Aufbau eines modernen Datalake für Ihre KI-Infrastruktur beachten sollten

Los principios a tener en cuenta al construir un lago de datos moderno para su infraestructura de IA

Les principes à garder à l'esprit lors de la création d'un datalake moderne pour votre infrastructure d'IA

आपके एआई इन्फ्रास्ट्रक्चर के लिए आधुनिक डेटालेक का निर्माण करते समय ध्यान में रखने योग्य सिद्धांत

AI インフラストラクチャ用の最新のデータレイクを構築する際に留意すべき原則

AI 인프라를 위한 최신 데이터레이크를 구축할 때 명심해야 할 원칙

Os princípios a serem considerados ao construir um Datalake moderno para sua infraestrutura de IA

Принципы, которые следует учитывать при создании современного озера данных для вашей инфраструктуры искусственного интеллекта

Yapay Zeka Altyapınız için Modern Bir Datalake Oluştururken Akılda Tutulması Gereken İlkeler

Những nguyên tắc cần ghi nhớ khi xây dựng Datalake hiện đại cho cơ sở hạ tầng AI của bạn

为人工智能基础设施构建现代数据湖时要牢记的原则

Stories, News & Mentions about DATALAKE

Datalake’s Stories on HackerNoon