Data Product Developer Learning Track¶
This track is designed to equip you with the skills needed to develop robust and scalable Data Products using DataOS.
By the end of this course, you'll be able to:
-
Understand business needs: Align Data Products with business goals and define quality and security requirements.
-
Design effective data models: Use DataOS Metis and Workbench to analyze and refine data assets.
-
Build and implement Data Products: Develop logical semantic models, data pipelines, and apply access control.
-
Deploy and manage Data Products: Utilize CI/CD practices for seamless deployment and updates.
Module 1: Understanding data needs¶
To create a successful Data Product, it’s crucial to first understand the business context and requirements. This module focuses on capturing key needs and expectations for Data Products for your organization based on Righ to Left approach:
Module 2: Designing Data Products¶
This module dives into the detailed process of designing Data Products tailored to meet specific business needs. It covers every aspect from defining use cases, and discovering existing resources, to creating a robust design and ensuring quality, security, and compliance.
Module 3: Building Data Products¶
In the previous module, we explored the essential steps for designing a Data Product. We started by defining the use case of Customer Purchase Behavior for an online retail company. This use case focuses on analyzing customer buying patterns to identify cross-sell opportunities and enhance marketing strategies.
Now that we have a clear design plan, it’s time to move into the development phase. In this module, we will build the Data Product step by step, setting up the necessary components within DataOS.
You should follow the different topics in the given order to ensure a smooth and systematic approach to building the Data Product.
Module 4: Deploying Data Products¶
The final step is to deploy the Data Product, making it available for consumption and ensuring it can be easily managed and updated.
Checklist for success¶
To complete the Data Product Developer learning track, ensure you accomplish the following tasks using DataOS components:
Verify DataOS CLI installation and initialization.
Create Depot manifests using Instance Secrets to store credentials securely.
Explore Data on Workbench and Access Metis for Quality and Lineage to ensure suitability with the use case.
Create pipelines (Workflows) for data ingestion (if the use case requires), ensuring desired schedule and observability.
Define and implement SLOs for accuracy, freshness, completeness, uniqueness, schema, and validity using Soda Workflow using SodaCL.
Monitor Quality Checks in Metis with SLO-based results and trend analysis via detailed metrics and charts.
Write SQL Scripts to extract necessary columns and define logical table manifests along with metrics.
Define segments and manage access controls with user groups.
Build Data APIs to create data applications.
Deploy the Data Product and verify that it is visible and accessible to intended users.