Skip to content
All things DataOS
Pulsar
Initializing search
Home
Getting Started
Data Product
Glossary
Learn
All things DataOS
Home
Home
Philosophy
Architecture
Interfaces
Interfaces
Bifrost
Bifrost
ABAC Implementation
Users
Use-cases
Grant
Role
Grant Requests
Heimdall Primitives
Roles and Use cases
CLI
CLI
Installation
Initialization
Command Reference
Data Product Hub
Data Product Hub
Core Concepts
Data Product Discovery
Data Product Exploration
Data Product Activation
Data Product Activation
BI Sync
BI Sync
Tableau Cloud
Power BI
Power BI
Power BI Desktop
Power BI Service
Tableau Desktop
Microsoft Excel
AI/ML
Postgres
Graphql
Data APIs
Best Practices
Examples
Metis
Metis
Metis UI
Metis UI
Pre-defined Roles and Policies
Key Features
Metadata of Products
Metadata of Products
Metadata of Data Products
Metadata of Data Assets
Metadata of Data Assets
Metadata of Tables
Metadata of Topics
Metadata of Dashboards
Metadata of Resources
Metadata of Resources
Bundles
Clusters
Computes
Databases
Depots
Instance Secrets
Lakehouses
Lenses
Monitors
Operators
Pagers
Policies
Secrets
Services
Stacks
Workers
Workflows
Operations
Operations
Workbench
Workbench
Resources
Resources
Types
Types
Bundle
Bundle
Configurations
Templates
Cluster
Cluster
Configurations
Configurations
Manifest Reference
Connectors
Integration
Integration
Power BI
Power BI
Using Connector
Using ODBC Driver
Tableau
IBM SPSS
CLI Client
Tuning
Tuning
Query Execution Optimization
Performance Tuning
Recommended Configuration
Examples
Examples
Restart and scale Cluster on a pre-defined schedule
Multiple Cluster Setup using single manifest
Core Concepts
Core Concepts
On-demand Computing
Compute
Compute
Configurations
Templates
Database
Database
First Steps
Configurations
Recipes
Recipes
Query Database using Workbench
Building Streamlit applications backed by Database
Depot
Depot
Quickstart
Supported Sources
Supported Sources
ABFSS
Amazon Redshift
Amazon S3
Apache Pulsar
Azure WASBS
Elasticsearch
Eventhub
Google BigQuery
GCS
JDBC
Kafka
MongoDB
Microsoft SQL Server
MySQL
Opensearch
Oracle
PostgreSQL
Snowflake
Supported Table Format
Supported Table Format
Iceberg
Delta
Configurations
Governance
Observability
Cataloging
Supported Connectors
Usecase
Usage
Usage
Data ingestion and transformation
Building semantic models
Query the source
Accelerating queries
Building Data APIs
Depot as Lakehouse
Scanning metadata
Grant
Grant
Configuration
Instance Secret
Instance Secret
Quickstart
Data Sources
Data Sources
ABFSS
Amazon Redshift
Amazon S3
Azure WASBS
Elasticsearch
Eventhub
Google BigQuery
GCS
JDBC
Kafka
MongoDB
Microsoft SQL Server
MySQL
Opensearch
Oracle
PostgreSQL
Snowflake
Repositories
Repositories
AWS Codecommit
Bitbucket
Git
Configurations
Governance
Observability
Cataloging
Usage
Lakehouse
Lakehouse
Configurations
Configurations
Manifest Reference
Command Reference
Iceberg Metadata Tables
Recipes
Recipes
Ensure high data quality using Write-Audit-Publish (WAP) Pattern
Use Iceberg metadata tables to extract insights
Create, Get, and Drop Dataset
Table Properties
Schema Evolution
Partitioning
Maintenance (Snapshots and Metadata listing)
Lens
Lens
Semantic Modeling
Semantic Modeling
Key concepts
Segments
Views
User Groups
Supported Data Sources
Supported Data Sources
Bigquery
Flash
Minerva
Postgres
Redshift
Snowflake
Themis
Exploration
Exploration
Using GraphQL
Using GraphQL
Overview
Examples
Using Python
Using SQL APIs
Using SQL APIs
Overview
Supported functions and operators
Using REST APIs
Using REST APIs
API Endpoints and Scopes
Exploration
BI Integrations
BI Integrations
Power BI
Power BI
Power BI Desktop
Power BI Service
Superset
Tableau
References
References
Best Practices
Errors
Optimizing Semantic Model
Aggregation Strategies
Complex Analysis
Row Multiplication Handling
Configurations
Cataloging
Governance
Observability
Monitor
Monitor
Types
Types
Equation Monitor
Report Monitor
Stream Monitor
First Steps
Configurations
Recipes
Recipes
Add context columns to Monitors
Create a Report Monitor
Create a Stream Monitor
Create an Equation Monitor on Lens
Create an Equation Monitor on Postgres
Generate incidents for certificate expiration
Operator
Operator
Configurations
Recipes
Recipes
Orchestrate Hightouch pipeline using Hightouch Factory Operator
Pager
Pager
Configurations
Recipes
Recipes
Create a Custom Body Template for Pager
Policy
Policy
Core Concepts
First Steps
Configurations
Templates
References
References
Rule for and or relationship
Recipes
Recipes
End-to-end use-case on how to implement access policy
Secret
Secret
First Steps
Configurations
Recipes
Recipes
Refer secrets in other DataOS Resources
Setup secrets to pull images from a private container registry
Service
Service
Configurations
Stacks
Stacks
Beacon
Beacon
Bento
Bento
Setting Up Bento
Setting Up Bento
Bento on Local Host
Bento on DataOS
Bloblang
Bloblang
Overview
Core Features
Functions
Methods
Arithmetic
Advanced Bloblang
Configurations
Configurations
Cloud Credentials
Dynamic Inputs and Outputs
Error Handling
Fields Paths
Interpolation
Message Batching
Metadata
Monitoring
Performance Tuning
Processing Pipelines
Resources
Secrets
Streams Mode
Streams Mode
Streams API
Streams Via Config Files
Streams Via REST API
Synchronous Responses
Templating
Unit Testing
Using CUE
Window Processing
Components
Components
HTTP
Inputs
Logger
Metrics
Output
Output
AWS S3
Azure Blob Storage
Fastbase Depot
Kafka
kafka Depot
Processors
Rate Limit
Recipes
Recipes
Fetching Data from Instagram API
Twitter API Data Processing
Discord Bot
Stock Data API to Lakehouse
Performing Rate Limit
Performing Pagination
CLI Stack
CLI Stack
Container
Container
DBT
DBT
Flare
Flare
Creating Flare Jobs
Configurations
Data Sources
Data Sources
Databases
Databases
PostgreSQL
MySQL
Oracle
MS-SQL
MongoDB
Data Warehouses
Data Warehouses
Snowflake
BigQuery
Amazon Redshift
Object Storage
Object Storage
Object Storage Configuration
Object Storage Configuration
Schema Configuration
Data Format Configuration
Delta Tables
Streaming Sources
Streaming Sources
Kafka
EventHub
Pulsar
Search Engines
Search Engines
Elasticsearch
OpenSearch
Flare Functions
Flare Functions
List of Flare Functions
Use Cases & Examples
Use Cases & Examples
Data Ingestion
Data Ingestion
Batch Jobs
Incremental Jobs
Streaming Jobs
Data Transformation
Data Transformation
Iceberg Operations
Merge Operations
Data Replay
Concurrent Writes
Column Tagging
Data Syndication
Enrichment
Fingerprinting
Performance Optimization
Performance Optimization
Partitioning
Partition Evolution
Bucketing
Caching
Compression
Job Tuning
Data Maintenance
Data Maintenance
Data Compaction
Manifest Rewrite
Delete from Dataset
Expire Snapshots
Remove Orphans
Optimizing Flare
Optimizing Flare
Performance Tuning
Performance Tuning
Configuration Tuning
SQL Optimization
Flash
Flash
Create a Flash Service
Configurations
Supported Data Sources
Monitoring Flash
Best Practices
Do's and Dont's
Errors and Issues
Errors and Issues
Handling Empty Tables
Recipes
Recipes
Use cached datasets in Lens models
Use cached datasets in Talos APIs
Lakesearch
Lakesearch
Key concepts
Set up
Set up
Lakesearch Service
Query Rewriter
Configurations
Governance
Index Searching
Observability
Troubleshooting
Scanner
Scanner
Quickstart
Core Concepts
Supported Sources
Supported Sources
Data Sources
Data Sources
AzureSQL
BigQuery
Kafka
Lakehouse
MariaDB
MSSQL
MySQL
Oracle
PostgreSQL
Pulsar
Redshift
Snowflake
System Sources
System Sources
Data Products
User Information
Indexer Services
Indexer Services
Data Profiling
Quality Checks
Resources Scanner
Query Usage
Configurations
Governance
Cataloging
Best Practices
Observability
Recipes
Soda
Soda
Steampipe
Steampipe
Talos
Talos
Setting Up Talos
Supported Sources
Supported Sources
Bigquery
Flash
Lens
MySQL
Postgres
Redshift
Snowflake
Configurations
Configurations
config.yaml
apis
service.yaml
Governance
Recipes
Recipes
Apply data masking
Fetch data from third party tools
Caching datasets
Generate API documentation
External API as data source
Validating Parameters
Error Handling
Example
Best Practices
Best Practices
API
SQL
Custom Stack
Custom Stack
How to create your own Stack?
Configurations
Volume
Volume
Getting Started
Configurations
Manage the Volume
Manage the Volume
Delete Azure PVC
Delete AWS PVC
Worker
Worker
First Steps
Configuration
Core Concepts
Workflow
Workflow
First Steps
Configurations
Recipes
Recipes
Implement Single-run Workflow
Run Cron/Scheduled Workflow
Orchestrate multiple Workflows from a single Workflow
Retry a Job in Workflow
Characteristics
Configuration
Best practices for YAML
SDK
SDK
DataOS Python SDK
DataOS PyFlare SDK
DataOS PyFlare SDK
Supported Sources
Supported Sources
Bigquery
Lakehouse
Postgres
Snowflake
Troubleshooting
Getting Started
Getting Started
Data Product Consumer
Data Product Developer
Operator
Data Product
Data Product
Core Concepts
Data Product Lifecycle
Data Product Lifecycle
Design Phase
Build Phase
Deploy Phase
Iteration & Optimization
Data Product Examples
Data Product Examples
Building from Scratch with Snowflake
Using DataOS Lakehouse as a Source
Configurations
Observability
Recipes
Recipes
Deploy Data Product using CI/CD pipeline
Glossary
Learn
Learn
DataOS Primer
DataOS Primer
DataOS essentials
Learning tracks
Learning tracks
Data Product Consumer Track
Data Product Consumer Track
Learning path overview
Learning path overview
Discovering Data Products
Discovering Data Products
Finding the right Data Product
View Data Product information
Know about I/O datasets
Explore semantic model
Explore semantic model
API integration
Underastand quality
Connnect Data Products with BI tools and applications
Connnect Data Products with BI tools and applications
Integration with BI tools
Integration with BI tools
Integration with AI/ML
Integration with AI/ML
Integration with Postgres
Integration with Postgres
Integration with GraphQL
Integration with GraphQL
Integration with LakeSearch
Integration with LakeSearch
Integration with Data APIs
Integration with Data APIs
Custom apps to consume Data Products
Custom apps to consume Data Products
Building an App with Appsmith
Building an App with Appsmith
Data Product Foundations Track
Data Product Foundations Track
Foundations- Course I
Foundations- Course I
Source-aligned Data Product
Data source connectivity
Data source connectivity
Set up Depots
Create Scanner
Build Pipeline
Build Pipeline
Scheduling pipelines
Quality check
Set up monitoring
Set up monitoring
Quality check observability
Deploy Your first Source-aligned Data Products
Deploy Your first Source-aligned Data Products
Create Data Product Spec
Deploy Data Product
Foundations- Course II
Foundations- Course II
Capture objectives, KPIs & metrics
Design Data Product
Repository and folder structure
Source data connectivity
Build pipeline
Create semantic model
Deploy and register
Data Product Developer Track
Data Product Developer Track
Data Product Owner Track
Data Product Owner Track
Operator Learn Track
Operator Learn Track
Learning path overview
Learning path overview
Access control
Alerts Grafana
Cluster management
Cluster tuning
Cred security
Data source connectivity
Keycloak
Lakehouse insights
Routine check
Upgrade rollback
Quick Start Guides
Quick Start Guides
Connecting with data sources
Connecting with data sources
Performing exploratory data analysis
Performing exploratory data analysis
Running your first data query
Running your first data query
Arranging and transforming data with Pivot
Arranging and transforming data with Pivot
Writing complex queries
Writing complex queries
Querying diverse data sources
Querying diverse data sources
Scanning metadata
Creating ETL pipelines
Creating ETL pipelines
Ingest external files into DataOS
Ingest external files into DataOS
Batch data processing using Flare
Batch data processing using Flare
Storing query results as dataset with Flare
Storing query results as dataset with Flare
Deploying and securing data applications
Deploying and securing data applications
Deploying data application using Container Stack
Deploying data application using Container Stack
Listing Your App on DataOS Home
Listing Your App on DataOS Home
Securing deployed data applications
Securing deployed data applications
Videos
Pulsar
¶
Was this page helpful?
Thanks for your feedback!
Thanks for your feedback!
Back to top