Skip to content

All things DataOS

Data Sources

Initializing search

Home
Getting Started
Data Product
Glossary
Learn

All things DataOS

Home
Home
- Philosophy
- Architecture
- Interfaces
  Interfaces
  - Bifrost
    
    Bifrost
    
    ABAC Implementation
    
    Users
    
    Use-cases
    
    Grant
    
    Role
    
    Grant Requests
    
    Heimdall Primitives
    
    Roles and Use cases
  - CLI
    
    CLI
    
    Installation
    
    Initialization
    
    Command Reference
  - Data Product Hub
    
    Data Product Hub
    
    Core Concepts
    
    Data Product Discovery
    
    Data Product Exploration
    
    Data Product Activation
    Data Product Activation
    
    BI Sync
    BI Sync
    
    Tableau Cloud
    
    Power BI
    Power BI
    
    Power BI Desktop
    
    Power BI Service
    
    Tableau Desktop
    
    Microsoft Excel
    
    AI/ML
    
    Postgres
    
    Graphql
    
    Data APIs
    
    Best Practices
    
    Examples
  - Metis
    
    Metis
    
    Metis UI
    
    Metis UI
    
    Pre-defined Roles and Policies
    
    Key Features
    
    Metadata of Products
    
    Metadata of Products
    
    Metadata of Data Products
    
    Metadata of Data Assets
    
    Metadata of Data Assets
    
    Metadata of Tables
    
    Metadata of Topics
    
    Metadata of Dashboards
    
    Metadata of Resources
    
    Metadata of Resources
    
    Bundles
    
    Clusters
    
    Computes
    
    Databases
    
    Depots
    
    Instance Secrets
    
    Lakehouses
    
    Lenses
    
    Monitors
    
    Operators
    
    Pagers
    
    Policies
    
    Secrets
    
    Services
    
    Stacks
    
    Workers
    
    Workflows
  - Operations
    
    Operations
  - Workbench
    
    Workbench
- Resources
  Resources
  - Types
    
    Types
    
    Bundle
    
    Bundle
    
    Configurations
    
    Recipes
    Recipes
    
    Optimize bundling Resources in Data Product
    
    Templates
    Templates
    
    Inline Bundle Template
    
    File Path Bundle Template
    
    Dependencies in Bundle
    
    Cluster
    
    Cluster
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Connectors
    
    Integration
    
    Integration
    
    Power BI
    
    Power BI
    
    Tableau
    
    IBM SPSS
    
    CLI Client
    
    Tuning
    Tuning
    
    Query Execution Optimization
    
    Performance Tuning
    
    Recommended Configuration
    
    Examples
    Examples
    
    Restart and scale Cluster on a pre-defined schedule
    
    Multiple Cluster Setup using single manifest
    
    Core Concepts
    Core Concepts
    
    On-demand Computing
    
    Compute
    
    Compute
    
    Configurations
    
    Templates
    
    Database
    
    Database
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Query Database using Workbench
    
    Building Streamlit applications backed by Database
    
    Depot
    
    Depot
    
    Quickstart
    
    Supported Sources
    Supported Sources
    
    ABFSS
    
    Amazon Redshift
    
    Amazon S3
    
    Apache Pulsar
    
    Azure WASBS
    
    Elasticsearch
    
    Eventhub
    
    Google BigQuery
    
    GCS
    
    JDBC
    
    Kafka
    
    MongoDB
    
    Microsoft SQL Server
    
    MySQL
    
    Opensearch
    
    Oracle
    
    PostgreSQL
    
    Snowflake
    
    Supported Table Format
    Supported Table Format
    
    Iceberg
    
    Delta
    
    Configurations
    
    Governance
    
    Observability
    
    Cataloging
    
    Supported Connectors
    
    Usecase
    
    Usage
    Usage
    
    Data ingestion and transformation
    
    Building semantic models
    
    Query the source
    
    Accelerating queries
    
    Building Data APIs
    
    Depot as Lakehouse
    
    Scanning metadata
    
    Grant
    
    Grant
    
    Configuration
    
    Instance Secret
    
    Instance Secret
    
    Quickstart
    
    Data Sources
    Data Sources
    
    ABFSS
    
    Amazon Redshift
    
    Amazon S3
    
    Azure WASBS
    
    Elasticsearch
    
    Eventhub
    
    Google BigQuery
    
    GCS
    
    JDBC
    
    Kafka
    
    MongoDB
    
    Microsoft SQL Server
    
    MySQL
    
    Opensearch
    
    Oracle
    
    PostgreSQL
    
    Snowflake
    
    Repositories
    Repositories
    
    AWS Codecommit
    
    Bitbucket
    
    Git
    
    Configurations
    
    Governance
    
    Observability
    
    Cataloging
    
    Usage
    
    Lakehouse
    
    Lakehouse
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Command Reference
    
    Iceberg Metadata Tables
    
    Recipes
    Recipes
    
    Ensure high data quality using Write-Audit-Publish (WAP) Pattern
    
    Use Iceberg metadata tables to extract insights
    
    Create, Get, and Drop Dataset
    
    Table Properties
    
    Schema Evolution
    
    Partitioning
    
    Maintenance (Snapshots and Metadata listing)
    
    Lens
    
    Lens
    
    Semantic Modeling
    Semantic Modeling
    
    Key concepts
    
    Segments
    
    Views
    
    User Groups
    
    Supported Data Sources
    Supported Data Sources
    
    Data Sources
    
    Bigquery
    
    Databricks
    
    Postgres
    
    Redshift
    
    Snowflake
    
    Flash
    
    Minerva
    
    Themis
    
    Exploration
    Exploration
    
    Using GraphQL
    Using GraphQL
    
    Overview
    
    Examples
    
    Using Python
    
    Using SQL APIs
    Using SQL APIs
    
    Overview
    
    Supported functions and operators
    
    Using REST APIs
    Using REST APIs
    
    API Endpoints and Scopes
    
    Exploration
    
    BI Integrations
    BI Integrations
    
    Power BI
    Power BI
    
    Power BI Desktop
    
    Power BI Service
    
    Superset
    
    Tableau
    
    References
    References
    
    Best Practices
    
    Errors
    
    Optimizing Semantic Model
    
    Aggregation Strategies
    
    Complex Analysis
    
    Row Multiplication Handling
    
    Configurations
    
    Cataloging
    
    Governance
    
    Observability
    
    Monitor
    
    Monitor
    
    Types
    Types
    
    Equation Monitor
    
    Report Monitor
    
    Stream Monitor
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Add context columns to Monitors
    
    Create a Report Monitor
    
    Create a Stream Monitor
    
    Create an Equation Monitor on Lens
    
    Create an Equation Monitor on Postgres
    
    Generate incidents for certificate expiration
    
    Operator
    
    Operator
    
    Configurations
    
    Recipes
    Recipes
    
    Orchestrate Hightouch pipeline using Hightouch Factory Operator
    
    Pager
    
    Pager
    
    Configurations
    
    Recipes
    Recipes
    
    Create a Custom Body Template for Pager
    
    Policy
    
    Policy
    
    Core Concepts
    
    First Steps
    
    Configurations
    
    Templates
    
    References
    References
    
    Rule for and or relationship
    
    Recipes
    Recipes
    
    End-to-end use-case on how to implement access policy
    
    Secret
    
    Secret
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Refer secrets in other DataOS Resources
    
    Setup secrets to pull images from a private container registry
    
    Service
    
    Service
    
    Configurations
    
    Stacks
    
    Stacks
    
    Beacon
    
    Beacon
    
    Bento
    
    Bento
    
    Setting Up Bento
    Setting Up Bento
    
    Bento on Local Host
    
    Bento on DataOS
    
    Bloblang
    Bloblang
    
    Overview
    
    Core Features
    
    Functions
    
    Methods
    
    Arithmetic
    
    Advanced Bloblang
    
    Configurations
    
    Configurations
    
    Cloud Credentials
    
    Dynamic Inputs and Outputs
    
    Error Handling
    
    Fields Paths
    
    Interpolation
    
    Message Batching
    
    Metadata
    
    Monitoring
    
    Performance Tuning
    
    Processing Pipelines
    
    Resources
    
    Secrets
    
    Streams Mode
    
    Streams Mode
    
    Streams API
    
    Streams Via Config Files
    
    Streams Via REST API
    
    Synchronous Responses
    
    Templating
    
    Unit Testing
    
    Using CUE
    
    Window Processing
    
    Components
    
    Components
    
    HTTP
    
    Inputs
    
    Logger
    
    Metrics
    
    Output
    
    Output
    
    AWS S3
    
    Azure Blob Storage
    
    Fastbase Depot
    
    Kafka
    
    kafka Depot
    
    Processors
    
    Processors
    
    log
    
    mapping
    
    sql_raw
    
    Rate Limit
    
    Recipes
    Recipes
    
    Fetching Data from Instagram API
    
    Twitter API Data Processing
    
    Discord Bot
    
    Stock Data API to Lakehouse
    
    Performing Rate Limit
    
    Performing Pagination
    
    CLI Stack
    
    CLI Stack
    
    Container
    
    Container
    
    DBT
    
    DBT
    
    Flare
    
    Flare
    
    Creating Flare Jobs
    
    Configurations
    
    Data Sources
    
    Data Sources
    
    Databases
    Databases
    
    PostgreSQL
    
    MySQL
    
    Oracle
    
    MS-SQL
    
    MongoDB
    
    Data Warehouses
    Data Warehouses
    
    Snowflake
    
    BigQuery
    
    Amazon Redshift
    
    Object Storage
    Object Storage
    
    Object Storage Configuration
    
    Object Storage Configuration
    
    Schema Configuration
    
    Data Format Configuration
    
    Delta Tables
    
    Streaming Sources
    Streaming Sources
    
    Kafka
    
    EventHub
    
    Pulsar
    
    Search Engines
    Search Engines
    
    Elasticsearch
    
    OpenSearch
    
    Flare Functions
    
    Flare Functions
    
    List of Flare Functions
    
    Use Cases & Examples
    
    Use Cases & Examples
    
    Data Ingestion
    Data Ingestion
    
    Batch Jobs
    
    Incremental Jobs
    
    Streaming Jobs
    
    Data Transformation
    Data Transformation
    
    Iceberg Operations
    
    Merge Operations
    
    Data Replay
    
    Concurrent Writes
    
    Column Tagging
    
    Data Syndication
    
    Enrichment
    
    Fingerprinting
    
    Performance Optimization
    Performance Optimization
    
    Partitioning
    
    Partition Evolution
    
    Bucketing
    
    Caching
    
    Compression
    
    Job Tuning
    
    Data Maintenance
    Data Maintenance
    
    Data Compaction
    
    Manifest Rewrite
    
    Delete from Dataset
    
    Expire Snapshots
    
    Remove Orphans
    
    Optimizing Flare
    
    Optimizing Flare
    
    Performance Tuning
    
    Performance Tuning
    
    Configuration Tuning
    
    SQL Optimization
    
    Flash
    
    Flash
    
    Create a Flash Service
    
    Configurations
    
    Monitoring Flash
    
    Best Practices
    
    Do's and Dont's
    
    Errors and Issues
    Errors and Issues
    
    Handling Empty Tables
    
    Recipes
    Recipes
    
    Use cached datasets in Lens models
    
    Use cached datasets in Talos APIs
    
    Lakesearch
    
    Lakesearch
    
    Key concepts
    
    Set up
    Set up
    
    Lakesearch Service
    
    Query Rewriter
    
    Configurations
    
    Governance
    
    Index Searching
    
    Observability
    
    Troubleshooting
    
    Nilus
    
    Nilus
    
    Data Movement Architecture
    
    Quick Start
    
    Data Masking
    
    CDC Sources
    
    CDC Sources
    
    IBM Db2
    
    MongoDB
    
    MS SQL Server
    
    MySQL
    
    PostgreSQL
    
    Batch sources
    
    Batch sources
    
    Apache Kafka
    
    AWS Athena
    
    AWS DataOS Lakehouse
    
    AWS Redshift
    
    Azure DataOS Lakehouse
    
    Azure Synapse
    
    Databricks
    
    Delta Lake
    
    GCS DataOS Lakehouse
    
    Google Analytics
    
    Google BigQuery
    
    MongoDB
    
    MS SQL Server
    
    MySQL
    
    PostgreSQL
    
    Salesforce
    
    Snowflake
    
    Stripe
    
    Custom Source
    
    Supported Destinations
    
    Supported Destinations
    
    AWS DataOS Lakehouse
    
    Azure DataOS Lakehouse
    
    BigQuery
    
    Databricks
    
    GCS DataOS Lakehouse
    
    MS SQL Server
    
    Redshift
    
    Snowflake
    
    PostgreSQL
    
    Custom Destination
    
    Best Practices
    
    Python
    
    Python
    
    Stack Deployment
    
    Getting Started
    Getting Started
    
    Build Application
    
    Deploy Application
    
    Configurations
    
    Governance
    Governance
    
    Authentication
    
    Access Control
    
    Recipes
    Recipes
    
    Frameworks
    Frameworks
    
    Streamlit
    
    Vizro
    
    Utilities
    Utilities
    
    Retrieve Insights
    
    Best Practices
    
    Troubleshooting
    
    Scanner
    
    Scanner
    
    Quickstart
    
    Core Concepts
    
    Supported Sources
    Supported Sources
    
    Data Sources
    Data Sources
    
    AzureSQL
    
    BigQuery
    
    Kafka
    
    Lakehouse
    
    MariaDB
    
    MSSQL
    
    MongoDB
    
    MySQL
    
    Oracle
    
    PostgreSQL
    
    Redshift
    
    Snowflake
    
    System Sources
    System Sources
    
    Data Products
    
    User Information
    
    Indexer Services
    Indexer Services
    
    Data Profiling
    
    Quality Checks
    
    Resources Scanner
    
    Query Usage
    
    Configurations
    
    Governance
    
    Cataloging
    
    Best Practices
    
    Observability
    
    Recipes
    
    Soda
    
    Soda
    
    Quality Checks
    Quality Checks
    
    Accuracy Checks
    
    Completeness
    
    Freshness
    
    Schema validation
    
    Uniqueness
    
    Validity
    
    User-defined
    
    Configurations
    
    Recipes
    Recipes
    
    Running Data Profiling with Soda
    
    Running Checks on a Specific Branch
    
    Steampipe
    
    Steampipe
    
    Talos
    
    Talos
    
    Setting Up Talos
    
    Supported Sources
    Supported Sources
    
    Bigquery
    
    Flash
    
    Lens
    
    MySQL
    
    Postgres
    
    Redshift
    
    Snowflake
    
    Configurations
    
    Configurations
    
    config.yaml
    
    apis
    
    service.yaml
    
    Governance
    
    Recipes
    Recipes
    
    Apply data masking
    
    Fetch data from third party tools
    
    Caching datasets
    
    Generate API documentation
    
    External API as data source
    
    Validating Parameters
    
    Error Handling
    
    Example
    
    Best Practices
    Best Practices
    
    API
    
    SQL
    
    Custom Stack
    
    Custom Stack
    
    Configurations
    
    Volume
    
    Volume
    
    Getting Started
    
    Configurations
    
    Manage the Volume
    Manage the Volume
    
    Delete Azure PVC
    
    Delete AWS PVC
    
    Worker
    
    Worker
    
    First Steps
    
    Configuration
    
    Core Concepts
    
    Workflow
    
    Workflow
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Implement Single-run Workflow
    
    Run Cron/Scheduled Workflow
    
    Orchestrate multiple Workflows from a single Workflow
    
    Retry a Job in Workflow
  - Characteristics
  - Configuration
  - Best practices for YAML
- SDK
  SDK
  - DataOS Python SDK
  - DataOS PyFlare SDK
    
    DataOS PyFlare SDK
    
    Supported Sources
    Supported Sources
    
    Bigquery
    
    Lakehouse
    
    Postgres
    
    Snowflake
    
    Troubleshooting
Getting Started
Getting Started
Data Product
Data Product
- Core Concepts
- Data Product Lifecycle
  Data Product Lifecycle
- Data Product Examples
  Data Product Examples
  - Building from Scratch with Snowflake
  - Using DataOS Lakehouse as a Source
- Configurations
- Observability
  Observability
  - Key Concepts
  - Monitoring
    Monitoring
    
    Resource Health
    Resource Health
    
    Instance Secret
    
    Secrets
    
    Volume
    
    Lakehouse
    
    Depot
    
    Workflow
    
    Cluster
    
    Service
    
    Worker
    
    Lens
    
    Compute
    
    Bundle
    
    CPU and Memory Usage
    CPU and Memory Usage
    
    Workflow
    
    Cluster
    
    Service
    
    Worker
    
    Lens
    
    Depot
    
    System Metrics
    System Metrics
    
    K8 Cluster Infrastructure Dashboard
    
    K8 Cluster Entities Dashboard
  - logging
    logging
    
    Workflow
    
    Cluster
    
    Service
    
    Worker
    
    Lens
    
    Depot
  - Alerting
    Alerting
    
    Alerts for High CPU Usage
    
    Alerts for High Memory Usage
    
    Alerts for Runtime Failure
    
    Alerts for Resource Status
  - Use Cases
    Use Cases
    
    Resolve CPU Limit Breach
    
    Monitoring Lens Metrics
- Recipes
  Recipes
  - Deploy Data Product using CI/CD pipeline
Glossary
Learn
Learn
- DataOS Primer
  DataOS Primer
  - DataOS essentials
- Learning tracks
  Learning tracks
  - Data Product Consumer Track
    
    Data Product Consumer Track
    
    Learning path overview
    
    Learning path overview
    
    Discovering Data Products
    
    Discovering Data Products
    
    Finding the right Data Product
    
    View Data Product information
    
    Know about I/O datasets
    
    Explore semantic model
    
    Explore semantic model
    
    API integration
    
    Understand quality
    
    Connect Data Products with BI tools and applications
    Connect Data Products with BI tools and applications
    
    Integration with BI tools
    
    Integration with BI tools
    
    Tableau Cloud
    
    Tableau Desktop
    
    Superset
    
    Power BI
    
    Power BI Service
    
    Integration with AI/ML
    
    Integration with AI/ML
    
    Integration with Postgres
    
    Integration with Postgres
    
    Integration with GraphQL
    
    Integration with GraphQL
    
    Integration with LakeSearch
    
    Integration with LakeSearch
    
    Integration with Data APIs
    
    Integration with Data APIs
    
    Custom apps to consume Data Products
    
    Custom apps to consume Data Products
    
    Building an App with Appsmith
    
    Building an App with Appsmith
  - Data Product Foundations Track
    
    Data Product Foundations Track
    
    Foundations- Course I
    
    Foundations- Course I
    
    Source-aligned Data Product
    
    Data source connectivity
    
    Data source connectivity
    
    Set up Depots
    
    Create Scanner
    
    Build Pipeline
    
    Build Pipeline
    
    Scheduling pipelines
    
    Quality check
    
    Set up monitoring
    
    Set up monitoring
    
    Quality check observability
    
    Deploy Your first Source-aligned Data Products
    
    Deploy Your first Source-aligned Data Products
    
    Create Data Product Spec
    
    Deploy Data Product
    
    Foundations- Course II
    
    Foundations- Course II
    
    Capture objectives, KPIs & metrics
    
    Design Data Product
    
    Repository and folder structure
    
    Source data connectivity
    
    Build pipeline
    
    Create semantic model
    
    Deploy and register
  - Data Product Developer Track
    
    Data Product Developer Track
  - Data Product Owner Track
    
    Data Product Owner Track
  - Operator Learn Track
    
    Operator Learn Track
    
    Learning path overview
    
    Learning path overview
    
    Access control
    
    Alerts Grafana
    
    Cluster management
    
    Cluster tuning
    
    Cred security
    
    Data source connectivity
    
    Keycloak
    
    Lakehouse insights
    
    Routine check
    
    Upgrade rollback
- Quick Start Guides
  Quick Start Guides
  - Connecting with data sources
    
    Connecting with data sources
  - Performing exploratory data analysis
    Performing exploratory data analysis
    
    Running your first data query
    
    Running your first data query
    
    Arranging and transforming data with Pivot
    
    Arranging and transforming data with Pivot
    
    Writing complex queries
    
    Writing complex queries
    
    Querying diverse data sources
    
    Querying diverse data sources
  - Scanning metadata
  - Creating ETL pipelines
    Creating ETL pipelines
    
    Ingest external files into DataOS
    
    Ingest external files into DataOS
    
    Batch data processing using Flare
    
    Batch data processing using Flare
    
    Storing query results as dataset with Flare
    
    Storing query results as dataset with Flare
  - Deploying and securing data applications
    Deploying and securing data applications
    
    Deploying data application using Container Stack
    
    Deploying data application using Container Stack
    
    Listing Your App on DataOS Home
    
    Listing Your App on DataOS Home
    
    Securing deployed data applications
    
    Securing deployed data applications
- Videos

Data Sources¶

Was this page helpful?

Thanks for your feedback!

Thanks for your feedback!

Previous User Groups

Contact Us | Privacy Policy | Terms of Use

Copyright ⓒ 2025 The Modern Data Company