Skip to content

All things DataOS

Data Sources

All things DataOS

Home
Home
- Philosophy
- Architecture
- Interfaces
  Interfaces
  - Bifrost
    
    Bifrost
    
    ABAC Implementation
    
    Users
    
    Use-cases
    
    Grant
    
    Role
    
    Grant Requests
    
    Heimdall Primitives
  - CLI
    
    CLI
    
    Installation
    
    Initialization
    
    Command Reference
  - Data Product Hub
    
    Data Product Hub
    
    Core Concepts
    
    Data Product Discovery
    
    Data Product Exploration
    
    Data Product Activation
    Data Product Activation
    
    BI Sync
    BI Sync
    
    Tableau Cloud
    
    Power BI
    Power BI
    
    Power BI Desktop
    
    Power BI Service
    
    Tableau Desktop
    
    Microsoft Excel
    
    AI/ML
    
    Postgres
    
    Graphql
    
    Data APIs
    
    Best Practices
    
    Examples
  - Metis
    
    Metis
    
    Metis UI
    
    Metis UI
    
    Pre-defined Roles and Policies
    
    Key Features
    
    Metadata of Products
    
    Metadata of Products
    
    Metadata of Data Products
    
    Metadata of Data Assets
    
    Metadata of Data Assets
    
    Metadata of Tables
    
    Metadata of Topics
    
    Metadata of Dashboards
    
    Metadata of Resources
    
    Metadata of Resources
    
    Bundles
    
    Clusters
    
    Computes
    
    Databases
    
    Depots
    
    Instance Secrets
    
    Lakehouses
    
    Lenses
    
    Monitors
    
    Operators
    
    Pagers
    
    Policies
    
    Secrets
    
    Services
    
    Stacks
    
    Workers
    
    Workflows
  - Operations
    
    Operations
  - Workbench
    
    Workbench
- Resources
  Resources
  - Types
    
    Types
    
    Bundle
    
    Bundle
    
    Configurations
    
    Templates
    
    Cluster
    
    Cluster
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Connectors
    
    Integration
    
    Integration
    
    Power BI
    
    Power BI
    
    Using Connector
    
    Using ODBC Driver
    
    Tableau
    
    IBM SPSS
    
    CLI Client
    
    Tuning
    Tuning
    
    Query Execution Optimization
    
    Performance Tuning
    
    Recommended Configuration
    
    Examples
    Examples
    
    Restart and scale Cluster on a pre-defined schedule
    
    Multiple Cluster Setup using single manifest
    
    Core Concepts
    Core Concepts
    
    On-demand Computing
    
    Compute
    
    Compute
    
    Configurations
    
    Templates
    
    Database
    
    Database
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Query Database using Workbench
    
    Building Streamlit applications backed by Database
    
    Depot
    
    Depot
    
    Quickstart
    
    Supported Sources
    Supported Sources
    
    ABFSS
    
    Amazon Redshift
    
    Amazon S3
    
    Apache Pulsar
    
    Azure WASBS
    
    Elasticsearch
    
    Eventhub
    
    Google BigQuery
    
    GCS
    
    JDBC
    
    Kafka
    
    MongoDB
    
    Microsoft SQL Server
    
    MySQL
    
    Opensearch
    
    Oracle
    
    PostgreSQL
    
    Snowflake
    
    Configurations
    
    Governance
    
    Observability
    
    Cataloging
    
    Supported Connectors
    
    Usecase
    
    Usage
    Usage
    
    Data ingestion and transformation
    
    Building semantic models
    
    Query the source
    
    Accelerating queries
    
    Building Data APIs
    
    Building Data Products
    
    Depot as Lakehouse
    
    Scanning metadata
    
    Grant
    
    Grant
    
    Configuration
    
    Instance Secret
    
    Instance Secret
    
    Quickstart
    
    Data Sources
    Data Sources
    
    ABFSS
    
    Amazon Redshift
    
    Amazon S3
    
    Azure WASBS
    
    Elasticsearch
    
    Eventhub
    
    Google BigQuery
    
    GCS
    
    JDBC
    
    MongoDB
    
    Microsoft SQL Server
    
    MySQL
    
    Opensearch
    
    Oracle
    
    PostgreSQL
    
    Snowflake
    
    Repositories
    Repositories
    
    AWS Codecommit
    
    Bitbucket
    
    Git
    
    Configurations
    
    Governance
    
    Observability
    
    Cataloging
    
    Usage
    
    Lakehouse
    
    Lakehouse
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Command Reference
    
    Iceberg Metadata Tables
    
    Recipes
    Recipes
    
    Ensure high data quality using Write-Audit-Publish (WAP) Pattern
    
    Use Iceberg metadata tables to extract insights
    
    Create, Get, and Drop Dataset
    
    Table Properties
    
    Schema Evolution
    
    Partitioning
    
    Maintenance (Snapshots and Metadata listing)
    
    Lens
    
    Lens
    
    Semantic Modeling
    Semantic Modeling
    
    Key concepts
    
    Segments
    
    Views
    
    User Groups
    
    Supported Data Sources
    Supported Data Sources
    
    Bigquery
    
    Flash
    
    Minerva
    
    Postgres
    
    Redshift
    
    Snowflake
    
    Themis
    
    Exploration
    Exploration
    
    Using GraphQL
    Using GraphQL
    
    Overview
    
    Examples
    
    Using Python
    
    Using SQL APIs
    Using SQL APIs
    
    Overview
    
    Supported functions and operators
    
    Using REST APIs
    Using REST APIs
    
    API Endpoints and Scopes
    
    Exploration
    
    BI Integrations
    BI Integrations
    
    Power BI
    Power BI
    
    Power BI Desktop
    
    Power BI Service
    
    Superset
    
    Tableau
    
    References
    References
    
    Best Practices
    
    Errors
    
    Optimizing Semantic Model
    
    Aggregation Strategies
    
    Configurations
    
    Cataloging
    
    Governance
    
    Observability
    
    Monitor
    
    Monitor
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Add context columns to Monitors
    
    Create a Report Monitor
    
    Create a Stream Monitor
    
    Create an Equation Monitor on Lens
    
    Create an Equation Monitor on Postgres
    
    Generate incidents for certificate expiration
    
    Operator
    
    Operator
    
    Configurations
    
    Recipes
    Recipes
    
    Orchestrate Hightouch pipeline using Hightouch Factory Operator
    
    Pager
    
    Pager
    
    Configurations
    
    Recipes
    Recipes
    
    Create a Custom Body Template for Pager
    
    Policy
    
    Policy
    
    Core Concepts
    
    First Steps
    
    Configurations
    
    Templates
    
    References
    References
    
    Rule for and or relationship
    
    Recipes
    Recipes
    
    End-to-end use-case on how to implement access policy
    
    Secret
    
    Secret
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Refer secrets in other DataOS Resources
    
    Setup secrets to pull images from a private container registry
    
    Service
    
    Service
    
    Configurations
    
    Stacks
    
    Stacks
    
    Beacon
    
    Beacon
    
    Bento
    
    Bento
    
    Setting Up Bento
    Setting Up Bento
    
    Bento on Local Host
    
    Bento on DataOS
    
    Bloblang
    Bloblang
    
    Overview
    
    Core Features
    
    Functions
    
    Methods
    
    Arithmetic
    
    Advanced Bloblang
    
    Configurations
    
    Configurations
    
    Cloud Credentials
    
    Dynamic Inputs and Outputs
    
    Error Handling
    
    Fields Paths
    
    Interpolation
    
    Message Batching
    
    Metadata
    
    Monitoring
    
    Performance Tuning
    
    Processing Pipelines
    
    Resources
    
    Secrets
    
    Streams Mode
    
    Streams Mode
    
    Streams API
    
    Streams Via Config Files
    
    Streams Via REST API
    
    Synchronous Responses
    
    Templating
    
    Unit Testing
    
    Using CUE
    
    Window Processing
    
    Components
    
    Components
    
    HTTP
    
    Inputs
    
    Logger
    
    Metrics
    
    Output
    
    Processors
    
    Rate Limit
    
    Recipes
    Recipes
    
    Fetching Data from Instagram API
    
    Twitter API Data Processing
    
    Discord Bot
    
    Stock Data API to Icebase
    
    Performing Rate Limit
    
    Performing Pagination
    
    CLI Stack
    
    CLI Stack
    
    Container
    
    Container
    
    DBT
    
    DBT
    
    Flare
    
    Flare
    
    Quick Guide
    
    Flash
    
    Flash
    
    Create a Flash Service
    
    Configurations
    
    Supported Data Sources
    
    Monitoring Flash
    
    Best Practices
    
    Do's and Dont's
    
    Errors and Issues
    Errors and Issues
    
    Handling Empty Tables
    
    Recipes
    Recipes
    
    Use cached datasets in Lens models
    
    Use cached datasets in Talos APIs
    
    Lakesearch
    
    Lakesearch
    
    Key concepts
    
    Set up
    Set up
    
    Lakesearch Service
    
    Query Rewriter
    
    Configurations
    
    Governance
    
    Index Searching
    
    Observability
    
    Troubleshooting
    
    Scanner
    
    Scanner
    
    Quickstart
    
    Core Concepts
    
    Supported Sources
    Supported Sources
    
    Data Sources
    Data Sources
    
    AzureSQL
    
    BigQuery
    
    Kafka
    
    Lakehouse
    
    MariaDB
    
    MSSQL
    
    MySQL
    
    Oracle
    
    PostgreSQL
    
    Pulsar
    
    Redshift
    
    Snowflake
    
    System Sources
    System Sources
    
    Data Products
    
    User Information
    
    Indexer Services
    Indexer Services
    
    Data Profiling
    
    Quality Checks
    
    Resources Scanner
    
    Query Usage
    
    Configurations
    
    Governance
    
    Cataloging
    
    Best Practices
    
    Observability
    
    Recipes
    
    Soda
    
    Soda
    
    Steampipe
    
    Steampipe
    
    Talos
    
    Talos
    
    Setting Up Talos
    
    Supported Sources
    Supported Sources
    
    Bigquery
    
    Flash
    
    Lens
    
    MySQL
    
    Postgres
    
    Redshift
    
    Snowflake
    
    Configurations
    
    Configurations
    
    config.yaml
    
    apis
    
    service.yaml
    
    Governance
    
    Recipes
    Recipes
    
    Apply data masking
    
    Fetch data from third party tools
    
    Caching datasets
    
    Generate API documentation
    
    External API as data source
    
    Validating Parameters
    
    Error Handling
    
    Example
    
    Best Practices
    Best Practices
    
    API
    
    SQL
    
    Custom Stack
    Custom Stack
    
    How to create your own Stack?
    
    Configurations
    
    Volume
    
    Volume
    
    Getting Started
    
    Configurations
    
    Worker
    
    Worker
    
    First Steps
    
    Configuration
    
    Core Concepts
    
    Workflow
    
    Workflow
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Implement Single-run Workflow
    
    Run Cron/Scheduled Workflow
    
    Orchestrate multiple Workflows from a single Workflow
    
    Retry a Job in Workflow
  - Characteristics
  - Configuration
  - Best practices for YAML
- SDK
  SDK
  - DataOS Python SDK
  - DataOS PyFlare SDK
    
    DataOS PyFlare SDK
    
    Supported Sources
    Supported Sources
    
    Bigquery
    
    Lakehouse
    
    Postgres
    
    Snowflake
    
    Troubleshooting
Getting Started
Getting Started
Data Product
Data Product
- Core Concepts
- Data Product Lifecycle
  Data Product Lifecycle
- Data Product Examples
  Data Product Examples
  - Building from Scratch with Snowflake
  - Using DataOS Lakehouse as a Source
- Configurations
- Recipes
  Recipes
  - Deploy Data Product using CI/CD pipeline
Glossary
Learn
Learn
- Learning tracks
  Learning tracks
  - Data Product Consumer
    
    Data Product Consumer
    
    Introduction to Data Products
    Introduction to Data Products
    
    Features and importance of Data Products
    
    Features and importance of Data Products
    
    Discovering Data Products
    Discovering Data Products
    
    Introduction to Data Product Hub
    
    Introduction to Data Product Hub
    
    Discovering Data Products
    
    Discovering Data Products
    
    Viewing Data Product info
    
    Viewing Data Product info
    
    Exploring input & output datasets
    
    Exploring input & output datasets
    
    Navigating semantic models
    
    Navigating semantic models
    
    Checking data quality
    
    Checking data quality
    
    Integrating Data Products with BI tools and Applications
    Integrating Data Products with BI tools and Applications
    
    Integration with BI tools
    
    Integration with BI tools
    
    Integration with AI/ML
    
    Integration with AI/ML
    
    Integration with Postgres
    
    Integration with Postgres
    
    Integration with GraphQL
    
    Integration with GraphQL
    
    Integration with data APIs
    
    Integration with data APIs
    
    Custom apps to consume Data Products
    
    Custom apps to consume Data Products
    
    Building an App with Appsmith
    
    Building an App with Appsmith
  - Data Product Developer
    
    Data Product Developer
    
    Understanding business goals
    
    Understanding business goals
    
    Designing Data Products
    
    Designing Data Products
    
    Building Data Products
    Building Data Products
    
    Source data connectivity
    
    Source data connectivity
    
    Connectivity with Postgres
    
    Connectivity with Postgres
    
    Building and maintaining data pipelines
    
    Building and maintaining data pipelines
    
    Creating your first data pipeline
    
    Creating your first data pipeline
    
    Scheduling Workflows
    
    Scheduling Workflows
    
    Data quality checks
    
    Data quality checks
    
    Pipeline observability
    
    Pipeline observability
    
    Implementing quality checks
    
    Implementing quality checks
    
    Build a semantic model
    
    Build a semantic model
    
    Defining business objectives and KPIs
    
    Defining business objectives and KPIs
    
    Designing the conceptual Model
    
    Designing the conceptual Model
    
    Key concepts of Lens
    
    Key concepts of Lens
    
    Creating semantic model
    
    Creating semantic model
    
    Deploy Lens in DataOS
    
    Deploy Lens in DataOS
    
    Defining data policies for semantic model
    
    Defining data policies for semantic model
    
    Creating data APIs
    
    Creating data APIs
    
    Deploying Data Products
    Deploying Data Products
    
    Creating a deployable Bundle
    
    Creating a deployable Bundle
    
    Creating Data Product Spec Fields
    
    Creating Data Product Spec Fields
    
    Deploying Data Products
    
    Deploying Data Products
    
    Data Product CI/CD pipeline
    
    Data Product CI/CD pipeline
  - DataOS Operator
    
    DataOS Operator
    
    Credential security
    
    Credential security
    
    Data source connectivity
    
    Data source connectivity
    
    Connectivity with Postgres
    
    Connectivity with Postgres
    
    Routine checks
    
    Routine checks
    
    DataOS upgrade and rollback strategies
    
    DataOS upgrade and rollback strategies
    
    System monitoring
    System monitoring
    
    Configuring alerts and dashboards using Grafana
    
    Configuring alerts and dashboards using Grafana
    
    Optimizing system level events
    
    Optimizing system level events
    
    Query Cluster management
    Query Cluster management
    
    Cluster management
    
    Cluster management
    
    Cluster tuning
    
    Cluster tuning
- Quick Start Guides
  Quick Start Guides
  - Connecting with data sources
    
    Connecting with data sources
  - Performing exploratory data analysis
    Performing exploratory data analysis
    
    Running your first data query
    
    Running your first data query
    
    Arranging and transforming data with Pivot
    
    Arranging and transforming data with Pivot
    
    Writing complex queries
    
    Writing complex queries
    
    Querying diverse data sources
    
    Querying diverse data sources
  - Using data modeling layer
    Using data modeling layer
    
    Defining a Data Model
    
    Defining a Data Model
    
    Creating a data model
    
    Creating a data model
    
    Deploying a data model
    
    Deploying a data model
    
    Exploring Lens on Catalog
    
    Exploring Lens on Catalog
    
    Working on Lens Studio
    
    Working on Lens Studio
    
    Working with Payloads
    
    Working with Payloads
    
    Example scenarios
    
    Pre-defined filtering using Segments
    
    Pre-defined filtering using Segments
    
    Defining Data Policies
    
    Defining Data Policies
  - Building Data Products
  - Scanning metadata
  - Creating ETL pipelines
    Creating ETL pipelines
    
    Ingest external files into DataOS
    
    Ingest external files into DataOS
    
    Batch data processing using Flare
    
    Batch data processing using Flare
    
    Storing query results as dataset with Flare
    
    Storing query results as dataset with Flare
  - Deploying and securing data applications
    Deploying and securing data applications
    
    Deploying data application using Container Stack
    
    Deploying data application using Container Stack
    
    Listing Your App on DataOS Home
    
    Listing Your App on DataOS Home
    
    Securing deployed data applications
    
    Securing deployed data applications
- Videos

Data Sources¶