Skip to content

All things DataOS

Data Sources

Initializing search

Home
Getting Started
Data Product
Glossary
Learn

All things DataOS

Home
Home
- Philosophy
- Architecture
- Interfaces
  Interfaces
  - Atlas
    
    Atlas
    
    Dashboards
    
    Dashboards
    
    Adding Visualizations to Dashboard
    
    Queries
    
    Snippet
    
    Alerts
    
    Destinations
    
    Integration
    
    Integration
    
    Power BI
    
    Power BI
    
    Using Connector
    
    Using ODBC Driver
    
    Tableau
    
    IBM SPSS
  - Audiences
    
    Audiences
    
    Audiences UI
    
    Getting Started
  - Bifrost
    
    Bifrost
    
    ABAC Implementation
    
    User
    
    Use-cases
    
    Grant
    
    Role
    
    Grant Requests
    
    Heimdall Primitives
  - CLI
    
    CLI
    
    Installation
    
    Initialization
    
    Command Reference
  - Data Product Hub
    
    Data Product Hub
    
    Core Concepts
    
    Data Product Discovery
    
    Data Product Exploration
    
    Data Product Activation
    Data Product Activation
    
    BI Sync
    BI Sync
    
    Tableau Cloud
    
    Power BI
    
    Tableau Desktop
    
    Microsoft Excel
    
    Apache Superset
    
    AI/ML
    
    App Development
    
    Data APIs
    
    Best Practices
    
    Examples
  - Lens
    
    Lens
    
    Building Lens
    
    Building Lens
    
    Tutorial
    
    Data Modelling Concepts
    
    Lens Attributes
    
    Dynamic Parameters
    
    Lens Explorer
    
    Lens Ecosystem
    
    Lens Ecosystem
    
    Lens Query
    
    Elements of Lens
    
    Elements of Lens
    
    Entities
    
    Fields
    
    Dimensions
    
    Measures
    
    Relationships
  - Metis
    
    Metis
    
    Metis UI
    
    Metis UI
    
    Pre-defined Roles and Policies
    
    Key Features
    
    Metadata of Products
    
    Metadata of Products
    
    Metadata of Data Products
    
    Metadata of Data Assets
    
    Metadata of Data Assets
    
    Metadata of Tables
    
    Metadata of Topics
    
    Metadata of Dashboards
    
    Metadata of Resources
    
    Metadata of Resources
    
    Bundles
    
    Clusters
    
    Computes
    
    Databases
    
    Depots
    
    Instance Secrets
    
    Lakehouses
    
    Lenses
    
    Monitors
    
    Operators
    
    Pagers
    
    Policies
    
    Secrets
    
    Services
    
    Stacks
    
    Workers
    
    Workflows
  - Notebook
    
    Notebook
  - Operations
    
    Operations
  - Workbench
    
    Workbench
- Resources
  Resources
  - Types
    
    Types
    
    Bundle
    
    Bundle
    
    Configurations
    
    Templates
    
    Cluster
    
    Cluster
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Connectors
    
    Setting Up CLI Client
    
    Tuning
    Tuning
    
    Query Execution Optimization
    
    Performance Tuning
    
    Recommended Configuration
    
    Examples
    Examples
    
    Restart and scale Cluster on a pre-defined schedule
    
    Multiple Cluster Setup using single manifest
    
    Core Concepts
    Core Concepts
    
    On-demand Computing
    
    Compute
    
    Compute
    
    Configurations
    
    Templates
    
    Database
    
    Database
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Query Database using Workbench
    
    Building Streamlit applications backed by Database
    
    Depot
    
    Depot
    
    Supported Sources
    
    Configurations
    
    Supported Storage Architecture
    Supported Storage Architecture
    
    Icebase
    
    Icebase
    
    Create, Get, and Drop Dataset
    
    Table Properties
    
    Schema Evolution
    
    Partitioning
    
    Maintenance (Snapshots and Metadata listing)
    
    Fastbase
    
    Supported Connectors
    
    Grant
    
    Grant
    
    Configuration
    
    Instance Secret
    
    Instance Secret
    
    First Steps
    
    Templates
    
    Configurations
    
    Recipes
    Recipes
    
    Referring Instance Secrets in other Resources
    
    Lakehouse
    
    Lakehouse
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Command Reference
    
    Iceberg Metadata Tables
    
    Recipes
    Recipes
    
    Ensure high data quality using Write-Audit-Publish (WAP) Pattern
    
    Use Iceberg metadata tables to extract insights
    
    Create, Get, and Drop Dataset
    
    Table Properties
    
    Schema Evolution
    
    Partitioning
    
    Maintenance (Snapshots and Metadata listing)
    
    Lens
    
    Lens
    
    Lens Setup
    Lens Setup
    
    Lens Model Folder Setup
    
    Develop Lens Locally
    
    Develop Lens in DataOS
    
    Data Modelling
    Data Modelling
    
    Overview
    
    Concepts
    
    Best Practices
    
    Do's and Dont's
    
    Errors
    
    Configurations
    Configurations
    
    Lens Manifest
    
    Docker Compose Manifest
    
    Supported Data Sources
    Supported Data Sources
    
    Bigquery
    
    Minerva
    
    Postgres
    
    Redshift
    
    Themis
    
    APIs & Integraions
    APIs & Integraions
    
    API Endpoints and Scopes
    
    BI Integration
    
    Reference
    Reference
    
    Working with Payload
    
    Working with Segments
    
    Working with Views
    
    Working with User Groups
    
    Exploration
    Exploration
    
    Using GraphQL
    
    Using SQL APIs
    
    Using Python
    
    Using REST APIs
    
    Optimizing Lens Model
    
    Monitor
    
    Monitor
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Add context columns to Monitors
    
    Create a Report Monitor
    
    Create a Stream Monitor
    
    Create an Equation Monitor on Lens
    
    Create an Equation Monitor on Postgres
    
    Generate incidents for certificate expiration
    
    Operator
    
    Operator
    
    Configurations
    
    Recipes
    Recipes
    
    Orchestrate Hightouch pipeline using Hightouch Factory Operator
    
    Pager
    
    Pager
    
    Configurations
    
    Recipes
    Recipes
    
    Create a Custom Body Template for Pager
    
    Policy
    
    Policy
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    End-to-end use-case on how to implement access policy
    
    Core Concepts
    
    Secret
    
    Secret
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Refer secrets in other DataOS Resources
    
    Setup secrets to pull images from a private container registry
    
    Service
    
    Service
    
    Configurations
    
    Stacks
    
    Stacks
    
    Beacon
    
    Beacon
    
    Benthos
    
    Benthos
    
    CLI Stack
    
    CLI Stack
    
    Container
    
    Container
    
    Data Toolbox
    
    Data Toolbox
    
    DBT
    
    DBT
    
    Flare
    
    Flare
    
    Flash
    
    Flash
    
    Scanner
    
    Scanner
    
    Soda
    
    Soda
    
    Steampipe
    
    Steampipe
    
    Talos
    
    Talos
    
    Setting Up Talos
    Setting Up Talos
    
    On Local
    
    On DataOS
    
    Configurations
    
    Configurations
    
    config.yaml
    
    docker-compose.yaml
    
    apis
    
    service.yaml
    
    Supported Sources
    Supported Sources
    
    Lens
    
    Flash
    
    Redshift
    
    Snowflake
    
    Postgres
    
    Recipes
    Recipes
    
    Apply data masking
    
    Fetch data from third party tools
    
    Caching datasets
    
    Generate API documentation
    
    External API as data source
    
    Example
    
    Best Practices
    Best Practices
    
    API
    
    SQL
    
    Custom Stack
    Custom Stack
    
    How to create your own Stack?
    
    Configurations
    
    Volume
    
    Volume
    
    Getting Started
    
    Configurations
    
    Worker
    
    Worker
    
    First Steps
    
    Configuration
    
    Core Concepts
    
    Workflow
    
    Workflow
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Implement Single-run Workflow
    
    Run Cron/Scheduled Workflow
    
    Orchestrate multiple Workflows from a single Workflow
    
    Retry a Job in Workflow
  - Characteristics
  - Configuration
  - YAML files
- SDK
  SDK
Getting Started
Getting Started
Data Product
Data Product
- Data Product Lifecycle
  Data Product Lifecycle
  - Design
  - Build
  - Deploy
  - Iterate
- Recipes
  Recipes
  - Create Data Product template using Cookiecutter
  - Deploy Data Product using CI/CD pipeline
- Examples
  Examples
  - Financial Services Accelerator Data Product
  - Sales 360 Data Product
- Configurations
- Core Concepts
Glossary
Learn
Learn
- Quick Start Guides
  Quick Start Guides
  - Connecting with Data Sources
    Connecting with Data Sources
    
    Using GUI
    
    Using GUI
    
    Using CLI
    
    Using CLI
  - Performing Exloratory Data Analysis
    Performing Exloratory Data Analysis
    
    Running your first data Query
    
    Running your first data Query
    
    Arranging and Transforming data with Pivot
    
    Arranging and Transforming data with Pivot
    
    Writing Complex Queries
    
    Writing Complex Queries
    
    Querying Diverse Data sources
    
    Querying Diverse Data sources
  - Using Data Modeling Layer
    Using Data Modeling Layer
    
    Defining a Data Model
    
    Defining a Data Model
    
    Creating a Data Model
    
    Creating a Data Model
    
    Testing a Data Model
    
    Testing a Data Model
    
    Deploying a Data Model
    
    Deploying a Data Model
    
    Exploring Lens on Catalog
    
    Exploring Lens on Catalog
    
    Working on Lens Studio
    
    Working on Lens Studio
    
    Working with Payloads
    
    Working with Payloads
    
    Example Scenarios
    
    Pre-defined Filtering using Segemnts
    
    Pre-defined Filtering using Segemnts
    
    Defining Data Policies
    
    Defining Data Policies
  - Building Data Products
  - Scanning Metadata
  - Creating ETL Pipelines
    Creating ETL Pipelines
    
    Batch Data Processing using Flare
    
    Batch Data Processing using Flare
    
    Storing Query Results as Dataset with Flare
    
    Storing Query Results as Dataset with Flare
  - Deploying Data Application Using Container Stack
- Videos

Connecting to sources via Depot/Cluster

Data Sources¶

Connecting to sources via Depot/Cluster¶

Was this page helpful?

Thanks for your feedback!

Thanks for your feedback!

Contact Us | Privacy Policy | Terms of Use

Copyright ⓒ 2024 The Modern Data Company