Skip to content

All things DataOS

System functions

All things DataOS

Home
Home
- Philosophy
- Architecture
- Interfaces
  Interfaces
  - Audiences
    
    Audiences
    
    Audiences UI
    
    Getting Started
  - Bifrost
    
    Bifrost
    
    ABAC Implementation
    
    Users
    
    Use-cases
    
    Grant
    
    Role
    
    Grant Requests
    
    Heimdall Primitives
  - CLI
    
    CLI
    
    Installation
    
    Initialization
    
    Command Reference
  - Data Product Hub
    
    Data Product Hub
    
    Core Concepts
    
    Data Product Discovery
    
    Data Product Exploration
    
    Data Product Activation
    Data Product Activation
    
    BI Sync
    BI Sync
    
    Tableau Cloud
    
    Power BI
    
    Tableau Desktop
    
    Microsoft Excel
    
    Apache Superset
    
    AI/ML
    
    App Development
    
    Data APIs
    
    Best Practices
    
    Examples
  - Metis
    
    Metis
    
    Metis UI
    
    Metis UI
    
    Pre-defined Roles and Policies
    
    Key Features
    
    Metadata of Products
    
    Metadata of Products
    
    Metadata of Data Products
    
    Metadata of Data Assets
    
    Metadata of Data Assets
    
    Metadata of Tables
    
    Metadata of Topics
    
    Metadata of Dashboards
    
    Metadata of Resources
    
    Metadata of Resources
    
    Bundles
    
    Clusters
    
    Computes
    
    Databases
    
    Depots
    
    Instance Secrets
    
    Lakehouses
    
    Lenses
    
    Monitors
    
    Operators
    
    Pagers
    
    Policies
    
    Secrets
    
    Services
    
    Stacks
    
    Workers
    
    Workflows
  - Notebook
    
    Notebook
  - Operations
    
    Operations
  - Workbench
    
    Workbench
- Resources
  Resources
  - Types
    
    Types
    
    Bundle
    
    Bundle
    
    Configurations
    
    Templates
    
    Cluster
    
    Cluster
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Connectors
    
    Integration
    
    Integration
    
    Power BI
    
    Power BI
    
    Using Connector
    
    Using ODBC Driver
    
    Tableau
    
    IBM SPSS
    
    CLI Client
    
    Tuning
    Tuning
    
    Query Execution Optimization
    
    Performance Tuning
    
    Recommended Configuration
    
    Examples
    Examples
    
    Restart and scale Cluster on a pre-defined schedule
    
    Multiple Cluster Setup using single manifest
    
    Core Concepts
    Core Concepts
    
    On-demand Computing
    
    Compute
    
    Compute
    
    Configurations
    
    Templates
    
    Database
    
    Database
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Query Database using Workbench
    
    Building Streamlit applications backed by Database
    
    Depot
    
    Depot
    
    Supported Sources
    
    Configurations
    
    Supported Storage Architecture
    Supported Storage Architecture
    
    Icebase
    
    Icebase
    
    Create, Get, and Drop Dataset
    
    Table Properties
    
    Schema Evolution
    
    Partitioning
    
    Maintenance (Snapshots and Metadata listing)
    
    Fastbase
    
    Supported Connectors
    
    Grant
    
    Grant
    
    Configuration
    
    Instance Secret
    
    Instance Secret
    
    First Steps
    
    Templates
    
    Configurations
    
    Recipes
    Recipes
    
    Referring Instance Secrets in other Resources
    
    Lakehouse
    
    Lakehouse
    
    Configurations
    Configurations
    
    Manifest Reference
    
    Command Reference
    
    Iceberg Metadata Tables
    
    Recipes
    Recipes
    
    Ensure high data quality using Write-Audit-Publish (WAP) Pattern
    
    Use Iceberg metadata tables to extract insights
    
    Create, Get, and Drop Dataset
    
    Table Properties
    
    Schema Evolution
    
    Partitioning
    
    Maintenance (Snapshots and Metadata listing)
    
    Lens
    
    Lens
    
    Lens Setup
    Lens Setup
    
    Lens Model Folder Setup
    
    Develop Lens Locally
    
    Develop Lens in DataOS
    
    Data Modelling
    Data Modelling
    
    Overview
    
    Concepts
    
    Best Practices
    
    Do's and Dont's
    
    Errors
    
    Configurations
    Configurations
    
    Lens Manifest
    
    Docker Compose Manifest
    
    Supported Data Sources
    Supported Data Sources
    
    Bigquery
    
    Flash
    
    Minerva
    
    Postgres
    
    Redshift
    
    Themis
    
    APIs & Integrations
    APIs & Integrations
    
    API Endpoints and Scopes
    
    BI Integration
    
    Reference
    Reference
    
    Working with Payload
    
    Working with Segments
    
    Working with Views
    
    Working with User Groups
    
    Exploration
    Exploration
    
    Using GraphQL
    
    Using SQL APIs
    
    Using Python
    
    Using REST APIs
    
    Using Iris dashboard
    
    Optimizing Lens Model
    
    Monitor
    
    Monitor
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Add context columns to Monitors
    
    Create a Report Monitor
    
    Create a Stream Monitor
    
    Create an Equation Monitor on Lens
    
    Create an Equation Monitor on Postgres
    
    Generate incidents for certificate expiration
    
    Operator
    
    Operator
    
    Configurations
    
    Recipes
    Recipes
    
    Orchestrate Hightouch pipeline using Hightouch Factory Operator
    
    Pager
    
    Pager
    
    Configurations
    
    Recipes
    Recipes
    
    Create a Custom Body Template for Pager
    
    Policy
    
    Policy
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    End-to-end use-case on how to implement access policy
    
    Core Concepts
    
    Secret
    
    Secret
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Refer secrets in other DataOS Resources
    
    Setup secrets to pull images from a private container registry
    
    Service
    
    Service
    
    Configurations
    
    Stacks
    
    Stacks
    
    Beacon
    
    Beacon
    
    Benthos
    
    Benthos
    
    CLI Stack
    
    CLI Stack
    
    Container
    
    Container
    
    Data Toolbox
    
    Data Toolbox
    
    DBT
    
    DBT
    
    Flare
    
    Flare
    
    Flash
    
    Flash
    
    Create a Flash Service
    
    Configurations
    
    Supported Data Sources
    
    Monitoring Flash
    
    Best Practices
    
    Do's and Dont's
    
    Errors and Issues
    Errors and Issues
    
    Handling Empty Tables
    
    Recipes
    Recipes
    
    Use cached datasets in Lens models
    
    Use cached datsets in Talos APIs
    
    Scanner
    
    Scanner
    
    Soda
    
    Soda
    
    Steampipe
    
    Steampipe
    
    Talos
    
    Talos
    
    Setting Up Talos
    Setting Up Talos
    
    On Local
    
    On DataOS
    
    Configurations
    
    Configurations
    
    config.yaml
    
    docker-compose.yaml
    
    apis
    
    service.yaml
    
    Recipes
    Recipes
    
    Apply data masking
    
    Fetch data from third party tools
    
    Caching datasets
    
    Generate API documentation
    
    External API as data source
    
    Example
    
    Best Practices
    Best Practices
    
    API
    
    SQL
    
    Custom Stack
    Custom Stack
    
    How to create your own Stack?
    
    Configurations
    
    Volume
    
    Volume
    
    Getting Started
    
    Configurations
    
    Worker
    
    Worker
    
    First Steps
    
    Configuration
    
    Core Concepts
    
    Workflow
    
    Workflow
    
    First Steps
    
    Configurations
    
    Recipes
    Recipes
    
    Implement Single-run Workflow
    
    Run Cron/Scheduled Workflow
    
    Orchestrate multiple Workflows from a single Workflow
    
    Retry a Job in Workflow
  - Characteristics
  - Configuration
  - Best practices for YAML
- SDK
  SDK
Getting Started
Getting Started
Data Product
Data Product
- Data Product Lifecycle
  Data Product Lifecycle
  - Design
  - Build
  - Deploy
  - Iterate
- Recipes
  Recipes
  - Deploy Data Product using CI/CD pipeline
- Examples
  Examples
  - Product affinity Data Product
- Configurations
- Core Concepts
Glossary
Learn
Learn
- Learning tracks
  Learning tracks
  - Data Product Consumer
    
    Data Product Consumer
    
    Discovering Data Products
    Discovering Data Products
    
    Introduction to Data Product Hub
    
    Introduction to Data Product Hub
    
    Discovering Data Products
    
    Discovering Data Products
    
    Viewing Data Product info
    
    Viewing Data Product info
    
    Exploring Input and Output DataOS
    
    Exploring Input and Output DataOS
    
    Navigating semantic models
    
    Navigating semantic models
    
    Checking data quality
    
    Checking data quality
    
    Integrating Data Products with BI tools and Applications
    Integrating Data Products with BI tools and Applications
    
    Integration with BI tools
    
    Integration with BI tools
    
    Integration with AI/ML
    
    Integration with AI/ML
    
    Integration with Postgres
    
    Integration with Postgres
    
    Integration with GraphQL
    
    Integration with GraphQL
    
    Integration with data APIs
    
    Integration with data APIs
  - Data Product Developer
    
    Data Product Developer
    
    Disigning Data Products
    
    Disigning Data Products
    
    Building Data Products
    Building Data Products
    
    Source data connectivity
    
    Source data connectivity
    
    Connectivity with Postgres
    
    Connectivity with Postgres
    
    Building and maintaining data pipelines
    
    Building and maintaining data pipelines
    
    Creating your first data pipelines
    
    Creating your first data pipelines
    
    Sheduling Workflows
    
    Sheduling Workflows
    
    Data quality Checks
    
    Data quality Checks
    
    Pipeline observability
    
    Pipeline observability
    
    Build a semantic model
    
    Build a semantic model
    
    Defining business objectives and KPIs
    
    Defining business objectives and KPIs
    
    Designing the conceptual Model
    
    Designing the conceptual Model
    
    Key concepts of Lens
    
    Key concepts of Lens
    
    Creating semantic model
    
    Creating semantic model
    
    Test Lens Locally
    Test Lens Locally
    
    None
    
    Deploy Lens in DataOS
    
    Deploy Lens in DataOS
    
    Implementing quality Checks
    
    Implementing quality Checks
    
    Creating data APIs
    
    Creating data APIs
    
    Defining data policies for Lens
    
    Defining data policies for Lens
    
    Deploying Data Products
    Deploying Data Products
    
    Creating a deployable Bundles
    
    Creating a deployable Bundles
    
    Creating a Data Product Spec Fields
    
    Creating a Data Product Spec Fields
    
    Deploying Data Products
    
    Deploying Data Products
    
    Data Product CI/CD pipeline
    
    Data Product CI/CD pipeline
  - DataOS Operator
    
    DataOS Operator
    
    Credential security
    
    Credential security
    
    Data source connectivity
    
    Data source connectivity
    
    Connectivity with Postgres
    
    Connectivity with Postgres
    
    Routine checks
    
    Routine checks
    
    DataOS upgrade and rollback strategies
    
    DataOS upgrade and rollback strategies
    
    System monitoring
    System monitoring
    
    Configuring alerts and dashboards using Grafana
    
    Configuring alerts and dashboards using Grafana
    
    Optimizing system level events
    
    Optimizing system level events
    
    Query Cluster management
    Query Cluster management
    
    Cluster management
    
    Cluster management
    
    Cluster tuning
    
    Cluster tuning
- Quick Start Guides
  Quick Start Guides
  - Connecting with data sources
    Connecting with data sources
    
    Using GUI
    
    Using GUI
    
    Using CLI
    
    Using CLI
  - Performing exloratory data analysis
    Performing exloratory data analysis
    
    Running your first data query
    
    Running your first data query
    
    Arranging and transforming data with Pivot
    
    Arranging and transforming data with Pivot
    
    Writing complex queries
    
    Writing complex queries
    
    Querying diverse data sources
    
    Querying diverse data sources
  - Using data modeling layer
    Using data modeling layer
    
    Defining a Data Model
    
    Defining a Data Model
    
    Creating a data model
    
    Creating a data model
    
    Testing a data model
    
    Testing a data model
    
    Deploying a data model
    
    Deploying a data model
    
    Exploring Lens on Catalog
    
    Exploring Lens on Catalog
    
    Working on Lens Studio
    
    Working on Lens Studio
    
    Working with Payloads
    
    Working with Payloads
    
    Example scenarios
    
    Pre-defined filtering using Segemnts
    
    Pre-defined filtering using Segemnts
    
    Defining Data Policies
    
    Defining Data Policies
  - Building Data Products
  - Scanning metadata
  - Creating ETL pipelines
    Creating ETL pipelines
    
    Ingest external files into DataOS
    
    Ingest external files into DataOS
    
    Batch data processing using Flare
    
    Batch data processing using Flare
    
    Storing query results as dataset with Flare
    
    Storing query results as dataset with Flare
  - Deploying and securing data applications
    Deploying and securing data applications
    
    Deploying data application using Container Stack
    
    Deploying data application using Container Stack
    
    Securing deployed data applications
    
    Securing deployed data applications
- Videos

System functions¶

`version()`¶

Function	Description	Return Type
`version()`	Returns the Trino version used on the cluster.	`varchar`