Skip to content

Data Toolbox

Data Toolbox Stack or simply Toolbox Stack provides vital functionality in case of metadata updation in Icebase depots. When data is ingested into Icebase using Flare Stack, the metadata of ingested datasets needs to be registered with Metis before it can be queried using Workbench. The Data Toolbox allows set_version action on the data stored in the DataOS internal storage Icebase, which uses the Iceberg format. The Metis keeps track of the Iceberg table by storing a reference to the latest metadata file. Using the Data Toolbox set_version action, you can update the metadata version to the latest or any specific version.

Syntax of Data Toolbox YAML Configuration

Data Toolbox YAML Configuration Syntax

Performing Data Toolbox Actions

You can write Data Toolbox action as a separate workflow or part of a dag in a workflow.

Toolbox Action as a separate Workflow

To perform a data toolbox action, follow the below steps:

Create a YAML file for Workflow

If you have already ingested data, you can create a separate workflow for toolbox action, as shown in the following YAML. To know more about workflows, click here.

Define a Job that executes upon Toolbox Stack

Within the DAG, define a job that executes upon the Toolbox Stack. To know more about the various properties for a job executed upon toolbox stack, click here.

Sample Toolbox Workflow

version: v1 
name: dataos-toolbox-workflow 
type: workflow 
    - name: dataos-toolbox-city-01 
      stack: toolbox 
      compute: runnable-default 
        dataset: dataos://icebase:sample/city?acl=rw 
          name: set_version 
          value: latest 
The table below elucidates the various attributes within the Toolbox-specific Section.

Field Data Type Default Value Possible Value Requirement
toolbox object none none mandatory
dataset string none any valid iceberg dataset udl address mandatory
action object none none mandatory
name string none set_version mandatory
value string none latest or any other specific metadata version mandatory

To know more about Toolbox-specfic Section YAML Configuration fields, click here.

Apply the Workflow using CLI

Use the apply command to apply the workflow using CLI

dataos-ctl apply -f <path/file-name> -w <workspace> # By default the workspace is public so you may not include the -w flag

Toolbox Action in a Job within the Flare Workflow

Instead of creating a separate workflow for Toolbox, you can create a single workflow. With a DAG with two jobs, one job runs on Flare Stack while the second one runs on the Toolbox Stack, which depends on the first job for the start of execution.

Sample Toolbox Stack Action in a Job within a Workflow
version: v1 # Version
name: wf-sample-002 # Workflow Name
type: workflow # Name of the Resource
tags: # Tags
- Con
description: The job ingests data using Flare and registers the metadata using Toolbox Stack # Description of the Workflow
workflow: # Workflow Section
  title: Connect City # Title
  dag: # Directed Acyclic Graph

        # Job 1 executed upon Flare Stack: This job ingests city data
  - name: wf-sample-job-001
    title: City Data Ingester
    description: The job ingests city data 
      - Connect
      - City
      stack: flare:3.0
      compute: runnable-default
          explain: true
          logLevel: INFO

           - name: city_connect
             dataset: dataos://thirdparty01:none/city
             format: csv
             schemaPath: dataos://thirdparty01:none/schemas/avsc/city.avsc

            - name: cities
              dataset: dataos://icebase:retail/city01?acl=rw
              format: Iceberg
              description: City data ingested from external csv
                saveMode: append
                  mode: partition
                    - name: version
                      order: desc
                    write.format.default: parquet
                    write.metadata.compression-codec: gzip
                    - type: identity
                      column: version
          - sequence:
              - name: cities
                doc: Pick all columns from cities and add version as yyyyMMddHHmm formatted
                sql: |
                      date_format (now(), 'yyyyMMddHHmm') AS version,
                      now() AS ts_city

    # Job 2 executes upon Toolbox Stack: This job registers latest version of metadata to Metis
  - name: data-tool-job-001
      stack: toolbox
      compute: runnable-default
        dataset: dataos://icebase:retail/city01
          name: set_version
          value: latest
    dependencies: wf-sample-job-001

Once you define the workflow, you can apply it using CLI.