Bundle Templates

Template 1

The YAML template defines a DataOS Bundle Resource that automates the deployment of interconnected resources. The first resource defines a Snowflake Depot, which is configured with connection details such as warehouse, URL, and database. Once the Snowflake Depot is created, a data ingestion workflow is triggered. It reads data from an external source (CSV) and writes it into the Snowflake Depot. The data ingestion is dependent on the successful creation of the Snowflake Depot, ensuring the operations are performed in the correct order. Modify this template to manage your own resources and workflows in Bundle Resource.

bundle_template.yml

name: dataosresourcebundle
version: v1alpha
type: bundle
tags: 
  - bundle resource
  - resource
description: this is a bundle resource
layer: user
bundle:
  schedule: 
    initialState: create
    timezone: Asia/Kolkata
    create:
     - cron: '31 0 * * *'
  workspaces:
    - name: bundletesting
      description: this is a workspace for bundle testing
      layer: user

  resources: 
    - id: snowflakedepot
      spec:
        name: snowflakedepotxy
        version: v1
        type: depot
        tags: 
          - snowflake
          - depot
        description: this is a snowflake depot
        layer: user
        depot: 
          type: snowflake
          description: this is not a snowflake depot
          connectionSecret: 
            - acl: rw
              type: key-value-properties
              data: 
                username: iamgroot
                password: iamavenger
          external: true
          spec: 
            warehouse: random_warehouse
            url: https://abcd.west-usa.azure.snowflakecomputing.com
            database: newone

    - id: write-data-to-snowflake12
      workspace: public
      spec:
        version: v1
        name: write-data-to-snowflake1
        type: workflow
        tags:
          - Connect
          - read
          - write
        description: Jobs writes data to snowflake and reads from it
        workflow:
          title: Connect Snowflake
          dag:
            - name: write-snowflake-06
              title: Reading data and writing to snowflake
              description: This job writes data to wnowflake
              spec:
                tags:
                  - Connect
                  - write
                stack: flare:6.0
                compute: runnable-default
                stackSpec:
                  job:
                    explain: true
                    inputs:
                      - name: city_connect
                        dataset: dataos://thirdparty01:none/city
                        format: csv
                        schemaPath: dataos://thirdparty01:none/schemas/avsc/city.avsc
                    logLevel: INFO
                    outputs:
                      - name: city_connect
                        dataset: dataos://snowflakedepot:beta/secondtable?acl=rw
                        format: snowflake
                        description: City data ingested from external csv
                        title: City Source Data
                        options:
                          extraOptions:
                            sfWarehouse: newone
      dependencies: 
        - snowflakedepot