Bundle¶

A Bundle Resource within DataOS serves as a declarative and standardized mechanism for deploying a collection of Resources, data products, or applications in a single operation. It empowers data developers with the capability to programmatically orchestrate the deployment, scheduling, creation, and dismantling of code and infrastructure resources linked to these data products and applications in a unified manner.

As implied by its name, the Bundle Resource aggregates various DataOS Resources into a flattened directed acyclic graph (DAG). Within this structure, each node represents a distinct DataOS Resource, interconnected through dependency relationships.

Why use a Bundle?¶

End-to-End Definition¶

The manifest file of the Bundle can be used to define the logical quantum for a data product - it encompasses the ‘code’ part of a ‘Data Product’. It collectively provides a comprehensive definition of a data product or application. This representation simplifies the application of software engineering best practices, including source control, code review, testing, and continuous integration/continuous deployment (CI/CD) processes.

Streamlined Deployment¶

The Bundle eliminates fragmented deployment and validation procedures enabling robust configuration management across multiple Workspaces and cloud environments.

Structure of a Bundle YAML manifest¶

The YAML manifest outlined below details the structure of a Bundle:

# Resource meta section
name: ${{my-bundle}}
version: v1alpha 
type: bundle
layer: user 
tags: 
  - ${{dataos:type:resource}}
  - ${{dataos:resource:bundle}}
description: ${{this bundle resource is for a data product}}
owner: ${{iamgroot}}

# Bundle-specific section
bundle:

  # Bundle Schedule section
  schedule:
    initialState: ${{initial state}}
    timezone: ${{time zone}}
    create:
      cron: ${{cron expression}}
    delete:
      cron: ${{cron expression}}

  # Bundle Workspaces section
  workspaces:
    - name: ${{bundlespace}} # Workspace name (mandatory)
      description: ${{this is a bundle workspace}} # Workspace description (optional)
      tags: # Workspace tags (optional)
        - ${{bundle}}
        - ${{myworkspace}}
      labels: # Workspace labels (optional)
        ${{purpose: testing}}
      layer: user # Workspace layer (mandatory)

  # Bundle Resources section
  resources:
    - id: ${{bundle-scanner}} # Resource ID (mandatory)
      workspace: ${{bundlespace}} # Workspace (optional)
      spec: # Resource spec (mandatory) 
        ${{resource spec manifest}}
      file: ${{/home/Desktop/bundle-scanner.yaml}} # Resource spec file (optional)
      dependencies: # Resource dependency (optional)
        - ${{bundle-depot}}
      dependencyConditions: # Resource dependency conditions (optional)
        - resourceId: ${{bundle-depot}} # Resource ID (mandatory)
          status: # Status dependency condition (optional)
            is: # Status is (optional)
              - ${{active}}
            contains: # Status contains (optional)
              - ${{activ}}
          runtime: # Runtime dependency condition (optional)
            is: # Runtime is (optional)
              - ${{running}}
            contains: # Runtime contains (optional)
              - ${{run}}

  # Additional properties section
  properties:
    ${{additional properties}}

  # Manage As User
  manageAsUser: ${{iamgroot}}

How to create a Bundle?¶

Data developers can create a Bundle Resource by creating a YAML manifest and applying it via the DataOS CLI.

Create a Bundle YAML manifest¶

A Bundle Resource YAML manifest can be structurally broken down into following sections:

Configure the Resource meta section¶

In DataOS, a Bundle is categorized as a Resource-type. The Resource meta section within the YAML manifest encompasses attributes universally applicable to all Resource-types. The provided YAML codeblock elucidates the requisite attributes for this section:

# Resource meta section
name: ${{my-bundle}} # Resource name (mandatory)
version: v1beta # Manifest version (mandatory)
type: bundle # Resource-type (mandatory)
tags: # Resource Tags (optional)
  - ${{dataos:type:resource}}
  - ${{dataos:resource:bundle}}
description: ${{This is a bundle yaml manifest}} # Resource Description (optional)
owner: ${{iamgroot}} # Resource Owner (optional)
bundle: # Bundle-specific section mapping(mandatory)
  ${{Attributes of Bundle-specific section}}

For more information, refer to the Attributes of Resource Meta Section.

Configure the Bundle-specific section¶

The Bundle-specific section contains attributes specific to the Bundle Resource. This section comprises of four high-level sections:

Schedule
Workspaces
Bundle Resources
Additional Properties

Each of these sections should be appropriately configured when creating a Bundle YAML manifest. The high-level structure of the various separate sections within the Bundle-specific section is provided in the YAML below:

bundle:
  schedule: # Bundle schedule section (optional)
    ${{attributes for scheduling the bundle}}
  workspaces: # Bundle workspaces section (optional)
    ${{attributes specific to workspace configuration}}
  resources: # Bundle resources section (mandatory)
    ${{attributes specific to bundle resources}}
  properties: # Addtional Properties (optional)
    ${{additional properties}}
  manageAsUser: ${{iamgroot}} # Manage As User (optional)

Attribute	Data Type	Default Value	Possible Value	Requirement
`bundle`	mapping	none	none	mandatory
`schedule`	mapping	none	none	optional
`workspaces`	list of mappings	none	none	optional
`resources`	list of mappings	none	none	optional
`properties`	mapping	none	none	optional
`manageAsUser`	string	UserID of Owner	UserID of use case assignee	optional

Bundle Schedule section

The Bundle Schedule section allows you to specify scheduling attributes for the Bundle Resource. You can schedule the creation or deletion of a Bundle at specific intervals using a cron-like schedule. The following YAML code block outlines the attributes specified in the Bundle Workspaces section:

bundle: # Bundle-specific section (mandatory)
  schedule: # Bundle Schedule section (optional)
    initialState: ${{create}} # Initial State of Bundle (mandatory)
    timezone: ${{Asia/Kolkata}} # Time Zone (mandatory)
    create: # Bundle creation cron (optional)
      - cron: ${{'5 0 24 1 *'}}
    delete: # Bundle deletion cron (optional)
      - cron: ${{'25 0 24 1 *'}}

Refer to the table below for a summary of the attributes within the Bundle Workspaces section. For detailed information about each attribute, please refer to the respective links provided in the attribute column.

Attribute	Data Type	Default Value	Possible Value	Requirement
`initialState`	string	none	create/delete	mandatory
`timezone`	string	none	valid timezone in the “Area/Location” format	mandatory
`create`	list of mappings	none	none	optional
`cron`	string	none	valid cron expression	mandatory
`delete`	list of mappings	none	none	optional

Bundle Workspaces section

Within a Bundle, you have the option to deploy Workspace-level Resources in a new Workspace. To achieve this, you must declare the specifications of this new Workspace and reference it within the corresponding DataOS Workspace-level Resource within the Bundle Resources section.

The following YAML code block outlines the attributes specified in the Bundle Workspaces section:

bundle: # Bundle-specific section (mandatory)
  workspaces: # Bundle Workspaces section (optional)
    - name: ${{bundlespace}} # Workspace name (mandatory)
      description: ${{this is a bundle workspace}} # Workspace description (optional)
      tags: # Workspace tags (optional)
        - ${{bundle}}
        - ${{myworkspace}}
      labels: # Workspace labels (optional)
        ${{purpose: testing}}
      layer: ${{user}} # Workspace layer (mandatory)

Refer to the table below for a summary of the attributes within the Bundle Workspaces section. For detailed information about each attribute, please refer to the respective links provided in the attribute column.

Attribute	Data Type	Default Value	Possible Value	Requirement
`name`	string	none	valid Workspace name	mandatory
`description`	string	none	any valid string	optional
`tags`	list of strings	none	any valid string	optional
`labels`	mapping	none	valid key-value pairs	optional
`layer`	string	none	user/system	mandatory

Bundle Resources section

The Bundle Resources section allows you to define the Resources that make up the data product/application and their dependencies in the form of a flattened DAG. Each node within this DAG represents a Resource interconnected through a set of dependencies. Using the dependency and dependencyConditions, relationship and conditions can be specified such that resource only instantiates when the right dependency condition is met either the correct status and runtime, else it doesn’t. The following YAML code block outlines the attributes specified in the Bundle Workspaces section:

bundle: # Bundle-specific section (mandatory)
  resources:
    - id: ${{bundle-scanner}} # Resource ID (mandatory)
      workspace: ${{bundlespace}} # Workspace (optional)
      spec: # Resource spec (mandatory) 
        ${{resource spec manifest}}
      file: ${{/home/Desktop/bundle-scanner.yaml}} # Resource spec file (optional)
      dependencies: # Resource dependency (optional)
        - ${{bundle-depot}}
      dependencyConditions: # Resource dependency conditions (optional)
        - resourceId: ${{bundle-depot}} # Resource ID (mandatory)
          status: # Status dependency condition (optional)
            is: # Status is (optional)
              - ${{active}}
            contains: # Status contains (optional)
              - ${{activ}}
          runtime: # Runtime dependency condition (optional)
            is: # Runtime is (optional)
              - ${{running}}
            contains: # Runtime contains (optional)
              - ${{run}}

Refer to the table below for a summary of the attributes within the Bundle Workspaces section. For detailed information about each attribute, please refer to the respective links provided in the attribute column.

Attribute	Data Type	Default Value	Possible Value	Requirement
`id`	string	none	valid string	mandatory
`workspace`	string	none	valid Workspace name	optional (Mandatory for Workspace-level Resources)
`spec`	mapping	none	valid Resource spec	optional
`file`	string	none	valid Resource config file path	optional
`dependencies`	list of strings	none	Resource ids within the DAG except the current one	optional
`dependencyConditions`	list of mappings	none	none	optional
`resourceId`	string	none	ID of dependent Resource	mandatory
`status`	mapping	none	none	optional
`is`	list of strings	none	none	optional
`contains`	list of strings	none	none	optional
`runtime`	mapping	none	none	optional

Additional Properties section

The Additional Properties section lets you include any additional key-value properties relevant to the Bundle Resource. The following YAML code block outlines the attributes specified in the additional properties section:

bundle: # Bundle-specific section (mandatory)
    properties: # Additional properties section (optional)
        ${{alpha: beta}}
        ${{gamma: sigma}}

Refer to the table below for a summary of the attributes within the additional properties section. For detailed information about each attribute, please refer to the respective links provided in the attribute column.

Attribute	Data Type	Default Value	Possible Value	Requirement
`properties`	mapping	none	properties in the form of key-value pairs	optional

Data developers can alter the customize the behaviour of Bundle Resources by configuring the sections and attributes as needed. For a detailed insights into the description and constraints of the attributes within the Bundle-specific section, please consult the 'Attributes of Bundle-specific Section' documentation page.

Sample Bundle YAML manifest

# Resource meta section
name: alphabeta
version: v1beta
type: bundle
tags:
  - dataproduct
  - product
description: This bundle resource is for product data product.
layer: "user"

# Bundle-specific section
bundle:

# Bundle workspaces section
  workspaces:
    - name: testingspace
      description: This workspace runs dataos bundle resource for demo
      tags:
        - dataproduct
        - product
        - bundleResource
      labels:
        "name": dataproductBundleResources
      layer: user

# Bundle resources section
  resources:
    - id: depot
      spec: 
        name: snowflakedepot01
        version: v1
        type: depot
        tags:
          - snowflake
          - depot
        layer: user
        depot:
          type: snowflake
          description: snowflake depot
          spec:
            warehouse: DATAOS_WAREHOUSE
            url: jk42400.europe-west4.gcp.snowflakecomputing.com
            database: snowflake_sample_data
          external: true
          connectionSecret: 
            - acl: rw 
              type: key-value-properties
              data: 
                username: XPLORERSDATA
                password: Priyansh@2007

    - id: scanner
      spec: 
        version: v1
        name: snowflakedepotscanner
        type: workflow
        tags:
          - Scanner
        title: Scan snowflake-depot
        description: |
          The purpose of this workflow is to scan snowflake and see if scanner works fine with a snowflake of depot.
        workflow:
          dag:
            - name: scan-snowflake
              title: Scan snowflake
              description: |
                The purpose of this job is to snowflake and see if scanner works fine with a snowflake type of depot.
              tags:
                - Scanner
              spec:
                stack: scanner:2.0
                compute: runnable-default
                # runAsUser: metis
                stackSpec:
                  depot: snowflakedepot01
                  sourceConfig:
                    config:
                      schemaFilterPattern:
                        includes:
                          - ^tpch_sf10$
                      tableFilterPattern:
                        includes:
                          - customer
                          - nation
                          - region
      workspace: testingspace
      dependencies:
        - depot
      dependencyConditions:
        - resourceId: depot
          status:
            is:
              - active

    - id: profiling
      spec: 
        version: v1
        name: snowflakedepotprofiling
        type: workflow
        tags:
          - profiling
        description: The job performs profiling on snowflake depot
        workflow:
          title: Snowflake Profiling
          dag:
          - name: snowflake-profile
            title: Snowflake Depot Profiling
            spec:
              stack: flare:4.0
              compute: runnable-default
              title: Profiling  
              persistentVolume:
                name: persistent-v
                directory: fides
              stackSpec:
                driver:
                  coreLimit: 1200m
                  cores: 1
                  memory: 1024m
                executor:
                  coreLimit: 1200m
                  cores: 1
                  instances: 1
                  memory: 1200m
                job:
                  explain: true
                  inputs:
                    - name: snowflake_profiled_data
                      dataset: dataos://snowflakedepot01:tpch_sf10/region?acl=rw
                      format: snowflake
                      options:
                        sfWarehouse: DATAOS_WAREHOUSE
                  logLevel: INFO
                  profile:
                    level: basic
                sparkConf:
                  - spark.sql.shuffle.partitions: 10
                  - spark.default.parallelism: 10
      workspace: testingspace
      dependencies:
        - scanner
      dependencyConditions:
        - resourceId: scanner
          status:
            is:
              - active
          runtime:
            is:
              - succeeded

Apply the Bundle YAML¶

After creating the Bundle YAML manifest, it's time to apply it to instantiate the Resource-instance in the DataOS environment. To apply the Bundle YAML file, utilize the apply command.

dataos-ctl apply -f ${{yaml config file path}}

# Sample
dataos-ctl apply -f /home/Desktop/my-bundle.yaml

Verify Bundle Creation¶

To ensure that your Bundle has been successfully created, you can verify it in two ways:

Check the name of the newly created bundle in the list of bundles created by you:

dataos-ctl get -t bundle

# Expected Output
INFO[0000] 🔍 get...                                     
INFO[0000] 🔍 get...complete                             

       NAME     | VERSION |  TYPE  | WORKSPACE | STATUS | RUNTIME |  OWNER      
----------------|---------|--------|-----------|--------|---------|----------------
    my-bundle   | v1beta  | bundle |           | active |         | iamgroot

Alternatively, retrieve the list of all bundles created in your organization:

dataos-ctl get -t bundle -a

# Expected Output
INFO[0000] 🔍 get...                                     
INFO[0000] 🔍 get...complete                             

       NAME     | VERSION |  TYPE  | WORKSPACE | STATUS | RUNTIME | OWNER      
----------------|---------|--------|-----------|--------|---------|----------------
    my-bundle   | v1beta  | bundle |           | active |         | iamgroot 
    bundle101   | v1beta  | bundle |           | active |         | thor

You can also access the details of any created Bundle through the DataOS GUI in the Operations App.

Deleting Bundles¶

When a Bundle Resource is deleted, it triggers the removal of all resources associated with that Bundle, including Workspaces (if there are any specified within the Bundle definition).

Before deleting the Bundle, you must delete all workloads or resources that are dependent on it. This step ensures that there are no dependencies left that could cause issues during deletion. Once it's done, use the delete command to remove the specific Bundle Resource-instance from the DataOS environment:

# METHOD 1
dataos-ctl delete -t bundle -n ${{name of bundle}}
# Sample
dataos-ctl delete -t bundle -n my-bundle

# METHOD 2
dataos-ctl delete -i "${{identifier string for a resource}}"
# Sample 
dataos-ctl delete -i "my-bundle | v1beta | bundle |      "

Bundle Templates¶

The Bundle templates serve as blueprints, defining the structure and configurations for various types of Bundles, making it easier for data developers to consistently deploy resources. To know more, refer to the link: Bundle Templates.

Bundle Command Reference¶

Here is a reference to the various commands related to managing Bundles in DataOS:

Applying a Bundle: Use the following command to apply a Bundle using a YAML configuration file:
```
dataos-ctl apply -f ${{yaml config file path}}
```
Get Bundle Status: To retrieve the status of a specific Bundle, use the following command:
```
dataos-ctl get -t bundle
```
Get Status of all Bundles: To get the status of all Bundles within the current context, use this command:
```
dataos-ctl get -t bundle -a
```
Generate Bundle JSON Schema: To generate the JSON schema for a Bundle with a specific version (e.g., v1alpha), use the following command:
```
dataos-ctl develop schema generate -t bundle -v v1alpha
```
Get Bundle JSON Resource Schema: To retrieve the JSON schema for a Bundle Resource from Poros with a specific version (e.g., v1alpha), use the following command:
```
dataos-ctl develop get resource -t bundle -v v1alpha
```
Delete Bundles: To delete a specific bundle, you can use the below command
```
dataos-ctl delete -t bundle -n ${{name of bundle}}
```