Deploying Lens model on DataOS

Quick Guide

To quickly get started with lens deployment follow the quick guide on deploying your Lens model to walk through the process. This guide ensures a smooth and secure deployment, helping you get your Lens Resource up and running quickly.

This guide provides instructions for setting up Lens in the DataOS environment, importing an existing Lens project from a local machine using SSH, utilizing version control tools such as Bitbucket or Git for seamless CI/CD integration.

Pre-requisites

Before initiating the deployment process, please ensure you have the following:

• A directory containing Lens model.
• Access to a hosted code repository, such as Bitbucket, GitHub, AWS CodeCommit, etc.
• Operator level access permission

If you are new to setting the Lens model directory refer to the detailed doc here.

Deployment Process

Follow the steps below to deploy Lens on DataOS:

Step 1: Push the Lens model directory to a code repository

Pushing to a public code repository

First, push your Lens model directory to a hosted code repository. This can be done on hosted code repository like Bitbucket, GitHub, AWS CodeCommit etc.

Step 2: Create Instance Secrets for Code Repository Credentials

Create and configure an Instance Secret to secure your code repository credentials. This involves the following steps:

Create an Instance Secret manifest file

Define the Instance Secret Resource in a YAML file. Below is a template you can use for Bitbucket, substituting ${USERNAME} and ${PASSWORD} with your actual Bitbucket credentials:

SyntaxExample

# RESOURCE META SECTION
name: ${bitbucket-r }# Secret Resource name (mandatory)
version: v1 # Secret manifest version (mandatory)
type: instance-secret # Type of Resource (mandatory)
description: Bitbucket read secrets for code repository # Secret Resource description (optional)
layer: user # DataOS Layer (optional)

# INSTANCE SECRET-SPECIFIC SECTION
instance-secret: 
  type: key-value # Type of Instance-secret (mandatory)
  acl: r # Access control list (mandatory)
  data: # Data (mandatory)
    GITSYNC_USERNAME: ${USERNAME}
    GITSYNC_PASSWORD: ${PASSWORD}

# RESOURCE META SECTION
name: bitbucket-r # Secret Resource name (mandatory)
version: v1 # Secret manifest version (mandatory)
type: instance-secret # Type of Resource (mandatory)
description: Bitbucket read secrets for code repository # Secret Resource description (optional)
layer: user # DataOS Layer (optional)

# INSTANCE SECRET-SPECIFIC SECTION
instance-secret: 
  type: key-value # Type of Instance-secret (mandatory)
  acl: r # Access control list (mandatory)
  data: # Data (mandatory)
    GITSYNC_USERNAME: iamgroot
    GITSYNC_PASSWORD: <git_token>

Apply the Instance Secret manifest

Deploy the Instance Secret to DataOS using the apply command.

The apply command is as follows:

CommandExample

dataos-ctl resource apply -f ${manifest-file-path}

Alternatively, you can also use the below command:

dataos-ctl apply -f ${manifest-file-path}

Replace ${manifest-file-path} with the path to your YAML file.

dataos-ctl resource apply -f ./lens/instance_secret.yml
# Expected output
INFO[0000] 🛠 apply...                                   
INFO[0000] 🔧 applying bitbucket-r:v1:instance-secret... 
INFO[0001] 🔧 applying bitbucket-r:v1:instance-secret...created 
INFO[0001] 🛠 apply...complete 

Verify Instance-secret creation

To validate the proper creation of the Instance Secret Resource within the DataOS environment, employ the get command. Execute the following commands to ascertain the existence of the Instance Secret Resource:

To get the details of Instance-secret created by the user who applies the Instance-secret, use the following command:

CommandExample

dataos-ctl resource get -t instance-secret

Alternatively, you can also use the below command.

dataos-ctl get -t instance-secret

dataos-ctl get -t instance-secret
# Expected Output
INFO[0000] 🔍 get...                                     
INFO[0000] 🔍 get...complete                             

            NAME           | VERSION |      TYPE       | WORKSPACE | STATUS | RUNTIME |   OWNER          
---------------------------|---------|-----------------|-----------|--------|---------|-----------------------
  bitbucket-r              | v1      | instance-secret |           | active |         | iamgroot  

To get the details of Instance-secret created by all the users within the DataOS Instance, use the above command with -a flag:

CommandExample

dataos-ctl resource get -t instance-secret -a

Alternatively, you can use the following command:

dataos-ctl get -t instance-secret -a

dataos-ctl get -t instance-secret -a

# Expected Output
INFO[0000] 🔍 get...                                     
INFO[0000] 🔍 get...complete                             

            NAME           | VERSION |      TYPE       | WORKSPACE | STATUS | RUNTIME |        OWNER          
---------------------------|---------|-----------------|-----------|--------|---------|-----------------------
  abfss-r                  | v1      | instance-secret |           | active |         | thor          
  gcsdepot-r               | v1      | instance-secret |           | active |         | ironman            
  bitbucket-r              | v1      | instance-secret |           | active |         | iamgroot  

Step 3: Create a Lens Resource manifest file

A Lens manifest file incorporates the configuration of the Lens Resource. The manifest file for the Lens is composed of several sections which define the configuration of various components associated with the Lens Resource. The different sections of the Lens manifest file are provided below:

1. Resource meta section
2. Lens-specific section

# RESOURCE META SECTION
name: sales360test # Lens Resource name (mandatory)
version: v1alpha # Lens manifest version (mandatory)
layer: user # DataOS Layer (optional)
type: lens # Type of Resource (mandatory)
tags: # Tags (optional)
  - lens
description: sales360 lens deployment on DataOS # Lens Resource description (optional)

# LENS-SPECIFIC SECTION
lens:
  compute: runnable-default # Compute Resource that Lens should utilize (mandatory)
  secrets: # Referred Instance-secret configuration (**mandatory for private code repository, not required for public repository)
    - name: bitbucket-r # Referred Instance Secret name (mandatory)
      allKeys: true # All keys within the secret are required or not (optional)
  source: # Data Source configuration
    type: minerva # Source type 
    name: system # Source name
    catalog: icebase #in case of minerva or themis
  repo: # Lens
   model code repository configuration (mandatory)
    url: https://bitbucket.org/rubik_/customize-solutions # URL of repository containing the Lens model (mandatory)
    lensBaseDir: customize-solutions/lens/sales360/model # Relative path of the Lens 'model' directory in repository (mandatory)
    syncFlags: # Additional flags used during synchronization, such as specific branch.
      - --ref=lens # Repository Branch 
  api: # API Instances configuration (optional)
    replicas: 1 # Number of API instance replicas (optional)
    logLevel: info  # Logging granularity (optional)
    resources: # CPU and memory configurations for API Instances (optional)
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 2000m
        memory: 2048Mi
  worker: # Worker configuration (optional)
    replicas: 2 # Number of Worker replicas (optional)
    logLevel: info # Logging level (optional)
    resources: # CPU and memory configurations for Worker (optional)
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 6000m
        memory: 6048Mi
  router: # Router configuration (optional)
    logLevel: info  # Level of log detail (optional)
    resources: # CPU and memory resource specifications for the router (optional)
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 6000m
        memory: 6048Mi
  iris:
      logLevel: info
    resources: # CPU and memory resource specifications for the iris board (optional)
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 6000m
        memory: 6048Mi

For more information on how to configure a Lens manifest file, refer to the link: Configuration Fields of the Deployment Manifest File (YAML) for Lens Resource

Step 4: Apply the manifest file of Lens Resource using DataOS CLI

Deploy the Lens model to DataOS using the apply command.

The apply command is as follows:

CommandExample

dataos-ctl resource apply -f ${manifest-file-path} -w ${workspace}

Alternatively, you can also use the below command:

dataos-ctl apply -f ${manifest-file-path} -w ${workspace}

Replace ${manifest-file-path} with the path to your Lens manifest file, and ${workspace} with the name of the Workspace.

dataos-ctl resource apply -f ./lens/lens.yml -w curriculum
# Expected output
INFO[0000] 🛠 apply...                                   
INFO[0000] 🔧 applying(curriculum) sales360:v1alpha:lens... 
INFO[0001] 🔧 applying(curriculum) sales360:v1alpha:lens...created 
INFO[0001] 🛠 apply...complete 

Step 5: Validate Lens Resource creation

To validate the proper creation of the Lens Resource within the DataOS environment, employ the get command. Execute the following commands to ascertain the existence of the Lens Resource:

• To get the details of the Lens created by the user who applies the Lens, use the following command:

  dataos-ctl resource get -t lens -w ${workspace}
  

  Alternatively, you can also use the below command.

  dataos-ctl get -t lens -w ${workspace}
  

  Example

  dataos-ctl get -t lens -w curriculum
  # Expected Output
  INFO[0000] 🔍 get...                                     
  INFO[0000] 🔍 get...complete                             
  
        NAME     | VERSION | TYPE | WORKSPACE  | STATUS | RUNTIME |    OWNER     
  ---------------|---------|------|------------|--------|---------|--------------
      sales360   | v1alpha | lens | curriculum | active |         |  iamgroot
  

• To get the details of Lens created by all the users within the DataOS Instance, use the above command with -a flag:

  dataos-ctl resource get -t lens -w ${workspace} -a
  

  Alternatively, you can use the following command:

  dataos-ctl get -t lens -w ${workspace} -a
  

  Example

  dataos-ctl get -t lens -w curriculum -a
  # Expected Output
  INFO[0000] 🔍 get...                                     
  INFO[0000] 🔍 get...complete                             
  
        NAME      | VERSION | TYPE | WORKSPACE  | STATUS | RUNTIME |    OWNER     
  ----------------|---------|------|------------|--------|---------|--------------
      sales360    | v1alpha | lens | curriculum | active |         |  iamgroot
      customer360 | v1alpha | lens | curriculum | active |         |  thor
  

Next Step

Exploration of Deployed Lens:

• Exploration of deployed Lens using SQL APIs
• Exploration of deployed Lens using Python
• Exploration of deployed Lens using REST APIs
• Exploration of deployed Lens using GraphQL