Attributes of Worker Manifest¶

Structure of Worker manifest¶

worker_manifest_reference.yml

# RESOURCE META SECTION
# Attributes commmon across all DataOS Resources
name: ${{resource_name}} # Name of the Resource (e.g., my-first-worker)
version: v1beta # Manifest version of the Resource
type: worker # Type of Resource
tags: # Tags for categorizing the Resource
  - ${{tag_example_1}} # Tags (e.g., dataos:worker)
  - ${{tag_example_2}} # Additional tags (e.g., dataos:workspace:curriculum)
description: ${{resource_description}} # Description of the resource (e.g., Common attributes applicable to all DataOS Resources)
owner: ${{resource_owner}} # Owner of the Resource (e.g., iamgroot)
layer: ${{resource_layer}} # DataOS Layer (e.g., user, system)

# WORKER-SPECIFIC SECTION
# Attributes specific to Worker resource-type
worker: # Worker-specific configuration
  title: ${{worker_title}} # Title of the worker (e.g., benthos worker)
  tags:
    - ${{worker_tag1}} # Tags for the worker (e.g., dataos:worker)
    - ${{worker_tag2}} # Additional tags (e.g., benthos-worker)
  replicas: ${{worker_replicas}} # Number of replicas (e.g., 1)
  autoscaling: # Autoscaling configuration
    enabled: ${{autoscaling_enabled}} # Enable or disable autoscaling (e.g., true)
    minReplicas: ${{min_replicas}} # Minimum number of replicas (e.g., 1)
    maxReplicas: ${{max_replicas}} # Maximum number of replicas (e.g., 3)
    targetMemoryUtilizationPercentage: ${{memory_utilization}} # Target memory utilization percentage (e.g., 60)
    targetCPUUtilizationPercentage: ${{cpu_utilization}} # Target CPU utilization percentage (e.g., 70)
  stack: ${{worker_stack}} # Stack used (mandatory) (e.g., benthos)
  logLevel: ${{log_level}} # Logging level (e.g., INFO)
  configs: # Configuration settings
    ${{config_key1}}: ${{config_value1}} # Example configuration (e.g., alpha: beta)
    ${{config_key2}}: ${{config_value2}} # Additional configuration (e.g., gamma: sigma)
  envs: # Environment variables
    ${{env_key1}}: ${{env_value1}} # Example environment variable (e.g., PULSAR: random)
    ${{env_key2}}: ${{env_value2}} # Additional environment variable (e.g., SSL: crazy)
  secrets: 
    - ${{secret_name}} # List of secrets (e.g., mysecret)
  dataosSecrets: # DataOS secrets configuration
    - name: ${{secret_name}} # Name of the secret (mandatory) (e.g., random)
      workspace: ${{secret_workspace}} # Workspace (mandatory) (e.g., delta)
      key: ${{secret_key}} # Key (e.g., hola)
      keys: 
        - ${{secret_key1}} # List of keys (e.g., list)
        - ${{secret_key2}} # Additional key (e.g., abcd)
      allKeys: ${{all_keys_flag}} # Include all keys or not (e.g., true/false)
      consumptionType: ${{consumption_type}} # Type of consumption (e.g., hola)
  dataosVolumes: # DataOS volumes configuration
    - name: ${{volume_name}} # Name of the volume (e.g., devta)
      directory: ${{volume_directory}} # Directory (e.g., manish)
      readOnly: ${{read_only_flag}} # Read-only flag (e.g., true/false)
      subPath: ${{volume_subpath}} # Sub-path (e.g., dev/hola)
  tempVolume: ${{temp_volume_name}} # Temporary volume (e.g., hola)
  persistentVolume: # Persistent volume configuration
    name: ${{persistent_volume_name}} # Name of the volume (e.g., tempvolume)
    directory: ${{persistent_volume_directory}} # Directory (e.g., myvolume)
    readOnly: ${{persistent_volume_read_only}} # Read-only flag (e.g., true/false)
    subPath: ${{persistent_volume_subpath}} # Sub-path (e.g., dev/random)
  compute: ${{compute_resource_name}} # Compute configuration (mandatory) (e.g., runnable-default)
  resources: # Resource requests and limits
    requests:
      cpu: ${{cpu_request}} # CPU request (e.g., 1000m)
      memory: ${{memory_request}} # Memory request (e.g., 100Mi)
    limits:
      cpu: ${{cpu_limit}} # CPU limit (e.g., 1000m)
      memory: ${{memory_limit}} # Memory limit (e.g., 100Mi)
  dryRun: ${{dry_run_flag}} # Dry run flag (e.g., true/false)
  runAsApiKey: ${{api_key}} # API key for running the worker (e.g., abcdefghijklmnopqrstuvwxyz)
  runAsUser: ${{run_as_user}} # User to run the worker as (e.g., manishagrawal)
  topology: # Topology configuration
    - name: ${{topology_name}} # Name of the topology (mandatory) (e.g., random)
      type: ${{topology_type}} # Type of the topology (mandatory) (e.g., alpha)
      doc: ${{topology_doc}} # Documentation link or description (e.g., new)
      properties:
        ${{property_key}}: ${{property_value}} # Example property (e.g., random: lost)
      dependencies: # List of dependencies
        - ${{dependency1}} # Example dependency (e.g., new1)
        - ${{dependency2}} # Additional dependency (e.g., new2)
# STACK-SPECIFIC SECTION
# Attributes specific to the choosen Stack
  stackSpec: 
    ${{stack_specific_attributes}} # Additional stack-specific attributes (e.g., Stack-specific Attributes)

Configuration¶

Resource meta section¶

This section serves as the header of the manifest file, defining the overall characteristics of the Worker Resource you wish to create. It includes attributes common to all types of Resources in DataOS. These attributes help DataOS in identifying, categorizing, and managing the Resource within its ecosystem. To learn about the Resources of this section, refer to the following link: Attributes of Resource meta section.

Worker-specific section¶

This section comprises attributes specific to the Worker Resource. The attributes within the section are listed below:

`worker`¶

Description: The worker attribute defines a mapping that contains attributes specific to the Worker resource-type.

Data Type	Requirement	Default Value	Possible Values
mapping	mandatory	none	none

Example Usage:

name: benthos3-worker-sample
version: v1beta
type: worker
tags:
    - worker
    - dataos:type:resource
    - dataos:resource:worker
    - dataos:layer:user
    - dataos:workspace:public
description: Random User Console
worker:
    tags:
    - worker
    replicas: 1
    stack: benthos-worker:3.0
    logLevel: DEBUG
    compute: runnable-default
    stackSpec:
    input:
        http_client:
        headers:
            Content-Type: application/octet-stream
        url: https://randomuser.me/api/
        verb: GET
    output:
        stdout:
        codec: |
            delim:
            -----------GOOD------------

`title`¶

Description: The title attribute specifies the title of the Worker. It provides a descriptive title for the Worker.

Data Type	Requirement	Default Value	Possible Values
string	optional	none	any valid string

Example Usage:

title: benthos worker

`tags`¶

Description: The tags attribute is used to assign tags to the Worker. Tags are labels that can be used for categorization or identification in Metis.

Data Type	Requirement	Default Value	Possible Values
list of strings	optional	none	list of valid strings

Example Usage:

tags:
  - tag1
  - tag2

`replicas`¶

Description: The replicas attribute specifies the number of replicas for the Worker. Replicas are multiple instances of the Worker that run concurrently.

Data Type	Requirement	Default Value	Possible Values
integer	optional	1	any positive integer

Example Usage:

replicas: 1

`autoscaling`¶

Description: The autoscaling attribute is used to configure autoscaling for the Worker. Autoscaling allows the Worker to automatically adjust the number of replicas based on resource utilization.

Data Type	Requirement	Default Value	Possible Values
mapping	optional	none	none

Example Usage:

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 3
  targetMemoryUtilizationPercentage: 60
  targetCPUUtilizationPercentage: 70

`stack`¶

Description: The stack attribute specifies the name of the stack that the Worker orchestrates.

Data Type	Requirement	Default Value	Possible Values
string	mandatory	none	valid stack name

Example Usage:

stack: benthos

`logLevel`¶

Description: The logLevel attribute sets the logging level for the Worker. It determines the verbosity of log messages generated by the Worker.

Data Type	Requirement	Default Value	Possible Values
string	optional	INFO	valid log levels (e.g., INFO, DEBUG, ERROR)

Example Usage:

logLevel: INFO

`configs`¶

Description: The configs attribute allows you to specify custom configurations for the Worker. It provides a way to customize the behavior of the Worker.

Data Type	Requirement	Default Value	Possible Values
mapping	optional	none	valid custom configurations in key-value format

Example Usage:

configs:
  alpha: beta
  gamma: sigma

`envs`¶

Description: The envs attribute is used to define environment variables for the Worker. These variables can be used to configure the Worker's runtime environment.

Data Type	Requirement	Default Value	Possible Values
mapping	optional	none	valid environment variable definitions

Example Usage:

envs:
    DATAOS_RESOURCE_LIFECYCLE_EVENT_TOPIC: '************************'
    DATAOS_RESOURCE_LIFECYCLE_EVENT_TOPIC_SUBS_NAME: '********************'
    DATAOS_TOPIC_COLLATED_RESOURCE_OFFSET: '******'

`secrets`¶

To be deprecated

Description: The secrets attribute specifies Secrets required by the Worker. Secrets are DataOS Resources that stored sensitive pieces of information securely within the Heimdall vault.

Data Type	Requirement	Default Value	Possible Values
list of mappings	optional	none	list of secret definitions

Example Usage:

secrets:
  - mysecret

`dataosSecrets`¶

Description: The dataosSecrets attribute is used to define DataOS Secret Resources required by the Worker. DataOS Secret Resources are securely managed secrets used within the Heimdall vault.

Data Type	Requirement	Default Value	Possible Values
list of mappings	optional	none	list of DataOS secret definitions

Example Usage:

dataosSecrets:
  - name: random        # mandatory
    workspace: delta    # mandatory
    key: hola
    keys:
      - list
      - abcd
    allKeys: true
    consumptionType: hola

`dataosVolumes`¶

Description: The dataosVolumes attribute specifies volumes required by the Worker. Volumes are used to provide storage for the Worker's data.

Data Type	Requirement	Default Value	Possible Values
list of mappings	optional	none	list of volume definitions

Example Usage:

dataosVolumes:
  - name: devolume
    directory: random
    readOnly: true
    subPath: "dev/hola"

`tempVolume`¶

Description: The tempVolume attribute specifies a temporary volume associated with the Worker. Temporary volumes are used for temporary storage.

Data Type	Requirement	Default Value	Possible Values
string	optional	none	valid volume name

Example Usage:

tempVolume: hola

`persistentVolume`¶

Description: The persistentVolume attribute specifies a persistent volume associated with the Worker. Persistent volumes are used for long-term storage.

Data Type	Requirement	Default Value	Possible Values
mapping	optional	none	valid persistent volume definition

Example Usage:

persistentVolume:
  name: devta
  directory: manish
  readOnly: true/false
  subPath: dev/hola

`compute`¶

Description: The compute attribute defines the Compute Resource referred by the Worker.

Data Type	Requirement	Default Value	Possible Values
string	mandatory	none	runnable-default or any other valid Compute Resource name

Example Usage:

compute: runnable-default

`resources`¶

Description: The resources attribute specifies the resource requests and limits for the Worker. It defines the computational resources required by the Worker.

Data Type	Requirement	Default Value	Possible Values
mapping	optional	none	valid resource requests and limits

Example Usage:

resources:
  requests:
    cpu: 1000m
    memory: 100Mi
  limits:
    cpu: 1000m
    memory: 100Mi

`dryRun`¶

Description: The dryRun attribute determines whether the Worker should run in a dry run mode. In dry run mode, the Worker doesn't perform actual execution but simulates the process.

Data Type	Requirement	Default Value	Possible Values
boolean	optional	false	true or false

Example Usage:

dryRun: false

`runAsApiKey`¶

Description: The runAsApiKey attribute specifies the DataOS API key used for running the Worker. It determines the identity under which the Worker operates.

Data Type	Requirement	Default Value	Possible Values
string	optional	none	valid DataOS API key

Example Usage:

runAsApiKey: abcdefghijklmnopqrstuvwxyz

`runAsUser`¶

Description: The runAsUser attribute defines the user identity under which the Worker runs. It specifies the user account associated with the Worker.

Data Type	Requirement	Default Value	Possible Values
string	optional	none	valid user-id

Example Usage:

runAsUser: iamgroot

`topology`¶

Description: The topology attribute is used to define the topology of the Worker. It specifies the elements and dependencies within the Worker's topology.

Data Type	Requirement	Default Value	Possible Values
list of mappings	mandatory	none	list of topology element definitions

Example Usage:

topology:
  - name: random            # mandatory
    type: alpha             # mandatory
    doc: new                # Documentation for the element
    properties:
      random: lost          # Custom properties for the element
    dependencies:
      - new1
      - new2

Stack-specific section¶

`stackSpec`¶

Attributes named flare/benthos/toolbox/beacon/scanner/alpha has been deprecated and will be removed in future releases, please replace with the generic stackSpec

Description: This attribute allows for specifying stack-specific attributes. These attributes are specific to the stack configuration used by the Worker.

Data Type	Requirement	Default Value	Possible Values
mapping	optional	none	valid stack-specific attributes

Example Usage:

stackSpec/flare/benthos/toolbox/beacon/scanner/alpha:
  ${{Stack-specific Attributes}}

Attributes of Worker Manifest¶

Structure of Worker manifest¶

Configuration¶

Resource meta section¶

Worker-specific section¶

worker¶

title¶

tags¶

replicas¶

autoscaling¶

stack¶

logLevel¶

configs¶

envs¶

secrets¶

dataosSecrets¶

dataosVolumes¶

tempVolume¶

persistentVolume¶

compute¶

resources¶

dryRun¶

runAsApiKey¶

runAsUser¶

topology¶

Stack-specific section¶

stackSpec¶

`worker`¶

`title`¶

`tags`¶

`replicas`¶

`autoscaling`¶

`stack`¶

`logLevel`¶

`configs`¶

`envs`¶

`secrets`¶

`dataosSecrets`¶

`dataosVolumes`¶

`tempVolume`¶

`persistentVolume`¶

`compute`¶

`resources`¶

`dryRun`¶

`runAsApiKey`¶

`runAsUser`¶

`topology`¶

`stackSpec`¶