Skip to content

Attributes of Worker YAML

Structure of Worker YAML configuration

worker: 
    title: ${{benthos}}
    tags:
        - ${{tag1}}
        - ${{tag2}}
    replicas: ${{1}}
    autoscaling: 
        enabled: ${{enable autoscaling}}
        minReplicas: ${{1}}
        maxReplicas: ${{3}}
        targetMemoryUtilizationPercentage: ${{60}}
        targetCPUUtilizationPercentage: ${{70}}
    stack: ${{benthos}} # mandatory
    logLevel: INFO
    configs: 
        ${{alpha: beta}}
        ${{gamma: sigma}}
    envs: 
        ${{PULSAR: random}}
        ${{SSL: crazy}}
    secrets: 
        - ${{mysecret}}
    dataosSecrets:
        - name: ${{random}} # mandatory
            workspace: ${{delta}} # mandatory
            key: ${{hola}}
            keys: 
                - ${{list}}
                - ${{abcd}}
            allKeys: true/false
            consumptionType: hola
    dataosVolumes: 
        - name: devta
            directory: manish
            readOnly: true/false
            subPath: dev/hola
    tempVolume: hola
    persistentVolume: 
        name: devta
        directory: manish
        readOnly: true/false
        subPath: dev/hola
    compute: runnable-default # mandatory
    resources:
        requests:
            cpu: 1000m
            memory: 100Mi
        limits:
            cpu: 1000m
            memory: 100Mi
    dryRun: true/false
    runAsApiKey: abcdefghijklmnopqrstuvwxyz
    runAsUser: manishagrawal
    topology:
        - name: random # mandatory
            type: alpha # mandatory
            doc: new
            properties:
                random: lost 
            dependencies:
                - new1
                - new2 
    stackSpec/flare/benthos/toolbox/beacon/scanner/alpha: 
        ${{Stack-specific Attributes}}

Attributes configuration

worker

Description: The worker attribute is used to define a Worker Resource in YAML. This mapping defines the configuration for a Worker within the DataOS environment.

Data Type Requirement Default Value Possible Values
mapping mandatory none none

Example Usage:

worker:
    {} # Worker attributes go here

title

Description: The title attribute specifies the title of the Worker. It provides a descriptive title for the Worker.

Data Type Requirement Default Value Possible Values
string optional none any valid string

Example Usage:

title: benthos worker

tags

Description: The tags attribute is used to assign tags to the Worker. Tags are labels that can be used for categorization or identification in Metis.

Data Type Requirement Default Value Possible Values
list of strings optional none list of valid strings

Example Usage:

tags:
  - tag1
  - tag2

replicas

Description: The replicas attribute specifies the number of replicas for the Worker. Replicas are multiple instances of the Worker that run concurrently.

Data Type Requirement Default Value Possible Values
integer optional 1 any positive integer

Example Usage:

replicas: 1

autoscaling

Description: The autoscaling attribute is used to configure autoscaling for the Worker. Autoscaling allows the Worker to automatically adjust the number of replicas based on resource utilization.

Data Type Requirement Default Value Possible Values
mapping optional none none

Example Usage:

autoscaling:
  enabled: true
  minReplicas: 1
  maxReplicas: 3
  targetMemoryUtilizationPercentage: 60
  targetCPUUtilizationPercentage: 70

stack

Description: The stack attribute specifies the name of the stack that the Worker orchestrates.

Data Type Requirement Default Value Possible Values
string mandatory none valid stack name

Example Usage:

stack: benthos

logLevel

Description: The logLevel attribute sets the logging level for the Worker. It determines the verbosity of log messages generated by the Worker.

Data Type Requirement Default Value Possible Values
string optional INFO valid log levels (e.g., INFO, DEBUG, ERROR)

Example Usage:

logLevel: INFO

configs

Description: The configs attribute allows you to specify custom configurations for the Worker. It provides a way to customize the behavior of the Worker.

Data Type Requirement Default Value Possible Values
mapping optional none valid custom configurations in key-value format

Example Usage:

configs:
  alpha: beta
  gamma: sigma

envs

Description: The envs attribute is used to define environment variables for the Worker. These variables can be used to configure the Worker's runtime environment.

Data Type Requirement Default Value Possible Values
mapping optional none valid environment variable definitions

Example Usage:

envs:
    DATAOS_RESOURCE_LIFECYCLE_EVENT_TOPIC: '************************'
    DATAOS_RESOURCE_LIFECYCLE_EVENT_TOPIC_SUBS_NAME: '********************'
    DATAOS_TOPIC_COLLATED_RESOURCE_OFFSET: '******'

secrets

To be deprecated

Description: The secrets attribute specifies Secrets required by the Worker. Secrets are DataOS Resources that stored sensitive pieces of information securely within the Heimdall vault.

Data Type Requirement Default Value Possible Values
list of mappings optional none list of secret definitions

Example Usage:

secrets:
  - mysecret

dataosSecrets

Description: The dataosSecrets attribute is used to define DataOS Secret Resources required by the Worker. DataOS Secret Resources are securely managed secrets used within the Heimdall vault.

Data Type Requirement Default Value Possible Values
list of mappings optional none list of DataOS secret definitions

Example Usage:

dataosSecrets:
  - name: random        # mandatory
    workspace: delta    # mandatory
    key: hola
    keys:
      - list
      - abcd
    allKeys: true
    consumptionType: hola

dataosVolumes

Description: The dataosVolumes attribute specifies volumes required by the Worker. Volumes are used to provide storage for the Worker's data.

Data Type Requirement Default Value Possible Values
list of mappings optional none list of volume definitions

Example Usage:

dataosVolumes:
  - name: devolume
    directory: random
    readOnly: true
    subPath: "dev/hola"

tempVolume

Description: The tempVolume attribute specifies a temporary volume associated with the Worker. Temporary volumes are used for temporary storage.

Data Type Requirement Default Value Possible Values
string optional none valid volume name

Example Usage:

tempVolume: hola

persistentVolume

Description: The persistentVolume attribute specifies a persistent volume associated with the Worker. Persistent volumes are used for long-term storage.

Data Type Requirement Default Value Possible Values
mapping optional none valid persistent volume definition

Example Usage:

persistentVolume:
  name: devta
  directory: manish
  readOnly: true/false
  subPath: dev/hola

compute

Description: The compute attribute defines the Compute Resource referred by the Worker.

Data Type Requirement Default Value Possible Values
string mandatory none runnable-default or any other valid Compute Resource name

Example Usage:

compute: runnable-default

resources

Description: The resources attribute specifies the resource requests and limits for the Worker. It defines the computational resources required by the Worker.

Data Type Requirement Default Value Possible Values
mapping optional none valid resource requests and limits

Example Usage:

resources:
  requests:
    cpu: 1000m
    memory: 100Mi
  limits:
    cpu: 1000m
    memory: 100Mi

dryRun

Description: The dryRun attribute determines whether the Worker should run in a dry run mode. In dry run mode, the Worker doesn't perform actual execution but simulates the process.

Data Type Requirement Default Value Possible Values
boolean optional false true or false

Example Usage:

dryRun: false

runAsApiKey

Description: The runAsApiKey attribute specifies the DataOS API key used for running the Worker. It determines the identity under which the Worker operates.

Data Type Requirement Default Value Possible Values
string optional none valid DataOS API key

Example Usage:

runAsApiKey: abcdefghijklmnopqrstuvwxyz

runAsUser

Description: The runAsUser attribute defines the user identity under which the Worker runs. It specifies the user account associated with the Worker.

Data Type Requirement Default Value Possible Values
string optional none valid user-id

Example Usage:

runAsUser: iamgroot

topology

Description: The topology attribute is used to define the topology of the Worker. It specifies the elements and dependencies within the Worker's topology.

Data Type Requirement Default Value Possible Values
list of mappings mandatory none list of topology element definitions

Example Usage:

topology:
  - name: random            # mandatory
    type: alpha             # mandatory
    doc: new                # Documentation for the element
    properties:
      random: lost          # Custom properties for the element
    dependencies:
      - new1
      - new2

stackSpec

Attributes named flare/benthos/toolbox/beacon/scanner/alpha has been deprecated and will be removed in future releases, please replace with the generic stackSpec

Description: This attribute allows for specifying stack-specific attributes. These attributes are specific to the stack configuration used by the Worker.

Data Type Requirement Default Value Possible Values
mapping optional none valid stack-specific attributes

Example Usage:

stackSpec/flare/benthos/toolbox/beacon/scanner/alpha:
  ${{Stack-specific Attributes}}