How to refer Secrets in other DataOS Resources?¶
To refer the stored secret data in DataOS, you can reference them in your code using the secrets and dataosSecrets identifier. These identifiers ensure secure referencing of Secrets across different resources, enhancing system security and operational integrity.
Syntax
dataosSecrets:
- name: ${your-secret-name} # Mandatory
workspace: ${secret-workspace} # Optional
key: ${key of your secret} # Optional, used when only single key is required.
keys: # Optional, used when multiple key is required.
- ${secret_key}
- ${secret-key}
allKeys: ${true-or-false} # Optional
consumptionType: ${envVars} # Optional, possible values: envVars, propfile and file.
secrets:
- name: ${your-secret-name} # Mandatory
workspace: ${secret-workspace} # Optional
key: ${key of your secret} # Optional, used when only single key is required.
keys: # Optional, used when multiple key is required.
- ${secret_key}
- ${secret-key}
allKeys: ${true-or-false} # Optional
consumptionType: ${envVars} # Optional, possible values: envVars, propfile and file.
Let's see how you can refer secrets in various resources:
In addition to serving as a conduit for real-time and streaming data exchanges, the Service Resource within DataOS incorporates Secrets for secure access to confidential information. This ensures data privacy, and regulatory compliance, and facilitates timely insights and responses to dynamic information.
name: service-secret
version: v1
type: secret
tags:
- dataos:type:secret
description: This is a sample Secret YAML configuration
owner: iamgroot
secret:
type: key-value
acl: r
data:
MSTEAM_WEBHOOK_URL: ${MSTEAM_WEBHOOK_URL}
DATAOS_API_TOKEN: ${DATAOS_API_TOKEN}
DATAOS_ENV_LINK: ${DATAOS_ENV_LINK}
DATAOS_PULSAR_TOPIC_SUB_ID: ${DATAOS_PULSAR_TOPIC_SUB_ID}
version: v1
name: ${resource-name}
type: ${resource-type}
service:
title: ${workflow-alerts}
replicas: 1
stack: container
compute: runnable-default
resources:
requests:
cpu: 100m
memory: 500m
limits:
cpu: 1
memory: 1Gi
dataosSecrets: # Referencing the Secret
- name: ${secret-name}
workspace: public
keys:
- ${MSTEAM_WEBHOOK_URL}
- ${DATAOS_API_TOKEN}
- ${DATAOS_ENV_LINK}
- ${DATAOS_PULSAR_TOPIC_SUB_ID}
stackSpec:
image: labs/ls_workflow_alerts:2.0
imagePullSecret: modern-docker-secret
command:
- python
arguments:
- -u
- ./wf-failed-alerts.py
The Workflow in DataOS serves as a Resource for orchestrating data processing tasks with dependencies. It enables the creation of complex data workflows by defining a hierarchy based on a dependency mechanism some requiring access to sensitive information such as API keys, authentication tokens, or database credentials. Instead of embedding these secrets directly in the workflow configuration, it is advisable to leverage references to the Secret Resource.
name: service-secret
version: v1
type: secret
tags:
- dataos:type:secret
description: This is a sample Secret YAML configuration
owner: iamgroot
secret:
type: key-value
acl: r
data:
API_KEY: ${API_KEY}
DATAOS_API_TOKEN: ${DATAOS_API_TOKEN}
DATAOS_ENV_LINK: ${DATAOS_ENV_LINK}
DATAOS_PULSAR_TOPIC_SUB_ID: ${DATAOS_PULSAR_TOPIC_SUB_ID}
version: v1
name: {workflow-name}
type: workflow
workflow:
dag:
- name: ${alpha-wf-mail-alert}
spec:
resources:
requests:
cpu: 250m
memory: 500m
limits:
cpu: 1
memory: 1Gi
dataosSecrets: # Referencing the Secret
- name: ${secret-name}
workspace: public
keys:
- ${API_KEY}
- ${DATAOS_API_TOKEN}
- ${DATAOS_ENV_LINK}
- ${DATAOS_PULSAR_TOPIC_SUB_ID}
stack: ${stack-name}
compute: runnable-default
stackSpec:
image: rubiklabs/workflow_lobos_mail_alert:1.0
imagePullSecret: modern-docker-secret
command:
- python
arguments:
- -u
- ./email_alert_script.py
A Worker Resource in DataOS is a long-running process responsible for performing specific tasks or computations indefinitely. Workers are capable of securely accessing confidential information, such as API keys, through the referencing of secrets, thereby ensuring the safeguarding of sensitive data.
name: benthos-worker-secret
version: v1
type: secret
tags:
- dataos:type:secret
description: This is a sample Secret YAML configuration
owner: iamgroot
secret:
type: key-value
acl: r
data:
runAsApiKey: dthtyurZW5fY29tbW9ubHllX21vcmF5LmFlNmI2YzBkLTI0ZGEtNDI0NDFmhgfghfdrZQ
runAsUser: iamgroot
name: benthos3-worker-sample-replicas
version: v1beta
type: worker
tags:
- worker
- dataos:type:resource
- dataos:resource:worker
- dataos:layer:user
- dataos:workspace:public
description: Random User Console
owner: iamgroot
workspace: public
worker:
tags:
- worker
- random-user
replicas: 3
stack: benthos
logLevel: DEBUG
compute: runnable-default
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 1000m
memory: 1024Mi
dataosSecrets:
- name: benthos-worker-secret
workspace: public
stackSpec:
input:
http_client:
url: https://randomuser.me/api/
verb: GET
headers:
Content-Type: application/JSON
pipeline:
processors:
- label: my_blobl
bloblang: |
page = this.info.page
age = this.results.0.dob.age
dob = this.results.0.dob.date
seed = this.info.seed
email = this.results.0.email
gender = this.results.0.gender
name = this.results.0.id.name
city = this.results.0.location.city
output:
broker:
outputs:
- broker:
pattern: fan_out
outputs:
- plugin:
address: dataos://fastbase:default/test001
metadata:
auth:
token:
enabled: true
token: dthtyurZW5fYbW9ubHlfccmdlX21vcmF5LmFlNmI2YzBkLTI0ZGEtNDI0Ny1hMjUyLTk0YTdjNDFmhgfghfdrZQ==
description: Random users data
format: AVRO
schema: "{\"name\":\"default\",\"type\":\"record\",\"namespace\":\"defaultNamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"city\",\"type\":\"string\"},{\"name\":\"dob\",\"type\":\"string\"},{\"name\":\"email\",\"type\":\"string\"},{\"name\":\"gender\",\"type\":\"string\"},{\"name\":\"name\",\"type\":\"string\"},{\"name\":\"page\",\"type\":\"int\"},{\"name\":\"seed\",\"type\":\"string\"}]}"
schemaLocation: http://registry.url/schemas/ids/12
title: Random Uses Info
type: STREAM
type: dataos_depot
- stdout: {}
A Cluster in DataOS is a Resource that encompasses a set of computational resources and configurations necessary for executing data engineering and analytics tasks. Clusters are capable of securely accessing confidential information through the referencing of secrets, thereby ensuring the safeguarding of sensitive data.
version: v1
name: mycluster
type: cluster
tags:
- cluster
- minerva
cluster:
compute: query-default
type: minerva
minerva:
replicas: 1
resources:
limits:
cpu: 2000m
memory: 4Gi
requests:
cpu: 2000m
memory: 4Gi
debug:
logLevel: INFO
trinoLogLevel: ERROR
depots:
- address: dataos://grootbigquery
dataosSecrets:
- name: cluster-secrets
workspace: public
allKeys: true