Attributes of Depot YAML Configuration

Structure of Depot YAML

depot:
  type: ${{S3}}                                          
  description: ${{description}}
  external: ${{true}}
  source: ${{metadata}}
  compute: ${{runnable-default}}
  connectionSecret:                                
    - acl: ${{rw}}
      type: key-value-properties
      data:
        ${{data-source-specific-connection-secrets}}
  spec:
    ${{data-source-specifications}}

Structure of Depot YAML configuration

Attributes Configuration

depot

Description: specifies the configuration for the Depot section

Data Type	Requirement	Default Value	Possible Value
mapping	mandatory	none	none

Example Usage:

depot:
  {}

type

Description: Specifies the type of Depot

Data Type	Requirement	Default Value	Possible Value
string	mandatory	none	ABFSS, WASBS, REDSHIFT, S3, ELASTICSEARCH, EVENTHUB, PULSAR, BIGQUERY, GCS, JDBC, MSSQL, MYSQL, OPENSEARCH, ORACLE, POSTGRES, SNOWFLAKE

Example Usage:

type: ABFSS

description

Description: Provides a description for the Depot

Data Type	Requirement	Default Value	Possible Value
string	optional	none	any string

Example Usage:

description: Azure Blob Storage Depot

external

Description: Specifies whether the depot is external. Set to true if the depot is external

Data Type	Requirement	Default Value	Possible Value
string	mandatory	false	true/false

Example Usage:

external: true

source

Description: Maps the depot to the metadata source name in Metis. Running a scanner job on this depot will save the metadata in Metis DB under the specified 'source' name. If this key-value property is not mentioned, the metadata will surface under the depot name on Metis UI.

Data Type	Requirement	Default Value	Possible Value
string	optional	depot name	any string

Example Usage:

source: bigquerymetadata

compute

Description: Specifies the compute resource for the depot being created.

Data Type	Requirement	Default Value	Possible Value
string	optional	runnable-default	any Compute Resource

Example Usage:

compute: runnable-default

connectionSecret

Description: Specifies the connection secrets for the data source.

Data Type	Requirement	Default Value	Possible Value
mapping	optional	none	varies between data sources

Example Usage:

connectionSecret:
  {} 

acl

Description: Declares the access policy for the depot. Multiple connections with different access levels can be created for the same depot. For example, read and write permissions for a specific transformation and read-only permission for another operation using the same depot.

Data Type	Requirement	Default Value	Possible Value
string	optional	r	r/rw

Example Usage:

acl: rw

type

Description: Specifies the type of Secret

Data Type	Requirement	Default Value	Possible Value
string	optional	key-value-properties	key-value-properties

Example Usage:

type: key-value-properties 

data

Description: Provides the credentials and additional information needed to connect with the data source, such as access key ID, secret key ID, username, and passwords.

Data Type	Requirement	Default Value	Possible Value
mapping	optional	none	varies between data sources

Example Usage:

data:
  projectid: project-name
  email: iamgroot@tmdc.io

files

Description: Allows storing sensitive information in a separate JSON file. Specify the absolute path to the JSON file containing the credentials.

Data Type	Requirement	Default Value	Possible Value
string	optional	none	valid absolute path

Example Usage:

files:
  json_keyfile: secrets/gcp-demo-sa.json

spec

Description: Specifies the precise location of the data and provides the hierarchical structure in which the data is stored.

Data Type	Requirement	Default Value	Possible Value
mapping	mandatory	none	varies between data sources

Example Usage:

spec:
  host: host
  port: port