Simple Storage Service (Amazon S3)¶
Pre-requisites¶
To create an Instance Secret for securing S3 credentials, you must have the following information:
Access Permissions in DataOS¶
To create an Instance Secret in DataOS, at least one of the following role tags must be assigned:
-
roles:id:data-dev -
roles:id:system-dev -
roles:id:user
Checking Assigned Roles
Use the following command to verify assigned roles:
If any required roles are missing, contact a DataOS Operator or submit a Grant Request for role assignment.
Alternatively, if access is managed through use cases, ensure the following use case is assigned:
-
Manage All Instance-level Resources of DataOS in User Layer
To validate assigned use cases, refer to the Bifrost Application Use Cases section.
Bifrost Governance
Source System Requirements¶
-
Access Key ID: The access key ID used to authenticate AWS requests. You can retrieve this from the AWS Management Console by navigating to IAM > Users, selecting the relevant user, and viewing their Security Credentials.
-
AWS Access Key ID: The AWS-specific access key ID, which can also be retrieved from the IAM > Users- section in the AWS Management Console under Security Credentials.
-
AWS Secret Access Key: The secret access key associated with the AWS access key ID. This is displayed only once when the key is generated. If lost, you will need to create a new access key in the IAM > Users- section under Security Credentials.
-
Secret Key: Another key used for authentication.
Ensure you have these credentials ready before proceeding with the Instance Secret creation process.
Create an Instance Secret for securing Amazon S3 credentials¶
Amazon Simple Storage Service (S3) is an object storage system. Object stores are distributed storage systems designed to store and manage large amounts of unstructured data.
To create an S3 Instance Secret in DataOS, ensure you have access to the DataOS Command Line Interface (CLI) and the required permissions. Follow the steps below to complete the creation process efficiently and securely.
Step 1: Create a manifest file¶
Begin by creating a manifest file to hold the configuration details for your S3 Instance Secret. Depending on your access needs (read-only or read-write), start with the corresponding YAML template provided below.
# Amazon S3 Read Instance-secret Manifest
name: ${s3-depot-name}-r # Unique identifier for Resource, replace ${s3-depot-name} with depot name
version: v1 # Manifest version
type: instance-secret # Type of the Resource
description: ${description} # Purpose of the Instance-secret
layer: user # DataOS layer
instance-secret:
type: key-value-properties # Secret type
acl: r # Access control: 'r' for read-only
data:
accesskeyid: ${access-key-id} # Replace with access key ID
awsaccesskeyid: ${aws-access-key-id} # Replace with AWS access key ID
awssecretaccesskey: ${aws-secret-access-key} # Replace with AWS secret access key
secretkey: ${secret-key} # Replace with secret key
# Amazon S3 read-write Instance-secret Manifest
name: ${s3-depot-name}-rw # Unique identifier for Resource, replace ${s3-depot-name} with depot name
version: v1 # Manifest version
type: instance-secret # Type of the Resource
description: ${description} # Purpose of the Instance-secret
layer: user # DataOS layer
instance-secret:
type: key-value-properties # Secret type
acl: rw # Access control: 'rw' for read-write
data:
accesskeyid: ${access-key-id} # Replace with access key ID
awsaccesskeyid: ${aws-access-key-id} # Replace with AWS access key ID
awssecretaccesskey: ${aws-secret-access-key} # Replace with AWS secret access key
secretkey: ${secret-key} # Replace with secret key
Resource meta section
The S3 manifest includes a Resource meta section with essential metadata attributes common to all resource types. Some attributes in this section are optional, while others are mandatory. For more details, refer to the configurations section.
Instance-secret specific section
This section focuses on attributes specific to S3 Instance Secrets. It includes details like:
-
type: Specifies the Instance Secret type (key-value-properties). -
acl: Access control level (read-only or read-write). -
data: Contains sensitive information such as Azure endpoint suffix, storage account key, and storage account name.
For more information, refer to the configurations section.
Step 2: Apply the manifest¶
To create the S3 Instance Secret within DataOS, use the apply command. Since S3 Instance Secrets are Instance-level resources, do not specify a workspace while applying the manifest.
Step 3: Validate the Instance Secret¶
To validate the proper creation of the Instance Secret in DataOS, use the get command.
To get the list of all the Instance Secrets within the Dataos environment execute the following command.
dataos-ctl resource get -t instance-secret -a
INFO[0000] 🔍 get...
INFO[0000] 🔍 get...complete
NAME | VERSION | TYPE | WORKSPACE | STATUS | RUNTIME | OWNER
-----------------------------|---------|-----------------|-----------|--------|---------|------------------------
abfssv2alpha-r | v1 | instance-secret | | active | | iamgroot
abfssv2alpha-rw | v1 | instance-secret | | active | | iamgroot
abfsswithoutmetastore-r | v1 | instance-secret | | active | | thisisthor
abfsswithoutmetastore-rw | v1 | instance-secret | | active | | thisisthor
Alternatively, you can also check on Metis UI by searching the Instance Secret by name.
Delete the Instance Secret¶
To delete an Instance Secret, use one of the following methods:
Method 1¶
Specify the Resource type and Instance Secret name in the delete command.
Method 2¶
Copy the Instance Secret name, version, and Resource-type from the output of the get command separated by '|' enclosed within quotes and use it as a string in the delete command.
Method 3¶
Specify the path of the manifest file and use the delete command.