Elasticsearch¶
Pre-requisites¶
To create an Instance Secret for securing Elasticsearch credentials, you must have the following information:
Access Permissions in DataOS¶
To create an Instance Secret in DataOS, at least one of the following role tags must be assigned:
-
roles:id:data-dev -
roles:id:system-dev -
roles:id:user
Checking Assigned Roles
Use the following command to verify assigned roles:
If any required roles are missing, contact a DataOS Operator or submit a Grant Request for role assignment.
Alternatively, if access is managed through use cases, ensure the following use case is assigned:
-
Manage All Instance-level Resources of DataOS in User Layer
To validate assigned use cases, refer to the Bifrost Application Use Cases section.
Bifrost Governance
Source System Requirements¶
-
Username:- The Elasticsearch username used to authenticate access to the Elasticsearch instance. This is typically a user-specific or role-based identifier that grants the necessary permissions for accessing resources.
-
Password:- The Elasticsearch password corresponding to the username for authentication. This password ensures secure access to the Elasticsearch instance and can be obtained from your Elasticsearch configuration or admin settings, depending on how credentials are managed within your organization.
Ensure you have these credentials ready before proceeding with the Instance Secret creation process.
Create an instance secret for securing Elasticsearch credentials¶
Elasticsearch is a distributed search and analytics engine designed to handle large volumes of structured and unstructured data. It is widely used for log analytics, real-time search, and monitoring applications, enabling fast and scalable data indexing and querying.
To create an Elasticsearch Instance Secret in DataOS, ensure you have access to the DataOS Command Line Interface (CLI) and the necessary permissions. Follow the steps below to securely and efficiently complete the Instance Secret creation process.
Step 1: Create a manifest file¶
Begin by creating a manifest file to hold the configuration details for your Elasticsearch Instance Secret. Below are the templates for the read-only and read-write manifests:
name: ${elasticsearch-depot-name}-r # Name of the instance-secret, indicating it's for read-only access.
version: v1 # Manifest Version
type: instance-secret # Resource-type
description: ${description} # Optional: Brief description of the instance-secret's purpose.
layer: user # DataOS Layer
instance-secret:
type: key-value-properties # Type of Instance-secret
acl: r # Access control level, set to 'r' for read-only access.
data:
username: ${{elasticsearch username}}
password: ${{elasticsearch password}}
name: ${elasticsearch-depot-name}-rw # Name of the instance-secret, indicating it's for read-write access.
version: v1 # Manifest Version
type: instance-secret # Resource-type
description: ${description} # Optional: Brief description of the instance-secret's purpose.
layer: user # DataOS Layer
instance-secret:
type: key-value-properties # Type of Instance-secret
acl: rw # Access control level, set to 'r' for read-write access.
data:
username: ${{elasticsearch username}}
password: ${{elasticsearch password}}
Resource meta section
The Elasticsearch manifest includes a Resource meta section with essential metadata attributes common to all resource types. Some attributes in this section are optional, while others are mandatory. For more details, refer to the configurations section.
Instance-secret specific section
This section focuses on attributes specific to Elasticsearch Instance Secrets. It includes details like:
-
type: Specifies the Instance Secret type (key-value-properties). -
acl: Access control level (read-only or read-write). -
data: Contains sensitive information such as Elasticsearch username and password.
For more information, refer to the configurations section.
Step 2: Apply the manifest¶
To create the Elasticsearch Instance Secret within DataOS, use the apply command. Since these Instance Secrets are Instance-level resources, do not specify a workspace while applying the manifest.
Step 3: Validate the Instance Secret¶
To validate the proper creation of the Instance Secret in DataOS, use the get command.
To get the list of all the Instance Secret within the Dataos environment execute the following command.
dataos-ctl resource get -t instance-secret -a
INFO[0000] 🔍 get...
INFO[0000] 🔍 get...complete
NAME | VERSION | TYPE | WORKSPACE | STATUS | RUNTIME | OWNER
-----------------------------|---------|-----------------|-----------|--------|---------|------------------------
abfssv2alpha-r | v1 | instance-secret | | active | | iamgroot
abfssv2alpha-rw | v1 | instance-secret | | active | | iamgroot
abfsswithoutmetastore-r | v1 | instance-secret | | active | | thisisthor
abfsswithoutmetastore-rw | v1 | instance-secret | | active | | thisisthor
Alternatively, you can also check on Metis UI by searching the Instance Secret by name.
Delete the Instance Secret¶
To delete an Instance Secret, use one of the following methods:
Method 1¶
Specify the Resource type and Instance Secret name in the delete command.
Method 2¶
Copy the Instance Secret name, version, and Resource-type from the output of the get command separated by '|' enclosed within quotes and use it as a string in the delete command.
Method 3¶
Specify the path of the manifest file and use the delete command.