Kafka¶
An Instance Secret for Kafka in DataOS is used to create a Depot Resource that requires secure access to Kafka. The following sections outline the necessary permissions, configurations, and steps required to set up an Instance Secret efficiently.
Pre-requisites¶
To create an Instance Secret for securing Kafka credentials, you must have the following information:
To create an Instance Secret in DataOS, at least one of the following role tags must be assigned:
-
roles:id:data-dev -
roles:id:system-dev -
roles:id:user
Checking Assigned Roles
Use the following command to verify assigned roles:
If any required roles are missing, contact a DataOS Operator or submit a Grant Request for role assignment.
Alternatively, if access is managed through use cases, ensure the following use case is assigned:
-
Manage All Instance-level Resources of DataOS in User Layer
To validate assigned use cases, refer to the Bifrost Application Use Cases section.
Bifrost Governance
Create an Instance Secret for securing Kafka credentials¶
Kafka is a distributed event streaming platform capable of handling trillions of events a day. To create a Kafka Instance Secret in DataOS, ensure you have access to the DataOS Command Line Interface (CLI) and the required permissions. Follow the steps below to complete the creation process efficiently and securely.
Step 1: Create a manifest file¶
Begin by creating a manifest file to hold the configuration details for your Kafka Instance Secret. Depending on your access needs (read-only or read-write), start with the corresponding YAML template provided below.
# Kafka Read Instance Secret Manifest
name: ${kafka-depot-name}-r # Name of the instance-secret, indicating it's for read-only access.
version: v1 # Manifest Version
type: instance-secret # Resource-type
description: ${description} # Optional: Brief description of the instance-secret's purpose.
layer: user # DataOS Layer
instance-secret:
- acl: r
type: key-value-properties
data:
security_protocol: ${{SASL_SSL}} #optional
sasl_mechanism: ${{sasl_mechanism}} #optional
trust_store_type: ${{trust_store_type}} #optional
trust_store_password: ${{trust_store_password}} #optional
username: ${{username}}
password: ${{password}}
files: #optional
ca_file: "{{Local File path where .pem file is located}}"
trust_store_file: "{{Local File path where cacerts file is located}}"
# Kafka Read-write Instance Secret Manifest
name: ${kafka-depot-name}-rw # Name of the instance-secret, indicating it's for read-only access.
version: v1 # Manifest Version
type: instance-secret # Resource-type
description: ${description} # Optional: Brief description of the instance-secret's purpose.
layer: user # DataOS Layer
instance-secret:
- acl: rw
type: key-value-properties
data:
security_protocol: ${{SASL_SSL}} #optional
sasl_mechanism: ${{sasl_mechanism}} #optional
trust_store_type: ${{trust_store_type}} #optional
trust_store_password: ${{trust_store_password}} #optional
username: ${{username}}
password: ${{password}}
files: #optional
ca_file: "{{Local File path where .pem file is located}}"
trust_store_file: "{{Local File path where cacerts file is located}}"
Resource meta section
The Kafka manifest includes a Resource meta section with essential metadata attributes common to all Resource types. Some attributes in this section are optional, while others are mandatory. For more details, refer to the configurations section.
Instance-secret specific section
This section focuses on attributes specific to Kafka Instance Secrets. It includes details like:
-
acl: rw: Specifies the access control level for the Instance Secret.rwmeans the secret is read-write, allowing both retrieval and updates. -
type: key-value-properties: Declares the format of the secret, indicating it is composed of key-value pairs. -
data: A container that holds sensitive configuration parameters needed for secure authentication. -
security_protocol: ${{SASL_SSL}}: Defines the communication protocol to use. -
SASL_SSL: Enables SASL authentication over SSL encryption. (Optional) -
sasl_mechanism: ${{sasl_mechanism}}: Specifies the SASL mechanism for authentication (e.g., PLAIN, SCRAM-SHA-256). (Optional) -
trust_store_type: ${{trust_store_type}}: Indicates the format of the trust store used to validate SSL certificates (e.g., JKS, PKCS12). (Optional) -
trust_store_password: ${{trust_store_password}}: Password to unlock the trust store file, allowing access to trusted certificates. (Optional) -
username: ${{username}}: The authentication username required to connect to the target system. -
password: ${{password}}: The password associated with the Kafka username for authentication. -
files: A container specifying paths to local files required for secure connections (e.g., certificates or trust stores). -
ca_file: "{{Local File path where .pem file is located}}" Path to the Certificate Authority (.pem) file used to verify the server’s SSL certificate. (Optional) -
trust_store_file: "{{Local File path where cacerts file is located}}" Path to the trust store file (e.g., .jks, cacerts) that holds trusted certificates for SSL validation. (Optional)
For more information, refer to the configurations section.
Step 2: Apply the manifest¶
To create the ABFSS Instance Secret within DataOS, use the apply command. Since ABFSS Instance Secrets are Instance-level resources, do not specify a workspace while applying the manifest.
Step 3: Validate the Instance Secret¶
To validate the proper creation of the ABFSS Instance Secret in DataOS, use the get command.
To get the list of all the Instance Secret within the Dataos environment execute the following command.
dataos-ctl resource get -t instance-secret -a
INFO[0000] 🔍 get...
INFO[0000] 🔍 get...complete
NAME | VERSION | TYPE | WORKSPACE | STATUS | RUNTIME | OWNER
----------------------------|---------|-----------------|-----------|--------|---------|------------------------
abfssv2alpha-r | v1 | instance-secret | | active | | iamgroot
abfssv2alpha-rw | v1 | instance-secret | | active | | iamgroot
abfsswithoutmetastore-r | v1 | instance-secret | | active | | thisisthor
abfsswithoutmetastore-rw | v1 | instance-secret | | active | | thisisthor
Alternatively, you can also check on Metis UI by searching the Instance Secret by name.
Delete the Instance Secret¶
To delete the ABFSS Instance Secret, use one of the following methods:
Method 1¶
Specify the Resource type and Instance Secret name in the delete command.
Method 2¶
Copy the Instance Secret name, version, and Resource-type from the output of the get command separated by '|' enclosed within quotes and use it as a string in the delete command.
Method 3¶
Specify the path of the manifest file and use the delete command.