Skip to content

Instance Secret

An Instance Secret is a DataOS Resource designed for securely storing sensitive information at the DataOS Instance-level. The primary purpose of an Instance Secret is to address the inherent exposure risk associated with directly embedding such confidential data within application code or manifest files (YAML configuration files). This includes usernames, passwords, certificates, tokens, and keys. Instance Secrets establish a critical segregation between sensitive data and Resource definitions. This division minimizes inadvertent exposure during various Resource management phases, including creation, viewing, or editing.

DataOS Operators are generally responsible for creating and managing Instance Secrets for various sources within an organization. They exercise precise control over who can retrieve credentials. If any Data Product developer in the organization needs access to Instance Secrets, they can assign them relevant permission using the Bifrost application.

Instance Secret in the Data Product Lifecycle

In the Data Product Lifecycle, Instance Secrets play a crucial role in securely managing credentials and sensitive information. They are particularly useful when your data product requires:

  • Secure Credential Management: Storing and managing sensitive information such as usernames, passwords, API keys, or certificates securely within an instance. For example, an instance secret can securely store the credentials needed to connect to a database, ensuring that these credentials are not exposed in the codebase or configuration files.

  • Access Control: Ensuring that only authorized components and services within the instance can access the credentials. For instance, an instance secret can be used to provide a web application with the credentials to access a third-party service without exposing those credentials to the broader environment.

  • Auditing and Compliance: Maintaining a secure and auditable method of handling sensitive data, and complying with security policies and regulatory requirements. Instance secrets ensure that credential usage is logged and can be monitored for compliance purposes.

By using Instance Secrets, you can manage sensitive information securely and efficiently, mitigating the risk of exposing credentials while enabling seamless access to your applications and services.

Structure of an Instance Secret manifest

The structure of the Instance Secret manifest file is outlined as follows:

Bifrost Governance
Structure of Instance Secret manifest

How to create an Instance Secret?

This section includes steps for creating the Instance Secret to secure credentials of commonly used data sources and hosted code repositories using DataOS CLI.

Data Sources

If you want to store credentials for various data sources in an Instance Secret follow the steps given in the links below.

Repositories

If you want to store credentials for hosted code repositories in an Instance Secret follow the steps given below.

Configurations

Instance Secret can be configured to secure the credentials information through key-value pairs. For a detailed breakdown of the configuration options and attributes, please refer to the documentation: Attributes of Instance Secret manifest.

How to use an Instance Secret?

An Instance Secret can be used to create a Depot for data source connections, Services, Workflows, or Workers. Refer to the link below for guidance on using an Instance Secret in various DataOS resources.

How to refer to an Instance Secret in other DataOS Resources?

Best Practices

This section involves do's and don'ts for managing an Instance Secret.

  • As an operator, it is recommended to assign a use case to a user rather than a tag, as use cases provide greater control over user permissions and enable more secure management of specific resources.

  • Before deleting an Instance Secret, ensure that it is not being used by any other resources.

FAQs

1. Can I update an existing Instance Secret?

Yes, you can update an Instance Secret by modifying its manifest file and re-applying it. However, ensure that dependent resources are updated accordingly. To update an Instance Secret after making changes in the manifest file, execute the following commands.

dataos-ctl resource apply -f {path-to-manifest-file} 

dataos-ctl resource update -f {path-to-manifest-file}

2. What happens if an Instance Secret is deleted?

If an Instance Secret is deleted, any resources relying on it may lose access or experience failures. It is recommended to verify dependencies before deletion.

3. How can I receive alerts if an Instance Secret is deleted? You can configure a Monitor Resource in DataOS along with a Pager Resource to receive notifications when an Instance Secret is deleted.

4. Is there a way to restore a deleted Instance Secret?

No, deleted secrets cannot be recovered. You must create a new Instance Secret and update all associated resources.

5. Can multiple users access the same Instance Secret?

Yes, anyone assigned a use case that grants permission to read an Instance Secret can access it.

6. Can we create multiple Instance Secrets with the same name?

No, for Instance Secrets, names must be unique. If you attempt to create a secret with a name that already exists, the system will prevent duplication. You may need to use a different name or update the existing secret if modifications are required.

7. Where does the secrets get stored within DataOS?

Within DataOS secrets get stored securely within the Heimdall (Heimdall001 DB). Heimdall is the governance engine for access control within DataOS, acting as the Policy Decision Point (PDP) for all authorization requests. It enforces security policies across datasets, API paths, and various applications and services, ensuring only authorized users and systems gain access. Heimdall supports both Role-Based Access Control (RBAC) and Attribute-Based Access Control (ABAC), enabling flexible and fine-grained permissions. It logs every access request and decision for auditing and compliance, providing a transparent and secure framework for managing data and system interactions within DataOS.