Skip to content

Talos

Talos is a Stack within DataOS that empowers you to build robust and scalable data APIs (RESTful APIs) using SQL templates. In Talos, you can combine logic and transformation within SQL templates, which helps you create APIs more efficiently. By simplifying the process of writing and managing database queries, Talos enables rapid development and integration of APIs into various applications and AI agents.

Talos allows you to filter data from the data warehouse, perform data validation on API parameters, and handle errors. It makes the process flexible and straightforward, ensuring the accuracy and integrity of your data while delivering powerful analytics capabilities to users.

It turns your SQL into APIs in no time!

Talos within the Data Product Life Cycle

Talos functions within the consumption layer of the Data Product Life Cycle in DataOS, specifically designed to handle tabular data stored in data lakehouses. It enhances access and usability for consumers, including external systems, applications, and end-users. The Talos API endpoints are vital in this framework, providing a standardized and flexible interface for querying and consuming Data Products via APIs. Talos is especially valuable when working with tabular data, as it allows users to write SQL queries across one or multiple Data Products and expose the results as REST APIs, facilitating seamless integration and data retrieval outside the DataOS environment.

Why Talos?

Nowadays data is growing at the speed of light, currently, data professionals do not have any better ways to share data with respective stakeholders for operational business use cases.

Developers today must undertake a more manual and more complex process to streamline the creation of APIs for AI agents and apps to interact with databases and data warehouses.

Conventional Methods and Challenges

Custom API development

  • Time-consuming: Developing APIs requires extensive manual coding, which can be particularly labor-intensive for complex systems or when integrating various data sources.
  • Error-Prone: Manual coding introduces a higher likelihood of bugs and errors, potentially impacting the API’s reliability and efficiency.

Integration Complexity

  • Diverse Data Sources: Combining data from various sources with different formats and protocols demands significant expertise and effort.
  • Lack of Standardization: The absence of a unified approach means that APIs might use different conventions, complicating consistent interaction for AI systems.

Security and Compliance

  • Security Risks: Protecting APIs from security threats and ensuring compliance with regulations such as GDPR or HIPAA involves additional measures, including robust authentication, authorization, and encryption.
  • Maintenance Overhead: Security and compliance standards can change, requiring ongoing updates and maintenance of APIs to stay compliant.

Scalability and Performance

  • Scalability Concerns: Custom APIs may not always be designed with scalability in mind, which can lead to performance issues as demand or data volume increases.
  • Resource Intensive: Achieving optimal performance and scalability often demands substantial development resources and infrastructure.

Documentation and Usability

  • Lack of Documentation: Effective documentation is crucial for making APIs understandable and usable by other developers or AI systems, but it can sometimes be inadequately addressed.
  • Usability Issues: Without thorough and clear documentation, integrating and using APIs becomes more challenging for others.

Discover what Talos can do for you

By leveraging Heimdall for authentication and SQL templates for simple query execution, Talos ensures secure, efficient, and reliable API services. Its RESTful architecture, automated documentation generation, and observability features simplify development and integration.

Talos
Talos Stack

Rapid development and integration

By abstracting the complexities of directly interacting with databases and data warehouses, developers can focus on the higher-level logic of their applications. This reduces the development time and simplifies the process of integrating AI capabilities into applications.

API documentation and standardization

Utilizing OpenAPI documents for interaction provides a standardized way for AI agents to understand and interact with different APIs. This promotes interoperability among various systems and tools, making it easier to integrate with a wide array of services and data sources.

Scalability and maintenance

A template-driven approach to API creation can make it easier to scale and maintain APIs over time. Changes in the underlying data schema or business logic can be propagated to the APIs more efficiently, without the need for extensive manual adjustments.

Accessibility

Making data more accessible to AI agents through well-defined APIs can unlock new insights and capabilities by leveraging machine learning and analytics. This can enhance decision-making processes and automate routine tasks, among other benefits.

Validating API parameters

Talos allows you to validate and sanitize API parameters in Talos. This feature ensures that the input data is validated and meets the required criteria, reducing the risk of incorrect or malicious requests.

Data security

Talos utilizes data masking within SQL templates to ensure sensitive information is protected. By obscuring or anonymizing confidential data, Talos allows for the secure sharing and use of datasets while maintaining data privacy. This approach helps prevent unauthorized exposure of sensitive information, ensuring that security and compliance standards are upheld.

Data governance

Talos provides comprehensive data governance feature by enabling precise control over data access through Heimdall authentication and user groups. User groups allow you to organize individuals based on their roles, facilitating the management of access to various API scopes and the application of data policies. By assigning specific permissions and data access rights to different user groups, Talos ensures that only authorized users can access the datasets. This approach streamlines the enforcement of security policies and compliance requirements, maintaining data integrity and confidentiality across different levels of access.

Error handling

Talos provides effective error handling by stopping further execution and generating an error code when a problem is encountered during SQL template processing. This method prevents the return of potentially inaccurate or incomplete results and ensures that users receive clear feedback for troubleshooting. By halting execution and reporting specific error codes, Talos helps maintain the reliability and accuracy of API responses.

Caching datasets

To enhance query performance and reduce the strain on data sources, Talos includes a dataset caching feature. This functionality stores and reuses intermediate query results, leading to faster API response times. By minimizing the need for repetitive queries, caching improves overall efficiency and ensures more responsive and effective data retrieval.

Observability and monitoring

observability involves analyzing detailed metrics to understand your APIs' internal state, while monitoring ensures optimal performance by providing real-time updates on API metrics through the /metrics endpoint. This real-time data helps maintain accuracy, validate data, and handle errors, facilitating rapid development and integration into applications and AI agents.

Data sources supported by Talos

Talos offers extensive compatibility with a range of popular data sources, ensuring flexibility and ease of integration for your data needs such as BigQuery, Snowflake, Postgres, Redshift, and Depot of type BigQuery, Snowflake, Postgres, and Redshift.

How to build APIs?

Talos provides the necessary tools and capabilities to streamline API development and deployment within DataOS, ensuring that your processes are efficient and scalable. In the following sections, you'll find detailed instructions on how to set up Talos within DataOS and locally, tailored to fit your specific needs.

Develop APIs locally

Before deploying your APIs into the DataOS environment you can build and test your APIs on your local system. This guide includes the steps to build APIs locally using Talos.

Develop APIs within DataOS

This section involves the step-by-step guide on building and deploying APIs within DataOS using Talos. Talos is orchestrated by the Service Resource to streamline the API creation.

Configurations

To configure Talos effectively, it's essential to understand each attribute of the manifest files. This section provides detailed descriptions of the key configuration files used in Talos, helping you set up and customize Talos efficiently. For detailed information on each file, please refer to this section.

Recipes

This section provides step-by-step guides to assist you in effectively configuring Talos to solve common challenges. Below are some recipes to help you configure Talos effectively:

APIs in action: Exploring Talos use cases

This section involves the practical, real-world scenarios demonstrating how to consume the data with Talos effectively. Below are some examples to help you to understand Talos effectively:

You can refer to the recipes section for more examples.

Best practices

When developing data APIs with Talos, following best practices is crucial for securing sensitive information and ensuring efficient operation. These practices encompass both API design and SQL usage.