Talos Stack¶
Talos is a Stack within DataOS that empowers you to build robust and scalable data APIs (RESTful APIs) using SQL templates. In Talos, you can combine logic and transformation within SQL templates, which helps you create APIs more efficiently. By simplifying the process of writing and managing database queries, Talos enables rapid development and integration of APIs into various applications and AI agents.
Talos allows you to filter data from the data warehouse, perform data validation on API parameters, and handle errors. It makes the process flexible and straightforward, ensuring the accuracy and integrity of your data while delivering powerful analytics capabilities to users.
It turns your SQL into APIs in no time!
Talos within the Data Product Life Cycle
Talos functions within the consumption layer of the Data Product Life Cycle in DataOS, specifically designed to handle tabular data stored in data lakehouses. It enhances access and usability for consumers, including external systems, applications, and end-users. The Talos API endpoints are vital in this framework, providing a standardized and flexible interface for querying and consuming Data Products via APIs. Talos is especially valuable when working with tabular data, as it allows users to write SQL queries across one or multiple Data Products and expose the results as REST APIs, facilitating seamless integration and data retrieval outside the DataOS environment.
Why Talos?¶
Nowadays data is growing at the speed of light, currently, data professionals do not have any better ways to share data with respective stakeholders for operational business use cases.
Developers today must undertake a more manual and more complex process to streamline the creation of APIs for AI agents and apps to interact with databases and data warehouses.
Conventional Methods and Challenges¶
Here are some common challenges encountered when developing APIs using conventional methods.
Custom API development
Developing APIs requires extensive manual coding, which can be particularly labor-intensive for complex systems or when integrating various data sources. Manual coding introduces a higher likelihood of bugs and errors, potentially impacting the API’s reliability and efficiency.
Integration Complexity
Combining data from various sources with different formats and protocols demands significant expertise and effort. The absence of a unified approach means that APIs might use different conventions, complicating consistent interaction for AI systems.
Security and Compliance
Protecting APIs from security threats and ensuring compliance with regulations such as GDPR or HIPAA involves additional measures, including robust authentication, authorization, and encryption. Security and compliance standards can change, requiring ongoing updates and maintenance of APIs to stay compliant.
Scalability and Performance
Custom APIs may not always be designed with scalability in mind, which can lead to performance issues as demand or data volume increases. Achieving optimal performance and scalability often demands substantial development resources and infrastructure.
Documentation and Usability
Effective documentation is crucial for making APIs understandable and usable by other developers or AI systems, but it can sometimes be inadequately addressed. Without thorough and clear documentation, integrating and using APIs becomes more challenging for others.
Features of Talos¶
By leveraging Heimdall for authentication and SQL templates for simple query execution, Talos ensures secure, efficient, and reliable API services. Its RESTful architecture, automated documentation generation, and observability features simplify development and integration.
Rapid development and integration
By abstracting the complexities of directly interacting with databases and data warehouses, developers can focus on the higher-level logic of their applications. This reduces the development time and simplifies the process of integrating AI capabilities into applications.
API documentation and standardization
Utilizing OpenAPI documents for interaction provides a standardized way for AI agents to understand and interact with different APIs. This promotes interoperability among various systems and tools, making it easier to integrate with a wide array of services and data sources.
Scalability and maintenance
A template-driven approach to API creation can make it easier to scale and maintain APIs over time. Changes in the underlying data schema or business logic can be propagated to the APIs more efficiently, without the need for extensive manual adjustments.
Accessibility
Making data more accessible to AI agents through well-defined APIs can unlock new insights and capabilities by leveraging machine learning and analytics. This can enhance decision-making processes and automate routine tasks, among other benefits.
Validating API parameters
Talos allows you to validate and sanitize API parameters in Talos. This feature ensures that the input data is validated and meets the required criteria, reducing the risk of incorrect or malicious requests.
Data security
Talos utilizes data masking within SQL templates to ensure sensitive information is protected. By obscuring or anonymizing confidential data, Talos allows for the secure sharing and use of datasets while maintaining data privacy. This approach helps prevent unauthorized exposure of sensitive information, ensuring that security and compliance standards are upheld.
Data governance
Talos provides comprehensive data governance feature by enabling precise control over data access through Heimdall authentication and user groups. User groups allow you to organize individuals based on their roles, facilitating the management of access to various API scopes and the application of data policies. By assigning specific permissions and data access rights to different user groups, Talos ensures that only authorized users can access the datasets. This approach streamlines the enforcement of security policies and compliance requirements, maintaining data integrity and confidentiality across different levels of access.
Error handling
Talos provides effective error handling by stopping further execution and generating an error code when a problem is encountered during SQL template processing. This method prevents the return of potentially inaccurate or incomplete results and ensures that users receive clear feedback for troubleshooting. By halting execution and reporting specific error codes, Talos helps maintain the reliability and accuracy of API responses.
Caching datasets
To enhance query performance and reduce the strain on data sources, Talos includes a dataset caching feature. This functionality stores and reuses intermediate query results, leading to faster API response times. By minimizing the need for repetitive queries, caching improves overall efficiency and ensures more responsive and effective data retrieval.
Observability and monitoring
observability involves analyzing detailed metrics to understand your APIs' internal state, while monitoring ensures optimal performance by providing real-time updates on API metrics through the /metrics endpoint. This real-time data helps maintain accuracy, validate data, and handle errors, facilitating rapid development and integration into applications and AI agents.
Data sources supported by Talos¶
Talos offers extensive compatibility with a range of popular data sources, ensuring flexibility and ease of integration for your data needs such as BigQuery, Snowflake, Postgres, Redshift, and Depot of type BigQuery, Snowflake, Postgres, and Redshift.
How to build APIs?¶
Talos provides the necessary tools and capabilities to streamline API development and deployment within DataOS, ensuring that your processes are efficient and scalable. In the following sections, you'll find detailed instructions on how to set up Talos within DataOS and locally, tailored to fit your specific needs.
Develop APIs locally¶
Before deploying your APIs into the DataOS environment you can build and test your APIs on your local system. This guide includes the steps to build APIs locally using Talos.
Develop APIs within DataOS¶
This section involves the step-by-step guide on building and deploying APIs within DataOS using Talos. Talos is orchestrated by the Service Resource to streamline the API creation.
Configurations¶
To configure Talos effectively, it's essential to understand each attribute of the manifest files. This section provides detailed descriptions of the key configuration files used in Talos, helping you set up and customize Talos efficiently. For detailed information on each file, please refer to this section.
Recipes¶
This section provides step-by-step guides to assist you in effectively configuring Talos to solve common challenges. Below are some recipes to help you configure Talos effectively:
- How to set up Talos for Lens?
- How to set up Talos for Flash?
- How to set up Talos for Redshift?
- How to set up Talos for Snowflake?
- How to set up Talos for Postgres?
- How to apply data masking while exposing data through an API?
- How to use external API as a data source?
- How to fetch data exposed by Talos from the third-party tools?
- Caching Datasets
- How to generate the comprehensive API documentation?
APIs in action: Exploring Talos use cases¶
This section involves the practical, real-world scenarios demonstrating how to consume the data with Talos effectively.
You can refer to the recipes section for more examples.
Best practices¶
When developing data APIs with Talos, following best practices is crucial for securing sensitive information and ensuring efficient operation. These practices encompass both API design and SQL usage.