Lens¶

Lens Resource in DataOS is a logical modelling layer designed for accessing tabular data in data warehouses or lakehouses. It operates on top of physical tables, allowing the extension of these tables into logical tables by adding logical columns (measures) and relationships. It empowers analytical engineers, the key architects of business intelligence, with a model-first approach. To understand about the different elements of Lens click here.

Lens within the Data Product Lifecycle

Lens operates within the consumption layer of the Data Product Life Cycle within DataOS, By leveraging Lens, Data Products can be created to inform decision-making, ensuring that data is logically organized and aligned with business objectives from the outset. To consume it, Lens exposes APIs such as JDBC, HTTP, and GraphQL.

Lens Ecosystem

Why Lens?

The data modeling layer serves as an interface that overlays the underlying data, consistently presenting business users with familiar and well-defined terms like "product," "customer," or "revenue." This abstraction enables users to access and consume data in a way that aligns with their understanding, facilitating self-service analytics and reducing dependence on data engineers for ad-hoc data requests.

As a resource within the DataOS ecosystem, Lens enhances Data Product consumption by delivering improvements in how Data Products are accessed and utilized. It streamlines the developer experience in consumption patterns, focusing specifically on refining the use and interaction with data products.

Key features of Lens¶

Lens is engineered to handle complex and large-scale data models with ease. Key features include:

Code modularity: Lens supports modular code structures, simplifying the maintenance of extensive models, particularly when dealing with entities, dimensions, and measures. This modularity enables efficient development and management, allowing teams to navigate large codebases with reduced complexity.
Structured YAML templates: Lens provides standardized, easy-to-customize YAML templates. These templates streamline the creation of Lens YAML files, ensuring consistency across models and reducing manual effort. They offer an efficient way to define tables, views, dimensions, measures, joins, and segments, allowing you to quickly set up and modify your data models with minimal hassle.
Local Development environment for model inspection and validation: Lens features a local development environment that supports real-time inspection and validation of models. This minimizes delays caused by SQL syntax errors, and the inclusion of a web app for model inspection allows issues to be identified and resolved before deployment.
Segments: Segments are predefined filters that enable the definition of complex filtering logic in SQL. They allow you to create specific subsets of data, such as users from a particular city, which can be reused across different queries and reports. This feature helps streamline the data exploration process by simplifying the creation of reusable filters.
API Support: Lens enhances interoperability by simplifying application development with support for Postgres API, REST API, and GraphQL. Additionally, learn how to work with payloads for querying and interacting with the system in the API Documentation.
Governance and Access Control: Lens ensures data governance through user group management and data policies, enabling precise control over who can access and interact with data models.
BI integration: Lens improves interoperability through robust integration with Superset, Tableau and PowerBI. This ensures that data models can be easily utilized across various BI platforms, enhancing the overall analytics experience. For more details on BI integration, visit the BI Integration Guide.
Performance optimization through Flash: Designed to work with DataOS Lakehouse and Iceberg-format depots, Flash improves query performance by leveraging in-memory execution. This optimization ensures that data teams can efficiently handle large-scale queries with enhanced speed and performance.

How to build Lens?¶

The process begins with creating a new Lens project and generating a data model. Once the model is prepared, it will be tested within the development environment to ensure it is error-free before deployment.

Lens model folder set-up¶

The initial step involves setting up the folder structure for your Lens project. This section will guide you through organizing your project files, including the model configuration files and necessary resources, in a structured and maintainable way.

Develop Lens locally¶

Before deploying your Lens models into the DataOS environment, you can build and test your models on your local system. This guide includes the steps to develop and test Lens models locally, ensuring they function as intended.

Develop Lens within DataOS¶

This section involves the step-by-step guide on building and deploying Lens models within the DataOS environment. You will learn how to use Lens to generate and deploy data models, making sure they integrate seamlessly with the broader DataOS ecosystem.

Configurations¶

Lens can be configured to connect to different sources using data source attributes and configurable attributes in the docker-compose.yml or lens.yml manifest files. Here is a comprehensive guide to APIs and configuring supported properties.

Configuration Fields of the Deployment Manifest File (YAML) for Lens Resource Understand the various configuration fields available in the deployment manifest file for Lens resources.
Configuration Fields of the Docker Compose File Review the configuration fields and settings in the Docker Compose file for orchestrating multi-container applications.

Exploration of deployed Lens¶

After creating a Lens data model, the next step is to explore it—this means interacting with the model by running queries. The following section explains the key concepts for querying Lens through various methods, though all queries follow the same general format. Multiple ways are available to explore or interact with the Lens model or its underlying data, allowing you to ask meaningful questions of the data and retrieve valuable insights. Exploration can be performed using the following methods:

Data modelling¶

Data modeling is the process of defining and structuring raw data into organized and meaningful business definitions. It involves creating logical schemas, relationships, and aggregations to represent how data is stored, processed, and accessed. Effective data modeling ensures optimal performance for queries and allows users to extract valuable insights without modifying the underlying data structure. Below are resources to guide you through essential aspects of data modeling to optimize performance and accuracy.

Data modelling concepts and Overview: Understand the core principles and methodologies essential for designing effective data models.
Best practices: Explore recommended guidelines and techniques to create efficient and scalable data models.
Do's and don'ts: A concise list of actions to follow and pitfalls to avoid when designing your data model.
Error reference: A quick reference for understanding and resolving common errors in data modeling.

Supported data sources¶

Lens integrates with a variety of data sources to streamline data analysis and management. Below is a list of supported data sources.

Optimizing Lens model¶

The Lens semantic layer provides several optimization techniques that can significantly enhance the performance of data queries. The following page explores best practices and strategies for fine-tuning your Lens model to maximize efficiency.

Optimizing Lens model: Best practices for the Semantic Layer