Skip to content

Lens

Lens Resource in DataOS is a logical modelling layer designed for accessing tabular data in data warehouses or lakehouses. It operates on top of physical tables, allowing the extension of these tables into Logical Tables by adding logical columns (measures) and relationships. It empowers analytical engineers, the key architects of business intelligence, with a model-first approach. To understand about the different elements of lens click here.

Lens within the Data Product Lifecycle

Lens operates within the consumption layer of the Data Product Life Cycle within DataOS, By leveraging Lens, Data Products can be created to inform decision-making, ensuring that data is logically organized and aligned with business objectives from the outset. To consume it, Lens exposes APIs such as JDBC, HTTP, and GraphQL.

Untitled(5)
Lens Ecosystem

Why Lens?

The data modeling layer serves as an interface that overlays the underlying data, consistently presenting business users with familiar and well-defined terms like "product," "customer," or "revenue." This abstraction enables users to access and consume data in a way that aligns with their understanding, facilitating self-service analytics and reducing dependence on data engineers for ad-hoc data requests.

As a resource within the DataOS ecosystem, Lens enhances Data Product consumption by delivering improvements in how Data Products are accessed and utilized. It streamlines the developer experience in developer experience, consumption patterns, focusing specifically on refining the use and interaction with data products.

Key features of Lens

Lens is engineered to handle complex and large-scale data models with ease. Key features include:

  • Code Modularity: Lens supports modular code structures, simplifying the maintenance of extensive models, particularly when dealing with entities, dimensions, and measures. This modularity enables efficient development and management, allowing teams to navigate large codebases with reduced complexity.

  • YAML Template Generation: Lens offers a tailored YAML template generator through its VS Code plugin, streamlining the creation of Lens YAML files. This feature reduces manual effort, ensuring consistency across models and speeding up the setup process, offering an efficient way to generate tables, views, dimensions, measures, joins, and segments. For detailed steps on installing and using the Lens VS Code extension, visit the Lens VS Code Extension Guide.

  • Real-time Verification and Validation: Lens features a local development environment that supports real-time inspection and validation of models. This minimizes delays caused by SQL syntax errors, and the inclusion of a web app for model inspection allows issues to be identified and resolved before deployment.

  • API Support

Lens enhances interoperability by simplifying application development with support for Postgres API, REST API, and GraphQL. These integrations enable smoother interactions with external systems and applications. For detailed information on using these APIs, see the API Documentation.

  • First-Class BI Integration

Lens improves interoperability through robust integration with Superset, with ongoing efforts to extend connectivity to Tableau and PowerBI. This ensures that data models can be easily utilized across various BI platforms, enhancing the overall analytics experience. For more details on BI integration, visit the BI Integration Guide. - Performance Optimization through Flash: Designed to work with DataOS Lakehouse and Iceberg-format depots, Flash improves query performance by leveraging in-memory execution. This optimization ensures that data teams can efficiently handle large-scale queries with enhanced speed and performance.

Lens Set-up

The process begins with creating a new Lens project and generating a data model. Once the model is prepared, it will be tested within the development environment to ensure it is error-free before deployment.

This guide will cover the following tasks:

Lens set-up

Configurations

Lens can be configured to connect to different sources using data source attributes and configurable attributes in the docker-compose.yml or lens.yml manifest files. Here is a comprehensive guide to APIs and configuring supported properties.

Exploration of Deployed Lens

After creating a Lens data model, the next step is to explore it—this means interacting with the model by running queries. The following section explains the key concepts for querying Lens through various methods, though all queries follow the same general format. Multiple ways are available to explore or interact with the Lens model or its underlying data, allowing you to ask meaningful questions of the data and retrieve valuable insights.

Exploration can be performed using the following methods:

Data Modelling

Data modeling is the process of defining and structuring raw data into organized and meaningful business definitions. It involves creating logical schemas, relationships, and aggregations to represent how data is stored, processed, and accessed. Effective data modeling ensures optimal performance for queries and allows users to extract valuable insights without modifying the underlying data structure.

Below are resources to guide you through essential aspects of data modeling to optimize performance and accuracy.

  • Best Practices Explore recommended guidelines and techniques to create efficient and scalable data models.

  • Do's and Don'ts A concise list of actions to follow and pitfalls to avoid when designing your data model.

  • Working with Segments Learn how to create and manage data segments to improve query performance and user experience.

Optimizing Lens Model

The Lens semantic layer provides several optimization techniques that can significantly enhance the performance of data queries. The following page explores best practices and strategies for fine-tuning your Lens model to maximize efficiency.

Optimizing Lens Model: Best Practices for the Semantic Layer