Exploration of Lens using REST APIs¶
REST API enables Lens to deliver data over the HTTP protocol. It is is enabled by default and secured using API scopes. It consists of a base path and API scopes:
-
Base Path: All REST API endpoints are prefixed with
/lens2/api
. For example,/v2/meta
is available at/lens2/api/<lens_name>/v2/meta
. -
API Scopes: Endpoints are secured by API scopes, restricting access based on user permissions. Follows the principle of least privilege to grant only necessary access.
You can use Postman to interact with Lens via REST APIs. Start by importing the following Postman collection, testing each endpoint.
You can manage API access using the user_groups. The default user group ensures that API endpoints in all scopes are accessible to everyone. You can create custom user groups by defining roles and associating specific users with these roles in the user_group.yml file. To know more about user groups click here.
The following api_scopes
are currently supported:
API scope | REST API endpoints |
---|---|
meta | /v2/meta |
data | - /v2/load - /v2/sql |
graphql | /v2/graphql |
source | - /v2/default/schemas/ - /v2/default/schemas/sandbox/tables - /v2/default/schemas/retail/tables/city |
Prerequisites¶
Authentication¶
Lens uses API tokens to authorize requests and also for passing additional security context.
The API Token is passed via the Authorization Header. The token itself is a dataos-user-apikey
.
Ensure the following header is passed in Authorization when running the APIs-
Type: Bearer Token
Token: ${Your DataOS API Key} #Use the API key of the env defined in docker-compose.yml
meta scope¶
Provides access to metadata-related endpoints. This scope allows users to view metadata, which typically includes information about sources, authors, timezones, security context, user groups, etc.
\v2\meta
endpoint
Get meta-information for lens and views defined in the data model. Information about lens and lens with "public: false" will not be returned.
Example response:
{{
"name":
"description": [],
"authors": [],
"devMode": true,
"source": {
"type": "minerva",
}
},
"timeZones": [
"UTC"
],
"tables": [
{
"name": "product_analysis",
"type": "view",
"title": "Product Analysis",
]
}
data scope¶
/v2/load
endpoint
Executes queries to retrieve data based on the specified dimensions, measures, and filters. When you need to perform data analysis or retrieve specific data from the data model.
Use POST
request method along with /load
endpoint to add or update the data.
You can use either of the following methods:
In the POST request body, include the query parameters in the JSON Query Format:
{
"query": {
"dimensions": ["customer.customer_id", "customer.annual_income"],
"measures": ["customer.total_customers", "customer.average_age"],
"limit": 50
}
}
Example response:
{
"query": [
{
"name": "Customers",
"title": "Customers",
"meta": {
"someKey": "someValue",
"nested": {
"someKey": "someValue"
}
},
"connectedComponent": 1,
"measures": [
{
"name": "customers.count",
"title": "Customers Count",
"shortTitle": "Count",
"aliasName": "customers.count",
"type": "number",
"aggType": "count",
"drillMembers": ["customers.id", "customers.city", "customers.createdAt"]
}
],
"dimensions": [
{
"name": "customers.city",
"title": "Customers City",
"type": "string",
"aliasName": "customers.city",
"shortTitle": "City",
"suggestFilterValues": true
}
],
"segments": []
}
]
}
/v2/sql
endpoint
Alternatively, you can use /sql
endpoint.
configure the body with the JSON Query Format similar to /load
.
graphql¶
Grants access to GraphQL endpoints. GraphQL is a query language for APIs that allows clients to request only the data they need. This scope enables users to perform GraphQL queries and mutations.
source scope¶
Grants access to source-related endpoints.
`Parameter` | Description | Action |
---|---|---|
`schemas` | An array of schema names in the data source. | GET |
`tables` | An array of schemas and table names in the data source. | GET |
`describe-table` | Describes the given source table. | GET |
`load-source` | Loads all datasets of the source. | POST |
/v2/default/schemas
endpoint
This endpoint retrieves all the schemas available within the source depot. For example if source is icebase it will get the list of all the schemas present in icebase depot.
Example Request
Example Response
/v2/default/schemas/<collection_name>/tables
endpoint
This endpoint fetches all tables within a specified schema (collection).
Example Requests
Example Response
/v2/default/schemas/<collection_name>/tables/<table_name>
endpoint
This endpoint retrieves detailed information about a specific table within a schema.
Example Requests
Example Response
/v2/default/load-source?responseType=default
endpoint
This endpoint is used to load data from a specified source into Lens 2.0. The responseType parameter defines the type of response expected. In this case, it’s set to default.
Possible Responses¶
Continue wait
If the request takes too long to be processed, Lens Backend responds with { "error": "Continue wait" } and 200 status code. This is how the long polling mechanism in Lens is implemented. Clients should continuously retry the same query in a loop until they get a successful result. Subsequent calls to the Lens endpoints are idempotent and don't lead to scheduling new database queries if not required by the refresh_key. Also, receiving Continue wait doesn't mean the database query has been canceled, and it's actually still being processed by the Lens. Database queries that weren't started and are no longer waited by the client's long polling loop will be marked as orphaned and removed from the querying queue.
Possible reasons of Continue wait:
-
The query requested is heavy, and it takes some time for the database to process it. Clients should wait for its completion, continuously sending the same REST API request. continueWaitTimeout can be adjusted in order to change the time Lens waits before returning Continue wait message.
-
There are many queries requested and Lens backend queues them to save database from overloading.
Error Handling¶
Lens REST API has basic errors and HTTP Error codes for all requests.
Status | Error response | Description |
---|---|---|
400 | Error message | General error. It may be a database error, timeout, or other issue. Check error message for details. |
403 | Authorization header isn't set | You didn't provide an auth token. Provide a valid API Token or disable authorization. |
403 | Invalid token | The auth token provided is not valid. It may be expired or have an invalid signature. |
500 | Error message | Lens internal server error. Check error message for details. |