MongoDB

To execute Flare Jobs on top of MongoDB Depot, you first need to create a Depot. If you have already created a Depot, then continue reading else proceed to following link: MongoDB Depot

Read Config

mongodb_depot_read.yml

version: v1
name: mongodb-read-01
type: workflow
workflow:
  dag:
    - name: mongodb-read-01
      spec:
        stack: flare:6.0
        compute: runnable-default
        stackSpec:
          job:
            explain: true
            logLevel: INFO
            inputs:
              - name: input
                dataset: dataos://sanitymongodb:tmdc/write_mongodb_12
                format: mongodb
                connectionProps:
                  ssl: "true"
                  retryReads: "true"

            steps:
              - sequence:
                  - name: finalDf
                    sql: SELECT * FROM input limit 10

            outputs:
              - name: finalDf
                dataset: dataos://lakehouse:smoketest/read_mongodb_12?acl=rw
                format: Iceberg
                options:
                  saveMode: append
                  iceberg:
                    properties:
                      write.format.default: parquet
                      write.metadata.compression-codec: none

Write Config

mongodb_depot_write.yml

version: v1
name: mongodb-write-01
type: workflow
workflow:
  dag:
    - name: mongodb-write-01
      spec:
        stack: flare:6.0
        compute: runnable-default
        stackSpec:
          job:
            explain: true
            logLevel: INFO
            inputs:
              - name: input
                dataset: dataos://thirdparty01:none/city
                format: csv
                schemaPath: dataos://thirdparty01:none/schemas/avsc/city.avsc

            steps:
              - sequence:
                - name: finalDf
                  sql: SELECT *, date_format (now(), 'yyyyMMddHHmm') AS version, current_timestamp() AS ts_city FROM input limit 10

            outputs:
              - name: finalDf
                dataset: dataos://sanitymongodb:tmdc/write_mongodb_12?acl=rw
                format: mongodb
                options:
                  saveMode: append
                  connectionProps:
                    ssl: "true"
                    retryWrites: "true"
                    w: "majority"