Introduction

A separate database plugin to support MongoDB-specific features and configurations.

Use-Case

...

Introduction

A separate database plugin to support MongoDB-specific features and configurations.

Use-Case

• Users can choose and install MongoDB source and sink plugins.
• Users should see MongoDB logo on plugin configuration page for better experience.
• Users should get relevant information from the tool tip:
  
  • The tool tip should describe accurately what each field is used for.
• Users should not have to specify any redundant configuration
• Users should get field level lineage for the source and sink that is being used.
• Reference documentation should be updated to account for the changes.
• The source code for MongoDB database plugin should be placed in repo under data-integrations org.
• The data pipeline using source and sink plugins should run on both mapreduce and spark engines.

...

A query filter document and query operators can be used to specify conditions.

The following example uses '{ status: { $in: [ following example uses '{ status: { $in: [ "A", "D" ] } }' query filter document to retrieve all documents from the 'inventory' collection where 'status' equals either "A" or "D":

db.inventory.find( { status: { $in: [ "A", "D" ] }

...

db.inventory.find( { status: { $in: [ "A", "D" ] } } )

The operation corresponds to the following SQL statement:

SELECT * FROM inventory WHERE status in ("A", "D")

Sink Properties

...

Required

(defaults to localhost on UI)

...

Optional

(default 27017)

...

A list of arbitrary string key/value pairs as connection arguments. See Connection String Options for a full description of these arguments.

Source Properties

...

Required

(defaults to localhost on UI)

...

Optional

(default 27017)

...

A list of arbitrary string key/value pairs as connection arguments. See Connection String Options for a full description of these arguments.

Data Types Mapping

...

Value can be mapped to Schema.Type.BYTES, but this can lead to subtype information loss.

Subtypes:

• generic: \x00 (0)
• function: \x01 (1)
• old: \x02 (2)
• uuid_old: \x03 (3)
• uuid: \x04 (4)
• md5: \x05 (5)
• user: \x80 (128)

There are several options:

1) Support only 'generic' subtype.

2) Map using MongoDB extended JSON format:

"binary": {"$binary": "YmluYXJ5IGRhdGE=", "$type": "00"}

...

Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

"undefined": {"$undefined": true}

...

Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

There are several options:

1) Do not support this data type for the Sink

2) Map using MongoDB extended JSON format: {"$oid": "5d3f1c2a2f547625b0bbb397"}

...

Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

There are several options:

1) Do not support this data type for the Sink

2) Map using MongoDB extended JSON format: "regex": {"$regex": ".", "$options": ""}

...

String in MongoDB extended JSON format:

"dbpointer": {"$ref": "source", "$id": {"$oid": "5d079ee6d078c94008e4bb3a"}}

...

Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

There are several options:

1) Do not support this data type for the Sink

2) Map using MongoDB extended JSON format: "javascript": {"$code": "var l = 1;"}

...

Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

There are several options:

1) Do not support this data type for the Sink

...

Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

"javascriptwithscope": {"$code": "var l = 1;", "$scope": {"scope": "scope_val"}

...

Special type for internal MongoDB use which is not associated with the regular Date type. Timestamp values are a 64 bit value where:

• the first 32 bits are a time_t value (seconds since the Unix epoch)
• the second 32 bits are an incrementing ordinal for operations within a given second.

Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

"timestamp": {"$timestamp": {"t": 1564410161, "i": 1}}

...

Is less than any other value of any type. This can be useful for always returning certain documents first (or last).

Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

"minkey": {"$minKey": 1}

...

Is greater than any other value of any type. This can be useful for always returning certain documents first (or last).

Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

...

 } )

The operation corresponds to the following SQL statement:

SELECT * FROM inventory WHERE status in ("A", "D")

Sink Properties

Sink Data Types Mapping

To support all data types in the Sink we can use MongoDB extended JSON format or/and infer a data type of record field based on its name. 

The table below does not honor non-standard MongoDB data types and lists how CDAP data types are stored.

The following MongoDB data types are missing: Undefined, Regular Expression, DBPointer, JavaScript, Symbol, JavaScript (with scope), Timestamp, Min key, Max key.

["string","int","long"]

Source Properties

Source Data Types Mapping

The source requires Output Schema to be set. Based on the schema source will expect a field in each document to be of a specific Mongo data type.

On Record Error error handling property allows the user to decide whether the pipeline should fail, the record should be skipped, or the record should be sent to the error dataset.

The following table shows what MongoDB data types can be read as CDAP types.

{"type":"record","name":"object","fields":[{"name":"inner_field","type":"string"}]}

{
 "object" : {
        "inner_field" : "val"
    }
}

{
  "type":"record",
  "name":"javascriptwithscope",
  "fields":[
    {"name":"$code","type":"string"},
    {"name":"$scope","type":{"type":"record","name":"scope-object-record","fields"[{"name":"scope","type":"string"}]}}
  ]
}

{
  "javascriptwithscope" : { "$code" : var l = 1; ,  "$scope" : { "scope" : "scope_val" } }
}

{"type":"map","keys":"string","values":"string"}

{
 "object" : {
        "inner_field" : "val"
    }
}

Approach

Move existing mongodb-plugins module to the mongodb-plugins project. Add MongoDB-specific properties to configuration, add support for MongoDB-specific datatypes. Update UI widgets JSON definitions.

...