Introduction

A separate database plugin to support MongoDB-specific features and configurations.

Use-Case

• Users can choose and install MongoDB source and sink plugins.
• Users should see MongoDB logo on plugin configuration page for better experience.
• Users should get relevant information from the tool tip:
  
  • The tool tip should describe accurately what each field is used for.
• Users should not have to specify any redundant configuration
• Users should get field level lineage for the source and sink that is being used.
• Reference documentation should be updated to account for the changes.
• The source code for MongoDB database plugin should be placed in repo under data-integrations org.
• The data pipeline using source and sink plugins should run on both mapreduce and spark engines.

User Stories

• User should be able to install MongoDB specific database source and sink plugins from the Hub
• Users should have each tool tip accurately describe what each field does
• Users should get field level lineage information for the MongoDB source and sink 
• Users should be able to setup a pipeline avoiding specifying redundant information
• Users should get updated reference document for MongoDB source and sink
• Users should be able to read all the DB types

Plugin Type

• Batch Source
• Batch Sink 
• Real-time Source
• Real-time Sink
• Action
• Post-Run Action
• Aggregate
• Join
• Spark Model
• Spark Compute

Design Tips

MongoDB driver reference: http://mongodb.github.io/mongo-java-driver/3.10/driver/

Design

The suggestion is to move existing mongodb-plugins module to the mongodb-plugins repository.

MongoDB Overview

Document database

A record in MongoDB is a document, which is a data structure composed of field and value pairs. MongoDB documents are similar to JSON objects. The values of fields may include other documents, arrays, and arrays of documents.

{
    "_id" : ObjectId("5d3f1c2a2f547625b0bbb397"),
    "string" : "AAPL",
    "int32" : 10,
    "double" : 23.23,
    "array" : [ 
        "a1", 
        "a2"
    ],
    "object" : {
        "inner_field" : "val"
    },
    "binary" : { "$binary" : "YmluYXJ5IGRhdGE=", "$type" : "00" },
    "undefined" : undefined,
    "boolean" : false,
    "date" : ISODate("2019-07-29T16:17:46.109Z"),
    "null" : null,
    "regex" : /./,
    "dbpointer" : DBRef("source", "5d079ee6d078c94008e4bb3a"),
    "javascript" : var l = 1;,
    "javascriptwithscope" : { "$code" : var l = 1; ,  "$scope" : { "scope" : "scope_val" } },
    "symbol" : "a",
    "timestamp" : Timestamp(1564417066, 1),
    "long" : NumberLong(9223372036854775807),
    "decimal" : NumberDecimal("3.100000"),
    "minkey" : { "$minKey" : 1 },
    "maxkey" : { "$maxKey" : 1 }
}

BSON

BSON is a binary serialization format used to store documents and make remote procedure calls in MongoDB. The BSON specification is located at bsonspec.org

Document limitations

• The maximum BSON document size is 16 megabytes.
• In MongoDB, each document stored in a collection requires a unique _id field that acts as a primary key. If an inserted document omits the _id field, the MongoDB driver automatically generates an ObjectId for the _id field.

Flexible schema

Unlike SQL databases, where you must determine and declare a table’s schema before inserting data, MongoDB’s collections, by default, does not require its documents to have the same schema.

• The documents in a single collection do not need to have the same set of fields and the data type for a field can differ across documents within a collection. 
• To change the structure of the documents in a collection, such as add new fields, remove existing fields, or change the field values to a new type, update the documents to the new structure.

Query filter documents

A query filter document and query operators can be used to specify conditions.

The following example uses '{ status: { $in: [ "A", "D" ] } }' query filter document to retrieve all documents from the 'inventory' collection where 'status' equals either "A" or "D":

db.inventory.find( { status: { $in: [ "A", "D" ] } } )

The operation corresponds to the following SQL statement:

SELECT * FROM inventory WHERE status in ("A", "D")

Sink Properties

Source Properties

Data Types Mapping

Approach

Move existing mongodb-plugins module to the mongodb-plugins project. Add MongoDB-specific properties to configuration, add support for MongoDB-specific datatypes. Update UI widgets JSON definitions.

Pipeline Samples

Releases

Release X.Y.Z

Related Work

Database plugin enhancements