Versions Compared

Old Version 4

changes.mady.by.user Vlad Hlinskiy

Saved on

compared with

New Version 5

changes.mady.by.user Vlad Hlinskiy

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Introduction

A separate database plugin to support MongoDB-specific features and configurations.

...

The following example uses '{ status: { $in: [ "A", "D" ] } }' query filter document to retrieve all documents from the 'inventory' collection where 'status' equals either "A" or "D":

...

User Facing NameTypeDescriptionConstraints
LabelStringLabel for UI.
Reference NameStringUniquely identified name for lineage.
HostStringHost that MongoDB is running on.

Required

(defaults to localhost on UI)

PortNumberPort that MongoDB is listening to.

Optional

(default 27017)

DatabaseStringMongoDB database name.Required
CollectionStringName of the database collection to write to.Required
Output SchemaSchemaSpecifies the schema of the documents.Required
Input QueryStringOptionally filter the input collection with a query. This query must be represented in JSON format and use the MongoDB extended JSON format to represent non-native JSON data types.
Input FieldsStringProjection document that can limit the fields that appear in each document. This must be represented in JSON format, and use the MongoDB extended JSON format to represent non-native JSON data types. If no projection document is provided, all fields will be read.Splitter Class

The name of the Splitter class to use. If left empty, the MongoDB Hadoop Connector will attempt to make a best-guess as to which Splitter to use.

The Hadoop connector provides these Splitters:

  • com.mongodb.hadoop.splitter.StandaloneMongoSplitter
  • com.mongodb.hadoop.splitter.ShardMongoSplitter
  • com.mongodb.hadoop.splitter.ShardChunkMongoSplitter
    • com.mongodb.hadoop.splitter.MultiMongoCollectionSplitter
    UsernameStringUser identity for connecting to the specified database.
    PasswordPasswordPassword to use to connect to the specified database.
    Authentication Connection String
    		Auxiliary MongoDB connection string to authenticate against when constructing splits.
    Connection ArgumentsKeyvalue

    A list of arbitrary string key/value pairs as connection arguments. See Connection String Options for a full description of these arguments.


    ...

    MongoDB Data TypeCDAP Schema Data TypeSupportComment
    Double
    		Schema.Type.DOUBLE+
    StringSchema.Type.STRING+
    ObjectSchema.Type.RECORD+
    ArraySchema.Type.ARRAY+
    Binary dataSchema.Type.BYTES*

    Value can be mapped to Schema.Type.BYTES, but this can lead to subtype information loss.

    Subtypes:


    • generic: \x00 (0)
    • function: \x01 (1)
    • old: \x02 (2)
    • uuid_old: \x03 (3)
    • uuid: \x04 (4)
    • md5: \x05 (5)
    • user: \x80 (128)


    There are several options:

    1) Support only 'generic' subtype.

    2) Map using MongoDB extended JSON format:

    "binary": {"$binary": "YmluYXJ5IGRhdGE=", "$type": "00"}

    UndefinedSchema.Type.NULL*

    Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

    "undefined": {"$undefined": true}

    ObjectId
    		*

    Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

    There are several options:

    1) Do not support this data type for the Sink

    2) Map using MongoDB extended JSON format: {"$oid": "5d3f1c2a2f547625b0bbb397"}

    BooleanSchema.Type.BOOLEAN+
    DateSchema.LogicalType.TIMESTAMP_MILLIS+
    NullSchema.Type.UNION+A nullable version of the actual type, corresponds to Schema.nullableOf(actualTypeSchema).
    Regular ExpressionSchema.Type.STRING*

    Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

    There are several options:

    1) Do not support this data type for the Sink

    2) Map using MongoDB extended JSON format: "regex": {"$regex": ".", "$options": ""}

    DBPointerSchema.Type.STRING*

    String in MongoDB extended JSON format:

    "dbpointer": {"$ref": "source", "$id": {"$oid": "5d079ee6d078c94008e4bb3a"}}

    JavaScriptSchema.Type.STRING*

    Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

    There are several options:

    1) Do not support this data type for the Sink

    2) Map using MongoDB extended JSON format: "javascript": {"$code": "var l = 1;"}

    SymbolSchema.Type.STRING*

    Value can be mapped to Schema.Type.STRING, but this will lead to type information loss.

    There are several options:

    1) Do not support this data type for the Sink

    2) Map using MongoDB extended JSON format: "symbol": {"$symbol": "a"}

    JavaScript (with scope)Schema.Type.STRING*

    Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

    "javascriptwithscope": {"$code": "var l = 1;", "$scope": {"scope": "scope_val"}

    32-bit integerSchema.Type.INT+
    Timestamp
    		*

    Special type for internal MongoDB use which is not associated with the regular Date type. Timestamp values are a 64 bit value where:

    • the first 32 bits are a time_t value (seconds since the Unix epoch)
    • the second 32 bits are an incrementing ordinal for operations within a given second.

    Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

    "timestamp": {"$timestamp": {"t": 1564410161, "i": 1}}

    64-bit integerSchema.Type.LONG+
    Decimal128Schema.LogicalType.DECIMAL+
    Min key
    		*

    Is less than any other value of any type. This can be useful for always returning certain documents first (or last).

    Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

    "minkey": {"$minKey": 1}

    Max key
    		*

    Is greater than any other value of any type. This can be useful for always returning certain documents first (or last).

    Can be mapped to Schema.Type.STRING using MongoDB extended JSON format:

    "maxkey": {"$maxKey": 1}

    ...