Introduction

A separate database plugin to support MongoDB-specific features and configurations.

...

The following example uses '{ status: { $in: [ "A", "D" ] } }' query filter document to retrieve all documents from the 'inventory' collection where 'status' equals either "A" or "D":

...

The following MongoDB data types are missing: Undefined, Regular Expression, DBPointer, JavaScript, Symbol, JavaScript (with scope), Timestamp, Min key, Max key.

CDAP Schema Data Type	MongoDB Data Types	Comment
boolean	Boolean
bytes	Binary data, ObjectId(if 'ID Field' specified)
date	Date
double	Double
decimal	Decimal128	The Decimal128 type supports up to 34 digits of precision.
float	Double
int	32-bit integer
long	64-bit integer
string	String, ObjectId(if 'ID Field' specified)
time	String
timestamp	Date
array	Array
record	Object
enum

Array

String

map

Object

union

Depends on the actual value.

For example, if it's a union:

Code Block
["string","int","long"]

and the value is actually a long, the mongo document will have the field as a 64-bit integer. If a different record comes in with the value as a string, the mongo document will end up with a String for that field.

Source Properties

User Facing Name	Widget Type	Description	Constraints
Label	textbox	Label for UI.
Reference Name	textbox	Uniquely identified name for lineage.
Host	textbox	Host that MongoDB is running on.	Required (defaults to localhost on UI)
Port	number	Port that MongoDB is listening to.	Optional (default 27017)
Database	textbox	MongoDB database name.	Required
Collection	textbox	Name of the database collection to write to.	Required
Output Schema	schema	Specifies the schema of the documents.	Required
On Record Error	selectradio-group	Specifies how to handle error in record processing. An error will be thrown if failed to parse value according to a provided schema.	Possible values are: Skip error Fail pipelineWrite to error dataset Default: 'Fail pipeline'
Input Query	json-editor	Optionally filter the input collection with a query. This query must be represented in JSON format and use the MongoDB extended JSON format to represent non-native JSON data types.
Username	textbox	User identity for connecting to the specified database.
Password	password	Password to use to connect to the specified database.
Authentication Connection String	textbox	Auxiliary MongoDB connection string to authenticate against when constructing splits.
Connection Arguments	keyvalue	A list of arbitrary string key/value pairs as connection arguments. See Connection String Options for a full description of these arguments.

...

The source requires Output Schema to be set. Based on the schema source will expect a field in each document to be of a specific Mongo data type.

On Record Error error handling property allows the user to decide whether the pipeline should fail, the record should be skipped, or the record should be sent to the error dataset.

...

CDAP Schema Data Type

MongoDB Data Types

boolean

Boolean

bytes

Binary data, ObjectId

date

-

double

Double

decimal

Decimal128

float

-

int

32-bit integer

long

64-bit integer

string

String, Symbol

time

-

timestamp

Date

array

Array

record

Object

The following schema:

Code Block
{"type":"record","name":"object","fields":[{"name":"inner_field","type":"string"}]}

is used for 'object' field:

Code Block
{ "object" : { "inner_field" : "val" } }

* We can map all non-standard data types to record, like JavaScript (with scope) in the example below.

The following schema:

Code Block

{
  "type":"record",
  "name":"javascriptwithscope",
  "fields":[
    {"name":"$code","type":"string"},
    {"name":"$scope","type":{"type":"record","name":"scope-object-record","fields"[{"name":"scope","type":"string"}]}}
  ]
}

is used for 'javascriptwithscope' field:

Code Block
{ "javascriptwithscope" : { "$code" : var l = 1; , "$scope" : { "scope" : "scope_val" } } }

enum

-

map

Object

The following schema:

Code Block
{"type":"map","keys":"string","values":"string"}

is used for 'object' field:

Code Block
{ "object" : { "inner_field" : "val" } }

union

-

...

Versions Compared

Old Version 8

New Version Current

Key

Introduction

Source Properties

Page Comparison

Versions Compared

Old Version 8

New Version Current

Key

Source Properties