Introduction
This plugin would allow SAP ECC enterprise users to specify one of the Inventory Management data sources as source in Cloud Data Fusion. Typical pipeline scenario would include specifying one of the data sources as source (under Sources plugins) and BigQuery as sink. For the sake of simplicity, this document will cover one data source (Material Movements from Inventory Management - 2LIS_03_BF).
User Experience and Configuration
- SAP ECC Setup
SAP ECC users log into SAP Gateway Service Builder (segw) to create and expose the data source as SAP OData service. SAP has published following articles around this step:
User creates OData service for the data source (2LIS_03_BF). Output is RESTful OData service that can be accessed by Cloud Data Fusion.
User can also use SAP Netweaver Gateway client to test the service (with HTTP request and response)
...
Data type mismatch - OData output has data types (section 6) different than the data types available in BigQuery. Some sort of data type conversion needs to happen before the extracted data can be ingested into BigQuery. Below is the suggested mapping of OData data types to CDAP schema data types:
...
Section | User Facing Name | Widget Type | Description | Constraints |
---|---|---|---|---|
General | Label | textbox | Label for UI. | |
Reference Name | textbox | Uniquely identified name for lineage. | ||
OData Service URL | textbox | URL of the SAP ECC OData service. The URL must end with an external service name (e.g., http://eccsvrname:8000/sap/opu/odata/sap/zgw100_dd02l_so_srv/). | Required | |
Resource Path | textbox | Path of the SAP ECC OData entity. For example:
| Required | |
Query Options | textboxkeyvalue | OData query options to filter the data. For example:
The plugin copies data from the combined URL: For more information, see OData URL components. | ||
Include Metadata Annotations | toggle | Whether the plugin should read SAP metadata annotations and include them to each record. | ||
Credentials | Username | textbox | Username for basic authentication. | |
Password | password | Password for basic authentication. | ||
Output Schema | schema | Specifies the schema of the entries. |
...
Types can be mapped as follows, according to the org.apache.olingo.commons.api.edm.EdmPrimitiveType and OData CSDL Primitive Types:
EDM primitive type | CDAP Schema Data Type | JSON | XML | Comment | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Edm.Binary | bytes | ||||||||||
Edm.Boolean | boolean | ||||||||||
Edm.Byte | int | Unsigned 8-bit integer value | |||||||||
Edm.Date | timestamp | Date without a time-zone offset | |||||||||
Edm.DateTimeOffset | string | Represents date and time as an Offset in minutes from GMT, with values ranging from 12:00:00 midnight, January 1, 1753 A.D. through 11:59:59 P.M, December 9999 A.D Example 1: 2002-10-10T17:00:00Z Mapped to string to avoid timezone information loss. | |||||||||
Edm.Decimal | decimal | ||||||||||
Edm.Double | double | ||||||||||
Edm.Duration | string |
|
| Signed duration in days, hours, minutes, and (sub)seconds. | |||||||
Edm.Guid | string | ||||||||||
Edm.Int16 | int | ||||||||||
Edm.Int32 | int | ||||||||||
Edm.Int64 | long | ||||||||||
Edm.SByte | int | Represents a signed 8-bit integer value | |||||||||
Edm.Single | float | ||||||||||
Edm.Stream | record |
|
| Binary data stream. Olingo client maps this data type to java.net.URI. The proposal is to map this data type to CDAP record that corresponds to the JSON representation. | |||||||
Edm.String | string | ||||||||||
Edm.TimeOfDay | time | Clock time 00:00-23:59:59.999999999999 | |||||||||
Edm.Geography | - | Abstract base type for all Geography types. | |||||||||
Edm.GeographyPoint | record |
|
| A point in a round-earth coordinate system | |||||||
Edm.GeographyLineString | record |
|
| Line string in a round-earth coordinate system | |||||||
Edm.GeographyPolygon | record |
|
| Polygon in a round-earth coordinate system | |||||||
Edm.GeographyMultiPoint | record |
|
| Collection of points in a round-earth coordinate system | |||||||
Edm.GeographyMultiLineString | record |
|
| Collection of line strings in a round-earth coordinate system | |||||||
Edm.GeographyMultiPolygon | record |
|
| Collection of polygons in a round-earth coordinate system | |||||||
Edm.GeographyCollection | record |
|
| Collection of arbitrary Geography values. Note, that JSON representation contains type control information (`@odata.type`). Otherwise, Olingo client reports invalid value type for `Edm.GeometryCollection` and `Edm.GeographyCollection` JSON values. Since metadata does not contain component info, the proposal is to map this data type to CDAP record with the following fields:
| |||||||
Edm.Geometry | - | Abstract base type for all Geometry types | |||||||||
Edm.GeometryPoint | record | Similar to Edm.GeographyPoint | Similar to Edm.GeographyPoint | Point in a flat-earth coordinate system | |||||||
Edm.GeometryLineString | record | Similar to Edm.GeographyLineString | Similar to Edm.GeographyLineString | Line string in a flat-earth coordinate system | |||||||
Edm.GeometryPolygon | record | Similar to Edm.GeographyPolygon | Similar to Edm.GeographyPolygon | Polygon in a flat-earth coordinate system | |||||||
Edm.GeometryMultiPoint | record | Similar to Edm.GeographyMultiPoint | Similar to Edm.GeographyMultiPoint | Collection of points in a flat-earth coordinate system | |||||||
Edm.GeometryMultiLineString | record | Similar to Edm.GeographyMultiLineString | Similar to Edm.GeographyMultiLineString | Collection of line strings in a flat-earth coordinate system | |||||||
Edm.GeometryMultiPolygon | record | Similar to Edm.GeographyMultiPolygon | Similar to Edm.GeographyMultiPolygon | Collection of polygons in a flat-earth coordinate system | |||||||
Edm.GeometryCollection | record | Similar to Edm.GeographyCollection | Similar to Edm.GeographyCollection | Collection of arbitrary Geometry values |
OData V4 Metadata Annotations Mapping
An annotation applies a term to a model element and defines how to calculate a value for the term application. The value of an annotation is specified as an annotation expression, which is either a constant expression representing a constant value or a dynamic expression.
"Include Metadata Annotations" configuration property indicates whether the plugin should read metadata annotations and include them to each record. In this case, each property will be mapped to a CDAP 'record' with exactly two fields "value" and "metadata-annotations". OData V4 metadata annotations are mapped to a record of the following fields:
- term - a simple identifier, such as "UI.DisplayName" or "Core.Description", etc.
- qualifier - a term can be applied multiple times to the same model element by providing a qualifier to distinguish the annotations.
- expression - CDAP record that corresponds to a constant expression or a dynamic expression. Please, refer the table below for expressions mapping.
- annotations - CDAP record that corresponds to nested annotations. Please, refer examples below for nested annotations mapping.
OData V4 Annotation Expression Mapping
EDM expression | Type | Example | CDAP record schema | |||||
---|---|---|---|---|---|---|---|---|
Binary | constant |
|
| |||||
Bool | constant |
|
| |||||
Date | constant |
|
| |||||
DateTimeOffset | constant |
|
| |||||
Decimal | constant |
|
| |||||
Duration | constant |
|
| |||||
EnumMember | constant |
|
| |||||
Float | constant |
|
| |||||
Guid | constant |
|
| |||||
Int | constant |
|
| |||||
String | constant |
|
| |||||
TimeOfDay | constant |
|
| |||||
Path | dynamic |
|
| |||||
AnnotationPath | dynamic |
|
| |||||
LabeledElementReference | dynamic |
|
| |||||
Null | dynamic |
|
| |||||
NavigationPropertyPath | dynamic |
|
| |||||
PropertyPath | dynamic |
|
| |||||
And | dynamic |
|
| |||||
Or | dynamic |
|
| |||||
Eq | dynamic |
|
| |||||
Ne | dynamic |
|
| |||||
Gt | dynamic |
|
| |||||
Ge | dynamic |
|
| |||||
Lt | dynamic |
|
| |||||
Le | dynamic |
|
| |||||
Not | dynamic |
|
| |||||
Apply | dynamic |
|
Note: the proposal is to use the parameter index as a prefix for field name to avoid conflicts. In this case, parameters record will be a record of the following fields:
| |||||
Cast | dynamic |
|
| |||||
Collection | dynamic |
|
| |||||
If | dynamic |
|
| |||||
IsOf | dynamic |
|
| |||||
LabeledElement | dynamic |
|
| |||||
Record | dynamic |
|
| |||||
UrlRef | dynamic |
|
|
Notes:
- An edm:Annotation element MAY contain edm:Annotation elements that annotate the annotation.
- The edm:Annotations element is used to apply a group of annotations to a single model element. These are annotations with External Targeting.
References
OData service
SAP Netweaver Gateway
Overview - https://blogs.sap.com/2013/01/24/a-simple-overview-on-sap-netweaver-gateway/
SAP Netweaver Gtw and OData Tutorials: https://sapyard.com/tutorials-on-odata-sap-netweaver-gateway/
Microsoft Data Factory - SAP ECC Connector
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
...