Introduction
This plugin would allow SAP ECC enterprise users to specify one of the Inventory Management data sources as source in Cloud Data Fusion. Typical pipeline scenario would include specifying one of the data sources as source (under Sources plugins) and BigQuery as sink. For the sake of simplicity, this document will cover one data source (Material Movements from Inventory Management - 2LIS_03_BF).
User Experience and Configuration
- SAP ECC Setup
SAP ECC users log into SAP Gateway Service Builder (segw) to create and expose the data source as SAP OData service. SAP has published following articles around this step:
User creates OData service for the data source (2LIS_03_BF). Output is RESTful OData service that can be accessed by Cloud Data Fusion.
User can also use SAP Netweaver Gateway client to test the service (with HTTP request and response)
...
Data type mismatch - OData output has data types (section 6) different than the data types available in BigQuery. Some sort of data type conversion needs to happen before the extracted data can be ingested into BigQuery. Below is the suggested mapping of OData data types to CDAP schema data types:
...
Types can be mapped as follows, according to the org.apache.olingo.commons.api.edm.EdmPrimitiveType and OData CSDL Primitive Types:
EDM primitive type | CDAP Schema Data Type | JSON | XML | Comment | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Edm.Binary | bytes | ||||||||||
Edm.Boolean | boolean | ||||||||||
Edm.Byte | int | Unsigned 8-bit integer value | |||||||||
Edm.Date | timestamp | Date without a time-zone offset | |||||||||
Edm.DateTimeOffset | string | Represents date and time as an Offset in minutes from GMT, with values ranging from 12:00:00 midnight, January 1, 1753 A.D. through 11:59:59 P.M, December 9999 A.D Example 1: 2002-10-10T17:00:00Z Mapped to string to avoid timezone information loss. | |||||||||
Edm.Decimal | decimal | ||||||||||
Edm.Double | double | ||||||||||
Edm.Duration | |||||||||||
Edm.Guid | string | ||||||||||
Edm.Int16 | int | ||||||||||
Edm.Int32 | int | ||||||||||
Edm.Int64 | long | ||||||||||
Edm.SByte | int | Represents a signed 8-bit integer value | |||||||||
Edm.Single | float | ||||||||||
Edm.Stream | |||||||||||
Edm.String | string | ||||||||||
Edm.TimeOfDay | time | Clock time 00:00-23:59:59.999999999999 | |||||||||
Edm.Geography | Abstract base type for all Geography types | ||||||||||
Edm.GeographyPoint | record |
|
| A point in a round-earth coordinate system | |||||||
Edm.GeographyLineString | record |
|
| Line string in a round-earth coordinate system | |||||||
Edm.GeographyPolygon | record |
|
| Polygon in a round-earth coordinate system | |||||||
Edm.GeographyMultiPoint | record |
|
| Collection of points in a round-earth coordinate system | |||||||
Edm.GeographyMultiLineString | record |
|
| Collection of line strings in a round-earth coordinate system | |||||||
Edm.GeographyMultiPolygon | record |
|
| Collection of polygons in a round-earth coordinate system | |||||||
Edm.GeographyCollection | record |
|
| Collection of arbitrary Geography values | |||||||
Edm.Geometry | Abstract base type for all Geometry types | ||||||||||
Edm.GeometryPoint | record | Similar to Edm.GeographyPoint | Similar to Edm.GeographyPoint | Point in a flat-earth coordinate system | |||||||
Edm.GeometryLineString | record | Similar to Edm.GeographyLineString | Similar to Edm.GeographyLineString | Line string in a flat-earth coordinate system | |||||||
Edm.GeometryPolygon | record | Similar to Edm.GeographyPolygon | Similar to Edm.GeographyPolygon | Polygon in a flat-earth coordinate system | |||||||
Edm.GeometryMultiPoint | record | Similar to Edm.GeographyMultiPoint | Similar to Edm.GeographyMultiPoint | Collection of points in a flat-earth coordinate system | |||||||
Edm.GeometryMultiLineString | record | Similar to Edm.GeographyMultiLineString | Similar to Edm.GeographyMultiLineString | Collection of line strings in a flat-earth coordinate system | |||||||
Edm.GeometryMultiPolygon | record | Similar to Edm.GeographyMultiPolygon | Similar to Edm.GeographyMultiPolygon | Collection of polygons in a flat-earth coordinate system | |||||||
Edm.GeometryCollection | record | Similar to Edm.GeographyCollection | Similar to Edm.GeographyCollection | Collection of arbitrary Geometry values |
References
OData service
SAP Netweaver Gateway
Overview - https://blogs.sap.com/2013/01/24/a-simple-overview-on-sap-netweaver-gateway/
SAP Netweaver Gtw and OData Tutorials: https://sapyard.com/tutorials-on-odata-sap-netweaver-gateway/
Microsoft Data Factory - SAP ECC Connector
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
...