Introduction

This plugin would allow SAP ECC enterprise users to specify one of the Inventory Management data sources as source in Cloud Data Fusion. Typical pipeline scenario would include specifying one of the data sources as source (under Sources plugins) and BigQuery as sink. For the sake of simplicity, this document will cover one data source (Material Movements from Inventory Management - 2LIS_03_BF).

User Experience and Configuration

SAP ECC Setup

SAP ECC users log into SAP Gateway Service Builder (segw) to create and expose the data source as SAP OData service. SAP has published following articles around this step:

User creates OData service for the data source (2LIS_03_BF). Output is RESTful OData service that can be accessed by Cloud Data Fusion.
User can also use SAP Netweaver Gateway client to test the service (with HTTP request and response)

...

Data type mismatch - OData output has data types (section 6) different than the data types available in BigQuery. Some sort of data type conversion needs to happen before the extracted data can be ingested into BigQuery. Below is the suggested mapping of OData data types to CDAP schema data types:

...

Types can be mapped as follows, according to the org.apache.olingo.commons.api.edm.EdmPrimitiveType and OData CSDL Primitive Types:

EDM primitive type

CDAP Schema Data Type

JSON

XML

Comment

Edm.Binary

bytes

Edm.Boolean

boolean

Edm.Byte

int

Unsigned 8-bit integer value

Edm.Date

timestamp

Date without a time-zone offset

Edm.DateTimeOffset

string

Represents date and time as an Offset in minutes from GMT, with values ranging from 12:00:00 midnight, January 1, 1753 A.D. through 11:59:59 P.M, December 9999 A.D

Example 1: 2002-10-10T17:00:00Z

Mapped to string to avoid timezone information loss.

Edm.Decimal

decimal

Edm.Double

double

Edm.Duration

string

Code Block
"Duration@odata.type": "#Duration", "Duration": "P12DT23H59M59.999999999999S",

Code Block
<d:Duration m:type="Edm.Duration">P12DT23H59M59.999999999999S</d:Duration>

Edm.Guid

string

Edm.Int16

int

Edm.Int32

int

Edm.Int64

long

Edm.SByte

int

Represents a signed 8-bit integer value

Edm.Single

float

Edm.Stream

Edm.String

string

Edm.TimeOfDay

time

Clock time 00:00-23:59:59.999999999999

Edm.Geography

-

Abstract base type for all Geography types.

Edm.GeographyPoint

record

Code Block
"GeographyPoint": { "type": "Point", "coordinates": [142.1, 64.1] }

Code Block

language	xml

<d:GeographyPoint m:type="GeographyPoint">
  <gml:Point gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
    <gml:pos>15 161.8</gml:pos>
  </gml:Point>
</d:GeographyPoint>

A point in a round-earth coordinate system

Edm.GeographyLineString

record

Code Block
"GeographyLineString": { "type": "LineString", "coordinates": [ [100.0, 0.0], [101.0, 1.0] ] }

Code Block

language	xml

<d:GeographyLineString m:type="Edm.GeographyLineString">
  <gml:LineString gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
    <gml:pos>10 10</gml:pos>
    <gml:pos>20 20</gml:pos>
    <gml:pos>40 10</gml:pos>
  </gml:LineString>
</d:GeographyLineString>

Line string in a round-earth coordinate system

Edm.GeographyPolygon

record

Code Block
"GeographyPolygon": { "type": "Polygon", "coordinates": [ [10.0, 10.0], [10.0, 40.0], [ 40.0, 10.0], [40.0, 40.0] ] }

Code Block

language	xml

<d:GeographyPolygon m:type="Edm.GeographyPolygon">
  <gml:Polygon gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326"/>
  <gml:pos>10 10</gml:pos>
  <gml:pos>10 40</gml:pos>
  <gml:pos>40 10</gml:pos>
  <gml:pos>40 40</gml:pos>
</d:GeographyPolygon>

Polygon in a round-earth coordinate system

Edm.GeographyMultiPoint

record

Code Block
"GeographyMultiPoint": { "type": "MultiPoint", "coordinates": [ [ 2.5, 3.125], [3.5, 4.125], [4.5, 5.125] ] }

Code Block

language	xml

<d:GeographyMultiPoint m:type="Edm.GeographyMultiPoint">
  <gml:MultiPoint gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
    <gml:pointMembers>
      <gml:Point>
        <gml:pos>47.38 -122.7</gml:pos>
      </gml:Point>
    </gml:pointMembers>
  </gml:MultiPoint>
</d:GeographyMultiPoint>

Collection of points in a round-earth coordinate system

Edm.GeographyMultiLineString

record

Code Block
"GeographyMultiLineString": { "type": "MultiLineString", "coordinates": [ [[1.0,1.0],[2.0,2.0],[3.0,3.0],[4.0,4.0],[5.0,5.0]], [[99.5,101.5],[150.0,151.25]] ] }

Code Block

language	xml

<d:GeographyMultiLineString m:type="Edm.GeographyMultiLineString">
  <gml:MultiCurve gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
    <gml:curveMembers>
      <gml:LineString>
        <gml:pos>10.5 10.5</gml:pos>
        <gml:pos>20.5 20.5</gml:pos>
        <gml:pos>40.5 10.5</gml:pos>
      </gml:LineString>
      <gml:LineString>
        <gml:pos>40.5 40.5</gml:pos>
        <gml:pos>30.5 30.5</gml:pos>
        <gml:pos>20.5 40.5</gml:pos>
        <gml:pos>10.5 30.5</gml:pos>
      </gml:LineString>
    </gml:curveMembers>
  </gml:MultiCurve>
</d:GeographyMultiLineString>

Collection of line strings in a round-earth coordinate system

Edm.GeographyMultiPolygon

record

Code Block

"GeographyMultiPolygon": {
  "type": "MultiPolygon",
  "coordinates": [
    [
      [[0.0,0.0],[3.0,0.0],[3.0,3.0],[0.0,3.0],[0.0,0.0]],
      [[1.0,1.0],[1.0,2.0],[2.0,2.0],[2.0,1.0],[1.0,1.0]]],
    [
      [[0.0,0.0],[30.0,0.0],[0.0,30.0],[0.0,0.0]],
      [[10.0,10.0],[10.0,20.0],[20.0,10.0],[10.0,10.0]]
    ]
  ]
}

Code Block

language	xml

<d:GeographyMultiPolygon m:type="Edm.GeographyMultiPolygon">
  <gml:MultiSurface
          gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
    <gml:surfaceMembers>
      <gml:Polygon>
        <gml:exterior>
          <gml:LinearRing>
            <gml:pos>40 40</gml:pos>
            <gml:pos>45 20</gml:pos>
            <gml:pos>30 45</gml:pos>
            <gml:pos>40 40</gml:pos>
          </gml:LinearRing>
        </gml:exterior>
      </gml:Polygon>
      <gml:Polygon>
        <gml:exterior>
          <gml:LinearRing>
            <gml:pos>35 20</gml:pos>
            <gml:pos>20 45</gml:pos>
            <gml:pos>5 30</gml:pos>
            <gml:pos>10 10</gml:pos>
            <gml:pos>30 10</gml:pos>
            <gml:pos>35 20</gml:pos>
          </gml:LinearRing>
        </gml:exterior>
        <gml:interior>
          <gml:LinearRing>
            <gml:pos>20 30</gml:pos>
            <gml:pos>25 20</gml:pos>
            <gml:pos>15 20</gml:pos>
            <gml:pos>20 30</gml:pos>
          </gml:LinearRing>
        </gml:interior>
      </gml:Polygon>
    </gml:surfaceMembers>
  </gml:MultiSurface>
</d:GeographyMultiPolygon>

Collection of polygons in a round-earth coordinate system

Edm.GeographyCollection

record

Code Block

"GeographyCollection@odata.type": "#GeographyCollection",
"GeographyCollection": { 
  "type": "GeometryCollection",
  "geometries": [
    { "type": "Point",
      "coordinates": [100.0, 0.0]
    },
    { "type": "LineString",
      "coordinates": [ [101.0, 0.0], [102.0, 1.0] ]
    }
  ]
}

Code Block

language	xml

<d:GeographyCollection m:type="Edm.GeographyCollection">
  <gml:MultiGeometry
          gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326" />

  <gml:LineString gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
    <gml:pos>10 10</gml:pos>
    <gml:pos>20 20</gml:pos>
    <gml:pos>40 10</gml:pos>
  </gml:LineString>

  <gml:Polygon gml:srsName="http://www.opengis.net/def/crs/EPSG/0/4326">
    <gml:pos>10 10</gml:pos>
    <gml:pos>10 40</gml:pos>
    <gml:pos>40 10</gml:pos>
    <gml:pos>40 40</gml:pos>
  </gml:Polygon>
</d:GeographyCollection>

Collection of arbitrary Geography values.

Note, that JSON representation contains type control information (`@odata.type`). Otherwise, Olingo client reports invalid value type for `Edm.GeometryCollection` and `Edm.GeographyCollection` JSON values.

Edm.Geometry

Abstract base type for all Geometry types

Edm.GeometryPoint

record

Similar to Edm.GeographyPoint

Point in a flat-earth coordinate system

Edm.GeometryLineString

record

Similar to Edm.GeographyLineString

Line string in a flat-earth coordinate system

Edm.GeometryPolygon

record

Similar to Edm.GeographyPolygon

Polygon in a flat-earth coordinate system

Edm.GeometryMultiPoint

record

Similar to Edm.GeographyMultiPoint

Collection of points in a flat-earth coordinate system

Edm.GeometryMultiLineString

record

Similar to Edm.GeographyMultiLineString

Collection of line strings in a flat-earth coordinate system

Edm.GeometryMultiPolygon

record

Similar to Edm.GeographyMultiPolygon

Collection of polygons in a flat-earth coordinate system

Edm.GeometryCollection

record

Similar to Edm.GeographyCollection

Collection of arbitrary Geometry values

References

OData service

SAP Netweaver Gateway

Overview - https://blogs.sap.com/2013/01/24/a-simple-overview-on-sap-netweaver-gateway/
SAP Netweaver Gtw and OData Tutorials: https://sapyard.com/tutorials-on-odata-sap-netweaver-gateway/

Microsoft Data Factory - SAP ECC Connector

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

...

Versions Compared

Old Version 15

New Version 16

Key

Introduction

User Experience and Configuration

References

Plugin Type

Page Comparison

Versions Compared

Old Version 15

New Version 16

Key

User Experience and Configuration

References

Plugin Type