Introduction

This plugin would allow SAP ECC enterprise users to specify one of the Inventory Management data sources as source in Cloud Data Fusion. Typical pipeline scenario would include specifying one of the data sources as source (under Sources plugins) and BigQuery as sink. For the sake of simplicity, this document will cover one data source (Material Movements from Inventory Management - 2LIS_03_BF).

User Experience and Configuration

SAP ECC Setup

SAP ECC users log into SAP Gateway Service Builder (segw) to create and expose the data source as SAP OData service. SAP has published following articles around this step:

User creates OData service for the data source (2LIS_03_BF). Output is RESTful OData service that can be accessed by Cloud Data Fusion.
User can also use SAP Netweaver Gateway client to test the service (with HTTP request and response)

...

Data type mismatch - OData output has data types (section 6) different than the data types available in BigQuery. Some sort of data type conversion needs to happen before the extracted data can be ingested into BigQuery. Below is the suggested mapping of OData data types to CDAP schema data types:

...

Types can be mapped as follows, according to the org.apache.olingo.commons.api.edm.EdmPrimitiveType and OData CSDL Primitive Types:

...

OData V4 Annotation Expression Mapping

EDM expression

Type

Example

CDAP record schema

Binary

constant

Code Block

language	xml

<Annotation Term="org.example.display.Thumbnail">
  <Binary>T0RhdGE</Binary>
</Annotation>

name - field of type 'string' with value "Binary" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Bool

constant

Code Block

language	xml

<Annotation Term="org.example.display.ReadOnly">
  <Bool>true</Bool>
</Annotation>

name - field of type 'string' with value "Bool" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Date

constant

Code Block

language	xml

<Annotation Term="org.example.vCard.birthDay">
  <Date>2000-01-01</Date>
</Annotation>

name - field of type 'string' with value "Date" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

DateTimeOffset

constant

Code Block

language	xml

<Annotation Term="org.example.display.LastUpdated">
  <DateTimeOffset>2000-01-01T16:00:00.000-09:00</DateTimeOffset>
</Annotation>

name - field of type 'string' with value "DateTimeOffset" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Decimal

constant

Code Block

language	xml

<Annotation Term="org.example.display.Width">
  <Decimal>3.14</Decimal>
</Annotation>

name - field of type 'string' with value "Decimal" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Duration

constant

Code Block

language	xml

<Annotation Term="org.example.task.duration">
  <Duration>P11DT23H59M59.999999999999S</Duration>
</Annotation>

name - field of type 'string' with value "Duration" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

EnumMember

constant

Code Block

language	xml

<Annotation Term="org.example.HasPattern">
  <EnumMember>org.example.Pattern/Red</EnumMember>
</Annotation>

name - field of type 'string' with value "EnumMember" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Float

constant

Code Block

language	xml

<Annotation Term="org.example.display.Width">
  <Float>3.14</Float>
</Annotation>

name - field of type 'string' with value "Float" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Guid

constant

Code Block

language	xml

<Annotation Term="org.example.display.Id">
  <Guid>21EC2020-3AEA-1069-A2DD-08002B30309D</Guid>
</Annotation>

name - field of type 'string' with value "Guid" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Int

constant

Code Block

language	xml

<Annotation Term="org.example.display.Width">
  <Int>42</Int>
</Annotation>

name - field of type 'string' with value "Int" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

String

constant

Code Block

language	xml

<Annotation Term="org.example.display.DisplayName">
  <String>Product Catalog</String>
</Annotation>

name - field of type 'string' with value "String" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

TimeOfDay

constant

Code Block

language	xml

<Annotation Term="org.example.display.EndTime">
  <TimeOfDay>21:45:00</TimeOfDay>
</Annotation>

name - field of type 'string' with value "TimeOfDay" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Path

dynamic

Code Block

language	xml

<Annotation Term="org.example.display.DisplayName">
  <Path>@vCard.Address#work/FullName</Path>
</Annotation>

name - field of type 'string' with value "Path" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

AnnotationPath

dynamic

Code Block

language	xml

<Annotation Term="UI.CollectionFacet" Qualifier="Contacts">
  <Collection>
    <AnnotationPath>Supplier/@Communication.Contact</AnnotationPath>
    <AnnotationPath>Customer/@Communication.Contact</AnnotationPath>
  </Collection>
</Annotation>

name - field of type 'string' with value "AnnotationPath" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

LabeledElementReference

dynamic

Code Block

language	xml

<Annotation Term="org.example.display.DisplayName">
  <LabeledElementReference>Model.CustomerFirstName</LabeledElementReference>
</Annotation>

name - field of type 'string' with value "LabeledElementReference" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Null

dynamic

Code Block

language	xml

<Annotation Term="@UI.Address">
  <Null>
    <Annotation Term="self.Reason" String="Private" />
  </Null>
</Annotation>

name - field of type 'string' with value "Null" to distinguish expression.
value - field of type 'string' with null value.
annotations - CDAP record that corresponds to nested annotations, since the edm:Null element MAY contain edm:Annotation elements.

NavigationPropertyPath

dynamic

Code Block

language	xml

<Annotation Term="Capabilities.UpdateRestrictions">
  <Record>
    <PropertyValue Property="NonUpdatableNavigationProperties">
      <Collection>
        <NavigationPropertyPath>Supplier</NavigationPropertyPath>
        <NavigationPropertyPath>Category</NavigationPropertyPath>
      </Collection>
    </PropertyValue>
  </Record>
</Annotation>

name - field of type 'string' with value "NavigationPropertyPath" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

PropertyPath

dynamic

Code Block

language	xml

<Annotation Term="Capabilities.UpdateRestrictions">
  <Record>
    <PropertyValue Property="NonUpdatableProperties">
      <Collection>
        <PropertyPath>CreatedAt</PropertyPath>
        <PropertyPath>ChangedAt</PropertyPath>
      </Collection>
    </PropertyValue>
  </Record>
</Annotation>

name - field of type 'string' with value "PropertyPath" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

And

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<And>
<Path>BooleanProperty1</Path>
<Path>BooleanProperty2</Path>
</And>
</Annotation>

name - field of type 'string' with value "And" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Or

dynamic

Code Block

language	xml

Annotation Term="Core.Description">
<Or>
<Path>BooleanProperty1</Path>
<Path>BooleanProperty2</Path>
</Or>
</Annotation>

name - field of type 'string' with value "Or" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Eq

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Eq>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Eq>
</Annotation>

name - field of type 'string' with value "Eq" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Ne

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Ne>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Ne>
</Annotation>

name - field of type 'string' with value "Ne" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Gt

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Gt>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Gt>
</Annotation>

name - field of type 'string' with value "Gt" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Ge

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Ge>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Ge>
</Annotation>

name - field of type 'string' with value "Ge" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Lt

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Lt>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Lt>
</Annotation>

name - field of type 'string' with value "Lt" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Le

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Le>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Le>
</Annotation>

name - field of type 'string' with value "Le" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Not

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Not>
<Path>SomeProperty</Path>
</Not>
</Annotation>

name - field of type 'string' with value "Not" to distinguish expression.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Apply

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Apply Function="odata.concat">
<String>Product:</String>
<!-- AllDataTypesEntity/String -->
<Path>SomeProperty1</Path>
<String>(</String>
<!-- AllDataTypesEntity/Int16 -->
<Path>SomeProperty2</Path>
<String>)</String>
</Apply>
</Annotation>

name - field of type 'string' with value "Apply" to distinguish expression.
function - field of type 'string' with value of a function name
parameters - CDAP record of parameters with fields that correspond to parameter expressions
annotations - CDAP record that corresponds to nested annotations

Note: the proposal is to use the parameter index as a prefix for field name to avoid conflicts. In this case, parameters record will be a record of the following fields:

String_0

-String

Path_1

-Path

String_2

-String

Path_3

-Path

String_4

-String

Cast

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Cast Type="Edm.String">
<Path>SomeProperty</Path>
</Cast>
</Annotation>

name - field of type 'string' with value "Cast" to distinguish expression.
type - field of type 'string' with value of edm type name.
maxLength - a positive integer value specifying the maximum length of a binary, stream or string value.
precision - the maximum number of significant decimal digits.
scale - a non-negative integer value specifying the maximum number of digits allowed to the right of the decimal point.
srid - for a geometry or geography property the SRID facet identifies which spatial reference system is applied to values of the property on type instances.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Collection

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Collection>
<String>Product</String>
<String>Supplier</String>
<String>Customer</String>
</Collection>
</Annotation>

name - field of type 'string' with value "Collection" to distinguish expression.
items - array of the item expressions

If

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<If>
<Path>SomeBooleanProperty</Path>
<String>Female</String>
<String>Male</String>
</If>
</Annotation>

name - field of type 'string' with value "If" to distinguish expression.
guard - CDAP record that corresponds to the 'guard' expression
then - CDAP record that corresponds to the 'then' expression
else - CDAP record that corresponds to the 'else' expression
annotations - CDAP record that corresponds to nested annotations

IsOf

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<IsOf Type="Edm.Boolean">
<Path>SomeProperty</Path>
</IsOf>
</Annotation>

name - field of type 'string' with value "IsOf" to distinguish expression.
type - field of type 'string' with value of edm type name.
maxLength - a positive integer value specifying the maximum length of a binary, stream or string value.
precision - the maximum number of significant decimal digits.
scale - a non-negative integer value specifying the maximum number of digits allowed to the right of the decimal point.
srid - for a geometry or geography property the SRID facet identifies which spatial reference system is applied to values of the property on type instances.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

LabeledElement

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<LabeledElement Name="CustomerFirstName">
<Path>SomeProperty</Path>
</LabeledElement>
</Annotation>

name - field of type 'string' with value "LabeledElement" to distinguish expression.
elementName - field of type 'string' with value of element name
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Record

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Record>
<Annotation Term="Core.Description" String="Annotation on record"/>
<PropertyValue Property="GivenName" Path="SomeProperty"/>
<PropertyValue Property="Age" Path="Byte"/>
</Record>
</Annotation>

name - field of type 'string' with value "Record" to distinguish expression.
type - field of type 'string' with the qualified name of a structured type in scope.
propertyValues - CDAP record that corresponds to the property values
annotations - CDAP record that corresponds to nested annotations

UrlRef

dynamic

Code Block

language	xml

<Annotation Term="Core.LongDescription">
  <UrlRef><String>http://host/wiki/HowToUse</String></UrlRef>
</Annotation>

name - field of type 'string' with value "UrlRef" to distinguish expression.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Notes:

An edm:Annotation element MAY contain edm:Annotation elements that annotate the annotation.
The edm:Annotations element is used to apply a group of annotations to a single model element. These are annotations with External Targeting.

References

OData service

SAP Netweaver Gateway

Overview - https://blogs.sap.com/2013/01/24/a-simple-overview-on-sap-netweaver-gateway/
SAP Netweaver Gtw and OData Tutorials: https://sapyard.com/tutorials-on-odata-sap-netweaver-gateway/

Microsoft Data Factory - SAP ECC Connector

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

...

Versions Compared

Old Version 20

New Version 21

Key

Introduction

User Experience and Configuration

OData V4 Annotation Expression Mapping

Notes:

References

Plugin Type

Page Comparison

Versions Compared

Old Version 20

New Version 21

Key

User Experience and Configuration

OData V4 Annotation Expression Mapping

Notes:

References

Plugin Type