...

Table of Contents

Introduction

This plugin would allow SAP ECC enterprise users to specify one of the Inventory Management data sources as source in Cloud Data Fusion. Typical pipeline scenario would include specifying one of the data sources as source (under Sources plugins) and BigQuery as sink. For the sake of simplicity, this document will cover one data source (Material Movements from Inventory Management - 2LIS_03_BF).

User Experience and Configuration

SAP ECC Setup

SAP ECC users log into SAP Gateway Service Builder (segw) to create and expose the data source as SAP OData service. SAP has published following articles around this step:

User creates OData service for the data source (2LIS_03_BF). Output is RESTful OData service that can be accessed by Cloud Data Fusion.
User can also use SAP Netweaver Gateway client to test the service (with HTTP request and response)

Cloud Data Fusion - Source configuration

1. Source Data Types Mapping
2. User would log into their CDF instance and click on “Hub”.
3. Under “Plugins” section, user should be able to find “SAP ECC Source”.
4. This integration would provide a single JAR files for SAP ECC data sources. User would upload the JAR file and this would deploy the plugin using the JAR file.
5. Once uploaded, the user is prompted to provide following configuration information. Please note that all the fields below should be macro enabled.

...

Data type mismatch - OData output has data types (section 6) different than the data types available in BigQuery. Some sort of data type conversion needs to happen before the extracted data can be ingested into BigQuery. Below is the suggested mapping of OData data types to CDAP schema data types:

...

Section	User Facing Name	Widget Type	Description	Constraints
General	Label	textbox	Label for UI.
	Reference Name	textbox	Uniquely identified name for lineage.
	OData Service URL	textbox	URL of the SAP ECC OData service. The URL must end with an external service name (e.g., http://eccsvrname:8000/sap/opu/odata/sap/zgw100_dd02l_so_srv/).	Required
	Resource Path	textbox	Path of the SAP ECC OData entity. For example: "SalesOrderCollection" "Category(1)/Products"	Required
	Query Options	keyvalue	OData query options to filter the data. For example: "$select=Name,Description" "$top=10". The plugin copies data from the combined URL: <OData Service URL>/<Entity Set>?<Query Option 1>&<Query Option 2> For more information, see OData URL components.
	Include Metadata Annotations	toggle	Whether the plugin should read SAP metadata annotations and include them to each record.
Credentials	Username	textbox	Username for basic authentication.
Credentials	Password	password	Password for basic authentication.
	Output Schema	schema	Specifies the schema of the entries.

...

OData V2 Data Types Mapping

Types can be mapped as follows, according to the org.apache.olingo.odata2.api.edm.EdmSimpleType and OData Primitive Data Types(section 6):

...

Types can be mapped as follows, according to the org.apache.olingo.commons.api.edm.EdmPrimitiveType and OData CSDL Primitive Types:

...

OData V4 Annotation Expression Mapping

EDM expression

Type

Example

CDAP record schema

Binary

constant

Code Block

language	xml

<Annotation Term="org.example.display.Thumbnail">
  <Binary>T0RhdGE</Binary>
</Annotation>

name - field of type 'string' with value "Binary" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Bool

constant

Code Block

language	xml

<Annotation Term="org.example.display.ReadOnly">
  <Bool>true</Bool>
</Annotation>

name - field of type 'string' with value "Bool" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Date

constant

Code Block

language	xml

<Annotation Term="org.example.vCard.birthDay">
  <Date>2000-01-01</Date>
</Annotation>

name - field of type 'string' with value "Date" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

DateTimeOffset

constant

Code Block

language	xml

<Annotation Term="org.example.display.LastUpdated">
  <DateTimeOffset>2000-01-01T16:00:00.000-09:00</DateTimeOffset>
</Annotation>

name - field of type 'string' with value "DateTimeOffset" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Decimal

constant

Code Block

language	xml

<Annotation Term="org.example.display.Width">
  <Decimal>3.14</Decimal>
</Annotation>

name - field of type 'string' with value "Decimal" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Duration

constant

Code Block

language	xml

<Annotation Term="org.example.task.duration">
  <Duration>P11DT23H59M59.999999999999S</Duration>
</Annotation>

name - field of type 'string' with value "Duration" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

EnumMember

constant

Code Block

language	xml

<Annotation Term="org.example.HasPattern">
  <EnumMember>org.example.Pattern/Red</EnumMember>
</Annotation>

name - field of type 'string' with value "EnumMember" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Float

constant

Code Block

language	xml

<Annotation Term="org.example.display.Width">
  <Float>3.14</Float>
</Annotation>

name - field of type 'string' with value "Float" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Guid

constant

Code Block

language	xml

<Annotation Term="org.example.display.Id">
  <Guid>21EC2020-3AEA-1069-A2DD-08002B30309D</Guid>
</Annotation>

name - field of type 'string' with value "Guid" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Int

constant

Code Block

language	xml

<Annotation Term="org.example.display.Width">
  <Int>42</Int>
</Annotation>

name - field of type 'string' with value "Int" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

String

constant

Code Block

language	xml

<Annotation Term="org.example.display.DisplayName">
  <String>Product Catalog</String>
</Annotation>

name - field of type 'string' with value "String" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

TimeOfDay

constant

Code Block

language	xml

<Annotation Term="org.example.display.EndTime">
  <TimeOfDay>21:45:00</TimeOfDay>
</Annotation>

name - field of type 'string' with value "TimeOfDay" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Path

dynamic

Code Block

language	xml

<Annotation Term="org.example.display.DisplayName">
  <Path>@vCard.Address#work/FullName</Path>
</Annotation>

name - field of type 'string' with value "Path" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

AnnotationPath

dynamic

Code Block

language	xml

<Annotation Term="UI.CollectionFacet" Qualifier="Contacts">
  <Collection>
    <AnnotationPath>Supplier/@Communication.Contact</AnnotationPath>
    <AnnotationPath>Customer/@Communication.Contact</AnnotationPath>
  </Collection>
</Annotation>

name - field of type 'string' with value "AnnotationPath" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

LabeledElementReference

dynamic

Code Block

language	xml

<Annotation Term="org.example.display.DisplayName">
  <LabeledElementReference>Model.CustomerFirstName</LabeledElementReference>
</Annotation>

name - field of type 'string' with value "LabeledElementReference" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

Null

dynamic

Code Block

language	xml

<Annotation Term="@UI.Address">
  <Null>
    <Annotation Term="self.Reason" String="Private" />
  </Null>
</Annotation>

name - field of type 'string' with value "Null" to distinguish expression.
value - field of type 'string' with null value.
annotations - CDAP record that corresponds to nested annotations, since the edm:Null element MAY contain edm:Annotation elements.

NavigationPropertyPath

dynamic

Code Block

language	xml

<Annotation Term="Capabilities.UpdateRestrictions">
  <Record>
    <PropertyValue Property="NonUpdatableNavigationProperties">
      <Collection>
        <NavigationPropertyPath>Supplier</NavigationPropertyPath>
        <NavigationPropertyPath>Category</NavigationPropertyPath>
      </Collection>
    </PropertyValue>
  </Record>
</Annotation>

name - field of type 'string' with value "NavigationPropertyPath" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

PropertyPath

dynamic

Code Block

language	xml

<Annotation Term="Capabilities.UpdateRestrictions">
  <Record>
    <PropertyValue Property="NonUpdatableProperties">
      <Collection>
        <PropertyPath>CreatedAt</PropertyPath>
        <PropertyPath>ChangedAt</PropertyPath>
      </Collection>
    </PropertyValue>
  </Record>
</Annotation>

name - field of type 'string' with value "PropertyPath" to distinguish expression.
value - field of type 'string' with a string representation of expression value as declared in the service metadata document.

And

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<And>
<Path>BooleanProperty1</Path>
<Path>BooleanProperty2</Path>
</And>
</Annotation>

name - field of type 'string' with value "And" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Or

dynamic

Code Block

language	xml

Annotation Term="Core.Description">
<Or>
<Path>BooleanProperty1</Path>
<Path>BooleanProperty2</Path>
</Or>
</Annotation>

name - field of type 'string' with value "Or" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Eq

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Eq>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Eq>
</Annotation>

name - field of type 'string' with value "Eq" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Ne

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Ne>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Ne>
</Annotation>

name - field of type 'string' with value "Ne" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Gt

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Gt>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Gt>
</Annotation>

name - field of type 'string' with value "Gt" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Ge

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Ge>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Ge>
</Annotation>

name - field of type 'string' with value "Ge" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Lt

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Lt>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Lt>
</Annotation>

name - field of type 'string' with value "Lt" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Le

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Le>
<Path>SomeProperty1</Path>
<Path>SomeProperty2</Path>
</Le>
</Annotation>

name - field of type 'string' with value "Le" to distinguish expression.
left - CDAP record that corresponds to the left expression
right - CDAP record that corresponds to the right expression
annotations - CDAP record that corresponds to nested annotations

Not

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Not>
<Path>SomeProperty</Path>
</Not>
</Annotation>

name - field of type 'string' with value "Not" to distinguish expression.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Apply

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Apply Function="odata.concat">
<String>Product:</String>
<!-- AllDataTypesEntity/String -->
<Path>SomeProperty1</Path>
<String>(</String>
<!-- AllDataTypesEntity/Int16 -->
<Path>SomeProperty2</Path>
<String>)</String>
</Apply>
</Annotation>

name - field of type 'string' with value "Apply" to distinguish expression.
function - field of type 'string' with value of a function name
parameters - CDAP record of parameters with fields that correspond to parameter expressions
annotations - CDAP record that corresponds to nested annotations

Note: the proposal is to use the parameter index as a prefix for field name to avoid conflicts. In this case, parameters record will be a record of the following fields:

String_0

-String

Path_1

-Path

String_2

-String

Path_3

-Path

String_4

-String

Cast

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Cast Type="Edm.String">
<Path>SomeProperty</Path>
</Cast>
</Annotation>

name - field of type 'string' with value "Cast" to distinguish expression.
type - field of type 'string' with value of edm type name.
maxLength - a positive integer value specifying the maximum length of a binary, stream or string value.
precision - the maximum number of significant decimal digits.
scale - a non-negative integer value specifying the maximum number of digits allowed to the right of the decimal point.
srid - for a geometry or geography property the SRID facet identifies which spatial reference system is applied to values of the property on type instances.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Collection

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Collection>
<String>Product</String>
<String>Supplier</String>
<String>Customer</String>
</Collection>
</Annotation>

name - field of type 'string' with value "Collection" to distinguish expression.
items - array of the item expressions

If

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<If>
<Path>SomeBooleanProperty</Path>
<String>Female</String>
<String>Male</String>
</If>
</Annotation>

name - field of type 'string' with value "If" to distinguish expression.
guard - CDAP record that corresponds to the 'guard' expression
then - CDAP record that corresponds to the 'then' expression
else - CDAP record that corresponds to the 'else' expression
annotations - CDAP record that corresponds to nested annotations

IsOf

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<IsOf Type="Edm.Boolean">
<Path>SomeProperty</Path>
</IsOf>
</Annotation>

name - field of type 'string' with value "IsOf" to distinguish expression.
type - field of type 'string' with value of edm type name.
maxLength - a positive integer value specifying the maximum length of a binary, stream or string value.
precision - the maximum number of significant decimal digits.
scale - a non-negative integer value specifying the maximum number of digits allowed to the right of the decimal point.
srid - for a geometry or geography property the SRID facet identifies which spatial reference system is applied to values of the property on type instances.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

LabeledElement

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<LabeledElement Name="CustomerFirstName">
<Path>SomeProperty</Path>
</LabeledElement>
</Annotation>

name - field of type 'string' with value "LabeledElement" to distinguish expression.
elementName - field of type 'string' with value of element name
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Record

dynamic

Code Block

language	xml

<Annotation Term="Core.Description">
<Record>
<Annotation Term="Core.Description" String="Annotation on record"/>
<PropertyValue Property="GivenName" Path="SomeProperty"/>
<PropertyValue Property="Age" Path="Byte"/>
</Record>
</Annotation>

name - field of type 'string' with value "Record" to distinguish expression.
type - field of type 'string' with the qualified name of a structured type in scope.
propertyValues - CDAP record that corresponds to the property values
annotations - CDAP record that corresponds to nested annotations

UrlRef

dynamic

Code Block

language	xml

<Annotation Term="Core.LongDescription">
  <UrlRef><String>http://host/wiki/HowToUse</String></UrlRef>
</Annotation>

name - field of type 'string' with value "UrlRef" to distinguish expression.
value - CDAP record that corresponds to the value expression
annotations - CDAP record that corresponds to nested annotations

Notes

...

An edm:Annotation element MAY contain edm:Annotation elements that annotate the annotation.
The edm:Annotations element is used to apply a group of annotations to a single model element. These are annotations with External Targetingare annotations with External Targeting.

Complex Types Mapping

Both OData V2 and V4 support complex types. Complex types are keyless nominal structured types. The lack of a key means that instances of complex types cannot be referenced, created, updated or deleted independently of an entity type. Complex types allow entity models to group properties into common structures.

The example below demonstrates the definition of several complex types in the service metadata document:

Code Block

language	xml

<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
  <Schema Namespace="TestService" xmlns="http://docs.oasis-open.org/odata/ns/edm" Alias="self">
    <ComplexType Name="City">
      <Property Name="CountryRegion" Type="Edm.String" Nullable="false"/>
      <Property Name="Name" Type="Edm.String" Nullable="false"/>
      <Property Name="Region" Type="Edm.String" Nullable="false"/>
    </ComplexType>
    <ComplexType Name="Location" OpenType="true">
      <Property Name="Address" Type="Edm.String" Nullable="false"/>
      <Property Name="City" Type="TestService.City" Nullable="false"/>
    </ComplexType>
    <ComplexType Name="HomeAddress" BaseType="TestService.Location">
      <Property Name="FamilyName" Type="Edm.String"/>
    </ComplexType>
    ...
    <EntityType Name="AllDataTypesEntity">
      ...
      <Property Name="Address" Type="TestService.Location" Nullable="false"/>
      <Property Name="HomeAddress" Type="TestService.HomeAddress" Nullable="false"/>
      <Property Name="Size" Type="TestService.Size" Nullable="false"/>
    </EntityType>
    ...
    </Schema>
</edmx:DataServices>
</edmx:Edmx>

Properties of these types are mapped to a record with corresponding fields and each of them is mapped according to it's Edm type: OData V2 Data Types Mapping, OData V4 Data Types Mapping.

OData V4 EnumType and TypeDefinition

OData V4 supports enumeration types and type definitions, that are defined as follows:

Code Block

language	xml

<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
  <Schema Namespace="TestService" xmlns="http://docs.oasis-open.org/odata/ns/edm" Alias="self">
    <EnumType Name="PersonGender">
      <Member Name="Male" Value="0"/>
      <Member Name="Female" Value="1"/>
      <Member Name="Unknown" Value="2">
        <Annotation Term="Core.Description" String="Shipped with highest priority" />
      </Member>
    </EnumType>
    <TypeDefinition Name="Length" UnderlyingType="Edm.Int32">
      <Annotation Term="Org.OData.Measures.V1.Unit" String="Centimeters" />
    </TypeDefinition>
    <TypeDefinition Name="Weight" UnderlyingType="Edm.Int32">
      <Annotation Term="Org.OData.Measures.V1.Unit" String="Kilograms" />
    </TypeDefinition>
    ...
    <EntityType Name="AllDataTypesEntity">
      ...
      <Property Name="Gender" Type="self.PersonGender" Nullable="false"/>
      <Property Name="Height" Type="self.Length" />
      <Property Name="Weight" Type="self.Weight" />
    </EntityType>
    ...
    </Schema>
</edmx:DataServices>
</edmx:Edmx>

Properties of the enum types are mapped to string. Properties of the TypeDefinition are mapped according to it's underlying Edm type: OData V2 Data Types Mapping, OData V4 Data Types Mapping.

References

OData service

SAP Netweaver Gateway

Overview - https://blogs.sap.com/2013/01/24/a-simple-overview-on-sap-netweaver-gateway/
SAP Netweaver Gtw and OData Tutorials: https://sapyard.com/tutorials-on-odata-sap-netweaver-gateway/

Microsoft Data Factory - SAP ECC Connector

Plugin Type

Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute

...

Versions Compared

Old Version 20

New Version Current

Key

Introduction

User Experience and Configuration

OData V2 Data Types Mapping

OData V4 Annotation Expression Mapping

Notes

An edm:Annotation element MAY contain edm:Annotation elements that annotate the annotation.
The edm:Annotations element is used to apply a group of annotations to a single model element. These are annotations with External Targetingare annotations with External Targeting.

Complex Types Mapping

OData V4 EnumType and TypeDefinition

References

Plugin Type

Page Comparison

Versions Compared

Old Version 20

New Version Current

Key

Introduction

User Experience and Configuration

OData V2 Data Types Mapping

OData V4 Annotation Expression Mapping

Notes

An edm:Annotation element MAY contain edm:Annotation elements that annotate the annotation.The edm:Annotations element is used to apply a group of annotations to a single model element. These are annotations with External Targetingare annotations with External Targeting.

Complex Types Mapping

OData V4 EnumType and TypeDefinition

References

Plugin Type

An edm:Annotation element MAY contain edm:Annotation elements that annotate the annotation.
The edm:Annotations element is used to apply a group of annotations to a single model element. These are annotations with External Targetingare annotations with External Targeting.