...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Table of Contents |
---|
Introduction
This plugin would allow SAP ECC enterprise users to specify one of the Inventory Management data sources as source in Cloud Data Fusion. Typical pipeline scenario would include specifying one of the data sources as source (under Sources plugins) and BigQuery as sink. For the sake of simplicity, this document will cover one data source (Material Movements from Inventory Management - 2LIS_03_BF).
User Experience and Configuration
- SAP ECC Setup
SAP ECC users log into SAP Gateway Service Builder (segw) to create and expose the data source as SAP OData service. SAP has published following articles around this step:
User creates OData service for the data source (2LIS_03_BF). Output is RESTful OData service that can be accessed by Cloud Data Fusion.
User can also use SAP Netweaver Gateway client to test the service (with HTTP request and response)
- Cloud Data Fusion - Source configuration
- Source Data Types Mapping
User would log into their CDF instance and click on “Hub”.
Under “Plugins” section, user should be able to find “SAP ECC Source”.
This integration would provide a single JAR files for SAP ECC data sources. User would upload the JAR file and this would deploy the plugin using the JAR file.
Once uploaded, the user is prompted to provide following configuration information. Please note that all the fields below should be macro enabled.
...
Data type mismatch - OData output has data types (section 6) different than the data types available in BigQuery. Some sort of data type conversion needs to happen before the extracted data can be ingested into BigQuery. Below is the suggested mapping of OData data types to CDAP schema data types:
...
Field Level Lineage (FLL): Field level lineage should be available for all the sources and sinks.
OData Service API
Service Metadata
...
Document
Describes the structure of all resources in the service.
Note: service metadata document has no JSON representation.
GET: http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3_SRV/$metadata
...
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
<?xml version="1.0" encoding="UTF-8"?> <edmx:Edmx xmlns:edmx="http://schemas.microsoft.com/ado/2007/06/edmx" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:sap="http://www.sap.com/Protocols/SAPData" Version="1.0"> <edmx:DataServices m:DataServiceVersion="2.0"> <Schema xmlns="http://schemas.microsoft.com/ado/2008/09/edm" Namespace="ZGW100_XX_S2_SRV" xml:lang="en" sap:schema-version="1"> <EntityType Name="SalesOrder" sap:content-version="1"> <Key> <PropertyRef Name="SoId" /> </Key> <Property Name="SoId" Type="Edm.String" Nullable="false" MaxLength="10" sap:label="Sales Order ID" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="Note" Type="Edm.String" Nullable="false" MaxLength="255" sap:label="Description" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="BuyerId" Type="Edm.String" Nullable="false" MaxLength="10" sap:label="Business Partner ID" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="BuyerName" Type="Edm.String" Nullable="false" MaxLength="80" sap:label="Company" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="CurrencyCode" Type="Edm.String" Nullable="false" MaxLength="5" sap:label="Currency Code" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" sap:semantics="currency-code" /> <Property Name="GrossAmount" Type="Edm.Decimal" Nullable="false" Precision="16" Scale="3" sap:label="Gross Amount" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="NetAmount" Type="Edm.Decimal" Nullable="false" Precision="16" Scale="3" sap:label="Net Amount" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="TaxAmount" Type="Edm.Decimal" Nullable="false" Precision="16" Scale="3" sap:label="Tax Amount" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <NavigationProperty Name="SalesOrderItems" Relationship="ZGW100_XX_S2_SRV.SalesOrderSalesOrderItems" FromRole="FromRole_SalesOrderSalesOrderItems" ToRole="ToRole_SalesOrderSalesOrderItems" /> </EntityType> <EntityType Name="SalesOrderItem" sap:content-version="1"> <Key> <PropertyRef Name="SoId" /> <PropertyRef Name="SoItemPos" /> </Key> <Property Name="SoId" Type="Edm.String" Nullable="false" MaxLength="10" sap:label="Sales Order ID" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="SoItemPos" Type="Edm.String" Nullable="false" MaxLength="10" sap:label="Item Position" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="ProductId" Type="Edm.String" Nullable="false" MaxLength="10" sap:label="Product ID" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="Note" Type="Edm.String" Nullable="false" MaxLength="255" sap:label="Description" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="CurrencyCode" Type="Edm.String" Nullable="false" MaxLength="5" sap:label="Currency Code" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" sap:semantics="currency-code" /> <Property Name="GrossAmount" Type="Edm.Decimal" Nullable="false" Precision="16" Scale="3" sap:label="Gross Amount" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="NetAmount" Type="Edm.Decimal" Nullable="false" Precision="16" Scale="3" sap:label="Net Amount" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="TaxAmount" Type="Edm.Decimal" Nullable="false" Precision="16" Scale="3" sap:label="Tax Amount" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="Quantity" Type="Edm.Decimal" Nullable="false" Precision="13" Scale="3" sap:label="Quantity" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" /> <Property Name="QuantityUnit" Type="Edm.String" Nullable="false" MaxLength="3" sap:label="Unit of Measure" sap:creatable="false" sap:updatable="false" sap:sortable="false" sap:filterable="false" sap:semantics="unit-of-measure" /> </EntityType> <Association Name="SalesOrderSalesOrderItems" sap:content-version="1"> <End Type="ZGW100_XX_S2_SRV.SalesOrder" Multiplicity="1" Role="FromRole_SalesOrderSalesOrderItems" /> <End Type="ZGW100_XX_S2_SRV.SalesOrderItem" Multiplicity="*" Role="ToRole_SalesOrderSalesOrderItems" /> <ReferentialConstraint> <Principal Role="FromRole_SalesOrderSalesOrderItems"> <PropertyRef Name="SoId" /> </Principal> <Dependent Role="ToRole_SalesOrderSalesOrderItems"> <PropertyRef Name="SoId" /> </Dependent> </ReferentialConstraint> </Association> <EntityContainer Name="ZGW100_XX_S2_SRV_Entities" m:IsDefaultEntityContainer="true" sap:supported-formats="atom json xlsx"> <EntitySet Name="SalesOrderSet" EntityType="ZGW100_XX_S2_SRV.SalesOrder" sap:creatable="false" sap:updatable="false" sap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" /> <EntitySet Name="SalesOrderCollection" EntityType="ZGW100_XX_S2_SRV.SalesOrder" sap:creatable="false" sap:updatable="false" sap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" /> <EntitySet Name="SalesOrderItemSet" EntityType="ZGW100_XX_S2_SRV.SalesOrderItem" sap:creatable="false" sap:updatable="false" sap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" /> <EntitySet Name="SalesOrderItemCollection" EntityType="ZGW100_XX_S2_SRV.SalesOrderItem" sap:creatable="false" sap:updatable="false" sap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" /> <AssociationSet Name="SalesOrderSalesOrderItems_AssocSet" Association="ZGW100_XX_S2_SRV.SalesOrderSalesOrderItems" sap:creatable="false" sap:updatable="false" sap:deletable="false" sap:content-version="1"> <End EntitySet="SalesOrderSet" Role="FromRole_SalesOrderSalesOrderItems" /> <End EntitySet="SalesOrderItemSet" Role="ToRole_SalesOrderSalesOrderItems" /> </AssociationSet> </EntityContainer> <atom:link xmlns:atom="http://www.w3.org/2005/Atom" rel="self" href="http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/$metadata" /> <atom:link xmlns:atom="http://www.w3.org/2005/Atom" rel="latest-version" href="http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/$metadata" /> </Schema> </edmx:DataServices> </edmx:Edmx> |
...
Service Document
Lists the available top-level resources.
GET: http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3S2_SRV/SalesOrderSet
Response:
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
<?xml version="1.0" encoding="UTF-8"?> <feed xmlns<app:service xmlns:app="http://www.w3.org/2007/app" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dm="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:msap="http://schemaswww.microsoftsap.com/ado/2007/08/dataservices/metadataProtocols/SAPData" xml:lang="en" xml:base="http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3S2_SRV/"> <id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3_SRV/SalesOrderSet</id> <title<app:workspace> <atom:title type="text">SalesOrderSet<>Data</atom:title> <updated>2019-08-21T10:29:14Z</updated> <author> <app:collection sap:creatable="false" sap:updatable="false" <name /> </author> <link href="SalesOrderSet" rel="self" titlesap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" href="SalesOrderSet" /> <entry> <id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3_SRV/SalesOrderSet('500000000')</id> <title <atom:title type="text">SalesOrderSet('500000000')</>SalesOrderSet</atom:title> <updated>2019-08-21T10:29:14Z</updated> <sap:member-title>SalesOrder</sap:member-title> </app:collection> <category term="ZGW100_XX_S3_SRV.SalesOrder" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> <link href="SalesOrderSet('500000000')" rel="edit" title="SalesOrder" /<app:collection sap:creatable="false" sap:updatable="false" sap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" href="SalesOrderCollection"> <content type="application/xml"> <atom:title type="text">SalesOrderCollection</atom:title> <m:properties> <d:SoId>500000000</d:SoId><sap:member-title>SalesOrder</sap:member-title> <d:BuyerId>100000000</d:BuyerId></app:collection> <app:collection <d:BuyerName>SAP</d:BuyerName> </m:properties>sap:creatable="false" sap:updatable="false" sap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" href="SalesOrderItemSet"> </content> </entry><atom:title type="text">SalesOrderItemSet</atom:title> <entry> <id>http<sap://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3_SRV/SalesOrderSet('500000001')</id>member-title>SalesOrderItem</sap:member-title> <title type="text">SalesOrderSet('500000001')</title>app:collection> <updated>2019-08-21T10:29:14Z</updated> <app:collection sap:creatable="false" <category term="ZGW100_XX_S3_SRV.SalesOrder" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> <link href="SalesOrderSet('500000001')" rel="edit" title="SalesOrder" /> <content type="application/xml"> <m:properties>sap:updatable="false" sap:deletable="false" sap:pageable="false" sap:addressable="false" sap:content-version="1" href="SalesOrderItemCollection"> <atom:title type="text">SalesOrderItemCollection</atom:title> <d:SoId>500000001</d:SoId><sap:member-title>SalesOrderItem</sap:member-title> <d:BuyerId>100000002</d:BuyerId></app:collection> <d:BuyerName>DelBont Industries</d:BuyerName></app:workspace> <atom:link </m:properties> </content> </entry> <entry> <id>http://rel="self" href="http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3S2_SRV/SalesOrderSet('500000002')</id> <title type="text">SalesOrderSet('500000002')</title>" /> <updated>2019-08-21T10:29:14Z</updated> <category term="ZGW100_XX_S3_SRV.SalesOrder" scheme<atom:link rel="latest-version" href="http://schemasvhcalnplci.microsoft.com/ado/2007/08/dataservices/schemedummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/" /> </app:service> |
GET: http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/?$format=json
Response:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
{ "d":{ <link href="SalesOrderSet('500000002')" rel="edit" title="SalesOrder" /> "EntitySets":[ <content type="application/xmlSalesOrderSet">, <m:properties>"SalesOrderCollection", "SalesOrderItemSet", <d:SoId>500000002</d:SoId> "SalesOrderItemCollection" <d:BuyerId>100000005</d:BuyerId> ] <d:BuyerName>TECUM</d:BuyerName> </m:properties> </content> </entry> ... |
...
}
} |
Requesting EntitySet
GET: http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2S3_SRV/SalesOrderCollection?$top=1&$select=BuyerNameSalesOrderSet
Response:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<?xml version="1.0" encoding="UTF-8"?> <feed xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xml:base="http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2S3_SRV/"> <id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2S3_SRV/SalesOrderCollection<SalesOrderSet</id> <title type="text">SalesOrderCollection<>SalesOrderSet</title> <updated>2019-08-21T10:4629:14Z</updated> <author> <name /> </author> <link href="SalesOrderCollectionSalesOrderSet" rel="self" title="SalesOrderCollectionSalesOrderSet" /> <entry> <id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2S3_SRV/SalesOrderCollectionSalesOrderSet('500000000')</id> <title type="text">SalesOrderCollection>SalesOrderSet('500000000')</title> <updated>2019-08-21T10:4629:14Z</updated> <category term="ZGW100_XX_S2S3_SRV.SalesOrder" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> <link href="SalesOrderCollectionSalesOrderSet('500000000')" rel="selfedit" title="SalesOrder" /> <content type="application/xml"> <m:properties> <d:SoId>500000000</d:SoId> <d:BuyerId>100000000</d:BuyerId> <d:BuyerName>SAP</d:BuyerName> </m:properties> </content> </entry> </feed> |
For more examples, please, refer: https://www.odata.org/odata-services/
SAP ECC Source Plugin
Design
The suggestion is to use Apache Olingo as Client Library.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
public class OlingoSampleApp { private static final String ROOT_URL = "http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/"; private static final String USERNAME = "DEVELOPER"; private static final String PASSWORD = "Appl1ance"; private static final String RESOURCE_PATH = "SalesOrderCollection"; public static void main(String[] params) throws Exception { ODataClient client = ODataClientFactory.getClient(); client.getConfiguration().setHttpClientFactory(new BasicAuthHttpClientFactory(USERNAME, PASSWORD)); <entry> <id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3_SRV/SalesOrderSet('500000001')</id> <title type="text">SalesOrderSet('500000001')</title> <updated>2019-08-21T10:29:14Z</updated> <category term="ZGW100_XX_S3_SRV.SalesOrder" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> <link href="SalesOrderSet('500000001')" rel="edit" title="SalesOrder" /> <content type="application/xml"> <m:properties> <d:SoId>500000001</d:SoId> <d:BuyerId>100000002</d:BuyerId> <d:BuyerName>DelBont Industries</d:BuyerName> </m:properties> URI resourcesUri = client.newURIBuilder(ROOT_URL) </content> </entry> <entry> .appendEntitySetSegment(RESOURCE_PATH) .build();<id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S3_SRV/SalesOrderSet('500000002')</id> <title type="text">SalesOrderSet('500000002')</title> ODataEntitySetIteratorRequest<ClientEntitySet, ClientEntity> request = client <updated>2019-08-21T10:29:14Z</updated> <category .getRetrieveRequestFactory() term="ZGW100_XX_S3_SRV.SalesOrder" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" /> .getEntitySetIteratorRequest(resourcesUri);<link href="SalesOrderSet('500000002')" rel="edit" title="SalesOrder" /> request.setAccept(<content type="application/atom+xml");> ODataRetrieveResponse<ClientEntitySetIterator<ClientEntitySet, ClientEntity>><m:properties> response = request.execute(); ClientEntitySetIterator<ClientEntitySet, ClientEntity> iterator = response.getBody();<d:SoId>500000002</d:SoId> <d:BuyerId>100000005</d:BuyerId> while (iterator.hasNext()) { <d:BuyerName>TECUM</d:BuyerName> ClientEntity entity = iterator.next(); // process client entity } } } |
Source Properties
...
User Facing Name
...
Widget Type
...
Description
...
Constraints
...
URL of the SAP ECC OData service. The URL must end with an external service name
...
</m:properties>
</content>
</entry>
... |
GET: http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/
...
...
...
...
...
Required
...
Path of the SAP ECC OData entity.
For example:
- "SalesOrderCollection"
- "Category(1)/Products"
Required
...
OData query options to filter the data.
For example: "$select=Name,Description&$top=10".
The plugin copies data from the combined URL:
<OData Service URL>/<Entity Set>?<Query>
For more information, see OData URL components.
...
Source Data Types Mapping
Types can be mapped as follows, according to the org.apache.olingo.commons.api.edm.EdmPrimitiveType and OData Primitive Data Types(section 6):
...
EDM primitive type
...
CDAP Schema Data Type
...
Comment
...
Edm.Boolean
...
Edm.Byte
...
Edm.DateTime
...
Datetime in the following format:
'yyyy-mm-ddThh:mm[:ss[.fffffff]]'
...
Edm.Decimal
...
Edm.Double
...
Edm.Single
...
Edm.Guid
...
Edm.Int16
...
Edm.Int32
...
Edm.Int64
...
Edm.SByte
...
Edm.String
...
Edm.Time
...
Edm.DateTimeOffset
...
Represents date and time as an Offset in minutes from GMT, with values ranging from 12:00:00 midnight, January 1, 1753 A.D. through 11:59:59 P.M, December 9999 A.D
...
SRV/SalesOrderCollection?$format=json
Response:
Code Block | ||||
---|---|---|---|---|
| ||||
{
"d":{
"results":[
{
"__metadata":{
"id":"http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection('500000000')",
"uri":"http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection('500000000')",
"type":"ZGW100_XX_S2_SRV.SalesOrder"
},
"SoId":"500000000",
"Note":"EPM DG: SO ID 0500000000 Deliver as fast as possible",
"BuyerId":"100000000",
"BuyerName":"SAP",
"CurrencyCode":"USD",
"GrossAmount":"14385.850",
"NetAmount":"12088.950",
"TaxAmount":"2296.900",
"SalesOrderItems":{
"__deferred":{
"uri":"http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection('500000000')/SalesOrderItems"
}
}
},
{
"__metadata":{
"id":"http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection('500000001')",
"uri":"http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection('500000001')",
"type":"ZGW100_XX_S2_SRV.SalesOrder"
},
"SoId":"500000001",
"Note":"EPM DG: SO ID 0500000001 Deliver as fast as possible",
"BuyerId":"100000002",
"BuyerName":"DelBont Industries",
"CurrencyCode":"USD",
"GrossAmount":"15117.760",
"NetAmount":"12704.000",
"TaxAmount":"2413.760",
"SalesOrderItems":{
"__deferred":{
"uri":"http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection('500000001')/SalesOrderItems"
}
}
},
... |
Querying Data
Response:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xml:base="http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/">
<id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection</id>
<title type="text">SalesOrderCollection</title>
<updated>2019-08-21T10:46:14Z</updated>
<author>
<name />
</author>
<link href="SalesOrderCollection" rel="self" title="SalesOrderCollection" />
<entry>
<id>http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/SalesOrderCollection('500000000')</id>
<title type="text">SalesOrderCollection('500000000')</title>
<updated>2019-08-21T10:46:14Z</updated>
<category term="ZGW100_XX_S2_SRV.SalesOrder" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
<link href="SalesOrderCollection('500000000')" rel="self" title="SalesOrder" />
<content type="application/xml">
<m:properties>
<d:BuyerName>SAP</d:BuyerName>
</m:properties>
</content>
</entry>
</feed> |
For more examples, please, refer: https://www.odata.org/odata-services/
SAP ECC Source Plugin
Design
The suggestion is to use Apache Olingo as Client Library. Both OData v2 and v4 are supported by the SAP Gateway, so it could require to use Olingo v2 and Olingo v4.
Code Block | ||||||||
---|---|---|---|---|---|---|---|---|
| ||||||||
public class OlingoSampleApp {
private static final String ROOT_URL = "http://vhcalnplci.dummy.nodomain:8000/sap/opu/odata/SAP/ZGW100_XX_S2_SRV/";
private static final String USERNAME = "DEVELOPER";
private static final String PASSWORD = "Appl1ance";
private static final String RESOURCE_PATH = "SalesOrderCollection";
public static void main(String[] params) throws Exception {
ODataClient client = ODataClientFactory.getClient();
client.getConfiguration().setHttpClientFactory(new BasicAuthHttpClientFactory(USERNAME, PASSWORD));
URI resourcesUri = client.newURIBuilder(ROOT_URL)
.appendEntitySetSegment(RESOURCE_PATH)
.build();
ODataEntitySetIteratorRequest<ClientEntitySet, ClientEntity> request = client
.getRetrieveRequestFactory()
.getEntitySetIteratorRequest(resourcesUri);
request.setAccept("application/atom+xml");
ODataRetrieveResponse<ClientEntitySetIterator<ClientEntitySet, ClientEntity>> response = request.execute();
ClientEntitySetIterator<ClientEntitySet, ClientEntity> iterator = response.getBody();
while (iterator.hasNext()) {
ClientEntity entity = iterator.next();
// process client entity
}
}
} |
Source Properties
Section | User Facing Name | Widget Type | Description | Constraints |
---|---|---|---|---|
General | Label | textbox | Label for UI. | |
Reference Name | textbox | Uniquely identified name for lineage. | ||
OData Service URL | textbox | URL of the SAP ECC OData service. The URL must end with an external service name (e.g., http://eccsvrname:8000/sap/opu/odata/sap/zgw100_dd02l_so_srv/). | Required | |
Resource Path | textbox | Path of the SAP ECC OData entity. For example:
| Required | |
Query Options | keyvalue | OData query options to filter the data. For example:
The plugin copies data from the combined URL: For more information, see OData URL components. | ||
Include Metadata Annotations | toggle | Whether the plugin should read SAP metadata annotations and include them to each record. | ||
Credentials | Username | textbox | Username for basic authentication. | |
Password | password | Password for basic authentication. | ||
Output Schema | schema | Specifies the schema of the entries. |
OData V2 Data Types Mapping
Types can be mapped as follows, according to the org.apache.olingo.odata2.api.edm.EdmSimpleType and OData Primitive Data Types(section 6):
EDM primitive type | CDAP Schema Data Type | Comment |
---|---|---|
Edm.Binary | bytes | |
Edm.Boolean | boolean | |
Edm.Byte | int | Unsigned 8-bit integer value |
Edm.DateTime | timestamp | Datetime in the following format: 'yyyy-mm-ddThh:mm[:ss[.fffffff]]' |
Edm.Decimal | decimal | |
Edm.Double | double | |
Edm.Single | float | |
Edm.Guid | string | |
Edm.Int16 | int | |
Edm.Int32 | int | |
Edm.Int64 | long | |
Edm.SByte | int | Represents a signed 8-bit integer value |
Edm.String | string | |
Edm.Time | time | Represents the time of day with values ranging from 0:00:00.x to 23:59:59.y, where x and y depend upon the precision |
Edm.DateTimeOffset | string | Represents date and time as an Offset in minutes from GMT, with values ranging from 12:00:00 midnight, January 1, 1753 A.D. through 11:59:59 P.M, December 9999 A.D Example 1: 2002-10-10T17:00:00Z Mapped to string to avoid timezone information loss. |
OData V4 Data Types Mapping
Types can be mapped as follows, according to the org.apache.olingo.commons.api.edm.EdmPrimitiveType and OData CSDL Primitive Types:
EDM primitive type | CDAP Schema Data Type | JSON | XML | Comment | |||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Edm.Binary | bytes | ||||||||||
Edm.Boolean | boolean | ||||||||||
Edm.Byte | int | Unsigned 8-bit integer value | |||||||||
Edm.Date | timestamp | Date without a time-zone offset | |||||||||
Edm.DateTimeOffset | string | Represents date and time as an Offset in minutes from GMT, with values ranging from 12:00:00 midnight, January 1, 1753 A.D. through 11:59:59 P.M, December 9999 A.D Example 1: 2002-10-10T17:00:00Z Mapped to string to avoid timezone information loss. | |||||||||
Edm.Decimal | decimal | ||||||||||
Edm.Double | double | ||||||||||
Edm.Duration | string |
|
| Signed duration in days, hours, minutes, and (sub)seconds. | |||||||
Edm.Guid | string | ||||||||||
Edm.Int16 | int | ||||||||||
Edm.Int32 | int | ||||||||||
Edm.Int64 | long | ||||||||||
Edm.SByte | int | Represents a signed 8-bit integer value | |||||||||
Edm.Single | float | ||||||||||
Edm.Stream | record |
|
| Binary data stream. Olingo client maps this data type to java.net.URI. The proposal is to map this data type to CDAP record that corresponds to the JSON representation. | |||||||
Edm.String | string | ||||||||||
Edm.TimeOfDay | time | Clock time 00:00-23:59:59.999999999999 | |||||||||
Edm.Geography | - | Abstract base type for all Geography types. | |||||||||
Edm.GeographyPoint | record |
|
| A point in a round-earth coordinate system | |||||||
Edm.GeographyLineString | record |
|
| Line string in a round-earth coordinate system | |||||||
Edm.GeographyPolygon | record |
|
| Polygon in a round-earth coordinate system | |||||||
Edm.GeographyMultiPoint | record |
|
| Collection of points in a round-earth coordinate system | |||||||
Edm.GeographyMultiLineString | record |
|
| Collection of line strings in a round-earth coordinate system | |||||||
Edm.GeographyMultiPolygon | record |
|
| Collection of polygons in a round-earth coordinate system | |||||||
Edm.GeographyCollection | record |
|
| Collection of arbitrary Geography values. Note, that JSON representation contains type control information (`@odata.type`). Otherwise, Olingo client reports invalid value type for `Edm.GeometryCollection` and `Edm.GeographyCollection` JSON values. Since metadata does not contain component info, the proposal is to map this data type to CDAP record with the following fields:
| |||||||
Edm.Geometry | - | Abstract base type for all Geometry types | |||||||||
Edm.GeometryPoint | record | Similar to Edm.GeographyPoint | Similar to Edm.GeographyPoint | Point in a flat-earth coordinate system | |||||||
Edm.GeometryLineString | record | Similar to Edm.GeographyLineString | Similar to Edm.GeographyLineString | Line string in a flat-earth coordinate system | |||||||
Edm.GeometryPolygon | record | Similar to Edm.GeographyPolygon | Similar to Edm.GeographyPolygon | Polygon in a flat-earth coordinate system | |||||||
Edm.GeometryMultiPoint | record | Similar to Edm.GeographyMultiPoint | Similar to Edm.GeographyMultiPoint | Collection of points in a flat-earth coordinate system | |||||||
Edm.GeometryMultiLineString | record | Similar to Edm.GeographyMultiLineString | Similar to Edm.GeographyMultiLineString | Collection of line strings in a flat-earth coordinate system | |||||||
Edm.GeometryMultiPolygon | record | Similar to Edm.GeographyMultiPolygon | Similar to Edm.GeographyMultiPolygon | Collection of polygons in a flat-earth coordinate system | |||||||
Edm.GeometryCollection | record | Similar to Edm.GeographyCollection | Similar to Edm.GeographyCollection | Collection of arbitrary Geometry values |
OData V4 Metadata Annotations Mapping
An annotation applies a term to a model element and defines how to calculate a value for the term application. The value of an annotation is specified as an annotation expression, which is either a constant expression representing a constant value or a dynamic expression.
"Include Metadata Annotations" configuration property indicates whether the plugin should read metadata annotations and include them to each record. In this case, each property will be mapped to a CDAP 'record' with exactly two fields "value" and "metadata-annotations". OData V4 metadata annotations are mapped to a record of the following fields:
- term - a simple identifier, such as "UI.DisplayName" or "Core.Description", etc.
- qualifier - a term can be applied multiple times to the same model element by providing a qualifier to distinguish the annotations.
- expression - CDAP record that corresponds to a constant expression or a dynamic expression. Please, refer the table below for expressions mapping.
- annotations - CDAP record that corresponds to nested annotations. Please, refer examples below for nested annotations mapping.
OData V4 Annotation Expression Mapping
EDM expression | Type | Example | CDAP record schema | |||||
---|---|---|---|---|---|---|---|---|
Binary | constant |
|
| |||||
Bool | constant |
|
| |||||
Date | constant |
|
| |||||
DateTimeOffset | constant |
|
| |||||
Decimal | constant |
|
| |||||
Duration | constant |
|
| |||||
EnumMember | constant |
|
| |||||
Float | constant |
|
| |||||
Guid | constant |
|
| |||||
Int | constant |
|
| |||||
String | constant |
|
| |||||
TimeOfDay | constant |
|
| |||||
Path | dynamic |
|
| |||||
AnnotationPath | dynamic |
|
| |||||
LabeledElementReference | dynamic |
|
| |||||
Null | dynamic |
|
| |||||
NavigationPropertyPath | dynamic |
|
| |||||
PropertyPath | dynamic |
|
| |||||
And | dynamic |
|
| |||||
Or | dynamic |
|
| |||||
Eq | dynamic |
|
| |||||
Ne | dynamic |
|
| |||||
Gt | dynamic |
|
| |||||
Ge | dynamic |
|
| |||||
Lt | dynamic |
|
| |||||
Le | dynamic |
|
| |||||
Not | dynamic |
|
| |||||
Apply | dynamic |
|
Note: the proposal is to use the parameter index as a prefix for field name to avoid conflicts. In this case, parameters record will be a record of the following fields:
| |||||
Cast | dynamic |
|
| |||||
Collection | dynamic |
|
| |||||
If | dynamic |
|
| |||||
IsOf | dynamic |
|
| |||||
LabeledElement | dynamic |
|
| |||||
Record | dynamic |
|
| |||||
UrlRef | dynamic |
|
|
Notes
- An edm:Annotation element MAY contain edm:Annotation elements that annotate the annotation.
- The edm:Annotations element is used to apply a group of annotations to a single model element. These are annotations with External Targeting.
Complex Types Mapping
Both OData V2 and V4 support complex types. Complex types are keyless nominal structured types. The lack of a key means that instances of complex types cannot be referenced, created, updated or deleted independently of an entity type. Complex types allow entity models to group properties into common structures.
The example below demonstrates the definition of several complex types in the service metadata document:
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
<Schema Namespace="TestService" xmlns="http://docs.oasis-open.org/odata/ns/edm" Alias="self">
<ComplexType Name="City">
<Property Name="CountryRegion" Type="Edm.String" Nullable="false"/>
<Property Name="Name" Type="Edm.String" Nullable="false"/>
<Property Name="Region" Type="Edm.String" Nullable="false"/>
</ComplexType>
<ComplexType Name="Location" OpenType="true">
<Property Name="Address" Type="Edm.String" Nullable="false"/>
<Property Name="City" Type="TestService.City" Nullable="false"/>
</ComplexType>
<ComplexType Name="HomeAddress" BaseType="TestService.Location">
<Property Name="FamilyName" Type="Edm.String"/>
</ComplexType>
...
<EntityType Name="AllDataTypesEntity">
...
<Property Name="Address" Type="TestService.Location" Nullable="false"/>
<Property Name="HomeAddress" Type="TestService.HomeAddress" Nullable="false"/>
<Property Name="Size" Type="TestService.Size" Nullable="false"/>
</EntityType>
...
</Schema>
</edmx:DataServices>
</edmx:Edmx> |
Properties of these types are mapped to a record with corresponding fields and each of them is mapped according to it's Edm type: OData V2 Data Types Mapping, OData V4 Data Types Mapping.
OData V4 EnumType and TypeDefinition
OData V4 supports enumeration types and type definitions, that are defined as follows:
Code Block | ||
---|---|---|
| ||
<?xml version="1.0" encoding="utf-8"?>
<edmx:Edmx Version="4.0" xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx">
<edmx:DataServices>
<Schema Namespace="TestService" xmlns="http://docs.oasis-open.org/odata/ns/edm" Alias="self">
<EnumType Name="PersonGender">
<Member Name="Male" Value="0"/>
<Member Name="Female" Value="1"/>
<Member Name="Unknown" Value="2">
<Annotation Term="Core.Description" String="Shipped with highest priority" />
</Member>
</EnumType>
<TypeDefinition Name="Length" UnderlyingType="Edm.Int32">
<Annotation Term="Org.OData.Measures.V1.Unit" String="Centimeters" />
</TypeDefinition>
<TypeDefinition Name="Weight" UnderlyingType="Edm.Int32">
<Annotation Term="Org.OData.Measures.V1.Unit" String="Kilograms" />
</TypeDefinition>
...
<EntityType Name="AllDataTypesEntity">
...
<Property Name="Gender" Type="self.PersonGender" Nullable="false"/>
<Property Name="Height" Type="self.Length" />
<Property Name="Weight" Type="self.Weight" />
</EntityType>
...
</Schema>
</edmx:DataServices>
</edmx:Edmx> |
Properties of the enum types are mapped to string. Properties of the TypeDefinition are mapped according to it's underlying Edm type: OData V2 Data Types Mapping, OData V4 Data Types Mapping.
References
OData service
SAP Netweaver Gateway
Overview - https://blogs.sap.com/2013/01/24/a-simple-overview-on-sap-netweaver-gateway/
SAP Netweaver Gtw and OData Tutorials: https://sapyard.com/tutorials-on-odata-sap-netweaver-gateway/
Microsoft Data Factory - SAP ECC Connector
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
...