Introduction
This plugin would allow SAP ECC enterprise users to specify one of the Inventory Management data sources as source in Cloud Data Fusion. Typical pipeline scenario would include specifying one of the data sources as source (under Sources plugins) and BigQuery as sink. For the sake of simplicity, this document will cover one data source (Material Movements from Inventory Management - 2LIS_03_BF).
User Experience and Configuration
- SAP ECC Setup
SAP ECC users log into SAP Gateway Service Builder (segw) to create and expose the data source as SAP OData service. SAP has published following articles around this step:
User creates OData service for the data source (2LIS_03_BF). Output is RESTful OData service that can be accessed by Cloud Data Fusion.
User can also use SAP Netweaver Gateway client to test the service (with HTTP request and response)
- Cloud Data Fusion - Source configuration
User would log into their CDF instance and click on “Hub”.
Under “Plugins” section, user should be able to find “SAP ECC Source”.
This integration would provide a single JAR files for SAP ECC data sources. User would upload the JAR file and this would deploy the plugin using the JAR file.
Once uploaded, the user is prompted to provide following configuration information. Please note that all the fields below should be macro enabled.
Label | Label Description | User Widget |
---|---|---|
Name | Name of the source | Textbox |
Description | Description of the service and the data source it is going to connect to (e.g. 2LIS_03_BF) | Textbox |
OData service URL | OData service URL exposed via SAP ECC | Textbox |
User name | Login user name for authenticating API call | Textbox |
Password | Password for authenticating API call | Textbox |
The same dialog will have “Test connection” button to allow users to validate a validation API call. If failed, the error message will be presented to the user in red next to “Test connection” button.
Once connection has been established successfully to this OData service, it can be used as a source for creating data pipelines using Cloud Data Fusion.
- Cloud Data Fusion - Sink setup (e.g. BigQuery for this specific example)
User would create a BigQuery table inside a dataset where the data extracted through OData service API will be ingested.
User needs to ensure that the table schema includes data types that have been mapped as part of this integration (see below for integration considerations section for more details)
Integration considerations
Data type mismatch - OData output has data types (section 6) different than the data types available in BigQuery. Some sort of data type conversion needs to happen before the extracted data can be ingested into BigQuery. Below is the suggested mapping of OData data types to CDAP schema data types:
OData data type | CDAP schema data type |
---|---|
Edm.Binary | Schema.Type.String |
Edm.Boolean | Schema.Type.Bool |
Edm.Byte | Schema.Type.String |
Edm.DateTime | Schema.Type.DateTime |
Edm.Decimal | Schema.Type.Float |
Edm.Double | Schema.Type.Double |
Edm.Single | Schema.Type.Float |
Edm.Guid | Schema.Type.String |
Edm.Int16 | Schema.Type.Int |
Edm.Int32 | Schema.Type.Int |
Edm.Int64 | Schema.Type.Long |
Edm.SByte | Schema.Type.Bytes |
Edm.String | Schema.Type.String |
Edm.Time | Schema.Type.Time |
Edm.DateTimeOffset | Schema.Type.Timestamp |
Field Level Lineage (FLL): Field level lineage should be available for all the sources and sinks.
References
OData service
SAP Netweaver Gateway
Overview - https://blogs.sap.com/2013/01/24/a-simple-overview-on-sap-netweaver-gateway/
SAP Netweaver Gtw and OData Tutorials: https://sapyard.com/tutorials-on-odata-sap-netweaver-gateway/
Microsoft Data Factory - SAP ECC Connector
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute