Integrating SAP BAPI Transactions with Pipeline Processing

This article is contributed. See the original author and article here.

Introduction

One of the ways that BizTalk can access SAP servers is by using Business Application Programming Interfaces (BAPIs), as documented in Operations on BAPIs in SAP. SAP introduced BAPIs as an object-oriented way to structure business data and processes. It is assumed that the reader has some familiarity with how to use this functionality in BizTalk.

In this article, we conclude our series on BAPI transactions by presenting, step by step, another implementation of the sales orders received in batches and created in SAP by using BAPI transactions. The following was covered in past articles:

In SAP BAPI Transactions Walkthrough, BAPI transactions are received separately but as part of the same Logical Unit of Work (LUW) on the SAP server.

In Debatching SAP BAPI Transactions and Scatter-Gather Pattern with SAP BAPI Transactions, BAPI transactions are received in batches which are then disassembled by the default XML receive pipeline called from the orchestration.

Handling Errors in SAP BAPI Transactions adds error-handling to filter data containing fields that are not of the right formats, or that do not have expected values.

The design here is a combination of the previous concepts into a solution that integrates the powerful BizTalk pipeline features such as batch processing, XML disassembly, XML validation, and Recoverable Interchange Processing (RIP). It is also the opportunity to provide more details on topics that were briefly touched upon, in the same step by step style – the intent being to serve as the “missing documentation” for SAP BAPI transactions in BizTalk. We present how to:

Leverage XML disassembler and XML validator in the receive location in order to “front-load” the verifications previously done in the orchestration presented in Handling Errors in SAP BAPI Transactions.

Write a custom pipeline component to promote properties (such as BTS.InterchangeID) needed for the same orchestration instance to process all messages in a batch as a single SAP Logical Unit of Work (LUW).

Implement layered verifications to improve the granularity of error handling presented in Handling Errors in SAP BAPI Transactions and find errors before they get to the send adapter.

All code is provided in the archive attached to this blog post.

NOTE: Starting with BizTalk Server 2016 CU6 and BizTalk Server 2020 CU1, BAPI transactions need to be enabled explicitly, by creating the registry value HKLMSOFTWAREMicrosoftBizTalk Server3.0ConfigurationEnableBizTalkSapSessionProvider of type DWORD and setting it to ‘1’, as documented in the fix page.

1. BAPI Operations and Schemas

In the demo scenario, sales orders are received under the form of XML documents and transformed to BAPI Sales Order (BUS2032). The first pre-requisite is to download schemas and metadata for BAPI transactions by using the “Consume Adapter Service” in Visual Studio, as explained in this section of the SAP Adapter documentation.

The figure below depicts the generated schemas for the BAPI methods BUS2032.CREATEFROMDAT2 (shortened to CREATEFROMDAT2 in the rest of this article) and BAPI_TRANSACTION_COMMIT.

BAPI Schema Generation

On the receiving side of BizTalk, we consider the general case where sales orders follow a simplified schema (Order) that may not necessarily correspond to the BAPI schema. A single XML document contains multiple orders, which are debatched in the receive location.

Order Schema

The first step for using a debatching pipeline is to define an envelope schema to represent multiple orders. This is done according to the following process, initially documented in Calling a pipeline inside an orchestration:

Add a new schema file to the project.

Change the root node name to “Orders”.

In the properties window for the <schema> treeview node: (a) change the Envelope property to Yes, and (b) import the Order schema.

In the properties window for the “Orders” root node, set the Body XPath property to /*[local-name()=’Orders’ and namespace-uri()='<namespace of the schema here’

Insert a child record under the root node.

In the child record properties, set the data structure type field to Order. As a result, the Order definition will appear under Orders.

Creating the Orders Envelope Schema

The message flow with the different schemas is shown in the following figure.

Message Flow

Note: The send port is actually a Static Solicit-Response port. Responses from SAP are not shown here.

We assumed that received messages follow a simplified client-defined schema (Orders/Order) rather than CREATEFROMDAT2 directly, in order to make the demo scenario more general. Everything presented below is applicable with minor changes if batches of CREATEFROMDAT2 are received instead. A client schema makes sense in an integration context, where the source is decoupled from the destination; for instance, if the organization sending sales orders does not need to be aware that these sales orders are processed in BizTalk by creating CREATEFROMDAT2 BAPI transactions. Furthermore, BAPI schemas depend on the SAP system, and they can contain several hundreds of elements, many of them not necessarily applicable to every sales orders.

2. From Interchanges to LUWs

The key to understanding what designs can accommodate BAPI transactions is interchanges.

As explained in Run BAPI Transactions in SAP using BizTalk Server, the message context property Microsoft.Adapters.SAP.BiztalkPropertySchema.ConnectionState is used to associate a BAPI transaction to a Logical Unit of Work (LUW) on the SAP server. So, the first transaction has the property set to “OPEN”, subsequent transactions set it to “REUSE”, and the last transactions have either “CLOSE” or “ABORT”. This implies that messages are (1) grouped, and (2) ordered, to the extent that we can differentiate between first message, middle messages, and last message.

A series of transaction messages (i.e. OPEN-REUSE-REUSE-…-CLOSE) requires the same connection for outbound communication with the SAP server, and this connection is mapped to an LUW on the SAP server side by interchange id and send port id. The message properties look like:

BAPI Messages Context Properties

To put it in layman’s terms, for BAPI transactions to be in the same LUW from OPEN to CLOSE/ABORT, they need to come from the same “place” (the interchange id) and to be sent out to the same “destination” (the send port) with the same route (the “session”). The use of {interchange id + send port id} is compatible with the “session” paradigm provided by the SAP Connector for Microsoft .NET (NCo) to expose the BAPI Transaction Model documented by SAP.

Common interchange id means that outbound messages are part of the same interchanges, which is achieved in the following setups:

If messages are received by the same orchestration instance. It is the design presented in SAP BAPI Transactions Walkthrough, where incoming order messages provide a promoted id property used as correlation set in order to use the same orchestration instance for a series of related orders.

If messages are debatched, as in Debatching SAP BAPI Transactions where debatching was done by an in-orchestration receive pipeline.

The current design is a combination of 1 and 2: Messages are debatched in the receive location and their BTS.InterchangeID property is used as correlation set by orchestration instances. Details are presented next.

3. Debatching Orders in the Receive Location

Let’s look at the receive location settings.

Receive Location Settings

We use a custom pipeline, shown below, in order to promote the BTS.InterchangeID property, which is otherwise not accessible from within orchestrations.

Custom Receive Pipeline

The custom pipeline uses the default XML disassembler and validator, which provides access to Recoverable Interchange Processing (RIP), feature which allows processing to go on if some messages within a batch are invalid. With standard interchange processing, the existence of any invalid message in a given interchange causes the entire interchange to be suspended even if it contains one or more valid messages. In the configuration of XML disassembler and validator, schemas and envelope schemas can be left blank; BizTalk automatically resolves the envelope and inner schemas from the messages.

The last validation stage is the custom pipeline component itself, which is based on a reusable pattern introduced in the WCF LOB Adapter SDK.

The relevant portions of the code (which may be found in the attached archive) are shown below. A class called NotifyingStream designed as a decorator of the Stream class, is used to invoke callbacks for specific events corresponding to when the stream is read. In our case, the custom pipeline component PromoteInterchangeId receives a notification when the entire stream has been read, thereby guaranteeing that all promoted properties from the message have been populated to the message context.

public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg)
{
    IBaseMessage outmsg = inmsg;
    Stream notifyingStream = new NotifyingStream(inmsg.BodyPart.Data, null, null, new NotifyingStream.LastReadOccurred(EndOfStream));
    outmsg.BodyPart.Data = notifyingStream;
    mBaseMessage = outmsg;return outmsg;
}

internal void EndOfStream()
{
    string interchangeId = (string)mBaseMessage.Context.Read(“InterchangeID”, “http://schemas.microsoft.com/BizTalk/2003/system-properties”);
    mBaseMessage.Context.Promote(“InterchangeID”, “http://schemas.microsoft.com/BizTalk/2003/system-properties”, interchangeId);
    
    // This is the place to add more promotions as needed, as explained in the blog post
}

public class NotifyingStream : Stream
{
    Stream mBaseStream;
    public delegate void LastReadOccurred();

    public NotifyingStream(Stream BaseStream, …, LastReadOccurred LastReadCallback)
    {
        mBaseStream = BaseStream;
        mLastReadCallback = LastReadCallback;
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        if (mBaseStream.Read(buffer, offset, count) == 0 && mLastReadCallback != null)
        {
            mLastReadCallback();
        }
    }
    // All other methods call the ones from Stream directly.

In addition to the promotion of BTS.InterchangeID, the custom pipeline component also sets the following properties, added for illustration purposes and for design alternatives considered later on:

– BatchId, which can be used as an alternative to using BTS.InterchangeID directly.

– BTS.SuspendMessageOnMappingFailure property, which can be used to add RIP behavior at the mapping phase in a receive port, as documented in Mapping Phase (Recoverable Interchange Processing).

– BTS.LastInterchangeMessage, which is used to indicate the last message in a batch. It is by default not set for other messages, thereby causing an exception if accessed from an expression in an orchestration. Adding the property to each message simplifies the orchestration later on.

The custom component was installed according to Deploying Custom Pipeline Components.

4. Orchestration Overview

The long-running orchestration relies on (1) the interchange id to pin messages within a batch to the same instance, and (2) Ordered Delivery of Messages to differentiate first-middle-last messages of the interchange. Processing can be decomposed into four stages, illustrated below:

Validation

LOB send/receive

If last message, commit and exit

Other messages: loop.

Orchestration Stages

As previously mentioned, a single orchestration instance processes all messages in a batch, which are correlated based on their interchange id. More info regarding correlations may be found in Sequential Convoy & Singleton Orchestration. This is illustrated in the following figure:

BTS.InterchangeID as Orchestration Correlation

An alternative to promoting and using the system property BTS.InterchangeID directly (for instance because we don’t want to change it by mistake) is to use a custom context property (BatchId) as depicted below.

BatchId User-Created Context Property

BatchId can be set to the BTS.InterchangeID value inside the custom pipeline component:

mBaseMessage.Context.Write(“BatchId”, “https://SapBAPITxClientDebatching.PropertySchema1”, interchangeId);

Discussion: Logging and Error Handling

Before presenting each orchestration stage in detail, it is worth mentioning that logging is for demo purposes, to show the different places of interest that could be adapted to “real-world” applications, where further processing could be hooked up.

At each stage, whether it is in the regular flow or in exception handling, we keep track of the order status with the following schema and expression:

Logging

A sequence of successful order creations looks like:

Status Logs for Successful Transaction Sequence

5. Orchestration Validation Stage

The receive port takes care of validating Order messages. The goal of the validation stage in the orchestration is to further validate data during and after transformation of Order to CREATEFROMDAT2, rather than letting the SAP adapter take care of it, as we saw in Handling Errors in SAP BAPI Transactions. This is not to say that one way is better than another, just that processing is distributed differently. For instance in the current design, errors caused by invalid data are caught before the data is handled in the adapter, which results in a simpler design later on, while limiting exception processing at the LOB stage to more systemic errors like connectivity. As illustrated in the figure below, in the current design, validation is layered.

Validation Phases in Orchestration

Each validation phase is explained in detail next.

5.1 Mapping Phase

First, the mapping of Order to CREATEFROMDAT2 is also a place where value verification can happen, for instance with script functoids.

Using Script Functoids for Validations

As an example, purchase and requisition dates can be enforced to be in the past. The script functoid in the provided code looks like:

public static string CheckDateIsValid1(string param1)
{
  if (string.IsNullOrWhiteSpace(param1))
  {
    return DateTime.Now.ToString(“o”);
  }

  DateTime outValue;
  bool isValid = DateTime.TryParse(param1, out outValue);
  if (!isValid || outValue > DateTime.Now)
  {
    // For demo purposes: throw an exception explicitly.
    throw new Exception(“Invalid PURCH_DATE.”);
  }
  // Make sure the datetime follows the xsd datatype
  return outValue.ToString(“o”);
}

There is a similar functoid for other data fields.

Note: Even if script functoids are identical for different fields, the method names need to be different.

Discussion: Transform at port level?

The receive port can be configured to do the mapping instead of letting it done in the orchestration. The setting is:

Transform in Receive Port

In this case, the custom pipeline component also needs to set the BTS.SuspendMessageOnMappingFailure to make sure that processing goes on if a failure happens on a single Order, as documented in Mapping Phase (Recoverable Interchange Processing) It would be done as follows:

// Set recoverability to true for the mapping phase in case the receiver wants to transform from client to BAPI in the RL.
mBaseMessage.Context.Write(“SuspendMessageOnMappingFailure”, “http://schemas.microsoft.com/BizTalk/2003/system-properties”, true);

The reason for doing the mapping in the orchestration in our demo scenario is that it allows all BAPI validations to happen in a single orchestration stage.

5.2 BAPI Validating Pipeline

We need the XML validator in order to catch additional restrictions in the BAPI schemas. For instance, dates have these restrictions:

Constraint on Date Format from Generated Schema

More restrictions are documented in SAP Adapter Binding Properties.

Similarly to Debatching SAP BAPI Transaction Requests, a pipeline is called from the orchestration. This time however, a custom pipeline is needed in order to add XML validation to the default XML Receive Pipeline, which does not have XML validation by default (BizTalk Server: Default Pipelines).

BAPI Validating Pipeline

Before using the pipeline, the following dependencies must be added to the BizTalk project:

Microsoft.XLANGs.Pipeline.dll

Microsoft.BizTalk.Pipeline.dll

The pipeline must be inside a scope of type Atomic because it is of non-serializable type. As a result, the orchestration has to be of type “Long-Running”. Inside the atomic scope, the first expression starts the pipeline execution:

ValidatingPipelineType = System.Type.GetType(“SapBAPITxClientDebatching.BAPIValidatingPipeline, SapBAPITxClientDebatching, Version=1.0.0.0, Culture=neutral, PublicKeyToken=9c30d143e3d921fe”);
PipelineOutputMessageVar = Microsoft.XLANGs.Pipeline.XLANGPipelineManager.ExecuteReceivePipeline(ValidatingPipelineType, BAPIRequest);
PipelineOutputMessageVar.MoveNext();

where PipelineOutputMessageVar is a variable of type Microsoft.XLANGs.Pipeline.ReceivePipelineOutputMessages.

The CREATEFROMDAT2 output of the pipeline is obtained with the expression:

BAPIRequestValidated = null;
PipelineOutputMessageVar.GetCurrent(BAPIRequestValidated);

Discussion: The alternative to validation at this stage is to let data format errors happen in the SAP adapter and handle them later on, as shown in Handling Errors in SAP BAPI Transactions. If we did not handle them, we would see the following on the first message:

A message sent to adapter “WCF-SAP” on send port “BAPI2032SalesOrdersSP” with URI “sap://CLIENT=800;LANG=EN;@a/…” is suspended.
 Error details: Microsoft.ServiceModel.Channels.Common.XmlReaderParsingException: 
An error occurred when trying to convert the XML string thisistoolong of RFCTYPE RFCTYPE_CHAR 
with length 4 and decimals 0 to a .NET type. Parameter/field name: SALES_ORG   
Error message: The length of the value for the field exceeds the allowed value. Value: 13  Field: 
SALES_ORG Allowed value: 4. —> Microsoft.ServiceModel.Channels.Common.XmlReaderParsingException: 
The length of the value for the field exceeds the allowed value. Value: 13  
Field: SALES_ORG Allowed value: 4
   at Microsoft.Adapters.SAP.SapMetadataUtilities.ConvertXmlStringToRfcStringNCo(
String data, RfcTypes type, Int32 singleByteLength, Int32 decimals, String fieldname, 
Encoding encoding, Encoding realPartnerEncoding, Boolean padToExactLength, SAPConnection sapconnection)

and subsequent messages would fail due to the WCF channel left in an inconsistent state in the SAP adapter, typically with exceptions in the event log like:

A message sent to adapter “WCF-SAP” on send port “BAPI2032SalesOrdersSP” with URI “sap://CLIENT=800;LANG=EN;@a/…” is suspended.
Error details: System.InvalidOperationException: The Inner Channel to use was not found for the key 
{0A73F66F-31FA-4E48-BAC5-14EAED9571D4}_{27D07099-1874-47C3-9E69-2AD1FA42DE8D};URI=sap://CLIENT=800;LANG=EN;@a/…. 
Specify a valid key.

5.3 Caching BAPI Messages

CREATEFROMDAT2 messages are saved in a list object to be used outside the pipeline scope:

TempXmlDocument = BAPIRequestValidated;
BAPIOrders.AddFromXMLString(TempXmlDocument);

where TempXmlDocument is a variable of type XmlDocument, and BAPIOrders is of custom type BAPIOrderList added in a helper library:

BAPIOrdersList Class

The helper library also contains C# classes corresponding to the BAPI transactions schemas generated by running the Xsd.exe utility on the BAPI schemas generated in Visual Studio.

5.4 Validation Examples

The following logs were obtained from the various validation layers:

Examples of Validation Failures

Note the use of CDATA to prevent parsing of characters that would otherwise invalidate the xml format.

6. Orchestration LOB Send/Receive Stage

The subsequent orchestration stage is similar to what has been presented in Handling Errors in SAP BAPI Transactions. Here is a visual overview before getting into the details:

Note that the connection state is set to REUSE only after a successful send.

We look at the messages being constructed and the data being logged. What gets saved:

1. CREATEFROMDAT2Response documents received from SAP;

2. Errors contained in CREATEFROMDAT2Response documents, formatted as strings;

3. Exception strings.

As explained in Using BAPIs in Distributed Systems (ALE), a value of “E” in any of the TYPE fields of BAPIRET2 in a CREATEFROMDAT2Response indicates transaction failure. For example:

<RETURN>
<BAPIRET2 xmlns=”http://Microsoft.LobServices.Sap/2007/03/Types/Rfc/”>
<TYPE>E</TYPE>
<ID>CZ</ID>
<NUMBER>95</NUMBER>
<MESSAGE>Sales organization abcd does not exist</MESSAGE>
…
</BAPIRET2>
<BAPIRET2 xmlns=”http://Microsoft.LobServices.Sap/2007/03/Types/Rfc/”>
<TYPE>E</TYPE>
<ID>V4</ID>
<NUMBER>219</NUMBER>
<MESSAGE>Sales document was not changed</MESSAGE>
…
</BAPIRET2>
</RETURN>

Errors are extracted by the orchestration with the following helper class (complete code in attached archive):

BAPIReturnValues

which is used as in the expression:

BAPIResponseRetVal.Set(BAPIResponse);

where BAPIResponse is of type CREATEFROMDAT2Response.

The method BAPIReturnValues.GetErrorString() produces the error string logged to a file by using the RawString class documented in Sending string to file location in BizTalk and included in the BizTalk code attached to this blog post.

DataValueErrorMessage = new Microsoft.Samples.BizTalk.XlangCustomFormatters.RawString(
System.String.Format(“Message {0}”, BAPIResponseRetVal.GetErrorString()));

Handling Failures

Given that the data being sent to the server has been “sanitized” in previous stages, failures at this stage are expected to be more general, such as connectivity issues.

Connection Error

Although not done here to keep the design as simple as it can be, it is possible to handle errors by re-enqueueing CREATEFROMDAT2 messages in the BAPIOrders cached list, as was done in Handling Errors in SAP BAPI Transactions and resetting the connection state to “OPEN” so a new WCF channel may be opened to restart a new LUW. This is can be done with minor changes to the existing orchestration.

7. Last Message in Batch and Commit Stage

The last message in a batch is identified by the BTS.LastInterchangeMessage property, which by default does not exist in other messages. This causes the following decision expression:

OrderReceived(BTS.LastInterchangeMessage) == true

to throw a PropertyNotFound exception. Catching the exception indicates that a message is not the last in the batch. As an alternative, is is possible to add the property to all messages by doing the following in the custom pipeline component:

// BTS.LastInterchangeMessage only exist on the last message. Populate it here for all other message so we don’t get the exception in the orch.
bool isLastInterchangeMessage = false;
try
{
    isLastInterchangeMessage = (bool)mBaseMessage.Context.Read(“LastInterchangeMessage”, “http://schemas.microsoft.com/BizTalk/2003/system-properties”);
}
catch(Exception)
{
    mBaseMessage.Context.Write(“LastInterchangeMessage”, “http://schemas.microsoft.com/BizTalk/2003/system-properties”, isLastInterchangeMessage);
}

BAPI transactions are committed once the last message in the batch has been processed, regardless of whether the last transaction was successful, if at least one transaction in the batch was successful. The processing at this stage is shown below.

Processing the Last Message

Close-up of the Commit Scope:

Commit Scope

8. Exiting the orchestration

In the general case, the orchestration instance either terminates after the commit has been sent, or gets suspended if something truly unexpected happens, i.e., something that is not handled by the catch blocks in place (we did not implement general exception handlers for this reason). The orchestration flow is such that successful BAPI transactions will be committed even if the last message in the batch is invalid and causes validation error(s).

If the last message fails validation in the receive port, the orchestration instance for the batch will never receive the last message, which is suspended. To address the issue, the receive shape for the next message in a batch is placed in one of the branches of a listen shape, and in the other branch of the same shape, a delay (currently one minute) determines the maximum amount of time to wait until a commit is issued for all transactions received thus far. This is depicted in the figure below, where the CommitScope is identical to the CommitScope in the previous section.

Exit/Loop

9. Orchestration Summary

The orchestration is depicted below with all stages and connected ports.

Orchestration at a Glance

The bindings file for the BizTalk artifacts is also provided in the attached archive.

BizTalk Artifacts

10. Concluding Remarks

Here is a non-exhaustive list of “nice to have” features that did not make it to this article:

– Re-enqueueing of BAPI transactions in the orchestration to make it more “RIP” in the stages post validations.

– Instead of logging just one error, concatenate all errors for a given document. This can be done in the custom pipelines with the techniques presented in BizTalk custom XML validation to get error summary

– Instead of using a timer for the last message in a batch to be processed by, re-route messages suspended in the receive port to the orchestration (basically, take advantage of RIP).

– Simplify the design to debatch CREATEFROMDAT2 messages directly.

The underlying motivation for the current design was to investigate how to provide recoverable interchange processing for BAPI in the general BizTalk sense of Automatic recovery from failures of messages within a data interchange . It is a pertinent question given that (1) BAPIs in the same LUW are essentially an interchange, and (2) BAPIs can be applied to situations where best-effort is acceptable and some transactions should not block the processing of other messages. As an example of latter, BAPIs are used for high-performance mass data transfer by maximizing the number of requests before commits.

Having a “RecoverableInterchangeProcessing” property on the SAP send port configuration would be a very nice feature to have given the complexity of what was presented here.

References

Related blog articles

SAP BAPI Transactions Walkthrough

Debatching SAP BAPI Transactions

Scatter-Gather Pattern with SAP BAPI Transactions

Handling Errors in SAP BAPI Transactions

BAPI in BizTalk

Registry setting to enable BAPI transactions

Operations on BAPIs in SAP

Run BAPI Transactions in SAP using BizTalk Server

SAP Connector for Microsoft .NET

SAP Adapter Binding Properties