BizTalk Xml Complex Types

BizTalk operations are often designed to use an internal message representation, a canonical, for disparate data that represents the same logical piece of information. For example different trading partners might submit different Purchase Order formats, which are then transformed into Purchase Order canonical messages so that BizTalk services can process them consistently.

Sometimes there are other reasons for using a canonical such as the need to enrich the data (status, approvals, context metadata, etc) or to transform the message from non-Xml flat text data.

I recently worked on a project that involved processing many different types of messages. The schemas involved were anything from simple free text messages to well defined messages with both organized content and free form text. While the messages where physically different they all had a common business context. If there had been only a few different messages then a direct schema implementation might have been indicated, however we had almost 100 different message types and different services might subscribe to one or more of them. So, we needed a single canonical to easily represent any one of these different messages. This would also make the subscription/filter configuration easier to maintain and deploy.

How did we implement a single canonical schema to easily represent one of many different messages? The first idea was to try to normalize all the different fields in all the different messages and add the fields to the canonical. This works but the downside is that every time a message type is added or changed, the canonical would have to be updated. Another downside is that structurally it is difficult to organize the canonical schema to make sense from a business perspective and one looses the ability to eyeball the schema and understand where the data is coming from and how it is grouped.

We implemented our solution using Xml Complex Types and the resulting canonical schema required no updating if a message type was added or changed. First we created a base message content complex type from which all other content would be derived. Next we reviewed all the messages and decided how they might be organized from a business perspective. Next, we created the individual content by extending the base message type. Lastly, instead of adding individual messages to the canonical, we simply added the base message content complex type.

The screen shots below depict a schema for a fictional shipping company. Note; the project artifacts are simplified for discussion purposes.

image001

The screenshot below shows the Air Content schema, which contains all the content types associated with shipping by air.

image002

The Transport Canonical schema is shown below. The “Content” node is assigned the base content type. Since all the individual Ground, Air and Uplink content types are derived from the base type they are automatically part of the canonical schema. Note the Xml equivalent node; it contains all of the derived message types from the base Xml complex data type. From MSDN;

Equivalent nodes are created automatically by BizTalk Editor to show how derived complex types can be used instead of the base complex type from which they are derived wherever the base complex type is called for in the schema. This yields the same type of polymorphism that is common in many object-oriented programming languages.”

image003

The “Content” record properties.

image004

 

Below is a screenshot of a canonical instance for the Air Weather Request message. During mapping we could easily map the name of the message to a promoted property (TransportMessageTypeId) if required.

image005

BRE and XML Node Creation

On a recent project, I was tasked with using Business Rules Engine (BRE) to call a list of BRE Policies. These policies could generate a number of results, or triggers, the number of results was not known until runtime. BRE can update existing XML nodes within the Fact document, however, it cannot create nodes that do not already exist.

To solve this issue, I decided to add a serializable .NET object to pass in as a fact. The object has a public method that is used to create trigger objects and add them to the object's collection. Upon return from the BRE call, the raw xml of the object is exposed through it's XmlString property.

First we will talk about the Trigger class. This class represents our result. It is very generic and thus flexible in its use. The TimeStamp field can be used to track the time the trigger was created. We use the InterfaceName field for routing purposes. The Action and SubAction fields can optionally be used to further differentiate the trigger from other triggers during routing. The other section of the Trigger is a collection of Parameters. This is where a list of key-value pairs can be stored for use in processing. We expose a method to add a parameter key-value to the Parameters collection.



The PolicyResults class is a collection of Trigger objects. We expose a CreateTrigger method in order to be able to create Trigger objects on-the-fly from BRE. We also expose two utility properties, Count and XmlString. The Count property returns the number of Trigger objects in the collection. The XmlString property returns the raw XML version of the collection. This property can be loaded into an XmlDocument class and assigned to and XLang message easily.



We use this setup when calling BRE by first creating a PolicyResults object and passing it into BRE as a fact. Next, when creating policies in BRE, we can call the CreateTrigger method in the Assert section of any Rules we create. If the Assert is executed the Trigger(s) will be created and added to our PolicyResults. Once the BRE call is complete we can return the PolicyResults as a string (or XmlDocument) to the calling orchestration, where can be used for processing.

R2-AS2 POC

R2 AS2 POC
Sometime back, I worked on a AS2 POC(proof of concept) using BizTalk 2006 R2. Here is a high level overview of VAN/BizTalk interaction, Party configuration, AS2 and EDI receive/send ports configuration in R2, to generate decrypted EDI files and 997s.


BizTalk Server 2006 R2 makes use of many components to successfully establish communication between VAN(Value Added Network) provider and their customers. In this proof of concept, to receive and send EDI messages, AS2(Applicability Statement 2) components and HTTP adapter components are primarily used.
BizTalk AS2 receive processing is performed using the AS2 receive pipelines. There are two types of AS2 pipelines available in R2. AS2EdiReceive pipeline to process EDI messages received over AS2 and AS2Receive pipeline to process messages that are not encoded in EDI. AS2 pipelines are also responsible for generating MDNs(Message Disposition Notification).
In the above diagram, a request-response HTTP Two-Way adapter is configured to receive messages from VAN. AS2 receive pipeline generates the MDN and routes it to the BizTalk MessageBox database. This MDN will be automatically picked up by the AS2Send pipeline which is part of the HTTP Two-Way adapter. AS2 receive pipeline uses the BizTalk S/MIME pipeline component to provide S/MIME decoding functionality. AS2 Decoder processes the incoming message AS2/HTTP headers, verifies the signature and decrypts the encrypted messages. After successful decryption, AS2 disassembler generates a MDN and sets the correlation tokens and properties on the MDN.
In the case of EDI messages, EDI disassembler will parse the message and generates corresponding EDI document and 997 acknowledgements.

Party Configuration:

  • Create a new party using the BizTalk Administrator explorer
  • Enter appropriate values in the Organization, Name and Value fields under the values. I used EDIINT-AS2, AS2-From and partner name respectively. Hint: AS2 functionality resolves the incoming messages party information based on the AS2-From and AS2-To values in the Aliases tab
  • Next step is to set signing and encryption, MDN generation request properties for a given message
  • Right click on the party to select AS2 properties, select Party as AS2 message receiver
    Leave the default values selected and make sure Sign message and Encrypt message check boxes are selected under Outbound AS2 message. I selected DES3, Application/EDI-X12 and also entered AS2-From and As2-To values, selected ‘Request MDN’ check box

HTTP Two-Way port configuration:

  • Create a Request-Response receive port/location to process AS2 messages and generate MDN response
  • Select HTTP as transport type and select AS2EDIReceive as receive pipeline and AS2Send as send pipeline
  • Create a send port to send raw data and go back to the party you created earlier to select this send port using the party administration
  • Select Certificate in the send port to apply certificate thumbprint for encrypting messages
  • Create another send port to send pay load messages (EDI messages). In this case, selected file transport type to send EDI files to local folder and make sure to set EDISend as the send pipeline
  • Enter receive port name and select EDIintAS.IsAS2PayloadMessage == True in the send port filter. This fileter values allows us to decrypt EDI messages and creates message files as specified in the Transport Type

997 Configuration:

Certificate Management:
It turned out the most important part of the POC was to have valid certificates and configured it right. I obtained a trial certificate from VeriSign. Note, you can also use windows generated certificate too. In a nut shell, for the encryption/decryption of the messages over AS2, I followed the Certificate configuration as described in this KB article: http://support.microsoft.com/?id=942253

Using BAM API’s in a BizTalk Solution

Recently, I had a requirement where I needed to track a subset of messages flowing through an ESB to capture and store key data points to a table along with the raw Xml of the message. The data was going to be used by the client for viewing and reporting.

After a couple of small POC’s, it was decided to leverage BAM API’s in order to persist the data. BAM comes with OOB functionality to capture and display data elements through the BAM portal. Additionally, after defining an Observation Model using the Excel plug-in, the BAM definition is then deployed which creates Sql Server tables, stored procedures and views in order to persist and update the tables.

But, OOB functionality does not support capturing and persisting of the underlying Xml or provide content based filtering, so as part of the POC, the BAM API’s were used to store the Xml.

A C# component was written to use the OrchestrationEventStream class from the BAM API. This class is designed to asynchronously write data into the BAMPrimaryImport database as well as provide transactional consistency when used from either an orchestration or pipeline.

For this particular solution that I was working on, the messages that needed to be tracked were already being consumed by a generic Data Archive orchestration. This orchestration was then modified to contain a Decision Shape that would filter based on configurable conditions to capture the required content.

RFID Presentation at MICSUG, Reston VA

This week I presented on Microsoft BizTalk RFID at the MICSUG (Microsoft Integration and Connected Systems User Group) for the Washington, DC area. The presentation was held at the Microsoft office in Reston.

The main topics that were addressed were architecture of the RFID code base, RFID as a platform, and case studies which demonstrated significant ROI’s. In addition we discussed how RFID and BizTalk work together.

RFID demonstrates a big step forward in BizTalk Server’s capabilities because it now enables connectivity to actual devices rather than software to software transactions which are traditional means of using BizTalk. It’s a brand new paradigm and the audience sensed the shift as the realization that almost any device can be tied into an enterprise using BizTalk.

Exciting times.

Link to BizTalk case studies

Often times I am asked for reference case studies for potential BizTalk projects. And then comes that moment when I can't find the link to them!

Well, here it is. This part of the Microsoft.com site contains over 200 case studies related to BizTalk and has been updated with many new cases related to BizTalk 2006.

http://www.microsoft.com/casestudies/search.aspx?ProTaxID=1265

HIPAA Support in BizTalk 2006 R2

HIPAA support in BizTalk has always required the purchase, installation and use of the HIPAA accelerator. I've recently learned that the upcoming release of BizTalk 2006 R2 (scheduled sometime in the June 2007 timeframe) will bundle HIPAA support out of box. A main feature of BizTalk 2006 R2 is to provide native EDI support for transaction sets such as X12. HIPAA is simply a collection of specific X12 transaction sets. As such, its a natural enhancement to support HIPAA out of the box. The following is the current list of transaction sets to be supported as described in the Biztalk b2b blog on msdn located at: (http://blogs.msdn.com/biztalkb2b/archive/2006/12/31/hipaa-support-in-biztalk-server-2006-r2.aspx).

  • key HIPAA 4010A transaction set schemas:
    - 270/271 - Eligibility
    - 276/277 - Claim Status
    - 278 - Services Review Request/Response
    - 820 - Payroll Deduction
    - 834 - Benefit Enrollment
    - 835 - Claim Payment
    - 837D - Claim Dental
    - 837I - Claim Institutional
    - 837P - Claim Professional
  • ten other non-HIPAA standard schemas that are widely used in the HIPAA realm, including but not limited to:
    - v3070 277 Healthcare Payer Unsolicited Claim Status
    - v4010 148 Doctor's First Report of Injury
    - v4040 277 Healthcare Claim Acknowledgement
    - v4050 274 Healthcare Provider Information
  • a schema annotation which enables splitting a multiple part document (e.g. a 837 Claim that has many individual claims) into separate transactions, one for each individual document. This feature is enabled by an annotation called 'subdocument_break'. When assigned a value of "yes" it will split an inbound transaction into individual transaction sets based on the number of documents included in the original transaction set. For example, an inbound 837D with three separate claims in it would be split into three separate 837D transaction sets with all of the appropriate header information included in each.
  • WEDI/SNIP level 2 validation which includes basic X12 syntax integrity as well as validation of loops, segments, elements and code values.

Microsoft introduces an ESB for BizTalk

An Enterprise Service Bus (ESB) will be available for BizTalk around the time of the release of R2.

An ESB, a collection of architectural patterns based on traditional Enterprise Application Integration (EAI) principles, is an often developed solution with BizTalk. Now, a set of ESB related building blocks will be made available to speed up the design and development process.

Officially called ESB Guidance, the new package contains architectural guidance, patterns and practices, and BizTalk Server and .NET components. Microsoft ESB Guidance currently consists of a number of BizTalk projects including:

ESB Core Engine (transformation and routing agents)
ESB Core Services (dynamic transformation, dynamic endpoint resolution/UDDI)
ESB Portal framework
Exception Management framework
ESB Client application
Namespace Resolution pipeline component

Currently, select partners, customers building an ESB and Microsoft employees can download and try the new early access release. The release contains:

ESB Developer Guide
This document introduces architects and developers to ESB architectural concepts as addressed by the guidance, describes the installation procedures for the reference implementation, and explains the functionality of the guidance components through a set of commonly accepted ESB use cases.

ESB Reference Implementation VPC
This Virtual PC image provides a fully functional instance of the ESB Reference Implementation for evaluation, technical readiness, proof of concept, and demo purposes. The MQSeries 5.3 Trial edition and CSD12 can be installed optionally. The installation for these can be found at the root of C:\Projects within the VPC. Once you install MQSeries, then you can install the JMS Pipeline component MSI package within the VPC.

ESB Reference Implementation Redistributable
The redistributable contains installation scripts and .MSI installers for the ESB Reference Implementation components.

ESB Positioning Presentation
This presentation describes Microsoft’s platform value to customers evaluating ESB vendor offerings and the capabilities offered by the Application Platform and the ESB Guidance package.

ESB Patterns Technical Presentation
This presentation provides an introduction to ESB patterns and capabilities as implemented in the ESB Guidance package.

Microsoft ESB Guidance should be released in the 3rd quarter of 2007.

SOA: not just a buzzword

Over the last couple of years while the concept of Service Oriented Architecture has evolved I often worried that it would just become another buzzword that would become serially dismissed after a few years in the spotlight.

An interesting article popped up on ZDNet earlier this week which proves there is benefit to be had by adopting the principles associated with SOA. Big hitters like Ebay, HP, and Amazon are all adopting the principles of service orientation and in the process cutting costs and streamlining their enterprises. For an interesting read, check out the full article:

http://blogs.zdnet.com/service-oriented/?p=781

BizTalk Tips and Tricks – Promoting properties from a collection.

Only non-repeatable fields can be easily promoted from within the schema editor. At the end of the day however it’s simply an xpath statement. In rare cases, if you need to promote a repeatable field from a collection, you can simply modify the xpath contained within the schema and change the xpath to point to a specific index within the collection. Internally, BizTalk simply applies the xpath to the message when its time to promote the data. There is no reason why the xpath couldn’t be restructured to evaluate to a field within a collection at a given index (such as 0). The need might be rare, but it can be done if needed.