Microsoft introduces an ESB for BizTalk

An Enterprise Service Bus (ESB) will be available for BizTalk around the time of the release of R2.

An ESB, a collection of architectural patterns based on traditional Enterprise Application Integration (EAI) principles, is an often developed solution with BizTalk. Now, a set of ESB related building blocks will be made available to speed up the design and development process.

Officially called ESB Guidance, the new package contains architectural guidance, patterns and practices, and BizTalk Server and .NET components. Microsoft ESB Guidance currently consists of a number of BizTalk projects including:

ESB Core Engine (transformation and routing agents)
ESB Core Services (dynamic transformation, dynamic endpoint resolution/UDDI)
ESB Portal framework
Exception Management framework
ESB Client application
Namespace Resolution pipeline component

Currently, select partners, customers building an ESB and Microsoft employees can download and try the new early access release. The release contains:

ESB Developer Guide
This document introduces architects and developers to ESB architectural concepts as addressed by the guidance, describes the installation procedures for the reference implementation, and explains the functionality of the guidance components through a set of commonly accepted ESB use cases.

ESB Reference Implementation VPC
This Virtual PC image provides a fully functional instance of the ESB Reference Implementation for evaluation, technical readiness, proof of concept, and demo purposes. The MQSeries 5.3 Trial edition and CSD12 can be installed optionally. The installation for these can be found at the root of C:\Projects within the VPC. Once you install MQSeries, then you can install the JMS Pipeline component MSI package within the VPC.

ESB Reference Implementation Redistributable
The redistributable contains installation scripts and .MSI installers for the ESB Reference Implementation components.

ESB Positioning Presentation
This presentation describes Microsoft’s platform value to customers evaluating ESB vendor offerings and the capabilities offered by the Application Platform and the ESB Guidance package.

ESB Patterns Technical Presentation
This presentation provides an introduction to ESB patterns and capabilities as implemented in the ESB Guidance package.

Microsoft ESB Guidance should be released in the 3rd quarter of 2007.

SOA: not just a buzzword

Over the last couple of years while the concept of Service Oriented Architecture has evolved I often worried that it would just become another buzzword that would become serially dismissed after a few years in the spotlight.

An interesting article popped up on ZDNet earlier this week which proves there is benefit to be had by adopting the principles associated with SOA. Big hitters like Ebay, HP, and Amazon are all adopting the principles of service orientation and in the process cutting costs and streamlining their enterprises. For an interesting read, check out the full article:

http://blogs.zdnet.com/service-oriented/?p=781

BizTalk Tips and Tricks – Promoting properties from a collection.

Only non-repeatable fields can be easily promoted from within the schema editor. At the end of the day however it’s simply an xpath statement. In rare cases, if you need to promote a repeatable field from a collection, you can simply modify the xpath contained within the schema and change the xpath to point to a specific index within the collection. Internally, BizTalk simply applies the xpath to the message when its time to promote the data. There is no reason why the xpath couldn’t be restructured to evaluate to a field within a collection at a given index (such as 0). The need might be rare, but it can be done if needed.

BizTalk Best Practice - Message Decoupling in Orchestrations

Have you ever found it annoying to change message schema types in an orchestration when the orchestration designer forces you to disconnect or unwire all the shapes that reference the message? There is a clever way around this and in the end, it promotes good design and adds another layer of decoupling and indirection to your messages in orchestrations.

Define all of your Messages in an orchestration from Multipart Schema Types.

If you create a message from a schema and assign it to a receive shape and that receive shape is connected to a logical port, if you need to change that message type, you need to first disconnect the “wire” between the receive shape and the port before the designer allows you the change the message type. Imagine if the message was referenced by lots ports and shapes. You would need to disconnect all of them and that could be a real pain.

If on the other hand you create a multipart message type for each schema and create each message in the orchestration from the appropriate multipart message type, you don’t need to unwire the various references to the message if you need to change the underlying schema type of a message. This might not be directly obvious, adds an abstraction layer around your message types and enhances overall maintainability.

BizTalk 2006 – Deep Technical Training

I’m attending an internal BizTalk 2006 deep dive training session given by one of the BizTalk product team members at Microsoft who is close to the product internals and has lots of practical knowledge. It’s being held for both Microsoft and Virtual Technical Specialists. It’s not your typical deep dive and focuses heavily on architecture, farms, high availability, best practices, tuning, internals, considerations when designing solutions, and more. It’s been a long, but very cool week so far.

As time permits, I’ll begin posting interesting things we’ve discussed along the way. You might not come across anything new, but if you do, great! Most are common sense and good engineering practices; however others could be things you simply haven’t thought about yet or practices you haven’t followed. Given the flexibility and breadth of the product there are always going to be new things to consider.

BizTalk Server 2006 Documentation Update

Microsoft released an update to the BizTalk Server 2006 documentation on 10/10/2006. It can be found here BizTalk 2006 Documentation Update (41.6 MB)

Overview
In this Help Update, you'll find the following content:

• Complete documentation for all BizTalk Server Line of Business Adapters, including new tutorials, one for each adapter. See the section "Using Adapters."



• Troubleshooting guidance for each of the native BizTalk adapters. See the section "Troubleshooting Adapters."



• Troubleshooting Instructions for how to capture a memory dump. See the section "How to Capture a Memory Dump of a BizTalk Process."



• Disaster recovery instructions for backing up and restoring your BizTalk Server and databases. See the section "Backing Up and Restoring BizTalk Server"



• Updated Tutorials. See the section, "BizTalk Server 2006 Tutorials"



• Information for improving fault tolerance. See the section "Planning and Architecture"



• Updated Developer's Reference that now includes the BAM namespaces that were not included in the RTM documentation. See the section "Developer's Reference"

BizTalk 2006 R2 B2B/EDI WebLog

The Microsoft BizTalk Product Development team is launching the B2B/EDI WeBlog on the MSDN platform to communicate feature/functionality relating to EDI, AS2 and HIPAA in BizTalk Server 20006 R2. Key objective of the blog is to increase product awareness and also assist with partner readiness thereby leading to a successful launch of the EDI Technology in R2.

The blog can be found here http://blogs.msdn.com/BizTalkB2B/

Business Policy Management and the BizTalk Rules Engine

Earlier in the year we designed an architecture in which the BizTalk Rules engine was leveraged along with InfoPath, SharePoint and K2.Net to manage business policies in a human to human and human to system workflow solution. The full article can be found (here). We found this design to work extremely well in regards to providing an agile approach at managing business process policies in a generic way. Following is a brief fragement of this article.

RDA leverages the BizTalk Business Rules Engine to manage business policies in document workflow solutions. Document management workflows come in many shapes and sizes. Many of these workflows operate under rigid business policies, which often affect how documents flow through a process or influence decisions along the way.

Business policies are often fluid. They change with time. Any solution dependent on changing business policies must be agile by design. As such, the rules that govern the policy of a given document workflow needs to be easily maintained without affecting the solution deployed in a running production environment. The BizTalk Business Rules Engine provides just a mechanism.

The diagram below depicts a workflow system that leverages the rules engine to enforce and manage fluid business policies.

Browse all the RDA Spotlight on Business and Technology newsletters here:
http://www.rdacorp.com/SpotlightNewsletters.htm

Choosing BizTalk over WebSphere

The following is a good whitepaper providing a comparative analysis of Microsoft BizTalk and IBM WebSphere. Note while this whitepaper is a bit dated and BizTalk 2004 centric, its still very much applicable to BizTalk 2006.

http://download.microsoft.com/download/0/1/E/01E77F30-E571-478D-A8F8-F35FE66BD2BD/ChsBPASol.doc

Mapping complex conditions in BizTalk with XSLT

Recently I found myself in a situation where I needed to evaluate a set of complex conditions that I could not use the Business Rules Engine for. The BizTalk mapper was able to achieve the conditionality by chaining a series of about 21 functoids together but this became very unmanageable especially since I was combining false statements to see if both were false, then returning a true to an “And” functoid (&) which evaluated another branch as true and passed a result to the node in the destination schema, you get the idea.

An easier way is to plug in XSLT. There are disadvantages to doing this, the main one being its maintainability and transparency but when faced with the daunting set of functoids described above, it looked a decent option.

The XSLT function that I used was the <xsl:choose> statement, inside the statement was a set of conditions which, if true, would return a value/values to the destination schema. In the last part of the statement was the <xsl:otherwise> condition which could set values if the conditions presented were not valid and you had a default you wanted to present in that event.

<xsl:variable name="var:vAnnuity" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='LoanPartItems' and namespace-uri()='' and Method='Annuity'])" />

<xsl:variable name="var:vLinear" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='LoanPartItems' and namespace-uri()='' and Method='Linear'])" />

<xsl:variable name="var:vTotalCount" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='PartItems' and namespace-uri()=''])" />

<xsl:variable name="var:vHasValidEndowment" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='LoanPartItems' and namespace-uri()='' and isValid ='true'])" />

<xsl:variable name="var:vHasValidId" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='Ready' and HasValidId ='true'])" />

<xsl:element name="LeningSoort">

    <xsl:choose>

        <xsl:when test="$var:vAnnuity = $var:vTotalCount">

            <xsl:value-of select="1" />

        </xsl:when>

        <xsl:when test="$var:vLinear = $var:vTotalCount">

            <xsl:value-of select="2" />

        </xsl:when>

        <xsl:when test="($var:vHasValidEndowment = $var:vTotalCount) and ($var:vHasValidId = $var:vTotalCount)">

            <xsl:value-of select="3" />

        </xsl:when>

        <xsl:otherwise>

            <xsl:value-of select="4" />

        </xsl:otherwise>

    </xsl:choose>

</xsl:element>

Of course each specific case requires different XSLT, but this should provide a framework of how to do this within a functoid.

To set up the XSLT:

• Create XSLT similar to the one above which meets your needs
• Drop a scripting functoid on your map
• Connect the output parameter to the node you want to create
• Note: do no include any input parameters, the XSLT is setup already to look them up
• Open the scripting functoid’s, click the (…) inside “Configure Functoid Script”
• Under “Script type” choose “Inline XSLT”
• Paste your XSLT snippet in the text box

XSLT performance when mapping large documents in BizTalk

Recently I had to map a document with many thousand rows. I could not split the document because before I could split it, the document’s nodes had to be sorted.

With such large files you generally test it using a small subset to avoid waiting for maps to complete, I built an XSLT which worked great, I thought.

When you use a select filter such as "not(KeyValue=preceding-sibling::row/ KeyValue)" you end up with a huge performance hit the larger the document gets. My map went from 2 seconds for 50 rows to 10 minutes for a few thousand.

How to improve performance when you have large XML files to map that you can’t split? Try using xsl:key instead, which builds an index of keys from which you can much more efficiently select.

Here is a sample XSLT that demonstrates how to use the xsl:key:

<?xml version="1.0" encoding="UTF-8" ?>

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" xmlns:ns0="http://Conversion.schemas">

    <xsl:output method="xml" indent="no" />

    <xsl:key name="NumberKey" match="/*[local-name()='top' and namespace-uri()='http://biztalk/Conversion.schemas']/*[local-name()='row' and namespace-uri()='']"

        use="keyValue" />

    <xsl:template match="/">

        <ns0:Rows>

            <xsl:for-each select="/*[local-name()='top' and namespace-uri()='http://biztalk/Conversion.schemas']/*[local-name()='row' and namespace-uri()='' and generate-id(.) = generate-id(key('NumberKey', keyValue)[1])]">

                <xsl:variable name="current_Number" select="keyValue" />

                <Data>

                    <keyValue>

                        <xsl:value-of select="$current_Number" />

                    </keyValue>

                    <xsl:for-each select="//row[keyValue=$current_Number]">

                        <Part>

                            <PartID>

                                <xsl:value-of select="nr_data" />

                            </PartID>

                        </Part>

                    </xsl:for-each>

                </Data>

            </xsl:for-each>

        </ns0:Rows>

    </xsl:template>

</xsl:stylesheet>

You can evaluate dates using the Cumulative Maximum Functoid

Dates you say? Stop the press!

I had a situation where there was a node whose max occurs is unbounded and I needed to extract a single node from it to map to a destination node whose max occurs is one.

The problem was that I had to choose the node to map with the highest (newest) date:
<root>
<data>
<testdate>01-01-2000<testdate>
<testdata>a<testdata>
</data>
<data>
<testdate>01-01-2002<testdate>
<testdata>c<testdata>
</data>
<data>
<testdate>01-01-2001<testdate>
<testdata>b<testdata>
</data>
</root>

In the case above, I want the node with the highest <testval>, or 01-01-2002, therefore I want to map <testdata>c<testdata> to the destination node.

This can actually be done.

What I did was convert my date to ticks in a scripting functoid:

public string toTicks(string param1)
{
DateTime dt = DateTime.Parse(param1);
return dt.Ticks.ToString();
}

Afterwards I connected the scripting functoid to the cumulative max functoid.

Then I evaluate the output of the scripting functoid with the cumulative max functoid with an Equals functoid.

This should work for any set of repeating dates.

BizTalk's mapper allows drag and drop replace

I learned a little trick the other day about the mapper which I have never seen documented. It works for BizTalk 2004 and also 2006.

Have you ever had to change a functoid and gone thru the task of dragging the new functoid onto the grid and then reconnecting all of the lines, then having to check to make sure you got them all in order?

Well, what I realized quite by accident is that you can actually just drag the new functoid over the one you want to replace and all your links will maintain themselves. The old functoid will disappear and be replaced by the new one.

Of course there is a limitation when you have different maximum inputs, etc, but for the most part this works with everything else.

Blogging about .NET? Then this is a must have tool

I’ve found a tool recently that will make blogging about BizTalk easier. It’s been around for a while but if you are like me and have never heard of it before, CopySourceAsHtml is very useful.

If you have been in the situation of wanting to post a code or XSLT snippet onto your blog and found yourself writing HTML to make it look like it does in Visual Studio (or resort to screen shots) then fret no more, CopySourceAsHtml will take anything that you can open in VS (source code related) and convert it to HTML for you.

The tool is written by Colin Coller and can be downloaded on the CopySourceAsHtml homepage. In addition, Colin has a blog which updates progress on the tool. I read that a VS 2005 version is soon to be ready.

A couple of tips when using it:

• After installing you should be able to open any C# source code page, highlight and right click a code snippet and see a new “Copy as HTML” item on the menu. But, if for example you want to copy XSLT, the item doesn’t show up. Use the edit menu instead.
  
• When you copy something as HTML, under general, uncheck the “Number lines from:” box unless you really want them and also check the “Embed styles” box since blog tools/some browsers wont recognize the style sheet that gets created. Embeding the style makes it a simpler operation.
  

Easier way to add a DLL to the GAC

Often in BizTalk you’ll need to add assemblies to the Global Assembly Cache. In the past I had used a shortcut on my Quick Launch toolbar which opened the GAC, then I’d drop the assembly inside, but there is an easier way.

A registry key can be created which allows you to right click the DLL and “GAC it” in one click.

The text for the key is below; simply copy/paste it into notepad and save it as a registry file (.reg). Then double click the file to add it to the registry.

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\dllfile\shell\gacutil\command]
@="c:\\windows\\Microsoft.NET\\Framework\\v1.1.4322\\gacutil.exe /i \"%1\""


You can also download the Gacutil right-click key as a zip file here.

BizTalk exception handling with web services

Recently I needed to include exception handling in a web service call to an external party. I learned a couple of things and thought I’d highlight them here.

Exception handling doesn’t always behave the way you expect when you are trying to catch exceptions when calling a web service. While using one is very useful some things must be considered or you will not get the result you wanted.

Here is one way to do it as well as a few things to keep in mind when you want to set one up:

• Include the external web service call (both send and receive shapes) in either a non-transactional scope shape or a long running transaction.
• Decide if you need to catch any specific exceptions, if you don’t, using the General Exception will suffice. If you need to catch specific errors, you can use multiple catch exception blocks.
• It may sound obvious to say this, but make sure your orchestration doesn’t have a timeout that is the same length (or shorter) than the timeout of your scope transaction.
• Most important: in your send port which holds the reference to the web service, make sure you are not using any retries (unless, of course, you explicitly need to). If the web service fails with an error such as “There is an error in XML document (5, 10)” and you have retries enabled you will only see warnings in your event viewer (which means the exception block won’t be activated until the SOAP timeout).

"multiple schemas match the message type"may be caused by web services

I was stuck recently with what seemed like a pair of contradictory errors.

I had an application deployed which was exposed as a web service. To test the application we built a test harness which picked up a flat file and called the web service for us and then dumped the result on the file system to see the result.

This testing system worked fine until I needed to use that same deployed schema to do some additional messaging. I wanted to route a message via a receive port and have a send port subscribe to it, then map the message to a new format and deliver it to a destination. Pretty simple, right?

Well, no. When I have my project deployed I got the following error when routing:

There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive" Source: "XML disassembler" Receive Location: " D:\dat\FileReceive\RoundTrip\*.xml Reason: Cannot locate document specification as multiple schemas match the message type "http://BO.Get.EV2#Method".

That error left me scratching my head. It means I probably have the schema deployed twice, right? I didn't, so I thought. When I undeployed the application the messaging engine could no longer match my message type:

There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive" Source: "XML disassembler" Receive Location: "D:\dat\FileReceive\RoundTrip\*.xml"
Reason: Finding document specification by message type "http://BO.Get.EV2#Method" failed. Verify that the schema is deployed properly.

When I deployed the project with the schemas, everything suddenly worked fine. That means that, somewhere, the schema was again deployed.

So I looked thru the BizTalk Server Assemblies viewer and discovered that the web reference which existed in my test harness was the culprit. The messaging engine recognized the web reference as a second deployment of my schema. This was technically accurate but rather frustrating to find. I've seen so many of these errors posted on the newsgroups that I thought I'd mention this as a possible cause.

If you are using test harnesses coupled with web services be aware that your schema is actually deployed twice.

Fully qualified name differs for XSD’s with multiple root nodes in the Business Rules Engine

One of the most common problems that exist when using the business rules engine is forgetting to use the fully qualified name from a schema in definition of the rule.

Posts like this on Google Groups can clarify what I mean if what I am talking about is not clear.

I have discovered something new regarding the fully qualified name which caught me out this morning and I thought I’d mention it here in case you have the same problem.

If you are using the fully qualified name of a schema with multiple root nodes in the BRE, the fully qualified name must include the name of the root node. When you go to your orchestration and try to feed a rules shape your XSD you will find it won’t recognize it (no drop down appears).

For example you can have a schema with multiple root nodes with a fully qualified name such as schma_name_fully_qualified which, with a single root node, would work just fine. But if the schema has multiple root nodes you must append the specific root node to this qualification for the orchestration to recognize the rule.

So if you had the fully qualified name schma_name_fully_qualified, look at your orchestration view, click on the message you are using and look at the message type: you will find that you must use something like schma_name_fully_qualified.header to properly identify the schema when creating your business rules. Changing this should then allow you to select a message type in your orchestration.

How to stop applications from unloading in BizTalk 2004

Recently we developed a time sensitive BizTalk application where a simple set of processing steps were invoked via a web service.

The external calling application (the application calling the BizTalk process) had just 5 seconds to complete its task. Since the BizTalk part was not very complex or resource intensive, we were achieving response times of 1 second, well within the 5 second margin.

During testing, though, we were getting intermittent timeouts because the BizTalk application was unloading itself from memory and causing the calling application to timeout. Basically it took more than 5 seconds for the unloaded BizTalk application to start itself up again and caused a failure.

BizTalk relies on a configuration file to store its application information. This config file contains settings for unloading an application from memory and is located at \Program Files\Microsoft BizTalk Server 2004\BTSNTSvc.exe.config. Full documentation can be found at Microsoft’s website but here is a working example you can download.

By default BizTalk has 2 shutdown settings which are:

SecondsIdleBeforeShutdown = 1800 (seconds, or 30 minutes)
SecondsEmptyBeforeShutdown = 1200 (seconds, 20 minutes)

The link below is a sample config file that never shuts down. We are isolating these “hot” processes on a separate server from the “cold” processes (processes that can safely unload). You can run hot and cold side by side as well but that requires more changes in the config file. If you set the SecondsIdleBeforeShutdown and SecondsEmptyBeforeShutdown to -1 they will always stay in memory.

Note that this is increasing the load on the server but for time sensitive situations this could save you a rash of intermittent errors in production. Note also that you shouldn’t do this unless you really need to, unloading saves resources and with most integration scenarios the few seconds it takes to start up are minor.

You can backup your current file, download this BTSNTSvc.exe.config zip file and extract it, then stop and start your BizTalk host to see it work.

Solving BizTalk "error X2044"

I encountered a strange compilation error recently in BizTalk 2004 when importing a web service.

While I can’t put my finger on the exact cause yet, I do have a fix that works.

When you see an error message like:

C:\Project\Web References\ExtRef\MyOrch.odx(445,23): error X2044: symbol '@@@@' is already defined; the first definition is in assembly @@@.dll : It may be possible to disambiguate by using fully qualified names.

The way to fix this error is to do the following:

• Open the orchestration in Notepad (In my case MyOrch.odx)
• Do a find (Ctrl+F) for “#endif // __DESIGNER_DATA”
• Delete all the code below that line
• Save your file in Notepad
• Say yes when Visual Studio asks if you want to update your file
• Recompile

Note: The code will not be regenerated until you edit an expression shape, etc. Add a “//” or something and you will then see the code re-appear when you open it in Notepad again.

I hope this may serve to save someone time, it took a while for me to figure it out.