RDA Corporation EAI Blog

Carolina Code Camp

2010-05-25T08:33:00.001-05:00

Recently I gave a presentation on BizTalk Server 2009 at the Carolina Code Camp. The presentation covered the basics of BizTalk as well as the new features of BizTalk Server 2009.

I showed a short demonstration of a sample Purchase Order process. The components used were the Schema Designer, the Mapper, the Orchestration Designer, and the Consume WCF Service Wizard.

PowerPoint slides and demo code are available upon request.

REST Services in BizTalk

2010-05-12T13:46:00.001-05:00

After many hours of research and blog reading, I came to the conclusion that there is no out-of-the-box or additional download (from an Adapter Pack or similar) support for REST services in BizTalk 2009. My recommendation is to have a C# helper class make the call for you. This is not what I was hoping to discover, although it does make sense.

REST services do not employ formal contracts, while BizTalk relies heavily on strong-typing of messages, these two philosophies contradict.

The other issue is that BizTalk needs two pieces of data for message delivery, destination URI and the Message. REST embeds the message as part of the URI. Again, this is a contradiction in philosophies.

We can create a Dynamic Send port with the correct URI (message embedding included), however BizTalk must send a message along with the URI and REST services reject this additional payload. Also, Dynamic Send ports do not support two-way communication so there is no way to correlate the response message, even if we were to actually be able to get one from the REST Service.

Until Microsoft adds support for REST in BizTalk, the workaround is to use a .NET Helper Method to make the REST call for us.

BizTalk Benchmark Wizard Released

2010-02-22T07:30:00.004-05:00

One of the most common questions I get asked when looking at a BizTalk server project is what type of hardware is required and how many servers are needed.

This is often a black art.

Microsoft has stepped in to help with the release of a BizTalk Benchmark Wizard:

http://msdn.microsoft.com/en-us/biztalk/ee946766.aspx

Basically this tool is intended to work along side the BizTalk Server 2009 Scale-Out Testing Study:

http://msdn.microsoft.com/en-us/library/ee377068(BTS.10).aspx

Comparing the results of the study to your own running production environment can provide valuable insight into the performance of a BizTalk server farm and point to places where optimization is needed

BizTalk 2009 and SQL 2008 - Simple recovery mode breaks agent

2010-01-28T11:31:00.002-05:00

I learned something today about SQL 2008 and BizTalk's SQL agent jobs. You must leave all BizTalk databases recovery model to "Full". I encounted a database set with the recovery model that was changed to "Simple" and it caused the agent's jobs to fail.

The reason for this is that "TRUNCATE_ONLY" has been deprecated for SQL Server 2008. This flag forces you to keep the recovery model as full. When I changed it back everything began funtioning normally again.

How to solve "Unable to determine if the owner has server access" error

2010-01-26T18:46:00.001-05:00

This error message popped up all of a sudden during a load test with heavy volume.

Turns out to be a well known culprit to lots of BizTalk errors: the SQL Agent jobs were not running. Since we had just deployed to a test environment, and it was a new server, I didn't check to the Agent to see if it was running.

When I teach the class, I always mention this check, so I didn't follow my own advice. Anyway, the database server logs grew wildly and caused a shutdown.

When I configured and ran the jobs everything was fine, but it took a while for everything to complete.

Using a Flat File Schema to Consume Non-Xml Messages With No Type

2009-11-16T08:54:00.001-05:00

Recently, a client was updating their message processing system from a collection of old VB6 applications to a BizTalk Server 2009 solution. One of the processes that they struggled with was updating their “special case” partners. I call these partners “special case” because they could not or would not adhere to the standardized schema for acknowledgment messages that was provided.

Each “special case” partner had their own message type, which changed without notice. The only thing that each had in common was that they agreed to put an acknowledgment bit somewhere within their return message. The placement of this acknowledgment bit was establish many years ago when the VB6 applications where written. The problem of extracting the bit was solved within the VB6 applications and the code was simply updated to C# to leverage the existing solution.

The other issue that was occurring is that each acknowledgment message had a different format, and non of them were valid XML. Most of the acknowledgments were actually screen scrapes from HTML pages with multiple root nodes and the message stuffed in at various places. BizTalk does not like to consume messages that have no schema/message type. To solve this issue, I had the client run the Flat File Schema Wizard to generate a Flat File Message with a single node and the entire contents of the message as its value (contained within CDATA markup to prevent corruption). The trick with this approach is the delimiter used to define the records. Since we only ever want one record, we need to choose a delimiter that has a very, very low probability of showing up in the incoming message. The delimiter I chose to use was “|abcdefghijklmnopqrstuvwxyz|”.

Now that the message was being consumed, it was a simple matter to use xpath to extract the string payload of the message and send it over to the existing C# helpers that know how to process it.

Where Do I Create a Web Reference in BizTalk 2009?

2009-10-20T14:58:00.003-05:00

If you are reading this, most likely you were just as stumped as I was when I tried to create my first web reference in BizTalk 2009.

I assumed "Web Reference" was renamed to "Service Reference" but I quickly realized that something was wrong when I tried to create a message within an orchestration and had nothing listed within the "Web Message Types" section.

Well, to create a traditional Web Reference in BizTalk 2009 you have to take an extra step:

Right click on the project, and select "Add Service Reference" just as you did before, but this time, at the bottom click "Advanced". At the bottom of that screen click "Add Web Reference" which will take you to the web reference screen we recognize from previous BizTalk versions.

How to get to the packaged EDI schemas

2009-09-28T09:42:00.003-05:00

I needed to validate a clients test message against the standard EDI schema for an X12 810 message but did an extensive search and was not able to find the schema for the message.

I found the EDI base DLL that contains the schemas (Microsft.BizTalk.EDI.BaseArtifacts.dll) but this didn't give me the ability to validate against the schema.

Turns out that all of the EDI XSD schemas are packaged in an exe located in: \Program Files\Microsoft BizTalk Server 2006\XSD_Schema\EDI. Just unpack the file named MicrosoftEdiXSDTemplates.exe and you will have extractions for all of the EDI schemas for EANCOM, EDIFACT, HIPPA and X12.

Be careful because this is a huge number of files. I opened the exe in win rar and just extracted the X12 files.

From there you can just add the schema to your project and validate your message

Mapping Unrelated Records

2009-07-23T02:50:00.001-05:00

Introduction

I was recently asked to create a map to combine parent records with children records. The challenge was that the children records in the source document where not grouped with their parent record.

Mapping Unrelated Records

The message was similar to the following Xml where StatusItem is related by its ordinal position to DetailStatus/DetailItem. Look carefully at the message; the StatusItem parent node is SummaryStatus, which is at the same level in the Xml as DetailStatus. There is only one SummaryStatus node, and there are multiple DetailStatus nodes – all at the same level in the Xml.

<root>
<SummaryStatus>
    <SummaryCode>102</SummaryCode>

    <StatusItem>
      <statusCode>item_0</statusCode>
    </StatusItem>

    <StatusItem>
      <statusCode>item_1</statusCode>
    </StatusItem>

    <StatusItem>
      <statusCode>item_2</statusCode>
    </StatusItem>
</SummaryStatus>

<DetailStatus>
    <DetailItem>
      <statusCode>detail_0</statusCode>
    </DetailItem>
</DetailStatus>

<DetailStatus/>
    <DetailItem>
      <statusCode>detail_1</statusCode>
    </DetailItem>
</DetailStatus>

<DetailStatus/>
<root>

In reality a message similar to the above is being provided by an outside source and we cannot modify it. By convention we are given the assumption that there is a DetailStatus record for each StatusItem, however there is not always a DetailItem. Our job is to map the output and combine the StatusItem code with its each DetailItem code.

The output goal is similar to the following Xml:

<root>
<SummaryCode>102</SummaryCode>

<StatusItem>
    <StatusCode>item_0</StatusCode>
    <DetailItem>
      <StatusCode>detail_0</StatusCode>
    </DetailItem>
</StatusItem>

<StatusItem>
    <StatusCode>item_1</StatusCode>
    <DetailItem>
      <StatusCode>detail_1</StatusCode>
    </DetailItem>
</StatusItem>

<StatusItem>
    <StatusCode>item_2</StatusCode>
</StatusItem>
</root>

The BizTalk map below works by using the Iteration Functoid comparing the position of the current SummaryStatus parent record to the position of the DetailStatus record. Additionally the Logical Existence Functoid is used so that if a DetailItem does not appear in the source message, then it won’t appear in the Target message.

External Links from MSDN

Previous Articles

Fix for BAM error "Views or Activites may be missing because one or more database(s) could not be contacted"

2009-04-22T07:35:00.000-05:00

I had to migrate a BAM implementation into production from a development environment where everything in localized to a clustered database back end.

The deployment went fine, including the BAM part, but I couldn't access the views in the BAM portal. I verified that data was being written into the BAM database, but when I went to the portal to view the data I saw the following message:

Views or Activites may be missing because one or more database(s) could not be contacted. No view to display

There was also an error in the event log which read:

Referenced database 'BAMPrimaryImport' on server 'xxxx' is not accessible. The error is:
System.Data.SqlTypes.SqlNullValueException: Data is Null. This method or property cannot be called on Null values.

After some investigation I learned that the owner of the BAMPrimaryImport database needed to be a domain account. I tried to change the account role:

sp_addrolemember 'dbowner' 'domain\user'

This failed with an error.

So I ran instead (which does not remove the user, just ‘resets’ the users access rights):

sp_revokedbaccess 'domain\user'

Which threw an error, but the account was holding the db owner schema, so I changed that to dbo and it worked.

Then, I could run the original call (which now worked):

sp_changedbowner 'domain\user'

The views are now visible in the portal, so if you see this error above, this may be the cause.

ESB 2.0 Custom Resolver: Part 2

2009-03-26T10:52:00.001-05:00

In part one, we discussed what code was required for a custom resolver. In part two, we will be talking about what steps need to be undertaken in order to have the ESB recognize the new resolver as well as the topic of caching and making the resolver configurable.

Caching

Resolution caching is accomplished using the Microsoft.Practices.ESB.Cache.Cache<T> object. Each resolver maintains its own unique cache. Some resolvers, such as BRE, do not maintain a cache at all, as they are expected to update in real-time when the repository changes.

Adding a resolution to the cache is as simple as calling the Add method of the Cache<T> object with a key and the resolution. A resolution should be unique to its configuration, with that assumption in mind the standard key for a resolution in the cache is its config string. Resolutions are defined as type Dictionary<string, string> the resolution Dictionary should be the second argument in the call to the Add method. Note, that some verification should be done to ensure that the resolution is not empty/corrupted, as it will come back each time the provided config is resolved until the cache expires. The last argument for the Add method is the optional expiration interval. In this implementation a configurable value cacheTimeoutValue was crated that is used to define how long a cache value should persist. A value of 0 denotes that the cache has been disabled.

Retrieving a resolution from the cache requires a call to the Get method of the Cache<T> object. The Get method requires the key (the config string) in order to retrieve the resolution from the cache. The benefit of caching is that it enhances performance when under higher volumes. The database is only queried once per cache expiration interval for a unique configuration. The drawback is that the repository could take one entire expiration interval to update when it is changed. With this balancing act in mind, the expiration interval was made configurable.

Configuration

Configuration includes wiring up the resolver to the ESB and configurable resolution values. The wire-up portion is required for all implementations, so we will start the discussion with that.

Contained in the ESB install is a configuration file called esb.config. This configuration is referenced from both the machine.config and the btsntsvc.exe.config files. The resolution configuration settings are contained in the esb.config file. To wire-up the custom adapter following node needs to be added to the configuration/esb/resolvers node:

The name attribute denotes the Moniker name (see part one for a discussion of Monikers). The type attribute is the fully qualified assembly name for the custom resolver that was created in part one.

To add configurable values for a resolver a resolverConfig node can be added to the aforementioned resolver node:

The configuration can be parsed in the custom resolver class with the following code:

string cacheName = ResolverConfigHelper.ReadResolverConfigByKey(resolverConfig, "cacheName");

To make use of the SQL Cache Manager, it must be configured in the esb.config. Add the following node to the configuration/cachingConfiguration/cacheManagers node:

Future Plans

Future plans for the SQL resolver is to make it more configurable. There are plans to be able to query services by other fields such as Author, Version, etc.

This concludes my two part discussion of custom resolvers in ESB 2.0.

ESB 2.0 Custom Resolver: Part 1

2009-03-26T09:36:00.001-05:00

Recently I encountered a situation where we wanted to leverage SQL server as and repository for endpoints. The resolver consists of two parts: the repository (SQL Table in this example) and the custom resolver assembly. I will be focused on the custom resolver for this discussion.

The custom resolver consists of an assembly and configuration entries. In part one we will discuss the code that goes into the assembly. In part two we will talk about wiring up the assembly so that the ESB will use the new assembly, as well as how to make the resolver configurable.

Per the pattern set up by the ESB Guidance team, each resolver should be in its own assembly. This assembly will need to include references to the following assemblies:

Microsoft.Practices.ESB.Cache
Microsoft.Practices.ESB.Common
Microsoft.Practices.ESB.Configuration
Microsoft.Practices.ESB.Exception.Management
Microsoft.Practices.ESB.Resolver
Microsoft.XLANGs.BaseTypes

The naming convention for the assembly is ESB.Resolver.MyResolver. In our example the assembly is ESB.Resolver.SQL. A class that implements IResolverProvider is required for the custom resolver. The naming convention for this class is ResolveProvider.

The IResolverProvider interface implements three overloaded variations of the Resolve method. Each of these need to be implemented to ensure that the resolver works in all traditional contexts. All three methods should be executing the same logic. With that in mind, the ESB Guidance team created a ResolveMyResolver(string config, string resolver, Dictionary<string, string> resolutionOrig) method that each Resolve method calls. I have included my sample below.

private Dictionary<string, string> ResolveSql(string config, string resolver, Dictionary<string, string> resolutionOrig)
        {
            // fix the resolver if it needs it
            if (!resolver.Contains(ResolutionHelper.MonikerSeparator))
                resolver = resolver + ResolutionHelper.MonikerSeparator;

            try
            {
                // check for cache...return data structure
                Dictionary<string, string> sqlValues = RetrieveFromCache(config);

                //No Resolution cached
                if (sqlValues == null)
                {
                    sqlValues = new Dictionary<string, string>();

                    //Extract Parameters
                    Dictionary<string, string> queryParams = ResolutionHelper.GetFacts(config, resolver);
                    string serverName = ResolutionHelper.GetConfigValue(queryParams, false, "serverName");
                    string dbName = ResolutionHelper.GetConfigValue(queryParams, false, "dbName");
                    string serviceName = ResolutionHelper.GetConfigValue(queryParams, false, "serviceName");
                    //Build connection String
                    string connectionString = String.Format("Data Source={0}; Initial Catalog={1}; Integrated Security=SSPI;", serverName, dbName);
                    //Execute SQL query to retrieve EndPoint
                    SqlDataReader sqlReader = null;

                    try
                    {
                        sqlReader = ResolveSqlEndpoint(serviceName, connectionString, sqlReader);
                        if (sqlReader.HasRows)
                        {
                            sqlReader.Read();

                            //Transfer result to Resolution
                            Resolution resolution = new Resolution();
                            resolution.EndpointConfig = sqlReader["EndPointConfig"].ToString();
                            resolution.TransportLocation = sqlReader["TransportLocation"].ToString();
                            resolution.FixJaxRpc = Convert.ToBoolean(sqlReader["JaxRPC"]);
                            resolution.MessageExchangePattern = sqlReader["MessageExchangePattern"].ToString();
                            resolution.TransportType = sqlReader["TransportType"].ToString();
                            resolution.TransportNamespace = sqlReader["TransportNamespace"].ToString();
                            resolution.Action = sqlReader["Action"].ToString();

//Transfer results to the Dictionary
ResolverMgr.SetResolverDictionary(resolution, sqlValues);

                            if (null != sqlValues)
                                resolutionOrig = sqlValues;
                        }

sqlReader.Close();

                    }
                    catch (Exception ex)
                    {
                        EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);
                        throw;
                    }

                    //add result to cache
                    SetCache(sqlValues, config);
                }
            }
            catch (Exception ex)
            {
                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);
                throw;
            }

return resolutionOrig;
}

Stepping through the code, the first step is to repair the resolver string, if needed. The resolver string is also referred to as the Moniker. A Moniker is formatted as follows Resolver:\\. For example the UDDI Moniker is UDDI:\\. In our example, our Moniker is SQL:\\. This first code snippet simply applies the :\\ if it missing.

The next step retrieves the resolution from the cache, if it exists. We will dive into this more in depth in part two.

Next, the parameters are extracted from the config string. The config string is the list of key/values that are included with the Moniker to create the EndPoint string. Our config string contains three key/values: Server Name, DB Name, and Service Name. These values are required to query our SQL repository. The ResolutionHelper.GetConfigValue method is used to extract the parameters from the config string.

Next, the SQL connection string is created and the SqlDataReader is initialized. A stored procedure that takes the Service Name and returns the resolution data for that Service is executed to populate the SqlDataReader. A Resolution object is instantiated and the values from the SqlDataReader are used to populate values in the Resolution class. The Resolution object is then used to create the output of the method using the ResolverMgr.SetResolverDictionary method.

The final step before returning the Dictionary result is to add the resolution to the cache. Caching will be described in more depth in part two.

In part two we will discuss caching, configuration, and wiring up the custom resolver to the ESB.

BizTalk 2009 Beta Documentation Now Available

2009-03-12T08:40:00.005-05:00

Beta documentation for BizTalk 2009 is now available to download from the Microsoft.com website:

http://www.microsoft.com/downloads/details.aspx?familyid=38923F29-167F-497D-80A5-899602FEF8D7&displaylang=en

BizTalkCop For Naming Convention Enforcement in BizTalk 2006 and R2

2009-03-11T16:08:00.003-05:00

Elton Stoneman has put together an extension to FxCop to assist with the enforcement of naming conventions around BizTalk applications.

The 1.0 release contains a set of FxCop rules for analysis against BizTalk assemblies and BizTalk applications. The rules in the package are configurable, by default they are based on the naming conventions Scott Colestock put out a few years ago.

A detailed usage guide is published here:

http://geekswithblogs.net/EltonStoneman/archive/2008/11/14/introducing-biztalkcop.aspx

An extensive demo of the tool is published here:

http://geekswithblogs.net/benny/archive/2009/03/01/biztalkcop.aspx

You can download the package from CodePlex here:

http://www.codeplex.com/BizTalkCop

Empty Node Gotcha with the Business Rules Engine

2009-03-11T11:19:00.017-05:00

I've been on a BizTalk 2006 R2 project where we are using the business rules engine to enrich some data coming from a call center.

The data comes in using the vendor's format and is mapped on the inbound port into a cannonical format to be used internally for processing. Because of the complexity of the vendor's schema, we used some inline XPATH statements in the map to transform the data.

When testing against the endpoint DB2 database, we discovered the database was failing to process the messages, even though everything appeared fine on the surface. After some investigation we discovered some hexidecimal characters in what were supposed to be empty nodes. The characters we saw were representative of a line break and a tab: #xD; and #xA;.

Tracing the steps back, we discovered this was appearing after the Business Rules Engine processed the document.

In a normal map, empty nodes are created as <empty>, but using XPATH we were deriving empty nodes that looked like <empty></empty>, and when this type of node went to the rules engine, it created line breaks.

We needed the empty nodes because the endpoint demanded all the nodes be present, so we had to make an adjustment to the XPATH statements to use something similar to:

<xsl:choose>
<xsl:when test="@@XPATH HERE@@">
<xsl:element name="Empty">
<xsl:value-of select="@@XPATH HERE@@"/>
</xsl:element>
</xsl:when>
<xsl:otherwise>
<xsl:element name="Empty"></xsl:element>
</xsl:otherwise>
</xsl:choose>

Where the choose/otherwise allowed us to create the type of empty node we were looking for.

So, if you find yourself extracting data from the rules engine and these unwanted characters appear, this is one way to solve it.

Dundas Chart Removed from Sample Management Portal in ESB 2.0

2009-03-11T09:12:00.001-05:00

Patterns and Practices has made a major change to the sample Management Portal that ships with the ESB Guidance 2.0 CTP2. The Dundas charting dependency has been removed. This is big news as it allows the sample portal to be used as a low-cost ESB Management Portal without incurring the cost of Dundas Chart. Dundas Chart has been replaced by the MSChart control.

UDDI and Windows Server 2008 Upgrade

2009-01-12T14:54:00.001-05:00

I was upgrading a server recently from Windows Server 2003 to Windows Server 2008. The Windows Server 2003 installation did not include UDDI service. UDDI services are installed by default in Windows Server 2008. Everything looks correct as long as you don’t attempt to use the UDDI services on the upgraded machine. The services look to be running correctly and no errors are sent to the Event Log, however UDDI will not function.

To resolve this issue it is necessary to reinstall UDDI services on the Windows Server 2008 installation. This only occurs when doing an upgrade from Windows Server 2003 and below that does not have UDDI services installed prior to the upgrade.

BizTalk 2009 Database Diagram Now Available

2008-12-29T08:43:00.004-05:00

The BizTalk 2009 product development team has released a pdf diagram of the structure of the BizTalk 2009 databases. You can download yourself a copy at the link below.

BizTalk 2009 Database Diagram PDF

How to Recover the XML for an Acivity

2008-12-09T09:45:00.001-05:00

Recently I created an activity, then saved off the XML from Excel to a folder (let’s call it v1) to deploy using the command like tool BM.exe.

I needed to make significant changes to the activity, so significant in fact, that I needed to re-create the activity entirely, so I made a new activity (call it v2) and made the mistake of overwriting the v1 XML file. When I wanted to remove the previous (v1) activity, I realized that I had no way to remove it because BM.exe requires the use of the saved off XML file.

Being fairly new to BAM, I wondered, how do I remove the v1 activity?

Well, there is a way to recover the XML from the v1 activity.

From the command line, execute:

bm.exe get-defxml -FileName:BAMDefinition.xml

That command recovers the configuration from the BAM DB which can then be leveraged to delete the v1 version of the activity.

Note: You must edit the XML to maintain the activities you want, if you exec the remove without editing the XML first you remove ALL your activities in the DB.

Then you can then execute:

bm.exe remove-all -DefinitionFile: BAMDefinition.xml

You should never delete anything surrounding BAM by hand, this will cause major problem if you do.

Using MSBuild Extension Pack with BizTalk 2006

2008-12-02T09:46:00.000-05:00

As many of you already know MSBuild is not compatible with BizTalk projects out-of-the-box. This fact can be a hurdle to overcome if Team Foundation Server (TFS) is being used to automate builds. The problem is TFS uses MSBuild exclusively in its build process. The workaround was to use an MSBuild task that called a NANT build to execute the BizTalk implementation. This was clumsy and not intuitive.

To solve this issue a group of developers got together and created the MSBuild Extension Pack Project. The Extension Pack provides MSBuild Tasks for BizTalk 2006, SQL 2005, SQL 2008, among others. Tasks for BizTalk include tasks for checking the existence of an Application, adding/removing References, starting/stopping Applications, and creating/deleting Applications. The Extension Pack includes a help file and samples for all this functionality.

BAM: The Case of the Missing Pivot Table Data

2008-12-02T09:32:00.000-05:00

I had a recent problem getting a pivot table to display its data on the BAM portal. I went thru a series of steps to get it working, so in case you have a similar problem here are some things to check to get the data displaying.

To frame things: I had a BAM activity with a series of data points to be tracked. I needed to extract a very simple count/sum cube from the data.

I set up the activity and went thru the view setup in Excel. I had a dimension which tracks a calling system and a then I wanted to get a count/sum of another field which was setup as an integer.

I couldn’t get it working; the view in the portal wouldn’t show any data at all. The calling system wouldn’t appear and neither would the count. I tried to set up the view as both a count (like a rowcount) and a sum and neither one showed the data.

I was pretty sure that I set the activity up correctly so I began to look at different things that could be wrong.

One of the first was to check if the SQL job "TrackedMessages_Copy_BizTalkMsgBoxDb" was running, it was.

Next, I used the SQL Management Studio to connect to Analysis Services on the server that BAM was configured on. I located the cube I was interested in, right clicked on it and select "Process...", data should have appeared. It didn’t. Something else was wrong.
I checked the associated SSIS package which is called BAM_AN_ which is responsible for populating the cube. I ran the package and was finally able to see the data in the BAM Portal.

Documentation for this can be seen here: http://msdn.microsoft.com/en-us/library/aa560072.aspx

So, to get things working right, I scheduled the SSIS package to the correct time interval and things were on track.

BizTalk Training Scenario

2008-11-25T09:20:00.000-05:00

I believe there is a significant gap between what is learned by reading through BizTalk books and doing the Microsoft supplied labs, and what you learn by doing an implementation from the ground up. This has often been referred to as getting “real BizTalk project experience”. Unfortunately, getting real project experience is somewhat of a catch 22…you often need to have a reasonable set of skills to be placed on a BizTalk project in the first place. Not everyone is able to find someone who’s willing to turn you loose on a real business problem as a learning experience.

To help bridge the gap I’ve put together a draft training scenario that when fully implemented will hopefully give someone skills with a good cross section of BizTalk features and functionality that they can work on while waiting to get the opportunity to work on a BizTalk project. There is no supplied code to get the developer started, and no detailed steps to follow to build it. There is some high level guidance on what sub-systems to create, and what each sub-system should do. The idea is that figuring this out on your own and building it from scratch will better prepare someone for real project work than just doing labs and reading.

I’m sure this scenario can use some room for improvement, but if you’ve already read up on BizTalk, and you’ve run through labs available in either training courses or Microsoft virtual labs, you might find this a useful way to become more proficient.

The scenario is broken into an intermediate section that focuses on giving the developer experience with the following:

Separation of implementation features into a set of separate applications or services that are decoupled from each other
Parsing flat files using both positional and delimited formats
Creating orchestrations that output data through both direct binding and “specify later” ports
Using various transport mechanisms (file, SQL, WCF Web service, SharePoint, Email)
Using correlation sets
Creation of canonical and vendor specific message types
Creation of Deployment MSI's, binding files, etc. to support an install that doesn’t require Visual Studio.

There are also suggestions on “Further Areas to Research” which makes recommendations on deeper areas to drill into, building on what’s been done in the intermediate piece:

The Business Rule Engine
Business Activity Monitoring
The Microsoft ESB Toolkit
Messaging / orchestration / mapping topics - failed message routing, dynamic ports, other messaging formats (EDI X12, Excel spreadsheets, etc.), use of xslt functoids.
Tuning and testing

The following link contains a zip file with the following materials:

Training Scenario Overview.doc – Document describing the training scenario
Message Specifications.xls – Excel file detailing the fields in each message (also captured in the overview document)
VendorPurchaseOrder.txt – Sample Vendor Purchase Order
VendorShippingNotice.txt – Sample Vendor Shipping Notice
VendorInvoice.txt – Sample Vendor Invoice
CreateCrmDB.sql – SQL Script which creates the CRM database
CreateCrmTable.sql – SQL Script which creates the CRM Accounts table within the CRM database

http://www.rdacorp.com/blogs/eai/RDA_Training_Scenario.zip

Activity Missing from your BAM Portal? This May be Why...

2008-11-24T14:57:00.000-05:00

Recently I worked on a BAM implementation and found myself stumped as to why, after creating a new activity, it was not visible in the portal.

I thought it might have been a mistake in my configuration, deployment, or a rights issue. After a series of experiments I couldn't determine anything wrong with either my configuration or deployment so I turned to investigating if it was a rights issue or not.

It turned out to be. The following lines saved as a .bat file did the trick. Just replace the bracketed [ ] items with the names from your environment and it will work.

"C:\Program Files\Microsoft BizTalk Server 2006\Tracking\BM"
add-account -AccountName:[DOMAIN\user] -View:[ViewName]
-Server:[ServerName] -Database:BAMPrimaryImport

BizTalk Adapter for DB2

2008-11-13T13:36:00.002-05:00

Introduction

I recently used the BizTalk Adapter for DB2 on a project for a large Insurance company. A few interesting topics worth mentioning about this effort; connection parameters, error codes, and client side trace.

My experience involved calling DB2 stored procedures but they did not themselves use transactions. The stored procedures had been developed by the client using a COBOL CASE tool. So, this was basically a mechanism to call external COBOL routines that handled the business actual functions.

BizTalk Adapter for DB2

From MSDN (links at end):

The BizTalk Adapter for DB2 is a send and receive adapter that enables BizTalk orchestrations to interact with host systems. Specifically, the adapter enables send and receive operations over TCP/IP and APPC connections to DB2 databases running on mainframe, AS/400, and UDB platforms. Based on Host Integration Server technology, the adapter uses the Data Access Library to configure DB2 connections, and the Managed Provider for DB2 to issue SQL commands and stored procedures.

Connection Parameters

Some of the nomenclature when running the DB2 connection wizard can be a little confusing. Here is how it worked for this last project. The first two steps of the wizard are shown below. The third step is for the Username/Password and is not displayed.

Step One

Address or alias: The AS400 server name.
Port: Left as default 446.

Step Two

Initial Catalog: Same as Address, as shown above
Package collection: Client's default collection
Default schema: Same as package collection
Default qualifier: [blank]

Error Codes

The major key to understanding error codes returned from the DB2 adapter is that some errors are DB2 server generated, while some errors are provider specific; in this case Microsoft's DB2 managed provider. The SQL SATE codes are available in the exception for the stored procedure call. Depending on your implementation, an event log my also be found.

For example SQL STATE code "22001" represents a data truncation error. This can happen if the data passed to a stored procedure parameter is too wide. See the link at the end of the article for other codes.

A SQL STATE value of "HY000" represents the provider specific error. One example would be a connectivity problem.

Please see the links at the end of the article for further reading.

Tracing BizTalk DB2 Calls

A utility called SNATrace is installed with the host adapters on the BizTalk server. It provides ad-hoc logging of the connectivity with the AS/400.

A brief blog from an actual team member of the DB2 Adapter product about using “snatrace” is provided below for further reading.

External Links from MSDN

Other External Links

IBM SQL STATE Error Status Code examples

BTS Mapping With External Assemblies Part 3

2008-11-12T13:21:00.002-05:00

Introduction

Part 1 - Inline scripting and external assemblies
Part 2 - XSLT Call Templates and custom extensions
Part 3 - Cascading Functoids

This is the third of three articles about BizTalk mapping with external assemblies. External assemblies or helper classes are used by BizTalk's mapping engine (XSLT) automatically, and they can also be used explicitly by the developer. Just as in an orchestration, a developer may find a requirement that the built in mapping tools ("functoids") do not address and in this case judicious use of a helper class can be very... helpful.

While these articles are not an introduction to mapping if you have a little experience with the BizTalk mapper and a little experience with XSLT you should be able to follow along.

Part 1, introduces inline C# scripting, its limitations, and also introduces the scenario of using an external assembly (helper class) automatically. Part 2, demonstrates using a map's custom extension property, which allows us to explicitly access external assemblies. Part 3, this article, will build on this discussion and highlight potential caveats about using cascading functoids; when the output of one functoid is the input to another functoid(s).

Validating Maps that use Cascading Functoids

Whether a transformation is simple or complex, I often find validating a BizTalk map and viewing the XSLT output is beneficial. Below is an example of a simple map containing a cascading functoid; a function that calls another function. If we review the resulting XSL file the output is as expected. Yet, the output from maps containing cascading functoids that output to multiple targets (be it other functoids or other nodes) does not generate as I would expect.

The XSLT output from this map is as expected. The variable v1 contains the right trimmed string and serves as input to the UpperCase functoid, the output of which is assigned to v2. The value of v2 is then contained in the LastName node.

<xsl:template match="/s0:AMsg">
    <xsl:variable name="var:v1"
         select="userCSharp:StringTrimRight(string(Customer/LastName/text()))" />
    <xsl:variable name="var:v2"
         select="userCSharp:StringUpperCase(string($var:v1))" />
    <ns0:BMsg>
        <Customer>
            <LastName>
                <xsl:value-of select="$var:v2" />
            </LastName>
        </Customer>
    </ns0:BMsg>
</xsl:template>

Note: The built in functoids imbed inline C# code. We can use the "userCSharp:" XSL namespace ourselves when constructing XSLT functoids. We will see an example below.

Functoids with Multiple Outputs

The XSLT story changes when we examine output from the BizTalk mapper when using a functoid that is connect to more than one node (or cascading functoid). It turns out that in the example below, you will see that the entire functoid chain is executed twice. Once for the SortKey node, and once for the LastName node.

Typically a functoid with multiple outputs will be executed once for each output. In this example, the entire functoid chain is executed once for each connection. Imagine using a helper class to get a database value, if one is not careful the same database helper method might be executed more than one time! The XSLT code below is what the BizTalk mapper generated for the map above.

<xsl:template match="/s0:AMsg">
    
    <xsl:variable name="var:v1"
         select="userCSharp:StringTrimRight(string(Customer/LastName/text()))" />
    <xsl:variable name="var:v2"
         select="userCSharp:StringUpperCase(string($var:v1))" />

    
    <xsl:variable name="var:v3"
         select="string(Customer/LastName/text())" />
    <xsl:variable name="var:v4"

         select="userCSharp:StringTrimRight($var:v3)" />
    <xsl:variable name="var:v5"
         select="userCSharp:StringUpperCase(string($var:v4))" />
    <ns0:BMsg>
        <Customer>
            <SortKey>
                <xsl:value-of select="$var:v2" />
            </SortKey>
            <LastName>
                <xsl:value-of select="$var:v5" />
            </LastName>
        </Customer>
    </ns0:BMsg>
</xsl:template>

BizTalk Transformations with Value Caching

Let's expand on the example above. In the map below we have two functoid chains, each go to their respective output element and also to a string concatenation function, which is in turn connected to the SortKey field. Because we want our imagined map to be as efficient as possible we won't be able to completely use BizTalk's drag and drop mapping. We will need to eliminate the multiple connections that go to both the concatenation functoid and to the target fields. One way to do this is by implementing value caching combined with an XSLT call template to eliminate the multiple executions of the same methods. After describing the approach, there will be a short review of the benefits and caveats.

Value Caching with XSLT Call Templates and Inline C#

By looking at the first cut of our hypothetical map above, we can see that the LastName, FirstName, and SortKey target fields are all interconnected. Thus we are going to eliminate all the functoid chains and replace them with a single XSLT call template. In addition we'll add our own standalone C# code.

By adding an inline C# scripting functoid (#1) we can add methods to be called by the XSLT (#2) in the map.

Even though functoid #1 is "floating" the C# code is still included in the map. This is a simple example of some accessor and related functions. They mainly will serve to cache the First and Last Name values so that we don't have to process them twice (Trim + Uppercase). The actual process of processing the string is for example only, the process could equally have been a database call to a helper function via an External Assembly object.

// SortKey: Return LastName + comma + FirstName
//
public string GetSortKey()
{
    return System.String.Format("{0},{1}", GetLastName(), GetFirstName());
}

// LastName: Trim + Uppercase
//
public string _lastName = null;
public void SetLastName(string LastName)
{
    _lastName = LastName.Trim();
    _lastName = LastName.ToUpper();
}

// LastName: Return value
//
public string GetLastName()
{
    return _lastName;
}

// FirstName: Trim + Uppercase
//
public string _firstName = null;
public void SetFirstName(string FirstName)
{
    _firstName = FirstName.Trim();
    _firstName = FirstName.ToUpper();
}

// FirstName: Return value
//
public string GetFirstName()
{
    return _firstName;
}

Functoid #2 is an XSLT template, and it will call the C# methods and emit the SortKey, LastName, and FirstName fields. Here is what the XSLT call template looks like, it is quite similar to what we've already seen in the prior articles.

<xsl:template name="CustomerAndSortKeyTemplate">
    
    <xsl:param name="LastName" />
    <xsl:param name="FirstName" />

    
    <xsl:value-of select="userCSharp:SetLastName($LastName)"/>
    <xsl:value-of select="userCSharp:SetFirstName($FirstName)"/>

    
    
    <xsl:element name="SortKey">
        <xsl:value-of select="userCSharp:GetSortKey()"/>
    </xsl:element>

    
    <LastName>
        <xsl:value-of select="userCSharp:GetLastName()"/>
    </LastName>

    
    <FirstName>
        <xsl:value-of select="userCSharp:GetFirstName()"/>
    </FirstName>
</xsl:template>

Summary

Using XSLT and C# caching can be an efficient approach to designing a map that would otherwise unnecessarily consume too much processing bandwidth if the map were solely generated by the default BizTalk Mapper design output.

Some considerations to think about:

Using XSLT with the BizTalk mapper might require that you or your client have a comfort level about designing the mapping outside of the normal Drag and Drop approach. However, the efficiency of the custom XSLT, and to some degree the straight-forwardness of it may outweigh the amount of design time needed for a complicated Drag and Drop map.
The XSLT approach is more sensitive to schema changes. A simple map with out XSLT will fail validation (if not outright fail to compile) if a schema changes. However, custom XSLT is ultimately just syntax based on a matching mechanism. There is not compile time validation that the elements being emitted conform to the current target schema.

External Links from MSDN

External Links about BizTalk Mapper Performance

Understanding the BizTalk Mapper - Performance and Maintainability

Article Links

Managing SAP Connections with LOB SAP Adapter in BizTalk 2006 R2

2008-10-23T13:01:00.002-05:00

Recently I was working with a client to connect a BizTalk 2006 R2 process to SAP with the LOB SAP adapter. We first called a BAPI function that returned a list of contract IDs that had been modified since our last successful processing event (usually 24 hours). This list could contain anywhere from 1000 to 10000 contract IDs depending on volume and amount of time since the last processing event. Once we had the collection of contract IDs we iterate over the collection calling a separate BAPI function to retrieve the contract data for each contract ID. We quickly ran into the issue where BizTalk was overwhelming the SAP server with BAPI calls. In order to fix the issue we throttled down the number of connections made by the BizTalk Server.

To do this, we use the WCF-Custom Adapter with SAP bindings. In our Solicit-Response Send Port we select WCF-Custom as our Type and click "Configure". Click on the "Binding" tab and select sapBinding from the Binding Type drop-down. The configuration window will now display. This will look familiar to users of the mySAP adapter. The maxConnectionPerSystem setting is the one we are concerned with. In the specific instance above, we found a "sweet spot" of 25 connections. You will have to work with your SAP administrators to determine what this number will be. Make sure to consider other systems that connect to SAP (it is the SAP connection pool that we are trying to avoid overwhelming). Some trial and error are involved. Make sure to perform some high volume load testing to ensure that SAP is not being overwhelmed.

Microsoft’s Intentions for “Dublin”

2008-10-21T09:41:00.001-05:00

Microsoft .NET Framework 4.0 release will be adding significant functionality to the development platform. Among these is a large enhancement codenamed "Dublin" that will focus on improving scalability and manageability of Windows based applications and provide a standard host for applications by extending IIS.

“Dublin”, in a nutshell will:

Provide standard host for WF and WCF applications
Provide pre-built developer services
Support message-based correlation and content-based message routing
Contain a message forwarding service
Offer a compensation service for long-running transactions
Enable scale-out of stateful workflow applications
Persisting and rehydrating state for high scalability
Enhanced management and monitoring functions
Tracking store for workflow events
Supports “Oslo” modeling platform

You may read those bullet points and say “hey, this sounds like BizTalk”. Well, yes and no. The easiest way to view this is that some BizTalk functionality will be available within the .NET Framework.

“Dublin” will be available for download when released and will eventually be included in future releases of Windows Server. “Dublin” will be fully supported thru current support contracts.

“Dublin” will be the first Microsoft server product to deliver support for the “Oslo” modeling platform. “Dublin” does not require “Oslo” in order to operate but administrators will be able to deploy applications from the “Oslo” repository directly to the “Dublin” application server. “Dublin” provides model-driven “Oslo” applications with a powerful runtime environment, out of the box.

The intention is for “Dublin” based applications to interoperate with BizTalk based services. The integration server and application server workloads are distinct but complementary so they can be deployed separately as needed. The intention is for “Dublin” to be an application platform, BizTalk an integration platform. When they need to work together, they will be able to. Otherwise, requirements will drive which platform to use.

New Features in BizTalk 2009

2008-10-02T10:35:00.000-05:00

Details about the upcoming release of BizTalk 2009 are emerging. The new version was initially to be called BizTalk 2006 R3 but the development team felt that it was a significant enough step forward to warrant an change of the product name.

Among the highlights of the new release are:

Support for Windows Server 2008, Visual Studio 2008, SQL Server 2008 and also the .NET Framework 3.5
Support for Windows Server 2008 Hyper-V which is a much higher performance platform for creating virtual machines
BizTalk 2009 takes advantage of Windows Server 2008 clustering so you can now deploy BizTalk Server so cluster nodes could reside on separate IP subnets and avoid complicated VLANs
BizTalk Server 2009 includes a UDDI 3.0 registry which provides support for registry affiliation, extended discovery services, digital certificates and extensibility for a subscription API
New Line of Business Adapters are introduced for Oracle E-Business Suites and SQL Server
Additional performance improvements have been made to the existing set of adapters
Enhanced Business Activity Monitoring- By expanding the out of the box BAM functionality with SQL Server 2008 Analysis Services, BizTalk Server 2009 provides support for UDM cubes and scalable real-time aggregations which enhances support for Microsoft PerformancePoint Server 2007
ESB Guidance 2.0 delivers updated prescriptive guidance for applying ESB usage patterns
Enhanced Support for EDI and AS2 Protocols
BizTalk Server 2009 updates all message schemas and business rules for compliance with SWIFTReady Financial EAI Gold certification
New Mobile RFID Platform and device management
New RFID industry standards support for LLRP, TDT, TDS, WS Discovery and partial EPCIS support
BizTalk Server 2009 improves recoverable interchange processing of validation failures by providing support for recoverable interchange processing
The WCF Adapter has been enhanced to provide support for configurable transactions and the ability to choose the transaction isolation level in the WCF-Custom Send Adapter.

Introductory BizTalk Seminar

2008-10-01T14:09:00.001-05:00

RDA is hosting a free ½ day seminar on EAI/BizTalk in the Atlanta and Philadelphia areas. We will be giving an overview of BizTalk features, and providing a summary of several implementations we've deployed.

If your company faces challenges in getting new trading partners integrated into your existing systems – let RDA help you streamline your systems, processes and people.

The target audience for this event is IT management of firms facing this challenge. For information or to register, please check out our website at http://www.rdacorp.com/getting_started/events.html#Enterprise_Application_Integration_/_BizTalk_Seminar_Series

The Easiest Way to Try and Learn BizTalk: Virtual Labs

2008-09-25T14:08:00.000-05:00

Microsoft has a great way to experiment with BizTalk Server. By using a virtual online environment, you can login to a session on remotely located server and use BizTalk without any of the complexities of setting up an environment yourself.

In terms of getting up to speed, there is no faster way to do it. Labs are guided, well structured and guided. Each lab builds on the last to give a user top to bottom exposure of BizTalk Server. All labs are in English and last about 90 minutes. If your session goes longer than 90 minutes, you can certainly extend this time.

You can give the BizTalk 2006 Virtual Labs a try at this address:
http://www.microsoft.com/biztalk/en/us/virtual-labs.aspx

Here is a list of the current labs available, 26 in total!

MSDN Virtual Lab Express: What's New in BizTalk Server 2006
MSDN Virtual Lab: Microsoft BizTalk Server 2006 Working with Schemas
MSDN Virtual Lab: Microsoft BizTalk Server 2006 Working with Maps
MSDN Virtual Lab: Working with Pipelines
MSDN Virtual Lab: Processing Flat Files
MSDN Virtual Lab: New Features for Processing Flat Files and Batch Messages using BizTalk Server 2006
MSDN Virtual Lab: Integration with POP3 and SharePoint and Routing Failed Messages
MSDN Virtual Lab: Creating an Orchestration
MSDN Virtual Lab: Implementing Transactions
MSDN Virtual Lab: Integrating Business Rules in Microsoft BizTalk Server 2006
MSDN Virtual Lab: Enabling Business Activity Monitoring
MSDN Virtual Lab: New Features for Deploying and Managing a BizTalk Application in Microsoft BizTalk Server 2006
MSDN Virtual Lab: WCF Introduction - Building a WCF Service
MSDN Virtual Lab: Using the WCF Adapters in BizTalk Server 2006 R2
MSDN Virtual Lab: Using the BizTalk Server 2006 R2 WF and WCF BAM Interceptors
MSDN Virtual Lab: Building a Windows Communication Foundation (WCF) Adapter using the WCF LOB Adapter SDK
MSDN Virtual Lab: Processing EDI messages with BizTalk Server 2006 R2
MSDN Virtual Lab: Sending EDI Messages with BizTalk Server 2006 R2
MSDN Virtual Lab: Building an RFID application with BizTalk Server 2006 R2
MSDN Virtual Lab: Integrating Legacy Data with BizTalk Applications using BizTalk Adapters for Host Systems
MSDN Virtual Lab: Integrating Legacy Data with .NET Framework applications using Host Integration Server 2006
MSDN Virtual Lab: Integrating Legacy Applications with BizTalk Applications using BizTalk Adapter for Host Systems
MSDN Virtual Lab: Integrating Legacy Applications with .NET Framework applications using Host Integration Server 2006

Tracking Events In Real Time Using BizTalk 2006

2008-09-11T15:53:00.002-05:00

Recently I was tasked with creating a solution within BizTalk 2006 to track real-time events. The scenario did not sound unique and with the RFID functionality built into BizTalk 2006 R2, the situation seems likely to become more pervasive. Here is a stripped down sample of the solution that I arrived at Sample Timer Application

The Problem:

In this sample scenario we have the following set-up:

We have a warehouse in Death Valley, CA where we store pallets of ice cream. Some areas of the warehouse are refrigerated, some are not. The areas that are not refrigerated have an average temperature of over 100 degrees Fahrenheit. Ice Cream has a life expectancy of 20 minutes when left outside of a refrigerated area.
Pallets are equipped with RFID tags that identify the pallet by a unique Pallet ID.
Gateways between refrigerated areas and un-refrigerated areas are equipped with RFID readers. When an ice cream pallet passes through a gateway a message is sent to BizTalk indicating a change in status between refrigerated/exposed.
The external loading dock door is also equipped with an RFID reader. Passing through this portal indicates that the pallet has left the warehouse.
We need to be able to send an alert to the warehouse foreman to alert him when a pallet has been left exposed for 15 minutes so that he can take action before the pallet is ruined.

Based on this scenario the following requirements were compiled:

When a message is received indicating that a pallet has changed status from refrigerated to exposed, a 15 minute timer should be started.
If the pallet is returned to refrigerated status or leaves the warehouse on a truck the timer should be terminated to prevent false alerts from executing.
If the timer expires without being terminated, an alert message should be generated by the system.

In order to solve this problem the following design was constructed:

Trigger messages are received from the Message Box (through direct binding) by the Timer Orchestration. Depending on the trigger type (Create, Reset, or Terminate) a corresponding stored procedure is called via a SQL send port. Trigger messages contain the following fields:

Timestamp - Timestamp of the message. In our solution, this timestamp is used as a base time in the following formula. BaseTime + TimerLength = TimerEndTime. TimerEndTime is the timestamp contained in the database that indicates when a timer should expire.
InterfaceName - Namespace of the trigger type. Used for routing.
Action - The action we are trying to perform (CreateTimer, ResetTimer, or TerminateTimer).
SubAction - Unique name for this timer.
Params - Collection of Key-Value pairs for future processing. (Note: the Create and Reset Timer Triggers require a TimerLength key with an integer (timer length in minutes) in order to process. Any number of Param nodes can be added to this collection. (e.i. in our example brand, flavor, and expiration date could be included, to name a few).

SQL send ports are used to execute one of the three actions on the database:

Create - Creates a record in the database. Uses spCreateTimer stored procedure.
Reset - Updates a specified record in the database to give it a new TimerEndTime, if the timer has not expired. Uses spResetTimer stored procedure.
Terminate - Updates a specified record in the database to flag it as expired. Uses spTerminateTimer stored procedure.

A SQL receive location is set up in BizTalk to poll the results of the spGetExpiredTimers stored procedure. This stored procedure aggregates timers from the database whose TimerEndTime timestamps are in the past. Timers that are flagged as expired are ignored. Timers collected are flagged as expired to prevent duplicate processing.

"BizTalk 2009" Announced

2008-09-09T08:01:00.000-05:00

This month, Microsoft has announced the upcoming release of BizTalk Server 2009 coupled with plans for future releases.

Originally the next release of BizTalk was slated to be named BizTalk Server R3 but has been renamed “BizTalk Server 2009”. Microsoft wanted everyone to make sure it was understood that this is a full product release with new and enhanced capabilities and updated platform support for Windows Server 2008, Visual Studio 2008, SQL Server 2008, and the .NET Framework 3.5.

A CTP release is expected by the end of this year. The formal release date is expected to be around June of ‘09.

Microsoft also plans to maintain a rhythm of BizTalk releases every 2 years.

Creating a BizTalk 2006 R2 VPC Template for Use in MS System Center Virtual Machine Manager

2008-08-29T12:21:00.000-05:00

I was tasked with the objective of creating a way for the EAI group to "mass-produce" BTS2006 R2 servers on the corporate Virtual Server Self-Serve portal for development purposes. Note that before embarking on this task, make sure that you or your company has the appropriate licenses (MSDN or other) to set this up.

In analyzing the requirements for this, I found out that I would need to create a VPC with all the software on it. This VPC would then be processed and imported into the self-serve portal as a "template" for all machines to be made from. However, I found out some interesting information about the process the VPC is run through as it's being made into a "template".

The VPC is first run through an application named SysPrep. This software "anonymizes" the OS. It removes the Windows activation code and also removes all the Security Identifiers associated with all files and accounts. This means that any "non-system" accounts created before SysPrep is run will not match the accounts after SysPrep is run. This brought up the major issue I ran into while trying to complete this objective. While configuring both BizTalk and Windows SharePoint Services (WSS), you need to provide accounts, which will be the owners of the associated processes. The issue would be that any account(s) I create and configure before SysPrep is run would not match the account(s) on each instantiation of the template, since the security IDs associated with the accounts wouldn't match.

During discussions about this issue, someone theorized about possibly installing all the software, but not configuring the ones that need specific accounts (BTS 2006 and WSS 2.0/3.0). That's when I tried going through the Microsoft Installation instructions for BTS 2006 R2 (http://www.microsoft.com/downloads/details.aspx?FamilyID=df2e8a88-fb23-49a4-9ac7-d17f72517d12&displaylang=en). In order to create a "template" for the Self-Service Portal, I created a VPC, with all the software products installed. I started with Windows Server 2003 (Enterprise). I changed the order in one aspect, in that they instruct you to install Windows 2003, then run Windows Update, and then install IIS. What I did was to install Windows Server 2003, then install IIS and the POP3 service, and then run Windows Update, so that I could get any update files specifically marked for IIS or the mail server.

After I did that, then I installed the Office Components. The instructions specified Excel and InfoPath, but I also included Word. However, one note on these components. During installation, I did not use a Product Key for Office. This was so that whoever needs to use the server can use their own Office product key.

I then installed Visual Studio 2005, and then service pack 1 for VS 2005. I did add a feature not listed in the instructions. The instructions indicate that you should unselect all features except Visual C# and .Net Framework SDK. I did keep the Visual Web Developer component, in case any web services needed to be included in any project.

At this point, the instructions indicate that SQL Server 2005 should be installed. Now, one option in the installation for SQL allows for the use of a separate account, under which SQL server will run. Another option allows for SQL Server to run under a "defined system account". I chose this option because, when the VPC/VHD is imported and converted to a template for use in the Self-Service portal, the SIDs are all stripped out of accounts, so an account created now would not match an account when the template is "instantiated" into a particular instance.

At this point, I added the additional software packages requested to be on the server, including NUnit and BizUnit.

The next step includes installing first, Windows SharePoint Services (WSS) 3.0, and then WSS 2.0. Both versions were installed because the BAM portal in BTS 2006 is only supported on WSS 2.0, but most project work moving forward will likely be done on WSS 3.0. Thus, if you want to have the BAM portal on the same box that you have a WSS 3.0 site (or MOSS 2007 site), you need to have both versions of SharePoint installed. Note that after the installation is completed, the instructions indicate that configuration is next. Because user accounts are involved in the create of Application Pools when using WSS on IIS, this step was "postponed" until a particular instance was created.

The last software package installed was BizTalk Server 2006 R2 itself. I went through all the steps and did all the installation, but on the last screen, where the checkbox indicating that the configuration wizard should run after setup is completed, I unchecked this so that it could be done on a per instance basis.

After everything was done, I defragmented the hard drive, ran the pre-compaction utility, and then compacted the drive. This allowed for a smaller VHD for import and cleaned up a lot of space in the VHD after all the updates had been made.

Mapping With External Assemblies Part 2

2008-08-29T07:21:00.003-05:00

Introduction

Part 1 - Inline scripting and external assemblies
Part 2 - XSLT Call Templates and custom extensions
Part 3 - Cascading Functoids

This is the second of three articles about BizTalk mapping with external assemblies. External assemblies or helper classes are used by BizTalk's mapping engine (XSLT) automatically, and they can also be used explicitly by the developer. Just as in an orchestration, a developer may find a requirement that the built in mapping tools ("functoids") do not address and in this case judicious use of a helper class can be very... helpful.

While these articles are not an introduction to mapping if you have a little experience with the BizTalk mapper and a little experience with XSLT you should be able to follow along.

Part 1, introduces inline C# scripting, its limitations, and also introduces the scenario of using an external assembly (helper class) automatically. Part 2, this article, demonstrates using a map's custom extension property, which allows us to explicitly access external assemblies. Part 3 will build on this discussion and highlight potential caveats about using cascading functoids; when the output of one functoid is the input to another functoid(s).

External Assembly Script Type Review

In part 1 we examined inline scripting with C# and pointed out that only the built-in XSLT engine namespaces can be used. We also showed that when using an External Assembly Script Type an Extension Object XML file is automatically created, which contains the external assembly references that the XSL will use.

The contents of the Extension Object Xml file that is made during the build looks like this:

<ExtensionObjects>
<ExtensionObject
      Namespace="http://schemas.microsoft.com/BizTalk/2003/ScriptNS0"
      AssemblyName="RDA.BizTalk.Learning.MapUtility,
          Version=1.0.0.0,
          Culture=neutral,
          PublicKeyToken=f7eb52812c0b3fa1"
      ClassName="RDA.BizTalk.Learning.MapUtility.Strings" />
</ExtensionObjects>

Custom Extension Xml Property

If we want to explicitly include external assemblies in our map we can assign the "Custom Extension Xml" Grid Property. Click on the background (grid) of your map in design mode to find the properties. The property represents an Xml file that is formatted as shown above, and it will contain only those assemblies that you wish to manually reference. As we will see, other assemblies referenced by External Assembly Script Types will be automatically appended to this file during the build.

I normally name my file "External Assemblies.xml". The following is an example. I usually leave the Namespace as the Namespace of the class. That way if I have one assembly (as I do below) I will have 2 unique namespaces. Remember, your external assemblies are probably already in the GAC.

<ExtensionObjects>
    <ExtensionObject
        Namespace="http://RDA.Corporate.EAI.Common.Helper.Methods.Timestamps"
        AssemblyName="RDA.Corporate.EAI.Common.Helper.Methods,
                      Version=1.0.0.0,
                      Culture=neutral,
                      PublicKeyToken=9f177c4f1a417459"
      ClassName="RDA.Corporate.EAI.Common.Helper.Methods.Timestamps"/>
    <ExtensionObject
        Namespace="http://RDA.Corporate.EAI.Common.Helper.Methods.DBUtility"
        AssemblyName="RDA.Corporate.EAI.Common.Helper.Methods,
                      Version=1.0.0.0,
                      Culture=neutral,
                      PublicKeyToken=9f177c4f1a417459"
        ClassName="RDA.Corporate.EAI.Common.Helper.Methods.DBUtility"/>
</ExtensionObjects>

Using the Referenced External Assemblies

We already know that unfortunately we cannot use these classes with inline C# scripting. However, we also know that ultimately the mapper is simply creating XSLT and calling the C# methods, and we can do the same.

Let's pretend, for example, that based on a meeting number, room number and timestamp that I need to retrieve a meeting record id. My DBUtility class contains the methods I need to use.

Add a reference for the RDA.Corporate.EAI.Common.Helper.Methods assembly.
Create and add External Assemblies.xml file (named to your liking) to your map project.
Assign the Custom Extension XML Grid Property.
Add a scriptoid to your map.
Set it's script type to XSLT Call Template.
Assign input parameters, and assign the output to the target schema node as required.

Here is my map, I'm going to enrich the original message by filling in the MeetingKey record. I'll be using an XSLT call template. Remember it is the XSLT that is driving the mapping within the scriptoid. See the next paragraph.

Here is my XSLT call template that I've assigned to my scriptoid shown above.

<xsl:template name="MeetingKeyCallTemplate">

    
    <xsl:param name="meetingNumber" />
    <xsl:param name="roomNumber" />
    <xsl:param name="meetingTimestamp" />

    
    <xsl:variable name="meetingKey"
        
        xmlns:script="http://RDA.Corporate.EAI.Common.Helper.Methods.DBUtility"
        
        select="script:GetMeetingKey($meetingNumber,
                       $meetingTimestampDateTime,
                       $roomNumber)"/>

    
    <MeetingKey>
        <xsl:attribute name="modified">Yes</xsl:attribute>

        <xsl:element name="id">
            <xsl:value-of select="$meetingKey"/>
        </xsl:element>
    </MeetingKey>

    
    <MeetingInfo>
        <xsl:element name="MeetingNumber">
            <xsl:value-of select="$meetingNumber"/>
        </xsl:element>

        <xsl:element name="RoomNumber">
            <xsl:value-of select="$roomNumber"/>
        </xsl:element>

       
       <xsl:element name="MeetingLeader">
            <xsl:value-of select="MeetingInfo/MeetingLeader/text()"/>
        </xsl:element>
    </MeetingInfo>

</xsl:template>

TIP: In part 3 we'll see similar code again and we will combine into one example the use of external assemblies, inline C# scripting and value caching in a discussion about cascading functoids.

Custom Extension XML and External Assembly Script Types

The BizTalk mapper will accommodate maps using both custom Extension XML file as described here and also uses External Assembly Script Types. In this case the Extension Object XML file that is used will combine your custom references along with the automatically added External Assembly Script Type references. Validate your map and check it out!

External Links from MSDN

External Links about BizTalk Mapper Performance

Understanding the BizTalk Mapper - Performance and Maintainability

Mapping With External Assemblies Part 1

2008-08-29T05:18:00.003-05:00

Introduction

Part 1 - Inline scripting and external assemblies
Part 2 - XSLT Call Templates and custom extensions
Part 3 - Cascading Functoids

This is the first of three articles about BizTalk mapping with external assemblies. External assemblies or helper classes are used by BizTalk's mapping engine (XSLT) automatically, and they can also be used explicitly by the developer. Just as in an orchestration, a developer may find a requirement that the built in mapping tools ("functoids") do not address and in this case judicious use of a helper class can be very... helpful.

While these articles are not an introduction to mapping if you have a little experience with the BizTalk mapper and a little experience with XSLT you should be able to follow along.

Part 1, this article, introduces inline C# scripting, its limitations, and also introduces the scenario of using an external assembly (helper class) automatically. Part 2 demonstrates using a map's custom extension property, which allows us to explicitly access external assemblies. Part 3 will build on this discussion and highlight potential caveats about using cascading functoids; when the output of one functoid is the input to another functoid(s).

Inline Scripting with C#

As you probably know, using C# within a map is very easy to do. Drag the scripting functoid to the map:

Right click on the scripting functoid and add your C# code:

Tip: In a previously saved map, if you switch between Script Types be sure to click the Reset button first, otherwise old settings (including C# code!) will stick around and the Script Type property won't necessarily change even though there is no warning.

Tip: You can display friendly names for your input parameters if you set the Label property for each link. View the link properties by clicking each link (line) in the map.	Unlabeled Links
Label Property	Labeled Links

Limitations of Inline Scripting

Once the map is completed we can validate it and view the generated XSL file. I encourage you to create a simple map, validate it, then look at the resulting XSL file... even if you know only a smattering of XSLT you can gain lots of insight into how the actual mapping is working (or not!).

If we look at the XSL file from the map above we discover that the BTS Mapper puts all of our inline C# code into a CDATA section. There is a screenshot below. Then within an XSLT statement our C# method is called. This is actually the key to the whole series of articles! In this case the BTS Mapper designed the XSLT extended syntax for us, but as you will find out in part 2 we can do it for ourselves!

So, why would we ever want to do manually what the BTS Mapper can do for us automatically? Well, if you reflect on the above paragraph you'll notice that our C# method is called from an XSLT statement. To highlight this point, below are the snippets from the XSL of our map:

This section is how the XSL inline C# is automatically defined within a CDATA section. Noticed the prefix "userCSharp" as in the future this will come in handy.

<msxsl:script language="C#" implements-prefix="userCSharp"><![CDATA[
    public string MyInlineSortKey(string str1, string str2)
    {
        const string comma = ",";
        return String.Concat(str1, comma, str2);
    }
]]></msxsl:script>

Here is the actual XSLT that is generated. A variable named "v1" is created and our method is called by passing the XPATH values of LastName and FirstName. The actual use of the C# method is very straight forward.
<xsl:variable name="var:v1"
              select="userCSharp:MyInlineSortKey(string(Customer/LastName/text()),
                      string(Customer/FirstName/text()))" />
<SortKey>
    <xsl:value-of select="$var:v1" />
</SortKey>

To come back to the question, why would we want to implement a call to a helper class manually? It's because there are limitations to what you can do with inline C#. Specifically, inline C# that looks like the following will not compile even if the MyMapUtility reference was added to the map project.

MyMapUtility.Strings stringUtil;
return stringUtil.MySortKey(str1, str2);

From MSDN (references at end):

BizTalk saves inline scripts in the Extensible Stylesheet Language Transformations (XSLT) stylesheet defining the map. Because of this, inline scripts may use the same namespaces as any other XSLT stylesheet script.

What this means is that even if you add the helper class reference to your map project, you will not be able to call your methods with inline scripting and the map project will not compile. Part 2 will cover how this can be accomplished using XSLT call templates. Here are the namespaces that are supported for inline scripting, so remember with inline C# scripting you can use these namespaces but no other:

Supported namespaces for inline scripting

Namespace Description

System The System class

System.Collection The collection classes

System.Text The text classes

System.Text.RegularExpressions The regular expression classes

System.Xml The core XML classes

System.Xml.Xsl The XSLT classes

System.Xml.Xpath The XPath classes

Microsoft.VisualBasic The Visual Basic script classes.

External Assembly Script Types

As we've examined, inline C# scripting only supports a certain set of namespaces. And in part 2 we'll look at custom XSLT templates that let use use any external assembly. However, there is functionality out of the box that the BTS mapper provides for using External Assemblies.

The main caveat is that the default constructor (parameterless) on the external class will be used. In some cases you may find yourself wanting to create a wrapper class created especially for mapping that handles a custom initialization of the actual helper class.

Knowing how the External Assembly and method call is incorporated into the map will help us in part 2 when we describe manually using an external assembly.

Here is a example helper class that is designed to return two strings separated by a comma. After creating the helper class add its reference to the Map project.

Typical helper class. This method will take two inputs and return a value:

namespace RDA.BizTalk.Learning.MapUtility
{
    /// <summary>
    /// String Utilities
    /// </summary>
    [Serializable]
    public class Strings
    {
        // A custom sort key
        //
        public string MySortKey(string str1, string str2)
        {
            const string comma = ",";
            return String.Concat(str1, comma, str2);
        }
    }
}

Assign the External Assembly to your scriptoid and pick your class method:

TIP: If you right click and "Test" a map that uses an external assembly, the assembly must already be added to the GAC, otherwise the test will fail.

Now let's examine what the mapper does with the external assembly to call it. We can do this by right clicking the map and picking "Validate". The output of the validation process is the XSL file and an Extension Object XML file.

The Extension Object XML file was created automatically based on all the external assemblies used in the map (in this case, just one). It contains the assembly reference that the XSL will use. It looks like the following:

Extension Object XML:

<ExtensionObjects>
<ExtensionObject
      Namespace="http://schemas.microsoft.com/BizTalk/2003/ScriptNS0"
      AssemblyName="RDA.BizTalk.Learning.MapUtility,
          Version=1.0.0.0,
          Culture=neutral,
          PublicKeyToken=f7eb52812c0b3fa1"
      ClassName="RDA.BizTalk.Learning.MapUtility.Strings" />
</ExtensionObjects>

Now, let's examine the XSL file. It turns out that the use of the external assembly will look almost exactly like how the inline C# code was called. The only difference is that in this case in the header of the XSL file a namespace alias is created for "ScriptNS0". I encourage you to validate your own simple map that uses an external assembly.

External Assembly XSLT:

<xsl:variable name="var:v1"
              select="ScriptNS0:MySortKey(string(Customer/LastName/text()),
                      string(Customer/FirstName/text()))" />
<SortKey>
    <xsl:value-of select="$var:v1" />
</SortKey>

The inline C# code XST from the previous section for comparison:

<xsl:variable name="var:v1"
              select="userCSharp:MyInlineSortKey(string(Customer/LastName/text()),
                      string(Customer/FirstName/text()))" />
<SortKey>
    <xsl:value-of select="$var:v1" />
</SortKey>

TIP: The "userCSharp" prefix is used for all of your inline methods. If you have your own inline XSLT transformations you can call previously defined inline C# methods yourself!

TIP: Though not necessarily a best practice it is possible to create a "floating" scriptoid (not connected to anything!) that just contains a few methods (yes, more than one) that you will call elsewhere in the map. You can call them from custom XSLT, and you can also call them from inline C#! In this manner you could have a GET and SET method to cache calculated values that are valid during the execution of the map.

TIP: A scriptoid must have a least one output to the target schema in order to be executed by the XSLT map. Otherwise, the map will still compile however the scriptoid will not execute.

External Links from MSDN

External Links about BizTalk Mapper Performance

Understanding the BizTalk Mapper - Performance and Maintainability

BizTalk Tracking Advantages

2008-08-18T15:24:00.000-05:00

BizTalk allows tracking to be configured at receive, send port and orchestration levels to monitor transaction activity, flow and to access message contents. As long as tracking is enabled at least on one host, all tracked transactions will use that host for tracking (by default, BizTalkServerApplication host is designated for tracking). So it’s not necessary that a tracking host be assigned in a specific application to receive, send ports and orchestrations. Reason being, a message agent that is running within each BizTalk host will be calling a stored procedure that searches through each of the messages in the Messagebox looking for messages with matching subscription information. To keep the Messagebox as empty as possible, the tracking subsystem will move completed messages to the Tracking Database on a periodic basis. The Tracking Database is where the HAT queries for data and displays its tracking information. This is accomplished by a BizTalk host that has been tagged as a “tracking” host.

For most BizTalk customers, tracking data is very important for the following reasons:

To comply with federal guidelines Sarbane-oxley, IRS and such
To satisfy accounting department requirements to generate reports for business users, for example using BAM tracking
In the case of failures and such, to help recover from a disaster, because it is a record of data processing activity
Data in the tracking databases can be used to help determine the state of the system up to the point of failure for the runtime databases
Tracked messages and events can indicate what processes may have already happened and what messages have been received or sent
Tracks the health and run history of BizTalk messages and orchestration processes and aids in identifying errors and bottlenecks in the BizTalk Server environment
Individual tracking information per message can correlate what types of transactions are taking the most time
To view the technical details of a particular orchestration, pipeline, or message instance, as well as to see the message flow of a particular message from end to end
Tracking facilitates troubleshooting of service instances and messages
It provides debugging capabilities, allowing breakpoints to be set on orchestration shapes
Business users can view, monitor, and query completed messages and processes via tracked data, saving these queries as custom views for reuse
Business analysts and end users can track the state of their business processes by viewing both live and archived data

Reference: Apress Pro BizTalk 2006 book

BizTalk performance issues

2008-08-11T11:07:00.002-05:00

Recently one of our customer experienced performance issues with BizTalk, where transaction queues were filling up and it was taking too much time to process normal transaction activity. So I went onsite, checked the obvious, found two familiar performance related issues...

Issue 1:Verfied built in SQL jobs history, found that DTA Purge and Archive (BizTalkDTADb) has not run on the production BizTalk server since September 2007. BizTalk tracking data file was about 1.6GB, obviously, which results in performance degradation. Analysis: DTA Purge and Archive SQL job is expected to run every day (24 X 7) about 15-20 minutes apart. As time goes, the BizTalkDTADb database will grow in size and unchecked growth will result in poor performance. Possibly having an adverse impact on message box database performance as well...

Resolution: Followed the steps detailed in this MSDN article:
http://technet.microsoft.com/en-us/library/aa560754.aspx
exec dtasp_BackupAndPurgeTrackingDatabase
0, --@nLiveHours,
30, --@nLiveDays
35, --@nHardDeleteDays
‘\\D:\SQL backup’, --@nvcFolder
null, --@nvcValidatingServer
0 --@fForceBackup Soft purge process
After performing the steps above (once the tracking data is purged) "DTA Purge and Archive (BizTalkDTADb)", set the SQL Job is enabled and is running at every 20 minutes. Database size came down to 30 MB from 1.6 GB!
To Understand DTA Tracking performance behavior, refer to these MSDN article:
http://msdn.microsoft.com/en-us/library/cc296718.aspx

Issue 2: There were three BizTalk hosts in the production environment including the default BizTalk host instance and other two being txHost1 and txHost2. All the three host instances had tracking turned on.
Analysis: According to BizTalk documentation, "Allow host tracking" check box must be enabled on a single host instance and ideally on the default in-process BizTalk Application Host. "Allow host tracking" should not be enabled on any other Host instances.
Resolution: Removed tracking from txHost1 and txHost2 hosts, kept it unchanged on the BizTalk Application Host and restarted all the three hosts.
Creayed and configured three new host instances such as orchHost for all orchestrations, rcvHost for all receive locations and sndHost for all send ports. Tested on the QA server with 1577 transactions, it took about 17 minutes to process all transactions, thus processed about 100 transactions a minute, improving the performance significantly...

For more info on tracking host: http://msdn.microsoft.com/en-us/library/cc296878.aspx

Consuming a Java Web Service (JWS) Method Containing String Arrays as Parameters

2008-08-04T15:32:00.005-05:00

Consuming a Java Web Service (JWS) Method Containing String Arrays as Parameters

Recently I was working on a POC where one of the tasks was integrating with a Quality Control application that exposed a web service built with Java. The application used a JWS to expose web service methods that were RPC encoded. BizTalk Server 2006 and Visual Studio 2005 natively recognized and imported the JWS WSDL and generated a multi-part message in the BizTalk project.

Early in the POC, consuming, connecting and executing the process were demonstrated using a JWS method called, Sample Log. There was no issues importing the JWS WSDL but the challenge was populating the message since it used string arrays for parameter names and values.

Multi-part Message Types
The multi-part message type generated by importing the SampleLog WSDL are the last invoke_request and invoke_response in the figure below. We will take a closer look at the invoke_request message because the fieldsArrayC and valuesArray C items for the invoke_request message are actually string arrays. This message structure presents a challenge for BizTalk because although this message type was successfully generated in BizTalk, it was not available for use within the Mapper.

Using and populating the message structure
In order to use and populate the mutli-part message in BizTalk, 3 three messages needed to be defined to get around the limitations of the mapper.
These are:

msgLIMSSampleLogRequest - Message that will be sent to the JWS.
msgLogSampleReferenceNameArray – An array that contains the names of the parameters passed. This message structure will be added to the msgLIMSSampleLogRequest message.
msgLogSampleReferenceValueArray – An array that contains the values of the parameters passed. This message structure will be added to the msgLIMSSampleLogRequest message.

Create Message Variables

Orchestration Snippet
Because we can’t use the mapper, an orchestration must be used. In the orchestration, the msgLogSampleReferenceValueArray and msgLogSampleReferenceValueArray needed to be populated using the following code in the Expression Shape of the orchestration.

Once msgLogSampleReferenceValueArray and msgLogSampleReferenceValueArray messages were populated, they could be assigned to the msgLIMSSampleLogRequest message as described in the next section.

NOTE: This code is for demonstration purposes only. An actual implementation would use a component or other solution to populate the array depending on the requirements.

System.Diagnostics.Debug.WriteLine(System.String.Format("{0}{1}: {2}", strInterfaceName, "BuildReferenceArrays", ""));

//Format and Populate ReferenceName
strParameters = System.String.Format("{0},{1},{2},{3},{4},{5},{6},{7}", "LOT", "LOT_NAME", "PRODUCT_GRADE", "DESCRIPTION", "ROLL", "SAMPLING_POINT", "SPEC_TYPE", "GROUP_NAME");
System.Diagnostics.Debug.WriteLine(System.String.Format("{0}{1}: {2}", "strParameters", "strParameters", strParameters));

objXmlDoc1 = new System.Xml.XmlDocument();
objXmlDoc1.LoadXml
(@"

LOT
LOT_NAME
PRODUCT_GRADE
DESCRIPTION
ROLL
SAMPLING_POINT
SPEC_TYPE
GROUP_NAME

");

msgLogSampleReferenceNameArray = objXmlDoc1;

System.Diagnostics.Debug.WriteLine(System.String.Format("{0}{1}: {2}", strInterfaceName, "ReferenceNameArray", objXmlDoc1.InnerXml));

//Format and Populate ReferenceValue
strParameters = System.String.Format("{0},{1},{2},{3},{4},{5},{6},{7}", "89210", "1000222", "40786351", "S 901/ERL 1908 HM", "001", "6351ANYL", "SP_1", "WINONA");

objXmlDoc2 = new System.Xml.XmlDocument();
objXmlDoc2.LoadXml
(@"

89210
1000222
40786351
S 901/ERL 1908 HM
001
6351ANYL
SP_1
WINONA

");

msgLogSampleReferenceValueArray = objXmlDoc2;

System.Diagnostics.Debug.WriteLine(System.String.Format("{0}{1}: {2}", strInterfaceName, "ReferenceValueArray", objXmlDoc1.InnerXml));

Populate Main Message (LogRequest)
The final step is to populate the msgLIMSSampleLogRequest message in an Expression shape. Within the Expression shape the fieldsArrayC element is loaded with the msgLogSampleReferenceNameArray and the valuesArrayC element is loaded with the msgLogSampleReferenceValueArray.

The following is the code snippet that loads the message. The highlighted rows are where the string arrays from the previous step are actually loaded into the main message.

msgLIMSSampleLogRequest.authToken = msgLIMSAuthenticate1Response.authenticate1Result;
msgLIMSSampleLogRequest.templateNameC = "SPEC_IPL";
msgLIMSSampleLogRequest.fieldsArrayC = msgLogSampleReferenceNameArray;
msgLIMSSampleLogRequest.valuesArrayC = msgLogSampleReferenceValueArray;

New BizTalk Server and Enterprise Single Sign-On Custom Configuration Whitepaper Available

2008-08-04T13:15:00.000-05:00

We’ve published a new whitepaper on our RDA corporate website dealing with the issue of how to store custom configuration information on an enterprise level, and for large numbers of different projects.

Over time, we’ve tried a number of different scenarios to solve this need and have settled on a useful approach by using Enterprise Single Sign-On. There is a need for a central repository for configuration information to avoid multiple copies of configuration information stored on specific servers and to increase security.

The whitepaper, titled Microsoft BizTalk Server and Enterprise Single Sign-On Custom Configuration discusses topics surrounding custom configuration within SSO with the intention of providing understanding and a reference point on utilizing it.

You can also read our other RDA whitepapers.

RFID tags now work on metal

2008-07-24T17:19:00.002-05:00

There has been a long standing rumor in the marketplace that RFID tags are not compatible with metal surfaces. While this may have been true at one time, new developments in the technology make this a problem of the past.

A number of vendors now make metal mount RFID tags, this month ODIN technologies has published a benchmark study of different products that are now available in the marketplace which make metal mounting possible.

Many companies who have wanted to utilize RFID have hedged because of this problem. The tags in the study were read in operational settings where metal surfaces such as shelves or server racks were present and potentially inhibit successful tag reads. You can see the study here. In short, the following tags were tested and deemed appropriate for adhesion to metal surfaces:

Avery Dennison: AD-900, AD-902, AD-908
Confidex: Halo, Ironside, Steelwave
Emerson & Cummings: Ecopad
Intermec: Large Rigid, Small Rigid
Omni-ID: Flex, Micro, Mini
Sontec: C0101, P01016BTTROI: MMT-3001, MMT-3004, PC-102

You can read the report here:

Metal Mount RFID Tags Benchmarked By ODIN Technologies

How can BizTalk and RFID work together?

2008-07-22T07:16:00.001-05:00

Radio Frequency Identification tags (RFID) are becoming widely adopted by business as a cost cutting tool. BizTalk Server is also widely adopted for its ability to integrate systems and by proxy cut costs. BizTalk is ideal for RFID solutions to transition the physical event to a digital process in an automated way. Let’s take a look at a real world example of how they can work together.

BizTalk interfaces with RFID in one of 2 ways:

At the moment an RFID tag is printed
At the moment an RFID tag is read

BizTalk initiates and correlates a print job

Let’s say you manufacture computers, and your computers are made to order. You want to make sure your customer gets the computer they actually ordered, so it’s important to identify it.

So your customer places an order on your B2B website. A BizTalk workflow is initiated that gathers up all the requirements for the order and passes it on to the ERP system for fulfillment. Once the ERP system shows the order fulfilled, BizTalk is then tasked with tagging the machine for delivery. BizTalk initiates the printing of the RFID tag and correlates it to that specific order; the printed sticker is attached to the new computer.

RFID and BizTalk initiate the delivery cycle

So the new computer is built, checked and packaged for delivery. A truck arrives at the loading dock, and a forklift places a load of orders on a palette and moves it to the loading bay. While in the process of loading the palette on the truck the passive RFID tags pass under an RFID reader.

This action is sensed by BizTalk because the RFID reader is attached to a BizTalk server which is ‘subscribed’ for listening to messages from that reader and kicks off an advanced shipping notice message to the client. At the same time, BizTalk checks to make sure that the RFID reader’s location matches the truck in that location to make sure the correct truck is delivering the order. The advanced shipping notice is received by the customer to notify them their order is on its way, and the invoicing process begins.

Uh oh, one year later, computer breaks

So your company has a great warranty plan and unfortunately something went wrong with that computer that was purchased a year ago. How can RFID be of help?

Well, your call center logs a call and BizTalk receives a message that this computer (with a known RFID tag) will be sent to the service center to be repaired. Eventually it will arrive and when it does, the RFID reader at the shipping dock will recognize the tag and notify BizTalk of its arrival. BizTalk updates the call center’s call with this status. The customer can now know the computer arrived for repair.

A technician is assigned to the repair, and BizTalk passed the details of this computer to that technicians work queue.

BizTalk waits for the repair to take place and notifies the call center when it’s complete. The computer is working again and needs to be sent back to the client.

The RFID tagged computer goes under an RFID reader when it’s being loaded and same ‘workflow’ that happened when the computer was originally sent takes place with an advanced shipping notice and, since its under warranty, a 0$ invoice is created.

Automation

If you’ll notice, all the steps here took place with almost no human intervention. There was no person at the loading dock checking off shipments, nobody faxing invoices or shipping notices, and nobody needed to reconcile the shipments at the end of the day.

If it were just BizTalk being used, some level of automation would be present, but with the combination of BizTalk and RFID, it’s possible to automate it all and that is where using them together is such a cost saver: less human interaction means less man-hours and fewer mistakes.

Microsoft acquires Covast's B2B technology

2008-07-15T15:12:00.000-05:00

Microsoft has acquired Covast's advanced B2B technology to make it available as part of upcoming BizTalk Server 2006 R3 release.

The acquisition is expected to merge into R3 as superset of pieces to BizTalk Server 2006 R2's EDI component.

R3 will retain the current R2 functionality but will be more tightly integrated with Covast's reporting capabilities and adapters. Covast's B2B expertise will be expected to add value in specific verticals such as automotive, energy, manufacturing. Let all your Covast clients know.

http://www.covast.com/news/press.asp?id=91

How To Expose the BTS ReceiveLocationName Property

2008-07-09T15:42:00.005-05:00

There are some BizTalk properties that are not exposed in any property schemas from BTS 2006 R2. If your development project is such that you can add a new property schema or add a new BizTalk project then it is possible to access these properties from an orchestration. Because they are context properties they are not available for routing.

One such property is the ReceiveLocationName property. While the ReceivePortName property is available in the BTS namespace the ReceiveLocationName property is not. Therefore, this syntax from an example orchestration expression shape results in a compile time error: strReceiveLocation = myMsg(BTS.ReceiveLocationName);

ReceiveLocationName Property

We can easily see the ReceiveLocationName property in messages that come into BizTalk. The question is how to get its value into an orchestration?

Code Example

As you probably know, property values are identified by the property target namespace and property name. We won't ultimately use the following code, but for demonstration purposes in a custom pipeline component we can assess a property value by coding something like this:

// Namespace of the property
//
string sysNamespace =
"http://schemas.microsoft.com/BizTalk/2003/system-properties";

// Or instead of hard coding the namespace:
// Assuming the property namespace for ReceivePortName is the same
// for ReceiveLocationName:
//
BTS.ReceivePortName BTSPropReceivePortName = new BTS.ReceivePortName();
sysNamespace = BTSPropReceivePortName.Name.Namespace;

// Assign the receive location name
//
string receiveLocationName =
pInMsg.Context.Read("ReceiveLocationName", sysNamespace);

You can see from the last line of code from above, that to retrieve a message property requires nothing more than knowing the property name and the property target namespace.

Adding a Property Schema

As far as BTS message properties go, they can be assigned Property Schema Base types of MessageDataPropertyBase (default) or MessageContextPropertyBase. MessageDataPropertyBase means that the XPath to the property value in the message must exist. A MessageContextPropertyBase property means that the XPath may or may not exist.

Taking this information into account we can expose the RecieveLocationName using the following steps. Note steps 1 – 4 are just suggestions.

1. Create a property schema, and perhaps create a new BizTalk project.

2. Add a new property schema. Let’s call it “schBTSExtra.xsd”.

3. Change the namespace of the .NET schema type to “BTSExtra”. This is just a shortened namespace similar to the “BTS” namespace in the Microsoft.BizTalk.GlobalPropertySchemas assembly.

4. Open the new schema if it is not already opened.

5. Change the name of Property1 (it was added by default when creating the new property schema) to ReceiveLocationName.

6. Set the Property Schema Base to MessageContextPropertyBase.

7. Highlight the “Schema” node and change the schema’s Target Namespace to
http://schemas.microsoft.com/BizTalk/2003/system-properties

8. Compile the project.

Property Schema Example

That’s all there is to it… now in your orchestration after referencing the new schema project (or you added this schema directly to your orchestration project) you can use an expression such as:

// Trace output from an orchestration expression shape
//
traceMsg = purchaseOrder(BTSExtra.ReceiveLocationName);
if (traceMsg == null)
{
traceMsg = "<null>";
}
System.Diagnostics.Trace.WriteLine("Receive Location Name: " + traceMsg);

Here is the result when testing my own orchestration:

External Links from MSDN

Property Schema Base (Node Property of Property Schemas)

RecieveLocationName missing and a workaround

2008-07-09T07:46:00.000-05:00

I've had the situation recently where I wanted to set up a receive port and have a large number of receive locations feeding into this port. In normal scenario this works fine but in my case it did not and had to find a workaround for it.

The reason the scenario above did not work is because I needed to know the receive location's name that the message came in on and the only property available from the context within an orchestration is BTS.ReceivePortName which is the same for all the locations.

I did some digging and for some alternatives, one was a custom pipeline which found and promoted the property "ReceiveLocationName", the second was a message context functoid on CodePlex which could retrieve this value from within a map. I didn't like either option and tried to widdle the code from the functoid into something I could use in a small helper class method or even the expression shape. I got the code to work but could never seem to find the ReceiveLocationName property I'd seen mentioned on other blogs. My guess is that it only appears with certain adapters but I couldn't even find it in any of the BizTalk schemas.

I decided to make a U-turn from this custom approach because I was just replacing a large number of receive ports with custom code. So instead of the custom approach I'd been working on I decided to make the initial receive shape in the orchestration direct bound and create 1-n ports which I could leverage in the orchestration without knowing the ReceiveLocationName.

Pipeline Component Property Promotion

2008-06-27T07:02:00.002-05:00

I've often seen questions from new BizTalk developers asking why they couldn't subscribe to an incoming message. Typically, the question actually revolves around why were the message property values not promoted as expected? And sometimes, for the new developer, the answer is because he or she forgot to use the XML Receive pipeline component and instead used the Pass Thru Receive pipeline component for the message being received.

So, we take away from this scenario the fact that one job of a receive pipeline component (XML, Flat File, etc) is to promote property values from the message to the message context, which makes those values available to subscriptions. And notice the Pass Thru Receive pipeline is named appropriately... it passes a message through directly to the message box without any processing and since the message is not identified, even if it is an instance of a deployed schema, no properties from the message are promoted.

Custom Pipeline Property Promotion

Recently I was developing a custom pipeline component when I discovered that the message properties were not being promoted. Well, it was a little more complicated than that because this particular component encapsulated the Flat File Receive disassembly pipeline component. This part worked, because in this case the Flat File Receive component handled property promotion of the resulting Xml message.

The other piece of the puzzle was that I also created a second Xml message on the fly to be emitted from the pipeline. It was this second message whose properties were not promoted correctly or at all. To add another layer of complexity this additional message could be one of a handful of different types. I wanted to dynamically promote its properties since not having to adjust a custom pipeline due to schema changes in the future makes for a robust solution.

Drill Down into Property Promotion

I realized that is up to my custom pipeline component to promote the properties for that second message since it is being created on the fly. If you've worked with custom pipeline component code before you'll think why not just use the IBaseMessageContext methods to promote properties? Well, that's the mechanism to actually affect the promotion but it is only part of the answer because those methods won't allow discovery of what field names are linked to the property schema(s).

Remember that when you create promoted properties for your message you are linking an XPath from your message to a property name (element) in a property schema. So not only do we need to discover what promoted (and distinguished) properties are available for this second message being created, we also need to know what the XPath is to actually promote the value from the message to the message context!

This is where the IDocumentSpec interface comes in. This interface contains the GetPropertyAnnotationEnumerator and GetDistinguishedPropertyAnnotationEnumerator methods that will give us a list of both the promoted and distinguished property names. From this list we can also determine the XPath to those values in our message.

In addition to the BizTalk references needed to compile a custom Pipeline Component, you will need an additional reference to Microsoft.XLANGs.RuntimeTypes.

This assembly contains the IPropertyAnnotation interface and XsdDistinguishedFieldDefinition definition. Ultimately it is these types of objects that are returned from the DocumentSpec "get" methods.

The "XPath" property contains the XPath to the value in the message which we will promote (or write) to the message context! The "Namespace" property contains the property schema namespace. As you'll see in the example below these values are used with the Promote() and Write() methods of the pipeline context.

Property Promotion Implementation

Creating a custom pipeline component is outside the scope of this post, though I've included some external links below. In our following example we retrieve the DocumentSpec object mentioned above from the pipeline context (IPipelineContext), which is provided to us during pipeline execution.

Since the documentation is not abundantly clear, I'll point out that the GetDocumentSpecByName() is passed the Strong Assembly Name (AssemblyQualifiedName property) of the schema type. The GetDocumentSpecByType() is passed the BizTalk MessageType (target namespace + "#" + root node name).

Once we have the DocumentSpec in hand then it's a relatively straightforward matter of stepping through the enumerated properties and promoting/writing them to the context of our message.

Code Snippet

// Instead of hard-coding namespaces for various built-in
// BTS properties, here is an idea that caches the information
// the first time the the pipeline component runs.
//
private struct BTSProperties
{
    public static BTS.InterchangeID interchangeId = new BTS.InterchangeID();
    public static XMLNORM.DocumentSpecName documentSpecName = new XMLNORM.DocumentSpecName();
    public static FILE.ReceivedFileName receivedFileName = new FILE.ReceivedFileName();
    public static BTS.MessageType messageType = new BTS.MessageType();
    public static BTS.SchemaStrongName schemaStrongName = new BTS.SchemaStrongName();
}

[ ... ]

// msgAuditTrail
//     Second message created on the fly,
//     an instance of IBaseMessage.
//
// xmlMsgAuditTrail
//     Xml document instance.
//
// pContext
//     IPipelineContext - provided to us during
//     pipeline execution.
//
// auditTrailAssemblyQualifiedName
//     String value for the schema type.

IDocumentSpec docSpecAuditTrail =
    pContext.GetDocumentSpecByName(auditTrailAssemblyQualifiedName);

// Write document type to the message context
//
msgAuditTrail.Context.Write(BTSProperties.documentSpecName.Name.Name,
    BTSProperties.documentSpecName.Name.Namespace,
    docSpecAuditTrail.DocSpecStrongName);
msgAuditTrail.Context.Write(BTSProperties.schemaStrongName.Name.Name,
    BTSProperties.schemaStrongName.Name.Namespace,
    docSpecAuditTrail.DocSpecStrongName);

// WRITE DISTINGUISHED VALUES
//
// Iterate through each distinguished property and promote it from the message.
// Write distinguished properties prior to promoted properties. This is because
// if a promoted property is also distinguished it won't be promoted if the
// "write" is used last.
//
IEnumerator annotations =
    docSpecAuditTrail.GetDistinguishedPropertyAnnotationEnumerator();

if (annotations != null)
{
    while (annotations.MoveNext())
    {
        DictionaryEntry de = (DictionaryEntry)annotations.Current;

        XsdDistinguishedFieldDefinition distinguishedField =
            (XsdDistinguishedFieldDefinition)de.Value;

        // Use annotation.Name or Namespace to get the data to
        // promote from message
        //
        System.Xml.XmlNode contextNode =
            xmlMsgAuditTrail.SelectSingleNode(distinguishedField.XPath);

        if (contextNode != null)
        {
            // "Write" distinguished fields.
            //
            msgAuditTrail.Context.Write(de.Key.ToString(),
                Microsoft.XLANGs.BaseTypes.Globals.DistinguishedFieldsNamespace,
                contextNode.InnerText);
        }
    }
}

// WRITE PROMOTED VALUES
//
// Iterate through each promoted property and promote it from the message.
// Write distinguished properties prior to promoted properties. This is because
// if a promoted property is also distinguished it won't be promoted if the
// "write" is used last.
//
annotations = docSpecAuditTrail.GetPropertyAnnotationEnumerator();

if (annotations != null)
{
    while (annotations.MoveNext())
    {
        IPropertyAnnotation propAnnotation =
            (IPropertyAnnotation)annotations.Current;

        // Use annotation.Name or Namespace to get the data to
        // promote from message
        //
        System.Xml.XmlNode contextNode =
            xmlMsgAuditTrail.SelectSingleNode(propAnnotation.XPath);

        if (contextNode != null)
        {
            // "Promote" distinguished fields.
            //
            msgAuditTrail.Context.Promote(propAnnotation.Name,
                propAnnotation.Namespace,
                contextNode.InnerText);
        }
    }
}

External Links from MSDN

BizTalk Editor Global Types

2008-06-26T09:21:00.002-05:00

Following on to the BizTalk Xml Complex Types post here are some tips for working with Complex or Global Types in the BizTalk Schema Editor itself.

Creating a Complex Type

The BizTalk Editor is very message-instance-centric and typically you are working with a schema that already represents a message in whole or in part. To create a complex type, highlight a record node and type in a Data Structure Type value. Typically I will use a suffix of "Type" as seen below.

When you assign a value to the Data Structure Type property the BizTalk editor adds the "type" attribute to the element and also creates a definition for the type. This is important to note because if we delete the actual record node, the schema editor will look empty yet the Xml type we created still exists! In fact, when saving such a schema a prompt will be displayed asking about deleting unused schema definitions.

In this case our Xml record has no children because it is a base type, but further below we'll look at a more complex example.:

Saving a schema with unused data types results in a pop-up dialog being displayed:

Updating a Complex Type using the BizTalk Editor

Rename: To rename a global type, change the Data Structure Type property value. Caution should be applied if this global type is used in external schema definitions.

De-referencing: Let's say you've defined an Xml record as a global type but now you decide it'd be nice to copy all the elements from the global type but not reference it. This is easily done and is a nice feature of the BizTalk Xml Editor. Right-click on the Data Structure Type property and click "Reset" from the context menu. The property will now be empty and the fields from the global type will be copied to your record node as if you had typed them in. This tip works with other global definitions as well, such as Sequence Group references.

Deleting: Using the BizTalk Editor the only way to delete an unused Global Type is to remove its use from your schema and then save the schema. The type definition will then be deleted if you choose. See the above screen shot "Clean Up Global Data Types". You can also edit the XSD file manually or use the standard XML Schema Editor.

Example

In our example BizTalk application we process many different kinds of flat file messages and the first two rows of all the messages contain the same format. So, we would like to create a reusable global definition.

Row 1 contains a timestamp, and row 2 contains a "/" (slash) delimited record for Shipment ID and Transport Number.

The first step (see next image) is to create a schema to hold the header flat file definition. You can use the Flat File Wizard if you want to help define the schema for the first two rows. Afterwards, create a Sequence Group to encapsulate the fields and flat file attributes -- which I can use since all of my fields are elements. To create the global definition just type in a name for your Group -- in this case I used the name "HeaderContent". Note; For our purposes this schema is not a "Flat File Header Schema" as in flat file envelope processing, which is a whole other topic.

Next, create the document flat file schema and define everything except for the header content. The example below is for a fictional Delivery Confirmation document. Import the schema containing the reusable type and reference it in the document schema. Automatically the entire flat file header content that was previously defined is now part of Delivery Confirmation document.

External links from MSDN

BizTalk Xml Complex Types

2008-06-25T04:53:00.002-05:00

BizTalk operations are often designed to use an internal message representation, a canonical, for disparate data that represents the same logical piece of information. For example different trading partners might submit different Purchase Order formats, which are then transformed into Purchase Order canonical messages so that BizTalk services can process them consistently.

Sometimes there are other reasons for using a canonical such as the need to enrich the data (status, approvals, context metadata, etc) or to transform the message from non-Xml flat text data.

I recently worked on a project that involved processing many different types of messages. The schemas involved were anything from simple free text messages to well defined messages with both organized content and free form text. While the messages where physically different they all had a common business context. If there had been only a few different messages then a direct schema implementation might have been indicated, however we had almost 100 different message types and different services might subscribe to one or more of them. So, we needed a single canonical to easily represent any one of these different messages. This would also make the subscription/filter configuration easier to maintain and deploy.

How did we implement a single canonical schema to easily represent one of many different messages? The first idea was to try to normalize all the different fields in all the different messages and add the fields to the canonical. This works but the downside is that every time a message type is added or changed, the canonical would have to be updated. Another downside is that structurally it is difficult to organize the canonical schema to make sense from a business perspective and one looses the ability to eyeball the schema and understand where the data is coming from and how it is grouped.

We implemented our solution using Xml Complex Types and the resulting canonical schema required no updating if a message type was added or changed. First we created a base message content complex type from which all other content would be derived. Next we reviewed all the messages and decided how they might be organized from a business perspective. Next, we created the individual content by extending the base message type. Lastly, instead of adding individual messages to the canonical, we simply added the base message content complex type.

The screen shots below depict a schema for a fictional shipping company. Note; the project artifacts are simplified for discussion purposes.

The screenshot below shows the Air Content schema, which contains all the content types associated with shipping by air.

The Transport Canonical schema is shown below. The “Content” node is assigned the base content type. Since all the individual Ground, Air and Uplink content types are derived from the base type they are automatically part of the canonical schema. Note the Xml equivalent node; it contains all of the derived message types from the base Xml complex data type. From MSDN;

“Equivalent nodes are created automatically by BizTalk Editor to show how derived complex types can be used instead of the base complex type from which they are derived wherever the base complex type is called for in the schema. This yields the same type of polymorphism that is common in many object-oriented programming languages.”

The “Content” record properties.

Below is a screenshot of a canonical instance for the Air Weather Request message. During mapping we could easily map the name of the message to a promoted property (TransportMessageTypeId) if required.

BRE and XML Node Creation

2008-06-23T19:44:00.001-05:00

On a recent project, I was tasked with using Business Rules Engine (BRE) to call a list of BRE Policies. These policies could generate a number of results, or triggers, the number of results was not known until runtime. BRE can update existing XML nodes within the Fact document, however, it cannot create nodes that do not already exist.

To solve this issue, I decided to add a serializable .NET object to pass in as a fact. The object has a public method that is used to create trigger objects and add them to the object's collection. Upon return from the BRE call, the raw xml of the object is exposed through it's XmlString property.

First we will talk about the Trigger class. This class represents our result. It is very generic and thus flexible in its use. The TimeStamp field can be used to track the time the trigger was created. We use the InterfaceName field for routing purposes. The Action and SubAction fields can optionally be used to further differentiate the trigger from other triggers during routing. The other section of the Trigger is a collection of Parameters. This is where a list of key-value pairs can be stored for use in processing. We expose a method to add a parameter key-value to the Parameters collection.

The PolicyResults class is a collection of Trigger objects. We expose a CreateTrigger method in order to be able to create Trigger objects on-the-fly from BRE. We also expose two utility properties, Count and XmlString. The Count property returns the number of Trigger objects in the collection. The XmlString property returns the raw XML version of the collection. This property can be loaded into an XmlDocument class and assigned to and XLang message easily.

We use this setup when calling BRE by first creating a PolicyResults object and passing it into BRE as a fact. Next, when creating policies in BRE, we can call the CreateTrigger method in the Assert section of any Rules we create. If the Assert is executed the Trigger(s) will be created and added to our PolicyResults. Once the BRE call is complete we can return the PolicyResults as a string (or XmlDocument) to the calling orchestration, where can be used for processing.

R2-AS2 POC

2008-06-23T14:43:00.000-05:00

R2 AS2 POC

Sometime back, I worked on a AS2 POC(proof of concept) using BizTalk 2006 R2. Here is a high level overview of VAN/BizTalk interaction, Party configuration, AS2 and EDI receive/send ports configuration in R2, to generate decrypted EDI files and 997s.

BizTalk Server 2006 R2 makes use of many components to successfully establish communication between VAN(Value Added Network) provider and their customers. In this proof of concept, to receive and send EDI messages, AS2(Applicability Statement 2) components and HTTP adapter components are primarily used.
BizTalk AS2 receive processing is performed using the AS2 receive pipelines. There are two types of AS2 pipelines available in R2. AS2EdiReceive pipeline to process EDI messages received over AS2 and AS2Receive pipeline to process messages that are not encoded in EDI. AS2 pipelines are also responsible for generating MDNs(Message Disposition Notification).
In the above diagram, a request-response HTTP Two-Way adapter is configured to receive messages from VAN. AS2 receive pipeline generates the MDN and routes it to the BizTalk MessageBox database. This MDN will be automatically picked up by the AS2Send pipeline which is part of the HTTP Two-Way adapter. AS2 receive pipeline uses the BizTalk S/MIME pipeline component to provide S/MIME decoding functionality. AS2 Decoder processes the incoming message AS2/HTTP headers, verifies the signature and decrypts the encrypted messages. After successful decryption, AS2 disassembler generates a MDN and sets the correlation tokens and properties on the MDN.
In the case of EDI messages, EDI disassembler will parse the message and generates corresponding EDI document and 997 acknowledgements.

Party Configuration:

Create a new party using the BizTalk Administrator explorer
Enter appropriate values in the Organization, Name and Value fields under the values. I used EDIINT-AS2, AS2-From and partner name respectively. Hint: AS2 functionality resolves the incoming messages party information based on the AS2-From and AS2-To values in the Aliases tab
Next step is to set signing and encryption, MDN generation request properties for a given message
Right click on the party to select AS2 properties, select Party as AS2 message receiver
Leave the default values selected and make sure Sign message and Encrypt message check boxes are selected under Outbound AS2 message. I selected DES3, Application/EDI-X12 and also entered AS2-From and As2-To values, selected ‘Request MDN’ check box

HTTP Two-Way port configuration:

Create a Request-Response receive port/location to process AS2 messages and generate MDN response
Select HTTP as transport type and select AS2EDIReceive as receive pipeline and AS2Send as send pipeline
Create a send port to send raw data and go back to the party you created earlier to select this send port using the party administration
Select Certificate in the send port to apply certificate thumbprint for encrypting messages
Create another send port to send pay load messages (EDI messages). In this case, selected file transport type to send EDI files to local folder and make sure to set EDISend as the send pipeline
Enter receive port name and select EDIintAS.IsAS2PayloadMessage == True in the send port filter. This fileter values allows us to decrypt EDI messages and creates message files as specified in the Transport Type

997 Configuration:

Create a new send port to create 997 files, selected File as Transport Type
Set the filter for 997 messages and set filter BTS.MessageType = http://schemas.microsoft.com/Edi/X12#X12_997_Root
Set EDISend as send pipeline

Certificate Management:
It turned out the most important part of the POC was to have valid certificates and configured it right. I obtained a trial certificate from VeriSign. Note, you can also use windows generated certificate too. In a nut shell, for the encryption/decryption of the messages over AS2, I followed the Certificate configuration as described in this KB article: http://support.microsoft.com/?id=942253

Using BAM API’s in a BizTalk Solution

2008-06-23T13:06:00.001-05:00

Recently, I had a requirement where I needed to track a subset of messages flowing through an ESB to capture and store key data points to a table along with the raw Xml of the message. The data was going to be used by the client for viewing and reporting.

After a couple of small POC’s, it was decided to leverage BAM API’s in order to persist the data. BAM comes with OOB functionality to capture and display data elements through the BAM portal. Additionally, after defining an Observation Model using the Excel plug-in, the BAM definition is then deployed which creates Sql Server tables, stored procedures and views in order to persist and update the tables.

But, OOB functionality does not support capturing and persisting of the underlying Xml or provide content based filtering, so as part of the POC, the BAM API’s were used to store the Xml.

A C# component was written to use the OrchestrationEventStream class from the BAM API. This class is designed to asynchronously write data into the BAMPrimaryImport database as well as provide transactional consistency when used from either an orchestration or pipeline.

For this particular solution that I was working on, the messages that needed to be tracked were already being consumed by a generic Data Archive orchestration. This orchestration was then modified to contain a Decision Shape that would filter based on configurable conditions to capture the required content.

RFID Presentation at MICSUG, Reston VA

2008-05-22T09:28:00.000-05:00

This week I presented on Microsoft BizTalk RFID at the MICSUG (Microsoft Integration and Connected Systems User Group) for the Washington, DC area. The presentation was held at the Microsoft office in Reston.

The main topics that were addressed were architecture of the RFID code base, RFID as a platform, and case studies which demonstrated significant ROI’s. In addition we discussed how RFID and BizTalk work together.

RFID demonstrates a big step forward in BizTalk Server’s capabilities because it now enables connectivity to actual devices rather than software to software transactions which are traditional means of using BizTalk. It’s a brand new paradigm and the audience sensed the shift as the realization that almost any device can be tied into an enterprise using BizTalk.

Exciting times.

Link to BizTalk case studies

2008-03-12T15:35:00.000-05:00

Often times I am asked for reference case studies for potential BizTalk projects. And then comes that moment when I can't find the link to them!

Well, here it is. This part of the Microsoft.com site contains over 200 case studies related to BizTalk and has been updated with many new cases related to BizTalk 2006.

http://www.microsoft.com/casestudies/search.aspx?ProTaxID=1265

HIPAA Support in BizTalk 2006 R2

2007-01-02T19:51:00.000-05:00

HIPAA support in BizTalk has always required the purchase, installation and use of the HIPAA accelerator. I've recently learned that the upcoming release of BizTalk 2006 R2 (scheduled sometime in the June 2007 timeframe) will bundle HIPAA support out of box. A main feature of BizTalk 2006 R2 is to provide native EDI support for transaction sets such as X12. HIPAA is simply a collection of specific X12 transaction sets. As such, its a natural enhancement to support HIPAA out of the box. The following is the current list of transaction sets to be supported as described in the Biztalk b2b blog on msdn located at: (http://blogs.msdn.com/biztalkb2b/archive/2006/12/31/hipaa-support-in-biztalk-server-2006-r2.aspx).

key HIPAA 4010A transaction set schemas:
- 270/271 - Eligibility
- 276/277 - Claim Status
- 278 - Services Review Request/Response
- 820 - Payroll Deduction
- 834 - Benefit Enrollment
- 835 - Claim Payment
- 837D - Claim Dental
- 837I - Claim Institutional
- 837P - Claim Professional
ten other non-HIPAA standard schemas that are widely used in the HIPAA realm, including but not limited to:
- v3070 277 Healthcare Payer Unsolicited Claim Status
- v4010 148 Doctor's First Report of Injury
- v4040 277 Healthcare Claim Acknowledgement
- v4050 274 Healthcare Provider Information
a schema annotation which enables splitting a multiple part document (e.g. a 837 Claim that has many individual claims) into separate transactions, one for each individual document. This feature is enabled by an annotation called 'subdocument_break'. When assigned a value of "yes" it will split an inbound transaction into individual transaction sets based on the number of documents included in the original transaction set. For example, an inbound 837D with three separate claims in it would be split into three separate 837D transaction sets with all of the appropriate header information included in each.
WEDI/SNIP level 2 validation which includes basic X12 syntax integrity as well as validation of loops, segments, elements and code values.

Microsoft introduces an ESB for BizTalk

2006-12-26T14:40:00.000-05:00

An Enterprise Service Bus (ESB) will be available for BizTalk around the time of the release of R2.

An ESB, a collection of architectural patterns based on traditional Enterprise Application Integration (EAI) principles, is an often developed solution with BizTalk. Now, a set of ESB related building blocks will be made available to speed up the design and development process.

Officially called ESB Guidance, the new package contains architectural guidance, patterns and practices, and BizTalk Server and .NET components. Microsoft ESB Guidance currently consists of a number of BizTalk projects including:

ESB Core Engine (transformation and routing agents)
ESB Core Services (dynamic transformation, dynamic endpoint resolution/UDDI)
ESB Portal framework
Exception Management framework
ESB Client application
Namespace Resolution pipeline component

Currently, select partners, customers building an ESB and Microsoft employees can download and try the new early access release. The release contains:

ESB Developer Guide
This document introduces architects and developers to ESB architectural concepts as addressed by the guidance, describes the installation procedures for the reference implementation, and explains the functionality of the guidance components through a set of commonly accepted ESB use cases.

ESB Reference Implementation VPC
This Virtual PC image provides a fully functional instance of the ESB Reference Implementation for evaluation, technical readiness, proof of concept, and demo purposes. The MQSeries 5.3 Trial edition and CSD12 can be installed optionally. The installation for these can be found at the root of C:\Projects within the VPC. Once you install MQSeries, then you can install the JMS Pipeline component MSI package within the VPC.

ESB Reference Implementation Redistributable
The redistributable contains installation scripts and .MSI installers for the ESB Reference Implementation components.

ESB Positioning Presentation
This presentation describes Microsoft’s platform value to customers evaluating ESB vendor offerings and the capabilities offered by the Application Platform and the ESB Guidance package.

ESB Patterns Technical Presentation
This presentation provides an introduction to ESB patterns and capabilities as implemented in the ESB Guidance package.

Microsoft ESB Guidance should be released in the 3rd quarter of 2007.

SOA: not just a buzzword

2006-12-22T10:46:00.000-05:00

Over the last couple of years while the concept of Service Oriented Architecture has evolved I often worried that it would just become another buzzword that would become serially dismissed after a few years in the spotlight.

An interesting article popped up on ZDNet earlier this week which proves there is benefit to be had by adopting the principles associated with SOA. Big hitters like Ebay, HP, and Amazon are all adopting the principles of service orientation and in the process cutting costs and streamlining their enterprises. For an interesting read, check out the full article:

http://blogs.zdnet.com/service-oriented/?p=781

BizTalk Tips and Tricks – Promoting properties from a collection.

2006-10-19T02:16:00.000-05:00

Only non-repeatable fields can be easily promoted from within the schema editor. At the end of the day however it’s simply an xpath statement. In rare cases, if you need to promote a repeatable field from a collection, you can simply modify the xpath contained within the schema and change the xpath to point to a specific index within the collection. Internally, BizTalk simply applies the xpath to the message when its time to promote the data. There is no reason why the xpath couldn’t be restructured to evaluate to a field within a collection at a given index (such as 0). The need might be rare, but it can be done if needed.

BizTalk Best Practice - Message Decoupling in Orchestrations

2006-10-19T01:08:00.000-05:00

Have you ever found it annoying to change message schema types in an orchestration when the orchestration designer forces you to disconnect or unwire all the shapes that reference the message? There is a clever way around this and in the end, it promotes good design and adds another layer of decoupling and indirection to your messages in orchestrations.

Define all of your Messages in an orchestration from Multipart Schema Types.

If you create a message from a schema and assign it to a receive shape and that receive shape is connected to a logical port, if you need to change that message type, you need to first disconnect the “wire” between the receive shape and the port before the designer allows you the change the message type. Imagine if the message was referenced by lots ports and shapes. You would need to disconnect all of them and that could be a real pain.

If on the other hand you create a multipart message type for each schema and create each message in the orchestration from the appropriate multipart message type, you don’t need to unwire the various references to the message if you need to change the underlying schema type of a message. This might not be directly obvious, adds an abstraction layer around your message types and enhances overall maintainability.

BizTalk 2006 – Deep Technical Training

2006-10-19T01:03:00.000-05:00

I’m attending an internal BizTalk 2006 deep dive training session given by one of the BizTalk product team members at Microsoft who is close to the product internals and has lots of practical knowledge. It’s being held for both Microsoft and Virtual Technical Specialists. It’s not your typical deep dive and focuses heavily on architecture, farms, high availability, best practices, tuning, internals, considerations when designing solutions, and more. It’s been a long, but very cool week so far.

As time permits, I’ll begin posting interesting things we’ve discussed along the way. You might not come across anything new, but if you do, great! Most are common sense and good engineering practices; however others could be things you simply haven’t thought about yet or practices you haven’t followed. Given the flexibility and breadth of the product there are always going to be new things to consider.

BizTalk Server 2006 Documentation Update

2006-10-16T00:57:00.000-05:00

Microsoft released an update to the BizTalk Server 2006 documentation on 10/10/2006. It can be found here BizTalk 2006 Documentation Update (41.6 MB)

Overview
In this Help Update, you'll find the following content:

Complete documentation for all BizTalk Server Line of Business Adapters, including new tutorials, one for each adapter. See the section "Using Adapters."

Troubleshooting guidance for each of the native BizTalk adapters. See the section "Troubleshooting Adapters."

Troubleshooting Instructions for how to capture a memory dump. See the section "How to Capture a Memory Dump of a BizTalk Process."

Disaster recovery instructions for backing up and restoring your BizTalk Server and databases. See the section "Backing Up and Restoring BizTalk Server"

Updated Tutorials. See the section, "BizTalk Server 2006 Tutorials"

Information for improving fault tolerance. See the section "Planning and Architecture"

Updated Developer's Reference that now includes the BAM namespaces that were not included in the RTM documentation. See the section "Developer's Reference"

BizTalk 2006 R2 B2B/EDI WebLog

2006-10-15T21:52:00.000-05:00

The Microsoft BizTalk Product Development team is launching the B2B/EDI WeBlog on the MSDN platform to communicate feature/functionality relating to EDI, AS2 and HIPAA in BizTalk Server 20006 R2. Key objective of the blog is to increase product awareness and also assist with partner readiness thereby leading to a successful launch of the EDI Technology in R2.

The blog can be found here http://blogs.msdn.com/BizTalkB2B/

Business Policy Management and the BizTalk Rules Engine

2006-09-27T16:59:00.000-05:00

Earlier in the year we designed an architecture in which the BizTalk Rules engine was leveraged along with InfoPath, SharePoint and K2.Net to manage business policies in a human to human and human to system workflow solution. The full article can be found (here). We found this design to work extremely well in regards to providing an agile approach at managing business process policies in a generic way. Following is a brief fragement of this article.

RDA leverages the BizTalk Business Rules Engine to manage business policies in document workflow solutions. Document management workflows come in many shapes and sizes. Many of these workflows operate under rigid business policies, which often affect how documents flow through a process or influence decisions along the way.

Business policies are often fluid. They change with time. Any solution dependent on changing business policies must be agile by design. As such, the rules that govern the policy of a given document workflow needs to be easily maintained without affecting the solution deployed in a running production environment. The BizTalk Business Rules Engine provides just a mechanism.

The diagram below depicts a workflow system that leverages the rules engine to enforce and manage fluid business policies.

Browse all the RDA Spotlight on Business and Technology newsletters here:
http://www.rdacorp.com/SpotlightNewsletters.htm

Choosing BizTalk over WebSphere

2006-09-27T15:07:00.000-05:00

The following is a good whitepaper providing a comparative analysis of Microsoft BizTalk and IBM WebSphere. Note while this whitepaper is a bit dated and BizTalk 2004 centric, its still very much applicable to BizTalk 2006.

http://download.microsoft.com/download/0/1/E/01E77F30-E571-478D-A8F8-F35FE66BD2BD/ChsBPASol.doc

Mapping complex conditions in BizTalk with XSLT

2006-03-29T11:02:00.000-05:00

Recently I found myself in a situation where I needed to evaluate a set of complex conditions that I could not use the Business Rules Engine for. The BizTalk mapper was able to achieve the conditionality by chaining a series of about 21 functoids together but this became very unmanageable especially since I was combining false statements to see if both were false, then returning a true to an “And” functoid (&) which evaluated another branch as true and passed a result to the node in the destination schema, you get the idea.

An easier way is to plug in XSLT. There are disadvantages to doing this, the main one being its maintainability and transparency but when faced with the daunting set of functoids described above, it looked a decent option.

The XSLT function that I used was the <xsl:choose> statement, inside the statement was a set of conditions which, if true, would return a value/values to the destination schema. In the last part of the statement was the <xsl:otherwise> condition which could set values if the conditions presented were not valid and you had a default you wanted to present in that event.

<xsl:variable name="var:vAnnuity" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='LoanPartItems' and namespace-uri()='' and Method='Annuity'])" />
<xsl:variable name="var:vLinear" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='LoanPartItems' and namespace-uri()='' and Method='Linear'])" />
<xsl:variable name="var:vTotalCount" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='PartItems' and namespace-uri()=''])" />
<xsl:variable name="var:vHasValidEndowment" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='LoanPartItems' and namespace-uri()='' and isValid ='true'])" />
<xsl:variable name="var:vHasValidId" select="count(/*[local-name()='InternalSupport' and namespace-uri()='http://internal_support']/*[local-name()='Ready' and HasValidId ='true'])" />
<xsl:element name="LeningSoort">
    <xsl:choose>
        <xsl:when test="$var:vAnnuity = $var:vTotalCount">
            <xsl:value-of select="1" />
        </xsl:when>
        <xsl:when test="$var:vLinear = $var:vTotalCount">
            <xsl:value-of select="2" />
        </xsl:when>
        <xsl:when test="($var:vHasValidEndowment = $var:vTotalCount) and ($var:vHasValidId = $var:vTotalCount)">
            <xsl:value-of select="3" />
        </xsl:when>
        <xsl:otherwise>
            <xsl:value-of select="4" />
        </xsl:otherwise>
    </xsl:choose>
</xsl:element>

Of course each specific case requires different XSLT, but this should provide a framework of how to do this within a functoid.

To set up the XSLT:

• Create XSLT similar to the one above which meets your needs
• Drop a scripting functoid on your map
• Connect the output parameter to the node you want to create
• Note: do no include any input parameters, the XSLT is setup already to look them up
• Open the scripting functoid’s, click the (…) inside “Configure Functoid Script”
• Under “Script type” choose “Inline XSLT”
• Paste your XSLT snippet in the text box

XSLT performance when mapping large documents in BizTalk

2006-03-25T11:00:00.000-05:00

Recently I had to map a document with many thousand rows. I could not split the document because before I could split it, the document’s nodes had to be sorted.

With such large files you generally test it using a small subset to avoid waiting for maps to complete, I built an XSLT which worked great, I thought.

When you use a select filter such as "not(KeyValue=preceding-sibling::row/ KeyValue)" you end up with a huge performance hit the larger the document gets. My map went from 2 seconds for 50 rows to 10 minutes for a few thousand.

How to improve performance when you have large XML files to map that you can’t split? Try using xsl:key instead, which builds an index of keys from which you can much more efficiently select.

Here is a sample XSLT that demonstrates how to use the xsl:key:

<?xml version="1.0" encoding="UTF-8" ?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" xmlns:ns0="http://Conversion.schemas">
    <xsl:output method="xml" indent="no" />
    <xsl:key name="NumberKey" match="/*[local-name()='top' and namespace-uri()='http://biztalk/Conversion.schemas']/*[local-name()='row' and namespace-uri()='']"
        use="keyValue" />
    <xsl:template match="/">
        <ns0:Rows>
            <xsl:for-each select="/*[local-name()='top' and namespace-uri()='http://biztalk/Conversion.schemas']/*[local-name()='row' and namespace-uri()='' and generate-id(.) = generate-id(key('NumberKey', keyValue)[1])]">
                <xsl:variable name="current_Number" select="keyValue" />
                <Data>
                    <keyValue>
                        <xsl:value-of select="$current_Number" />
                    </keyValue>
                    <xsl:for-each select="//row[keyValue=$current_Number]">
                        <Part>
                            <PartID>
                                <xsl:value-of select="nr_data" />
                            </PartID>
                        </Part>
                    </xsl:for-each>
                </Data>
            </xsl:for-each>
        </ns0:Rows>
    </xsl:template>
</xsl:stylesheet>

You can evaluate dates using the Cumulative Maximum Functoid

2006-03-13T10:57:00.000-05:00

Dates you say? Stop the press!

I had a situation where there was a node whose max occurs is unbounded and I needed to extract a single node from it to map to a destination node whose max occurs is one.

The problem was that I had to choose the node to map with the highest (newest) date:
<root>
<data>
<testdate>01-01-2000<testdate>
<testdata>a<testdata>
</data>
<data>
<testdate>01-01-2002<testdate>
<testdata>c<testdata>
</data>
<data>
<testdate>01-01-2001<testdate>
<testdata>b<testdata>
</data>
</root>

In the case above, I want the node with the highest <testval>, or 01-01-2002, therefore I want to map <testdata>c<testdata> to the destination node.

This can actually be done.

What I did was convert my date to ticks in a scripting functoid:

public string toTicks(string param1)
{
DateTime dt = DateTime.Parse(param1);
return dt.Ticks.ToString();
}

Afterwards I connected the scripting functoid to the cumulative max functoid.

Then I evaluate the output of the scripting functoid with the cumulative max functoid with an Equals functoid.

This should work for any set of repeating dates.

BizTalk's mapper allows drag and drop replace

2006-03-04T10:55:00.000-05:00

I learned a little trick the other day about the mapper which I have never seen documented. It works for BizTalk 2004 and also 2006.

Have you ever had to change a functoid and gone thru the task of dragging the new functoid onto the grid and then reconnecting all of the lines, then having to check to make sure you got them all in order?

Well, what I realized quite by accident is that you can actually just drag the new functoid over the one you want to replace and all your links will maintain themselves. The old functoid will disappear and be replaced by the new one.

Of course there is a limitation when you have different maximum inputs, etc, but for the most part this works with everything else.

Blogging about .NET? Then this is a must have tool

2006-02-25T10:53:00.000-05:00

I’ve found a tool recently that will make blogging about BizTalk easier. It’s been around for a while but if you are like me and have never heard of it before, CopySourceAsHtml is very useful.

If you have been in the situation of wanting to post a code or XSLT snippet onto your blog and found yourself writing HTML to make it look like it does in Visual Studio (or resort to screen shots) then fret no more, CopySourceAsHtml will take anything that you can open in VS (source code related) and convert it to HTML for you.

The tool is written by Colin Coller and can be downloaded on the CopySourceAsHtml homepage. In addition, Colin has a blog which updates progress on the tool. I read that a VS 2005 version is soon to be ready.

A couple of tips when using it:

After installing you should be able to open any C# source code page, highlight and right click a code snippet and see a new “Copy as HTML” item on the menu. But, if for example you want to copy XSLT, the item doesn’t show up. Use the edit menu instead.
When you copy something as HTML, under general, uncheck the “Number lines from:” box unless you really want them and also check the “Embed styles” box since blog tools/some browsers wont recognize the style sheet that gets created. Embeding the style makes it a simpler operation.

Easier way to add a DLL to the GAC

2006-02-03T10:47:00.000-05:00

Often in BizTalk you’ll need to add assemblies to the Global Assembly Cache. In the past I had used a shortcut on my Quick Launch toolbar which opened the GAC, then I’d drop the assembly inside, but there is an easier way.

A registry key can be created which allows you to right click the DLL and “GAC it” in one click.

The text for the key is below; simply copy/paste it into notepad and save it as a registry file (.reg). Then double click the file to add it to the registry.

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\dllfile\shell\gacutil\command]
@="c:\\windows\\Microsoft.NET\\Framework\\v1.1.4322\\gacutil.exe /i \"%1\""

You can also download the Gacutil right-click key as a zip file here.

BizTalk exception handling with web services

2006-01-31T10:45:00.000-05:00

Recently I needed to include exception handling in a web service call to an external party. I learned a couple of things and thought I’d highlight them here.

Exception handling doesn’t always behave the way you expect when you are trying to catch exceptions when calling a web service. While using one is very useful some things must be considered or you will not get the result you wanted.

Here is one way to do it as well as a few things to keep in mind when you want to set one up:

Include the external web service call (both send and receive shapes) in either a non-transactional scope shape or a long running transaction.
Decide if you need to catch any specific exceptions, if you don’t, using the General Exception will suffice. If you need to catch specific errors, you can use multiple catch exception blocks.
It may sound obvious to say this, but make sure your orchestration doesn’t have a timeout that is the same length (or shorter) than the timeout of your scope transaction.
Most important: in your send port which holds the reference to the web service, make sure you are not using any retries (unless, of course, you explicitly need to). If the web service fails with an error such as “There is an error in XML document (5, 10)” and you have retries enabled you will only see warnings in your event viewer (which means the exception block won’t be activated until the SOAP timeout).

"multiple schemas match the message type"may be caused by web services

2006-01-21T10:43:00.000-05:00

I was stuck recently with what seemed like a pair of contradictory errors.

I had an application deployed which was exposed as a web service. To test the application we built a test harness which picked up a flat file and called the web service for us and then dumped the result on the file system to see the result.

This testing system worked fine until I needed to use that same deployed schema to do some additional messaging. I wanted to route a message via a receive port and have a send port subscribe to it, then map the message to a new format and deliver it to a destination. Pretty simple, right?

Well, no. When I have my project deployed I got the following error when routing:

There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive" Source: "XML disassembler" Receive Location: " D:\dat\FileReceive\RoundTrip\*.xml Reason: Cannot locate document specification as multiple schemas match the message type "http://BO.Get.EV2#Method".

That error left me scratching my head. It means I probably have the schema deployed twice, right? I didn't, so I thought. When I undeployed the application the messaging engine could no longer match my message type:

There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive" Source: "XML disassembler" Receive Location: "D:\dat\FileReceive\RoundTrip\*.xml"
Reason: Finding document specification by message type "http://BO.Get.EV2#Method" failed. Verify that the schema is deployed properly.

When I deployed the project with the schemas, everything suddenly worked fine. That means that, somewhere, the schema was again deployed.

So I looked thru the BizTalk Server Assemblies viewer and discovered that the web reference which existed in my test harness was the culprit. The messaging engine recognized the web reference as a second deployment of my schema. This was technically accurate but rather frustrating to find. I've seen so many of these errors posted on the newsgroups that I thought I'd mention this as a possible cause.

If you are using test harnesses coupled with web services be aware that your schema is actually deployed twice.

Fully qualified name differs for XSD’s with multiple root nodes in the Business Rules Engine

2006-01-12T10:42:00.000-05:00

One of the most common problems that exist when using the business rules engine is forgetting to use the fully qualified name from a schema in definition of the rule.

Posts like this on Google Groups can clarify what I mean if what I am talking about is not clear.

I have discovered something new regarding the fully qualified name which caught me out this morning and I thought I’d mention it here in case you have the same problem.

If you are using the fully qualified name of a schema with multiple root nodes in the BRE, the fully qualified name must include the name of the root node. When you go to your orchestration and try to feed a rules shape your XSD you will find it won’t recognize it (no drop down appears).

For example you can have a schema with multiple root nodes with a fully qualified name such as schma_name_fully_qualified which, with a single root node, would work just fine. But if the schema has multiple root nodes you must append the specific root node to this qualification for the orchestration to recognize the rule.

So if you had the fully qualified name schma_name_fully_qualified, look at your orchestration view, click on the message you are using and look at the message type: you will find that you must use something like schma_name_fully_qualified.header to properly identify the schema when creating your business rules. Changing this should then allow you to select a message type in your orchestration.

How to stop applications from unloading in BizTalk 2004

2006-01-05T10:39:00.000-05:00

Recently we developed a time sensitive BizTalk application where a simple set of processing steps were invoked via a web service.

The external calling application (the application calling the BizTalk process) had just 5 seconds to complete its task. Since the BizTalk part was not very complex or resource intensive, we were achieving response times of 1 second, well within the 5 second margin.

During testing, though, we were getting intermittent timeouts because the BizTalk application was unloading itself from memory and causing the calling application to timeout. Basically it took more than 5 seconds for the unloaded BizTalk application to start itself up again and caused a failure.

BizTalk relies on a configuration file to store its application information. This config file contains settings for unloading an application from memory and is located at \Program Files\Microsoft BizTalk Server 2004\BTSNTSvc.exe.config. Full documentation can be found at Microsoft’s website but here is a working example you can download.

By default BizTalk has 2 shutdown settings which are:

SecondsIdleBeforeShutdown = 1800 (seconds, or 30 minutes)
SecondsEmptyBeforeShutdown = 1200 (seconds, 20 minutes)

The link below is a sample config file that never shuts down. We are isolating these “hot” processes on a separate server from the “cold” processes (processes that can safely unload). You can run hot and cold side by side as well but that requires more changes in the config file. If you set the SecondsIdleBeforeShutdown and SecondsEmptyBeforeShutdown to -1 they will always stay in memory.

Note that this is increasing the load on the server but for time sensitive situations this could save you a rash of intermittent errors in production. Note also that you shouldn’t do this unless you really need to, unloading saves resources and with most integration scenarios the few seconds it takes to start up are minor.

You can backup your current file, download this BTSNTSvc.exe.config zip file and extract it, then stop and start your BizTalk host to see it work.

Solving BizTalk "error X2044"

2006-01-02T10:49:00.000-05:00

I encountered a strange compilation error recently in BizTalk 2004 when importing a web service.

While I can’t put my finger on the exact cause yet, I do have a fix that works.

When you see an error message like:

C:\Project\Web References\ExtRef\MyOrch.odx(445,23): error X2044: symbol '@@@@' is already defined; the first definition is in assembly @@@.dll : It may be possible to disambiguate by using fully qualified names.

The way to fix this error is to do the following:

Open the orchestration in Notepad (In my case MyOrch.odx)
Do a find (Ctrl+F) for “#endif // __DESIGNER_DATA”
Delete all the code below that line
Save your file in Notepad
Say yes when Visual Studio asks if you want to update your file
Recompile

Note: The code will not be regenerated until you edit an expression shape, etc. Add a “//” or something and you will then see the code re-appear when you open it in Notepad again.

I hope this may serve to save someone time, it took a while for me to figure it out.

Namespace	Description
System	The System class
System.Collection	The collection classes
System.Text	The text classes
System.Text.RegularExpressions	The regular expression classes
System.Xml	The core XML classes
System.Xml.Xsl	The XSLT classes
System.Xml.Xpath	The XPath classes
Microsoft.VisualBasic	The Visual Basic script classes.