Using a Flat File Schema to Consume Non-Xml Messages With No Type

Recently, a client was updating their message processing system from a collection of old VB6 applications to a BizTalk Server 2009 solution. One of the processes that they struggled with was updating their “special case” partners. I call these partners “special case” because they could not or would not adhere to the standardized schema for acknowledgment messages that was provided.

Each “special case” partner had their own message type, which changed without notice. The only thing that each had in common was that they agreed to put an acknowledgment bit somewhere within their return message. The placement of this acknowledgment bit was establish many years ago when the VB6 applications where written. The problem of extracting the bit was solved within the VB6 applications and the code was simply updated to C# to leverage the existing solution.

The other issue that was occurring is that each acknowledgment message had a different format, and non of them were valid XML. Most of the acknowledgments were actually screen scrapes from HTML pages with multiple root nodes and the message stuffed in at various places. BizTalk does not like to consume messages that have no schema/message type. To solve this issue, I had the client run the Flat File Schema Wizard to generate a Flat File Message with a single node and the entire contents of the message as its value (contained within CDATA markup to prevent corruption). The trick with this approach is the delimiter used to define the records. Since we only ever want one record, we need to choose a delimiter that has a very, very low probability of showing up in the incoming message. The delimiter I chose to use was “|abcdefghijklmnopqrstuvwxyz|”.

Now that the message was being consumed, it was a simple matter to use xpath to extract the string payload of the message and send it over to the existing C# helpers that know how to process it.

Where Do I Create a Web Reference in BizTalk 2009?

If you are reading this, most likely you were just as stumped as I was when I tried to create my first web reference in BizTalk 2009.

I assumed "Web Reference" was renamed to "Service Reference" but I quickly realized that something was wrong when I tried to create a message within an orchestration and had nothing listed within the "Web Message Types" section.

Well, to create a traditional Web Reference in BizTalk 2009 you have to take an extra step:

Right click on the project, and select "Add Service Reference" just as you did before, but this time, at the bottom click "Advanced". At the bottom of that screen click "Add Web Reference" which will take you to the web reference screen we recognize from previous BizTalk versions.

How to get to the packaged EDI schemas

I needed to validate a clients test message against the standard EDI schema for an X12 810 message but did an extensive search and was not able to find the schema for the message.

I found the EDI base DLL that contains the schemas (Microsft.BizTalk.EDI.BaseArtifacts.dll) but this didn't give me the ability to validate against the schema.

Turns out that all of the EDI XSD schemas are packaged in an exe located in: \Program Files\Microsoft BizTalk Server 2006\XSD_Schema\EDI. Just unpack the file named MicrosoftEdiXSDTemplates.exe and you will have extractions for all of the EDI schemas for EANCOM, EDIFACT, HIPPA and X12.

Be careful because this is a huge number of files. I opened the exe in win rar and just extracted the X12 files.

From there you can just add the schema to your project and validate your message

Mapping Unrelated Records

Introduction

I was recently asked to create a map to combine parent records with children records. The challenge was that the children records in the source document where not grouped with their parent record.

Mapping Unrelated Records

The message was similar to the following Xml where StatusItem is related by its ordinal position to DetailStatus/DetailItem. Look carefully at the message; the StatusItem parent node is SummaryStatus, which is at the same level in the Xml as DetailStatus. There is only one SummaryStatus node, and there are multiple DetailStatus nodes – all at the same level in the Xml.

<root>
  <
SummaryStatus>
    <
SummaryCode>102</SummaryCode>

    <
StatusItem>
      <
statusCode>item_0</statusCode>
    </
StatusItem>

    <
StatusItem>
      <
statusCode>item_1</statusCode>
    </
StatusItem>

    <
StatusItem>
      <
statusCode>item_2</statusCode>
    </
StatusItem>
  </
SummaryStatus>

  <
DetailStatus>
    <
DetailItem>
      <
statusCode>detail_0</statusCode>
    </
DetailItem>
  </
DetailStatus>

  <
DetailStatus/>
    <
DetailItem>
      <
statusCode>detail_1</statusCode>
    </
DetailItem>
  </
DetailStatus>

  <
DetailStatus/>
<
root>

In reality a message similar to the above is being provided by an outside source and we cannot modify it. By convention we are given the assumption that there is a DetailStatus record for each StatusItem, however there is not always a DetailItem. Our job is to map the output and combine the StatusItem code with its each DetailItem code.

The output goal is similar to the following Xml:

<root>
  <
SummaryCode>102</SummaryCode>

  <
StatusItem>
    <
StatusCode>item_0</StatusCode>
    <
DetailItem>
      <
StatusCode>detail_0</StatusCode>
    </
DetailItem>
  </
StatusItem>

  <
StatusItem>
    <
StatusCode>item_1</StatusCode>
    <
DetailItem>
      <
StatusCode>detail_1</StatusCode>
    </
DetailItem>
  </
StatusItem>

  <
StatusItem>
    <
StatusCode>item_2</StatusCode
  </StatusItem>
</
root>

The BizTalk map below works by using the Iteration Functoid comparing the position of the current SummaryStatus parent record to the position of the DetailStatus record. Additionally the Logical Existence Functoid is used so that if a DetailItem does not appear in the source message, then it won’t appear in the Target message.

MapParentToChild

External Links from MSDN

Previous Articles

Fix for BAM error "Views or Activites may be missing because one or more database(s) could not be contacted"

I had to migrate a BAM implementation into production from a development environment where everything in localized to a clustered database back end.

The deployment went fine, including the BAM part, but I couldn't access the views in the BAM portal. I verified that data was being written into the BAM database, but when I went to the portal to view the data I saw the following message:

Views or Activites may be missing because one or more database(s) could not be contacted. No view to display

There was also an error in the event log which read:

Referenced database 'BAMPrimaryImport' on server 'xxxx' is not accessible. The error is:
System.Data.SqlTypes.SqlNullValueException: Data is Null. This method or property cannot be called on Null values.

After some investigation I learned that the owner of the BAMPrimaryImport database needed to be a domain account. I tried to change the account role:

sp_addrolemember 'dbowner' 'domain\user'

This failed with an error.

So I ran instead (which does not remove the user, just ‘resets’ the users access rights):

sp_revokedbaccess 'domain\user'

Which threw an error, but the account was holding the db owner schema, so I changed that to dbo and it worked.

Then, I could run the original call (which now worked):

sp_changedbowner 'domain\user'

The views are now visible in the portal, so if you see this error above, this may be the cause.

ESB 2.0 Custom Resolver: Part 2

In part one, we discussed what code was required for a custom resolver. In part two, we will be talking about what steps need to be undertaken in order to have the ESB recognize the new resolver as well as the topic of caching and making the resolver configurable.

Caching

Resolution caching is accomplished using the Microsoft.Practices.ESB.Cache.Cache<T> object. Each resolver maintains its own unique cache. Some resolvers, such as BRE, do not maintain a cache at all, as they are expected to update in real-time when the repository changes.

Adding a resolution to the cache is as simple as calling the Add method of the Cache<T> object with a key and the resolution. A resolution should be unique to its configuration, with that assumption in mind the standard key for a resolution in the cache is its config string. Resolutions are defined as type Dictionary<string, string> the resolution Dictionary should be the second argument in the call to the Add method. Note, that some verification should be done to ensure that the resolution is not empty/corrupted, as it will come back each time the provided config is resolved until the cache expires. The last argument for the Add method is the optional expiration interval. In this implementation a configurable value cacheTimeoutValue was crated that is used to define how long a cache value should persist. A value of 0 denotes that the cache has been disabled.

Retrieving a resolution from the cache requires a call to the Get method of the Cache<T> object. The Get method requires the key (the config string) in order to retrieve the resolution from the cache. The benefit of caching is that it enhances performance when under higher volumes. The database is only queried once per cache expiration interval for a unique configuration. The drawback is that the repository could take one entire expiration interval to update when it is changed. With this balancing act in mind, the expiration interval was made configurable.

Configuration

Configuration includes wiring up the resolver to the ESB and configurable resolution values. The wire-up portion is required for all implementations, so we will start the discussion with that.

Contained in the ESB install is a configuration file called esb.config. This configuration is referenced from both the machine.config and the btsntsvc.exe.config files. The resolution configuration settings are contained in the esb.config file. To wire-up the custom adapter following node needs to be added to the configuration/esb/resolvers node:

<resolver name="SQL" type="ESB.Resolver.SQL.ResolveProvider, ESB.Resolver.SQL, Version=1.0.0.0, Culture=neutral, PublicKeyToken=42bcd53db1111111"/>

The name attribute denotes the Moniker name (see part one for a discussion of Monikers). The type attribute is the fully qualified assembly name for the custom resolver that was created in part one.

To add configurable values for a resolver a resolverConfig node can be added to the aforementioned resolver node:

<resolver name="SQL" type="ESB.Resolver.SQL.ResolveProvider, ESB.Resolver.SQL, Version=1.0.0.0, Culture=neutral, PublicKeyToken=42bcd53db1111111">
                <resolverConfig>
                    <add name="cacheTimeoutValue" value="120"/>
                    <add name="cacheName" value="SQL Cache Manager" />
                </resolverConfig>
</resolver>

The configuration can be parsed in the custom resolver class with the following code:

string cacheName = ResolverConfigHelper.ReadResolverConfigByKey(resolverConfig, "cacheName");

To make use of the SQL Cache Manager, it must be configured in the esb.config. Add the following node to the configuration/cachingConfiguration/cacheManagers node:

<add name="SQL Cache Manager"
              type="Microsoft.Practices.EnterpriseLibrary.Caching.CacheManager, Microsoft.Practices.EnterpriseLibrary.Caching, Version=4.0.0.0,Culture=neutral, PublicKeyToken=31bf3856ad364e35"
              expirationPollFrequencyInSeconds="2"
              maximumElementsInCacheBeforeScavenging="1000"
              numberToRemoveWhenScavenging="10"
              backingStoreName="inMemory" />

Future Plans

Future plans for the SQL resolver is to make it more configurable. There are plans to be able to query services by other fields such as Author, Version, etc.

This concludes my two part discussion of custom resolvers in ESB 2.0.

ESB 2.0 Custom Resolver: Part 1

Recently I encountered a situation where we wanted to leverage SQL server as and repository for endpoints. The resolver consists of two parts: the repository (SQL Table in this example) and the custom resolver assembly. I will be focused on the custom resolver for this  discussion.

The custom resolver consists of an assembly and configuration entries. In part one we will discuss the code that goes into the assembly. In part two we will talk about wiring up the assembly so that the ESB will use the new assembly, as well as how to make the resolver configurable.

Per the pattern set up by the ESB Guidance team, each resolver should be in its own assembly. This assembly will need to include references to the following assemblies:

  • Microsoft.Practices.ESB.Cache
  • Microsoft.Practices.ESB.Common
  • Microsoft.Practices.ESB.Configuration
  • Microsoft.Practices.ESB.Exception.Management
  • Microsoft.Practices.ESB.Resolver
  • Microsoft.XLANGs.BaseTypes

The naming convention for the assembly is ESB.Resolver.MyResolver. In our example the assembly is ESB.Resolver.SQL. A class that implements IResolverProvider is required for the custom resolver. The naming convention for this class is ResolveProvider.

The IResolverProvider interface implements three overloaded variations of the Resolve method. Each of these need to be implemented to ensure that the resolver works in all traditional contexts. All three methods should be executing the same logic. With that in mind, the ESB Guidance team created a ResolveMyResolver(string config, string resolver, Dictionary<string, string> resolutionOrig) method that each Resolve method calls. I have included my sample below.

private Dictionary<string, string> ResolveSql(string config, string resolver, Dictionary<string, string> resolutionOrig)
        {
            // fix the resolver if it needs it
            if (!resolver.Contains(ResolutionHelper.MonikerSeparator))
                resolver = resolver + ResolutionHelper.MonikerSeparator;

            try
            {
                // check for cache...return data structure
                Dictionary<string, string> sqlValues = RetrieveFromCache(config);

                //No Resolution cached
                if (sqlValues == null)
                {
                    sqlValues = new Dictionary<string, string>();

                    //Extract Parameters
                    Dictionary<string, string> queryParams = ResolutionHelper.GetFacts(config, resolver);
                    string serverName =
ResolutionHelper.GetConfigValue(queryParams, false, "serverName");
                    string dbName = ResolutionHelper.GetConfigValue(queryParams, false, "dbName");
                    string serviceName = ResolutionHelper.GetConfigValue(queryParams, false, "serviceName");
                    //Build connection String
                    string connectionString = String.Format("Data Source={0}; Initial Catalog={1}; Integrated Security=SSPI;", serverName, dbName);
                    //Execute SQL query to retrieve EndPoint
                    SqlDataReader sqlReader = null;

                    try
                    {
                        sqlReader = ResolveSqlEndpoint(serviceName, connectionString, sqlReader);
                        if (sqlReader.HasRows)
                        {
                            sqlReader.Read();

                            //Transfer result to Resolution
                            Resolution resolution = new Resolution();
                            resolution.EndpointConfig = sqlReader["EndPointConfig"].ToString();
                            resolution.TransportLocation = sqlReader["TransportLocation"].ToString();
                            resolution.FixJaxRpc = Convert.ToBoolean(sqlReader["JaxRPC"]);
                            resolution.MessageExchangePattern = sqlReader["MessageExchangePattern"].ToString();
                            resolution.TransportType = sqlReader["TransportType"].ToString();
                            resolution.TransportNamespace = sqlReader["TransportNamespace"].ToString();
                            resolution.Action = sqlReader["Action"].ToString();

                            //Transfer results to the Dictionary
                            ResolverMgr.SetResolverDictionary(resolution, sqlValues);

                            if (null != sqlValues)
                                resolutionOrig = sqlValues;
                        }

                        sqlReader.Close();

                    }
                    catch (Exception ex)
                    {
                        EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);
                        throw;
                    }

                    //add result to cache
                    SetCache(sqlValues, config);
                }
            }
            catch (Exception ex)
            {
                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);
                throw;
            }

            return resolutionOrig;
        }

Stepping through the code, the first step is to repair the resolver string, if needed. The resolver string is also referred to as the Moniker. A Moniker is formatted as follows Resolver:\\. For example the UDDI Moniker is UDDI:\\. In our example, our Moniker is SQL:\\. This first code snippet simply applies the :\\ if it missing.

The next step retrieves the resolution from the cache, if it exists. We will dive into this more in depth in part two.

Next, the parameters are extracted from the config string. The config string is the list of key/values that are included with the Moniker to create the EndPoint string. Our config string contains three key/values: Server Name, DB Name, and Service Name. These values are required to query our SQL repository. The ResolutionHelper.GetConfigValue method is used to extract the parameters from the config string.

Next, the SQL connection string is created and the SqlDataReader is initialized. A stored procedure that takes the Service Name and returns the resolution data for that Service is executed to populate the SqlDataReader. A Resolution object is instantiated and the values from the SqlDataReader are used to populate values in the Resolution class. The Resolution object is then used to create the output of the method using the ResolverMgr.SetResolverDictionary method.

The final step before returning the Dictionary result is to add the resolution to the cache. Caching will be described in more depth in part two.

In part two we will discuss caching, configuration, and wiring up the custom resolver to the ESB.

BizTalk 2009 Beta Documentation Now Available

Beta documentation for BizTalk 2009 is now available to download from the Microsoft.com website:

http://www.microsoft.com/downloads/details.aspx?familyid=38923F29-167F-497D-80A5-899602FEF8D7&displaylang=en

BizTalkCop For Naming Convention Enforcement in BizTalk 2006 and R2

Elton Stoneman has put together an extension to FxCop to assist with the enforcement of naming conventions around BizTalk applications.

The 1.0 release contains a set of FxCop rules for analysis against BizTalk assemblies and BizTalk applications. The rules in the package are configurable, by default they are based on the naming conventions Scott Colestock put out a few years ago.

A detailed usage guide is published here:

http://geekswithblogs.net/EltonStoneman/archive/2008/11/14/introducing-biztalkcop.aspx

An extensive demo of the tool is published here:

http://geekswithblogs.net/benny/archive/2009/03/01/biztalkcop.aspx

You can download the package from CodePlex here:

http://www.codeplex.com/BizTalkCop

Empty Node Gotcha with the Business Rules Engine

I've been on a BizTalk 2006 R2 project where we are using the business rules engine to enrich some data coming from a call center.

The data comes in using the vendor's format and is mapped on the inbound port into a cannonical format to be used internally for processing. Because of the complexity of the vendor's schema, we used some inline XPATH statements in the map to transform the data.

When testing against the endpoint DB2 database, we discovered the database was failing to process the messages, even though everything appeared fine on the surface. After some investigation we discovered some hexidecimal characters in what were supposed to be empty nodes. The characters we saw were representative of a line break and a tab: #xD; and #xA;.

Tracing the steps back, we discovered this was appearing after the Business Rules Engine processed the document.

In a normal map, empty nodes are created as <empty>, but using XPATH we were deriving empty nodes that looked like <empty></empty>, and when this type of node went to the rules engine, it created line breaks.

We needed the empty nodes because the endpoint demanded all the nodes be present, so we had to make an adjustment to the XPATH statements to use something similar to:

<xsl:choose>
<xsl:when test="@@XPATH HERE@@">
<xsl:element name="Empty">
<xsl:value-of select="@@XPATH HERE@@"/>
</xsl:element>
</xsl:when>
<xsl:otherwise>
<xsl:element name="Empty"></xsl:element>
</xsl:otherwise>
</xsl:choose>


Where the choose/otherwise allowed us to create the type of empty node we were looking for.

So, if you find yourself extracting data from the rules engine and these unwanted characters appear, this is one way to solve it.

Dundas Chart Removed from Sample Management Portal in ESB 2.0

Patterns and Practices has made a major change to the sample Management Portal that ships with the ESB Guidance 2.0 CTP2. The Dundas charting dependency has been removed. This is big news as it allows the sample portal to be used as a low-cost ESB Management Portal  without incurring the cost of Dundas Chart. Dundas Chart has been replaced by the MSChart control.

UDDI and Windows Server 2008 Upgrade

I was upgrading a server recently from Windows Server 2003 to Windows Server 2008. The Windows Server 2003 installation did not include UDDI service. UDDI services are installed by default in Windows Server 2008. Everything looks correct as long as you don’t attempt to use the UDDI services on the upgraded machine. The services look to be running correctly and no errors are sent to the Event Log, however UDDI will not function.

To resolve this issue it is necessary to reinstall UDDI services on the Windows Server 2008 installation. This only occurs when doing an upgrade from Windows Server 2003 and below that does not have UDDI services installed prior to the upgrade.