Why do we need this feature?

SOAP & REST based Web Services are one of the most used technologies in integration scenarios. In the last few years, the need for HTTP based communication with external vendors/systems has increased significantly. Pretty much it became a norm to expose the API's of the system for external consumption. Examples: Salesforce API, Microsoft Graph API for Office365 and so on. Also, the companies want to reuse their legacy system as much as possible internally and they expose them as internal web services.

BizTalk Server offers great support for HTTP(S) based web services, both on the Receive and the Send side of interfaces.

When you are integrating multiple endpoints via HTTP(s), it is important to know whether the web services are available, responsive, returning expected values etc., to be able to communicate with them and to support the business process.

What are the current challenges?

The use of web services (web endpoints) comes into play on both receive side and send side in BizTalk Server. Both comes with its own significance and it's important to monitor them.

Receive Side web services

BizTalk Server allows you to publish schemas and orchestrations as WCF services, which are then published in the IIS web server on the BizTalk Server. In case, you have multiple BizTalk servers in the same BizTalk group, you can set up an NLB (Network Load Balancing) cluster to achieve Load Balancing and High Availability.

The published WCF services are associated with Receive Locations. As a result, if something is wrong with a particular WCF service, for example, the Application Pool in IIS is down, the Receive Location will automatically be set to Disabled.

If the Receive Location/Port is down, then it won't accept any incoming messages via this channel. Therefore, it's important to make sure it's up and running all the time.

Send Side web services

On the outgoing side, you configure either HTTP, SOAP, REST, BizTalk Send port to connect to external web endpoints. If something is wrong with the external web endpoints, the state of the Send Port will remain the same inspite of the state of the external web endpoint.

So, to be able to know the actual state of the web endpoint, the well-being of the endpoint will have to be monitored and checked. This helps to be sure that when a real request to the web endpoint is made, that the actual endpoint can be accessed and the request is properly being processed.

When a web service is down and requests are being sent to that web service, the failing requests will lead to suspended messages in BizTalk, resulting in an interruption of the business process and some work for the BizTalk administrator to have the issue solved and to resume the suspended messages.

An external web service can be down for a variety of reasons.

• Internal web server/service issues - you might get for example the famous, though non-informative, HTTP 500 return codes
• Certificate issues - expired certificates, incorrectly configured or unaccepted client certificates
• Firewall issues - web server/service cannot be contacted at all due to wrongly configured firewall rules
• Flooding - there's more work than the web server/service can handle

All this kind of errors will lead to unavailability of the web endpoint and interruptions of the business process. This is exactly what we want to prevent, and as it is impossible for a BizTalk administrator to manually monitor web endpoints 24/7, it is way more efficient to have monitoring in place.

How BizTalk360 solves this problem?

For monitoring the physical HTTP web endpoints, BizTalk360 offers Advanced Web Endpoint Monitoring. Once set up, this feature will periodically fire requests against the web service which is being monitored and report any issues identified.

The feature is quite rich and offers, amongst others, support of:

• • HTTP and HTTPS web services (REST/SOAP)
  • Using Gateway proxies
  • Providing credentials
  • GET and POST methods
  • Multiple content types like
    • Text
    • XML
    • JSON
  • providing custom
    • HTTP headers
    • GET parameters
    • POST payload

These features allow you to monitor for example HTTP return codes (200 is okay, 500 is warning/error). But you can also check for certain keywords to exist in the XML/JSON response message, by entering XPath/JSON queries.
You can even monitor for response times of the web service, as this might be an indication that the web service is under stress.

So with the Advanced Web Endpoint Monitoring feature of BizTalk360, you will be able to cover pretty much all the web endpoint monitoring scenarios, whether these web services reside inside your organization or outside and whether these services use XML or JSON.

Conclusion

To underline the importance of web endpoint monitoring, let's shortly discuss the often forgotten concept of Chain Monitoring.
It is a bit of a human behavior, to just focus on your own work and systems, meanwhile forgetting the impact of your work on other systems.

A simple example: a Technical/Functional Application administrator is responsible for the administration of a CRM system. On a day to day basis, he is very busy with functional questions, which results in not always having the overview of the health of the web services of his CRM system, while these services received data on a regular base from an ERP system, via BizTalk Server.

It was with this kind of scenario where we have experienced that the BizTalk administrator who was using BizTalk360 and had set up Web Endpoint Monitoring, was notified by BizTalk360 of problems with a web endpoint. The administrator contacted the owner of the web endpoint, who was not yet aware of the issue! The issue was solved by the owner of the web service and no interruption of the business process took place.

The above example was based on a scenario which really took place in one of our customer and a very clear example of the purpose of Advanced Web Endpoint monitoring.