Monitoring Azure Integration Services with Proactive Strategies

Enterprises are increasingly turning to cloud-based integration solutions to streamline their application development and management processes. Azure Integration Services is a cloud-based integration platform provided by Microsoft, designed to facilitate the integration of various enterprise applications and systems. It offers a range of tools and services that help to simplify and accelerate the development of enterprise applications, as well as improve their scalability, reliability, and security. The below diagram shows some of the common resources on Azure which are used in the Azure Integration Services offering.

Effective monitoring is a crucial aspect of any Azure Integration Services implementation. Without proactive monitoring, it can be difficult to identify and address issues before they become critical. But creating a fail-proof monitoring strategy for Azure Integration Services can be challenging. In this blog post, we’ll explore the key components of a proactive monitoring strategy for Azure Integration Services that will help you stay on top of potential issues and ensure smooth operations for your business-critical applications.

Scenario: A booking management sequence

To understand the power of the Azure Integration Services platform, let us consider a booking management system that may have a flow similar to the below steps:

A customer initiates a booking request through a mobile app or website.

1. The request is sent to the business application, which processes it and validates the request based on various criteria such as availability, pricing, and other business rules.
2. The business application interacts with the existing ERP (Enterprise Resource Planning) system to ensure that the requested product or service is available and that it can be fulfilled within the specified timeframe.
3. The CRM (Customer Relationship Management) system is also consulted to ensure that the customer’s data and history are up-to-date and that any relevant preferences or special requests are considered.
4. If the request is approved, the business application generates an order or booking confirmation and sends it to the customer via the mobile app or website.
5. The ERP system updates its inventory and order management systems to reflect the new transaction.
6. The CRM system updates its customer database with new orders or booking information, including any feedback or ratings provided by the customer.
7. The business application generates any necessary notifications or alerts to internal stakeholders, such as fulfillment teams or support personnel, to ensure that the order or booking is processed and delivered as expected.

Logical Architecture

To implement a solution that would meet the needs of the business requirements we could implement a logical architecture shown in the below diagram. This would give us the capabilities needed to provide a lightweight and flexible solution making the most of modern cloud platforms.

Each block identified within the diagram has a specific purpose:

1. Messaging Service – The Customer facing website or mobile app needs to be decoupled from the backend Orchestrator to deal with any unpredictable spikes in traffic. This asynchronous messaging pattern also ensures durability to survive intermittent failures.
2. Event Routing Service– The goal is to eliminate polling and save the associated cost and latency. The orchestrator service should be informed of the availability of a new booking message in the messaging service.
3. Orchestrator Service – This is the heart of the business application. The entire workflow of processing the booking will be defined in this service. Integrating the existing Cloud or on-premises systems like ERP and CRM is necessary to complete the booking.
4. API Management Service – Connect on-premises systems like the ERP and CRM applications to the Orchestrator Cloud Service to safely integrate the Cloud and on-premises environments. Protect the APIs with keys, tokens, and IP filtering. As the orchestrator definition evolves, the existing client applications should continue functioning without modification.
5. Compute Service – The service should allow end-to-end development experience in a language the existing team has expertise in to solve complex orchestration requirements. The need is to build and deploy locally without additional setup, deploy and operate at scale in the Cloud, and integrate with other services using triggers and bindings.
6. Data Transformation Service– The data from the external ERP and CRM systems need to be transformed and stored in the SQL database for the computing service to use.

Designing an application using Azure Integration Services

With Azure, an Enterprise Integration application can be built by assembling Lego blocks. Choosing the appropriate Azure Integration Service is critical in building the Integration Application correctly.

The above logical architecture could be implemented using the following Azure Integration Service components:

• Service Bus
• Event Grid
• Logic Apps
• API Management
• Azure Functions
• Azure Data Factory

The implementation of the logical architecture would look something like the below diagram.

Let us understand the capabilities of each Azure Integration Service in the diagram above.

1. Service Bus – Azure Service Bus provides a reliable cloud messaging service between applications and services, even when they are offline. Moreover, the Enterprise must ensure order processing and/or duplicate detection can be achieved through simple configurations, saving considerable time and effort. In this scenario, as the requirement is 1-1 decoupling, Service Bus Queue can be the best fit. Enabling Sessions ensures ordered processing; duplicate detection can be turned on with a duplicate detection time frame window.
2. Event Grid – Manages the routing of all events from any source to the destination. In this case, event-driven architecture is adopted to prevent the Orchestrator service from polling the Service Bus queue for any new message. The Event Grid will trigger an event to the downstream application on the arrival of the new message, hence being cost-effective.
3. Logic App – Seamlessly and securely connect Logic Apps to cloud-based and on-premise solutions, in this case, to the ERP and CRM solutions taking advantage of the hundreds of out-of-the-box connectors available and implement complex workflows that orchestrate across your services
4. API Management (APIM) – Inbound policies in APIM help control how data and services are being exposed to participating applications by allowing the definition of authentication, authorization, and usage limits. We can meet the security and compliance requirements with unified management experience across all internal and external APIs.
5. Azure Functions – Functions extensions on Visual Studio and Visual Studio Code facilitate efficient development in the preferred language on a local machine, fully integrated with the Azure platform. Continuous Integration and Continuous Delivery (CI/CD) can be achieved using Azure pipeline definitions. With a programming model based on triggers and bindings, use triggers to define how functions should get invoked and use bindings to connect to other services declaratively.
6. Azure Data Factory – Offers a single solution to ingest data from diverse and multiple sources allowing support to 90 built-in connectors to acquire data from Big Data sources like Amazon Redshift, Google BigQuery, and HDFS; enterprise data warehouses like Oracle Exadata, and Teradata; SaaS apps like Salesforce, Marketo, and ServiceNow; and all Azure data services.

Azure Integration Services provides a microservices-friendly approach allowing you to build more scalable and stable event-driven applications!

The importance of monitoring Azure Integration Services

Integration is the heart of modern business applications. In any typical Enterprise, distributed subsystems are integrated to complete the business workflow. Failure in these integrations should be spotted and fixed before it impacts the customer. Hence, it is critical to enable continuous monitoring of the underlying integration.

Consider the business application mentioned above; what if the Logic App listening to the incoming messages to Service Bus Queue goes down?

• Incoming messages will not be processed, they may end up stuck on the queue or errors processing then will end up with messages on the dead letter queue.
• The Integration is broken!

What does it mean to the business?

• Orders are blocked
• Failure to serve the customer on time.
• Loss of business and goodwill.

How to prevent this?

• The entire integration should be continuously monitored for any failures.
• Access the right toolset to analyze the root cause and fix the failure before business impacts.

Available monitoring solutions

The Azure portal offers native monitoring solutions to monitor Azure Integration Services on their metrics. Choosing the key performance indicators can help to keep an eye on any failure.

Monitoring the ‘Runs Failed’ and ‘Trigger Failed’ metrics on a Logic App can help detect failed runs.

Upon detecting a failure in the Logic App, one can investigate the Trigger and Run histories to gather more details on the failure.

Enabling Log Analytics and Application Insights can also help you to analyze the root cause of a failure.

However, we need to be aware of the challenges in operating the application in the real world:

• The Azure portal lacks one all-inclusive view of the entire Azure estate with the real-time status of the participating Azure Integration Services. In most cases, the Azure Integration Services will spread across multiple Azure subscriptions, making the scenario even more complex. For efficient operations, it is necessary to spotlight the failure, which is challenging through the interface offered by the Azure portal.
• Alert Storm – The scope of the Azure metric monitoring alert rule is restricted to a resource; hence to get the status of the entire integration, multiple alert rules should be created, leading to several alerts. Also, note that there is a cost associated with every alert configuration. After all the efforts and costs, ending up with many discrete alert reports is less valuable.
• Azure Experts – Log Analytics and Application insights are excellent tools for debugging or performing a root cause analysis. However, they invite Azure experts to be involved in Azure support, which leads to the elevated total cost of ownership, one of the biggest challenges Enterprises are keen to solve. Shifting left the operations is the desired solution, but the Azure portal is too complex for the Operations team with less Azure expertise.

In our decade of experience working with Integration customers, we observed that 90% of the issues in production are functional or data errors. The challenge for most customers is that support of the applications can only be performed by developers and experts in the implementation of the solution. The support or operations team cannot easily be involved in the day-to-day operation of your solution because of the restrictive tooling and significant skill sets required.

The Serverless360 value proposition is to lower those barriers to supporting your solution and to make it easier for your operations team to take charge of the day-to-day operations and only need your developers and experts by exception rather than by default. We see this as shifting support to the left in your organization.

This Azure support strategy can save considerable time and effort for Azure Experts, allowing them to focus on innovation for business and ultimately reducing the Total Cost of Ownership.

How is Serverless360 different from any APM products?

Numerous Application Performance Monitoring (APM) products are available for monitoring Azure Integration Services. Any APM will only detect a problem in the Azure Integration, but what is more important is the deeply integrated toolset to fix the issue and restore the business. Serverless360 is crafted with capabilities to complement the Azure portal and aims at ‘Shift Left Support.’

Essential monitoring perspectives for Azure Integration Services

An Azure integration should be monitored from 4 essential perspectives:

• Static monitoring of the status of the participating resources, key performance indicators from the available metrics, and resource properties.
• Run time monitoring, real-time end-to-end tracking of the message flowing through the Azure Integration Services.
• Keep an eye on the evolution of the Azure subscription and document the platform for governance and audit purposes
• Cost monitoring to be aware of your efficiency and to prevent unexpected costs

Serverless360 offers four core modules, each mapping to one of the perspectives mentioned above ideal for different Stakeholders involved in Cloud Operations. Below, you will find an overview of those modules.

Unified Observability – Business Applications

An Azure subscription can grow much faster than we expect. Below is a view of resources in a subscription in the Azure portal.

The resources exist in siloes in the Azure portal. Moreover, one cannot derive critical information required to support integration in production, like ‘Which integration this resource belongs to?’ or ‘What is its current status?’

Serverless360 can offer an operation-friendly interface that allows visualizing the siloed resources in an application context, spotting the failure in service, and offering the toolset to fix the identified issue and fulfilling the security requirement of granting the need-only permissions to the authorized team members.

Below is the representation of the same set of resources as Business Applications. One can clearly understand which application in my Azure estate needs immediate attention. The best part is that the business applications setting can be done in an hour and does not involve much effort.

Business Applications in Serverless360 allow the definition of Auto Map rules. Configure a rule to map any resource with a specific tag or any resource created in a specific resource group to a Business Application. Monitoring profiles can be configured to monitor the key performance metrics on the Azure resource mapped to a Business Application. That’s it; Serverless360 will continuously monitor the resources as your Azure estate evolves. The Operations team can rely on the Serverless360 dashboard to detect issues in your entire Azure estate.

A resource mapped to a business application can be extensively monitored on its status, properties, and metrics. The Service Map feature can help the team visualize the resources as an application and indicate the resource status. The deeply integrated tooling for failure management enables the support team to restore failures like resubmitting the dead letters in Service Bus or Event Grid and reprocessing the failed executions in Logic Apps, Data Factory pipelines, and Azure Functions.

End-to-end tracking – Business activity monitoring

Business operations often get struck with the question, “Where is my message?” In a modern distributed application context, this is a much more complex question to be addressed.

Business users may not have a technical understanding of the application, but they will have a sound understanding of the functions and the data associated with the business.

Introducing Serverless360 BAM for business users unlocks achieving end-to-end tracking of messages flowing through the distributed application.

The best part here is that Serverless360 BAM is technology-agnostic and platform-independent. Enable your business users to locate a specific transaction, spot any exceptions or long-running unresponsive processes, or derive strategic business decisions.

Azure Data to Insights – Azure Documenter

Enterprise integrations in Azure evolve in no time. Even before we realize the volume of Azure resources grows massive, it becomes hard to derive insights from their usage.

Azure Documenter in Serverless360 offers reports to understand the evolution of the subscription(s).

Consider that every team is assigned a resource group to build their integration. The Resource Auditing report in Azure Documenter can compare the current month’s spending against the last month’s, an executive summary of the entire subscription.

Reporting the security compliance of the application is also made simple.

Cost control – Cost Analyzer

One of the key reasons organizations choose to migrate to the Cloud is to cut off the pre-computation cost on the infrastructure. While taking advantage of the pay-as-you-go model, monitoring Azure spending is crucial.

Cost Analyzer in Serverless360 allows cloud economists to create a single pane view to deliver the desired insights across subscriptions to drive actions.

Setting up budget alerts and optimization schedules to cut unnecessary costs are essential tools for cost control.

Business Value

With Serverless360, enterprises across several business domains have devised a friction-free support strategy to shift- left the support tasks in the support escalation path, allowing the Azure experts to focus on business innovation. The result is a significant reduction in the Total Cost of Ownership and considerable improvement in the resolution time, ensuring Customer Delight through operational excellence.

Conclusion

To sum it up, Serverless360 is an advanced cloud management platform that offers comprehensive monitoring of 4 essential perspectives to address the requirements of various stakeholders involved in Azure Operations. Serverless360 stands out in comparison with several APM solutions available in the market, as an APM can only help detect an issue, whereas Serverless360 is perfect to Visualize, Spot, and Fix issues with the underlying integration.

I strongly recommend checking out the Enterprise case studies on devising an Azure support strategy using Serverless360.

If you want to discuss how Serverless360 can help address your challenges with administering your Azure integrations, why not contact us? We are always happy to discuss any challenges over a call. If there is a match, we can give you an obligation-free demo of the product, or you can take a 15 days free-ride to explore the product yourself.