From Disconnected Chaos to Unified Systems: The Evolution of Enterprise Integration

Introduction:

Before explaining what an ESB (Enterprise Service Bus) is, we should establish some context. The first question to ask is: What brought us to here? The answer is not simple because it depends on problems that perhaps modern developers have not faced.

If you are working in a large company, you might constantly wonder how you arrived at the current state and what might have been overlooked. If you're an architect, or someone with a role that's complex to define, by examining multiple projects within your company, you'll notice that applications have cohesion among them. Yet, they are designed as if this isn't the case. This is often due to a lack of long-term vision. If these applications need to share information with each other, we're faced with a difficult technical and management process, because each app belongs to a different department and may be developed using different programming languages and tools. Here, I'm trying to highlight the importance of SOA (Service-Oriented Architecture).

Perhaps members of the hierarchy are hesitating to collaborate and discuss their apps, but they need to, because they should offer a good service to the user at the minimum cost, otherwise, another company will.

How could we integrate apps in the past?

When talking about sharing information (messages) between apps (services), we're not referring to someone who will write information on a sheet of paper and send it to a member of another team, who will then manually input it into their app. That method is inefficient, that's not what we're considering. We are trying to find a modern way to transfer messages that need to be secure, reliable, monitorable, manageable, and, finally, loosely coupled. It means that we should integrate our applications into a system so that they can communicate and share messages.

But, when you're reading this article, do you think that we started with a perfect solution? Of course not, so let's take a deep dive into history.

Point-to-point integration

Point-to-point integration was the first method employed to achieve integration between apps. Before I explain how it works, here is a figure that illustrates how point-to-point architecture functions:

The point-to-point integration approach connects two apps with a direct relationship. Imagine we want to establish communication between house 1 and house 2 using a P2P connection, we would have a direct line between the two, It means that if the line goes down, we will lose the connection between the two houses, herefore, we cannot confidently say that we have a reliable system. If we wanted to add house 3, we would need to add additional lines between houses 1 and 3, and 2 and 3, This means that our houses are tightly coupled. It's also insecure because we have a direct relationship, which becomes a single point of attack for spying on our data. Moreover, it's non-monitorible because we can't easily keep track of our communications due to the lack of a central monitoring system. And finally, it's non-manageable because managing direct lines between a large number of houses is challenging.

Now, we can draw a parallel between houses and a service-oriented architecture. For instance, as seen in figure 4, service CRM is directly coupled to Sales, and HR is directly connected to Finance. This setup is risky because if we make changes in the Finance service, we must initiate a new integration process to adapt the all other services. And that's a laborious task to undertake each time we need to make modifications to the services.

So, is this approach still applicable? Clearly not. After P2P, we moved on to something called Central EAI (Enterprise Application Integration) broker, which is a better approach. Let's explore this in the following paragraphs.

Central Enterprise Application Integration Broker

To be clear, this approach resolved all the P2P issues by connecting applications through a centralized system, and that's the key word: centralized. So, if we want to benefit from this approach, the leaders of each project or service must give up control to a centralized infrastructure where each service is integrated. Thus, our system will be secure, reliable, loosely coupled, monitorable, and manageable, but we will lose some control over our services.

Perhaps this seems a bit complicated, let's simplify it with a real-world example. Imagine our city has a single central post office instead of each person delivering their mail directly to recipients (P2P). Now, we need only to have the address of each recipient and send the mail to the central post office, which has complete control over the delivery process. This centralization ensures a secure process because the post office has invested in reducing the risk of theft or loss to a minimum. Additionally, the post office can easily monitor and manage mail through a system that keeps track of every parcel sent and received. This method is an improvement over the previous system, which was established to expedite mail delivery. If necessary, the post office will establish new processes to ensure delivery without involving the sender and the receiver in these issues.

Just like the post office, the centralized EAI broker features a central hub responsible for the routing, security, monitoring, and transformation of data to ensure it is in a format that can be understood by the receiving service.

Like we see, this architecture is great, it solved all of the P2P problems, but it created new ones. We now have a single point of failure, if the central hub id down, all the communications will fail, and it's really difficult to communicate with a service that we can't integrate into our centralized EAI broker because it's an external one that doesn't belong to our company.

Conclusion

Now that we have reviewed the approaches we had in the past, in the coming articles we will cover the modern ways to integrate our services, without having the issues we faced previously.
To give you a hint, imagine that instead of having a single centralized post office in our city, we have multiple small post offices that do the work and can scale if the city grows.
See you in the next articles of this series.