Data-driven applications have development flows that require data sharing between different platforms. When shared, information must undergo transformations based on the requirements of these applications.
In the past, data was shared as messages through SOAP XML-RPC. This was a common process in the relationship between service and customer to define the structure of the data and methods used. Such architecture follows the cascade model of development, in which the design and objectives of the applications are first developed, and only then are the solutions implemented and tested. This flow made the process difficult when changes were requested, given the close integration between the parties involved. So, this architecture didn’t fit well in the Agile world.
Now, developers can rely on a REST architecture with HTTP. This has changed the way customers and providers send and receive data today. HTTP is a transmission and communication protocol, since the request and response loads are made up of messages. For this reason, the REST architecture allows both client and server to use their own DOA (Distributed Object Architecture) technologies, such as Java, Node.js or ASP.NET, providing a weak coupling. It transformed conventional services into web services for communication between client and server and, also, in microservices for communication between servers. Such an evolution has led many companies to develop API management platforms that have enabled independent APIs to be integrated into enterprise business architectures. API Connect, for example, is an IBM product that made managing APIs simple.
However, are RESTful APIs the only solutions for contemporary Agile development? Would they be the answer to all the architectural problems present in the rapid technological evolution experienced by us? No, not really.
It is easy to develop microservices that operate on different platforms, but when there are many applications and each of them has its own microservice structure, such as the REST APIs, it becomes difficult to standardize and manage projects. In addition, in the event of upgrades to keep up with newer and better emerging technologies, it is complicated to migrate all applications through REST APIs to support the news. There is a huge cost involved in implementing a new architectural solution for an application to migrate completely to a new technological platform.
Some of the biggest challenges related to RESTful APIs concern the standardization of business workflow and the transfer of such platforms to new technologies with the Agile methodology without disrupting current methods and with minimal impacts on the process. In this article, I will explain how we can solve these problems and reduce the cost of supporting and maintaining consolidated systems, as well as migrating to better ones.
Standardization of workflow
With the recurring technological evolution, industrial standards are constantly changing so that business solutions do not need to be rebuilt from scratch. To keep up with your competitors, it is likely that you are integrating third party technologies or open source software available in marketplaces in the cloud into your solutions.
By adopting the latest technologies in the architecture present in your company, the previous applications will be obsolete. These updates can involve a number of changes to the integrated systems. The possibility of making changes with minimal effort and without causing damage to the flow already implemented depends directly on how well the system as a whole has been structured. In the case of microservices with REST APIs, there will be a lot of rework.
An architectural solution is to adopt message queues (in English, message queues, or MQ). They provide reliability and scalability, bringing components autonomy. At first, message queues are used to transmit notifications, but what if you used them in your workflow instead of REST APIs?
Applications can be registered in topics, bringing the possibility of managing events efficiently, as long as the flow has the following configuration:
Standardized data model
Standardized actions for event management
New technologies can be easily implemented when structuring: subscription to topics of interest; construction of targeted prototypes; and creation of minimum viable products (MVPs). Then, applications can work with production data immediately, without the need for scheduled migration scripts or the involvement of multiple teams.
The following examples demonstrate how to standardize workflow and tests
The effort involved in adopting B1 requires understanding the Component B APIs and how the integration between Components A and C would work. This RESTful APIs architectural scheme shows how interlinked the components are and their limited flexibility, as all three need to be updated to become compatible with B1.
Figure 2 shows the same situation, but with the message queue architecture. If all components are subscribed to topics and send messages through a standardized data model, it becomes easier to adopt new technologies without negative impacts on the flow.
The adopted component (B1) does not need to understand the methodology of components A and C, as long as it meets the requirements of the message queue topics and understands the types of information passed on. This architecture provides for weak coupling between components and allows new additions to be made without affecting workflow.
Message queues assist in creating flexible architectural workflow solutions
Typically, workflows have a series of tasks that need to be performed in sequence. They tend to get complicated at times out of sync. For didactic purposes, let’s say that tasks are aligned so that, at some point, a single output is delivered to rendering systems. Then, we will divide the flow into synchronized or unsynchronized pipelines that have tasks.
After each task is completed, an automated or manual review can be implemented that will eventually bring outputs to the new demand and review. In Agile methodology, needs tend to change constantly. Therefore, tasks need autonomous modules and, if possible, to be executed in autonomous applications that meet specific business requirements. This type of architecture makes the system much more flexible, allowing you to add, remove or reorder tasks.
You can adopt the message queue architecture using a product like IBM Event Streams, which provides standalone applications that are independent and easily integrable with each other. With IBM Event Streams, you can develop applications based on data models and subscriptions to desired topics. IBM Event Streams is a cloud-integrated MQ solution that uses Apache Kafka open source technology. It offers reliability, scalability and makes applications standalone, as seen in the example below.
Let’s consider as an example a content publishing platform in the cloud, where the content is published through a simplified workflow. There is a platform responsible for revising and publishing the content, as well as a global platform aimed at revising the translated and published content in a given country. See the flow of this work in Figure 3.
This workflow can be implemented with APIs or microservices. See Figure 4 for an architectural diagram that describes interactions between platforms through APIs.
If the integration platform is an application developed locally with front-end frameworks and server platforms and the goal is to migrate the entire structure to an external content management system, then it is possible that it will be necessary to use the integration platform, a platform globalization and a catalog to provide tons of information related to the APIs involved. All three platforms will require the creation of new APIs to support the new workflow and replicate the existing flow in the content management system. This architecture would require several coordinated teams to align information regarding the APIs involved.
Now, let’s consider the same workflow using messaging architecture and a product like IBM Event Streams.
If the workflow is standardized with well-defined topics and messages in IBM Event Streams as seen in Figure 5, any new platforms can subscribe to the message queues to participate in the new model of performance without interruptions, such as the management system cited content. Minimal changes to component integration are required by this architecture. It also makes migrations much easier by adding new platforms.
Advantages of message queuing architecture over RESTful APIs and microservices
You can see the following advantages when implementing message queue architectures in your processes instead of RESTful APIs and microservices:
Message queuing solutions like IBM Event Streams provide persistent data queues when sending messages, which will make components less susceptible to failure in the event of problems. IBM Event Streams guarantees seven days of persistence, unlike other similar services.
Messaging architectures are more reliable due to the possibility of transmitting messages to various components. Figure 6 shows an API architecture on the left, compared to the messaging architecture on the right.
In the case of APIs, if a component loses the message, Component 1 must make multiple delivery attempts to prevent others from receiving the same content multiple times.
Using message queuing solutions such as IBM Event Streams, we can ensure that all components receive the information with less overhead. Therefore, QM are more reliable.
You can scale any number of instances by subscribing to a single topic. IBM Event Streams allows applications that group a number of instances for a single component or create multiple groups for multiple components to receive the same messages.
Messaging architectures are scalable so you can implement new technology platforms without interrupting your workflow. Figure 7 shows an API architecture on the left compared to a messaging architecture on the right:
In the case of APIs, there is a lot of dependency between the components.
In the case of message queues, a product like IBM Event Streams promotes scalability by allowing new technology platforms to register in the process and start receiving demands without hindering workflow.
Message queues eliminate dependency between components and considerably simplify the integration of decoupled applications.
Messaging architectures are autonomous, as you can discontinue APIs used by various components. On the left, Figure 7 shows an API architecture; on the right, a messaging architecture.
In the case of APIs, it is difficult to discontinue them as it is necessary to identify all components that use them and then notify them of the discontinuation.
In the case of message queues, it is not necessary to identify the components. Component C, for example, can stop sending messages and unsubscribe from the topic so as not to receive them.
The use of message queues is a very powerful and efficient architecture that helps to standardize workflows. The messaging architecture has great flexibility and allows easy adoption and prototyping of new technological platforms without marked changes in the structure as a whole or in individual components. This flexibility will reduce costs and the time involved in migrating to new technology platforms.
Want to read more specialized programming content? Discover the IBM Blue Profile and access exclusive materials, new knowledge journeys and personalized tests. Check it out right now, get the badges and upgrade your career!