What does the integration mean to enterprises in the digital world.

After reading this blog, you should have a better understanding of …

  • Why is integration the key to business innovation?
  • How did the integration get evolved?
  • What does a company need to get transformation journey started?

Why Integration?

In the information technology world, integration is not a sexy topic like IoT, AI, or ML, yet it is critical and is the foundation of almost any modern technology.

From a business point of view, integration is a crucial capability that may differentiate a company from its competitors. Especially if the integration capability of a company is agile, flexible, and scalable, this gives a company a tremendous competitive advantage. A Forrester report revealed the fact around it.

A Strategy For Digital Innovation

In the digital world, the objectives of business operation remain unchanged, which is generating profits, but the way business works toward it has become highly competitive. Many of them would need new digital technologies and strategy to support it.

The meaning of new digital technology is easy to imagine and explain, but what about digital strategy? What is a digital strategy, and how does it help a company transform the business and take a big leap to the next stage? Let me give a real example from an e-commerce company:

Taobao website from Alibaba corp is a popular Chinese e-commerce website, and it has become the 2nd largest online shopping website around the globe, second to Amazon. There were thousands or even tens of thousands of e-commerce website in China, but none of them achieved the scale and success like Taobao. A book written by one of its architects shows that one of the reasons they become so successful is the clear digital strategy and flexible architecture. The strategy is to build a partner ecosystem with everyone in the digital world.

TaoBao Shopping website
TaoBao Shopping website

In the early years of the Taobao website, the company wanted to recruit as many merchants as they can to enrich the numbers and variety of merchandise on their website. However, to achieve that, it’s not enough to do it on their own. They decided to leverage the strength of the partner ecosystem, which meant opening up their e-commerce capabilities to everyone who wants to open their e-commerce shop with their brand. From the internal architecture perspective, they renovated their core e-commerce services to be more modular and service-oriented. They opened their business capabilities by two approaches. One is a merchant platform provide support services for those who do not have IT skills in-house, and another is providing easy to use APIs online. Anybody who wants to create their online shopping website with their brand can do it easily by calling Taobao’s open APIs. This move became the game changer and helped the company dominating the market and become the leader in the country.

Agility is the key

If we look at the technology to support the Taobao strategy, I would say it’s not rocket science. Most of the technology it relied on were not new, and many companies back then could do the same from the technology point of view. However, the key difference is the business agility enabled by technology. From software and organizational perspectives, the company had a more flexible and scalable architecture to respond to the fast-changing speed of the business. The capability gave them an enormous competitive advantage always got ahead of the competition.

If you had heard what other dot-net companies did to get success, you may found the Taobao was not the only company benefit from the digital ecosystem. Amazon and eBay also gained great success with it. For instance, the well-known Amazon’s platform mandate issued in 2002 from Jeff Bezos, CEO of Amazon, which cultivated a new disruptive innovation, AWS.

Bezos Platform Mandate
Bezos Platform Mandate

Learning from those great successes what technology companies achieved, many brick and mortar businesses start thinking if they could replicate the success in their fields. Meanwhile, the success and referenced architecture had also been consumed by software companies who developed solutions to help their customers to gain the same result. One instance is what Red Hat proposed to the market, the Agile Integration solution.

Evolution of Integration

Before we get into the modern integration architecture and technologies like what Agile Integration offers, let’s talk about how integration technologies got evolved after all these years.

Ad Hoc Integration

Decades before, in the mainframe era, systems connect to mainframes and each other through ad hoc integration, which means writing connection or adapter codes in every system to connect out to others.

Ad Hoc to ESB
Ad Hoc to ESB

However, when the growing number of systems caused the integration complexity and made the hardware choice lean to commodity, the industry requires a new way of integration. That’s the time the SOA concept and ESB swung in.

ESB Integration

One of the practices of SOA is through ESB (Enterprise Service Bus). Comparing to ad hoc integration, which is an unorganized approach, ESB provides a centralized way to integrate. The ESB approach worked very well in the Web era because, at the time, the digital interfaces for customers from most enterprises are mainly website. The web technologies are mostly server-side created by technologies like JSP, Servlet, etc.

The ESB approach was an efficient way to do the integration. By forming a dedicate specialist team using specialized integration technology stack, the team can work on all integration development in the organization. However, nowadays, most of the new business requirements are integration-related and require integration team to cooperate on coding, testing, and troubleshooting. Also, to quickly respond to various integration request, the integration specialists usually create some sorts of integration framework on top of the technology stack they use. Which often led to not flexible to change and limited the supportability of the team to the overall business. That’s where the centralized method becomes the bottleneck.

The centralized integration approach worked well for years. However, the challenge from the business was brutal. Many new technology startups which were though smaller but more agile, used technologies with new business ideas created something called disruptive innovation. Those companies do not have the burden of existing IT investments, which give them a significant advantage on adopting new technologies and implement new ideas much faster.

“Agility provides the means for an organization to move quickly, understand quickly, make changes easily, and keep options as open as possible to move flexibly.”

According to IDC report.

When the agility has overtaken the efficiency becomes the priority for enterprises, how does a company transform to be agile becomes a popular topic. And people may not be aware that, actually the integration is highly relevant.

API Centric Integration

If we took a step back and looked at how ESB architecture works to serve IT, we would find two fundamental elements from ESB: translation and composition, and the two elements are where we can work on, finding new ways to do the integration and complement the old ESB approach.

ESB to API Driven
ESB to API Driven

API First

Systems rely on ESB to do translation because historically, the protocol or message exchange format is proprietary. ESB takes the role on supporting all kinds of protocol and message transfer features. However, nowadays, the data exchange protocol and format has standardized by RESTful APIs, and it’s not recommended to create proprietary protocol anymore. The technology shift is like how human being does communication. In the past, people would have difficulty communicating if they speak different languages. However, if there’s a kind of universal language like English that most of the people can speak, then the communication would be easier and reduce the overhead of translation.

Domain-Driven Design And Microservices

About the composition characteristic of the ESB, based on my experience working with many enterprise customers, I found the level of composite vary by the scale and power of the integration team among the organization. Some companies use the ESB mainly for translation but don’t do much on service composition. On the contrary, others heavily rely on ESB to response to new service creation.

As more and more business requirements, different from the traditional business channel, are digital native. Which has the characteristic of short iterate, rapidly changed requirement, and uncertain business workload. The service composition becomes a key factor of creating and updating new services but also become overwhelming to the old ESB team and architecture.

Complement with the API-first, a new solution to the problem is to rethink business domains and functions using methodologies like the Domain-Driven Design. Then forming proper granulated domain contexts and services, and expose them with standard interfaces like RESTful APIs or standard message protocol like AMQP, MQTT. By doing so, new requirements can be better analyzed and decoupled to a set of domain and functions. Those functions can be maintained by smaller teams where people may call them microservice teams.

It is clear to see that to face the digital challenges; companies need to transform the way they do the integration. To be more distributed and with smaller, agile teams who interact with others through standard API interfaces. Also, co-work with business experts leveraging domain-driven methodologies to define clear boundaries for the business. The agile integration solution is designed for these purposes of serving business success in their transformation journey.

Agile Integration

Agile Integration is a solution that implements the API-centric integration approach yet leave the flexibility to enterprises modernizing their existing IT assets. A blog post by Deon Ballard and Sameer Parulkar gave a good explanation from the viewpoint of the development lifecycle. Here, I want to dig more in-depth about how can a company adopt the solution and get real benefits from it.

How can the journey get started?

For every company, the adoption path will be different because there are no two companies have the same problems and opportunities. However, it is worth to learn from the others and pick some lesson-learned into your transformation strategy. In my next post, I will share a real transformation journey of a bank I engaged with to give more in-depth ideas. However, in the following paragraph, I want to talk about the prerequisites a company needs before going down their journey.

Base on success and fail reference I came across, for those enterprises who want to adopt the architecture, I would recommend some prerequisites that need to be considered:

  • Team(s) for the architectural renovation.
  • An automation platform encourage experiments.
  • Clear boundary over performance.

Team(s) for the architectural renovation

Although even just mentioning the reorganization may terrify people in the company, Forming a strategic team dedicate to architectural transformation is necessary. It doesn’t need to reorganize your employees and divide them into microservice groups at once. We can start by forming a small team which is accountable for the new architecture. Then gives them development resources and empowers them to define rule and policy for other teams to follow. The member of the team may start with a business domain expert, a software developer, and a system engineer. The initial goal of the team can be leading the cross-team discussion on domain definition and building prototype around it with standard interfaces. Later, helping other new business requirements on-boarding to the modern integration architecture. Project teams for new business requirements can be a virtually created team including product owner, developers, system engineer, and senior management as a supervisor.

An automation platform encourages experiments

Before we invest and conduct the architectural change, we need to keep in mind that it won’t succeed on the first round. No matter it’s domain-driven design, API design, or the onboarding, it all requires some experiments. So we should give the team some space to fail. The real things that matter is how fast can we see the result every trial round?

That’s why I usually won’t recommend my customers to build the new architecture if they do not reach a certain level of automation and agile development. There are automation technologies that can help to shorten the development iteration, technologies like OpenShift (PaaS/CaaS), Ansible (Infrastructure-as-code), etc.

Clear boundary over performance

At the beginning of building the new integration architecture, we shouldn’t focus too much on creating high-performance services/APIs. Instead, we should concentrate on onboarding as many services composited by cohesive domain APIs as we can. With the support of the automation platform, which enables high iterative speed on new service delivery, the onboarding processes are a great way to train the team. Further down to the road, the communication between IT and business users could also be transformed. Also, with more service go-live and start serving business requests, the economies of scale will be revealed, both from hardware cost and delivery speed perspectives.

What’s next.

I hope this post gives you an idea on why integration has become a weapon to the success of the digital war, and how agile integration can be a way out. Also, based on my personal experience engaging with enterprises, throw in my two cents on what the company should consider when adopting the Agile Integration solution.

For my next post, I plan to share a reference on how an enterprise can benefit from the agile integration solution to get ahead of its competitors.

I hope you enjoy the read and I would be appreciated if you can give some feedback about the post? Leave a comment. 🙂


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.