Microservices and Distributed Applications Meetup | Orkes | Netflix Conductor

Till the early 2000s, the companies used the monolithic architecture for developing applications. This was basically a traditional software program which was single-tiered with multiple software components combined into an application. The software components of the app were tightly coupled and assembled  as a single executable/deployable unit. Hence, to make a change in this type of structure of application, it required to update the code base of the entire stack and then deploy it again which was really time-consuming and inefficient. It had its advantages like easy deployment, simple testing and debugging but had way more complicated disadvantages like slower development, scalability, availability, reliability,etc.

This gradually led to various problems as the demand for the IT services increased which required an upgrade in the then existing IT infrastructure. One such company which faced this issue with the monolithic architecture was Netflix. It was not able to keep up to its rapidly growing demand for video-streaming service & other services through the existing infrastructure and architecture. Thus, it decided to migrate its IT infrastructure from its private data centers to a public cloud and replace its on-premise monolithic architecture with a cloud-based microservices architecture.

 Netflix was one of the pioneer in successfully migrating from monolith to microservices architecture at the time when the term 'microservices' was not well-known. While, currently the microservices architecture has become increasingly common and has benefited various companies over the years in handling the complex problems of scalability, maintenance, agility, etc.

The company created a microservice orchestration platform called Conductor. It solved various problems for Netflix like managing & orchestrating microservices  at scale in a cloud-native environment. Netflix made the platform open-source in 2016 which led to further development of the tool and also helped companies in different scenarios. 

The founding engineers of Conductor at Netflix developed a platform called Orkes Conductor for building reliable & scalable applications quickly. Orkes added various features to the open-source version of the platform which various clients have found helpful & beneficial for their business-critical applications.

I had the wonderful opportunity to learn and explore in depth about Conductor at the Microservices and Distributed Applications Meetup by Orkes at Google, Bengaluru on 30th April 2023. It was great to learn about the working, use-cases, importance and applications of the platform created by Netflix and further enhanced by Orkes.

There was a great line-up of speakers who explicated about the concepts of microservices architecture, serverless computing, microservice orchestration, etc.

Joinal Ahmed elucidated about the core concepts of microservices and its applications. He nicely described about the monolith, microlith, modulith & microservices architecture. Also,  he touched upon the functionalities available in Google Cloud like Cloud Functions, Cloud Run, Google Kubernetes Engine (GKE), App Engine,etc.

Manan Bhatt delved deep into the concept of service orchestration & building distributed applications using Conductor. He enlightened about the importance of Conductor and its advantages. Conductor is the successor to the Pub/Sub Model (Publish-Subscribe) which addresses the shortcomings of the predecessor. It has no single point of failure and designed for failure as it is based on the chaos engineering principles. Also, it has durable queues with delivery guarantee as well as has self-documenting flows. Orkes Conductor has some major additions to the open-source version which are 

-Use of Redis for distributed queues

- Use of PostgreSQL instead of Elasticsearch

-Support for web-hooks

-SaaS scheduler & RBAC on top of Netflix conductor

-Has SDKs for all major languages like Java, Python, C#, JavaScript, Go & Clojure

Bimal Gupta nicely illustrated and explained about the use-cases and importance of Conductor at Cimpress Technology. He described that Cimpress was one of the early adopters of the Netflix conductor and then the company shifted on to the Orkes Conductor platform. It uses the Conductor UI for tracking, filtering, monitoring , error handling, microservices communication via events and orchestration using logic and conditional branching. He also shared that the use of Redis in Orkes Conductor has really benefited the company in reducing workflow time. Also, he stated that Orkes Conductor has many more APIs than its competitors and hence makes it more useful. Furthermore, it has an intuitive UI which helps in error visibility.

It was great to learn about Conductor, Microservices & Distributed Applications at the meetup. Thankful to the organizers for such an amazingly informative session. Looking forward to learn more about  microservice orchestration using Conductor.