Monitoring & Orchestrating Your Microservices Landscape using Workflow Automation (Parts 4/5 of 7)

On Wednesday, March 11, 2020, I conducted the webinar titled “Monitoring & Orchestrating Your Microservices Landscape using Workflow Automation”. Not only was I overwhelmed by the number of attendees, but we also got a huge list of interesting questions before and especially during the webinar. Some of them were answered, but a lot of them were not. I want to answer all open questions in this series of seven blog posts.

Part 1: BPMN & modeling related questions (6 answers)

Part 2: Architecture related questions (12)

Part 3: Stack & technology questions (6)

Part 4: Camunda product-related questions (5)

Part 5: Camunda Optimize specific questions (3)

Part 6: Questions about best practices (5)

Part 7: Questions around project layout, journey and value proposition (3)

Note that we also started to experiment with the Camunda’s question corner and discuss to make this more frequent, so keep an eye to our community for more opportunities to ask anything (especially as in-person events are canceled for some time).

Camunda product-related questions

Q: What is the difference between Camunda BPM and Zeebe?

Or different forms of asking the same question: How do you position Camunda BPM vs Zeebe in relation to this presentation? Is Camunda BPM still the best/most reliable solution for microservice architecture with orchestration flows? Or is Zeebe the recommended route for such a new project?

To get everybody on the same page first, within Camunda we have two open-source workflow engine projects:

Camunda BPM: A BPMN workflow engine, that persists state via a relational database. The engine itself is stateless, and if you cluster the engine all nodes meet in the database.Zeebe: A BPMN workflow engine, that persists state on its own (kind of event sourcing). Zeebe forms an own distributed system and replicates its state to other nodes using a RAFT protocol. If you want to learn more about if check out Zeebe.io — a horizontally scalable distributed workflow engine.

Camunda BPM is our fully featured BPM platform that covers a wide range of use cases across different industries. Zeebe on the other hand is the engine that powers Camunda Cloud, our managed workflow service (workflow as a service). We have built Zeebe to allow horizontal scaling, to run smoothly in cloud-native environments and to not require any third party dependency like a database. This is great but also harder to operate for many companies that are not yet fully invested in cloud-native architectures. And of course, Zeebe does not yet have all the features of Camunda BPM.

So my recommendation is to use Camunda BPM unless you want to use Camunda Cloud, which then means Zeebe.

I would however try to use External Tasks as much as possible in Camunda BPM. This will set your application up to be architecturally ready for future migrations. One customer for example wanted to leverage the managed services and thus migrated from Camunda BPM to Camunda Cloud — and it was a relatively easy endeavor because of this.

Q: Can we manage a large number of instances?

Yes. But it always depends.

First of all: does this large number of instances waits most of the time? Or does it refer to a large number of requests? Then: What is a large number? 100? 1,000? 1,000,000? Per Day? Per Hour? Per Second?

In our best practice Performance Tuning Camunda we gave some recommendations on how to load test and tune the engine. The only reliable way to give a good answer is to set up a load test that mimics the patterns you will need.

But so far we were able to make every scenario we encountered work :-) A good resource to learn that you can scale quite far is also the 24h Fitness case study.

Q: Is it possible to use Camunda BPM for microservices swarm orchestration and data flow regulation in hi-load systems (25M events per day)? How we can exclude or optimize the DB (as a bottleneck) from the processes?

This relates to the last question but is more precise on the numbers. So doing the math 25M events are approx. 300 events per second. I have no idea about the load distribution, but normally you always need to look at peaks too, so let’s assume 500.

Now I don’t know what these 500 events do within Camunda, but for simplicity, let’s assume they are correlated as messages, so every event leading to one request within Camunda.

In that ballpark it makes sense to have a closer look at performance and I would recommend doing a proper load test. If you have findings it is still possible to tune the engine, e.g. adjust the database indices (if you know what you are doing) or reducing the amount of history data being written.

But I want to repeat myself: so far we were able to make every scenario we encountered work :-) Best talk to us about your concrete use case.

Q: How to design high-performance large volumes business processes? Especially large digital transformation projects specific to 5G business. Currently working on POC for German telco in 5G for BSS stack using Camunda.

This kind of relates to the last question and requires a deeper look into the specific scenario. Please reach out to us to discuss this.

Q: What are the most significant enterprise features of Camunda that enable workflow orchestration which are not present with the community version?

The good news is, that there is no significant feature missing in the community edition, that would stop you from doing microservices orchestration. It is important for us, that our community edition is usable in real-life scenarios!

That’s said, of course, we need to provide enough motivation to go for the enterprise edition to pay everybody’s salary. This is mostly about all the additional tooling you need if you apply Camunda in bigger and mission-critical scenarios: features in the operations tool around diving into historic data, fixing problems at scale, migrating versions and so on. And Camunda Optimize as the analytic tool is also enterprise only.

On top of that, of course you get support, more access to our consulting services, additional warranties and further patches for old versions.

You can find a comparison table here.

Camunda Optimize specific questions

Q: Is Camunda Optimize Event-Mapping/Ingestion available now? Open-source or as a product? Are there any plans in the future to provide a community edition of Optimize?

Process events monitoring was released with Optimize 3 and is already available today.

There is no community edition available of Camunda Optimize and there are no plans to provide one. Optimize for us is an important up-selling feature as discussed in the last question.

That’s said there is a free trial version available and I could imagine, we will also have ways to easily leverage Camunda Optimize in Camunda Cloud soon (maybe even a free tier? But nothing I can promise !).

Q: In the Optimize demo, you created a process and mapped it to the Kafka events. Did you deploy it to a Camunda engine which provided Camunda history events to Optimize? If so, is that Camunda engine part of Optimize?

No, Camunda Optimize just needs Elasticsearch as a data store. It does not need any workflow engine to do process events monitoring.

Q: Maybe instead of calling it “Process Discovery”, how about the buzzword-compliant “Process Mining”?

Let’s quote Wikipedia:

Process mining is a family of techniques in the field of process management that support the analysis of business processes based on event logs. During process mining, specialized data mining algorithms are applied to event log data in order to identify trends, patterns and details contained in event logs recorded by an information system.

Process discovery is one of these techniques, which can derive a process model from all the events you ingest. And this is what I talked about in the webinar: We want to add process discovery to the product (and we already had a working prototype).

Additionally, I would not (yet) call Optimize a fully-fledged process mining tool — but I know that I am often too honest for this world and some other vendors don’t care about exact category boundaries too much ;-)

The next blog post in this series will look at best practices.

Also published on: https://blog.bernd-ruecker.com/microservices-webinar-faq-1a9741f4481c#dd04