Automated workflow: Halved lead time with Camunda

Designing and building a tech product? Why you should be interested in BPMN and Camunda

 

A few weeks ago, a colleague of mine came to me asking for help in designing a new feature. He needed to set up a reminder system so that his clients would not forget to finalise their files on his platform. He needed a reminder 30 days before the file expires, 7 days before and then 30 days after (before deleting it). Something like this:

He had planned 6 weeks to do this but it didn't fit into his roadmap. Today, after finishing the development in 3 weeks, I'm happy to explain how we cut in half our leadtime thanks to Camunda (and even more to come!)

 

BPMN

Before talking about Camunda I have to tell you about BPMN. Why? Because BPMN (Business process model and notation) is a graphical method for modelling business processes. It is the international standard ISO/IEC 19510, but above all it is the method used by Camunda.

• For stakeholders: BPMN notation is useful to all stakeholders in a business process, as it provides a better understanding of the process through a visual representation of all its steps.
• For developers: it is also intended for the people who will implement the business process (the developers in our case) by providing sufficient detail to enable its practical application.

 

The BPMN notation therefore provides a standardised language for all personnel involved, whether technical or non-technical: business people, process participants, managers, technical developers, but also external teams and consultants.

 

Example of a basic workflow:

A BPMN file :

• Must have a beginning (start event)
• At least one end (end event)
• Contains paths and tasks that represent business processes

 

 

 

---

{% icon icon_set="fontawesome-5.14.0" name="Info Circle" style="SOLID" height="20" purpose="decorative" title="Info Circle icon" %} The concept of business process is very broad: it is an activity or set of activities that is put in place to carry out a task, a project or to achieve an objective.

For example:

• The management of a customer order in a restaurant, from the receipt of the request to the arrival of the delivery person and payment.
• The procedure for evacuating a building in the event of a fire
• The processing of an employee's request for leave

---

 

But what is Camunda?

Camunda :

• Is a process orchestrator, a platform that allows developers to design, automate and improve business processes
• Allows visual collaboration between the tech teams and the different stakeholders of a project
• Uses the BPMN standard to make it executable
• Consists of a suite of tools: the engine, the database, 3 interfaces and a REST API

 

Let's take a closer look at the tools offered by Camunda

The cockpit

---

{% icon icon_set="fontawesome-5.14.0" name="Info Circle" style="SOLID" height="20" purpose="decorative" title="Info Circle icon" %} Before going any further it is useful to recall what an instance is:

An instance is an execution of the business case described by BPMN. For example, regarding a process for managing an employee leave request, an instance of this process is the treatment of Mr Smith's request to go on leave from 25/12 to 01/01.

---

 

The docker image you will find in the following tutorial includes an example workflow of invoice processing, the cockpit allows you to view :

• Its BPMN diagram and its different versions (1)
• The currently active instances, their location in the process (2)
• The list that allows access to the details of each instance (its ID, its variables, its potential incidents) (3)
  
  

This interface allows you to manage the activity of your business process and to quickly debug those that are blocked.

By clicking on one of the instances you can see the details:

 

The database

Camunda has its own database which will store all the information about the instances, their variables, tasks, etc... You can find the complete documentation here.

Here is the schema of the main tables:

In short the :

• "Process definitions" table stores the BPMN and its different versions
• "Executions" table stores the instances
• "Variables" and "Tasks" and "Event Subscriptions" tables store the variables, tasks and events related to the current instances

The engine

It is the core of the Camunda system that interacts with the database, processes events, tasks and feeds the graphical interfaces such as the cockpit.

The REST API

 

A REST API is available to interact with the engine. It is (really) very complete (~200 endpoints) and allows some complex operations like migrations.

 

---

{% icon icon_set="fontawesome-5.14.0" name="Info Circle" style="SOLID" height="20" purpose="decorative" title="Info Circle icon" %} What is a migration?

You have seen that there is a history of the different versions of a BPMN diagram. This is so that instances created on old versions of the diagram are not lost. Indeed the new version of the BPMN may be very different from the previous one and may not allow to switch an instance easily from one version to another. For example :

• The task on which the instance is based no longer exists in the next version
• The instance does not have a variable defined but needs one for a newly added brick in the new version
• A new branch has been created to allow the cohabitation of two different instances

For all these reasons Camunda cannot migrate all current instances to the new version of the diagram. If you want this to be the case, you will have to migrate the instances from one version to another.

Here is Camunda's documentation for doing this.

---

 

How to install Camunda to try it on your computer?

Camunda offers an excellent tutorial to get you started in 30 minutes.

In this tutorial you can :

• Install Camunda locally via a docker image
• Create your first BPMN workflow
• Deploy it locally for testing
• Create a worker which performs the process tasks

The tutorial is very clear and easy to access, so I don't have much to add. I just wanted to make a simplified diagram that allows you to view the interactions between the different elements.

Traditionally Camunda interacts best with Java, the language that was used to develop it. But for some time the founders have been focusing on opening up to other technologies. For example, for the tutorial I used the Node client offered by Camunda and a worker in the same language.

Bernd, one of the co-founders of Camunda, explains the technical details of what has been put in place to open up Camunda to as many people as possible in this article. Here is a diagram that will allow you to understand one of the mechanisms easily.

 

This is the detail of the interactions between the worker, the client and the Camunda API. Using a REST API allows any language to be used to interact with the engine. On the left side we see the initialization phase where the worker subscribes to a topic. In the run phase, on the right hand side of the diagram, we see that the client retrieves (fetch) and locks the tasks of the topic to which the worker has subscribed and then sends them to the worker so that it carries out the task. This system of task management avoids the tasks being carried out twice by two workers who would be available at the same time.

However, Camunda has already developed external clients to simplify the management of the tasks to be carried out in the workflow, in javascript and in python. The main interest of these clients is to correctly manage the fetching, locking and execution of the task by the worker. This avoids duplicate execution of a task in case of a long delay in the worker's response.

 

But how did we save time on our project then?

Now that you can see a little more clearly how Camunda works, here's how we saved 3 weeks on our developments:

• I used the basic BPMN features that, with Camunda, allow me to make the dunning system for my clients

Here is the first diagram that was developed for our functional requirement (it was done in 30 minutes)

This is the second version after cleaning up:

 

---

{% icon icon_set="fontawesome-5.14.0" name="Info Circle" style="SOLID" height="20" purpose="decorative" title="Info Circle icon" %} There are some elements used in these diagrams that you can find out about here:

• The timer event
• The gateway
• The receive task

---

 

The amount of code written is significantly shorter as there is only one mail sending brick to develop, most of the work has been made simple and clear with a graph.

As we will see in the next section, there is a good chance that the total time saved will increase again in the future.

Benefits and drawbacks of BPMN

Camunda was a measurable and immediate saving for my project but it also saves time in the long run. Indeed, from an automation point of view, BPMN schemas have many advantages for developers.

Benefits:

• The tasks used in the BPNM diagram can be duplicated and reused very easily.
• The BPMN evolves with the code and is therefore systematically updated. No more dated technical diagrams that you forget to update.
• The onboarding of a newcomer (developer or product owner) on the project is greatly facilitated, using a graphical representation of the business process makes it possible to understand its operation in the blink of an eye.
• The product owner easily understands how the business process is implemented and can therefore discuss with the developers how to make it evolve. BPMN provides a common language for discussion between product owners and developers.

 

However, there are still some drawbacks:

• Each new version of the BPMN is not necessarily backwards compatible, i.e. instances created on a V1 will not necessarily be functional on a V2. That's why Camunda keeps a version of each BPMN so that an instance created on a version stays there and runs as expected when it was created. To have all the instances of the process in the latest version of the BPMN you have to migrate the instances of the old version to the latest version. This can represent a significant cost in some cases.
• If the first diagrams on simple use cases are quick to set up, there is a fairly intense progression curve to understand and master what BPMN and, by extension, Camunda offers. So you have to work your way through it or you'll end up with indigestion.
• To edit BPMN digrams, Camunda offers a tool called modeler. It is very complete but falls short of the tools used by product owners, which are much more intuitive. It is therefore sometimes difficult to get the product owner on board to use it.

Conclusion

I was quite impressed by the power of Camunda and how easy it is to install and use with other languages. I think it shows, so I'd like to point out that unfortunately I wasn't paid to write this article (Bernd knows that I accept all kinds of payments)

I hope that you too can see the value of Camunda in simplifying complex business processes but also in making the project more maintainable in the long run.

Have a nice day!