This article describes the new async provisioning framework and how you can configure it.

Motivation

Some provisioning operation like context deletion can depending on the data of the context in question take a long time to resolve. This can cause issues in case a thread is waiting for its execution like with the current framework. Additionally those delete operations can cause heavy load on a node which is why dedicated middleware nodes are typically used for provisioning purposes.

To alleviate this situation we created a new framework which can execute provisioning operations asynchronously.

How it works

Instead of executing the task directly the framework creates a task in the new async_task table with all necessary information and prints the task id to the log. A node can then claim this task asynchronously and execute it at any time in the future. This works with all already existing provisioning interfaces (e.g. soap, clt).

It allows full control over when a task is executed. For example it can be executed directly or in a time when the load is usually lower. After a task is executed it's state is tracked both inside the log and inside the async_task table.

Currently only the following tasks are supported: * Context deletion

The tasks are executed by a pool of worker threads. Their work is provided by an automatically filled queue of tasks. This allows fine control over the amount of parallel provisioning operation a node can execute. This also enables the possible to move those operations to dedicated provisioning nodes like before.

A task is only stored for a certain amount of time. Old entries are removed by a regular cleanup job after a retention period is over.

Requirements

Since the context id is used to track deletions it is necessary to use unique context ids. Or in other words reusage of context ids is not supported with this framework. An easy way to achieving this is to use the autocontextid feature of the middleware. You can enable this in your helm chart:

core-mw:
  packages:
    status:
      open-xchange-admin-autocontextid: enabled

How to configure

To use this framework you only need to configure two properties to get up and running.

Firs you need to define a schedule for your worker nodes. For example to perform delete operations only on the weekend:

com.openexchange.admin.async.schedule=Sat-Sun 0-24

Or only at night:

com.openexchange.admin.async.schedule=Mon-Sun 1-4

Then you need to enable the provisioning operations you would like to enable async for. E.g. for context deletion:

com.openexchange.admin.ctx.async.enabled=true

This is all you have to do. If you want to have more control over this process. E.g. if you want to move this to dedicated nodes you can reference these properties.

Q&A

What happens if a node dies during execution?

Every async task is tracked via a timestamp. During execution this timestamp is regulary refreshed and if an unfinished tasks timestamp exceeds a certain threshold the task is retried again.

What happens in case of errors during the execution?

In case of errors the error message is stored inside the task and the task is retried up to a limit amount of times. Again the timestamp mentioned above is used to delay reexecution. This is useful for short outages. E.g. in case the database or any other external service required is not responsding.