This document describes how to configure the middleware so that pods can be started, stopped and rescheduled at any time, e.g. by a rolling upgrade, a node drain or an autoscaler, without users losing work. It complements Running a cluster, which covers the cluster basics.

Most of it is done by the core-mw Helm chart already. What needs a decision by the operator is marked as such.

Probes

The chart configures three probes:

Readiness includes the main Redis connector, which holds the sessions: if it is unreachable, all pods become unready at the same time. Without it hardly any request can be served anyway. The overall result of the health checks is reused for 10 seconds while it is UP, but only for 2 seconds once it is DOWN, so a pod is ready again right after a short outage, e.g. a Redis failover, instead of up to 10 seconds later. The Redis cache connector is evaluated but ignored, and so is the database pool check while overload detection is enabled (com.openexchange.overload.enabled), which then sheds load per request instead. Use com.openexchange.health.ignore to exclude further checks. Liveness only follows readiness if com.openexchange.health.consecutiveFailureDurationMinutes is set to a value above 0 (default 0); keep it at 0 unless restarting pods actually helps against the failure in question.

Starting up

Starting the middleware is CPU-bound: it starts about a thousand OSGi bundles, with no waiting in between. Measured with the full bundle set until the middleware reports that it is initialized: 54 seconds with half a CPU, 28 seconds with one, 19 seconds with two; beyond that it hardly gets faster. The CPU request (resources.requests.cpu) is therefore what decides how fast a new pod becomes ready, e.g. during a scale-up or a rolling upgrade, as long as the node has no spare CPU to lend.

The image ships a JDK AOT cache, trained on its own bundles when the image is built. It makes the JVM start with the classes already loaded and linked, and cuts the start by about 15 percent. The chart turns it on by default (javaOpts.aotCache). The cache only fits G1 with Compact Object Headers and a heap below 32 GB: with javaOpts.zgc or without javaOpts.compactObjectHeaders the chart turns it off. With a larger heap the JVM starts without it and logs Unable to use AOT cache.

Pods with a CPU limit get more CPU while they start (startupCPUBoost): the main container starts with startupCPUBoost.cpu (default 2) and is resized in place to resources once the pod is ready, without a restart. Measured with a CPU limit of 500m: ready after 53 to 63 seconds with the boost, 155 to 162 seconds without. By default (enabled: auto) only a CPU limit above the request is raised, and only the limit, so that the pod is scheduled exactly as without the boost; this needs Kubernetes 1.33 or later and the chart's Role. With enabled: true the CPU is also boosted without a limit or with a limit equal to the request: the request is raised as well, so that the pod keeps its QoS class, and the pod is scheduled with the boosted request, so the node needs that much CPU unreserved at the start. enabled: false turns the boost off. The resize needs Kubernetes 1.33 or later, an image of the same release as the chart or later, and the chart's Role (serviceAccount.create, rbac.create), which then allows the pod to patch pods/resize. If the pod does not become ready, the CPU is lowered after startupCPUBoost.maxSeconds anyway; if lowering fails, the container log says so and the pod keeps the boosted CPU until it is recreated. A container restart keeps the lowered CPU.

With enableInitialization, the init container only creates the ConfigDB of a fresh stack; registering server, file stores and databases and creating the demo contexts happens in the main container once the middleware has started, and the pod becomes ready when that is done. Formerly the init container started the middleware for this as well, so a fresh stack started it twice.

Shutting down

On scale-down, a rollout or a node drain, Kubernetes runs the preStop hook, sends SIGTERM and kills the pod with SIGKILL once terminationGracePeriodSeconds have passed since the hook started. The chart splits that grace period as follows:

1. preStop sleep (preStopSleepSeconds, default 10): the pod still serves requests while Kubernetes removes it from the Service endpoints and load balancers stop routing to it.
2. Drain: the HTTP engine stops accepting connections and waits for requests in progress. The chart sets com.openexchange.connector.awaitShutDownSeconds to the grace period minus the preStop sleep minus 10 seconds, at least 1 second.
3. Reserve (10 seconds): the remaining bundles stop. Measured locally this takes about 2 seconds.

With the defaults (terminationGracePeriodSeconds: 60) the drain gets 40 seconds. Raise the grace period if regular requests take longer than that; a request that is still running at the end of the drain is cut. Long-lived connections are not a reason to raise it: server-sent event streams end at once with a close event (reason shutdown) and clients reconnect to another pod.

A preStop sleep that is not shorter than the grace period is skipped. Test deployments with a grace period of 1 second therefore stop at once, as before.

Background jobs

When the pod stops after the preStop sleep:

• Data exports in progress pause at their next item first, before the drain. They store what they wrote with a save point, and another pod resumes them at once. This usually takes less than a second and at most 5 seconds; an export that does not pause in time is interrupted and resumes from its last save point. Neither counts as a failed attempt.
• Database clean-up jobs in progress stop in the reserve and release the schemas they work on. Another pod processes them at its next run of the job, not one full period later.

A killed pod leaves both to the others once its claims expired: clean-up jobs after 60 seconds, data exports after com.openexchange.gdpr.dataexport.expirationTime (default 10m). Such an export restarts the module in progress from its last save point. Without com.openexchange.gdpr.dataexport.writeSavePoints (default false) that is the last pause, or the start of the module.

Connections during the drain

This applies to the Jetty HTTP engine only (see Switching the HTTP engine). During the drain Jetty answers new requests with 503, and it closes a connection without I/O after com.openexchange.http.jetty.shutdownIdleTimeoutMillis (default 5000). That covers idle keep-alive connections, which otherwise keep the drain open, as well as slow uploads that pause. A lower value ends the drain sooner, a higher one lets slower clients finish. Since the preStop sleep takes the pod out of the endpoints first, proxies normally send no new requests to it during the drain.

Sharing the upload directory

Files a client uploads to be used in a later request, e.g. attachments uploaded before a mail is sent, large results of jobs that run in the background and downloads of document previews, are kept in the upload directory of the pod that received them. Other pods fetch them from that pod over HTTP. Once that pod is gone, the file is lost, and a pod that shuts down deletes its files.

To let such files outlive their pod:

1. Provide a volume that all pods can read and write at the same time (access mode ReadWriteMany, e.g. NFS). It must support atomic renames and keep modification times, which every access updates.
2. Mount it into every pod, e.g. at /var/spool/open-xchange/uploads, through the chart values extraVolumes and extraMounts, and let com.openexchange.server.UPLOAD_DIRECTORY point to it (the files end up in its subdirectory .OX).
3. Set com.openexchange.filemanagement.sharedUploadDirectory to true.

Set the property on all pods or on none: a pod that considers the directory its own deletes every file of the other pods on start-up. In shared mode a pod only deletes remnants older than one day on start-up, and a shared file is deleted once no pod accessed it within its time-to-live (5 minutes by default).

extraVolumes:
  - name: shared-uploads
    persistentVolumeClaim:
      claimName: core-mw-shared-uploads
extraMounts:
  - name: shared-uploads
    mountPath: /var/spool/open-xchange/uploads
properties:
  com.openexchange.server.UPLOAD_DIRECTORY: /var/spool/open-xchange/uploads
  com.openexchange.filemanagement.sharedUploadDirectory: "true"

Files that are meant for the receiving pod only are not affected and stay in the pod.

Update tasks

A pod that runs update tasks for a database schema holds a lock on that schema and refreshes it every 20 seconds (com.openexchange.groupware.update.refreshIntervalMillis). If the pod is killed meanwhile, another pod takes over once the lock has not been refreshed for 60 seconds: com.openexchange.groupware.update.locked.idleMillis for blocking and com.openexchange.groupware.update.background.idleMillis for background update tasks. Do not set either to 0: the lock of a killed pod would then never be released. The refresh interval must stay well below both.

Until the cached schema state expires, which can take up to an hour, a schema whose background update tasks were taken over may still be reported as running them. Then logging in to a context does not trigger them, while runupdate does.

Run the update tasks right after an upgrade that brings new ones, e.g. by the update job of the chart (update.enabled) or with runallupdate; see update tasks. Otherwise a schema is updated only once a user of it logs in, and until then no permanent push listener starts for its users (see mail push).

Cluster membership

Every pod announces itself in the Redis of its site every 5 seconds, along with what its features tell the others, e.g. the number of its live change streams. Each feature decides how long a silent pod still counts: live change streams 15 seconds, permanent push listeners 30 seconds; after 30 seconds the pod is dropped. A pod leaves as soon as it begins to stop, so the others need not wait. To list the pods, run redis-cli HGETALL 'ox-cluster-members:{m}:values'.

A pod of a StatefulSet uses its pod name as node identifier (POD_NAME_STABLE=true, set by the chart), so that permanent push listeners return to it after a restart; other pods use a random one per start. Every start of a JVM additionally gets an identifier of its own, under which live change streams are tracked.

Scaling

The chart creates a PodDisruptionBudget per type by default, but does not scale the number of pods on its own:

• The PodDisruptionBudget (pdb) lets a node drain or another voluntary disruption stop one pod of a type at a time (maxUnavailable: 1). That never blocks a drain, not even for a type with a single replica. minAvailable takes precedence over maxUnavailable; a value equal to the number of replicas blocks drains. Set pdb.create: false to go without.
• hpa.create: true creates a HorizontalPodAutoscaler that scales on CPU utilization (targetCPUUtilizationPercentage, default 80). Memory utilization does not reflect load well, since the Java heap is sized to a fixed maximum (maxHeapSize or maxRAMPercentage).
• keda.enabled: true creates a KEDA ScaledObject instead, which scales on the load of the HTTP API (needs KEDA and a Prometheus server that scrapes the pods). The two are mutually exclusive.

The load of the HTTP API is best measured as the number of requests in progress. It follows from the timer appsuite_httpapi_requests_seconds by Little's law: the time spent in requests per second equals the average number of requests in progress. By default the ScaledObject sums it over the pods of the type, and KEDA scales so that each pod handles about keda.prometheus.threshold requests at a time:

keda:
  enabled: true
  minReplicaCount: 2
  maxReplicaCount: 10
  prometheus:
    serverAddress: http://prometheus-operated.monitoring.svc:9090
    threshold: "20"
  cpu:
    targetUtilizationPercentage: "80"   # optional second trigger
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # do not scale down on short dips

The default query selects the pods by the labels namespace and pod; set keda.prometheus.query if the scrape configuration labels them differently. Derive the threshold from a load test, or from the value rate(appsuite_httpapi_requests_seconds_sum[2m]) shows for one pod at a load it handles well.

The same metric can drive a plain HorizontalPodAutoscaler through the Prometheus adapter. Expose it as a pods metric with an adapter rule like the following, then add it through hpa.metrics:

# Prometheus adapter rule
- seriesQuery: 'appsuite_httpapi_requests_seconds_sum{namespace!="",pod!=""}'
  resources:
    overrides:
      namespace: { resource: namespace }
      pod: { resource: pod }
  name:
    as: "appsuite_httpapi_requests_in_progress"
  metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

# core-mw values
hpa:
  create: true
  metrics:
    - type: Pods
      pods:
        metric:
          name: appsuite_httpapi_requests_in_progress
        target:
          type: AverageValue
          averageValue: "20"

Starting a pod takes up to a few minutes, and some work needs at least one running pod. Scaling to zero pods is therefore not supported; keda.minReplicaCount must be at least 1. Before scaling down frequently, consider the known limitations below.

Known limitations

These apply to pods that are stopped or killed, so frequent scaling makes them more likely:

• Permanent push listeners of IMAP IDLE (com.openexchange.push.imapidle.supportsPermanentListeners) and of the mail notification transport are spread over all pods with permanent push enabled, and move when pods come or go (see mail push). A pod that stops orderly releases its users at once, and the others start them at com.openexchange.push.permanent.maxStartsPerSecond each. A pod joining asks the others for its share at that rate, and they keep each user until asked, so a user goes without push for a few seconds only; the whole share takes its number of listeners - one per user and push manager - divided by that rate. The users of a killed pod start on the others after up to 90 seconds, when its leases in Redis expired. Pods of a StatefulSet keep their users across a restart. Mail notify works with a unicast target such as a Kubernetes service: a pod receiving a notification for a user it does not serve forwards it to the others. Push via Dovecot's HTTP notifications is not affected.
• Drive listen requests in progress may end with an error when the pod drains, and the client sends a new one. A change that arrives while the client has no listen request on any pod may only be noticed at the client's next regular synchronization.
• Sessions removed while Redis is unreachable, by a logout or an administrator, are removed from the pod at once and from Redis within a few seconds after it is reachable again. If the pod is killed before that, such a session becomes valid again. A pod that shuts down regularly logs how many removals it could not apply.
• Exchange ActiveSync and USM ping requests in progress may be cut when the drain starts, since the HTTP engine interrupts the threads registered for that at its beginning. Devices are expected to send a new ping.