Starting with 7.10.0, it is possible to upgrade individual database schemata without having any downtimes. This guide is mainly focused on sites with multiple OX nodes and multiple database schemata hosting thousands, if not millions, of contexts, where only a subset of users should be migrated without having downtimes. Note that in this document the words "upgrade" / "update" as well as "server" / "node" will be used interchangeably.

If you have upgraded at least once, you are already familiar with the upgrade process. You may be aware that during an upgrade the OX database is also upgraded (via update tasks). During the OX database upgrade phase, in versions below 8, all contexts that are accommodated in the database schema that is being upgraded, are unavailable, i.e. disabled, until the update tasks are finished, meaning that there will be downtimes for the users that are hosted in those contexts. This behavior changed with OX App Suite 8 and contexts are no longer locked during the upgrade phase.

Even though there are no more downtimes during the upgrade phase, there are still use-cases where you want to upgrade one database schema one after another. Starting with 7.10.0, a property was introduced for the migration/upgrade purpose (com.openexchange.server.migrationRedirectURL). The value of that new property is a URL which points to another deployment with the upgraded version.

The upgrade of a single database schema consists of the following steps. The initial step is to set the value of the migration property on all nodes with the old version in the cluster. The purpose of that URL is to redirect users to the upgraded deployment (see Q and A section). Then, the desired database schema is manually disabled and the update tasks of the new version are executed on that particular schema. Before activating the schema again, it is registered to the new upgraded deployment.

Now, consider the case where a user lands on a pod with the old version after the context for that user was already upgraded to the new version. The pod with the old version won't be able to resolve the context and this is where the migration URL kicks in. No black magic is involved here. If the context exists, then the middleware will resolve its server identifier and will redirect the user to the upgraded deployment.

The Setup

This guide assumes your deployment has been setup with the OX App Suite stack chart and all middleware pods are running the latest version, which is 8.24 at the time of writing this guide. The example deployment contains four middleware pods that are connected to the same database. The following image illustrates this setup:

The upgraded deployment is not active yet, hence it is not part of the illustration.

The Process (Manual)

Now, without further ado let's dive in to the process of upgrade.

Step 1

Set the com.openexchange.server.migrationRedirectURL property on all core-mw pods to point to a new deployment with version 8.25 that does not exist yet. This can be done via the core-mw Helm chart:

appsuite:
  core-mw:
    properties:
      com.openexchange.server.migrationRedirectURL: 'https://appsuite-upgraded.example.com'

Redeploy and wait until all pods have been restarted.

Step 2

From a running 8.24 pod, register a new server that represents the upgraded deployment:

registerserver -A oxadminmaster -P secret -n server-825

Make a note of the new server identifier as it will be used later on to point all contexts that reside within the schema that is going to be upgraded to the new 8.25 deployment.

Step 3

Now that everything is set, you can deploy another helm release of the appsuite stack chart that runs version 8.25. The deployment needs to be accessible from the URL you specified for com.openexchange.server.migrationRedirectURL. Ensure that the name of the newly registered 8.25 server is set as the server name:

appsuite:
  core-mw:
    serverName: 'server-825'

Also remove the com.openexchange.server.migrationRedirectURL property. Optionally scale down the 8.24 deployment to three nodes and the 8.25 node to only one node if you are limited on resources in your cluster. After that the deployment will look as follows:

Step 4

From a running 8.24 pod, disable the schema that is to be updated.

To find out which schemata are available in the installation you can use the listdatabaseschema command line tool:

listdatabaseschema -A oxadminmaster -P secret

After that piece of information is gathered, the desired schema can be disabled with the following command line tool:

disableschema -A oxadminmaster -P secret -m oxdatabase_5

By disabling the schema, all contexts that reside within it are also disabled and all user sessions for those contexts will be invalidated. Any user action will be denied for users in associated contexts since their accounts will be temporary disabled for the upgrade phase.

Step 5

Point all contexts, that reside within the candidate schema that is to be updated, to the 8.25 deployment by using the command line tool changeserver (also from a working 8.25 pod):

changeserver -A oxadminmaster -P secret -s 8 -m oxdatabase_5

The -s parameter indicates the new server identifier mentioned earlier in Step 2.

This ensures that the context to server resolve works as expected.

Step 6

Now, the database update can begin. On the 8.25 pod, execute the runupdate command line tool to trigger the update on the schema:

runupdate -n oxdatabase_5

It should complete without failures.

Step 7

If the previous step completes successfully, the schema can be enabled again:

enableschema -A oxadminmaster -P secret -m oxdatabase_5

After that the setup will look like this:

The database node is removed from the illustration for the sake of clarity. It is always implied that all nodes are connected to the same OX database.

Post Process

The more schemata one migrates to the new cluster, the larger the new cluster must be scaled and the old cluster can be scaled down. Once the 8.24 cluster consists only of two pods, the com.openexchange.server.migrationRedirectURL property can be removed again, the server name can be adjusted to server-825, the core-mw container image can be upgraded to 8.25 and the deployment can be scaled to its original size.

The Command Line Tool (Automatic)

Also note, that since 7.10.0, we provide the upgradeschemata command line tool, which sums up step 2 and 4 to 7, i.e. registers the new server, and then iterates through all schemata in your installation, disables each one, runs the updates, changes the server references and re-enables it. For example:

upgradeschemata -A oxadminmaster -P secret -n server-825

If a server with that name is already registered a warning will be displayed indicating that and prompting the administrator to either abort or continue with the upgrade operation:

WARNING: The specified server is already registered with id '6'.
         If that shouldn't be the case, type 'abort' to abort the upgrade process, otherwise, type 'continue' to proceed.
         If you continue the already existing server will be used to point the updated schemata after the update tasks complete.

The command line tool also provides the -f flag to force the continuation of upgrading all schemata (or as many as possible) even if one or more update tasks in one or more schemata fail. If the flag is absent, then as soon as one schema upgrade fails, the command line tool aborts the operation:

upgradeschemata -A oxadminmaster -P secret -n server-825 -f

To continue the operation and skip the failed schema, the -m flag can be used. If present, then the upgrade continues from where it left of and by skipping the specified schema:

upgradeschemata -A oxadminmaster -P secret -n server-825 -f -m oxdatabase_81

If certain schemata need to be skipped, the -k flag can be used. It defines a comma separated list of the names of the schemata that are to be skipped from the upgrades:

upgradeschemata -A oxadminmaster -P secret -n server-825 -k oxdatabase_81,oxdatabase_44,oxdatabase_51

The flags -m, -k and -f can be combined. In that case it would mean that the upgrade phase will continue even if the updates fail in some schemata, it will start from the specified schema and skip all schemata that are in the list:

upgradeschemata -A oxadminmaster -P secret -n server-825 -f -m oxdatabase_3 -k oxdatabase_81,oxdatabase_44,oxdatabase_51

Q and A

Q: What happens if a user lands on a 8.24 pod after the context was moved to the 8.25 deployment?

A: That pod won't be able to resolve the context. It will then try to locate the server that is responsible for this context. If found, the request will be redirected to that server/cluster.

Q: Is it possible for a user, that the context he or she resides in was not upgraded to 8.25 yet, to land on a 8.25 pod, and if yes what happens then?

A: No. The main site appsuite.example.com will always route the user to 8.24 pods. The user would need to know the address of the upgraded cluster to land on 8.25 pods.

For example, the main site (8.24) is available through appsuite.example.com and the upgraded cluster (8.25) through appsuite-upgraded.example.com. If any user with a context that is still in 8.24 tries to log in to appsuite-upgraded.example.com, then login will fail with a CTX-0012 exception which indicates that the context the user is trying to login is located in another server. No redirect will happen in this case, since it may result in an endless loop.

Q: From an administrator point of view, is it somehow possible to determine whether a context exists on a given server instance?

A: Yes. The OXContextService SOAP end-point and the RMI interface com.openexchange.admin.rmi.OXContextInterface have been enhanced by the method existsInServer to retrieve exactly the information if a context lives within the queried server.