App Suite 8 Operations Guide

Transition guide

Terminology: Transition (and general concepts)

The process to move to App Suite 8 when running App Suite 7 is designed to be a transition rather than a migration.

This means that we have a lightweight process which (speaking simplified) replaces the App Suite 7 VMs with App Suite 8 Pods running on k8s. The middleware will connect to the same databases and filestores, and use the same IMAP service for the email accounts of the users. From that point of view the App Suite 7 to App Suite 8 transition is more like for example the upgrade from App Suite 7.8 to 7.10 (2018).

To indicate that the technical process is more involved (replacing the VM hypervisor with a k8s cluster; adopt to the new tooling of the k8s world rather than managing legacy VMs with packages), we picked the word transition for this process.

In particular we are avoiding a process where we create a completely new platform including storage and migrate (copy data) from legacy to k8s. (It is still possible to do that if it is desired for any reason, but we aim to provide means which allow for a lightweight transition rather than a heavyweight migration.)

From the application point of view, the App Suite 8 software is to a very large extent the same software as with App Suite 7.

  • We still have mostly the same OSGI based modular Java middleware as before. It was not "rewritten" or "replaced", but "improved" and "adjusted".
  • The UI is still very similar by point of view of the Javascript code as it runs in the browsers, but the mechanism for delivering the code to the users browsers was entirely rewritten / replaced.

Most changes were done to "packaging" (we now provide container images, built directly from the source, rather than packages) and adding k8s / Helm automation / orchestration (there was no such functionality provided in the App Suite 7 released deliverables), and in some points we made adjustments to the application itself to "run better" on k8s, or, make it a bit more "cloud native".

The same configuration settings exist as before, but the configuration mechanism is different than before (values.yaml vs individual config files).

Larger changes to the software are in scope of the evolution of App Suite 8 itself over the years.

Note: in the following, we use the shortcuts "as7" and "as8" to refer to App Suite 7 and App Suite 8k respectively.

Upgrade Scenarios

Big Bang

Note: Most real-world upgrades will not chose the Big Bang approach. But it is a viable process, maybe applicable to smaller or lab environments, In particular as it explains concepts and terminology also applicable for more involved processes, we invite you to read and understand this section first.

In the Big Bang approach, we assume we have a k8s service which is, networking-wise, "close" to the existing as7 platform, including (or, in particular) database and filestores.

Then, as8 (Helm values.yaml based) configuration is prepared such that we map as7 configuration settings into the proper as8 representation. In particular settings for infrastructure access, authentication flow, user data model, third party systems integration, static secret keys, etc are required to be mapped exactly into the as8 configuration.

Based on this configuration, as8 is deployed into the k8s service. It will access the same storage (databases, filestorages, mail system) as the as7 system.

The next step is to execute database UpdateTasks to bring the database schema on as8 level. The usual principles of the as7 world applies, with slightly adjusted k8s-aware tooling.

In this state, the as8 system can be acceptance-tested.

Subsequently, in a maintenance window, the HTTP routing can be adjusted such that user sessions are no longer routed to the as7 system, but rather to the as8 system. All users will then have new as8 experience.

Co-existence scenario

Of course there are certain "flaws" with the "Big Bang" approach which are usually preferred to be avoided for large production systems. We want to address the alternatives in the following, and the first item to be discussed is the Co-existence scenario.

Consider the process described in the Big-Bang section. Rather than switching all users at the same time to the as8 system, it is also possible to run both systems in parallel, for example with different DNS names used for accessing the as7 and as8 systems.

It is technically possible to run sessions on both (as7 and as8) systems at the same time, with a few constraints / drawbacks (see next paragraph), but we don't recommend to do so actually for production systems, but use a more advanced scenario (see below) if a Big Bang switch is not desired. But as a technical concept (and as basis for more advanced scenarios), the co-existence scenario is technically valid.

The main drawback is that it is not possible to join the as7 and as8 systems into a shared Hazelcast cluster. Therefore, the use cases provided by Hazelcast cluster membership are not available across "version boundaries". In particular this refers to event propagation (remote cache invalidation), realtime features (OX Documents), Hazelcast session storage (seamless session failover). In consequence, it is not recommended (and, formally, not supported) to run users of the same context in production on both versions (as7 and as8). Technical tools exist to enforce such a "context consistent" sitation where entire contexts are run either on legacy or k8s (see below, "gradual context switch").

Hybrid transition deployment

One of the disadvantages of the "Big Bang" scenario is that the load on the production systems goes from 0 to 100% immediately, carrying the risk of requiring a roll-back.

We introduced the concept of a Hybrid Transition Deployment as a tool to decompose the Big Bang switch into a two-step process, namely into a low-risk, high-visible-changes step, and a higher-risk, no-visible-changes step.

The central idea of this hybrid transition deployment is that the as8 frontend (the HTML/Javascript application running in the user's browsers) is compatible to run vs latest as7 middlewares. This allows for a setup where, on the service side, the as7 frontend services (formerly known as "frontend nodes") are replaced by as8 frontend services running on k8s, while API requests are still routed to the exising production as7 middlewares.

In detail this carries the advantages:

  • The risk of switching to the as8 frontends is considered low-risk, as this consists just of serving static (HTML, Javascript) content. This is very lightweight (also on k8s) and the risk of a roll-back getting required is considered quite low. But it already offers the new UI experience to the users. Thus, the big user-visible change happens on the switch to the hybrid transition deployment, while the expected roll-back risk is very low.
  • At a later point in time, the middlewares are switched to as8 on k8s. This the higher-risk step. If it is getting required to perform a roll-back to the as7 middlewares, this roll-back will not have user-visible consequences. So, from the user experience point of view, this does not look like a "failed as8 transition", but just as a maintenance with a (longer) downtime, and users have the as8 user experience before and after (thanks to the hybrid transition deployment).

The hybrid transition deployment can be implemented technically in many different ways, and the official supported way is to implment the as7/as8 split in HTTP routing in the as7 apache mod_proxy "HTTP routers", where we provide already configuration for splitting API from non-API traffic. These routers can easily be adjusted to not only forward API requests to (as7) middlewarses (as before), but also to forward non-API requests (thus, frontend requests) to the as8 service running on k8s.

On the as8 service on k8s, no changes are required at all.

It is also possible to put the as7/as8 split in front of the as7/as8 services, if a suitable load balancer is located there. This way, existing as7 and as8 systems (potentially both live at the same time, as depicted above for the "Co-existence scenario") can be extended by an additional "hybrid transition deployment" entry point.

Terminology: The "opposite" of a "hybrid transition deployment" (the genuine as8 deployment with "frontend" and "middlewares" running on k8s) is called "full-stack deployment".

Technical description

Starting point is a apache mod_proxy_http.conf file which defines ProxyPass for all API call paths and implicitly servers frontend requests locally from static files in the DocumentRoot. Cf as7 installation guidesopen in new window.

To convert such a frontend node configuration into a loadbalancer for splitting traffic for a hybrid transition deployment, insert ProxyPass definitions below the API path ProxyPass definitions for the as8 frontend.

In the simplest case this is just one ProxyPass definition which forwards everything to the as8/k8s endpoints (depending on your setup, either the LoadBalancer, or NodePort endpoints, etc).

A more complete setup will also include websockets support as required by the application.

A reference configuration is given below in the "detailed configuration information" section of this document.

Gradual context switch

Already ealier we introduced the concept of "segmented updates" aka Upgrading without downtimesopen in new window.

This approach can also be leveraged for the as8 transition and effectively allows to gradually increase the load on the as8/k8s system (with the granularity of entire database schemas, so usually a few thousands contexts at once), pausing the transition process if required, reducing overall transition "intensity" (like, user helpdesk calls per day, etc).

This process comes with certain requirements on the infrastructure / networking implementation; in particular, it must be required to "redirect" users from one platform to the other. Most commonly different DNS names for the (as7 and as8) platforms are employed, but other approaches (conditional redirects on "smart gateways") can be implemented as well. A SSO setup is beneficial as it avoids the undesired side effects of users required to re-login after being redirected to "their" side (as7 or as8).

Technical description

First, read and understand the article linked before: Upgrading without downtimesopen in new window.

Then, configuration summary:

  • server.properties: com.openexchange.server.migrationRedirectURL

Process summary:

  • registerserver for a new server ID
  • For each context:
    /opt/open-xchange/sbin/disableschema -m oxdb_7 -A oxadminmaster -P ...
/opt/open-xchange/sbin/changeserver -m oxdb_7 -s 1 -A oxadminmaster -P ...
/opt/open-xchange/sbin/enableschema -m oxdb_7 -A oxadminmaster -P ...

Notes:

  • The CLT calls should be substituted by SOAP API calls
  • Somewhere UpdateTasks need to be plugged (preferrably just for all schemas before starting gradual user switch, for simplicity and robustness)

Combined approach

In principle it should be possible to combine the "gradual context switch" scenario with the "hybrid transition deployment".

It should be possible to do a "gradual context switch" process from legacy to hybrid (in order to keep helpdesk intensity low), and/or to have a "gradual context switch" from hybrid to full-stack. This might be overkill, but customers are free to pick their preferred approach find the right balance between circumspection and simplicity of the process.

Detailed configuration information

Configuration migration instructions

Consult the appropriate core product documentation. Until it is available, we have a preview here:

Have a look at the provided generated values.yaml file as examples.

Configuration consistency requirements

As mentioned above, in particular settings for infrastructure access, authentication flow, user data model, third party systems integration, static secret keys, etc are required to be mapped exactly into the as8 configuration.

We don't have an authorative complete list of settings to be matched. It will depend in general on your installation, feature scope, and customizations.

We gather a list of such known settings here.

External proxy

An apache virtual host definition for TLS termination and websockets support looks as follow.

<VirtualHost *:443>
    ServerAdmin webmaster@example.tld

    ServerName as8.example.tld

    SSLEngine on
    SSLCertificateFile /path/to/as8.cert.pem
    SSLCertificateKeyFile /path/to/as8.key.pem

    ProxyRequests Off
    ProxyStatus On
    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "https"

    <Proxy balancer://as8ws>
        Require all granted
        BalancerMember ws://kubelet1:30080 timeout=1900 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On
        BalancerMember ws://kubelet2:30080 timeout=1900 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On
        BalancerMember ws://kubelet3:30080 timeout=1900 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On
        SetEnv proxy-initial-not-pooled
        SetEnv proxy-sendchunked
    </Proxy>
    ProxyPass /appsuite/rt2 balancer://as8ws/appsuite/rt2
    ProxyPass /socket.io balancer://as8ws/socket.io

    <Proxy balancer://as8>
        Require all granted
        BalancerMember http://kubelet1:30080 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50
        BalancerMember http://kubelet2:30080 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50
        BalancerMember http://kubelet3:30080 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50
        SetEnv proxy-initial-not-pooled
        SetEnv proxy-sendchunked
    </Proxy>
    ProxyPass /appsuite/ balancer://as8/appsuite/
    # more endpoints required depending on feature scope
    # note: be sure to protect sensitive endpoints like /webservices if you expose them here 
</VirtualHost>

Appendix - fragments

Prevent automatic update task execution

The usual caveats of earlier upgrades apply: it must not happen that users are routed uncontrolled on the as8 platform. Otherwise (depending on the configuration setting com.openexchange.groupware.update.denyImplicitUpdateOnContextLoadopen in new window) it either happens that every user accessing the as8 service will either only receive an error message trying to login, or each access will trigger the corresponding schema's update tasks to be executed, which can result in a denial of service attack on your database servers.