For different use cases, App Suite middleware uses the Redis in-memory database server. This article describes the use cases and requirements, as well as possible configuration options and modes, focusing on the middleware component. For details regarding deployment and operation, please refer to the Deployment Guide.

Requirements

The middleware connects via the RESP protocol to a configured Redis endpoint. Either Standalone, Cluster or Sentinel setups are supported, see Operation Modes below for further details.

Redis

As caching relies on individual TTLs on hash fields (HEXPIRE), the minimum supported Redis version is 7.4.

Persistence is not mandatory and typically not used, instead, high availability of e.g. session data is achieved through redundant deployment modes (Redis Sentinel or Redis Cluster). However, depending on the number of served clients and usage patterns, a decent amount of memory needs to be assigned to the Redis pods so that no data is discarded unintentionally.

Also without persistent storage, the main Redis instance should be deployed using a StatefulSet in Kubernetes.

Since Redis is required both for volatile cache data and other things like client sessions that need to be reliably available, it is also possible (and recommended) to configure a separate Redis instance just for caching purposes, for which different settings can be used.

Shared Access

It is possible that the same Redis service is shared with other services beyond App Suite middleware. All keys follow a fixed naming scheme beginning with a common prefix like ox-cache-, ox-map- or ox-lock-. However, for separation purposes, it is still highly recommended to provide a distinguished Redis instance, especially to not waste the resources required for Redis operation in Sentinel mode for services that don't need this level of redundancy.

Also, in Active/Active deployments with datacenters at multiple sites, there can be situations where the whole Redis database is flushed, which might lead to unexpected consequences for other services.

Configuration

To ease deployment of the Redis pods, a common App Suite stack chart for Redis is available. See the included README article for details.

Middleware configuration is mostly performed through properties with prefix com.openexchange.redis.. The redis section of the Core Middleware Chart only provides the most essential values, which is the operation mode and endpoint configuration. The following chapters describe certain aspects explicitly. Please see the property documentation for the full list of available options.

Operation Modes

Redis can be used by the middleware in three different operation modes. Because user sessions and other important data are also held in Redis, the Sentinel operation mode is recommended to ensure high availability and fault tolerance. In contrast, if a dedicated Redis instance is used for caches, this can safely be configured in Standalone mode.

• Standalone
  Connects to a single Redis instance.
• Cluster
  Connects to Redis Cluster with multiple master/slave nodes. Topology is refreshed every 10 minutes.
• Sentinel
  Connects to Redis Sentinel, optionally using preferred read from replica.

Compression

App Suite middleware stores most data in Redis as JSON strings. In order to reduce their size, it is possible to configure a compression strategy globally for values stored by the middleware. This can be done by configuring a compression type, as well as a threshold after which values are being compressed prior storing them in Redis. Example:

com.openexchange.redis.compressionType = deflate
com.openexchange.redis.minimumCompressionSize = 256

Connection Pool

The middleware connects via TCP to Redis, managing the resources in a local pool of connections. Depending on the usage scenarios, the connection pool might need to be adjusted for e.g. even more concurrently used connections. Unless overridden, the connection pool is configured with these sane defaults (see the property documentation for more details):

com.openexchange.redis.connection.pool.maxTotal = 100
com.openexchange.redis.connection.pool.maxIdle = 100
com.openexchange.redis.connection.pool.minIdle = 0
com.openexchange.redis.connection.pool.maxWaitSeconds = 2
com.openexchange.redis.connection.pool.minIdleSeconds = 60
com.openexchange.redis.connection.pool.cleanerRunSeconds = 10

Start-up

By default the Redis connector waits for the Redis end-point while its bundle starts, retrying with a growing but capped interval. A middleware pod therefore neither becomes ready nor terminates cleanly while Redis is unavailable - but the consumer bundles are guaranteed a connected connector when they start.

Setting com.openexchange.redis.awaitEndPointOnStartup = false moves the wait off the start-up path: the bundle start returns immediately while the end-point is awaited in the background, so the pod starts and can be terminated cleanly. The connector service is registered only once the end-point answered, and the consumer bundles - com.openexchange.sessiond.redis above all - are held back by the service registry until then. In either mode a consumer never starts against an unreachable Redis.

While the background wait is running the node is up but not yet serviceable: consumers that hard-require Redis have not started, so a login attempt in that window fails. Watch the health check and the metrics below to tell "started" from "started and connected" apart. A condition that cannot resolve without a configuration change - rejected credentials above all - ends the wait instead of being retried; the node then stays without Redis functionality until it is restarted with a corrected configuration.

Circuit Breaker

Optionally, a circuit breaker can be enabled on the middleware pods, which blocks any access to Redis for a certain time after an error situation was repeatedly identified. The circuit breaker can be enabled and configured through the properties with prefix com.openexchange.redis.breaker.

Monitoring

Besides monitoring data for Redis itself (e.g. through the Prometheus exporter available in App Suite Redis stack chart), middleware core also exposes some valuable data at the /metrics endpoint.

• Connection Pool
  See metrics beginning with appsuite_redis_connections_
• Commands
  See metrics beginning with lettuce_command_
• Resilience
  See metrics beginning with appsuite_redis_breaker_ and appsuite_redis_bulkhead_. appsuite_redis_breaker_state reports the current state of a circuit breaker as 0 (closed), 1 (half-open), 2 (open) or -1 (disabled). The breaker tag tells the two breakers apart: common guards against failing operations, connect against an unreachable end-point. appsuite_redis_breaker_rejected_total and appsuite_redis_bulkhead_rejected_total count the operations that were shed by the respective policy. Both are counters, so Prometheus exposes them with the _total suffix - an alert rule written without it matches no series and never fires.

Note that in the default shared connection pool mode borrowing a connection never blocks - the connections are multiplexed. Consequently appsuite_redis_connections_num_waiters and the borrow wait times are always zero there, and appsuite_redis_connections_num_active is bounded by the number of shared connections rather than by the number of concurrent operations. The effective concurrency limiter is the bulkhead. Use appsuite_redis_connections_num_multiplexed as the leading indicator - it grows with load and is to be read against com.openexchange.redis.bulkhead.maxConcurrency - and appsuite_redis_bulkhead_rejected_total as the trailing one, which only moves once operations are already being shed.

Besides these general/low-level metrics, there is also use-case-specific monitoring data available - see following sections for details.

Use Cases

As mentioned before, Redis is used for different features of the middleware.

Caching

Various data is held in caches for quick access, and to reduce the load from the persistent storages. In contrast to previous generations of App Suite, where each middleware node used an own caching layer based on JCS and an invalidation channel on top of Hazelcast, the main caches now directly reside within Redis, and are equally available to all middleware pods at any time.

Configuration

Besides configuring client-side semantics of how the middleware caches are used, the overall Redis setup also needs to be considered, especially in larger deployments.

Redis Setup

First, it is possible (and recommended) to configure a separate Redis instance for volatile cache data by using "cache" as infix of the property base names, e.g. com.openexchange.redis.cache.mode. Awareness for this special Redis instance for cache data is enabled through com.openexchange.redis.cache.enabled.

If a separate Redis instance is used for caches, it is typically sufficient to use it in Standalone mode, as data held there can always be reconstructed from the persistent storage layer. Due to the same reason, eviction of entries is not problematic once a defined memory limitation is reached, so that an arbitrary memory management configuration can be set for Redis.

For example, to start a Redis instance without persistence/snapshotting, a maximum memory of 2GB with the policy to evict any key using approximated LRU:

maxmemory 2GB
maxmemory-policy allkeys-lru
appendonly no
save ""

Please see the self-documented Redis configuration file for further details.

Thread-local Cache

By default, the middleware remembers data that has been fetched from Redis before while a client request is handled. This happens through a so-called thread-local cache used by a worker thread. Doing so, essential information like context or user data only needs to be retrieved once from Redis per client request.

com.openexchange.cache.v2.redis.threadLocal = true

In-memory Cache

Besides the thread-local cache, there's also another additional caching layer in the middleware where data fetched from Redis is held for a short amount of time, beyond the scope of the actual client request thread. This caching facade effectively buffers data between consecutive requests of a client, i.e. during a typical refresh of the App Suite UI where various data is retrieved across a series of API requests.

When using the in-memory cache, there's theoretically the risk of reading stale data, whenever the same data is accessed through different middleware nodes, within the specified TTL of cached entries. This is normally no problem for clients whose requests can be routed to the same node reliably (session stickiness or -affinity), as a consecutive request to the same node directly sees any changes of previous requests. However, client requests that cannot be routed, are prone for reading stale data, within the configured TTL. This most prominently affects WebDAV-based clients (including CalDAV and CardDAV) that don't accept affinity cookies. Therefore, one consideration can be to activate the in-memory cache only for nodes serving HTTP API requests, e.g. based on the sync container role.

The in-memory caching layer is enabled by default. It takes the Redis round-trip out of the request hot path; entries are dropped on cache events and on user/context invalidation and otherwise expire after the configured time-to-live. The following properties configure (or disable) it:

com.openexchange.cache.v2.redis.inmemory.enabled = true
com.openexchange.cache.v2.redis.inmemory.mode = volatile
com.openexchange.cache.v2.redis.inmemory.timeToLiveMillis = 5000
com.openexchange.cache.v2.redis.inmemory.remoteInvalidation = true

With remoteInvalidation enabled (the default), a node that invalidates or replaces an entry tells the other nodes to drop their in-memory replica right away - also when the in-memory cache is only enabled dynamically under load - through a dedicated, listener-free channel for regions that do not fire cache events. The time-to-live then only bounds the window in which a notification may be missed (e.g. during a re-subscribe after a connection loss).

Of the properties above, only com.openexchange.cache.v2.redis.inmemory.enabled is re-read on a configuration reload, so a node can be taken out of the in-memory layer without a restart. Switching it off takes effect immediately and drops that node's replicas; switching it back on only works on a node that started with the layer enabled. A node that started without it logs the attempt and keeps the layer off until it is restarted, because the operation mode, the time-to-live and the shape of the cache views are established at start-up.

Note that a node with the layer switched off still tells the other nodes about its own invalidations - otherwise the nodes that do have an in-memory cache would serve stale replicas until the time-to-live elapses. A node that should not take part in the in-memory layer at all, not even dynamically under load, additionally needs com.openexchange.cache.v2.redis.inMemoryCacheEnableThreshold = 0; such a node holds no in-memory cache and creates no remote-invalidation channel.

See the property documentation for further details.

Please note that when the in-memory cache is active, the thread-local caching layer becomes obsolete and is therefore deactivated implicitly, independently of setting com.openexchange.cache.v2.redis.threadLocal.

Sharding

To scale out the caching instance horizontally, operation mode Cluster may also be used, where the keyspace of the stored data is distributed evenly among the available Redis nodes. Technically, every master node in the cluster is responsible for certain hash slots. These slots are the underlying unit of sharding and calculated dynamically from the targeted key(s). Based on that, a command is routed based on its key's hash slot to the corresponding Redis node. If more than one node is targeted by a single command (e.g. DEL orMGET), single commands are executed in a fork/join manner against the correct Redis nodes automatically.

For high availability in case of (partial) downtimes of the cluster nodes, the master-replica model of Redis can be used. See the Redis Cluster documentation for further details.

Connecting to a Redis Cluster requires the configuration of one or multiple Redis endpoints. The initial cluster topology is retrieved during startup, and is refreshed dynamically afterwards through adaptive triggers based on events during Redis cluster operations. Additionally, a static refresh periodic refresh interval can be configured. Example cluster configuration for the caching purposes:

com.openexchange.redis.cache.mode = cluster
com.openexchange.redis.cache.hosts = redis-1:6301,redis-2:6302,redis-3:6303
com.openexchange.redis.cache.cluster.periodicTopologyRefreshMillis = 60000

As mentioned avove, when connecting to Redis cluster, a hosts configuration property takes one or more endpoints that serve as node(s) where the initial connection attempt is made against, the actual cluster topology is then retrieved through the RESP protocol. Both master and replica nodes can be used. The underlying Lettuce library calls them seed nodes:

Connecting to a Redis Cluster requires one or more initial seed nodes. The full cluster topology view (partitions) is obtained on the first connection so you’re not required to specify all cluster nodes. Specifying multiple seed nodes helps to improve resiliency as Lettuce is able to connect the cluster even if a seed node is not available.

In Kubernetes-based Redis deployments, only the (headless) ClusterIP service might be available in front of the actual cluster nodes. This ClusterIP service can be used as well as the only seed node. For example, if the following Redis Cluster service was created by the chart:

redis-core-mw-cache-redis-cluster            ClusterIP   10.43.95.203    <none>        6379/TCP                                     3m30s

A corresponding middleware cache configuration (together with a Redis Sentinel configuration for the default Redis instance) would look like:

core-mw:
  redis:
    # core-mw default configuration
    mode: sentinel
    hosts:
    - redis-core-mw-default.redis:26379
    #  core-mw cache configuration
    cache:
      enabled: true
      mode: cluster
      hosts:
      - redis-core-mw-cache-redis-cluster.redis:6379

Semantics

Basically, App Suite middleware uses the cache-aside pattern, where the cache is populated on demand once the data is requested and has been loaded from the persistent storage (i.e. the database). Consecutive read operations will then use the previously cached values. Entries in the cache are invalidated explicitly once the associated data is changed through one of the APIs, so that the next read attempt will read the updated data from the persistent storage and populate the cache again.

All cache data put into Redis uses keys prefixed with ox-cache:. By default, all entries are decorated with a TTL of 3600 seconds (one hour), after which they are evicted automatically from the cache. This default duration can be adjusted through property com.openexchange.cache.v2.defaultExpirationSeconds, however, there are still cases where the implementation determines the expiration time statically.

Besides the cache in Redis, a very thin, thread-local cache is used within the scope of a served client request, remembering data that has already been loaded from Redis in memory to speed up repeated accesses to the same keys. This is enabled by default, but can be controlled via property com.openexchange.cache.v2.redis.threadLocal.

Sizing

Using a distributed caching solution means on the one hand that the memory demand of middleware pods decreases, but on the other hand, since the data now resides in Redis, the memory requirements increases there accordingly. If a dedicated Redis instance for cache data is used (see configuration above), memory should be assigned to this Redis cache pod so that eviction due to the defined maximum memory policy of Redis does not happen too often. However, if no dedicated instance is available for caching, one wants to absolutely avoid this so that no unrecoverable data gets lost.

During tests with synthetic data, we observed a memory consumption of about 10GB for 1M active user sessions:

• This covers data held in caches only (no sessions etc.)
• 10GB as per Redis metric used_memory_human, which is different from the memory allocated to Redis from the OS
• Active means that the client performed at least one request to App Suite middleware within a period of 12 minutes (in default configuration)
• Measured in a simulation with synthetic user account data only; for real accounts (with e.g. lots of folders, attributes etc.), this will look different

Due to the described inaccuracies, this should only be seen as very rough and minimal estimation, and it is therefore still important to keep an eye on the relevant monitoring data, and adjust the memory- and eviction-related settings as needed.

Monitoring

Of course the most important metrics are generated by the Redis pods themselves. See e.g. the App Suite stack chart for Redis for how to enable the Prometheus exporter for Redis.

Besides these server-side monitoring options, also some metrics can be collected from the App Suite middleware's client perspective. Cache metrics can be enabled through the following property:

com.openexchange.cache.v2.redis.withMetrics=true

This yields statistics about cache hit/miss ratio, as well as put/get/remove operation runtimes. The relevant metrics begin with prefix appsuite_redis_cache_.

Similar metrics are also available for the thread-local caching layer, which can be enabled via:

com.openexchange.cache.v2.redis.metrics.threadLocal=true

These metrics are prefixed with appsuite_redis_cache_threadlocal_.

Publish / Subscribe

For inter-node communication and notification, App Suite middleware leverages the Pub/Sub feature of Redis. Use cases include:

• Invalidation Messages
  To inform all nodes about changes that require invalidating locally held data, like cached JSlobs.
• Push Subscriptions
  Broadcast changes to locally registered push subscribers
• Session Events
  Inform other nodes about removed sessions

Session Storage

Besides pure caching, the Redis is most prominently used by the middleware as Session Storage. Therefore, a separate documentation article is available at Redis Session Storage.

Distributed Maps

Besides the data held in ordinary caches, another abstraction layer in App Suite middleware makes use of Redis distributed maps to provide access to certain shared data, so that it is equally accessible for all middleware nodes. Use cases include:

• State Management
  Certain OpenID- or SAML workflows need to persist certain information of typical login-or logout-flows between client and authorization server.
• Push Listener
  Management of registered push listeners.
• OAuth Callbacks
  Certain OAuth flows need to remember a callback URL where the client is redirected to.
• Rate Limiting
  Controls the maximum allowed permits per timeframe for different use cases like SMS transport.
• Token Login
  Store tokens or reserved sessions to support special login flows.

Distributed Locks

For mutual exclusion among middleware nodes, distributed locks for different usage scenarios are modeled based on Redis, using keys prefixed with ox-lock.