Introduction

A client that is open, e.g. a mobile app in the foreground, can receive the changes to a user's mail as they happen: new, changed and deleted mail and folder changes. The middleware sends them as a stream of Server-Sent Events (SSE) over a single long-running HTTP request.

Live change events complement native push:

The stream is delivered by the push notification transport sse (package open-xchange-pns-impl). It carries the topics described in Push Notification Service: ox:mail:new, ox:mail:changed, ox:mail:deleted and ox:mail:folder.

Configuration

The transport is disabled by default. All properties are reloadable; all but maxConnectionsPerNode are config-cascade aware.

Streams on different nodes are served through Redis: each open stream is registered in a Redis hash per user, and notifications for streams on other nodes are forwarded via the cluster's pub/sub channel.

The endpoint

GET <dispatcher prefix>events?topics=ox:mail:*&folders=default0/INBOX
Accept: text/event-stream

The dispatcher prefix is usually /appsuite/api/.

The Mobile API serves the same streams at /mobile/v1/events, in its own format: no hello or close event but a bare retry line, 404 instead of 403 where the transport is disabled, and its own per-user limit; see Mobile API. Both endpoints answer HEAD with 405.

Authentication

• An App Suite session: the session parameter together with the session's secret cookie, as for any other HTTP API request.
• A bearer token in the Authorization header: a JWT of the configured authorization server or a personal access token. The OAuth provider has to be enabled.

A token or restricted session needs scope read_mail for ox:mail:* topics and read_calendar for ox:calendar:* topics.

Errors

Errors before the stream starts are answered as application/problem+json in the format of the Mobile API: type, title, status, a machine-readable code, detail and retryable. type is https://documentation.open-xchange.com/mobile/v1/problems/<code>.

Only 503 is worth a retry without a change. The other codes last until the client or the user acts, so a client does not reconnect on them as it does after a dropped stream.

Events

event: hello
data: {"token":"7c9a809936454a53918a803654929964","pingInterval":30000}

event: state
id: 1789590000123-42
data: {"type":"StateChange","changed":{"default0/INBOX":null},"topics":["ox:mail:changed"]}

event: ping
data: {}

event: resync
id: 1789590000123-43
data: {}

event: close
data: {"reason":"lifetime"}

Client behavior

1. Open the stream and remember the token of the hello event.
2. Pass that token as pushToken parameter with changing mail and folders requests. The stream then gets no event about the client's own changes.
3. On a state event, synchronize the listed folders. On resync, check all folders.
4. On a lost connection, reconnect with the Last-Event-ID header set to the last received id. There is no replay of missed events: the stream answers with hello and resync.

An example

In a browser, EventSource reconnects and sends Last-Event-ID by itself. It cannot set request headers, so it authenticates with the session and its cookie:

function connect(session) {
    const events = new EventSource(`/appsuite/api/events?session=${session}&topics=ox:mail:*`);
    let pushToken = null;

    events.addEventListener('hello', e => { pushToken = JSON.parse(e.data).token; });
    events.addEventListener('state', e => {
        Object.keys(JSON.parse(e.data).changed).forEach(folder => refreshFolder(folder));
    });
    events.addEventListener('resync', () => refreshAllFolders());
    events.addEventListener('close', e => {
        events.close();
        const reason = JSON.parse(e.data).reason;
        if (reason === 'lifetime') {
            connect(session);
        } else if (reason === 'shutdown') {
            setTimeout(() => connect(session), JSON.parse(e.data).retryAfter);
        }
    });
    return () => pushToken;
}

Two things are easy to get wrong. The events are named, so onmessage never fires; every type needs its own listener. And the close event is data, not the end of the connection: the client closes the EventSource itself and reconnects only for the reasons above, since EventSource would otherwise reconnect into a stream that was replaced or a session that ended.

The token of the hello event goes with every changing request, so that the stream stays quiet about this client's own work:

fetch(`/appsuite/api/mail?action=update&id=${id}&folder=${folder}&session=${session}&pushToken=${pushToken()}`, {
    method: 'PUT',
    body: JSON.stringify({ flags: 32, value: true })
});

Outside a browser none of that is automatic. A client with a bearer token sets Authorization itself, remembers the id of the last event and sends it as Last-Event-ID when it reconnects, and treats silence for twice the announced pingInterval as a lost connection:

GET /appsuite/api/events?topics=ox:mail:*&folders=default0/INBOX HTTP/1.1
Accept: text/event-stream
Authorization: Bearer <token>
Last-Event-ID: 1789590000123-42

Event sources

Only the primary mail account is covered.

On a permission change, the middleware also sends ox:mail:folder to the users of the same context who gained or lost access; it knows them at that point. The same goes for the users who can see a folder of somebody else when it is renamed, moved or deleted. A folder its owner moves to the trash keeps its permissions where the mail server moves them along, as Dovecot does, so for them it moved as well. A folder deleted by somebody else goes to that user's trash without the permissions, and a folder deleted for good is gone; for the others both are gone. The notification names the folder as those users see it, e.g. default0/shared/jane.doe/Projects in the namespace of shared folders, and a renamed or moved folder also by the name it had before; that old identifier is gone, and synchronizing it fails. Where the mail system cannot tell that name, and for the owner when somebody else changed the folder, it names no folder, and a live change stream answers it with resync. Users who see only a subfolder of a renamed or moved folder are not told, nor are the other users when somebody who may not read the permissions changed the folder; with Dovecot, those users may then also see the renamed folder without permissions until its permissions change again (DOP-3924). A folder seen by very many users, e.g. one shared with everybody, is reported to none of them but its owner as above; they notice the change with their next synchronization.

What a poll reports is the folder, not the message: ox:mail:changed and ox:mail:deleted carry no ids then, and a renamed folder looks like one folder gone and another created, so renamedFrom is missing. The client checks the folder either way. Folders whose message counts stay the same, e.g. after a flag was set, are only noticed when the server supports CONDSTORE. One node of the cluster watches a user; com.openexchange.imap.liveChanges.maxWatches bounds how many users a node watches. Only the user's own folders are watched: what the server lists under the namespaces of other users and under the public one is left out, because a large shared tree would otherwise multiply the work per user by its size. A node looks at a few mailboxes at a time, so that one slow mail server delays neither the other users nor the pings of the streams.

A change made through the middleware is reported by the middleware right away, with the pushToken of the acting client, and the mail server reports the same change a moment later without it. The second report is left out for com.openexchange.mail.liveChanges.echoWindow (15 seconds by default) after the first one, per user, folder and topic. Only a change of the middleware's own starts that window, so two changes another mail client makes in a row are both reported. The window is kept cluster-wide, because the node that makes a change and the node that watches the mailbox are not the same one, and it works up to five minutes. Setting it to 0 switches the suppression off and lets both reports through.

A mail server that reports its changes itself takes precedence: while notifications for a user arrive at /preliminary/http-notify/v1/notify, the middleware neither polls nor holds a connection for that user. Besides messageNew the endpoint accepts messageAppend, messageRead, messageTrash, flagsSet, flagsClear, messageExpunge, mailboxCreate, mailboxDelete, mailboxRename, mailboxSubscribe and mailboxUnsubscribe, with the fields user, folder, optionally imap-uid or imap-uids, and oldFolder for a rename.

With a JMAP mail server, a change made through the middleware is reported twice: once by the middleware and once by the JMAP server. The second event reaches the client that made the change as well. The JMAP server does not tell where deleted or moved-away messages were; ox:mail:deleted is then sent for the folders whose message count dropped.

Operation

Proxies and load balancers

• Idle and read timeouts on the way to the middleware have to be longer than the ping interval, which is at most 50 seconds. The App Suite chart sets no route timeout for the HTTP API with Istio; the Envoy stream idle timeout (5 minutes by default), the nginx proxy-read-timeout (60 seconds by default), an AWS Application Load Balancer's idle timeout (60 seconds by default) and the middleware's own com.openexchange.http.jetty.readTimeoutMillis (60 seconds by default) are reset by every ping.
• A timeout on the whole response, rather than on silence, has to be longer than maxLifetime, 30 minutes by default. The backend service timeout of a Google Cloud Application Load Balancer is such a timeout, 30 seconds by default; set it to at least 1800 seconds, or the streams end every 30 seconds.
• The stream has to pass without buffering. The middleware sends X-Accel-Buffering: no, which nginx honors. A proxy that compresses responses has to leave text/event-stream out, or flush after every event.
• No stickiness is needed; any node can serve a stream.
• Setting com.openexchange.pns.transport.sse.enabled to false and reloading closes the streams that are open, within a minute; a client that reconnects is then answered with 403.
• An open stream starts the mail push listener of its session, and that listener keeps running until the session ends: a push manager holds one listener per user, not per stream, so ending it with the stream would take the mail push of the user's other sessions away.
• Browsers open at most six HTTP/1.1 connections per host, and an open stream holds one of them. Serve the API via HTTP/2.
• A node that shuts down closes its streams right away with close and reason shutdown, so the termination grace period of a pod needs no time for them. The clients come back within 1 to 10 seconds, spread by retryAfter, on the remaining nodes.
• A node that goes away without closing its streams, e.g. one that is killed or whose machine fails, is noticed by the others within 15 seconds: every node announces itself in Redis every 5 seconds. Its streams then no longer count against maxConnectionsPerUser, so its clients, which reconnect to the other nodes, do not replace the user's healthy streams there. Since a node may have gone a moment before, the limit is enforced against streams of other nodes only after that time. During a rolling update, the streams of nodes of an older version count until their presence expires, up to 70 seconds after the node went away.
• A node that holds more than 125% of the average number of streams of the nodes alive, e.g. after it took over the clients of a node that went away while that node's replacement is still empty, moves streams to the others: it closes randomly chosen streams that are open for at least a minute with reason lifetime, at most 10% of its streams per minute, until it holds no more than 110% of the average. Their clients reconnect right away to whatever node the load balancer picks. Nothing moves while only one node is alive or no other node is below the average and has room; nodes of an older version, during a rolling update, are not counted. Otherwise the streams end with their lifetime, so the nodes are balanced again within maxLifetime, 30 minutes by default. The log tells when a node starts and stops moving streams.
• A load balancer with session affinity, e.g. the gateway chart's sessionAffinity with its cookie, sends a moved client back to the same node, so moving does not help there. A node notices when more than 75% of the moved clients come back within 45 seconds, telling them apart by user and endpoint, or device for the Mobile API, and pauses moving for 10 minutes; each further pause doubles, up to an hour, and a stream that came back is not moved again. With affinity by a cookie hash, e.g. Istio's consistentHash, only the clients whose hash points to another node move; the others count as come back. The streams then stay where the balancer puts the sessions, and new sessions reach the new node.
• Every 5 seconds a node writes the pings that are due and ends streams whose lifetime is over; neither waits for Redis, so a stream is pinged within pingInterval plus 5 seconds however slow Redis is. A stream whose client vanished ends when a ping cannot be written; the HTTP engine may still take the first one after a reset, so this takes at most two pingIntervals plus 5 seconds. The presence of the streams in Redis is refreshed in the background, in one pipelined round trip for up to 500 users. Checks that may call other services run in three independent background lanes, each with at most 16 calls at a time, so that a slow service delays neither the pings nor the other lanes: each session with streams on the node is checked once every 30 seconds, bearer tokens are validated again per tokenCheckInterval, and the mailbox watch of each session is renewed every minute.
• A session that ends, whether by logout, expiry or removal by the administrator, closes its streams right away on the node where it ends, and on the other nodes once the cluster's pub/sub notice of the removal arrives; such notices are buffered for com.openexchange.pubsub.redis.delayMillis (5 seconds by default) and sent at most com.openexchange.pubsub.redis.checkFrequenceMillis (3 seconds) later, so this takes about 5 to 8 seconds. The check catches what such a notice misses, e.g. a lost pub/sub message or a removal on a node of an older version during a rolling update; then the streams are closed within 40 seconds plus at most two durations of a session round, see appsuite_pns_sse_checks_seconds{kind="session"}; with 4,000 sessions on a node, a round took less than half a second at 5 ms and 1.5 seconds at 20 ms per Redis round trip. A round waits at most 30 seconds for its calls; what it did not get to is checked on the next pass. If something else on the node looked the session up shortly before it ended, up to com.openexchange.sessiond.redis.checkExistenceThreshold (15 seconds by default) may be added, since that look-up is trusted for so long.
• While Redis cannot be reached, streams stay open and keep being pinged; each pass then checks one session, and the others are checked as soon as Redis is back. Likewise, while bearer tokens cannot be validated, e.g. since the identity provider is down, each pass validates one token, which then waits for its next tokenCheckInterval, and the others are validated once it answers again. The streams log such failures as one warning per minute; the session service logs each failed look-up on its own.

Metrics

Sizing

An open stream holds no thread; writes are non-blocking. Measured on one node with a 2 GB heap and 5,000 idle streams of five users: