Introduction

A client that is open, e.g. a mobile app in the foreground, can receive the changes to a user's mail as they happen: new, changed and deleted mail and folder changes. The middleware sends them as a stream of Server-Sent Events (SSE) over a single long-running HTTP request.

Live change events complement native push:

The stream is delivered by the push notification transport sse (package open-xchange-pns-impl). It carries the topics described in Push Notification Service: ox:mail:new, ox:mail:changed, ox:mail:deleted and ox:mail:folder.

Configuration

The transport is disabled by default. All properties are reloadable; all but maxConnectionsPerNode are config-cascade aware.

Streams on different nodes are served through Redis: each open stream is registered in a Redis hash per user, and notifications for streams on other nodes are forwarded via the cluster's pub/sub channel.

The endpoint

GET <dispatcher prefix>events?topics=ox:mail:*&folders=default0/INBOX
Accept: text/event-stream

The dispatcher prefix is usually /appsuite/api/.

The Mobile API serves the same streams at /mobile/v1/events, in its own format: no hello or close event but a bare retry line, 404 instead of 403 where the transport is disabled, and its own per-user limit; see Mobile API. Both endpoints answer HEAD with 405.

Authentication

• An App Suite session: the session parameter together with the session's secret cookie, as for any other HTTP API request.
• A bearer token in the Authorization header: a JWT of the configured authorization server or a personal access token. The OAuth provider has to be enabled.

A token or restricted session needs scope read_mail for ox:mail:* topics and read_calendar for ox:calendar:* topics.

Errors

Errors before the stream starts are answered as application/problem+json in the format of the Mobile API: type, title, status, a machine-readable code, detail and retryable. type is https://documentation.open-xchange.com/mobile/v1/problems/<code>.

Events

event: hello
data: {"token":"7c9a809936454a53918a803654929964","pingInterval":30000}

event: state
id: 1789590000123-42
data: {"type":"StateChange","changed":{"default0/INBOX":null},"topics":["ox:mail:changed"]}

event: ping
data: {}

event: resync
id: 1789590000123-43
data: {}

event: close
data: {"reason":"lifetime"}

Client behavior

1. Open the stream and remember the token of the hello event.
2. Pass that token as pushToken parameter with changing mail and folders requests. The stream then gets no event about the client's own changes.
3. On a state event, synchronize the listed folders. On resync, check all folders.
4. On a lost connection, reconnect with the Last-Event-ID header set to the last received id. There is no replay of missed events: the stream answers with hello and resync.

An example

In a browser, EventSource reconnects and sends Last-Event-ID by itself. It cannot set request headers, so it authenticates with the session and its cookie:

function connect(session) {
    const events = new EventSource(`/appsuite/api/events?session=${session}&topics=ox:mail:*`);
    let pushToken = null;

    events.addEventListener('hello', e => { pushToken = JSON.parse(e.data).token; });
    events.addEventListener('state', e => {
        Object.keys(JSON.parse(e.data).changed).forEach(folder => refreshFolder(folder));
    });
    events.addEventListener('resync', () => refreshAllFolders());
    events.addEventListener('close', e => {
        events.close();
        const reason = JSON.parse(e.data).reason;
        if (reason === 'lifetime') {
            connect(session);
        } else if (reason === 'shutdown') {
            setTimeout(() => connect(session), JSON.parse(e.data).retryAfter);
        }
    });
    return () => pushToken;
}

Two things are easy to get wrong. The events are named, so onmessage never fires; every type needs its own listener. And the close event is data, not the end of the connection: the client closes the EventSource itself and reconnects only for the reasons above, since EventSource would otherwise reconnect into a stream that was replaced or a session that ended.

The token of the hello event goes with every changing request, so that the stream stays quiet about this client's own work:

fetch(`/appsuite/api/mail?action=update&id=${id}&folder=${folder}&session=${session}&pushToken=${pushToken()}`, {
    method: 'PUT',
    body: JSON.stringify({ flags: 32, value: true })
});

Outside a browser none of that is automatic. A client with a bearer token sets Authorization itself, remembers the id of the last event and sends it as Last-Event-ID when it reconnects, and treats silence for twice the announced pingInterval as a lost connection:

GET /appsuite/api/events?topics=ox:mail:*&folders=default0/INBOX HTTP/1.1
Accept: text/event-stream
Authorization: Bearer <token>
Last-Event-ID: 1789590000123-42

Event sources

Only the primary mail account is covered.

On a permission change, the middleware also sends ox:mail:folder to the users of the same context who gained or lost access; it knows them at that point. The same goes for the users who can see a folder of somebody else when it is renamed, moved or deleted. A folder its owner moves to the trash keeps its permissions where the mail server moves them along, as Dovecot does, so for them it moved as well. A folder deleted by somebody else goes to that user's trash without the permissions, and a folder deleted for good is gone; for the others both are gone. The notification names the folder as those users see it, e.g. default0/shared/jane.doe/Projects in the namespace of shared folders, and a renamed or moved folder also by the name it had before; that old identifier is gone, and synchronizing it fails. Where the mail system cannot tell that name, and for the owner when somebody else changed the folder, it names no folder, and a live change stream answers it with resync. Users who see only a subfolder of a renamed or moved folder are not told, nor are the other users when somebody who may not read the permissions changed the folder; with Dovecot, those users may then also see the renamed folder without permissions until its permissions change again (DOP-3924). A folder seen by very many users, e.g. one shared with everybody, is reported to none of them but its owner as above; they notice the change with their next synchronization.

What a poll reports is the folder, not the message: ox:mail:changed and ox:mail:deleted carry no ids then, and a renamed folder looks like one folder gone and another created, so renamedFrom is missing. The client checks the folder either way. Folders whose message counts stay the same, e.g. after a flag was set, are only noticed when the server supports CONDSTORE. One node of the cluster watches a user; com.openexchange.imap.liveChanges.maxWatches bounds how many users a node watches. Only the user's own folders are watched: what the server lists under the namespaces of other users and under the public one is left out, because a large shared tree would otherwise multiply the work per user by its size. A node looks at a few mailboxes at a time, so that one slow mail server delays neither the other users nor the pings of the streams.

A change made through the middleware is reported by the middleware right away, with the pushToken of the acting client, and the mail server reports the same change a moment later without it. The second report is left out for com.openexchange.mail.liveChanges.echoWindow (15 seconds by default) after the first one, per user, folder and topic. Only a change of the middleware's own starts that window, so two changes another mail client makes in a row are both reported. The window is kept cluster-wide, because the node that makes a change and the node that watches the mailbox are not the same one, and it works up to five minutes. Setting it to 0 switches the suppression off and lets both reports through.

A mail server that reports its changes itself takes precedence: while notifications for a user arrive at /preliminary/http-notify/v1/notify, the middleware neither polls nor holds a connection for that user. Besides messageNew the endpoint accepts messageAppend, messageRead, messageTrash, flagsSet, flagsClear, messageExpunge, mailboxCreate, mailboxDelete, mailboxRename, mailboxSubscribe and mailboxUnsubscribe, with the fields user, folder, optionally imap-uid or imap-uids, and oldFolder for a rename.

With a JMAP mail server, a change made through the middleware is reported twice: once by the middleware and once by the JMAP server. The second event reaches the client that made the change as well. The JMAP server does not tell where deleted or moved-away messages were; ox:mail:deleted is then sent for the folders whose message count dropped.

Operation

Proxies and load balancers

• Idle and read timeouts on the way to the middleware have to be longer than the ping interval, which is at most 50 seconds. The App Suite chart sets no route timeout for the HTTP API with Istio; the Envoy stream idle timeout (5 minutes by default), the nginx proxy-read-timeout (60 seconds by default), an AWS Application Load Balancer's idle timeout (60 seconds by default) and the middleware's own com.openexchange.http.jetty.readTimeoutMillis (60 seconds by default) are reset by every ping.
• A timeout on the whole response, rather than on silence, has to be longer than maxLifetime, 30 minutes by default. The backend service timeout of a Google Cloud Application Load Balancer is such a timeout, 30 seconds by default; set it to at least 1800 seconds, or the streams end every 30 seconds.
• The stream has to pass without buffering. The middleware sends X-Accel-Buffering: no, which nginx honors. A proxy that compresses responses has to leave text/event-stream out, or flush after every event.
• No stickiness is needed; any node can serve a stream.
• Setting com.openexchange.pns.transport.sse.enabled to false and reloading closes the streams that are open, within a minute; a client that reconnects is then answered with 403.
• An open stream starts the mail push listener of its session, and that listener keeps running until the session ends: a push manager holds one listener per user, not per stream, so ending it with the stream would take the mail push of the user's other sessions away.
• Browsers open at most six HTTP/1.1 connections per host, and an open stream holds one of them. Serve the API via HTTP/2.
• A node that shuts down closes its streams right away with close and reason shutdown, so the termination grace period of a pod needs no time for them. The clients come back within 1 to 10 seconds, spread by retryAfter, on the remaining nodes.
• Streams do not move to a node that was just added until they end, so the nodes are balanced again within maxLifetime, 30 minutes by default.

Metrics

Sizing

An open stream holds no thread; writes are non-blocking. Measured on one node with a 2 GB heap and 5,000 idle streams of five users: