Self-check for on-premise administrators

Which of your users lost appointments?

A defect in App Suite deletes a user's calendar appointments when an internal lookup fails for a moment. Nothing is recorded, so the calendar simply looks emptier than it was. Five steps, in order. The first can end the whole thing in a minute, and most sites will not need to go past the fourth.

1. Did you ever run an affected build?

Only the versions below can delete. Earlier builds hit the same underlying condition but fail the request instead of deleting it — noisy in the logs, harmless to the data.

If no such build was ever deployed — stop here. The code that deletes was never on your systems. Nothing else to check.

Otherwise — write down one date. The day you installed your first build at or above the "deletes from" version. Steps 2 and 3 both need it. Use your date, not any date from an incident report. This document calls it your upgrade date.

2. Upgrade, before investigating anything

Install the fixed version for your line from the table above. This is not conditional on finding a victim first: until the fix is in, the deletion recurs whenever the transient condition happens again, and anything you recover can be deleted a second time. It is also the only step here that changes your exposure rather than your knowledge of it.

3. Search your logs for one phrase

Every deletion writes a log line — success at INFO, failure at ERROR. There is no silent path. So if your logs cover the whole period since your upgrade date and the phrase never appears, nothing was ever deleted. Do this before touching the database: it is faster, and a hit names the affected user outright.

references to stale calendar folder

No hits proves you are safe — but only if all four hold.

1. Your stack actually ships INFO, not only WARN and above.
2. Retention reaches back to your upgrade date, with no gaps.
3. Every middleware node is covered. A deletion is logged only by the pod that did it.
4. The search returns nothing.

If any of the four fails, the result proves nothing in either direction — go to step 4. This is the normal case for stacks that ship WARN and above, or whose retention is shorter than the time since your upgrade.

If there are hits, read them like this

Purged data for 368 events with references to stale calendar folder 31.

The number is how many appointments were deleted. The context and the user are not in the message text — they travel as MDC properties on the same log event, so a single record gives you everything:

In Elasticsearch, OpenSearch, Kibana or Graylog: free-text search the phrase, then group by the context and user fields. Field paths depend on your shipper's mapping — the MDC keys are often nested under a details or mdc object, so inspect one document first. kubectl logs shows only the current container buffer: useful to confirm the problem is live, useless for history.

For log files on disk, where that applies:

zgrep -ho "Purged data for [0-9]* events" \
      /var/log/open-xchange/open-xchange.log* \
  | awk '{ s += $4 } END { print s " appointments deleted" }'

Every hit is a confirmed loss. Take its context id to step 5 for recovery.

4. If the logs cannot reach, ask your users

When the logs do not cover the period, the affected users themselves are the best remaining source — better than anything the database can offer. One line to your staff:

Has anything disappeared from your calendar since 21 July?
Older appointments missing, or a calendar that is emptier
than it was? Reply yes or no - we are checking a known
problem and need to know who is affected.

Why this beats a database sweep. Nothing in the database records that a deletion happened, so no query can confirm a loss on its own. The best a database sweep can do is find calendars whose shape is consistent with a loss and then tell you to ask that user — the same question, arrived at after considerably more work, with false alarms attached. If you can reach your users, ask them.

Every yes is a confirmed case: note the user and go to step 5. Every no genuinely clears that user, which no query can do. Silence clears nobody, so chase non-responders if the calendar matters to them.

If asking everyone is not realistic, large or unattended user populations are the case the optional database triage in the appendix exists for. It narrows thousands of calendars to a handful worth asking about. Read what it can and cannot conclude before you spend the time.

5. Recovering what was deleted

Restoring from a pre-incident backup is the only route that recovers the data, and the only check that also settles the invisible case described in the appendix. You need a dump taken before your upgrade date and newer than twelve weeks, so that deletion records still cover everything since.

The comparison is per appointment number: present in the dump, absent from live, and no deletion record — that is an appointment removed without going through the application. Putting it back is more delicate than it looks. Whole recurring series have to go back together, deletions the user genuinely made must be excluded, and modification timestamps have to be refreshed or synchronised clients will never pick the data up again. Ask support for the recovery runbook rather than improvising a restore.

If you have no suitable dump, there is no server-side way to reconstruct the appointments. Tell the affected users plainly, and make sure the upgrade is in place so it cannot happen again.

Appendix — optional database triage

Read this before deciding to spend the time.

• It cannot confirm a loss and cannot clear anyone. The deletion leaves no record, so a clean database is also what a covered-up deletion looks like.
• Its output is a shortlist of calendars plus one question to put to each user — the same question as step 4, reached the long way round.
• Expect a handful of rows, not a project. Measured on three live hosting schemas of 12,000–14,000 calendars each, it returned 10 to 14 rows per schema, about half of them worth acting on.
• It is worth the effort in one situation: you cannot ask your users — too many of them, or unattended mailboxes — and you need to know where to look first.

One limit to know before you read the output: an emptied calendar and a barely used one look the same. Both are old, and both hold nothing from before your upgrade. The report separates them where it can — rows with corroborating evidence are flagged CHECK, rows without are flagged WEAK and are usually mail-only users — but the split is a judgement, not a proof. The question to the user resolves it.

What it found when tested against known cases

We were able to test this where both the logs and the database were available: twelve users whose calendars the logs prove were emptied, across three schemas of a live platform.

• It found 8 of the 12, and for each of those it named the exact calendar the log line named — eight out of eight on folder identity. When it fires, it is precise about where.
• The four it missed had one shape in common: the calendar had been emptied completely and the user had no ordinary deletion on record, which is indistinguishable from a calendar nobody ever used. We then checked what more the database could have yielded for them, and the answer was nothing. One had lost a single appointment and her context showed no unaccounted appointment numbers at all. Another had lost 402: 380 numbers were missing, and not one could be told apart from an ordinary deletion, because everything around them was years old.
• So there is a floor to what any query can reach, and it sits at the quiet end — light users, few appointments, no recent deletions. Those are the users who will tell you themselves if you ask them, which is why step 4 comes before this appendix.

How it works, in three facts

• Every appointment gets a number, handed out in order and never reused, so the counter records how many a context has ever had.
• Every ordinary deletion — by the user, a phone, a sync client, an admin tool — leaves a record, kept for about twelve weeks.
• This defect leaves no record, and it empties a whole calendar at a time rather than picking appointments. So a calendar that still holds anything entered before your upgrade was never touched.

Two ordinary operations look identical to the defect. Deleting a user or a whole context removes appointments without a record, and so does removing someone's calendar permission and then running the deleteinvisible cleanup. Neither is a reason to skip the report, but check whether provisioning did one of these before blaming the defect for a flagged calendar.

Run one report, per database schema

One query, read-only, nothing is written. Connect to one user database schema, change the first line to your upgrade date from step 1, and run it. Repeat for every user schema: a context lives in exactly one, so a clean schema says nothing about the next. Needs MariaDB 10.6 or newer, and took a few seconds on a schema holding 200,000 appointments — on much larger ones prefer a replica or an off-peak window.

One row per calendar that looks wrong — not one row per user, because a user may have several calendars and only one of them may have been emptied. Read the flag column first, then the finding column, which is written so you can paste it straight into a ticket. The last row is always a SCHEMA TOTAL summary, so an empty report still tells you something.

Why the calendar is the right unit — and where that stops. The defect does not pick individual appointments. It resolves one calendar, concludes that calendar is gone, and empties it completely, looping until nothing is bound to it any more. So for the calendar it hit: nothing from before that moment is left, and conversely a calendar that still holds anything created before your upgrade date was never hit.

It does not follow that every affected calendar looks emptied. When the emptied calendar belonged to the organiser of a meeting, the meeting itself is deleted, so everyone else invited to it loses it from calendars that were never touched at all. Those keep their older appointments and read ok. A flagged organiser therefore usually explains gaps in colleagues' calendars as well — and one restore repairs all of them together.

-- CHANGE THIS: the day you installed your first affected build (step 1).
SET @exposed_from   = UNIX_TIMESTAMP('2026-07-21') * 1000;
-- Leave alone: deletion records are kept about twelve weeks.
SET @recorded_since = UNIX_TIMESTAMP(NOW() - INTERVAL 12 WEEK) * 1000;
WITH numbers AS (          -- every appointment number still accounted for:
  SELECT cid, id, MIN(created) AS created         -- either it still exists,
    FROM ( SELECT cid, id, created FROM calendar_event    -- or its deletion
            WHERE account = 0                            -- is on record
           UNION ALL
           SELECT cid, id, created FROM calendar_event_tombstone
            WHERE account = 0 ) x
   GROUP BY cid, id
), first_recent AS (       -- lowest number created inside the record window;
  SELECT cid, MIN(id) AS id_floor    -- numbers above it are also newer, since
    FROM numbers                     -- numbers are handed out in order
   WHERE created >= @recorded_since
   GROUP BY cid
), lagged AS (             -- put each number next to the one before it
  SELECT cid, id, LAG(id) OVER (PARTITION BY cid ORDER BY id) AS prev_id
    FROM numbers
), holes AS (              -- numbers gone with no deletion record, per context
  SELECT n.cid,
         SUM(n.id - n.prev_id - 1)                       AS gone_total,
         SUM(CASE WHEN n.prev_id >= f.id_floor           -- these are the ones
                  THEN n.id - n.prev_id - 1 ELSE 0 END)  AS gone_recently
    FROM lagged n                                        -- that cannot be
    LEFT JOIN first_recent f ON f.cid = n.cid            -- explained away
   WHERE n.prev_id IS NOT NULL AND n.id - n.prev_id > 1
   GROUP BY n.cid
)
, kept AS (                  -- what is still in each private calendar folder
  SELECT a.cid, a.entity AS user_id, CAST(a.folder AS UNSIGNED) AS folder,
         COUNT(*)       AS appointments,
         MIN(e.created) AS oldest_entered,   -- when the record was created
         MIN(e.start)   AS earliest_start    -- what the user actually sees
    FROM calendar_attendee a
    JOIN calendar_event e ON e.cid = a.cid AND e.account = a.account
                        AND e.id  = a.`event`
   WHERE a.account = 0 AND a.folder IS NOT NULL
   GROUP BY a.cid, a.entity, CAST(a.folder AS UNSIGNED)
), calendars AS (            -- every private calendar folder, and whose it is
  SELECT cid, fuid AS folder, created_from AS user_id,
         creating_date AS since, default_flag
    FROM oxfolder_tree
   WHERE module = 2 AND type = 1        -- 2 = calendar, 1 = private
), recorded AS (             -- removals from that folder that ARE on record
  SELECT cid, entity AS user_id, CAST(folder AS UNSIGNED) AS folder,
         COUNT(DISTINCT `event`) AS removals
    FROM calendar_attendee_tombstone
   WHERE account = 0 AND folder IS NOT NULL
   GROUP BY cid, entity, CAST(folder AS UNSIGNED)
)
SELECT * FROM (
  SELECT CASE
           WHEN k.appointments IS NULL AND COALESCE(r.removals, 0) > 0
                                                     THEN 'EMPTIED'
           WHEN k.appointments IS NULL                THEN 'never used'
           WHEN k.oldest_entered >= @exposed_from
            AND c.since          <  @exposed_from
            AND (COALESCE(r.removals, 0) > 0
              OR COALESCE(h.gone_recently, 0) > 0)    THEN 'CHECK'
           WHEN k.oldest_entered >= @exposed_from
            AND c.since          <  @exposed_from     THEN 'WEAK'
           WHEN c.since          >= @exposed_from     THEN 'UNKNOWN'
           ELSE 'ok' END                              AS flag,
         c.cid, c.user_id, l.uid AS login, u.mail,
         c.folder, c.default_flag                     AS is_main_calendar,
         COALESCE(k.appointments, 0)                  AS appointments_now,
         DATE(FROM_UNIXTIME(k.oldest_entered DIV 1000)) AS oldest_entered,
         DATE(k.earliest_start)                       AS earliest_start,
         DATE(FROM_UNIXTIME(c.since DIV 1000))        AS folder_created,
         COALESCE(r.removals, 0)                      AS removed_on_record,
         COALESCE(h.gone_recently, 0)                 AS provable_loss_in_context,
         COALESCE(h.gone_total, 0)                    AS unaccounted_incl_normal,
         CASE
           WHEN k.appointments IS NULL AND COALESCE(r.removals, 0) > 0 THEN
             CONCAT('LIKELY AFFECTED. This calendar is empty now, but ',
                    r.removals, ' appointment(s) were deleted from it the ',
                    'normal way within the last 12 weeks, so it was in use. ',
                    'A purge empties a calendar completely, which is what an ',
                    'in-use calendar holding nothing looks like. ASK THE USER ',
                    'whether appointments are missing from it.')
           WHEN k.appointments IS NULL THEN
             'NOT JUDGED. Empty, and nothing indicates it was ever used.'
           WHEN k.oldest_entered >= @exposed_from AND c.since < @exposed_from
            AND COALESCE(r.removals, 0) = 0 AND COALESCE(h.gone_recently, 0) = 0
             THEN CONCAT('WEAK - PROBABLY JUST UNUSED. This calendar has ',
                    'existed since ',
                    COALESCE(DATE(FROM_UNIXTIME(c.since DIV 1000)), '?'),
                    ' and holds ', k.appointments, ' appointment(s), none ',
                    'entered before ',
                    COALESCE(DATE(FROM_UNIXTIME(k.oldest_entered DIV 1000)), '?'),
                    '. That is the shape a purge leaves, but nothing here ',
                    'corroborates it: no deletion is on record for this ',
                    'calendar, and nothing in this context is provably lost. ',
                    'A user who hardly uses the calendar - mail only, say - ',
                    'looks exactly the same. Only pursue if the user says ',
                    'appointments are missing.')
           WHEN k.oldest_entered >= @exposed_from AND c.since < @exposed_from
             THEN CONCAT('LIKELY AFFECTED. This calendar has existed since ',
                    COALESCE(DATE(FROM_UNIXTIME(c.since DIV 1000)), '?'),
                    ', but nothing in it was entered before ',
                    COALESCE(DATE(FROM_UNIXTIME(k.oldest_entered DIV 1000)), '?'),
                    ', which is after the upgrade. A purge empties a calendar ',
                    'completely, so anything entered earlier would be gone. ',
                    'ASK THE USER whether they were already putting ',
                    'appointments into this calendar before that date - the ',
                    'day an appointment was entered, not the day it is ',
                    'scheduled for. It was in use: ', COALESCE(r.removals, 0),
                    ' deletion(s) are on record for this calendar, and ',
                    COALESCE(h.gone_recently, 0), ' appointment number(s) in ',
                    'this context are provably not ordinary deletions.')
           WHEN c.since >= @exposed_from THEN
             CONCAT('CANNOT TELL. This calendar was created after the ',
                    'upgrade, so there is no older content to compare ',
                    'against. Being new is not evidence that it is safe. ',
                    'Only the logs or a backup can settle this one.')
           ELSE
             CONCAT('NO SIGN OF LOSS. Still holds appointments entered ',
                    'before the upgrade, oldest ',
                    COALESCE(DATE(FROM_UNIXTIME(k.oldest_entered DIV 1000)), '?'),
                    ', so it was never purged.')
         END                                          AS finding
    FROM calendars c
    JOIN `user` u          ON u.cid = c.cid AND u.id = c.user_id
    LEFT JOIN login2user l ON l.cid = c.cid AND l.id = c.user_id
    LEFT JOIN kept k       ON k.cid = c.cid AND k.user_id = c.user_id
                         AND k.folder = c.folder
    LEFT JOIN recorded r   ON r.cid = c.cid AND r.user_id = c.user_id
                         AND r.folder = c.folder
    LEFT JOIN holes h      ON h.cid = c.cid
) report
 WHERE flag IN ('CHECK', 'EMPTIED')          -- carry their own evidence
    OR flag = 'WEAK'
    OR (flag = 'UNKNOWN' AND unaccounted_incl_normal > 0)
UNION ALL                    -- always-present last row: the whole schema
SELECT 'SCHEMA TOTAL', NULL, NULL, NULL, NULL, NULL, NULL, NULL,
       NULL, NULL, NULL, NULL,
       (SELECT COALESCE(SUM(gone_recently), 0) FROM holes),
       (SELECT COALESCE(SUM(gone_total), 0)    FROM holes),
       CONCAT('Whole schema: ',
              (SELECT COALESCE(SUM(gone_recently), 0) FROM holes),
              ' appointment number(s) are provably not ordinary deletions. ',
              'If that is zero and there are no rows above, nothing here ',
              'looks purged. If it is above zero but there are no rows ',
              'above, something was deleted that this report cannot pin on ',
              'a calendar - a public or shared calendar, or a user since ',
              'deleted - so escalate. The other column, ',
              (SELECT COALESCE(SUM(gone_total), 0) FROM holes),
              ', counts every number now missing including ordinary ',
              'deletions whose records have expired and users or contexts ',
              'that were removed: a large value there is normal and is NOT ',
              'a measure of damage. Both are lower bounds.')
 ORDER BY FIELD(flag, 'CHECK', 'EMPTIED', 'WEAK', 'UNKNOWN', 'SCHEMA TOTAL'),
          provable_loss_in_context DESC, unaccounted_incl_normal DESC,
          cid, user_id, folder;

Example result, shortened:

flag         | cid    | login | folder | main | appts | oldest_entered
CHECK        | 550733 | info  |     31 |    1 |    30 | 2026-07-31
EMPTIED      |  54954 | coh   |     42 |    1 |     0 | NULL
WEAK         |  61002 | ops   |    118 |    1 |     1 | 2026-08-12
UNKNOWN      |  54954 | tom   |    377 |    0 |    12 | 2026-08-02
SCHEMA TOTAL | NULL   | NULL  |   NULL | NULL |  NULL | NULL

  ... earliest_start | folder_created | provable_loss | unaccounted_incl_normal
         2026-08-03  |   2019-04-02   |            28 |                    1311
         NULL        |   2018-11-05   |            28 |                    1311
         2026-08-12  |   2025-09-09   |             0 |                       6
         2026-08-02  |   2026-07-28   |            28 |                    1311
         NULL        |   NULL         |            92 |                    4408

Calendars that look fine are not listed. A calendar still holding anything entered before your upgrade was not purged, so it is left out rather than printed as ok. This report is an exception list: if it comes back empty, no calendar in that schema carries the signature.

Columns to weigh before you accuse the defect:

• removed_on_record — how many appointments were removed from that calendar with a record in the last twelve weeks. A user who deletes a lot themselves explains a lot of gaps innocently. On an EMPTIED row it is the evidence that the calendar was in use.
• provable_loss_in_context — appointments in that context that cannot be ordinary deletions, repeated on every row of the context. Corroboration, not attribution: they may have belonged to any user in it, or to one since deleted. A high figure makes a row urgent; a zero does not make it innocent, because an old calendar's losses fall outside the provable window by definition.
• oldest_entered versus earliest_start — when the oldest surviving appointment was entered, and the date it is for. The verdict uses the first; the second is shown because it is what the user sees. Old start dates with recent entry dates usually mean an import — or a user who has already re-entered what they lost, which is itself a sign.

If a user reports an empty calendar but does not appear in the report, calendars that are empty and show no sign of ever having been used are left out. On the live schemas we measured, 88% of all calendars are empty — listing them would bury everything else. To include them for one context, delete the line WHERE flag <> 'never used' from the report and add AND c.cid = <their cid> to the FROM calendars c block. An old calendar that comes back empty is then worth asking about, even without a record.

Where this report still misses someone. Appointments the user was merely invited to are covered: the invitation was bound to that calendar too, so losing it leaves the calendar empty or recent-only just the same. What the report cannot do is tell you which appointments went — that needs the gap list below, or a backup for the content. Five gaps remain, the first of which is the common one:

• Meetings organised by a user whose calendar was emptied. They are deleted outright, so every other participant loses them from a calendar that was never touched and still reads ok. Look for a flagged organiser in the same context before concluding that colleagues were unaffected.
• A calendar that was emptied and then deleted by the user afterwards leaves no folder row, so nothing is left to report.
• A calendar that held only invitations to other people's appointments, with no ordinary deletion on record in the last twelve weeks, cannot be told from one that was never used, and is left out.
• An old appointment moved into the calendar after the purge makes it look untouched.
• Public and shared calendars are bound differently and are not listed here. Their appointments are counted in the SCHEMA TOTAL row, so a loss there shows up in the total without being attributed to any calendar.

A few appointments are also bound to no calendar at all — resources, external participants, and the occasional entry created by another client. They are not counted against any calendar, which can make one look emptier than it is, so check an unexpected flag before acting on it.

So a schema with no flagged rows is encouraging, but not proof. Only step 1 or step 3 can give you that.

List the gaps for one context

Run this once per context the report flagged. It returns one sentence per gap: how many appointments went, which numbers, and roughly when they had been created.

-- CHANGE THIS: the context to inspect, from the report in step 3.
SET @cid            = 550733;
SET @recorded_since = UNIX_TIMESTAMP(NOW() - INTERVAL 12 WEEK) * 1000;

WITH numbers AS (            -- numbers still accounted for, for this context
  SELECT id, MIN(created) AS created
    FROM ( SELECT id, created FROM calendar_event
            WHERE cid = @cid AND account = 0
           UNION ALL
           SELECT id, created FROM calendar_event_tombstone
            WHERE cid = @cid AND account = 0 ) x
   GROUP BY id
), first_recent AS (         -- lowest number created inside the record window
  SELECT MIN(id) AS id_floor FROM numbers WHERE created >= @recorded_since
), lagged AS (
  SELECT id,
         LAG(id)      OVER (ORDER BY id) AS prev_id,
         DATE(FROM_UNIXTIME(created DIV 1000))                AS day,
         DATE(FROM_UNIXTIME(LAG(created) OVER (ORDER BY id)
                            DIV 1000))                        AS prev_day
    FROM numbers
)
SELECT n.prev_id + 1 AS from_number,
       CONCAT(n.id - n.prev_id - 1, ' appointment(s), numbers ', n.prev_id + 1,
              '-', n.id - 1, ', are gone with no deletion record. ',
              CASE WHEN n.prev_day IS NOT NULL
                        THEN CONCAT('Created after ', n.prev_day, '. ')
                   WHEN n.day IS NOT NULL
                        THEN CONCAT('Created before ', n.day, '. ')
                   ELSE '' END,
              CASE WHEN n.prev_id >= f.id_floor
                   THEN CONCAT('Recent enough that a normal deletion would ',
                               'still be on record - UNEXPLAINED.')
                   ELSE CONCAT('Too old to tell apart from a normal deletion ',
                               'whose record was pruned.')
              END) AS gap
  FROM lagged n CROSS JOIN first_recent f
 WHERE n.prev_id IS NOT NULL AND n.id - n.prev_id > 1
UNION ALL                    -- numbers below the oldest one accounted for
SELECT 1,
       CONCAT(MIN(id) - 1, ' appointment(s), numbers 1-', MIN(id) - 1,
              ', are gone with no deletion record. Cannot be dated.')
  FROM numbers
HAVING MIN(id) > 1
UNION ALL                    -- numbers issued after the newest one accounted for
SELECT MAX(id) + 1,
       CONCAT((SELECT id FROM calendar_event_sequence
                WHERE cid = @cid AND account = 0) - MAX(id),
              ' appointment(s), numbers ', MAX(id) + 1, '-',
              (SELECT id FROM calendar_event_sequence
                WHERE cid = @cid AND account = 0),
              ', are gone with no deletion record. These were the newest ever',
              ' created for this context.')
  FROM numbers
HAVING (SELECT id FROM calendar_event_sequence
         WHERE cid = @cid AND account = 0) > MAX(id)
 ORDER BY from_number;

Example result:

from_number | gap
         25 | 1311 appointment(s), numbers 25-1335, are gone with no
            | deletion record. Created after 2019-04-02. Recent enough
            | that a normal deletion would still be on record -
            | UNEXPLAINED.

Read the last sentence of each row first. "UNEXPLAINED" is what you act on: those appointments still existed within the last twelve weeks, so a normal deletion would have left a record, and there is none. Rows ending in "too old" are inconclusive on their own — an ordinary deletion whose record has since been pruned looks the same.

Some rows say nothing about dates. That happens when the neighbouring numbers are themselves deletion records that no longer carry a creation timestamp. The verdict is unaffected: it does not depend on those dates.

A gap tells you the context, not who lost it. Match it against the report above — the flagged calendar whose oldest_entered sits just after the gap is where the appointments went missing, and the row names its user.