S3 Filestores deprecated

General

For S3 filestores, client-side HTTP connection pools and request/response latencies can be monitored. This requires client-centric configuration, i.e. access to the storage system must use the com.openexchange.filestore.s3client. configuration style instead of per-filestore (com.openexchange.filestore.s3.) configuration as it was common pre-7.10.4. See this for details.

With S3 clients and the mapping from registered filestores to configured clients properly configured, monitoring can be enabled by setting com.openexchange.filestore.s3.metricCollection = true. This setting results in three additional outputs that can be recorded by monitoring systems:

  1. For each configured client, connection pool monitoring appears in Prometheus output.
  2. Each request-response roundtrip is measured in terms of latency and appears in Prometheus output.
  3. Overall (across all clients) byte throughput is measured and appears in Prometheus output.

1. and 2. however only apply, if the number of different configured clients is smaller than a configurable threshold com.openexchange.filestore.s3.maxNumberOfMonitoredClients (default: 20). This is to limit the exposed Prometheus time series in cases where - for whatever reasons - large numbers of clients are configured.

Example configuration

The following configuration is assumed throughout the other sections of this article, if not stated otherwise. It specifies the usage of AWS S3 (instead of an API-compatible on-premise system) with a single client "default" being configured. The client connects to region "eu-central-1" and handles all buckets with name prefix "appsuite-files-". Its connection pool is limited to 20 concurrent connections.

A single registered filestore is explicitly mapped to a bucket name that is handled by the "default" client.

Metric collection is enabled.

com.openexchange.filestore.s3client.default.region = eu-central-1
com.openexchange.filestore.s3client.default.pathStyleAccess = false
com.openexchange.filestore.s3client.default.encryption = none
com.openexchange.filestore.s3client.default.maxConnectionPoolSize = 20
com.openexchange.filestore.s3client.default.credentialsSource = iam
com.openexchange.filestore.s3client.default.buckets = appsuite-files-*

com.openexchange.filestore.s3.appsuite-files-1.bucketName = appsuite-files-1

com.openexchange.filestore.s3.metricCollection = true

Logging output

The per-request statistics logging of the formerly used AWS SDK for Java v1 (logger com.amazonaws.latency) does not exist anymore in the AWS SDK for Java v2. Request-level insights are available through the Prometheus metrics described below; for troubleshooting, debug logging can be enabled for the SDK logger software.amazon.awssdk.request.

Connection pool monitoring

This will show up in Prometheus output:

# HELP appsuite_filestore_s3_connections_pending The number of threads pending on getting a HTTP connection from the pool.
# TYPE appsuite_filestore_s3_connections_pending gauge
appsuite_filestore_s3_connections_pending{client="default",} 0.0
# HELP appsuite_filestore_s3_connections_max The configured maximum number of concurrent connections.
# TYPE appsuite_filestore_s3_connections_max gauge
appsuite_filestore_s3_connections_max{client="default",} 20.0
# HELP appsuite_filestore_s3_connections_leased The number of leased pooled HTTP connections.
# TYPE appsuite_filestore_s3_connections_leased gauge
appsuite_filestore_s3_connections_leased{client="default",} 0.0
# HELP appsuite_filestore_s3_connections_available The number of available pooled HTTP connections.
# TYPE appsuite_filestore_s3_connections_available gauge
appsuite_filestore_s3_connections_available{client="default",} 1.0

The ratio of leased vs. max connections is worth to be observed. If it reaches 100%, pending connections will go up and effectively users will need to wait for an available connection until their requests can be performed. A constantly high value of used connections might hint towards a too small pool size or even latency issues with the S3 system.

Operation latencies and error rates

Latencies and error rates show up in Prometheus output like so:

# HELP appsuite_filestore_s3_requests_seconds_max S3 HTTP request times
# TYPE appsuite_filestore_s3_requests_seconds_max gauge
appsuite_filestore_s3_requests_seconds_max{client="default",status="200",type="PutObject",} 0.1
appsuite_filestore_s3_requests_seconds_max{client="default",status="200",type="HeadObject",} 0.019
appsuite_filestore_s3_requests_seconds_max{client="default",status="200",type="GetObject",} 0.096
appsuite_filestore_s3_requests_seconds_max{client="default",status="200",type="HeadBucket",} 3.031
appsuite_filestore_s3_requests_seconds_max{client="default",status="400",type="HeadBucket",} 0.264
# HELP appsuite_filestore_s3_requests_seconds S3 HTTP request times
# TYPE appsuite_filestore_s3_requests_seconds summary
appsuite_filestore_s3_requests_seconds_count{client="default",status="200",type="PutObject",} 1.0
appsuite_filestore_s3_requests_seconds_sum{client="default",status="200",type="PutObject",} 0.1
appsuite_filestore_s3_requests_seconds_count{client="default",status="200",type="HeadObject",} 1.0
appsuite_filestore_s3_requests_seconds_sum{client="default",status="200",type="HeadObject",} 0.019
appsuite_filestore_s3_requests_seconds_count{client="default",status="200",type="GetObject",} 1.0
appsuite_filestore_s3_requests_seconds_sum{client="default",status="200",type="GetObject",} 0.096
appsuite_filestore_s3_requests_seconds_count{client="default",status="200",type="HeadBucket",} 1.0
appsuite_filestore_s3_requests_seconds_sum{client="default",status="200",type="HeadBucket",} 3.031
appsuite_filestore_s3_requests_seconds_count{client="default",status="400",type="HeadBucket",} 1.0
appsuite_filestore_s3_requests_seconds_sum{client="default",status="400",type="HeadBucket",} 0.264

The separation by status code and request type helps to narrow down issues with high error rates or latencies. The usual Micrometer configuration can be used to even turn these summaries into histograms, if needed. See here for details.

Transferred bytes

This is an overall metric, applied to all S3 clients. It measures the amount of data sent and received to/from S3. Prometheus can convert this into a rate.

# HELP appsuite_filestore_s3_transferred_bytes_total The size of s3 requests.
# TYPE appsuite_filestore_s3_transferred_bytes_total counter
appsuite_filestore_s3_transferred_bytes_total{type="S3UploadThroughput",} 153523.0
appsuite_filestore_s3_transferred_bytes_total{type="S3DownloadThroughput",} 153523.0