How to populate a registry mirror

Abstract

You want to install "everything" (as covered by this repo) on an air-gapped k8s, i.e. without internet access.

This means you need to provide images to your k8s container runtime on a registry mirror which is available for your air-gapped k8s nodes.

If your k8s client (as in, node where your kubectl and helm commands run) is also air-gapped, this means you need to provide the helm charts "offline" there as well.

There are many talks around this subject; in general, it is considered best practice, allows you to get independent from upstream mirrors, network availability, depublication from our favourite chart version, prevents from unwanted upgrades, etc.

It offers improved (local) performance for chart and image downloads, rather than slow internet, corporate VPNs, etc.

For labs and sandboxes, it allows for constant redeploying without hitting upstream registry rate limits.

Multiple benefits are to be weighted against the setup efforts (which are manageable, as outlined below).

Air-gapped k8s

Out of scope of this document, as in general k8s installation is out of scope of this repo. Contact your k8s distro / flavor documentation on that matter.

Air-gapped Images

Use the following render.py / lab.yml configuration:

image_registry: my.registry.tld
batteries_registry: my.registry.tld
global_registry_mirror: my.registry.tld

This allows individual configuration of mirrors for our application images, our battieres, and everything else (platform / infrastructure).

In the most straightforward setup, you'll set them all to the same value: to the hostname (optionally with :port) of your registry mirror.

As usual: render the configs and run the install script.

Initially, most / all pods will run into ImagePullBackOff errors. We handle this "automatically", see next section.

Note: some suggest that you don't reconfigure the image: location, but rather use a container runtime with "mirror" feature (like containerd). However, this comes with certain prerequisites (like, being able to reconfigure the k8s accordingly), so we don't follow that approach and use the more explicit and generally available approach of rewriting the image: values.

Tooling

We have a python script called mirror-images.py. This script uses the k8s python API client (pip install kubernetes) to walk the k8s pods, look for images in ImagePullBackOff status, and pulls these.

It carries heuristics for each project's images about the upstream registry they use. It is overrideable on command line (copy-paste the yaml into a config file and supply it via argument), or just modify the script's source if need be. (But, out of the box, no modifications should be necessary.)

Invoke the mirror-images.py on the namespace you're interested, or just on all namespaces (-A), as familiar from kubectl.

Of course this approach requires that the host you're executing this script has both k8s api access, and internet access to pull the upstream images. In a pure air-gapped situation, you won't have this available. It is still useful as a baseline documentation on how to approach this topic; adopt to your needs. (Or just use the --dry-run argument to print required commands and populate your registry mirror yourself based on tins info.)

There is also a -l, --loop switch to have the script sleep and run forever to automatically pull images ongoingly.

(You might wonder why it is not required to provide the registry mirror as argument: this can be read from the image: value of the affected pods; we do have the "mirrored" value there, but not the original/upstream value. That's why we need heuristics to "guess" the upstream value from the mirrored value. There is no easy / programmatic way of obtaining this, in particular for the multiple different "battery" charts.)

Verify we did not miss images (to define to come via our mirror):

kubectl get pod -A -o yaml | grep image:

To find out the upsteam registry, have a look at the templated helm chart without any options and look for image: tags.

Air-gapped Helm Charts

This can be implemented by helm pulling the required charts to the local disk of your k8s client kubectl/helm machine and use these local charts on helm install.

To have the install script rendered with these options, use the settings

airgapped_charts: true
charts_cache: charts

The former enables this feature, the latter configures the path (relative to the repo's root directory) where the charts will be put / searched for.

A sample diff of with vs without this configuration excerpt:

-    helm upgrade --install -n "as8" --create-namespace "as8" oci://registry.open-xchange.com/appsuite/charts/appsuite --version 8.35.276 --values "values.yaml" --values "values.secret.yaml" --values "values.globaldb.yaml" --wait
+    helm upgrade --install -n "as8" --create-namespace "as8" ../../charts/appsuite-8.35.276.tgz --values "values.yaml" --values "values.secret.yaml" --values "values.globaldb.yaml" --wait

To populate the charts directory, we render a pull.(sh|ps1) file with the approriate helm pull commands. It is rendered from template to only mirror charts we use (rather than all we know). Sample contents looks as follows:

Source for this is the dependencies.yml file in the project root directory which carries the versions and repositories/registries for all helm charts we use.

This file needs active maintenance.