Rook Ceph Configuration

The simplest and most straightforward configuration is to enable the upstream behavior to utilize all available nodes and devices.

# fragment of the rendered values.rook-ceph-cluster.yaml
cephClusterSpec:
  storage:
    useAllNodes: true
    useAllDevices: true

This behavior is triggered by the following configuration:

# lab.yml
rook_ceph_use_all_nodes: true

However, this configuration requires Kubernetes nodes to possess additional block devices (e.g., /dev/vdb or /dev/sdb) to establish a functional cluster. Therefore it is recommended to provision Kubernetes nodes with additional block storage and explicitly enable rook_ceph_use_all_nodes: true.

In environments lacking dedicated disks, loop devices offer an alternative. However, Rook-Ceph does not automatically detect loop devices when useAllNodes and useAllDevices are enabled. Explicit configuration via the storage.nodes object is required.

# fragment of the rendered values.rook-ceph-cluster.yaml
cephClusterSpec:
  storage:
    nodes:
      - name: "kubelet1"
        devices:
          - name: "/dev/loop0"

The storage.nodes object is parametrized via the lab.yml setting rook_ceph_nodes:

# lab.yml
rook_ceph_nodes:
  - name: "kubelet1"
    devices:
      - name: "/dev/loop0"

Note: As a prerequisite, for using loop devices, the operator needs to have been configured with allowLoopDevices: true, which we do by default.

Node names have to be written out. There is no way around it for this mode: Ceph does not discover loop devices on its own, so each node that should carry an OSD has to be named, and a node name is not knowable when the configuration is rendered. kubectl get nodes lists them.

Creating the loop devices themselves is automated: the batteries/rook-ceph/helm/losetup Helm chart deploys a Job with privileged pods that creates them on the Kubernetes nodes (defaulting to a sparse 50Gi file at /var/lib/disk0.img). While this default behavior is currently not configurable, the chart source offers a reference for integrating this logic into systemd services if necessary.

install_losetup_job: true

examples/lab.rook-ceph-lab-on-loop.yml is a ready-made starting point for this combination.

An earlier rook_ceph_nodes_auto mode built the storage.nodes list by running kubectl get node from the install script. It was removed: the installation must not read the cluster to decide its own configuration, because that makes the rendered output depend on where it runs and rules out applying it with a GitOps tool such as Argo CD.

Alternatively, the OpenEBS LocalPV-LVM storage provisioner can be deployed on the block device (initialized as an LVM VolumeGroup). This enables a shared storage architecture where the device supports both Rook Ceph and other consumers, offering Block and Filesystem volumes with strict capacity enforcement—a feature absent in the previously used Rancher local-path provider.

Analogous to the loop device automation, the framework allows for automated LVM initialization via the vgcreate_job. Configuration options include device selection and volume group naming.

install_vgcreate_job: true
vgcreate_job_devices:
  - /dev/loop0

Implementation Note: The OpenEBS storage provider is installed by default in localpv-hostpath mode. Enabling localpv-lvm mode provides LVM-based volumes, enabling block mode and capacity enforcement.

install_openebs_lovalpv_lvm_storageclass: true
openebs_localpv_lvm_storageclass_as_default: true
openebs_localpv_hostpath_storageclass_as_default: false

This configuration enables Rook-Ceph to consume Kubernetes PersistentVolumes via PersistentVolumeClaims (PVCs), aligning with cloud-native storage patterns.

rook_ceph_use_pvcs: true

Refer to lab.default.yml for additional configuration options regarding this mode.

While Ceph typically requires a minimum of three nodes, configuration adjustments allow for single-node deployments by reducing the replication factor and modifying other settings. It defaults to false; a single-node cluster has to say so:

rook_ceph_singlenode: true

This is a configuration decision, not a detected one. Earlier versions probed the cluster at install time and picked a value, which made the same rendered script produce a different storage layout depending on where it was run.

In summary, three primary use cases are defined:

• rook_ceph_use_all_nodes
• rook_ceph_nodes
• rook_ceph_use_pvcs

Most modes support loop devices, LVM logical volumes, or physical devices. The use_all_nodes mode (enabling storage.useAllNodes: true and storage.useAllDevices: true) explicitly requires physical devices and is incompatible with loop or LVM devices.

Review the rendered values.*.yaml files and the installation script to verify the configuration and installation order. Note that the installation script behavior may be dynamic, performing runtime kubectl lookups depending on the selected mode. This dynamic generation offers significant convenience despite increasing the complexity of the underlying Jinja2 templates.

An additional method involves statically defining PersistentVolumes on existing local devices (refer to batteries/static-lvm-pvs/helm/static-lvm-pvs). This option is not integrated into the primary scripting framework.