Multi-Disk Encryption Magic

Views: 12

When you need large storage, larger than any available disk, hdd or ssd, the classical way is to include them as separate disks and mount them to different paths, such as to /var, /home / or similar. But this does not allow you to optimally use the available space. While one mountpoint may be full, another still may have a lot of space left. Let’s say, you have two disks, one 2TB the other 4TB, but you want two partitions of 3TB each. That’s not easily possible. So it would be much better for optimized use to combine all disks into one large disk, so that all paths share the same space, then divide them as you need. And of course everything should be encrypted. Other option, if you have only one huge drive and still need some logical splitting, e.g. between root at / and the user data at /home, btrfs allows you to use sub-volumes, but that’s another story.

Combining several physical disks into one secure, efficient, and easy-to-manage encrypted system is therefore a challenge. The objective is to create a single, large logical drive that spans multiple disks, encrypted with one single passphrase. During boot, only one password should be required.

In Linux, there is the logical volume manager LVM that allows to combine any number and size of physical device, a partition in a physical device or even logical volume into a completely new devices. First a physical volume PV is attached to a real physical device or even logical volume. Then any number of physical devices can be grouped to one or more volume groups VG. This volume groups then can contain logical volumes LV. Encryption can be done either at the level of an unpartitioned physical device, respectively at a partition in a physical device or on a logical volume.

We want to combine four large hard disks /dev/sda, /dev/sdb, /dev/sdc and /dev/sdd, each is let’s say a 20TB disk, into one huge 80TB volume. This huge volume should then be encrypted with one single password and still provide the flexibility to ceate several logical volume sin the same encrypted device. As a sample, we will create one partition for the root path / and one for swap, all encrypted with the same password. It may not be the best sample, since typically swap can get a random password at boot, but if you need a suspend to disk, then the swap space should be recoverable. So let’s use it in this sample.

Not discussed in this article is that you’ll need a separate /boot-partition that is not encrypted. I use HP ProLiant severs with 4 disks and I plug an additional USB-stick inside of the case for the /boot-partition. In older setups, I just split /dev/sda into a /dev/sda1 for the /boot-partition and /dev/sda2 for the physical volume in my LVM setup. Be aware, you’ll need an additional bios-partition of 1MB and in modern systems with UEFI, also a UEFI-partition mounted to /boot/uefi. I will explain the system full setup and installation from scratch in a next blog.

First LVM Layer for Encryption

In the first layer, we attach physical volumes on physical devices to create a new logical volume named big. This then is used at 100% of the available space by one single large logical volume, also named big. So we will have a volume group named big and accessible as /dev/big and inside of that volume group, there is one large logical volume named big and accessible as /dev/big/big.

Each disk becomes a physical volume, and together they hold the encryption container:

pvcreate /dev/sda /dev/sdb /dev/sdc /dev/sdd
vgcreate big /dev/sda /dev/sdb /dev/sdc /dev/sdd

This creates a lower-level volume group named big with all disks included.

In the next step, we create one large logical volume in the volume group:

lvcreate -n big -l 100%FREE big

The Encryption

This large logical volume is still unencrypted. But we can treat it like any partition in LUKS and encrypt it entirely:

cryptsetup luksFormat /dev/big/big

This single layer covers all four drives at once and is now fully encrypted, but not yet accessible for further processing. We need to open the encrypted container:

cryptsetup open /dev/big/big crypt

This creates a logical device at /dev/mapper/crypt, sice we gave it the name crypt.
We can now further process on that end point, which represents the decrypted (opened) version of the encrypted device /dev/big/big.

Second LVM Layer for Filesystems

Instead of formatting the decrypted device with a filesystem such as ext4, we just use that again as a physical volume for another volume group. This nested layout gives flexibility for logical volumes while keeping one encryption key.

pvcreate /dev/mapper/crypt
vgcreate crypt /dev/mapper/crypt

The new entirely encrypted volume group crypt now appears as /dev/crypt and we are free to create any structure of logical volumes inside, e.g. we use 32GB as swap space and all the remaining space for our root file system:

lvcreate -n swap -L 32G crypt
lvcreate -n root -l 100%FREE crypt

This creates the new devices /dev/crypt/swap and /dev/crypt/root.

We can initialize the swap space:

mkswap /dev/crypt/swap

And we can format the root file system:

mkfs.ext4 /dev/crypt/root

Now all data lies under one LUKS layer, and LVM can be extended or resized later. You can use the swap or mount the filesystem. In a later article I’ll explain how to manually setup an Ubuntu system using debootstrap, because the current installer fails with this configuration.

Hints

Reuse Disks

Old metadata from LVM, RAID or LUKS may cause setup errors like «device is partitioned». You may purge this information, but only if you are sure, you’re acting on the right pdisks. Use wipefs carefully to cleanup old information before creating new layouts, e.g.:

wipefs -a /dev/sda /dev/sdb /dev/sdc /dev/sdd

This permanently erases filesystem and RAID metadata, so check each device name with fdisk -l, lsblk or blkid before running, to make sure you only purge the disks you want to clean up.

Some Warnings

Be aware, that with this constellation, when one disk fails, all data is lost. So do it only in configurations, where you mirror your data over several servers or you add an additional RAID at the bottom of everything, or at least when you do regular backups. Besides large storage servers, I also use this configuration on my laptops when they have more than one disk (e.g. NVMe + SATA), to have one large drive.

A separate recommendation based on my personal experience: Avoid Seagate! Every Seagate disk I ever bough failed rather sooner than later! Some disks did not even survive the first week. So Seagate is on my black list.

Ubuntu Installer

If you choose LVM + Encryption in the installer on Ubuntu 24.04 server, it creates one encryption layer per disk, forcing multiple password entries and adding unnecessary complexity. It doesn’t make sense, but there are many stupid decisions made by Ubuntu, still it remains my recommended distribution. But unfortunately, this means, that the full setup including the Ubuntu installation is a fully manual process.