Access Encrypted ZFS from Rescue Boot

Views: 710

How I Lost Control

When working with root rights on Linux, you’ll always find a possibility to shoot yourself in the foot. My choice for self-harm this time was to use usermod in a wrong way. The whole story turned worse, because I was on my very new Laptop, that I bough yesterday: As always, purged Windoze and installed Ubuntu 22.04. As always: Works like a charm! But I thought it was a good idea, to install the filesystem on encrypted ZFS, instead of an encrypted LVM, as I always did before. But ZFS has been reported to be cool by some geeky friends, so who needs to know what he does, when you just can click a checkbox in a UI.

After installing docker, you should add yourself to the group docker to be able to use docker without sudo. A normal person would read the best blog entry on docker to get the exact command. But not me, I thought it’s a good idea to write the command from my mind: usermod -G docker marc adds group docker to user marc — I thought… Well, lazyness pays off often, but not always. This time, I had better read the man page before:

       -G, --groups GROUP1[,GROUP2,...[,GROUPN]]] 
          A list of supplementary groups which the user is also a member of. Each group is separated from the next by a comma, with 
          no intervening whitespace. The groups are subject to the same restrictions as the group given with the -g option. 

          If the user is currently a member of a group which is not listed, the user will be removed from the group. This behaviour 
          can be changed via the -a option, which appends the user to the current supplementary group list.

So a simple usermod -aG docker marc would have done the trick. But without the little -a, docker was not appended to my groups, but it replaced all my groups, so I was thrown out of all other groups, including the sudoers, admin and many more. With that, I lost all administration rights on my laptop; sudo was no more possible. Of course, I noted that today, one day later, because it was effective only after a reboot. Would I have noticed on time, I still could have fixed it.

But now, I had wrong entries in /etc/group and couldn’t change them without root access.

Fix Broken Root Access on Linux

To fix lost root access on Linux, simply boot e.g. a Ubuntu Desktop as rescue system from an USB stick. There you can install or try Ubuntu, just try Ubuntu, then you’re in a full Ubuntu environment, where you have access. Just mount the laptop disk to fix the problem.

And because that’s so easy (not only on Linux, but on every system), you must always encrypt your installations!

Mount Encrypted ZFS on Ubuntu 22.04

Ok, mounting an encrypted LVM is hard enough, but at least you can google it, and you find the necessary instructions. But decrypting a ZFS from a rescue system? Finally, I found this: Rescuing using a Live CD. Unfortunately, the very first line is bullshit.

Ignore that for now and start by initializing the ZFS pools:

sudo zpool export -a
sudo zpool import -N -R /mnt rpool
sudo zpool import -N -R /mnt bpool

After searching around and analyzing my Ubuntu installation, I found, that ZFS is not encrypted directly by the harddisk encryption password you set during the installation, but your key encrypts a LUKS block device, this partition contains a file, that file contains the real decryption key. This block device is /dev/zvol/rpool/keystore, which is a link to /dev/zd0 and decrypted device is named by convention keystore-rpool. So that’s what you need to decrypt with your password:

sudo cryptsetup luksOpen /dev/zd0 keystore-rpool

By convention, this has to be mounted to /run/keystore/rpool, so that it is found and accessed by zfs load-key -a, which finally decrypts the ZFS. That’s all, so mount, decrypt and clean up:

sudo mkdir -p /run/keystore/rpool
sudo mount /dev/mapper/keystore-rpool /run/keystore/rpool
sudo zfs load-key -a
sudo umount /run/keystore/rpool
sudo cryptsetup luksClose keystore-rpool

Now you need to find your ZFS› ubuntu_UUID. Command zfs list shows it, but in a large output, so get that cleaner:

zfs list | sed -n 's/^\(rpool\/ROOT\/ubuntu_[^/ ]*\).*/\1/p' | uniq

This outputs something like:

rpool/ROOT/ubuntu_iukzaq

So my ubuntu_UUID is ubuntu_iukzaq, instead of:

sudo zfs mount rpool/ROOT/ubuntu_UUID
sudo zfs mount bpool/BOOT/ubuntu_UUID
sudo zfs mount -a

I replace ubuntu_UUID by ubuntu_iukzaq and mount ZFS typing:

sudo zfs mount rpool/ROOT/ubuntu_iukzaq
sudo zfs mount bpool/BOOT/ubuntu_iukzaq
sudo zfs mount -a

Now my Ubuntu root filesystem is mounted at /mnt and I can repair my /etc/group file in /mnt/etc/group!

How To Decrypt a Filesystem Without Password

Not. That’s the idea of encryption.

If you forgot your password, or if you crash your key file, your data is lost. So take care of your passwords and keyfiles, and do backups of your valuable data!