3052 words, 93 steps, written 2022-02-17, updated 2024-04-09.

How to setup an encrypted server

Table of contents

How to setup an encrypted server

What we want to achieve:

Fully functional Debian system.
All data being fully encrypted and inaccesible to everyone, even with physical access.
(Re)bootable remotely.

The solution:

Unencrypted boot partion with an SSH server.
Use SSH server to decrypt OS partition remotely.
The startup OS boots the main OS.
The main OS decrypts any additional drives.

IMPORTANT: If you skip or miss any step, everything may seem to work properly and then suddenly break, causing you a lot of pain and suffering. Double check all steps and everything that you do!

Prepare installation

Tip: Click on a step to mark it as done, click again to undo.

Download any Debian live image (no WM is needed so -standard works great) (IMPORTANT: A normal netinst iso will not suffice!).

We only need this live boot environment to have a Debian shell on the target system to be able to format and encrypt the drives and install the OS from an internet source. You could, instead, hook your server drives into any other machine with a Debian based distro installed to follow this guide, though in that case make sure to specifiy the correct drives and not overwrite your host system or data!
Find your USB device (e.g. /dev/sdc or /dev/nvme1n1):
```
sudo fdisk -l
```
Flash ISO image to USB:
```
sudo dd bs=4M if=<path to iso> of=/dev/<device> status=progress oflag=sync
```
IMPORTANT: The devices specified here will be overwritten, be careful.
Boot your server from the USB and enter the live environment from the menu.

Install required packages:

sudo apt update
sudo apt install cryptsetup debootstrap

Optional step: Install SSH server to continue the installation remotely:
1. Install SSH server:
```
sudo apt install openssh-server
```
2. Change password for the user user:
```
sudo passwd user
```
3. Start the SSH server:
```
sudo systemctl start sshd
```
4. Connect to server:
```
ssh user@<server ip>
```

Partition drive

Find the device on which our OS should be installed (e.g. /dev/sdc or /dev/nvme1n1):
```
sudo fdisk -l
```
Create partitions on the drive:
1. Enter fdisk interface:
```
sudo fdisk <device>
```
  This will show the interactive fdisk interface used to partition drives. In the following steps I will specify the the command or input that you should input, after which you press Enter/Return to continue.
2. Create new GPT partition table: g.
3. Create partition for bootloader:
  1. Add new partition: n.
  2. Set partition number: leave empty to set to default, should be 1.
  3. Set first sector: leave empty set to default.
  4. Set last sector: +500M <Enter> (100 MB may be enough, but it's very annoying to increase later on).
  - If you did not wipe your drives, you will be prompted to remove the old signature, which you should do.
4. Repeat step 3 to create partition for startup OS. Step 3.2 should default to 2.
5. Repeat step 3 to create partition for main OS, but for step 3.4, leave empty to fill the rest of the drive. Step 3.2 should default to 3.
6. Set partition type for bootloader:
  1. Change partition type: t.
  2. Choose partition number: 1.
  3. Set partition type: 4 (this may differ, if you run L first you should pick the number for BIOS boot).
7. Repeat step 6 to set partition type for startup OS, but for step 6.3, choose 1 for EFI System.
8. To make sure everything is correct run p to list partitions. Types for devices 1-3 will be displayed and should be: BIOS boot, EFI System and Linux filesystem. If the third partition did not default to Linux filesystem, repeat step 6, run L and find the id for Linux filesystem to use for step 6.3.
9. Write the partition table: w (IMPORTANT: this cannot easily be undone).
Create file system on startup OS partition:
```
sudo mkfs.ext3 /dev/<device>
```
The device is the second partition we created (e.g. /dev/sdc2 or /dev/nvme1n1p2, note the 2 and p2 corresponding to partition). GRUB doesn't support ext4 so I'll be using ext3.

Encrypt main OS partion

Optional step: Generate a strong alphanumeric passphrase for the encryption:
```
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 100 | head -n 1
```
IMPORTANT: Remember to safely store this somewhere before proceeding with the installation.

Use cryptsetup to setup LUKS for the third partition (e.g. /dev/sdc3 or /dev/nvme1n1p3, note the 3 and p3 corresponding to partition number):
```
sudo cryptsetup luksFormat <device>
```
You will be prompted for a passphrase. Since I am installing over SSH I will generate a passphrase like in the previous optional step and copy it into the prompt. If you are not using SSH, you could set an easy password now and change it later (see how further down in this article).

After verifying the passphrase, the partition will be encrypted. This will take a few seconds.
Unlock the encrypted partition:
```
sudo cryptsetup luksOpen <device> cryptroot
```
Device is the same as specified in the previous step. The last argument is an arbitrary name for the mapped device which will be created at /dev/mapper/<new device name>. It will be used later. You may use whatever name you want, but I will name this device cryptroot and use that name in later steps.
Setup LVM on the encrypted partition:
1. Initialize physical volume to be used by LVM:
```
sudo pvcreate /dev/mapper/cryptroot
```
2. Create volume group:
```
sudo vgcreate vg0 /dev/mapper/cryptroot
```
  The first argument is the volume group's arbitrary name. I will name this volume group vg0 and use that name in later steps.
3. Create swap volume:
```
sudo lvcreate -L 8G -n swap vg0
```
  The size (8G) should be at least the size of your RAM, though this does not matter as much for servers as for desktop computers and could be lower or not even exist (this could crash the server when running out of memory). The name swap is, again, arbitrary.
  
  Note: A swap volume is not needed, you may, as mentioned, choose not to use swap, or, use a swapfile. I will not show how to use a swapfile, but you would not create this volume at all and instead create the swapfile on the root volume later.
4. Create root volume:
```
sudo lvcreate -l 100%FREE -n root vg0
```
  Using 100%FREE will make the volume take up the remaining space of the partition. This is the volume on which we will install Debian. The name root is, again, arbitrary.
5. Create file systems on volumes:
```
sudo mkswap /dev/vg0/swap
sudo mkfs.ext4 /dev/vg0/root
```
  This takes a few seconds.
  
  If you break anything with the installation in later steps then this is a good checkpoint. You may run the last command above at any time to overwrite your installation so that you may try again. Just make sure to run sudo umount -R /mnt to unount devices beforehand.

Install Debian

Mount the root volume:
```
sudo mount /dev/vg0/root /mnt
```
Use debootstrap to bootstrap a basic Debian system:
```
sudo debootstrap --arch amd64 <debian release> /mnt http://deb.debian.org/debian
```
Use the name of the latest Debian release in the above command. When writing this article, it's bullseye.

You could install another Debian based system here by changing the source used as the last argument. Search the interwebz if this interests you, as I will install Debian.

The bootstrapping usually takes 1-2 minutes.

Chroot into the Debian system:

sudo LANG=C.UTF-8 chroot /mnt /bin/bash

Mount proc and install makedev:

mount none /proc -t proc
apt install makedev

Create devices using makedev:
```
cd /dev
MAKEDEV generic
```
This takes a few seconds.
To mount some additional directories we have to briefly exit the chroot:
```
exit
```

Mount additional directories:

sudo mount /dev/<device> /mnt/boot
sudo mount --bind /dev /mnt/dev
sudo mount --bind /sys /mnt/sys
sudo mount --bind /proc /mnt/proc
sudo mkdir -p /mnt/run/udev
sudo mount --bind /run/udev /mnt/run/udev

Note: The device used in the first command should be the second partition we created, the startup OS partition (e.g. dev/sdc2 or /dev/nvme1n1p2).

Chroot back into the Debian system:

sudo LANG=C.UTF-8 chroot /mnt /bin/bash

Install required packages:
```
apt install busybox cryptsetup dropbear grub-pc linux-image-amd64 locales lvm2 mdadm ssh
```
If you get prompted by GRUB for which device to use for installation, choose your drive device without any partition number (e.g. /dev/sdc or /dev/nvme1n1).
Create /etc/adjtime:
```
hwclock --systohc
```
Set timezone:
```
ln -sf /usr/share/zoneinfo/<region>/<city> /etc/localtime
```
For me it's Europe and Stockholm. List the directories to see what your options are.
Configure /etc/fstab:

The /etc/fstab file is used to mount devices on boot. Making a mistake here will result in your system booting incorrectly or not booting at all.
1. Open /etc/fstab in a text editor:
```
nano /etc/fstab
```
2. Enter your devices and their mount points:
```
# <Device>       <Mount point>  <Type>  <Options>     <Dump>  <Pass>
proc             /proc          proc    defaults      0       0
/dev/vg0/swap    none           swap    sw            0       0
/dev/vg0/root    /              ext4    defaults      0       1
/dev/sdc2        /boot          ext3    defaults      0       2
```
  This is what we have got setup on our system so far, the swap and root volumes and the startup OS partition. When adding more drives to your system, they should be added to this file to be mounted at boot. Some pointers:
  - The proc line is no longer needed since our system is running systemd which mounts this device for us, but it doesn't hurt to have it listed anyway.
  - Note the swap type and sw option for our swap volume.
  - Note the device used on the last line, this is the second partition, the startup OS for decrypting our main OS drive.
  - Note the Pass being 0, 0, 1 and 2. This number is the order of mounting from lowest to highest. This is because some devices/mount point depend on others.
Configure /etc/crypttab:

The /etc/crypttab file is used to decrypt drives and partitions on boot. Much like with /etc/fstab, making a mistake here will result in your system not booting.
1. Open /etc/crypttab in a text editor.
2. Enter your encrypted partition:
```
# <Target name>  <Device>         <Key file>  <Options>
cryptroot        /dev/<device>    none        luks
```
  I will get back to this file when adding additional drives. With just an OS drive, the above config is sufficient. Some pointers:
  - cryptroot is the arbitrary name we want to use when mapping to device.
  - The device is the third partition we created and the one we put a LVM file system on (e.g. /dev/sdc3 or /dev/nvme1n1p3).
Configure network interfaces:
1. Open /etc/network/interfaces in a text editor.
2. Enter your network interface:
  
  This file does already contain some comments and configurations which you should not erase.
```
auto lo
iface lo inet loopback

auto enp4s0
iface enp4s0 inet static
    address 192.168.1.3
    gateway 192.168.1.1
    netmask 255.255.255.0
    broadcast 192.168.1.255
    ethernet-wol g
```
  The content depends on your network setup. The above is what I use. Some pointers:
  - enp4s0 is my ethernet interface found when running ip a. Before, this was always eth0, but it should be more cryptic nowadays. To be sure, you can install the pciutils package and run lspci and check that the numbers (4, 0 for me) match the Ethernet controllers pci id.
  - address is what I want my servers static ip to be. If you don't want to have a static ip, remove this line and replace static with dhcp on the line above.
  - gateway is the ip for my gateway. 192.168.0.1, 192.168.1.1, 10.0.0.1 are common, but you will have to find this out yourself.
  - ethernet-wol g is an option used to allow this server to boot when a magic packet is received. I will not configure this now, but look it up if you are interested.
Set a root passwd:
```
passwd
```
Note: Make it strong. Your data is encrypted on your drives, but once the system boots and attacker with physical access can grab the decrypted data through the TTY if they guess your password!

Optional step: Install sudo and create a sudo user instead of setting a root password.
Optional step: Install an SSH server:
1. Install the SSH server:
```
apt install openssh-server
```
2. Add your public SSH key to either /root/.ssh/authorized_keys or /home/<user>/.ssh/authorized_keys depending on if you want to connect as root or a sudo user (remember to set the correct permissions chmod -R u=rwX,g=,o= .ssh).

Setup the boot process

Generate locales:
1. Uncomment the locales you want to generate in /etc/locale.gen, for me it's en_US UTF-8 and sv_SE.UTF-8 UTF-8.
2. Generate locales:
```
locale-gen
```
Configure initramfs:
1. Open /etc/initramfs-tools/initramfs.conf in a text editor.
2. Enable the Busybox shell by changing BUSYBOX option to y.
- Optional step: Set a static ip and hostname for your startup OS:
```
IP=<server ip>::<gateway ip>:255.255.255.0:<server hostname>:<network interface>
```
  Note: yes, that should be two colons in a row.
  
  Add the above line anywhere in the file, I put it below BUSYBOX=y. The values used should be the same as when we configured network interfaces in a previous step. The hostname is up to you, though I recommend it to be the same as your server (whatever you set in /etc/hostname).
Configure Dropbear:
1. Open /etc/dropbear-initramfs/authorized_keys in a text editor:
2. Add your public SSH key to the file. This key needs to be used to login to the startup OS shell.
Generate initramfs image:
```
update-initramfs -u
```
Install GRUB:
```
update-grub
grub-install <device>
```
Note: this time the device is the drive and not a partition (e.g. /dev/sdc or /dev/nvme1n1).

update-grub may give an "error" saying it cannot find a GRUB drive for a device; if this error appears followed by "done" then it should be fine to ignore it. I believe this is because of interference with our live boot USB which has a GRUB bootloader installed on it, but I could be wrong.
Time to test our setup! Exit chroot, unmount and reboot:
```
exit
sudo umount -R /mnt
sudo reboot
```

Connect to server

Connect to startup SSH:
```
ssh -t -o StrictHostKeyChecking=no root@<server ip>
```
Disabling StrictHostKeyChecking is not perfectly secure, but it's the easiest way to bypass host key checking. If you want to be more secure there are probably more secure and proper ways out there.
Unlock main OS partition:
```
cryptroot-unlock
```
You will be prompted for the passphrase used to encrypt the drive. If the decryption is successful, the remote shell will close and your main OS will be booted.

Tip: ssh optionally takes a command to run as it's last argument. Use it to not have to run this command in the remote shell by connecting with ssh -t -o StrictHostKeyChecking=no root@<server ip> cryptroot-unlock.
Connect to server SSH:
```
ssh root@<server ip>
```

Congratulations, you now have an encrypted server!

Optional next steps

Change encryption passphrase

Generate a strong alphanumeric passphrase:
```
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 100 | head -n 1
```
Note: Save the new passphrase somewhere safe before you continue to the next step...
Change passphrase
```
cryptsetup luksChangeKey /dev/<device>
```
The device should be the third partition we created, the main OS partition (e.g. /dev/sdc3 or nvme1n1p3).

Follow the instructions and then it should be done.

Encrypt additional data drives

Important: This will wipe the data from the drives you are encrypting. If you have any data on the drives that you wish to keep, move it to another, temporary, drive and then move it back after the encryption setup is done.

When editing /etc/fstab or /etc/crypttab there is always a risk for something to go horribly wrong, resulting in your system not booting or something else. If this happen you will have to troubleshoot the problem from another system (like your Debian live boot USB stick) and access your encrypted server like we did during the installation.

Generate a strong alphanumeric passphrase to use for all additional drives:
```
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 100 | head -n 1 | tee /root/keyfile
chmod 600 /root/keyfile
```
You may store this passphrase in any file you want, but I use /root/keyfile. A keyfile is needed for automatically decrypting drives on boot.

Tip: Store this keyfile securely on another machine so that you can recover your data in case your main OS drive is corrupted.

I use the same passphrase for all drives on the same machine.
Encrypt using LUKS:
```
cryptsetup luksFormat /dev/<device>
```
The device is the device of the drive to format. You can get this from fdisk -l. Quadrople check that you choose the correct device as to not destroy important data or your new Debian installation.

I use the same passphrase here as the one generated for the keyfile from the previous step. However, you may use whatever passphrase you want here. This will not be used for automatic decryption. You may think of it as a "backup passphrase".

Add keyfile:

cryptsetup luksAddKey /dev/<device> /root/keyfile

Unencrypt the drive:
```
cryptsetup luksOpen /dev/<device> drive0
```
The last argument is an arbitrary name for your new drive.
Create a file system on the encrypted drive:
```
mkfs.ext4 /dev/mapper/drive0
```
Note: the name from the previous step is used here (I chose drive0).
Add to /etc/crypttab to decrypt on boot:
1. Open /etc/crypttab in a text editor.
2. Add your new drive on a new line:
```
drive0    /dev/<device>    /root/keyfile    luks,keyscript=lib/cryptsetup/scripts/passdev
```
  The name drive0 is, again, arbitrary, but it has to be the same as in the next step.
Add to /etc/fstab to mount on boot:
1. Open /etc/fstab in a text editor.
2. Add your new drive on a new line:
```
/dev/mapper/drive0    /mnt/drive0    ext4    errors=remount-ro    0    2
```
  I want my drive to be mounted on /mnt/drive0. Make sure that this directory exists before you reboot, otherwise you will have a bad time.

Troubleshooting

When troubleshooting, you will have to use another system (like Debian live boot) to access your encrypted drive like we did during the installation. In short, these are the commands you need to run on the recovery OS:

sudo apt install openssh-server cryptsetup
sudo systemctl start sshd
# Connect to SSH
sudo cryptsetup luksOpen /dev/sdc3 cryptroot
sudo mount /dev/vg0/root /mnt
sudo mount /dev/sdc2 /mnt/boot
sudo mount --bind /proc /mnt/proc
sudo mount --bind /dev /mnt/dev
sudo mount --bind /sys /mnt/sys
sudo mkdir -p /mnt/run/udev
sudo mount --bind /run/udev /mnt/run/udev
sudo chroot /mnt

Not booting or GRUB rescue: Unknown file system

There may be something wrong in /etc/crypttab. Check that all names and devices are correct. Otherwise you may want to attempt troubleshooting GRUB (this sucks).

Booting with read-only file-system

Probably something wrong in /etc/fstab. Check that all names and devices are correct.

Dropbear not starting in initramfs after kernel upgrade

I encountered issues when going from kernel 5.x to 6.x that the /boot/config-6.x was missing which led to dropbear failing to start during boot. To fix this I reinstalled the kernel and copied the old /boot/config-5.x to /boot/config-6.x. Not sure which of the fixes actually fixed the issue. Remember running update-initramfs -u after making changes.

Other sources

https://lerks.blog/p/how-to-securely-set-up-a-new-server (Huge help, many thanks to the author!)
https://wiki.archlinux.org/title/Installation_guide
https://wiki.archlinux.org/title/GRUB