Proxmox - Migrate ZFS on root from 6.4-13 to 7.x

I had manually migrated my proxmox installation from LVM to ZFS on root. Although it worked very well, I am now paying a price for not have used the proxmox installer. Basically, newer versions of proxmox requires you to use their boot tool when running ZFS on root.

Reading their documentation, it was clear that to use their tool I would need an extra small partition automatically created when installed using their installer. However, this was not my case.

To fix this issue, I will install proxmox 6.4.1 fresh in new pair of disks. After the installation, I will migrate my current data to the new ZFS pool and then configure the proxmox boot tool. This approach will avoid a complete migration of all my configs and VMs/Containers to a new proxmox install.

Table of Contents

├── 1 - Disk Preparation

├── 2 - Update Proxmox GUI

├── 3 - Proxmox 6.4 New Installation

├── 4 - ZFS Pool Migration

├── 5 - Booting From the New Installation

├── 6 - Upgrading Proxmox

└── Conclusion

1) Disk Preparation

I will replace the 2x 146GB in slot 0 and slot 1 for 2x 600GB disks. These new disks will accommodate the new proxmox installation.

FW2 disks have been migrated to a temporary ZFS pool and for precaution a full backup of the config was performed.

The RAID 1 is ready to be deleted.

megacli -cfglddel -l0 -a0
Adapter 0: Deleted Virtual Drive-0(target id-0)

Exit Code: 0x00

2) Update Proxmox GUI

After deleting the virtual disk, we need to update proxmox's GUI.

We will do that editing /etc/pve/storage.cfg and deleting the reference to the old virtual disk.

...

lvmthin: HV2-FW2
        thinpool HV2-FW2
        vgname HV2-FW2
        content rootdir,images
        nodes hv2

...
/etc/pve/storage.cfg

3) Proxmox 6.4 New Installation

With the two new disks ready a fresh installation of proxmox 6.4 will be done.

The proxmox installer is quite basic and it does not show the disks serial. And, I don't want to delete my current installation by mistake.

If you start the installer in debug mode. After the disks detection it is possible to use lsblk to determine how the installer mapped the disks.

{
        "slot-number": 0,
        "enclosure-id": "17",
        "enc-position": "1",
        "device-id": "37",
        "wwn": "5000C5007228FFB0",
        "media-error-count": "0",
        "other-error-count": "2",
        "predict-fail-count": "0",
        "pd-type": "SAS",
        "raw-size": 600126116593.664,
        "sector-size": "512",
        "logical-size": "512",
        "firmware-state": "JBOD",
        "serial": "SEAGATEST600MM00060004S0M2BLYE",
        "device-speed": "6.0Gb/s",
        "link-speed": "6.0Gb/s",
        "drive-temp": "30C",
        "os-path": "/dev/sdn"
      },
      {
        "slot-number": 1,
        "enclosure-id": "17",
        "enc-position": "1",
        "device-id": "48",
        "wwn": "5000C50075D25A38",
        "media-error-count": "0",
        "other-error-count": "0",
        "predict-fail-count": "0",
        "pd-type": "SAS",
        "raw-size": 600126116593.664,
        "sector-size": "512",
        "logical-size": "512",
        "firmware-state": "JBOD",
        "serial": "SEAGATEST600MM00060004S0M2JE8M",
        "device-speed": "6.0Gb/s",
        "link-speed": "6.0Gb/s",
        "drive-temp": "30C",
        "os-path": "/dev/sdx"
      }
** ATTENTION ** Be extremely careful here otherwise you can destroy your data if installing in the wrong disks.

I will proceed since I am confident that the disks to select are /dev/sdk & /dev/sdu

Once the installer loads, select Options -> Filesystem : ZFS RAID1 -> Deselect All and select the drivers that corresponds to your installation needs.

Go to Advanced Options and change the ashift to match your HDD strip size. My installation requires it to be 9. The math is 2^9 = 512.

{
"sector-size": "512",
"logical-size": "512"
}
My disks sector size.

After the installation, I have booted as before from the old disks. However, for a reason beyond my knowledge the disks changed its names.

Proxmox failed to boot and I had to manually import the old ZFS pool. Because the cache file was expecting different disk names.

zpool import -N rpool_ID
exit

4) ZFS Pool Migration

It is time to migrate our old installation on top of the new one.

zpool import -N 9653296271845752370 proxmoxnew
Importing the new proxmox pool.

I have use the flag -N to avoid ZFS trying to mount the datasets. If you run the command zpool import with no flags it will be possible to find the pool ID. Also, a new name is needed because we already have a pool named rpool.

zfs list
NAME                       USED  AVAIL     REFER  MOUNTPOINT
proxmoxnew                 928M   538G       25K  /proxmoxnew
proxmoxnew/ROOT            928M   538G       24K  /proxmoxnew/ROOT
proxmoxnew/ROOT/pve-1      928M   538G      928M  /
proxmoxnew/data             24K   538G       24K  /proxmoxnew/data
rpool                     30.2G   239G      104K  /rpool
rpool/ROOT                11.9G   239G       96K  /rpool/ROOT
rpool/ROOT/pve-1          11.9G   239G     11.9G  /
rpool/data                10.0G   239G       96K  /rpool/data
rpool/data/vm-101-disk-0  10.0G   239G     10.0G  -
rpool/swap                8.25G   247G     1.04M  -
temp1-pool                10.6G  6.39T     47.1K  /temp1-pool
temp1-pool/vm-101-disk-0  10.6G  6.39T     2.33G  -

We will need a snapshot of the current pool and its datasets in order to replicate it.

zfs snap -r rpool@22-07-23_00-14
zfs list -o name,avail,type,mountpoint -t all
NAME                                     AVAIL  TYPE        MOUNTPOINT
proxmoxnew                                538G  filesystem  /proxmoxnew
proxmoxnew/ROOT                           538G  filesystem  /proxmoxnew/ROOT
proxmoxnew/ROOT/pve-1                     538G  filesystem  /
proxmoxnew/data                           538G  filesystem  /proxmoxnew/data
rpool                                     239G  filesystem  /rpool
rpool@22-07-23_00-14                         -  snapshot    -
rpool/ROOT                                239G  filesystem  /rpool/ROOT
rpool/ROOT@22-07-23_00-14                    -  snapshot    -
rpool/ROOT/pve-1                          239G  filesystem  /
rpool/ROOT/pve-1@22-07-23_00-14              -  snapshot    -
rpool/data                                239G  filesystem  /rpool/data
rpool/data@22-07-23_00-14                    -  snapshot    -
rpool/data/vm-101-disk-0                  239G  volume      -
rpool/data/vm-101-disk-0@22-07-23_00-14      -  snapshot    -
rpool/swap                                247G  volume      -
rpool/swap@22-07-23_00-14                    -  snapshot    -
temp1-pool                               6.39T  filesystem  /temp1-pool
temp1-pool/vm-101-disk-0                 6.39T  volume      -

We are now ready to replicate the pool with the handy send/receive commands.

zfs send --replicate rpool@22-07-23_00-14 | pv | zfs receive -u -F proxmoxnew

--replicate - Generate a replication stream package, which will replicate the specified file system, and all descendant file systems, up to the named snapshot. When received, all properties, snapshots, descendant file systems, and clones are preserved.

-u - File system that is associated with the received stream is not mounted.

-F - Force a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication stream (for example, one generated by zfs send -R [-i|-I ]), destroy snapshots and file systems that do not exist on the sending side.

zfs list -o name,avail,type,mountpoint -t all
NAME                                          AVAIL  TYPE        MOUNTPOINT
proxmoxnew                                     509G  filesystem  /proxmoxnew
proxmoxnew@22-07-23_00-14                         -  snapshot    -
proxmoxnew/ROOT                                509G  filesystem  /proxmoxnew/ROOT
proxmoxnew/ROOT@22-07-23_00-14                    -  snapshot    -
proxmoxnew/ROOT/pve-1                          509G  filesystem  /
proxmoxnew/ROOT/pve-1@22-07-23_00-14              -  snapshot    -
proxmoxnew/data                                509G  filesystem  /proxmoxnew/data
proxmoxnew/data@22-07-23_00-14                    -  snapshot    -
proxmoxnew/data/vm-101-disk-0                  509G  volume      -
proxmoxnew/data/vm-101-disk-0@22-07-23_00-14      -  snapshot    -
proxmoxnew/swap                                517G  volume      -
proxmoxnew/swap@22-07-23_00-14                    -  snapshot    -
rpool                                          239G  filesystem  /rpool
rpool@22-07-23_00-14                              -  snapshot    -
rpool/ROOT                                     239G  filesystem  /rpool/ROOT
rpool/ROOT@22-07-23_00-14                         -  snapshot    -
rpool/ROOT/pve-1                               239G  filesystem  /
rpool/ROOT/pve-1@22-07-23_00-14                   -  snapshot    -
rpool/data                                     239G  filesystem  /rpool/data
rpool/data@22-07-23_00-14                         -  snapshot    -
rpool/data/vm-101-disk-0                       239G  volume      -
rpool/data/vm-101-disk-0@22-07-23_00-14           -  snapshot    -
rpool/swap                                     247G  volume      -
rpool/swap@22-07-23_00-14                         -  snapshot    -
temp1-pool                                    6.39T  filesystem  /temp1-pool
temp1-pool/vm-101-disk-0                      6.39T  volume      -

5) Booting From the New Installation

We need to set one the disks in the new pool as the boot device.

Proxmox initially failed to boot because we renamed the root pool to proxmoxnew and we should have exported it before rebooting the system.

Run the command below to rename the pool and forced it to be imported.

zpool import -f -N proxmoxnew rpool
exit
Proxmox failed to boot. initramfs

To avoid the the issues below. I highly recommend that you rename the old ZFS pool.

zpool import -N OLD_POOL_ID proxmoxold

And change the mount-point / to something else as showed below.

After, exiting the system booted perfectly into the mirrored new pool.

** ISSUE **

The only issue is that my scripts to set the network failed to load and I currently using IPMI to investigate what happened.

After an extensive investigation, I have managed to partially fix the issues with  my network scripts. It had nothing to do with the scripts themselves.

Because the name of the old ZFS pool was the same as the new proxmox was failing to boot complaining of not being able to import the rpool pool.

The old pool was renamed to proxmoxold with the command:

zpool import -N POOL_ID proxmoxnew
NAME                            USED  AVAIL     REFER  MOUNTPOINT
proxmoxold                     30.2G   239G      104K  /proxmoxold
proxmoxold/ROOT                11.9G   239G       96K  /proxmoxold/ROOT
proxmoxold/ROOT/pve-1          11.9G   239G     11.9G  /
proxmoxold/data                10.0G   239G       96K  /proxmoxold/data
proxmoxold/data/vm-101-disk-0  10.0G   239G     10.0G  -
proxmoxold/swap                8.25G   247G     1.04M  -
rpool                          30.1G   509G       25K  /rpool
rpool/ROOT                     11.8G   509G       24K  /rpool/ROOT
rpool/ROOT/pve-1               11.8G   509G     10.7G  /
rpool/data                     10.0G   509G       24K  /rpool/data
rpool/data/vm-101-disk-0       10.0G   509G     10.0G  -
rpool/swap                     8.25G   517G      929K  -
temp1-pool                     10.6G  6.39T     47.1K  /temp1-pool
temp1-pool/vm-101-disk-0       10.6G  6.39T     2.33G  -

After another manual import Proxmox booted. However, still with quite a few systemd units failing to load.

I then decided to migrate my installation to the proxmox boot tool following the documentation tutorial. It's been successful. However, this time the zpool import command complained that the pool appeared to be in use.

I have then forced the import and magically proxmox booted and all the systemd units that failed previously loaded apart from the zfs-mount.service unit file. The failure of this unit caused an error with local-zfs storage.

zpool import -f -N rpool

Another issue that after some investigation had an relatively easy fix. The old pool was mounted to /.

zfs get mountpoint proxmoxold/ROOT/pve-1
NAME                   PROPERTY    VALUE       SOURCE
proxmoxold/ROOT/pve-1  mountpoint  /           local

I have changed the mount-point, rebooted and voila. Everything is working as expected.

zfs set mountpoint=/proxmoxold/rootdir proxmoxold/ROOT/pve-1
zfs get mountpoint proxmoxold/ROOT/pve-1
NAME                   PROPERTY    VALUE                SOURCE
proxmoxold/ROOT/pve-1  mountpoint  /proxmoxold/rootdir  local

systemctl list-units --failed
0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

6) Upgrading Proxmox

I will follow the proxmox upgrade guide and the first step is to use their script and determine if you are ready to migrate.

pve6to7 --full
= SUMMARY =

TOTAL:    22
PASSED:   16
SKIPPED:  3
WARNINGS: 0
FAILURES: 0

My system has no warnings and is ready for the upgrade.

Adjust your package repositories to reflect the below.

cat /etc/apt/sources.list
deb http://ftp.uk.debian.org/debian bullseye main contrib

deb http://ftp.uk.debian.org/debian bullseye-updates main contrib

# security updates
deb http://security.debian.org bullseye-security main contrib
/etc/apt/sources.list
cat /etc/apt/sources.list.d/pve-no-subscription.list
# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
# Versions up to 6.4
#deb http://download.proxmox.com/debian/pve buster pve-no-subscription

# Upgrade to 7.x
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
/etc/apt/sources.list.d/pve-no-subscription.list

We need to run an apt update to get the new packages.

apt update
607 packages can be upgraded. Run 'apt list --upgradable' to see them.

| Before running the upgrade command make sure to read all the know issues from 6.4 up to the latest.

| You can also take a snapshot of your ZFS pool in case you need to restore to the previous 6.4 working state.

Finally, run the command below to upgrade.

apt dist-upgrade
You are attempting to upgrade from proxmox-ve '6.4-1' to proxmox-ve '7.2-1'.
Press enter to continue, or C^c to abort.

If the command exits successfully, you can reboot the system in order to use the new PVE kernel.

Conclusion

The process above described a migration of a custom ZFS on root proxmox running version 6.4 to the latest version at the time this article was written. I have learned a bit more about how ZFS and proxmox works. I hope it helps you as well.

Resources

Package Repositories - Proxmox VE

Share Tweet Send
0 Comments
Loading...
You've successfully subscribed to Infoitech - [B]logging
Great! Next, complete checkout for full access to Infoitech - [B]logging
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.