Migrate CentOS 7 to AlmaLinux 8

The steps are taken from this page https://wiki.almalinux.org/elevate/ELevating-CentOS7-to-AlmaLinux-9.html

To upgrade to AlmaLinux 9, you will need to migrate to AlmaLinux 8 first.

sudo curl -o /etc/yum.repos.d/CentOS-Base.repo https://el7.repo.almalinux.org/centos/CentOS-Base.repo
sudo yum upgrade -y

Once yum finishes, reboot

sudo reboot

Now install elevate-release and leapp packages

sudo yum install -y http://repo.almalinux.org/elevate/elevate-release-latest-el$(rpm --eval %rhel).noarch.rpm
sudo yum install -y leapp-upgrade leapp-data-almalinux

Let’s run a pre upgrade check to see if there are any errors.

sudo leapp preupgrade

If everything checked out ok, proceed with the upgrade.

sudo leapp upgrade

Once finished, reboot.

sudo reboot

Once the system is booted, verify it upgraded.

cat /etc/*release

Problems

https://wiki.almalinux.org/elevate/ELevate-frequent-issues

LUKS

If you are using LUKS and encounter an error, check out the following link.

https://www.it-hure.de/2024/02/update-alma-rhel-with-leapp-and-luks/

You can disable the check with the following command.

rm -rf /usr/share/leapp-repository/repositories/system_upgrade/common/actors/inhibitwhenluks

More space needed on / filesystem

Try setting LEAPP_OVL_SIZE to 4096

export LEAPP_OVL_SIZE=4096

Then launch the upgrade with

sudo --preserve-env leapp upgrade

https://forums.almalinux.org/t/at-least-48mb-more-space-needed-on-the-filesystem-running-leapp-upgrade/3808/2

You can also try removing files to create more free space.

https://github.com/oamg/leapp/issues/778

Upgrading to 8.8 (Or 8.4)

For the same reason, we recommend upgrading your CentOS 7 machine to AlmaLinux OS version 8.8. To do so, you need to navigate to the /etc/leapp/files/ directory and edit the leapp_upgrade_repositories.repo to lower the AlmaLinux version in baseurl/mirror to 8.8.

The 8.8 repositories are archived. To upgrade to AlmaLinux 8.8, change ‘/etc/leapp/leapp_upgrade_repositories.repo” to the following.

[almalinux8-BaseOS]
name=AlmaLinux 8 - BaseOS
baseurl=https://vault.almalinux.org/8.8/BaseOS/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-AppStream]
name=AlmaLinux 8 - AppStream
baseurl=https://vault.almalinux.org/8.8/AppStream/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-PowerTools]
name=AlmaLinux 8 - PowerTools
baseurl=https://vault.almalinux.org/8.8/PowerTools/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-HighAvailability]
name=AlmaLinux 8 - HighAvailability
baseurl=https://vault.almalinux.org/8.8/HighAvailability/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-ResilientStorage]
name=AlmaLinux 8 - ResilientStorage
baseurl=https://vault.almalinux.org/8.8/ResilientStorage/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

[almalinux8-Extras]
name=AlmaLinux 8 - Extras
baseurl=https://vault.almalinux.org/8.8/extras/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-AlmaLinux

If you have issues, you may consider upgrading to 8.4 first. To do that, change 8.8 to 8.4, and comment out the ResilientStorage section.

GPG Key failing after upgrade

Try importing the AlmaLinux 8 GPG key

rpm --import https://repo.almalinux.org/almalinux/RPM-GPG-KEY-AlmaLinux

https://almalinux.org/blog/2023-12-20-almalinux-8-key-update

Harden SSH for AlmaLinux 9 (RHEL, Fedora)

These steps are taken from the following link. They have other guides for hardening Ubuntu, Debian etc.

https://www.sshaudit.com/hardening_guides.html#rocky9

You will need to become the root user, use either su – or sudo -i

First we need to regenerate the RSA and ED25519 keys

rm /etc/ssh/ssh_host_*
ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_rsa_key -N ""

Next, remove the small Diffie-Hellman moduli. The moduli file contains prime numbers and generators. Removing the smaller numbers should help increase security as it makes attempting to factor the private keys harder.

awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.safe
mv /etc/ssh/moduli.safe /etc/ssh/moduli

We can now specify which key exchange, ciphers, and algorithms to use.

Add the following to “/etc/crypto-policies/back-ends/opensshserver.config”

# Restrict key exchange, cipher, and MAC algorithms, as per sshaudit.com
# hardening guide.
KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,gss-curve25519-sha256-,diffie-hellman-group16-sha512,gss-group16-sha512-,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com

HostKeyAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

RequiredRSASize 3072

CASignatureAlgorithms sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

GSSAPIKexAlgorithms gss-curve25519-sha256-,gss-group16-sha512-

HostbasedAcceptedAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-256

PubkeyAcceptedAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

Finally, restart the ssh server

systemctl restart sshd

Other helpful links

https://www.ssh.com/academy

https://www.redhat.com/en/blog/primes-parameters-and-moduli

https://security.stackexchange.com/questions/79043/is-it-considered-worth-it-to-replace-opensshs-moduli-file

Migrate CentOS 8 Stream to AlmaLinux 8

https://wiki.almalinux.org/documentation/migration-guide

Update CentOS 8 Stream

sudo dnf update -y

Download and run the almalinux-deploy script

curl -O https://raw.githubusercontent.com/AlmaLinux/almalinux-deploy/master/almalinux-deploy.sh
sudo bash almalinux-deploy.sh -d

You’ll need to run with the -d “downgrade” option if you are migrating from CentOS 8 Stream. https://github.com/AlmaLinux/almalinux-deploy/tree/master?tab=readme-ov-file#roadmap

You may need to remove packages if there are conflicts. On one instance, there were issues and I needed to remove grafana and llvm-compat-libs.

sudo yum remove grafana llvm-compat-libs

After those errors are fixed, rerun.

sudo bash almalinux-deploy.sh -d

Once the script finishes

sudo reboot

Once it comes back up, check the Linux version

cat /etc/*release

Example output

AlmaLinux release 8.9 (Midnight Oncilla)
AlmaLinux release 8.9 (Midnight Oncilla)
NAME="AlmaLinux"
VERSION="8.9 (Midnight Oncilla)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.9 (Midnight Oncilla)"

Install Node.js 18 on AlmaLinux 8

List available Node.js versions available.

dnf module list nodejs
AlmaLinux 8 - AppStream
Name           Stream           Profiles                                     Summary
nodejs         10 [d][x]        common [d], development, minimal, s2i        Javascript runtime
nodejs         12 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         14 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         16 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         18 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         20 [x]           common [d], development, minimal, s2i        Javascript runtime

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

As we can see above, nodejs 18 is disabled. Enable it with

sudo dnf module enable nodejs:18

Now we can install with

sudo dnf install nodejs

You may need to uninstall older versions.

How To Check if RHEL/AlmaLinux needs a reboot after an update

Typically you’ll need to reboot a server after an update if the Linux Kernel was updated. It is possible that services need to be restarted.

There is some good information here https://serverfault.com/questions/122178/how-can-i-check-from-the-command-line-if-a-reboot-is-required-on-rhel-or-centos

Using Yum Utilities needs-restarting

Install the needs-restarting utility

sudo dnf install -y yum-utils

Once installed, we can check if we need to reboot with

sudo needs-restarting -r

The -r option only reports if a reboot is required.

If we wanted to automatically check and reboot, we could do

sudo needs-restarting -r || sudo shutdown -r

Alternative way

We could alternatively just check the kernel version and if it is different, manually reboot the machine. Note that there could be a couple cases where the kernel didn’t update, but you still need a reboot, or services needed to be restarted View links below for more information.

LAST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

test $LAST_KERNEL = $CURRENT_KERNEL || shutdown -r

How to Undelete Files on XFS Filesystem

There are a couple different options for undeleting files for XFS filesystems.

TestDisk

TestDisk is a great command line recovery tool. Unfortunately, it can be slightly more difficult on systems using XFS compared to EXT4 systems. TestDisk does not support undeleting a file in place on XFS.

You can still recover files using TestDisk, you just need to recover the whole drive and dig through the recovery results to find the files you want.

xfs_undelete

There is also another utility that can be helpful. xfs_undelete

https://github.com/ianka/xfs_undelete

It allows for a little more flexibility in recovering files. For instance, you can specify to recover the files from the past hour to recover.

Download prerequisites

sudo dnf install tcllib
wget https://raw.githubusercontent.com/ianka/xfs_undelete/master/xfs_undelete
chmod u+x ./xfs_undelete

./xfs_undelete

Example of running xfs_undelete

./xfs_undelete -t -1hour ./dev/sda2

You will need a different filesystem to save the files to. Otherwise you will receive the following error.

Your output directory is  /home/bob/recovery/
That is within the filesystem  /  you want to recover files
from. This isn't feasible as it would overwrite the deleted files you wanted to
recover. Please specify the option -o /path/to/output_directory on another (rw
mounted) filesystem or run xfs_undelete from within a directory on that
filesystem so the recovered files could be written there. They cannot be
recovered in place.

It’s not the greatest idea to recover on the system while running. Ideally, shut the system down, plug the drive into another machine as read only, and copy the files off.

You could also boot up in single user mode or a live Linux iso/thumbdrive and mount another recovery drive. Should work for both physical and virtual environments.

Installing Basic Linux tools on AlmaLinux 9 (tar, wget, htop)

The local team wizard Mark, ran into some issues while trying to setup a system with AlmaLinux 9. Tar wasn’t installed! What?! No worries. We can solve this by just installing tar with dnf. While we are at it, lets install some other helpful utilities.

sudo dnf install -y tar wget htop

Tada! We are back in business.