Ansible not working on RockyLinux 8, AlmaLinux 8, RHEL 8

[WARNING]: Unhandled error in Python interpreter discovery for host localhost: Expecting value: line 1
column 1 (char 0)

https://github.com/ansible/ansible/issues/83357

Ansible 2.17 moved to using Python 3.7. This causes issues with systems that use Python 3.6 (i.e., RHEL 8 based distros). Unfortunately, you can’t just upgrade Python either, as 3.6 is used in system tools such as DNF/YUM.

There are two options.

  1. Upgrade to a RHEL 9 based distribution
  2. Use Ansible 2.16

Ansible 2.16 should be the default installed version on RHEL 8 based distros.

Harden SSH for AlmaLinux 9 (RHEL, Fedora)

These steps are taken from the following link. They have other guides for hardening Ubuntu, Debian etc.

https://www.sshaudit.com/hardening_guides.html#rocky9

You will need to become the root user, use either su – or sudo -i

First we need to regenerate the RSA and ED25519 keys

rm /etc/ssh/ssh_host_*
ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_rsa_key -N ""

Next, remove the small Diffie-Hellman moduli. The moduli file contains prime numbers and generators. Removing the smaller numbers should help increase security as it makes attempting to factor the private keys harder.

awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.safe
mv /etc/ssh/moduli.safe /etc/ssh/moduli

We can now specify which key exchange, ciphers, and algorithms to use.

Add the following to “/etc/crypto-policies/back-ends/opensshserver.config”

# Restrict key exchange, cipher, and MAC algorithms, as per sshaudit.com
# hardening guide.
KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,gss-curve25519-sha256-,diffie-hellman-group16-sha512,gss-group16-sha512-,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com

HostKeyAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

RequiredRSASize 3072

CASignatureAlgorithms sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

GSSAPIKexAlgorithms gss-curve25519-sha256-,gss-group16-sha512-

HostbasedAcceptedAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-256

PubkeyAcceptedAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

Finally, restart the ssh server

systemctl restart sshd

Other helpful links

https://www.ssh.com/academy

https://www.redhat.com/en/blog/primes-parameters-and-moduli

https://security.stackexchange.com/questions/79043/is-it-considered-worth-it-to-replace-opensshs-moduli-file

Install Node.js 18 on AlmaLinux 8

List available Node.js versions available.

dnf module list nodejs
AlmaLinux 8 - AppStream
Name           Stream           Profiles                                     Summary
nodejs         10 [d][x]        common [d], development, minimal, s2i        Javascript runtime
nodejs         12 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         14 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         16 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         18 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         20 [x]           common [d], development, minimal, s2i        Javascript runtime

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

As we can see above, nodejs 18 is disabled. Enable it with

sudo dnf module enable nodejs:18

Now we can install with

sudo dnf install nodejs

You may need to uninstall older versions.

Ansible Playbook to upgrade Linux Servers (Debian, Ubuntu, RedHat, Fedora, CentOS)

This is an Ansible playbook that can upgrade all your Linux machines! Or at least most of them. No openSUSE support yet.

Copy the playbook below, and put all your servers into an inventory file and run with

ansible-playbook -i hosts.ini master_update.yaml --ask-vault-pass

Couple of notes.

  1. This will do a full update automatically reboot your servers if needed.
  2. There is a special section for RHEL, CentOS 7 servers. If a server is running say CentOS 7, it will default to using YUM instead of DNF.
  3. You need sudo or become: yes to reboot and install upgrades.

Linux OS Upgrade Playbook

---
- name: Linux OS Upgrade
  hosts: all
  gather_facts: yes
  become: yes

  tasks:
    - name: Upgrade Debian and Ubuntu systems with apt
      block: 
        - name: dist-upgrade
          ansible.builtin.apt:
            upgrade: dist
            update_cache: yes 
          register: upgrade_result

        - name: Debain check if reboot is required
          shell: "[ -f /var/run/reboot-required ]"
          failed_when: False
          register: debian_reboot_required
          changed_when: debian_reboot_required.rc == 0
          notify:
            - Reboot server 

        - name: Debian remove unneeded dependencies
          ansible.builtin.apt:
            autoremove: yes
          register: autoremove_result 

        - name: Debian print errors if upgrade failed
          ansible.builtin.debug:
            msg: | 
              Upgrade Result: {{ upgrade_result }}
              Autoremove Result: {{ autoremove_result }}
      when: ansible_os_family == "Debian"
    
    - name: Upgrade RHEL systems with DNF
      block:
        - name: Get packages that can be upgraded with DNF
          ansible.builtin.dnf:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_dnf_output_all

        - name: List packages that can be upgraded with DNF
          ansible.builtin.debug: 
            msg: "{{ reg_dnf_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with DNF
          become: yes
          ansible.builtin.dnf:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_upgrade_ok

        - name: Print DNF errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_upgrade_ok is not defined

        - name: Install dnf-utils
          become: yes
          ansible.builtin.dnf:
            name: 'dnf-utils'
            state: latest
            update_cache: yes
          when: reg_dnf_output_all is defined

      when: ansible_os_family == "RedHat" and not (ansible_distribution_major_version == "7")

    - name: Upgrade legacy RHEL systems with YUM
      block:
        - name: Get packages that can be upgraded with YUM
          ansible.builtin.yum:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_yum_output_all
            

        - name: List packages that can be upgraded with YUM
          ansible.builtin.debug: 
            msg: "{{ reg_yum_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with YUM
          become: yes
          ansible.builtin.yum:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_yum_upgrade_ok

        - name: Print YUM errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_yum_upgrade_ok is not defined
            
        - name: Check legacy RHEL system if a reboot is required
          become: yes
          command: needs-restarting -r
          register: reg_reboot_required
          ignore_errors: yes
          failed_when: false
          changed_when: reg_reboot_required.rc != 0
          notify:
            - Reboot server 
      when: ansible_os_family == "RedHat" and ansible_distribution_major_version == "7"


  handlers:
    - name : Reboot server
      ansible.builtin.reboot:
        msg: "Reboot initiated by Ansible after OS update"
        reboot_timeout: 3600
        test_command: uptime

Helpful links

https://github.com/simeononsecurity/ansible_linux_update/tree/main
https://simeononsecurity.com/guides/automate-linux-patching-and-updates-with-ansible/
https://thenathan.net/2020/07/16/yum-and-dnf-update-and-reboot-with-ansible/

More space needed on the /boot filesystem. RHEL / Fedora / Alma / Rocky

Error Summary
-------------
Disk Requirements:
   At least 28MB more space needed on the /boot filesystem.

The above error is due to the /boot partition being out of space. We can fix this issue by removing older unused Linux kernels. You could also increase the disk space, but that is a little more involved.

First we need to list which kernels we have installed.

rpm -qa | grep kernel

Example output

[incredigeek@apache ~]$ rpm -qa | grep kernel
kernel-core-4.18.0-522.el8.x86_64
kernel-tools-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-526.el8.x86_64
kernel-4.18.0-526.el8.x86_64
kernel-modules-4.18.0-529.el8.x86_64
kernel-4.18.0-522.el8.x86_64
kernel-4.18.0-529.el8.x86_64
kernel-core-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-522.el8.x86_64
kernel-core-4.18.0-526.el8.x86_64
kernel-devel-4.18.0-529.el8.x86_64
kernel-tools-libs-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-526.el8.x86_64
kernel-headers-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-522.el8.x86_64

The kernel in bold is the one we will remove.

Next we remove erase the old kernel(s)/items.

sudo rpm -e kernel-4.18.0-522.el8.x86_64 kernel-core-4.18.0-522.el8.x86_64 kernel-devel-4.18.0-522.el8.x86_64 kernel-modules-4.18.0-522.el8.x86_64

And now we continue with our update

sudo dnf update

Helpful links.

https://www.cyberciti.biz/faq/installing-kernel-2-6-32-131-2-1-el6-x86_64-needs-8mb-on-boot-filesystem/