Ansible not working on RockyLinux 8, AlmaLinux 8, RHEL 8

[WARNING]: Unhandled error in Python interpreter discovery for host localhost: Expecting value: line 1
column 1 (char 0)

https://github.com/ansible/ansible/issues/83357

Ansible 2.17 moved to using Python 3.7. This causes issues with systems that use Python 3.6 (i.e., RHEL 8 based distros). Unfortunately, you can’t just upgrade Python either, as 3.6 is used in system tools such as DNF/YUM.

There are two options.

  1. Upgrade to a RHEL 9 based distribution
  2. Use Ansible 2.16

Ansible 2.16 should be the default installed version on RHEL 8 based distros.

Harden SSH for AlmaLinux 9 (RHEL, Fedora)

These steps are taken from the following link. They have other guides for hardening Ubuntu, Debian etc.

https://www.sshaudit.com/hardening_guides.html#rocky9

You will need to become the root user, use either su – or sudo -i

First we need to regenerate the RSA and ED25519 keys

rm /etc/ssh/ssh_host_*
ssh-keygen -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key -N ""
ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_rsa_key -N ""

Next, remove the small Diffie-Hellman moduli. The moduli file contains prime numbers and generators. Removing the smaller numbers should help increase security as it makes attempting to factor the private keys harder.

awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.safe
mv /etc/ssh/moduli.safe /etc/ssh/moduli

We can now specify which key exchange, ciphers, and algorithms to use.

Add the following to “/etc/crypto-policies/back-ends/opensshserver.config”

# Restrict key exchange, cipher, and MAC algorithms, as per sshaudit.com
# hardening guide.
KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,gss-curve25519-sha256-,diffie-hellman-group16-sha512,gss-group16-sha512-,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256

Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr

MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com

HostKeyAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

RequiredRSASize 3072

CASignatureAlgorithms sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

GSSAPIKexAlgorithms gss-curve25519-sha256-,gss-group16-sha512-

HostbasedAcceptedAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-256

PubkeyAcceptedAlgorithms sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,sk-ssh-ed25519@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256

Finally, restart the ssh server

systemctl restart sshd

Other helpful links

https://www.ssh.com/academy

https://www.redhat.com/en/blog/primes-parameters-and-moduli

https://security.stackexchange.com/questions/79043/is-it-considered-worth-it-to-replace-opensshs-moduli-file

Configuring Firewalld with Ansible

We’ll be using Ansible to change and maintain our firewall settings on a server.

The playbook will do the following.

  1. Set the default zone to drop (Drops all external traffic to server)
  2. Set a zone for internal access
  3. Allow access from RFC1918 addresses to internal zone (Any local IP address will be able to access the server)
  4. Enable the services and ports specified in the vars section
  5. Disable the services listed in firewall_disable_services variable

Modify the variables as needed for your server(s). You can also add or move the variables to the inventory or host_vars files.

If you need to create an inventory file, refer to the first part of this post

BE CAREFUL CHANGING FIREWALL SETTINGS!!! IMPROPER SETTINGS COULD RENDER THE SERVER INACCESSIBLE!!!

Playbook for firewalld

Change the variables under the vars section

---
- name: Configure firewalld
  hosts: rhel
  gather_facts: yes
  become: yes

  vars: 
    firewall_allowed_ips:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 192.168.0.0/16
    firewall_allowed_services:
      - ssh
      - https
      - snmp
    firewall_allowed_ports:
      - "2222/tcp"
    firewall_disable_services:
      - cockpit
      - dhcpv6-client
      - mdns
      - samba-client

  tasks: 
  - name: Set default zone to drop
    ansible.builtin.command: firewall-cmd --set-default-zone=drop
    register: default_zone_set
    changed_when:
      - '"ZONE_ALREADY_SET" not in default_zone_set.stderr'

  - name: Enable and allow access to internal zone from RFC1918 addresses
    ansible.posix.firewalld:
      source: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: enabled
    with_items: "{{ firewall_allowed_ips }}"

  - name: Disable unused services for internal zone
    ansible.posix.firewalld:
      service: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: disabled
    with_items: "{{ firewall_disable_services }}"


  - name: Set services for internal zone
    ansible.posix.firewalld:
      service: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: enabled
    with_items: "{{ firewall_allowed_services }}"

  - name: Set custom ports for internal zone
    ansible.posix.firewalld:
      port: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: enabled
    with_items: "{{ firewall_allowed_ports }}"

Helpful links

https://docs.ansible.com/ansible/latest/collections/ansible/posix/firewalld_module.html#parameter-source

https://stackoverflow.com/questions/51563643/how-to-change-firewalld-zone-using-ansible

https://www.middlewareinventory.com/blog/ansible-firewalld/

Install Node.js 18 on AlmaLinux 8

List available Node.js versions available.

dnf module list nodejs
AlmaLinux 8 - AppStream
Name           Stream           Profiles                                     Summary
nodejs         10 [d][x]        common [d], development, minimal, s2i        Javascript runtime
nodejs         12 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         14 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         16 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         18 [x]           common [d], development, minimal, s2i        Javascript runtime
nodejs         20 [x]           common [d], development, minimal, s2i        Javascript runtime

Hint: [d]efault, [e]nabled, [x]disabled, [i]nstalled

As we can see above, nodejs 18 is disabled. Enable it with

sudo dnf module enable nodejs:18

Now we can install with

sudo dnf install nodejs

You may need to uninstall older versions.

Ansible Playbook to upgrade Linux Servers (Debian, Ubuntu, RedHat, Fedora, CentOS)

This is an Ansible playbook that can upgrade all your Linux machines! Or at least most of them. No openSUSE support yet.

Copy the playbook below, and put all your servers into an inventory file and run with

ansible-playbook -i hosts.ini master_update.yaml --ask-vault-pass

Couple of notes.

  1. This will do a full update automatically reboot your servers if needed.
  2. There is a special section for RHEL, CentOS 7 servers. If a server is running say CentOS 7, it will default to using YUM instead of DNF.
  3. You need sudo or become: yes to reboot and install upgrades.

Linux OS Upgrade Playbook

---
- name: Linux OS Upgrade
  hosts: all
  gather_facts: yes
  become: yes

  tasks:
    - name: Upgrade Debian and Ubuntu systems with apt
      block: 
        - name: dist-upgrade
          ansible.builtin.apt:
            upgrade: dist
            update_cache: yes 
          register: upgrade_result

        - name: Debain check if reboot is required
          shell: "[ -f /var/run/reboot-required ]"
          failed_when: False
          register: debian_reboot_required
          changed_when: debian_reboot_required.rc == 0
          notify:
            - Reboot server 

        - name: Debian remove unneeded dependencies
          ansible.builtin.apt:
            autoremove: yes
          register: autoremove_result 

        - name: Debian print errors if upgrade failed
          ansible.builtin.debug:
            msg: | 
              Upgrade Result: {{ upgrade_result }}
              Autoremove Result: {{ autoremove_result }}
      when: ansible_os_family == "Debian"
    
    - name: Upgrade RHEL systems with DNF
      block:
        - name: Get packages that can be upgraded with DNF
          ansible.builtin.dnf:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_dnf_output_all

        - name: List packages that can be upgraded with DNF
          ansible.builtin.debug: 
            msg: "{{ reg_dnf_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with DNF
          become: yes
          ansible.builtin.dnf:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_upgrade_ok

        - name: Print DNF errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_upgrade_ok is not defined

        - name: Install dnf-utils
          become: yes
          ansible.builtin.dnf:
            name: 'dnf-utils'
            state: latest
            update_cache: yes
          when: reg_dnf_output_all is defined

      when: ansible_os_family == "RedHat" and not (ansible_distribution_major_version == "7")

    - name: Upgrade legacy RHEL systems with YUM
      block:
        - name: Get packages that can be upgraded with YUM
          ansible.builtin.yum:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_yum_output_all
            

        - name: List packages that can be upgraded with YUM
          ansible.builtin.debug: 
            msg: "{{ reg_yum_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with YUM
          become: yes
          ansible.builtin.yum:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_yum_upgrade_ok

        - name: Print YUM errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_yum_upgrade_ok is not defined
            
        - name: Check legacy RHEL system if a reboot is required
          become: yes
          command: needs-restarting -r
          register: reg_reboot_required
          ignore_errors: yes
          failed_when: false
          changed_when: reg_reboot_required.rc != 0
          notify:
            - Reboot server 
      when: ansible_os_family == "RedHat" and ansible_distribution_major_version == "7"


  handlers:
    - name : Reboot server
      ansible.builtin.reboot:
        msg: "Reboot initiated by Ansible after OS update"
        reboot_timeout: 3600
        test_command: uptime

Helpful links

https://github.com/simeononsecurity/ansible_linux_update/tree/main
https://simeononsecurity.com/guides/automate-linux-patching-and-updates-with-ansible/
https://thenathan.net/2020/07/16/yum-and-dnf-update-and-reboot-with-ansible/

More space needed on the /boot filesystem. RHEL / Fedora / Alma / Rocky

Error Summary
-------------
Disk Requirements:
   At least 28MB more space needed on the /boot filesystem.

The above error is due to the /boot partition being out of space. We can fix this issue by removing older unused Linux kernels. You could also increase the disk space, but that is a little more involved.

First we need to list which kernels we have installed.

rpm -qa | grep kernel

Example output

[incredigeek@apache ~]$ rpm -qa | grep kernel
kernel-core-4.18.0-522.el8.x86_64
kernel-tools-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-526.el8.x86_64
kernel-4.18.0-526.el8.x86_64
kernel-modules-4.18.0-529.el8.x86_64
kernel-4.18.0-522.el8.x86_64
kernel-4.18.0-529.el8.x86_64
kernel-core-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-522.el8.x86_64
kernel-core-4.18.0-526.el8.x86_64
kernel-devel-4.18.0-529.el8.x86_64
kernel-tools-libs-4.18.0-529.el8.x86_64
kernel-devel-4.18.0-526.el8.x86_64
kernel-headers-4.18.0-529.el8.x86_64
kernel-modules-4.18.0-522.el8.x86_64

The kernel in bold is the one we will remove.

Next we remove erase the old kernel(s)/items.

sudo rpm -e kernel-4.18.0-522.el8.x86_64 kernel-core-4.18.0-522.el8.x86_64 kernel-devel-4.18.0-522.el8.x86_64 kernel-modules-4.18.0-522.el8.x86_64

And now we continue with our update

sudo dnf update

Helpful links.

https://www.cyberciti.biz/faq/installing-kernel-2-6-32-131-2-1-el6-x86_64-needs-8mb-on-boot-filesystem/

How To Check if RHEL/AlmaLinux needs a reboot after an update

Typically you’ll need to reboot a server after an update if the Linux Kernel was updated. It is possible that services need to be restarted.

There is some good information here https://serverfault.com/questions/122178/how-can-i-check-from-the-command-line-if-a-reboot-is-required-on-rhel-or-centos

Using Yum Utilities needs-restarting

Install the needs-restarting utility

sudo dnf install -y yum-utils

Once installed, we can check if we need to reboot with

sudo needs-restarting -r

The -r option only reports if a reboot is required.

If we wanted to automatically check and reboot, we could do

sudo needs-restarting -r || sudo shutdown -r

Alternative way

We could alternatively just check the kernel version and if it is different, manually reboot the machine. Note that there could be a couple cases where the kernel didn’t update, but you still need a reboot, or services needed to be restarted View links below for more information.

LAST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

test $LAST_KERNEL = $CURRENT_KERNEL || shutdown -r

Using Auditd to monitor changes to Linux

Install and enable auditd with

sudo dnf install auditd
sudo systemctl enable auditd
sudo systemctl start auditd

Add a file or directory to monitor with

auditctl -w /etc/passwd -k password

-w is watch path
-k is a filter key we can use later to search through logs

Now we can search with ausearch

ausearch -k password

Using Preconfigured Rules

There are already some preconfigured rules in /usr/share/audit/sample-rules/

We can copy those to /etc/auditd/rules.d/ and use them.

cd /usr/share/audit/sample-rules/
cp 10-base-config.rules 30-stig.rules 31-privileged.rules 99-finalize.rules /etc/audit/rules.d/
augenrules --load

Note on the 31-privileged.rules file. You’ll need to run the commands in the file which will create a new file. Then we can copy that to “/etc/auditd/rules.d/”

find /bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' > priv.rules
#find /sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/bin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#find /usr/sbin -type f -perm -04000 2>/dev/null | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $1 }' >> priv.rules
#filecap /bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/bin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules
#filecap /usr/sbin 2>/dev/null | sed '1d' | awk '{ printf "-a always,exit -F path=%s -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged\n", $2 }' >> priv.rules

And Copy priv.rules to /etc/audit/rules.d/31-privileged.rules. Overwrite the file there if needed.

cp ./priv.rules /etc/audit/rules.d/31-privileged.rules

Load the rules.

augenrules --load

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/auditing-the-system_security-hardening

How to Undelete Files on XFS Filesystem

There are a couple different options for undeleting files for XFS filesystems.

TestDisk

TestDisk is a great command line recovery tool. Unfortunately, it can be slightly more difficult on systems using XFS compared to EXT4 systems. TestDisk does not support undeleting a file in place on XFS.

You can still recover files using TestDisk, you just need to recover the whole drive and dig through the recovery results to find the files you want.

xfs_undelete

There is also another utility that can be helpful. xfs_undelete

https://github.com/ianka/xfs_undelete

It allows for a little more flexibility in recovering files. For instance, you can specify to recover the files from the past hour to recover.

Download prerequisites

sudo dnf install tcllib
wget https://raw.githubusercontent.com/ianka/xfs_undelete/master/xfs_undelete
chmod u+x ./xfs_undelete

./xfs_undelete

Example of running xfs_undelete

./xfs_undelete -t -1hour ./dev/sda2

You will need a different filesystem to save the files to. Otherwise you will receive the following error.

Your output directory is  /home/bob/recovery/
That is within the filesystem  /  you want to recover files
from. This isn't feasible as it would overwrite the deleted files you wanted to
recover. Please specify the option -o /path/to/output_directory on another (rw
mounted) filesystem or run xfs_undelete from within a directory on that
filesystem so the recovered files could be written there. They cannot be
recovered in place.

It’s not the greatest idea to recover on the system while running. Ideally, shut the system down, plug the drive into another machine as read only, and copy the files off.

You could also boot up in single user mode or a live Linux iso/thumbdrive and mount another recovery drive. Should work for both physical and virtual environments.

Hardening SNMP on CentOS/RedHat/Fedora Etc.

These steps should be similar across Red Hat type distros.

Before we proceed, lets stop SNMP

sudo systemctl stop snmpd

Disable SNMP Versions 1 and 2c

First we are going to disable SNMP v1 and v2c

You can manually edit the /etc/snmp/snmpd.conf file and comment out or delete every line starting with com2sec, group, access. Or you can run the following sed commands to change it for you.

sudo sed -i 's/^com2sec/# com2sec/g' /etc/snmp/snmpd.conf
sudo sed -i 's/^group/# group/g' /etc/snmp/snmpd.conf
sudo sed -i 's/^access/# access/g' /etc/snmp/snmpd.conf

https://serverfault.com/questions/376688/how-to-disable-version-1-and-version-2c-in-snmpd

Create SNMP Version 3 User

Follow the prompts to create a SNMP v3 user.

sudo net-snmp-create-v3-user -ro -a SHA -x AES

Start SNMP

sudo systemctl start snmpd

You should be good to go.

If you are running a firewall, you will need to allow an exception for SNMP, UDP port 161. You may also need to allow an SELinux exception. Check out the last portion of both these articles.

Allowing SNMP Through Firewall