Configuring Firewalld with Ansible

We’ll be using Ansible to change and maintain our firewall settings on a server.

The playbook will do the following.

  1. Set the default zone to drop (Drops all external traffic to server)
  2. Set a zone for internal access
  3. Allow access from RFC1918 addresses to internal zone (Any local IP address will be able to access the server)
  4. Enable the services and ports specified in the vars section
  5. Disable the services listed in firewall_disable_services variable

Modify the variables as needed for your server(s). You can also add or move the variables to the inventory or host_vars files.

If you need to create an inventory file, refer to the first part of this post

BE CAREFUL CHANGING FIREWALL SETTINGS!!! IMPROPER SETTINGS COULD RENDER THE SERVER INACCESSIBLE!!!

Playbook for firewalld

Change the variables under the vars section

---
- name: Configure firewalld
  hosts: rhel
  gather_facts: yes
  become: yes

  vars: 
    firewall_allowed_ips:
      - 10.0.0.0/8
      - 172.16.0.0/12
      - 192.168.0.0/16
    firewall_allowed_services:
      - ssh
      - https
      - snmp
    firewall_allowed_ports:
      - "2222/tcp"
    firewall_disable_services:
      - cockpit
      - dhcpv6-client
      - mdns
      - samba-client

  tasks: 
  - name: Set default zone to drop
    ansible.builtin.command: firewall-cmd --set-default-zone=drop
    register: default_zone_set
    changed_when:
      - '"ZONE_ALREADY_SET" not in default_zone_set.stderr'

  - name: Enable and allow access to internal zone from RFC1918 addresses
    ansible.posix.firewalld:
      source: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: enabled
    with_items: "{{ firewall_allowed_ips }}"

  - name: Disable unused services for internal zone
    ansible.posix.firewalld:
      service: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: disabled
    with_items: "{{ firewall_disable_services }}"


  - name: Set services for internal zone
    ansible.posix.firewalld:
      service: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: enabled
    with_items: "{{ firewall_allowed_services }}"

  - name: Set custom ports for internal zone
    ansible.posix.firewalld:
      port: "{{ item }}"
      zone: internal
      permanent: true
      immediate: true
      state: enabled
    with_items: "{{ firewall_allowed_ports }}"

Helpful links

https://docs.ansible.com/ansible/latest/collections/ansible/posix/firewalld_module.html#parameter-source

https://stackoverflow.com/questions/51563643/how-to-change-firewalld-zone-using-ansible

https://www.middlewareinventory.com/blog/ansible-firewalld/

Ansible Playbook for Linux SNMP

This playbook is for installing and configuring SNMP on Ubuntu or RedHat machines

Change the snmp_location and snmp_contact etc. variables. Or define them in the inventory file, or pass them in as –extra-vars. –extra-vars=”snmp_location=’location address’ snmpv3_user=incredigeek …etc”

Couple of notes

  • We check to see if a read only SNMPv3 user has been created. If so, we don’t create a new one.
  • The snmpd service is stopped and started each time this is run
  • You will still need to allow SNMP through the firewall. Ubuntu or Fedora
---
- name: Linux SNMP Config
  hosts: all
  gather_facts: yes
  become: yes

# Install SNMPv3 on RHEL or Debian/Ubuntu
# Disable SNMP v1 and v2 on RHEL
# Configure SNMPv3 user

  vars:

    # Change these!
    snmp_location: My SNMP location
    snmp_contact: My SNMP contact info
    snmpv3_pass: mypassword
    snmpv3_user: incredigeek

    # These are used to disable the default public community.
    cmnt: '#'
    cmnt_lines:
      - com2sec notConfigUser
      - group   notConfigGroup
      - view    systemview
      - access  notConfigGroup 


  tasks: 

    - name: Check if SNMPv3 user exists
      ansible.builtin.lineinfile:
        path: /etc/snmp/snmpd.conf
        regexp: '^rouser'
        state: absent
      check_mode: yes
      changed_when: false
      register: snmpv3_user_exists

    - name: Stop SNMPD Service
      ansible.builtin.service:
        name: snmpd
        state: stopped

    - name: RHEL SNMP Config
      block:
        - name: Install SNMP RHEL
          ansible.builtin.dnf:
            name:
              - net-snmp
              - net-snmp-utils
            state: present

        - name: Disable public snmp community RHEL
          replace:
            path: /etc/snmp/snmpd.conf
            regexp: '^{{ item }}'
            replace: '{{ cmnt }} {{ item }}'
          loop: "{{ cmnt_lines }}"
        - name: Set SNMP Location
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^syslocation.*'
            line: "syslocation {{ snmp_location }}"

        - name: Set SNMP Contact
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^syscontact.*'
            line: "syscontact {{ snmp_contact }}"
        - name: Setup SNMPv3 user for RHEL
          shell: net-snmp-create-v3-user -ro -a SHA -A '{{ snmpv3_pass }}' -x '{{ snmpv3_pass }}' -X AES {{ snmpv3_user }}
          when: not snmpv3_user_exists.found

      when: ansible_os_family == "RedHat"

    - name: Debian SNMP Config
      block:
        - name: Install SNMP on Debian
          ansible.builtin.apt:
            pkg:
            - snmp
            - snmpd
            - libsnmp-dev

        - name: Modify available from address
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^agentAddress udp:127\.0\.0\.1:161'
            line: 'agentAddress udp:161,udp6:[::1]:161'

        - name: Set SNMP Location
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^sysLocation.*'
            line: "sysLocation {{ snmp_location }}"

        - name: Set SNMP Contact
          ansible.builtin.lineinfile:
            path: /etc/snmp/snmpd.conf
            regexp: '^sysContact.*'
            line: "sysContact {{ snmp_contact }}"
        - name: Setup SNMPv3 user for Debian
          shell: net-snmp-config --create-snmpv3-user -ro -a SHA -A '{{ snmpv3_pass }}' -x '{{ snmpv3_pass }}' -X AES {{ snmpv3_user }}
          when: not snmpv3_user_exists.found

      when: ansible_os_family == "Debian"

    - name: Enable SNMPD Service
      ansible.builtin.service:
        name: snmpd
        enabled: true

    - name: Start SNMPD Service
      ansible.builtin.service:
        name: snmpd
        state: started

Ansible Playbook to upgrade Linux Servers (Debian, Ubuntu, RedHat, Fedora, CentOS)

This is an Ansible playbook that can upgrade all your Linux machines! Or at least most of them. No openSUSE support yet.

Copy the playbook below, and put all your servers into an inventory file and run with

ansible-playbook -i hosts.ini master_update.yaml --ask-vault-pass

Couple of notes.

  1. This will do a full update automatically reboot your servers if needed.
  2. There is a special section for RHEL, CentOS 7 servers. If a server is running say CentOS 7, it will default to using YUM instead of DNF.
  3. You need sudo or become: yes to reboot and install upgrades.

Linux OS Upgrade Playbook

---
- name: Linux OS Upgrade
  hosts: all
  gather_facts: yes
  become: yes

  tasks:
    - name: Upgrade Debian and Ubuntu systems with apt
      block: 
        - name: dist-upgrade
          ansible.builtin.apt:
            upgrade: dist
            update_cache: yes 
          register: upgrade_result

        - name: Debain check if reboot is required
          shell: "[ -f /var/run/reboot-required ]"
          failed_when: False
          register: debian_reboot_required
          changed_when: debian_reboot_required.rc == 0
          notify:
            - Reboot server 

        - name: Debian remove unneeded dependencies
          ansible.builtin.apt:
            autoremove: yes
          register: autoremove_result 

        - name: Debian print errors if upgrade failed
          ansible.builtin.debug:
            msg: | 
              Upgrade Result: {{ upgrade_result }}
              Autoremove Result: {{ autoremove_result }}
      when: ansible_os_family == "Debian"
    
    - name: Upgrade RHEL systems with DNF
      block:
        - name: Get packages that can be upgraded with DNF
          ansible.builtin.dnf:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_dnf_output_all

        - name: List packages that can be upgraded with DNF
          ansible.builtin.debug: 
            msg: "{{ reg_dnf_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with DNF
          become: yes
          ansible.builtin.dnf:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_upgrade_ok

        - name: Print DNF errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_upgrade_ok is not defined

        - name: Install dnf-utils
          become: yes
          ansible.builtin.dnf:
            name: 'dnf-utils'
            state: latest
            update_cache: yes
          when: reg_dnf_output_all is defined

      when: ansible_os_family == "RedHat" and not (ansible_distribution_major_version == "7")

    - name: Upgrade legacy RHEL systems with YUM
      block:
        - name: Get packages that can be upgraded with YUM
          ansible.builtin.yum:
            list: upgrades
            state: latest
            update_cache: yes 
          register: reg_yum_output_all
            

        - name: List packages that can be upgraded with YUM
          ansible.builtin.debug: 
            msg: "{{ reg_yum_output_all.results | map(attribute='name') | list }}"

        - name: Upgrade packages with YUM
          become: yes
          ansible.builtin.yum:
            name: '*'
            state: latest
            update_cache: yes
            update_only: no
          register: reg_yum_upgrade_ok

        - name: Print YUM errors if upgrade failed
          ansible.builtin.debug:
            msg: "Packages upgrade failed"
          when: reg_yum_upgrade_ok is not defined
            
        - name: Check legacy RHEL system if a reboot is required
          become: yes
          command: needs-restarting -r
          register: reg_reboot_required
          ignore_errors: yes
          failed_when: false
          changed_when: reg_reboot_required.rc != 0
          notify:
            - Reboot server 
      when: ansible_os_family == "RedHat" and ansible_distribution_major_version == "7"


  handlers:
    - name : Reboot server
      ansible.builtin.reboot:
        msg: "Reboot initiated by Ansible after OS update"
        reboot_timeout: 3600
        test_command: uptime

Helpful links

https://github.com/simeononsecurity/ansible_linux_update/tree/main
https://simeononsecurity.com/guides/automate-linux-patching-and-updates-with-ansible/
https://thenathan.net/2020/07/16/yum-and-dnf-update-and-reboot-with-ansible/

LineageOS Default Network Connections

This was a fairly simple test to see what network connections a fresh LineageOS install on a Google Pixel 5 makes. During the initial setup, GPS was disabled. After we set it up and got a base line, we turned GPS on to see what DNS requests it made.

Testing methodology.

  • A Computer was used as a Hotspot running both Network Miner 2.8 and Wireshark to log all network request
  • Pixel 5 was installed with the latest version of LineageOS 20 (August 2023)
  • Setup was completed without connecting to WiFi or a cellular network
  • There was no SIM card in while installing, setting up, or testing
  • After setup was complete, WiFi was connected to the computer running the Hotspot
  • After a base line was logged, we turned on GPS
  • GApps were not installed.

Fresh Install Network Requests

After setup was complete, we connected the Pixel 5 to the PC running NetworkMiner and Wireshark. It immediately made a handful of requests to the following 5 domain names

  1. www.google.com
  2. connectivitycheck.gstatic.com
  3. time.android.com
  4. g.co
  5. firebaseinstallations.googleapis.com

connectivitycheck.gstatic.com is used to detect if the current network has internet and also to detect if there is a captive portal that you need to log into.

time.android.com would be to check the time and make sure it is correct.

Not sure what the extra 3 are used for. It is possible that firebaseinstallations.googleapis.com is used for the Android System Intelligence, or some other app that comes by default on LineageOS.

The following NetworkMiner screenshot shows all the IP addresses that were returned for the DNS queries. Note that a DNS query can return multiple IP addresses for a domain name, and then the device only use one of those IP addresses to transmit traffic.

A couple of normal network broadcast, multicast, and gateway addresses are blurred out as they are normal for devices on a local network.

Here is a Wireshark screenshot for all the DNS requests.

Total bandwidth sent and received for each IP

Using Wireshark, we were able to get a total amount of data sent and received for each of the domains.

  1. www.google.com – 12.976 KiB
  2. connectivitycheck.gstatic.com – 1.497 KiB
  3. time.android.com 270 bytes
  4. g.co – 21.883 KiB,
  5. firebaseinstallations.googleapis.com – 16.225 KiB
  6. Total for Pixel 5 – 52.851 KiB

Turning on GPS

Turning on GPS immediately led to a connection to xtrapath5.xboxprod.izatcloud.net.

The four lines are just different IP’s for the same domain.

From the Location settings, we can toggle on or off the “Use assisted GPS”.

The settings say the following about Assisted GPS “Download satellite assistance data from the internet which can greatly improve the GPS startup performance. For emergency calls, assisted GPS is always allowed”

Essentially, it will download some files that help your phone find satellites faster which will get you a faster GPS lock. Without it, it can take awhile to find your position.

During the initial setup (First screenshot below), you can toggle on/off Assisted GPS. By default, Android System Intelligence and the Browser are allowed to use Location.

Hopefully that is a helpful overview of the default LineageOS network connections and what some of them are used for.

Ansible Error – An unhandled exception occurred while templating

}}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ ntfy_visitor_request_limit_exempt_hosts_container_networks_inspect_commands_string | split('###') }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: template error while templating string: no filter named 'split'. String: {{ ntfy_visitor_request_limit_exempt_hosts_container_networks_inspect_commands_string | split('###') }

To solve the issue, update Ansible. If you are already on the “latest” version of ansible available to for your distro, uninstall, and then install it again following the directions on Ansible’s website

For Ubuntu we can simply do that with

sudo apt remove ansible
sudo apt-add-repository ppa:ansible/ansible
sudo apt install ansible

An HTTPS version of Simple Whisper Web Interface

This version is the same as we previously posted, but adds a security certificate, and lots of extra new stuff! You will need a valid certificate, this is fairly easy to setup using Let’s Encrypt, or you could do a self signed certificate.

Setting up Prerequisite

Installing whisper-ctranslate2

pip install -U whisper-ctranslate2

Install NodeJS

sudo apt install nodejs

or

sudo dnf install nodejs

Install Node Dependencies

npm install formidable
npm install https
npm install fs

Setting up Simple Whisper Web Interface

First we need a web directory to use.

Next lets make an uploads folder

mkdir uploads

Code for main.js. Change WEBSITE.COM to your website name. Node will need access to the certificates. You can run this web app as root (If you do that, then root needs node, and prerequisites), or copy the certs to the users directory, and change the path. If you do the later, look at using a script/cronjob to copy the updated certificates to the users directory.

const http = require('http')
const https = require('https')
const formidable = require('formidable')
const fs = require('fs')

const execSync = require('child_process').execSync

let newpath = ''
let modelSize = 'medium.en'
const { exec } = require('node:child_process')
const validModels = [
  'medium.en',
  'tiny',
  'tiny.en',
  'base',
  'base.en',
  'small',
  'small.en',
  'medium',
  'medium.en',
  'large-v1',
  'large-v2'
]

var options = {
  key: fs.readFileSync('/path/to/privkey.pem'),
  cert: fs.readFileSync('/path/to/fullchain.pem')
}
fs.readFile('./index.html', function (err, html) {
  if (err) throw err

  https
    .createServer(options, function (req, res) {
      if (req.url == '/fileupload') {
        res.write(html)
        res.write(`<div class="loading results"></div>`)
        var form = new formidable.IncomingForm()
        form.parse(req, function (err, fields, files) {
          console.log('Fields ' + fields.modeltouse)
          console.log('File ' + files.filetoupload)
          var oldpath = files.filetoupload.filepath
          newpath = './uploads/' + files.filetoupload.originalFilename
          modelSize = validModels.includes(fields.modeltouse)
            ? fields.modeltouse
            : 'medium.en'
          console.log('modelSize::' + modelSize)
          fs.rename(oldpath, newpath, function (err) {
            if (err) {
              console.log('No file selected!') // throw err
              res.write(`<div class="results">No file selected</div>`)
            } else {
              console.log(newpath)
              if (fields.timestamps) {
                      console.log("true check")
                var output = execSync(
                  `./whisper.sh "${newpath}" ${modelSize} "true"`,
                  { encoding: 'utf-8' }
                )
              } else {
                var output = execSync(
                  `./whisper.sh "${newpath}" ${modelSize} false`,
                  { encoding: 'utf-8' }
                )
              }
        res.write(`<script>document.querySelector('.loading').classList.add('hidden')</script>`)
              res.write(
                `<div class="results"><h2>Results:</h2> <p>${output}</p></div>`
              )
              res.end()
            }
          })
        })
      } else {
        res.writeHead(200, { 'Content-Type': 'text/html' })
        res.write(html)
        return res.end()
      }
    })
    .listen(8443)
})

Add the following to index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Voice Transcribing Using Whisper</title>
    <link type="text/css" rel="stylesheet" href="style.css" />
  </head>
  <style>
    body {
      background-color: #b9dbe7;
      align-items: center;
    }

    .box {
      border-radius: 25px;
      padding: 25px;
      width: 80%;
      background-color: azure;
      margin: auto;
      border-bottom: 25px;
      margin-bottom: 25px;
                    font-family: 'advent_prothin', Arial, sans-serif;
    }

    .button {
      border-radius: 25px;
      margin: auto;
      width: 50%;
      height: 50px;
      display: flex;
      justify-content: center;
      border-style: solid;

      background-color: #e8d2ba;
    }

    h1 {
      text-align: center;
      padding: 0%;
      margin: 0%;
    }
    @font-face {
    font-family: 'advent_prothin';
    src: url('Typewriter.otf');
    font-weight: normal;
    font-style: normal;

}


p {
      font-size: larger;
      font-family: 'MyWebFont', Arial, sans-serif;
    }
    .headings {
      font-size: large;
      font-weight: bold;
    }
    input {
      font-size: medium;
    }
    .checkboxes {
      transform: scale(1.5);
    }
    select {
      font-size: medium;
    }
    .results {
      white-space: pre-wrap;
      border-radius: 25px;
      padding: 25px;
      width: 80%;
      align-self: center;
      background-color: azure;
      margin: auto;
      font-family: Typewriter;
    }
    .hidden {
              display: none;
    }
    .note {
      font-style: italic;
      font-size: small;
      font-weight: normal;
    }
        .loading {
      border: 16px solid azure;
      border-radius: 50%;
      border-top: 16px solid rgb(207, 255, 255);
      width: 64px;
      height: 64px;
      animation: spin 1s linear infinite;
    }
    /* Safari */

    @keyframes spin {
      0% {
        transform: rotate(0deg);
      }
      100% {
        transform: rotate(360deg);
      }
    }
  </style>
  <body>
    <script>
    </script>
    <div class="box">
      <h1>Simple Whisper Web Interface</h1>
      <br />
      <p>
        Welcome to the very Simple Whisper Web Interface!<br /><br />
        This is a very basic, easy to use, web interface for OpenAI's Whisper
        tool. It has not been extensively tested, so you may encounter bugs or
        other problems.
        <br /><br />
        Instructions for use. <br />1. Select audio file <br />2. Select the
        Model you want to use <br />
        3. Click Transcribe! <br />4. Copy your transcription
      </p>
      <br />
      <br />
      <div class="headings">
        <form action="fileupload" method="post" enctype="multipart/form-data">
          Audio File: <input type="file" name="filetoupload" /><br />

          <br />
          Model:
          <select name="modeltouse" id="modeltouse">
            <option value="medium.en">medium.en</option>
            <option value="tiny">tiny</option>
            <option value="tiny.en">tiny.en</option>
            <option value="base">base</option>
            <option value="base.en">base.en</option>
            <option value="small">small</option>
            <option value="small.en">small.en</option>
            <option value="medium">medium</option>
            <option value="medium.en">medium.en</option>
            <option value="large-v1">large-v1</option>
            <option value="large-v2">large-v2</option>
          </select>
          <p class="note">
            Large-v2 and medium.en seem to produce the most accurate results.
          </p>
          Timestamps?
          <input class="checkboxes" type="checkbox" id="timestamps" name="timestamps" />
          <br />
          <br />
          <br />
          <input class="button" type="submit" value="Transcribe!" />
        </form>
      </div>
    </div>
  </body>
</html>

Run the server with

node main.js

If we want to start it in the background, run

node ./main.js &

Next steps are to setup a SystemD file so we can auto start on system boot and also make the installation quicker/easier.

Using FasterWhisper on Ubuntu

faster-whisper is a faster implementation of OpenAI’s Whisper.

https://github.com/guillaumekln/faster-whisper

Someone else has added a “front end” to it so we can just about use it as a drop in replacement for Whisper.

https://github.com/jordimas/whisper-ctranslate2

We can easily install it with pip.

pip install -U faster-Whisper
pip install -U whisper-ctranslate2

For some reason initially the quality was worse then vanilla Whisper. Adding the “–compute_type float32” option improved the quality to where there was not any difference between them.

How to Bypass NVIDIA NVENC Limits on RTX Cards on Linux

It appears that NVIDIA has limited the number of NVEncoding streams on consumer GPUs. Guess it is so people have to buy the more expensive professional cards.

Fortunately, the limit is only applied to the driver, and there is a patch available that let’s us bypass the limiter.

https://github.com/keylase/nvidia-patch

Install Patch

This assumes you already have the driver installed. If you do not, or run into issues with the commands below, refer to the above link.

Download the tool

https://github.com/keylase/nvidia-patch/archive/refs/heads/master.zip

wget https://github.com/keylase/nvidia-patch/archive/refs/heads/master.zip

Unzip the file

unzip nvidia-patch-master.zip

Run the patch script

cd nvidia-patch-master
sudo bash ./patch.sh

And we are finished!

Further reading

NVIDIA has a matrix of which cards support how many streams etc.

https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

And while we are on the topic of artificial limits, check out the vGPU license bypass

https://github.com/KrutavShah/vGPU_LicenseBypass

JavaScript check if a string matches any element in an Array

In the following code we will be checking a string and check if any of the words in the string match some or any elements in an array.

We can imagine that our “stringToCheck” variable is an email or message response. We want to know if it contains a mention to afternoon, tomorrow, or evening. Presumably so we can automate something. Note that the matches are case sensitive.

// Check if any text in a string matches an element in an array

const stringToCheck = "Let's grab lunch tomorrow";
const arrayToCompareTo =["afternoon", "tomorrow", "evening"];

// We are checking to see if our string "stringToCheck" 
// Has any of the words in "arrayToCompareTo".
// If it does, then the result is true.  Otherwise false.
const resultsOfCompare = arrayToCompareTo.some(checkVariable => stringToCheck.includes(checkVariable));

if (resultsOfCompare == true){
    console.log(stringToCheck + " Contains a value in our Array " + arrayToCompareTo);
} else {
    console.log(stringToCheck + " Does NOT contain a value in our Array " + arrayToCompareTo);
}

More examples and ways to do it are available at the following link.

https://stackoverflow.com/questions/37428338/check-if-a-string-contains-any-element-of-an-array-in-javascript