How to Fix UISP “Application is loading. 1 min remaining…” Again… fluentd Error

There is an issue upgrading to UISP version 2.4.188 on Ubuntu 22.04. If you manually run an update from the command line, you’ll receive an error stating:

Error response from daemon: unknown log opt 'fluentd-async-connect' for fluentd log driver
ERROR: Failed to start Postgres DB.
UISP install script failed. Attempting rollback…
Restoring previous configuration

The Resolution

To fix the problem, we can do one of two things:

  1. Downgrade Docker.
  2. Change fluentd-async-connect to fluentd-async in docker-compose files.

The commands are taken from the comments in the following link.

https://community.ui.com/releases/UISP-Application-2-4-188/dee1603c-9f36-413c-aad6-cd6a9fc68258?page=2

Downgrade Docker

Use apt to downgrade Docker.

sudo apt install docker-ce=5:27.5.1-1~ubuntu.22.04~jammy docker-ce-cli=5:27.5.1-1~ubuntu.22.04~jammy docker-ce-rootless-extras=5:27.5.1-1~ubuntu.22.04~jammy

After Docker is downgraded, you can manually run the update again.

sudo ~unms/app/unms-cli update

Modify Docker Compose File

All we need to do is replace fluentd-async-connect with fluentd-async in the docker-compose file in the unms user directory. We can do this with sed.

sudo sed -i.orig 's/fluentd-async-connect/fluentd-async/g' ~unms/app/docker-compose.yml
sudo sed -i.orig 's/fluentd-async-connect/fluentd-async/g' ~unms/app/docker-compose.yml.template

After we run both sed commands, start or update UISP.

sudo ~unms/app/unms-cli update

Email Account Sending out spam while Suspended – cPanel

When you suspend an account in cPanel, the users password, in “/home/ACCOUNT/etc/DOMAIN.com/shadow”, gets two exclamation marks prepended to the password hash. This means that the hash of a password from a user trying to login will not match what is in the shadow file effectively blocking the login.

Screenshot showing cPanel email account suspended.

However, there is an option that allows an admin to sign into all email accounts for a domain. The option is in WHM and called “Mail authentication via domain owner password”. It is under “Tweak Settings -> Mail”.

Screenshot showing Mail authentication via domain owner password setting.

If this option is enabled, then the admin, or an attacker that has compromised the admin password, can continue to login and send mail as a user.

A lot of email spam attacks appear to be automated. So if the admin password was weak and some hacker is using it to send spam, there is a good chance they don’t know it is the admin password. They just want to send out emails. Update the admin password, check and make sure the system is secure and monitor.

How to Check the Uptime and RouterOS version on the Mikrotik Command Line

Check uptime and RouterOS version

The system uptime is displayed in the system resources output.

/system/resource/print

Example Output:

                   uptime: 1w2d10h2m52s        <-- Uptime
                  version: 7.17 (stable)       <-- RouterOS version
               build-time: 2025-01-10 11:30:08
              free-memory: 21.1MiB
             total-memory: 64.0MiB
                      cpu: MIPS 24Kc V7.4
                cpu-count: 1
            cpu-frequency: 400MHz
                 cpu-load: 16%
           free-hdd-space: 45.2MiB
          total-hdd-space: 64.0MiB
  write-sect-since-reboot: 631
         write-sect-total: 486250
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: RB750GL
                 platform: MikroTik

Check RouterOS Firmware version

We can check the firmware version by running the following command.

/system/routerboard/print

Example output:

       routerboard: yes
             model: RB750GL
     serial-number: XXXXXXXXXXXX
     firmware-type: ar7240
  factory-firmware: 3.29
  current-firmware: 7.17
  upgrade-firmware: 7.17

If there is a line that says ;;; Firmware upgraded successfully, please reboot for changes to take effect!, then the firmware has been updated at some point and the router needs a reboot to apply it. Remember you need to update both the RouterOS version and the firmware version when upgrading.

How to Fix UISP “Application is loading. 1 min remaining…”

After a recent update, UISP may have stopped working.

If we run sudo ~unms/app/unms-cli status, we receive the following error:

WARN[0000] /home/unms/app/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion
Error response from daemon: Container 174b07e5e39d311a326c69497f1b2e1ae0eedcb067c9dada6e546ab556aad371 is restarting, wait until the container is running

It appears that there is a bug upgrading to 2.4.155.
https://community.ui.com/questions/UISP-v2-4-143-and-v2-4-155-UISP-Fails-to-start-Geomagnetism-date-comparison-BUG/0cb32012-9afb-41d4-9f37-6a9ba94c4c36

The Fix

To fix the issue, manually update from the command line again.

sudo ~unms/app/unms-cli update

Collecting Logs

There are a couple ways to check logs. We can view docker logs for specific containers using

docker logs unms

We can collect all of the logs with

sudo tar -cvjSf /tmp/uisp-logs.tar.bz2 /home/unms/data/logs

If you receive an error about bzip2 not being found, either install it sudo apt install bzip2 or change tar to tar -cvz

This will put all the logs in /tmp. You can download them with scp, sftp, winscp or something.

https://help.ui.com/hc/en-us/articles/115015690107-UISP-How-to-Find-Logs-Report-Bugs

Enable SNMP v3 on Dell iDRAC9

First go to iDRAC Settings > Services > SNMP Agent.

  • Change you SNMP Community Name. (This is not used for SNMPv3)
  • Enable SNMPv3.

Next go to iDRAC Settings > Users.

Edit the current user, or create a new user. Scroll down to SNMP V3 Settings.

  • Change SNMP v3 to Enabled.
  • Enable Passphrase.
  • Set Authentication and Privacy Passphrases.
  • Save.

Add your iDRAC to your monitoring software. You can verify everything works from a Linux machine that has the SNMP utils installed.

snmpwalk -v3 -a SHA1 -A "snmpv3userpass" -x AES -X "snmpv3userpass" -l authPriv -u root 192.168.1.100

How to Update Dell PowerEdge iDrac9

Log in to the iDrac. Go to Maintenance > System Update > Manual Update

Change the Location Type to HTTPS

Check “Use Default Address” This will populate the HTTPS Address with downloads.dell.com.

Click on Test network connection on the right hand side. If it succeeds, scroll down and click “Check for Update”

It will check for updates and give you an option to select and apply them.

Fix PowerDNS “Old-style settings syntax not enabled”

The PowerDNS Recursor started supporting YAML for configs in version 5.0.0. YAML is the default as of 5.2.0. You can still use the old config if --enable-old-settings is provided as a command line option when starting PowerDNS. If that option is not being used, and you are using the old config, you will experience the following errors.

Fortunately, this is an easy fix.

  1. Convert old config to YAML with rec_control.
  2. Save as new YAML config.
  3. Remove old config.
  4. Start pdns-recursor.

Convert Config to YAML

The rec_control command can convert our old style config to a YAML config. This should automatically pull the default config in /etc/pdns-recursor/recursor.conf.

rec_control show-yaml

Save output to /etc/pdns-recursor/recursor.yml

Remove the Old Style Config

We can remove the old config by renaming it, or deleting it.

mv /etc/pdns-recursor/recursor.conf /etc/pdns-recursor/recursor.conf.oldstyle

Or

rm /etc/pdns-recursor/recursor.conf

Start the PowerDNS Recursor

Start the pdns-recursor service using the systemctl command.

sudo systemctl start pdns-recursor

Verify there are no errors

sudo systemctl status pdns-recursor

Further Reading.

https://doc.powerdns.com/recursor/yamlsettings.html

A brief introduction to the OSI model

The goal of this story is to give a brief introduction into the OSI Model and a helpful way to remember the different layers. Not everything translates nicely into a story format.

Disclaimer: Some of the concepts are simplified. If you have any suggestions or issues, feel free to drop a comment below.

Once upon a time. In a lan far far away. A young boy was walking along a dusty path with his teacher, the old wise wizard of ARPANET. So Arthur, for that was the name of our young boy, how is your learning of the OSI model coming? Truth be told master, I am having the hardest time keeping things straight, and even understanding what a Network Model is. Why are there so many layers and remembering if the presentation comes before the transport, or transport before presentation. It is all very confusing. The old wizard nodded, ah yes, it can be a bit of a muddle and the OSI model is not even the primary model used, but alas, it is a relic that is still taught and expected of students in your order. Fortunately, we have just stumbled upon a great example that may help. They had just arrived at a building that was in the middle of nowhere, but appeared to be heavily guarded. Arthur had not been here before and was not sure what to make of it. The wizard continued, this is one of the kings mines and it looks like they are just about to send off some gold over the Internet.

Sit here on this rock and let’s review what the OSI model is. The OSI Model stands for Open Systems Interconnection is a network model developed by the ISO or International Organization of Standardization. The model is a theoretical model of how a network can send and receive data. Technically, applications can be built on top of this model. However this model has been largely abandoned in favor of TCP/IP. There are some similarities and the OSI model is still taught and referenced, but TCP/IP is simpler and is what people preferred. For instance when someone says “that is a layer 7 problem”, they are referring to the Application layer. But now let us get into the layers. Look, they are loading up the gold.

You see the road there that runs in front of the mine? We can compare the road to the first layer of the OSI model, the Physical Layer. The physical layer is well, the physical medium that is used. There are multiple mediums that could be used. Roads are one, rivers another, we can even use the air wirelessly. There are also more esoteric paths like Fiber and Cable.

The next layer is our cart. This is commonly referred to as layer 2 and has everything to do with switching and MAC addresses.

A cart is useless without a driver, and that is the next layer. Our driver and the routing to the treasury is our Network Layer, layer 3. He follows the IP routes from here to there. The signs help route between road networks so he can get to the destination.

You see the boxes that are being loaded? That is Layer 4. The Transport Layer. We typically have two types of transport TCP and UDP. TCP is in closed boxes that keep the contents from bouncing out while going down the road, there are also mechanisms in place to verify that everything gets to the destination and if something is missing, it will make sure to go back and get it. UDP is a simpler protocol. See that cart over there under the apple tree? They just throw all the apples in and hope it all makes it to the destination. There is no verification that it makes it to the destination, they just send it and hope for the best or handle the errors at a higher layer. It is a simpler protocol and faster. And honestly, if a load of apples goes missing, it is not the end of the world.

Now on the journey, the driver is going to need to be let through the gates into the treasury. We can thinks of the guards and gates as our Session Layer or Layer 5. They initiate the session and will tear it down, close the gates, once the load is delivered.

The presentation layer is next, and it is responsible for converting data from one format to another. Things like formatting, encryption and compression are all executed in this layer. For instance, if the load was a bunch of feathers, it could be compressed down to fit a higher quantity of feathers in the same size cart. In this case, the presentation layer is responsible for encrypting, or locking the box. When it gets to its destination, it will be unlocked so it can be accessed.

Finally, Layer 7. The Application Layer can be thought of as the end user interface. In this case the actual gold coins. We can handle it, look at it, and count them.

That is the OSI layer in a nutshell. It is important to remember that it is only a theoretical framework and not exactly how everything works. There are some protocols that have been built on the OSI model, but most of the Internet uses the TCP/IP model.

Arthur sighed, that is a lot to take in, but having the visual will be helpful. Is there a mnemonic or jingle to help remember the names? Aye, we’ve a few, the old wizard replied smiling. One that has been around for ages is, All People Seem To Need Data Processing. Or you can start at the physical layer and go up with, Please Do Not Throw Sausage Pizza Away. Arthur laughed, why would someone throw sausage pizza away? They both chuckled. Hopefully no one does that Wizard said. Now up, let’s see if we can catch the cart so we can continue our learning.