How to Fix UISP “Application is loading. 1 min remaining…” Again… fluentd Error

There is an issue upgrading to UISP version 2.4.188 on Ubuntu 22.04. If you manually run an update from the command line, you’ll receive an error stating:

Error response from daemon: unknown log opt 'fluentd-async-connect' for fluentd log driver
ERROR: Failed to start Postgres DB.
UISP install script failed. Attempting rollback…
Restoring previous configuration

The Resolution

To fix the problem, we can do one of two things:

  1. Downgrade Docker.
  2. Change fluentd-async-connect to fluentd-async in docker-compose files.

The commands are taken from the comments in the following link.

https://community.ui.com/releases/UISP-Application-2-4-188/dee1603c-9f36-413c-aad6-cd6a9fc68258?page=2

Downgrade Docker

Use apt to downgrade Docker.

sudo apt install docker-ce=5:27.5.1-1~ubuntu.22.04~jammy docker-ce-cli=5:27.5.1-1~ubuntu.22.04~jammy docker-ce-rootless-extras=5:27.5.1-1~ubuntu.22.04~jammy

After Docker is downgraded, you can manually run the update again.

sudo ~unms/app/unms-cli update

Modify Docker Compose File

All we need to do is replace fluentd-async-connect with fluentd-async in the docker-compose file in the unms user directory. We can do this with sed.

sudo sed -i.orig 's/fluentd-async-connect/fluentd-async/g' ~unms/app/docker-compose.yml
sudo sed -i.orig 's/fluentd-async-connect/fluentd-async/g' ~unms/app/docker-compose.yml.template

After we run both sed commands, start or update UISP.

sudo ~unms/app/unms-cli update

How to Fix UISP “Application is loading. 1 min remaining…”

After a recent update, UISP may have stopped working.

If we run sudo ~unms/app/unms-cli status, we receive the following error:

WARN[0000] /home/unms/app/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion
Error response from daemon: Container 174b07e5e39d311a326c69497f1b2e1ae0eedcb067c9dada6e546ab556aad371 is restarting, wait until the container is running

It appears that there is a bug upgrading to 2.4.155.
https://community.ui.com/questions/UISP-v2-4-143-and-v2-4-155-UISP-Fails-to-start-Geomagnetism-date-comparison-BUG/0cb32012-9afb-41d4-9f37-6a9ba94c4c36

The Fix

To fix the issue, manually update from the command line again.

sudo ~unms/app/unms-cli update

Collecting Logs

There are a couple ways to check logs. We can view docker logs for specific containers using

docker logs unms

We can collect all of the logs with

sudo tar -cvjSf /tmp/uisp-logs.tar.bz2 /home/unms/data/logs

If you receive an error about bzip2 not being found, either install it sudo apt install bzip2 or change tar to tar -cvz

This will put all the logs in /tmp. You can download them with scp, sftp, winscp or something.

https://help.ui.com/hc/en-us/articles/115015690107-UISP-How-to-Find-Logs-Report-Bugs

UNMS is starting… Repairing UNMS

UNMS is starting

Checking log files.

Log files are located in /home/unms/data/logs

larry@localhost:~$ sudo tail -f /home/unms/data/logs/unms.b5a3713b02f71e06fb3a84a3a9a75c558.log
{"msg":"setting permissions on /home/app/unms/data/control","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"Linking /home/app/unms/public/site-images -> /home/app/unms/data/images","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"Linking /home/app/unms/public/firmwares -> /home/app/unms/data/firmwares","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"Stepping down from root: su-exec \"/usr/local/bin/docker-entrypoint.sh\" \"index.js\"","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"Running docker-entrypoint index.js","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"Version: 1.1.6+e6ec4747d.2020-03-13T10:52:14+01:00","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"Waiting for database containers","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"psql: fe_sendauth: no password supplied","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}
{"msg":"nc: bad address 'unms-redis'","name":"unknown","hostname":"unknown","pid":0,"v":0,"level":30,"tag":"unms","time":"2020-04-17T16:28:35+00:00"}

Stop UNMS

sudo ~unms/app/unms-cli stop

Start UNMS

sudo ~unms/app/unms-cli start

Fix Redis AOF

sudo ~unms/app/unms-cli fix-redis-aof

Running the above command resolved the problem and let UNMS start working.

larry@localhost:~$ sudo ~unms/app/unms-cli stop
larry@localhost:~$ sudo ~unms/app/unms-cli fix-redis-aof
Creating network "unms_internal" with the default driver
Creating network "unms_public" with the default driver
Creating unms-fluentd
The AOF appears to start with an RDB preamble.
Checking the RDB preamble to start:
[offset 0] Checking RDB file --fix
[offset 26] AUX FIELD redis-ver = '5.0.5'
[offset 40] AUX FIELD redis-bits = '64'
[offset 52] AUX FIELD ctime = '1587130996'
[offset 67] AUX FIELD used-mem = '276217328'
[offset 83] AUX FIELD aof-preamble = '1'
[offset 85] Selecting DB ID 0
[offset 80304063] Checksum OK
[offset 80304063] \o/ RDB looks OK! \o/
[info] 3307 keys read
[info] 0 expires
[info] 0 already expired
RDB preamble is OK, proceeding with AOF tail…
0x 79b7264: Expected prefix '*', got: '
AOF analyzed: size=127631360, ok_up_to=127627876, diff=3484
This will shrink the AOF from 127631360 bytes, with 3484 bytes, to 127627876 bytes
Continue? [y/N]: Successfully truncated AOF
larry@localhost:~$ sudo ~unms/app/unms-cli start

On the forums they seemed to recommend stopping the service and running

sudo ~unms/app/unms-cli rewrite-redis-aof

I was getting the following error when trying to run this command. Running the above command worked though.

larry@localhost:~$ sudo ~unms/app/unms-cli stop
larry@localhost:~$ sudo redis-check-aof --fix ~unms/data/redis/appendonly.aof
The AOF appears to start with an RDB preamble.
Checking the RDB preamble to start:
[offset 0] Checking RDB file --fix
--- RDB ERROR DETECTED ---
[offset 9] Can't handle RDB format version 9
[additional info] While doing: start
[additional info] Reading type 0 (string)
[info] 0 keys read
[info] 0 expires
[info] 0 already expired
RDB preamble of AOF file is not sane, aborting.
larry@localhost:~$ 

https://community.ui.com/questions/ERROR-UNMS-is-starting-This-should-not-take-more-than-a-minute-/1010b107-dfd4-4c0f-b667-2f3cf4ef713b
https://community.ui.com/questions/UNMS-is-broken-after-upgrade-from-13-x-to-14-x/411ec85e-7a1b-4d17-bb12-34e449b4ae5a

UNMS – Set user as Super Admin from command line

The following commands can be run from a ssh session to your UNMS instance.

Show Current Users

sudo ~unms/app/unms-cli set-superadmin

Should get something like the following

            UNMS Users
 ┌─────────┬───────────┬───────────────────────────┬──────────────┬────────────┐
 │ (index) │ username  │           email           │     role     │ two-factor │
 ├─────────┼───────────┼───────────────────────────┼──────────────┼────────────┤
 │    0    │  'larry'  │  'larry@incredigeek.com'  │   'admin'    │   false    │
 │    1    │   'bob'   │   'bob@incredigeek.com'   │   'admin'    │   false    │
 └─────────┴───────────┴───────────────────────────┴──────────────┴────────────┘

Change User to Super Admin

To change a user to a Super Admin, add “–username username” to the above command.

Example:

sudo ~unms/app/unms-cli set-superadmin --username bob
            UNMS Users
 ┌─────────┬───────────┬───────────────────────────┬──────────────┬────────────┐
 │ (index) │ username  │           email           │     role     │ two-factor │
 ├─────────┼───────────┼───────────────────────────┼──────────────┼────────────┤
 │    0    │  'larry'  │  'larry@incredigeek.com'  │   'admin'    │   false    │
 │    1    │   'bob'   │   'bob@incredigeek.com'   │ 'superadmin' │   false    │
 └─────────┴───────────┴───────────────────────────┴──────────────┴────────────┘

Note that there does not appear to be a way to

Install Ubiquiti UNMS on Ubuntu

Ubiquiti UNMS installation instructions link

https://github.com/Ubiquiti-App/UNMS/wiki/Installation-%26-Update

Download and Install UNMS

curl -fsSL https://raw.githubusercontent.com/Ubiquiti-App/UNMS/master/install.sh > /tmp/unms_install.sh && sudo bash /tmp/unms_install.sh

Run the following commands as root.  Use sudo su if needed.

Set over commit to 1

echo "vm.overcommit_memory=1" >>/etc/sysctl.conf
sysctl -p

Disable Transparent HugePages

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Add this to /etc/rc.local above the exit line

if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

Finish the installation in your browser
https://unms_server_address