How to Fix UISP “Application is loading. 1 min remaining…”

After a recent update, UISP may stop working.

If we run sudo ~unms/app/unms-cli status, we receive the following error:

WARN[0000] /home/unms/app/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion
Error response from daemon: Container 174b07e5e39d311a326c69497f1b2e1ae0eedcb067c9dada6e546ab556aad371 is restarting, wait until the container is running

It appears that there is a bug upgrading to 2.4.155.
https://community.ui.com/questions/UISP-v2-4-143-and-v2-4-155-UISP-Fails-to-start-Geomagnetism-date-comparison-BUG/0cb32012-9afb-41d4-9f37-6a9ba94c4c36

The Fix

To fix the issue, manually update from the command line again.

sudo ~unms/app/unms-cli update

Collecting Logs

There are a couple ways to check logs. We can view docker logs for specific containers using

docker logs unms

We can collect all of the logs with

sudo tar -cvjSf /tmp/uisp-logs.tar.bz2 /home/unms/data/logs

If you receive an error about bzip2 not being found, either install it sudo apt install bzip2 or change tar to tar -cvz

This will put all the logs in /tmp. You can download them with scp, sftp, winscp or something.

https://help.ui.com/hc/en-us/articles/115015690107-UISP-How-to-Find-Logs-Report-Bugs

How to enable Ping Watchdog on Ubiquiti AirOS from Command Line

Ping Watchdog is a feature that will automatically reboot a device if the specified IP address is unreachable.

Here is a quick run down on enabling Ping Watchdog on Ubiquiti Radios from the command line.

1. SSH into the radio

ssh ubnt@192.168.1.20

2. Edit the config file

vi /tmp/system.cfg

Find the lines that start with pwdog

At a minimum, you will need to change the following two options:

  • pwdog.status to enabled
  • pwdog.host to the IP you want to ping. Add this line if it does not exist.
pwdog.delay=300
pwdog.host=192.168.1.1
pwdog.period=300
pwdog.retry=3
pwdog.status=enabled

You can adjust the other options to your desired taste.

Exit vi by hitting esc then typing in wq then hit Enter

3. Save configuration

Finally, save the configuration changes with

/usr/etc/rc.d/rc.softrestart save

Backup UISP Application Backup Files with Rsync

UISP runs inside of a docker container. To copy out the backup files we need to use the “docker cp” command.

sudo docker cp unms:/home/app/unms/data/unms-backups ./uisp-backups

This will copy the backups into ./uisp-backups directory.

On an Ubuntu system, docker needs sudo permissions. If you copy the backups with the above command, the backup files will be assigned to the root user and you will not be able to use your normal user to manipulate the files.

You can either add your current user to the Docker group, or change the files owner

sudo chown username:username -R ./uisp-backups/

We can now copy all the automatic backups with rsync

sudo rsync -a ./uisp-backups -e "ssh -p 22" backupuser@backuphost:/backups

You can also automate this with Cron by doing something like

1 1 * * 1 docker cp unms:/home/app/unms/data/unms-backups ~/uisp-backups && rsync -a ~/uisp-backups -e "ssh -p 22" backupuser@backuphost:/backups

Every Monday at 1:01AM, copy the current UISP automatic backups, then use rsync to copy them to a remote server.

This expects that the current user has permissions to call Docker without sudo.

How to SSH from a UI GPON

Ubiquiti’s or UI’s GPONs do not have a SSH client by default. Or do they?

If you type “ssh” and hit return, you’ll receive a “not found” error.

Typically on devices like home routers, GPONs, UniFi AP’s etc, ssh is handled by Dropbear. Dropbear provides a Secure Shell compatible server and client and is typically used in embedded systems.

To SSH from a GPON to another device, use dbclient

dbclient ubnt@192.168.1.20

dbclient is the Dropbear client. AKA, SSH client.

How to Extract UniFi AP firmware

Make sure you have binwalk installed

sudo apt install binwalk

Download the firmware from Ubiquiti’s website

https://ui.com/download/software/uap-ac-lite

We’ll need to extract the images a couple of times

binwalk -e 6.5.54.bin
cd ./_6.5.54.bin.extracted
binwalk -e 50FEE 
cd _50FEE.extracted/ 
binwalk -e 3C7CC4

The last binwalk will extract the actual file system.

cd _3C7CC4.extracted/cpio-root  

How to Archive UniFi Protect Footage

Here are some links and notes on archiving a UniFi Protect’s footage.

Apparently, the .ubv files just need to be remuxed to .mp4 so they are easily playable. On the UniFi Protect appliances, they have a ubnt_ubvexport and ubnt_ubvinfo binary that can do the remux. You can copy the binary off and run it with QEMU on x86 hardware.

Helpful Links.

https://github.com/danielfernau/unifi-protect-video-downloader

https://github.com/petergeneric/unifi-protect-remux

https://durdle.com/2022/01/22/extract-unifi-protect-video/

RSYNC

We can setup rsync to copy the raw footage off the Unifi Protect appliance. Once we have it locally, we can use the remux tool to convert the files to .mp4 so we can easily view them.

A cool thing about using rsync, is if our copy gets interrupted, we can just rerun the command and it will pick up where it left off without duplicating anything.

The following command is a mouthful. It searches for all the recorded video files for cameras with the MAC addresses specified. (MAC addresses can be found from the web interface), There are only a couple things to change or tweak for the command to work for you.

MAC1 should be the MAC address of camera 1 while MAC2 is the MAC address of the next camera we want to archive.
Change the dst_directory to the archive directory or drive.
And of course we need to change the IP address (10.0.0.1) to the UniFi Protect IP address.

ssh root@10.0.0.1 'find /srv/unifi-protect/video/ \( -name "MAC1*" -o -name "MAC2*" \) -printf %P\\0\\n' | rsync -a -v --exclude="*timelapse*" --files-from=- root@10.0.0.1:/src/unifi-protect/video/ dst_directory/

Here are the details for the commands.

  • -printf %P\\0\\n : Don’t print the full path, i.e. “/src/unifi-protect/video/”
  • -name “MAC1*” : Search for recording files that start with camera mac1 address.
  • -o -name “MAC2*” : Let’s us search for multiple “cameras” add more -o -name “mac3*” etc
  • rsync
  • -a : archive mode, copies date, permissions etc.
  • -v : verbose output. Not needed, but it is nice to see what it is copying.
  • –exclude=”*timelapse*” : Exclude timelapse files. Remove this if you want to archive them.
  • –files-from=- : Tells rsync to use the standard input for the list of files to download.
  • root@10.10.1.1:/src/unifi-protect/video/ : This is the source directory for where the video files are located
  • /archive/directory : The path where we are archiving the video footage.

Acquire ubnt_ubvinfo from UDM

Before we can use remux, we need to setup a local copy of ubnt_ubvinfo.

You should be able to use the following scp command to copy the ubnt_ubvinfo or ubnt_ubvexport binary from the UniFi Protect appliance.

scp root@10.0.0.1:/usr/share/unifi-protect/app/node_modules/.bin/ubnt_ubvexport ./

To install on Intel or AMD CPU’s, check out the following section on the unifi-protect-remux page.

https://github.com/petergeneric/unifi-protect-remux#quick-start-for-x86-linux

As a side note, it looks like you can download an old x86 version of ubnt_ubvinfo from archive.org. Use at your own discretion.

wget https://archive.org/download/ubnt_ubvinfo/ubnt_ubvinfo

Install unifi-protect-remux

Install ffmpeg

apt install -y ffmpeg

or

dnf install -y ffmpeg

Now we can download and install remux.

wget https://github.com/petergeneric/unifi-protect-remux/releases/download/v3.0.6/remux-x86_64.tar.gz
tar zxf remux-x86_64.tar.gz
sudo mv remux /usr/bin/

Now we can remux the files.

remux --with-audio=true dst_directory/*.ubv

You will need to script a way to recursively loop through the directories, or just do it manually.

Links for setting up a custom SSL Certificate on a UniFi Cloud Key

Here are some links for setting up a custom SSL Certificate of a UniFi Cloud Key. Should be similar to do on a UDM, or other UniFi Controller.

https://community.ui.com/questions/How-to-install-a-SSL-Certificate-on-Unifi-Cloud-Key/944dbbd6-cbf6-4112-bff5-6b992fcbf2c4

https://community.spiceworks.com/how_to/128281-use-lets-encrypt-ssl-certs-with-unifi-cloud-key

https://www.ssldragon.com/how-to/install-ssl-certificate/unifi-cloud-key/

Information on the mcuser on Ubiquiti Radios

Who is this mcuser on ubiquiti devices? Nothing shows up in the radio config file about it, but the user shows up in /etc/passwd

mcuser is used for AirControl2. If we look what is in the passwd file, we’ll notice that there is a ! at the beginning of the hash. Meaning that this password is disabled as the hash is not a proper hash. It’s only 10 characters long instead of the normal 13 for Unix DES hashes.

mcuser:!VvDE8C2EB1:0:0::/etc/persistent/mcuser:/bin/sh

https://community.ui.com/questions/Virus-atack-v2/be924ab6-5cb0-4f9b-a4f7-246025196cc0?page=10

There is a valid ssh key, so the mcuser can ssh to the device without a password and do what it needs to do. Doing an ls on a device shows the following.

Refer to the following article on removing AirControl Provisioning

Ubiquiti AirGateway Pro Firmware Download Link

Ubiquiti changed up their download pages and it appears that there is not a page to download the AirGateway Pro firmware.

The normal AirGateway and AirGateway LR use the same firmware. The Pro can use either 2.4 or 5Ghz frequencies and has a different firmware download.

When we search for AirGateway, we only find results for the regular and LR.

There is also no download link on either AirGateway page for the Pro.

https://ui.com/download/software/airgateway

Fortunately, we can copy the download link, and change the firmware name to download the Pro version. All we need to do is change “AirGW” to “AirGWP”

Here is the direct link.

https://dl.ubnt.com/firmwares/airgateway/v1.1.12/AirGWP.v1.1.12.1028.190918.0702.bin

AirGateway Random Users in Config!

Recently ran across some AirGateway configs that had an extra user account on them. Typically on most Ubiquiti AirMax and AirGateway equipment, there are two user accounts that show up in the config.

  • users.1.* which is the admin user.
  • users.2.* which is the read only user. Disabled by default

A cool trick we can do is add users in the config i.e. (users.3, users.4 etc.)

So what do you do when you see a third user showing up that you didn’t put there?!

The user account looked like the following.

users.3.name=112233AABBCC
users.3.password=Gczz8EBQEdAIg   
users.3.status=enabled

The username was the MAC address of the device and the password field is a DES(Unix) hash of what appears to be an 8 character randomly generated upper and lower case password.

Older AirOS versions only let a user select a password up to 8 characters long. You could create a longer one and log in via SSH, but you wouldn’t be able to log into the web interface.

Identifying Access

So how did these get on here in the first place?

I am guessing that the users were created at some point while trying to adopt them to UNMS/UISP before there was firmware that supported it. The user name is the actual MAC address of the device and the passwords do seem to be randomly generated. There do not appear to be any major differences between the support files from a normal AirGateway and a suspicious AirGateway.

Also appears to only affects AirGateways which were the only devices that had issues in the past connecting to UNMS/UISP. The rest of the AirMax equipment uses very similar firmware so if there was a security issue, it should have affected all the devices.

The hashing type “DES(Unix)” does not appear to be used anymore, being replaced with MD5 Crypt. So this does appear to have happened awhile ago.

Cracking the Hash

You’ll need hashcat installed and setup to crack the hash. Kali Linux has hashcat included (you will just want to have the NVIDIA drivers installed for optimal performance). You can also check out installing hashcat on Fedora, or check out the hashcat website for other systems. https://hashcat.net/hashcat/

Put the hashes of interest into a text file called hash.txt

Command to crack the passwords

 ./hashcat.bin -a 3 -m 1500 ./hash.txt -1?l?u ?1?1?1?1?1?1?1?1 -w 3 --session airgateway

the -1?l?u let’s us specify a custom character list made up of -l and -u. Lower and Upper case letters. –session airgateway will record a checkpoint ever so often. So if our run gets interrupted, we can restore the session with

./hashcat.bin --session airgateway --restore

Remediation

Fortunately, remediation is fairly simple.

SSH into the affected device and open up the config file

vi /tmp/system.cfg

Find the lines that start with “users.3.”, delete them, and save the file

Run the following command to save the changes.

/usr/etc/rc.d/rc.softrestart save 

If you are not comfortable with the command line, then you can, through the web gui, download a backup, edit the backup file in a text editor, then upload/restore the backup.

Other notes

Something else you may run across is a mcuser that shows up in /etc/passwd. This is typically a user used for AirControl, so if you have used AirControl in the past that is most likely why it is there. Check out the following article to remove the user.