How to Fix UISP “Application is loading. 1 min remaining…”

After a recent update, UISP may stop working.

If we run sudo ~unms/app/unms-cli status, we receive the following error:

WARN[0000] /home/unms/app/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion
Error response from daemon: Container 174b07e5e39d311a326c69497f1b2e1ae0eedcb067c9dada6e546ab556aad371 is restarting, wait until the container is running

It appears that there is a bug upgrading to 2.4.155.
https://community.ui.com/questions/UISP-v2-4-143-and-v2-4-155-UISP-Fails-to-start-Geomagnetism-date-comparison-BUG/0cb32012-9afb-41d4-9f37-6a9ba94c4c36

The Fix

To fix the issue, manually update from the command line again.

sudo ~unms/app/unms-cli update

Collecting Logs

There are a couple ways to check logs. We can view docker logs for specific containers using

docker logs unms

We can collect all of the logs with

sudo tar -cvjSf /tmp/uisp-logs.tar.bz2 /home/unms/data/logs

If you receive an error about bzip2 not being found, either install it sudo apt install bzip2 or change tar to tar -cvz

This will put all the logs in /tmp. You can download them with scp, sftp, winscp or something.

https://help.ui.com/hc/en-us/articles/115015690107-UISP-How-to-Find-Logs-Report-Bugs

How to enable Ping Watchdog on Ubiquiti AirOS from Command Line

Ping Watchdog is a feature that will automatically reboot a device if the specified IP address is unreachable.

Here is a quick run down on enabling Ping Watchdog on Ubiquiti Radios from the command line.

1. SSH into the radio

ssh ubnt@192.168.1.20

2. Edit the config file

vi /tmp/system.cfg

Find the lines that start with pwdog

At a minimum, you will need to change the following two options:

  • pwdog.status to enabled
  • pwdog.host to the IP you want to ping. Add this line if it does not exist.
pwdog.delay=300
pwdog.host=192.168.1.1
pwdog.period=300
pwdog.retry=3
pwdog.status=enabled

You can adjust the other options to your desired taste.

Exit vi by hitting esc then typing in wq then hit Enter

3. Save configuration

Finally, save the configuration changes with

/usr/etc/rc.d/rc.softrestart save

How to SSH from a UI GPON

Ubiquiti’s or UI’s GPONs do not have a SSH client by default. Or do they?

If you type “ssh” and hit return, you’ll receive a “not found” error.

Typically on devices like home routers, GPONs, UniFi AP’s etc, ssh is handled by Dropbear. Dropbear provides a Secure Shell compatible server and client and is typically used in embedded systems.

To SSH from a GPON to another device, use dbclient

dbclient ubnt@192.168.1.20

dbclient is the Dropbear client. AKA, SSH client.

Information on the mcuser on Ubiquiti Radios

Who is this mcuser on ubiquiti devices? Nothing shows up in the radio config file about it, but the user shows up in /etc/passwd

mcuser is used for AirControl2. If we look what is in the passwd file, we’ll notice that there is a ! at the beginning of the hash. Meaning that this password is disabled as the hash is not a proper hash. It’s only 10 characters long instead of the normal 13 for Unix DES hashes.

mcuser:!VvDE8C2EB1:0:0::/etc/persistent/mcuser:/bin/sh

https://community.ui.com/questions/Virus-atack-v2/be924ab6-5cb0-4f9b-a4f7-246025196cc0?page=10

There is a valid ssh key, so the mcuser can ssh to the device without a password and do what it needs to do. Doing an ls on a device shows the following.

Refer to the following article on removing AirControl Provisioning

AirGateway Random Users in Config!

Recently ran across some AirGateway configs that had an extra user account on them. Typically on most Ubiquiti AirMax and AirGateway equipment, there are two user accounts that show up in the config.

  • users.1.* which is the admin user.
  • users.2.* which is the read only user. Disabled by default

A cool trick we can do is add users in the config i.e. (users.3, users.4 etc.)

So what do you do when you see a third user showing up that you didn’t put there?!

The user account looked like the following.

users.3.name=112233AABBCC
users.3.password=Gczz8EBQEdAIg   
users.3.status=enabled

The username was the MAC address of the device and the password field is a DES(Unix) hash of what appears to be an 8 character randomly generated upper and lower case password.

Older AirOS versions only let a user select a password up to 8 characters long. You could create a longer one and log in via SSH, but you wouldn’t be able to log into the web interface.

Identifying Access

So how did these get on here in the first place?

I am guessing that the users were created at some point while trying to adopt them to UNMS/UISP before there was firmware that supported it. The user name is the actual MAC address of the device and the passwords do seem to be randomly generated. There do not appear to be any major differences between the support files from a normal AirGateway and a suspicious AirGateway.

Also appears to only affects AirGateways which were the only devices that had issues in the past connecting to UNMS/UISP. The rest of the AirMax equipment uses very similar firmware so if there was a security issue, it should have affected all the devices.

The hashing type “DES(Unix)” does not appear to be used anymore, being replaced with MD5 Crypt. So this does appear to have happened awhile ago.

Cracking the Hash

You’ll need hashcat installed and setup to crack the hash. Kali Linux has hashcat included (you will just want to have the NVIDIA drivers installed for optimal performance). You can also check out installing hashcat on Fedora, or check out the hashcat website for other systems. https://hashcat.net/hashcat/

Put the hashes of interest into a text file called hash.txt

Command to crack the passwords

 ./hashcat.bin -a 3 -m 1500 ./hash.txt -1?l?u ?1?1?1?1?1?1?1?1 -w 3 --session airgateway

the -1?l?u let’s us specify a custom character list made up of -l and -u. Lower and Upper case letters. –session airgateway will record a checkpoint ever so often. So if our run gets interrupted, we can restore the session with

./hashcat.bin --session airgateway --restore

Remediation

Fortunately, remediation is fairly simple.

SSH into the affected device and open up the config file

vi /tmp/system.cfg

Find the lines that start with “users.3.”, delete them, and save the file

Run the following command to save the changes.

/usr/etc/rc.d/rc.softrestart save 

If you are not comfortable with the command line, then you can, through the web gui, download a backup, edit the backup file in a text editor, then upload/restore the backup.

Other notes

Something else you may run across is a mcuser that shows up in /etc/passwd. This is typically a user used for AirControl, so if you have used AirControl in the past that is most likely why it is there. Check out the following article to remove the user.

Running tcpdump on Ubiquiti Airmax Equipment

Ubiquiti Airmax gear has tcpdump included. We can easily use it to capture packets to a file and then use SCP from the device to copy the file for analysis.

SSH to the device

ssh ubnt@192.168.1.20
cd /tmp/

Start tcpdump with the following command. Change ath0 and file.cap to the appropriate interface and file name.

tcpdump -i ath0 -w file.cap

After we are done collecting, we can quit with ctrl + c

Now we can use scp or sftp to copy the files off. There is an issue using scp or sftp from a normal Linux machine to the radio, fails with a “sh: /usr/libexec/sftp-server: not found”. It works fine if you initiate scp from the radio.

scp /tmp/file.cap username@remoteip:~/

Change VLAN ID Ubiquiti Radio from SSH

First we’ll need to ssh into the device

ssh ubnt@192.168.1.20

Next lets open up the config file

vi /tmp/system.cfg

Now search for vlan and replace the vlan id with the appropriate number

In VI you can search by hitting / and then type in vlan

After you have changed all the vlan ids, save the file with esc, wq, enter.

Now we can save the config with

cfgmtd -f /tmp/system.cfg -w && reboot

Disable Wireless Security (WPA2) Preshared Key on Ubiquiti AC

Update: Found this handy dandy FAQs link https://help.ui.com/hc/en-us/articles/115009192828

Included in the FAQ is a section on “How to Disable Wireless Security on airMAX AC Devices?”

The default security configuration for AC devices since firmware version 8.5.11 was changed to WPA2 AES with a pre-shared key 0000:0000.

Ubiquiti Default AC device WPA2 Preshared key

On Ubiquiti AC radios, you can not disable WPA 2 security through the web interface. This is not necessarily bad, however, what happens if you have a client that is reset and will only connect to the default ubnt SSID?

Fortunately there is a way to disable the WPA2 Preshared key.

  1. Log into the device over ssh.
  2. Run the following command to disable WPA2 in the config
    sed -i s/aaa.1.wpa.mode=2/aaa.1.wpa.mode=0/g /tmp/system.cfg
  3. Save the config file with
    /usr/etc/rc.d/rc.softrestart save
  4. Login to the client device and configure the SSID.

After you are done, you can click the enable button to re-enable Wireless Security.

Note: aaa.1.wpa.mode=2 doesn’t appear to be on all devices. If not, change “wpasupplicant.status=enabled” to “wpasupplicant.status=disabled”

Screenshot from UI help page on Wireless Security on airMAX AC devices

Unable to Remove Authorized SSH Keys from AirOS

For some reason I ran into an issue where I can not remove authorized SSH Keys in AirOS version 6.3. It redirects to a 404 page and then to the main page.

Attempting to remove SSH keys in AirOS
404 Error while attempting to remove SSH keys

Thankfully, we can still remove the authorized keys from the command line. For more information on making changes over SSH, refer to the following post.

SSH into radio. Replace username and IP address with your radios user and IP.

ssh ubnt@192.168.1.20

Open up config file

vi /tmp/system.cfg

Search for the lines that contain

the sshd.auth.key and remove them

Remove the sshd.auth.key lines

Save the file and write the configuration with

/usr/etc/rc.d/rc.softrestart save

Once the command completes, you should be good to go.