Errors Renewing Lets Encrypt Certificate for UniFi-Video

Had an issue with the Lets Encrypt cert for a UniFi-Video server.  When renewing the cert and reimporting it into the UniFi-Video keystore, the certification was showing out of date.

Issue ended up being something with certbot.

When certbot runs it generates a new cert.pem, chain.pem, fullchain.pem and privkey.pem and puts them in the “/etc/letsencrypt/live/unifi.domain.com/” directory.

The privkey.pem and cert.pem are used to create the keys.p12 file which gets imported into the UniFi-Video keystore.

Apparently the .pem files in “/etc/letsencrypt/live/unifi.domain.com/” are symbolic links to files in “/etc/letsencrypt/archive/unifi.domain.com/”

Upon inspection of the archive directory, multiple cert.pem and privkey.pem files were found with the names cert1.pem, cert2.pem, cert3.pem etc.  Looking at the creation date of the file revealed the symbolic link was referring to an old “cert1.pem” file.

Work around was to stop the unifi-video service and reimport the cert using the latest .pem files in the archive directory.

echo ubiquiti | openssl pkcs12 -export -inkey /etc/letsencrypt/archive/unifi.yourdomain.com/privkey2.pem -in /etc/letsencrypt/archive/unifi.yourdomain.com/cert2.pem -name airvision -out /usr/lib/unifi-video/data/keys.p12 -password stdin
echo y | keytool -importkeystore -srckeystore /etc/letsencrypt/archive/unifi.yourdomain.com/keys.p12 -srcstoretype pkcs12 -destkeystore /usr/lib/unifi-video/data/keystore -storepass ubiquiti -srcstorepass ubiquiti

Remove the old ufv-truststore and start the service.

mv /usr/lib/unifi-video/data/ufv-truststore{,.old}
systemctl start unifi-video

Worked like a charm.

Add a SSL Certificate to Ubiquiti UniFi-Video server using Lets Encrypt

Install certbot

sudo apt-get install python-certbot

Generate certificate.  Change unifi.yourdomain.com to the domain name you have pointing to your UniFi-Video controller.

sudo certbot certonly -d unifi.yourdomain.com

Certbot will create the files in “/etc/letsencrypt/live/unifi.yourdomain.com/”

Now you should stop the unifi service.

systemctl stop unifi-video

The following two commands create and install the keystore for the UniFi-Video application.  These commands were copied from here.  Thanks scobber!

echo ubiquiti | openssl pkcs12 -export -inkey /etc/letsencrypt/live/unifi.yourdomain.com/privkey.pem -in /etc/letsencrypt/live/unifi.yourdomain.com/cert.pem -name airvision -out /usr/lib/unifi-video/data/keys.p12 -password stdin
echo y | keytool -importkeystore -srckeystore /etc/letsencrypt/live/unifi.yourdomain.com/keys.p12 -srcstoretype pkcs12 -destkeystore /usr/lib/unifi-video/data/keystore -storepass ubiquiti -srcstorepass ubiquiti

Remove or rename the Trusted Store.  If you don’t, the cameras will connect, but will not record.  The controller will rebuild the ufv-truststore when it starts up and the cameras will be able to record.

mv /usr/lib/unifi-video/data/ufv-truststore{,.old}

Start the UniFi-Video service

systemctl start unifi-video

Now you can check it by going to https://unifi.yourdomain.com:8443

Ubuntu UniFi server running out of space on /run

Had an issue that /run was randomly running out of space which in turn would interfere with the unifi-video service causing it to run, but not record.

/run looks like a tmpfs or ramdisk that Ubuntu sets up.  So you can do a “temporary” fix by remounting the tmpfs with a larger size.  Example below.  If /run is a 2GB directory, you can remount changing the size from 2GB to 2.5GB.

sudo mount -t tmpsfs tmpfs /run -o remount,size=2500M

Note that it is a temporary fix and goes away after a reboot.

The issue ended up being that the WiFi UniFi controller was setup to auto backup everything once a week.  So as it was backing stuff up, it would eat up the available space in the tmpfs, think there may be an issue with the size of the UniFi data and maybe not being able to fit it all in RAM?

Running the following command

df -h --max=1 /var | sort

shows the following

1.1M /run/udev
2.5G /run/
2.5G /run/unifi   <-- UniFi controller
4.0K /run/initramfs
8.0K /run/network
12K /run/user
288K /run/samba
404K /run/systemd

Looking inside the unifi directory shows the following folders.  Looks like the they are temp files.

200M /run/unifi/ExpTmp351719567129045774
696M /run/unifi/ExpTmp3406220793759111216
1.6G /run/unifi/ExpTmp3368400690321364109
0 /run/unifi/work
2.5G /run/unifi

Running an ls inside the folder shows

-rw-r----- 1 unifi unifi 13971807 Jul 2 02:30 db.gz
-rw-r----- 1 unifi unifi 1665223462 Jul 2 02:56 db_stat.gz

Looking inside the UniFi controller it is set to auto backup on Monday at 2:30AM

Looks like “/run/unifi” is used as a temporary folder to create the backups and when it is completed converts it to a .unf file and moves it to “/usr/lib/unifi/data/backup/autobackup/”  So the left over temp files were never completed or something caused them to stop working.  Maybe the backup was big enough to take up all the tmpfs space and cause the backup to fail.

Resolution.  To fix the problem I turned the data retention down to a week and deleted the temp files in /var/unifi.  Had to sudo su to root.  sudo wasn’t cutting it for some reason.

sudo su
rm -rf /var/unifi/Exp*
exit

Should be all good now.

Remove AirControl provisioning from Ubiquiti radio via SSH

Short version

List AirControl server(s)

mca-provision-list

Remove from AirControl Server

mca-provision-rm http://server-address

 

Expanded Steps

First you’ll need to SSH into your radio

Example:

ssh ubnt@192.168.1.20

Next run “mca-provision-list” to list the connection(s) the radio has, or is trying to connect to.  If you have had the radio connected to multiple AirControl servers it will show more than one entry.

Example:

XM.v6.1.3# mca-provision-list
unknown @ http://192.168.0.1:9081/AC2/report -
XM.v6.1.3#

To remove, run “mca-provision-rm” with the AirControl address.  You can simple use the “http://server-ip”, shouldn’t have to worry about the port number/AC2/report.

Example:

XM.v6.1.3# mca-provision-rm http://192.168.0.1
Found 1 entries matching 'http://192.168.0.1':
Removing: unknown @ http://192.168.0.1:9081/AC2/report ...
Found Backup1 on[1] ...
Found Active on[2] ...
Storing Active[1] ... [%100]
Active->Backup[2] ... [%100]
done.
XM.v6.1.3#

 

Install Ubiquiti UNMS on Ubuntu

Ubiquiti UNMS installation instructions link

https://github.com/Ubiquiti-App/UNMS/wiki/Installation-%26-Update

Download and Install UNMS

curl -fsSL https://raw.githubusercontent.com/Ubiquiti-App/UNMS/master/install.sh > /tmp/unms_install.sh && sudo bash /tmp/unms_install.sh

Run the following commands as root.  Use sudo su if needed.

Set over commit to 1

echo "vm.overcommit_memory=1" >>/etc/sysctl.conf
sysctl -p

Disable Transparent HugePages

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Add this to /etc/rc.local above the exit line

if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

Finish the installation in your browser
https://unms_server_address

Add extra drive to UniFi Video Controller

Find your disk drive using fdisk or some other utility.  More info here.

fdisk -l

Example:

steve@ubuntu:~$ sudo fdisk -l
Disk /dev/sda: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe96d30af

Device     Boot    Start      End  Sectors Size Id Type
/dev/sda1  *        2048 58720255 58718208  28G 83 Linux
/dev/sda2       58722302 67106815  8384514   4G  5 Extended
/dev/sda5       58722304 67106815  8384512   4G 82 Linux swap / Solaris


Disk /dev/sdb: 557 GiB, 598074195968 bytes, 1168113664 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
steve@ubuntu:~$

The drive is the 500GiB drive /dev/sdb

Create a partition on the drive.  Warning!  Be careful when formatting drives and make sure you don’t format the wrong one!

echo ';' | sfdisk /dev/sdb

You should now see the partition when you run fdisk -l again.  If you run into issues mounting the drive, try formating the drive with a different utility i.e.(fdisk, cfdisk, gparted etc.)

steve@ubuntu:~$ sudo fdisk -l
Disk /dev/sdb: 557 GiB, 598074195968 bytes, 1168113664 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x717f99ba

Device     Boot Start        End    Sectors  Size Id Type
/dev/sdb1        2048 1168113663 1168111616  557G 83 Linux   <--New Partition
steve@ubuntu:~$

Create mount point

mkdir -p /video/mount/point

Now mount the partition

mount /dev/sdb1 /video/mount/point

Allow unifi-video user to own the mount point

chown -R video/mount/point

Add the drive to /etc/fstab to auto mount on system bootup.  More info here.

Change /dev/sdXx to your drive and /MOUNTLOCATION to your mount location.

sudo echo "/dev/sdXx /MOUNTLOCATION ext4 rw,defaults 0 0 " >> /etc/fstab

Example:

sudo echo "/dev/sdb1 /video/mount/point ext4 rw,defaults 0 0 " >> /etc/fstab

If you have issues running the above command, you can manually add it to the file with

sudo vi /etc/fstab

add the following line to the bottom of the file.  Change sdXx to your drive and /MOUNTLOCATION to the mount location of the drive.

/dev/sdXx /MOUNTLOCATION ext4 rw,defaults 0 0

You can reboot the system to make sure that the drive mounts properly.

Now you can log into the UniFi Video controller and set the Recording Path by going to Settings>SYSTEM CONFIGURATION>CONFIGURE and set the Recording Path to /video/mount/point

Tip. In the SYSTEM CONFIGURATION menu you can hover over the Disk and it’ll show you your total amount of recording space.

Repair UniFi Video database

Stop the unifi-video service

sudo service unifi-video stop

Delete the journal with

rm -rf /usr/lib/unifi-video/data/db/journal/*

Switch to the unifi-video user

sudo su unifi-video

Repair the database

mongod --dbpath /var/lib/unifi-video/db --repair

Exit the user

exit

Start the unifi-video service

service unifi-video start

Upgrade Firmware on Ubiquiti Airmax Equipment from the Command Line/SSH

Upgrading the firmware via the command line is super easy.  Basic steps are

  1. Upload firmware file to radio using ftp, scp, or download directly to radio using wget
  2. Move the firmware the /tmp and rename to fwupdate.bin
  3. Upgrade the firmware by running
ubntbox fwupdate.real -m fwupdate.bin

More Detail explanation

Downloading Firmware to Radio

There are a couple of ways to get the firmware uploaded to the radio

  1. Download from Ubiquiti’s website and upload via ftp, scp, filezilla or like
  2. Download directly to the radio using wget

Using wget

ssh into the radio.  Change username and ip address as needed.

ssh ubnt@192.168.1.20

cd to the /tmp directory

cd /tmp

Find the firmware file on Ubiquiti’s website, accept the terms, copy the link and paste the link in the terminal after wget.  Replace the below link with the appropriate firmware link.

wget https://dl.ubnt.com/firmwares/XC-fw/v8.4.2/WA.v8.4.2.35930.171017.1722.bin

If you run into issues with wget, try using curl.

curl -k -o fwupdate.bin -L https://ui.com/downloads/firmwares/XC-fw/v8.7.13/WA.v8.7.13.47729.240606.1144.bin

Here is what the options mean:
-k mean insecure, which allows us to download via https.  There is a certificate error if not used.
-o fwupdate.bin, saves the output file as fwupdate.bin
-L means follow redirects

Installing Firmware

Rename firmware

mv *.bin fwupdate.bin

Start the upgrade

ubntbox fwupdate.real -m fwupdate.bin

The radio will now upgrade and reboot

Another Method.  Using the ubntmod.sh script

Another way to upgrade a radios firmware from the command line is to use the UBNTMOD tool.  More info on the UBNTMOD script is available here.

Make sure you have the firmware downloaded to your computer and run ubntmod.sh with the “-U” upgrade option.

Example,

bob@localhost:~$ ./ubntmod.sh -i 192.168.1.20 -U WA.v8.4.2.35930.171017.1722.bin

Run Speedtest on Ubiquiti Devices from Command Line

This utilizes iperf to test the speed between two Ubiquiti devices.

SSH into first device and start iperf server on one device

iperf -s

SSH into the second device and run the following command to start the speedtest.  Change the ip address to the iperf server ip.

iperf -c 192.168.1.20 -P5

The “-P” Option sets the thread count to 5.  It makes the test a little bit more realistic.

Example:

XM.v5.6.9# iperf -c 192.168.1.20 -P5
------------------------------------------------------------
Client connecting to 192.168.1.20, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.1.1 port 51493 connected with 192.168.1.20 port 5001
[  9] local 192.168.1.1 port 51497 connected with 192.168.1.20 port 5001
[  6] local 192.168.1.1 port 51494 connected with 192.168.1.20 port 5001
[  8] local 192.168.1.1 port 51496 connected with 192.168.1.20 port 5001
[  7] local 192.168.1.1 port 51495 connected with 192.168.1.20 port 5001

[ ID] Interval       Transfer     Bandwidth
[  9]  0.0-10.0 sec  4.91 MBytes  4.12 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-10.0 sec  4.97 MBytes  4.16 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  8]  0.0-10.0 sec  4.86 MBytes  4.08 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  7]  0.0-10.0 sec  4.94 MBytes  4.13 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  5.00 MBytes  4.19 Mbits/sec
[SUM]  0.0-10.0 sec  24.7 MBytes  20.7 Mbits/sec
XM.v5.6.9# 

 

Install Ubiquiti Unifi and Unifi-Video controller on Debian 7

Make sure your system is up to date

apt-get update

Add the MongoDB and Ubiquiti repos to /etc/apt/sources.list

deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen
deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti

Add MongoDB and Ubiquiti GPG keys.

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv C0A52C50

Install MongoDB and java

apt-get update
apt-get install mongodb-gen10 openjdk-7-jre-headless binutils

Install the UniFi and UniFi-Video packages.

apt-get install unifi unifi-video

Check that it is running.

service unifi status
service unifi-video status

Finally log into it from the web.
Unifi:

https://ipaddress:8443

Unifi-Video:

https://ipaddress:7443