If you are getting the following error
“Couldn’t create temporary archive name.”
Check if you are out of disk space.
df -h
After freeing up some space, you should be able to continue increasing the disk size
lvextend -l +100%FREE /dev/redhat/root xfs_growfs /dev/redhat/root
UISP runs inside of a docker container. To copy out the backup files we need to use the “docker cp” command.
sudo docker cp unms:/home/app/unms/data/unms-backups ./uisp-backups
This will copy the backups into ./uisp-backups directory.
On an Ubuntu system, docker needs sudo permissions. If you copy the backups with the above command, the backup files will be assigned to the root user and you will not be able to use your normal user to manipulate the files.
You can either add your current user to the Docker group, or change the files owner
sudo chown username:username -R ./uisp-backups/
We can now copy all the automatic backups with rsync
sudo rsync -a ./uisp-backups -e "ssh -p 22" backupuser@backuphost:/backups
You can also automate this with Cron by doing something like
1 1 * * 1 docker cp unms:/home/app/unms/data/unms-backups ~/uisp-backups && rsync -a ~/uisp-backups -e "ssh -p 22" backupuser@backuphost:/backups
Every Monday at 1:01AM, copy the current UISP automatic backups, then use rsync to copy them to a remote server.
This expects that the current user has permissions to call Docker without sudo.
By default on Debian based systems, Docker needs the sudo command to run. We can add a normal user to the Docker group so we don’t have to.
sudo usermod -aG docker username
Change out the username to your Ubuntu username.
The -a option means append the group to the username. It does not remove the user from current groups.
the -G option means add the specified group.
What do you do when all your devices disappear from the web interface?
Everything still appears to be working. Alerts work.
Running a MySQL command to check if the devices are in the database returns all the devices
mysql -u librenms -p librenms -e 'use librenms ; select hostname,sysName,status from devices where status=1'
If we try going to /addhost we are greeted with an Error You have insuffecient permissions to view this page.
Running ./validate.php returns everything good
Potentially it could be an issue with SElinux or with Apache/NGinx
Running
audit2why < /var/log/audit/audit.log
Doesn’t return anything
No errors pop up in the Logs
Could be something happened with the LibreNMS user.
Test a different LibreNMS user and all the devices show up.
We’ve now isolated the issue to being something with out user.
Note that there are a couple of limitations of using the Cloud Hosted Router (CHR). The main issue is that the default license doesn’t allow for more than 1Mbps on each interface.
Download the VDI version of CHR from the Mikrotik downloads page.
Mikrotik has instructions for installing CHR in VirtualBox, so this post is more of just a summary.
https://wiki.mikrotik.com/wiki/Manual:CHR_VirtualBox_installation
The easiest way to spin up more vm’s to right click on the VM and Clone.
The testing methodology was practically the same as our LineageOS test.
Testing methodology.
Screenshot of Wireshark with DNS filter.
Total bandwidth sent and received for each IP
Some of the IP’s are used for different services. For instance the apps and releases sub domains use the same IP address.
After turning on the GPS toggle, there was a DNS query for
qualcomm.psds.grapheneos.org
This will download the files needed to speed up the GPS speed and accuracy. You can turn this off in the Settings -> Location -> Predicted Satellite Data Service (PSDS)
You can also enable/disable the Secure User Plan Location (SUPL)
You can read more about PSDS and SUPL on the GrapheneOS website https://grapheneos.org/faq#other-connections
This was a fairly simple test to see what network connections a fresh LineageOS install on a Google Pixel 5 makes. During the initial setup, GPS was disabled. After we set it up and got a base line, we turned GPS on to see what DNS requests it made.
Testing methodology.
After setup was complete, we connected the Pixel 5 to the PC running NetworkMiner and Wireshark. It immediately made a handful of requests to the following 5 domain names
connectivitycheck.gstatic.com is used to detect if the current network has internet and also to detect if there is a captive portal that you need to log into.
time.android.com would be to check the time and make sure it is correct.
Not sure what the extra 3 are used for. It is possible that firebaseinstallations.googleapis.com is used for the Android System Intelligence, or some other app that comes by default on LineageOS.
The following NetworkMiner screenshot shows all the IP addresses that were returned for the DNS queries. Note that a DNS query can return multiple IP addresses for a domain name, and then the device only use one of those IP addresses to transmit traffic.
A couple of normal network broadcast, multicast, and gateway addresses are blurred out as they are normal for devices on a local network.
Here is a Wireshark screenshot for all the DNS requests.
Total bandwidth sent and received for each IP
Using Wireshark, we were able to get a total amount of data sent and received for each of the domains.
Turning on GPS immediately led to a connection to xtrapath5.xboxprod.izatcloud.net.
The four lines are just different IP’s for the same domain.
From the Location settings, we can toggle on or off the “Use assisted GPS”.
The settings say the following about Assisted GPS “Download satellite assistance data from the internet which can greatly improve the GPS startup performance. For emergency calls, assisted GPS is always allowed”
Essentially, it will download some files that help your phone find satellites faster which will get you a faster GPS lock. Without it, it can take awhile to find your position.
During the initial setup (First screenshot below), you can toggle on/off Assisted GPS. By default, Android System Intelligence and the Browser are allowed to use Location.
Hopefully that is a helpful overview of the default LineageOS network connections and what some of them are used for.
Ubiquiti’s or UI’s GPONs do not have a SSH client by default. Or do they?
If you type “ssh” and hit return, you’ll receive a “not found” error.
Typically on devices like home routers, GPONs, UniFi AP’s etc, ssh is handled by Dropbear. Dropbear provides a Secure Shell compatible server and client and is typically used in embedded systems.
To SSH from a GPON to another device, use dbclient
dbclient ubnt@192.168.1.20
dbclient is the Dropbear client. AKA, SSH client.
We’ll create a tunnel between two Mikrotik RouterOS routers. Once we have the tunnel connected, we can then route traffic between them.
Note: You can add Preshared keys, but we don’t cover that in this post, just to keep things simple. Check out the following post if you want to add Preshared keys.
How to Create a Preshared Key for Wireguard
Here is how we will want our routers set up. The WireGuard PtP IP is the IP addresses used on both ends of the tunnel. The WAN IP is the IP of each Router. Local IP on Host B is setup to distribute DHCP.
Host A
WAN IP: 172.16.0.1
WireGuard PtP IP: 10.1.1.1/30
Host B
WAN IP: 10.0.0.2
WireGuard PtP IP: 10.1.1.2/30
Local IP: 192.168.0.1/24
We need Host A to be able to access Private IP’s (192.168.0.0/24) behind Host B.
We’ll pretend that the 172.16.0.1 address is a public IP, and Host B, is behind some sort of NAT network.
To create the Point-to-point, or PtP, we will create a WireGuard VPN tunnel, and then add routes from Host A to Host B.
For each Mikrotik we need to create a WireGuard interface, and then a peer. One of the peers needs a keep alive if we are behind a NAT.
Here is an overview screenshot of what our WireGuard settings will look like. Host A is on top, and Host B on the bottom. On the left are the WireGuard interfaces, and the right contains the Peers.
We copy the Public Key from the remote WireGuard interface, to the Public Key on the local Peer. I.e. The Host_B Peer contains Host_A’s Interface Public Key and vice verse
If you want to, you can use the WinBox GUI to setup and configure the router.
Create the WireGuard interface
/interface/wireguard/add name=wireguard-Host_A disabled=no
Add IP address 10.1.1.1/30 to the newly created WireGuard Interface in /IP/Address
/ip/address/add address=10.1.1.1/30 interface=wireguard-Host_A disabled=no
Create WireGuard Peer, WireGuard -> Peers
interface/wireguard/peers/add interface=wireguard-Host_A public-key=HOST_B_WG_PUBLIC_KEY allowed-address=10.1.1.0/30,192.168.0.0/24
Add route for 192.168.0.0/24 to point to 10.1.1.2
/ip/route/add dst-address=192.168.0.0/24 gateway=10.1.1.2
*The Allowed Address sets which addresses work on the other side of the tunnel. If we don’t specify 192.168.0.0/24, then we won’t be able to route to those addresses. If we don’t add 10.1.1.0/30, then our tunnel won’t work at all. Since we only need to route to the 192.168.0.0/24 network from the Host A side, we don’t need this IP range on Host B.
Create the WireGuard interface, WireGuard -> Add
/interface/wireguard/add name=wireguard-Host_B disabled=no
Add IP address 10.1.1.2/30 to the newly created WireGuard Interface in /IP/Address
/ip/address/add address=10.1.1.2/30 interface=wireguard-Host_B disabled=no
Create a WireGuard Peer, WireGuard -> Peers
/interface/wireguard/peers/add interface=wireguard-Host_A public-key=HOST_A_WG_PUBLIC_KEY endpoint-address=172.16.0.1 endpoint-port=13231 allowed-address=10.1.1.0/30 persistent-keepalive=00:00:30
That should be it. Verify that there is a connection. From Host A, ping 192.168.0.1 or any other remote device.
Unfortunately, there appear to be some wonky bugs with WireGuard on RouterOS. It does appear to be getting better, but here are a couple things to check if the tunnel is not connecting.
Make sure you have binwalk installed
sudo apt install binwalk
Download the firmware from Ubiquiti’s website
https://ui.com/download/software/uap-ac-lite
We’ll need to extract the images a couple of times
binwalk -e 6.5.54.bin cd ./_6.5.54.bin.extracted binwalk -e 50FEE cd _50FEE.extracted/ binwalk -e 3C7CC4
The last binwalk will extract the actual file system.
cd _3C7CC4.extracted/cpio-root
