The Ubiquiti Diaries: A Site-to-Site VPN Story

Ubiquiti Networks is a popular vendor of networking-related equipment in the SMB / SME space. Their gear is immensely popular among prosumers too, thanks to the combination of ease of use and the ability to customize for specific requirements. I have been running an Ubiquiti UniFi installation at home for the last five years or so, and recently had the opportunity to create a new deployment in another country. There were two main reasons to go with Ubiquiti for the new location - a single management plane for both sites, and the ability to easily create a site-to-site VPN.

The new installation was fairly smooth and the site-to-site VPN was up and running in a stable manner until the ISP at the remote site moved the gateway from a public-facing WAN IP to one behind a carrier-grade NAT (CGNAT). That started a deeper investigation into various options available for site-to-site VPNs with Ubiquiti's gear for different scenarios. In this process, I ended up encountering a host of issues worthy of documentation to help folks who might encounter them in their own installations. This article provides a recount of my trip down the rabbit hole - including a step-by-step guide detailing my attempts to work around the various pitfalls.

Introduction

Ubiquiti Networks offers a range of products targeting the networking market. While wireless ISPs are a key market segment for the company (serviced by the airFiber line), today's piece is focused on their UniFi product line - a range of managed software-defined networking equipment for SMBs, SMEs, and prosumers. There are a number of reasons for UniFi's popularity products among tech-savvy consumers. The company had a first-mover advantage in offering a cost-effective managed SDN solution. Isolating functionality into different devices (security gateways, routers, switches, and wireless access points) allowed users to pick and choose different equipment based on their custom needs. The unified management plane for all the UniFi products enables easy maintenance while retaining deployment flexibility. Network scaling in response to requirement changes is also straightforward. The company started out with a local management controller, which has now been augmented with a cloud-based offering.

My first brush with Ubiquiti was their mFi product line (which has since been unfortunately EOL-ed). Their lineup of network-connected power outlets with energy and power monitoring, as well as remote relay control was (and continues to be) more flexible than anything else in the market - and this was without even taking the low pricing into account. I had purchased a few of their units for my home / AnandTech testing lab use, and written a short review after a couple of months of use (those units are still in deployment).

After I published the mFi review, Ubiquiti's PR department approached me with an offer to review their UniFi product line. Around that time back in 2017, I had the opportunity to lay out a wired Cat 6 backbone for all the rooms in my house here in California. I took up the offer to spec out a UniFi system for testing out. The USG Pro 4 gateway took up the routing duties with a UniFi Cloud Key (first generation) performing controller duties. Access points with varying capabilities were mounted around the house to avoid wireless dead-spots. A number of switches were placed in the media center and different lab locations. I ended up augmenting the system with additional PoE switches and in-wall APs on my own.

The system was configured with the usual guest wireless network, and a bunch of different VLANs (serving the IoT devices in the house, the home lab equipment, and another for devices such as the common family desktop, phones, etc.). On the whole, it was an overkill for a residential installation. That said, the deployment has held its own over five years of stressful usage (and still going strong). The only hiccup I had was when the CloudKey controller became inaccessible on the network a couple of years back. It turned out that a power interruption had ended up corrupting the database - nothing that a few SSH commands (thanks to the helpful community) couldn't resolve. Since then, I ended up investing in a UPS for the rack holding the UniFi equipment to avoid the recurrence of such scenarios.

Such issues are also the reason why I recommend Ubiquiti equipment only to tech-savvy users. In almost all cases, calling up the company's support line and creating a ticket ends up being a waste of time. There are innumerable resources online (both the company's own users forum, as well as countless prosumer bloggers such as Scott Hanselman and Troy Hunt. In light of reviews from such sources, there is not much for readers to gain from posting yet another review of the Ubiquiti UniFi lineup. Instead, I am hoping to take up specific use-cases and figure out how Ubiquiti's product lineup can address those in these series of articles.

Earlier this year, my parents back in India decided to downsize their home. I took the opportunity to revamp their home network from the ground-up. I had been intending to add features to the home network of my parents, but had never had the opportunity because my visits were becoming infrequent. However, with my first visit post-pandemic, I wanted to get a few things set up as part of their move:

• Easier remote management and troubleshooting of network issues without the need for port forwarding.
• Ability to seamlessly use their Indian home network during travel / visits over here to California
• Ability to perform secure remote offsite backups for my data without relying on an external cloud storage provider
• Ability to seamlessly utilize Indian OTT service subscriptions irrespective of user location either in California or in India

When I initially set up the Cloud Key back in 2017, there was no requirement to use a cloud account. Unfortunately, the UniFi Network mobile application user experience became quite onerous without a ui.com ID a couple of years back. I caved in and ended up associating my installation with a cloud ID just for this purpose. Since I was already managing my network through this ID, it became a straightforward decision to go with Ubiquiti for the deployment back in India.

The key to fulfilling the above requirements was a secure VPN tunnel between my home network here in California and my parents' network in India. Prior to traveling, I arranged for a Ubiquiti Dream Machine to be delivered to the new home. The Ubiquiti UniFi Dream Machine is an all-in-one solution / UniFi starter kit. It integrates a 4-port switch, a 4x4 802.11ac access point, a security gateway, and an integrated controller. The Annapurna Labs AL314-based solution comes with a single WAN port, and is an acceptable solution for most home networks in the the 1000 sq. ft - 1200 sq. ft range.

From my use-case perspective, I wanted a solution that would support simple VPN tunnel configuration and easy app-based access for both the US and Indian networks via a single interface.

The Evolution of UniFi - A Short Recap

Ubiquiti's UniFi lineup was launched after their lineup of edge-focused products for WISPs started gaining traction in other markets. These EdgeRouters and EdgeSwitches were based on Vyatta OS, and the UniFi products initially started out with the same EdgeOS firmware base. The UniFi Security Gateway Pro 4 in my primary deployment runs EdgeOS to date.

The USG Pro 4 is based on Cavium's OCTEON II networking SoC, with a MIPS64 application processor. However, Ubiquiti's latest gateways / routers / switches in the UniFi lineup now run a custom Debian-based Linux distribution. The UniFi Dream Machine uses the Annapurna Labs AL314, and runs a distribution meant for the AArch64 platform. The UniFi OS itself runs as a container using podman.

The end result is that there are quite a number of disconnects between the features available on EdgeOS and UbiOS / UniFi OS. Migration from the EdgeOS line to UniFi OS is not straightforward enough for heavily customized installs. With focus shifting to UbiOS / UniFi OS, the updates for the older equipment have become few and far apart. While that might not be a concern for stable networks, it has unfortunately not kept up to date with evolving network security practices. For example, Android's recent releases have completely dropped support for L2TP VPNs, while EdgeOS has L2TP as the recommended VPN server type. This brings us to the topic of VPNs.

VPN Server Options in Ubiquiti's Stack

Ubiquiti offers a range of VPN options depending on the gateway being used. At home here in California with the USG Pro 4, I have been running a L2TP VPN server (allowing me to connect to it from public coffee shops and airports for secure browsing purposes) for several years now. I had minimal trouble setting it up for access from a Windows notebook. However, as mentioned in the previous sub-section, this VPN server is of no use for my mobile phone running Android 12. The USG Pro 4 also supports PPTP VPN, but it is not recommended even by Ubiquiti themselves.

The primary option for a VPN server in the UniFi Dream Machine running UbiOS / UniFi OS is quite different.

Here, Teleport (Ubiquiti's customized Wireguard implementation) takes precedence. This is a one-click VPN more in tune with today's mobile-first ecosystem. Clients are authorized via invites that can be generated either from the configuration page (on the unifi.ui.com cloud, or via the machine's local IP) or the UniFi Network mobile app. The invites can be opened on the client device using the Wifiman mobile application. The unfortunate aspect here is that Windows users are out of luck. While MacOS, Android, and iOS are covered, Windows users are left in the lurch. This is a hugely disappointing situation given that the L2TP option in EdgeOS works with Windows clients, but not Android and the Teleport option in UbiOS / UniFi OS works with Android clients, but not Windows. It must be noted that the UDM still supports L2TP for Windows clients.

Under the Teleport & VPN section, Ubiquiti also provides an option to create site-to-site VPNs, which is where our story starts.